Process Reinforcement through Implicit Rewards

ArXiv: 2502.01456

🎯 Pitch

Dense process rewards—ideal for efficient credit assignment—are prohibitively expensive to collect at scale, forcing industry models to default to sparse outcome rewards. PRIME cracks this open by deriving token-level rewards for online RL using only response-level outcome labels, eliminating the need for manual step annotations or dedicated reward model training phases. The result, Eurus-2-7B-PRIME, outperforms Qwen2.5-Math-7B-Instruct on multiple reasoning benchmarks while training on just 10% of the data.

---

1. Executive Summary

This paper proposes PRIME (Process Reinforcement through IMplicit rEwards), a scalable online reinforcement learning framework that generates token-level dense rewards for LLM reasoning using only outcome labels — eliminating the prohibitive cost of collecting step-level process annotations. Training Qwen2.5-Math-7B-Base on competition-level math and coding benchmarks with PRIME yields a 15.1% average improvement over the SFT model and a 6.9% performance gain with 2.5× sample efficiency compared to outcome-reward-only RL, while the resulting model, Eurus-2-7B-PRIME, surpasses Qwen2.5-Math-7B-Instruct on seven reasoning benchmarks using only 10% of its training data. The framework's central insight is that using the SFT model itself to initialize an Implicit Process Reward Model — which computes per-step rewards as the log-ratio between the PRM and a reference policy at each token — and updating this PRM online on policy rollouts fundamentally mitigates reward hacking, establishing that dense process rewards can be scaled to online RL without dedicated reward model training or step-level labels.

2. Context and Motivation

The Core Problem: Dense Process Rewards Are Desirable but Impractical at Scale

The fundamental question this paper tackles is: how do we provide dense, token-level rewards for reinforcement learning of LLMs without collecting prohibitively expensive step-level labels? This matters because dense rewards — feedback at every intermediate step rather than only at the end — have well-established theoretical advantages in reinforcement learning (Sutton & Barto, 2018): they improve training efficiency by providing more frequent learning signals, enable better credit assignment by distinguishing which individual actions in a long trajectory contributed to success versus failure, and reduce the variance of policy gradient estimates. Despite these advantages, the practical deployment of dense rewards in LLM training has been extremely limited.

The gap between theoretical desirability and practical adoption is stark. As the paper notes in Section 1, "successful applications of dense rewards in RL for LLMs are limited (Setlur et al., 2024), as current industry-leading models primarily depend on verifiable outcome rewards and have not yet demonstrated meaningful progress with dense rewards (DeepSeek-AI et al., 2025; Team et al., 2025)." The most prominent example is DeepSeek-R1 (DeepSeek-AI et al., 2025), which explicitly chose to forgo PRMs in large-scale RL training due to the scalability challenges this paper identifies. This is not because dense rewards are theoretically undesirable — quite the opposite — but because the practical barriers to acquiring and maintaining high-quality process-level feedback at scale are formidable.

Why This Gap Is Growing More Urgent

The paper's motivation is amplified by two converging trends in the LLM landscape:

The shift from outcome-supervised to process-supervised reasoning. As LLMs are increasingly applied to complex multi-step reasoning tasks — competition-level mathematics, algorithmic coding, scientific problem-solving — the limitations of outcome-only feedback become more severe. Consider a 50-step mathematical derivation that arrives at an incorrect answer due to a subtle sign error in step 3. An outcome reward model (ORM) can only signal "wrong," providing no information about where the reasoning went off track. The policy receives the same penalty whether it made an arithmetic mistake on step 1 or a conceptual error on step 49. This credit assignment problem (Leike et al., 2018) means the policy gradient signal is diluted across all tokens, and the model may learn spurious correlations — solutions with incorrect processes but correct final answers — rather than genuine reasoning skills. As models tackle problems requiring thousands of reasoning tokens, this sparsity bottleneck becomes increasingly acute.

The emergence of online RL for reasoning. The success of OpenAI o1 (Jaech et al., 2024), DeepSeek-R1 (DeepSeek-AI et al., 2025), and Kimi k1.5 (Team et al., 2025) has demonstrated that large-scale RL with verifiable outcome rewards can dramatically improve reasoning capabilities. However, these results also reveal the ceiling of outcome-only methods. DeepSeek-AI et al. (2025) explicitly discusses the limitations and notes they were "impeded from incorporating PRMs into large-scale RL training" (Section 2.2). This is not a niche concern — it is a first-order bottleneck acknowledged by the leading research groups in the field. As the community pushes toward even harder reasoning problems, the absence of a scalable dense reward mechanism becomes the primary obstacle to further progress.

The data exhaustion horizon. The paper situates its work within a broader historical framing, citing Sutton (2019)'s "bitter lesson" about the primacy of general methods that leverage computation over methods that leverage human knowledge. The paper argues we are "entering a new era of experience, which is exemplified by RL" (Section 7), where the bottleneck is no longer human-generated demonstration data (which fueled SFT-based approaches) but rather the feedback signal that guides RL. If the LLM community is to sustain progress as pretraining data becomes scarce, scalable reward mechanisms — ones that can be computed automatically rather than painstakingly annotated — become essential infrastructure.

Where Existing Approaches Fall Short

The paper identifies three specific, interrelated challenges (C1–C3 in Section 2.2) that collectively explain why dense rewards have not yet been successfully deployed at scale:

C1: Process Rewards Are Hard to Define

The first challenge is definitional. Process rewards require assigning a meaningful correctness score to each intermediate step in a reasoning trajectory, but what constitutes a "step," and what does "correctness" mean for an incomplete derivation?

Concretely, reasoning steps do not naturally occur as discrete, well-segmented units in autoregressive token sequences. A model might express a single logical inference across 15 tokens or 150 — there is no universal segmentation boundary. Prior work on PRMs has typically relied on human annotators to segment solutions into steps and label each as correct or incorrect (Lightman et al., 2023). This approach, while producing high-quality labels, is fundamentally unscalable for online RL: each training iteration generates new rollouts from an updated policy, requiring fresh annotations. The cost and latency of human annotation pipelines make this impossible to sustain over hundreds or thousands of RL steps.

Token-level labeling is even more problematic. There are orders of magnitude more tokens than steps in a typical reasoning trace, and individual tokens rarely have well-defined "correctness." A token like "2" might be correct in one context and incorrect in another — its correctness is only interpretable in the context of the full reasoning path. Moreover, the paper makes a subtle but important point about ambiguity of intermediate correctness: "some incorrect steps can also positively contribute to the final answer by pruning searching branches" (Section 2.2, C1). This aligns with how modern reasoning models work — they explore, backtrack, and eliminate dead ends, meaning a "wrong" step that eliminates a fruitless direction is actually valuable. Assigning binary correctness labels to intermediate steps may fundamentally misrepresent their contribution.

C2: PRM Online Updates Are Not Scalable

Even if process labels could be obtained, a separate but equally critical challenge emerges: the reward model must be updated online alongside the policy. The reason is reward overoptimization, a phenomenon well-documented in the RLHF literature (Gao et al., 2022) but equally applicable here.

When a policy is optimized against a static reward model, it eventually learns to exploit the reward model's blind spots — generating outputs that score highly under the reward model but are actually incorrect (a form of Goodhart's Law). This is not a theoretical edge case; it is empirically observed in the inference-time scaling literature (see the executive summary's discussion of verifier over-optimization in beam search) and is exactly what the DeepSeek-AI team sought to avoid. The principled solution is to update the reward model online on the policy's latest outputs, keeping it calibrated to the shifting distribution (Schulman et al., 2017; Leike et al., 2018).

For outcome reward models, online update is relatively straightforward: collect rollouts, check final answers against ground-truth labels, and update the ORM via standard classification. For process reward models, however, online update requires collecting step-level annotations on every new batch of policy rollouts — because each rollout has potentially different intermediate reasoning, and you need to label whether each step within those rollouts is correct. This multiplies the annotation burden beyond what is feasible. If you need human labels per step, online update is impossible at RL training speeds. If you use automated estimation methods like Monte Carlo rollout (Wang et al., 2023; Kazemnejad et al., 2024) — where you sample 10–16 completions from each intermediate step to estimate its correctness probability — you incur approximately 10× more generation cost than what is already required for policy rollouts alone. At the scale of modern RL training (hundreds of steps, thousands of rollouts per step), this is computationally prohibitive.

C3: Explicit Reward Modeling Brings Extra Development Overhead

Beyond the per-step annotation cost, traditional approaches require a dedicated reward model training phase before RL even begins. This involves:

• Curating a large dataset of prompts and candidate solutions (potentially from multiple model families to ensure coverage)
• Collecting annotations — outcome labels for ORMs, step-level labels for PRMs
• Training a separate reward model (often different from the policy model in architecture or scale)
• Validating the reward model's calibration and coverage

This development overhead is substantial in terms of both computation and engineering effort. For ORMs, the DeepSeekMath approach (Shao et al., 2024) used a dedicated 72B reward model trained on 618K in-house data samples (Table 3). For PRMs, the overhead is even larger: Lightman et al. (2023) relied on an "extensive nuanced step-level annotation" pipeline with human labelers, and even automated PRM training methods like Math-Shepherd (Wang et al., 2023) require generating and scoring multiple rollouts per step. This makes the "time to first RL training step" long and expensive, and it creates a dependency on the quality of the reward model training data: if the reward model was not exposed to a sufficiently broad distribution of solutions, it will be poorly calibrated when the policy explores novel reasoning patterns.

Prior Work's Attempts and Their Limitations

The paper positions PRIME against a landscape of prior approaches, each of which addresses some aspect of the dense reward problem but fails to provide a complete, scalable solution:

Human-annotated PRMs. Lightman et al. (2023) demonstrated that dense process rewards can be more effective than sparse outcome rewards for inference-time verification (selecting the best among multiple sampled solutions). However, their approach relies on human annotators labeling each reasoning step as correct or incorrect — a pipeline that is inherently non-scalable for online RL where new annotations are required each iteration. This work established the value of dense rewards but not a path to deploying them in training.

Automated PRM training via Monte Carlo rollouts. Math-Shepherd (Wang et al., 2023) and VinePPO (Kazemnejad et al., 2024) developed methods to train PRMs without human labels by using the model's own completions from each intermediate step. For each step in a solution, they sample multiple rollouts (completions from that step forward), compute the fraction that reach the correct answer, and use that fraction as a soft correctness label. This eliminates the need for human annotation but introduces a severe computational overhead: estimating the correctness of one step requires generating ~10–16 rollouts. For a solution with 10 steps, this means generating 100–160 rollouts per training example — compared to the 1 rollout needed for outcome-only training. At RL training scale (thousands of prompts per iteration, multiple iterations), this overhead is prohibitive. The paper notes that these methods "require about 10× more rollouts for each step than sampling only the response-level trajectories" (Section 1), making them infeasible for online RL.

Implicit reward models for alignment. DPO (Rafailov et al., 2023) and its variants popularized the idea that language models contain implicit reward functions — the log-ratio between a policy and a reference policy can be interpreted as a reward signal. However, these implicit rewards were used primarily for offline preference optimization (no online RL) and were computed at the outcome level (entire responses compared, not individual tokens). The connection between implicit rewards and dense, token-level process feedback was not established in the alignment literature.

Implicit Process Reward Models. The direct precursor to PRIME is the Implicit PRM proposed by Yuan et al. (2024b). That work showed that a language model fine-tuned to predict outcome correctness (i.e., trained as an ORM) can be repurposed to provide per-token process rewards by computing β log(π_ϕ(y_t | y_<t) / π_ref(y_t | y_<t)) at each timestep. This is a remarkable finding: the same model that was trained only on whole-trajectory correctness signals can, at inference time, produce meaningful per-step scores. Critically, this requires only outcome labels for training — exactly the same kind of labels already available in outcome-verifier-based RL. The paper identifies this property as the key enabler for scalable dense rewards in online RL.

However, the original Implicit PRM work (Yuan et al., 2024b) focused primarily on inference-time verification (selecting among candidate solutions) rather than on training-time RL. The paper does not explore whether Implicit PRMs can be updated online, whether they remain calibrated under the distribution shift induced by policy optimization, or how to integrate their token-level rewards into standard RL advantage estimation frameworks. PRIME fills precisely this gap.

Value models as a partial solution. Some RL algorithms, notably PPO (Schulman et al., 2017), train a separate value model that predicts expected future returns at each state (token), providing a form of dense supervision through the advantage function: A_t = r(y_T) + γV(y_<t+1) - V(y_<t). The value model's predictions could, in principle, mitigate reward sparsity by providing non-zero signals at intermediate timesteps. However, the paper cites previous work (Shao et al., 2024; Ahmadian et al., 2024) showing that "value models may not be able to solve the reward sparsity issue effectively due to training challenges, despite the additional computation overhead" (Section 2.1). The empirical validation of this claim appears in Section 5.4 (Figure 8), where PRIME's process-reward-based advantage estimation substantially outperforms both a standard PPO value model and using the Implicit PRM as a value model. The value model's difficulty stems from the challenge of accurately predicting expected returns in long, complex reasoning trajectories where future outcomes are highly uncertain.

How PRIME Positions Itself Relative to Existing Work

PRIME's contribution is not a fundamentally new idea — process rewards, implicit rewards, and online RL are all established concepts — but rather a synthesis that makes online dense reward RL practical for the first time. The paper's positioning can be understood along several dimensions:

From inference-time to training-time implicit rewards. Yuan et al. (2024b) showed that Implicit PRMs provide meaningful per-step scores at inference time. PRIME asks: what if we take this same mechanism and embed it into the RL training loop? The answer requires solving the online update problem — and this is where the Implicit PRM's key property (outcome-label-only training) becomes transformative. Unlike traditional PRMs that require step labels for each update, the Implicit PRM can be updated online using exactly the same outcome labels already collected for the policy update. This closes the loop: the policy generates rollouts, the outcome verifier grades them, the Implicit PRM updates on those same outcome labels, and the updated PRM provides token-level rewards for the next policy iteration. No additional annotation pipeline is needed.

Eliminating the dedicated reward model phase (C3). The paper makes a surprising empirical finding: the SFT model itself, without any reward model training, serves as a "decent initialization of PRM" (Section 3.3). Because the Implicit PRM is just a language model (same architecture as the policy), it can be initialized from the same checkpoint. The paper shows that this outperforms a dedicatedly trained PRM (EurusPRM, trained on 500K additional samples) because starting from the same distribution as the policy eliminates initial distribution shift. This means the entire pipeline — SFT, PRM training, RL — collapses into a single model lineage: start from the base model, SFT it, then use the SFT model as both the initial policy and the initial PRM, and update both online. The development overhead (C3) essentially vanishes.

Compatibility with diverse RL algorithms. The paper is careful to position PRIME not as a new RL algorithm but as a reward mechanism that can be plugged into existing advantage estimation frameworks. Section 3.2 shows how implicit process rewards combine with outcome rewards in the advantage calculation of RLOO, REINFORCE, GRPO, and PPO. The formula (Equation 5) is additive: compute the return from process rewards (with leave-one-out baseline) and add it to the return from outcome rewards. This generality is important because it means practitioners do not need to adopt a specific RL framework to benefit from dense rewards — PRIME can be layered onto whatever advantage estimator they already use.

Addressing the definitional problem (C1) through token-level granularity. Rather than segmenting solutions into steps and assigning step-level scores, the Implicit PRM produces a reward for every token. This sidesteps the problem of defining where steps begin and end, and it avoids the ambiguity of whether a "wrong" intermediate step is nonetheless productive. The token-level rewards naturally capture the notion that some tokens contribute more to the final outcome than others, without requiring binary correctness judgments at intermediate points. Tokens that tend to appear in trajectories leading to correct answers receive positive rewards; tokens that tend to appear in trajectories leading to incorrect answers receive negative rewards. The magnitude of the reward reflects the strength of this association, learned statistically from outcome labels across many rollouts.

Acknowledging and addressing the fundamental tension. The paper is explicit about the central paradox it resolves: dense rewards are theoretically desirable for RL but practically infeasible to obtain at scale. The resolution comes from recognizing that if the reward model can be expressed as a function of outcome-labeled data alone, then it can be updated online using only the data already collected for policy updates. The Implicit PRM parameterization makes this possible: the reward at each token is derived from the PRM's token-level log-probability ratio, and the PRM itself is trained on whole-trajectory outcome labels via cross-entropy loss (Equation in Algorithm 1, step 8). No step labels appear anywhere in the pipeline.

This framing positions PRIME as an enabling technology rather than a point improvement. It doesn't just claim to improve sample efficiency — it claims to make a previously impractical class of methods (online dense-reward RL) practical at all. The paper's subtitle, "Process Reinforcement through Implicit Rewards," captures this: the word "through" indicates that implicit rewards are the mechanism by which process-level reinforcement becomes feasible.

3. Technical Approach

This is primarily a systems and methods paper whose core idea is that dense process rewards for online RL can be made scalable by deriving them from an Implicit Process Reward Model — a language model trained on outcome labels alone that produces token-level rewards through a log-ratio parameterization, eliminating the need for step-level annotations while still enabling online PRM updates to prevent reward hacking.

3.1 Reader Orientation

PRIME is a reinforcement learning framework that automatically generates dense, token-level reward signals for every step of an LLM's reasoning process during training, using only the final-answer correctness label — no human step annotations, no expensive Monte Carlo step rollouts, and no dedicated reward model training phase. The system solves the problem of scalable dense rewards for RL: previously, providing per-step feedback required either expensive human labeling (Lightman et al., 2023) or generating ~10× more rollouts to estimate step quality (Wang et al., 2023), making online RL infeasible; PRIME resolves this by recognizing that the SFT model itself, updated online on outcome labels only, can serve as a token-level reward model through a specific mathematical parameterization — the log-ratio between two language model probabilities — making the entire pipeline lightweight, update-efficient, and compatible with any policy-gradient RL algorithm.

3.2 Big-Picture Architecture (Diagram in Words)

The PRIME system has five interconnected components operating in a single training loop, all initialized from the same model checkpoint:

1. Policy Model ($\pi_\theta$) — the LLM being trained to reason; generates multiple candidate solutions per prompt and is updated via policy gradient using advantage estimates computed from the combined process-and-outcome rewards.

2. Outcome Verifier ($r_o$) — a rule-based function that grades complete solutions as correct or incorrect using ground-truth answers (for math: string match on boxed LaTeX expressions; for code: fraction of test cases passed). This is the only source of ground-truth supervision in the entire pipeline.

3. Implicit Process Reward Model ($\pi_\phi$) — a language model initialized from the same checkpoint as the policy, updated online on outcome-labeled rollouts via cross-entropy loss, and used to produce token-level dense rewards through the formula $r_\phi(y_t) = \beta \log \frac{\pi_\phi(y_t|y_{<t})}{\pi_{\text{ref}}(y_t|y_{<t})}$.

4. Reference Model ($\pi_{\text{ref}}$) — a frozen copy of the initial SFT model (or the old policy parameters) that provides a baseline probability distribution against which the PRM's token probabilities are compared to compute rewards.

5. Advantage Estimator and Policy Updater — a module that combines the token-level process rewards $r_\phi$ with the trajectory-level outcome reward $r_o$ into a per-timestep advantage $A_t$ using a leave-one-out baseline, then applies a PPO-clipped policy gradient to update $\pi_\theta$.

Information flows cyclically: prompts are sampled → the policy generates K responses (default K=4) → the outcome verifier checks each response against ground truth → the Implicit PRM is updated on these (prompt, response, outcome_label) tuples via cross-entropy → the updated PRM and reference model compute token-level process rewards → these combine with outcome rewards to compute advantages → the policy is updated via PPO loss → the cycle repeats.

3.3 Roadmap for the Deep Dive

• First, the Implicit Process Reward Model (Equation 3) — the mathematical parameterization that enables training a reward model on outcome labels while extracting per-token rewards at inference time, which is the core enabler of the entire framework.
• Second, the Implicit PRM training objective (the cross-entropy loss in Algorithm 1) — how the PRM is updated online using only outcome-labeled rollouts, and why this specific parameterization means no step labels are ever needed.
• Third, the advantage estimation formula (Equation 5) — how token-level process rewards and trajectory-level outcome rewards are combined with a leave-one-out baseline to produce a per-timestep advantage signal, and why they are computed separately before summing.
• Fourth, the policy update objective (Equation 6) — the PPO-clipped surrogate loss that stabilizes policy updates by preventing the new policy from diverging too far from the old one.
• Fifth, the auxiliary techniques — SFT-model initialization of the PRM, online prompt filtering, and the SFT warmup stage — which together reduce distribution shift, stabilize training, and eliminate the dedicated reward model training phase that prior work required.

3.4 Detailed, Sentence-Based Technical Breakdown

The Implicit Process Reward Model: Obtaining Per-Token Rewards from Outcome-Labeled Training

The central technical mechanism that makes PRIME possible is the Implicit Process Reward Model (Implicit PRM) parameterization, introduced in Yuan et al. (2024b) and repurposed here for online RL training rather than inference-time verification. To understand why this parameterization is powerful, we need to start with what a traditional process reward model does and why it is expensive to train.

A conventional PRM takes a partial solution $(x, y_{<t})$ and outputs a scalar score indicating how likely the final answer will be correct if the model continues reasoning from this point. Training such a model typically requires step-level supervision: for each intermediate step in training solutions, we need a label indicating whether that step is correct. Collecting these labels is the bottleneck — either through human annotation (Lightman et al., 2023) or through Monte Carlo rollouts where we sample many completions from each intermediate step and check what fraction reach the correct answer (Wang et al., 2023). Both approaches are too expensive for online RL, where the policy generates new rollouts at every training iteration and the PRM must be updated on these new distributions.

The Implicit PRM takes a fundamentally different approach. It is trained exactly like an outcome reward model (ORM) — it receives a full solution and a binary outcome label (correct or incorrect) — but its internal parameterization allows it to be decomposed into per-token contributions at inference time. The key insight is a specific form for representing the reward:

$r_\phi(y) = \beta \log \frac{\pi_\phi(y)}{\pi_{\text{ref}}(y)}$

where $\pi_\phi$ is the PRM (a causal language model), $\pi_{\text{ref}}$ is a frozen reference model (also a language model, typically the initial SFT checkpoint), $y$ is the complete generated response, and $\beta > 0$ is a temperature-like scaling hyperparameter (set to 0.05 in all PRIME experiments).

What this states: the total reward for a complete solution is proportional to the log-ratio between the PRM's token-averaged probability of generating that solution and the reference model's probability. If $\pi_\phi$ assigns higher probability to a response than $\pi_{\text{ref}}$ does, the reward is positive; if lower, the reward is negative.

Why this form matters for scalability: because both $\pi_\phi$ and $\pi_{\text{ref}}$ are autoregressive language models, the joint probability decomposes as a product of conditional probabilities:

$\beta \log \frac{\pi_\phi(y)}{\pi_{\text{ref}}(y)} = \beta \log \frac{\prod_{t=1}^T \pi_\phi(y_t | y_{<t})}{\prod_{t=1}^T \pi_{\text{ref}}(y_t | y_{<t})} = \beta \sum_{t=1}^T \log \frac{\pi_\phi(y_t | y_{<t})}{\pi_{\text{ref}}(y_t | y_{<t})}$

This decomposition reveals that the total reward is a sum of per-token contributions. Therefore, we can define the implicit process reward at each token $y_t$ as:

$r_\phi(y_t) = \beta \log \frac{\pi_\phi(y_t | y_{<t})}{\pi_{\text{ref}}(y_t | y_{<t})}$

where $r_\phi(y_t)$ is the scalar reward assigned to token $t$ in the response, $\pi_\phi(y_t | y_{<t})$ is the PRM's predicted probability of token $y_t$ given the prefix $y_{<t}$, and $\pi_{\text{ref}}(y_t | y_{<t})$ is the reference model's predicted probability of that same token given the same prefix.

What it computes operationally: to obtain the process reward at token $t$, we (1) run a forward pass of the PRM on the prefix $y_{<t}$ to get its predicted log-probability for the actual next token $y_t$, (2) run the same forward pass through the frozen reference model to get its log-probability for $y_t$, (3) subtract and multiply by $\beta$. This is computationally cheap — two forward passes over the complete sequence give us all token-level rewards at once, since each model's forward pass naturally produces per-token log-probabilities as a byproduct of autoregressive generation.

Why this form works: Yuan et al. (2024b) proved that this parameterization arises naturally from a specific formulation of reward modeling where the reward is represented as $r_\phi(y) = \beta \log \frac{\pi_\phi(y)}{\pi_{\text{ref}}(y)}$ and $\pi_\phi$ is optimized to maximize the likelihood of outcome labels. Under this parameterization, the Q-value — the expected cumulative reward from token $t$ onward — has a closed-form solution:

$q^\phi_t(y_{<t}, y_t) = \sum_{i=1}^t \beta \log \frac{\pi_\phi(y_i | y_{<i})}{\pi_{\text{ref}}(y_i | y_{<i})}$

This means the cumulative reward up to any point in the sequence can be computed exactly (not estimated) using only the token-level log-ratios. There is no need for Monte Carlo estimation or bootstrapping — the Q-value is an analytic function of the PRM's and reference model's token probabilities.

What this enables that was previously impossible: because $\pi_\phi$ is trained on whole-trajectory outcome labels (Algorithm 1, line 8: straightforward cross-entropy between $\sigma(r_\phi(y))$ and the binary outcome label $r_o(y)$), andyet its per-token decomposition produces meaningful token-level rewards, we obtain dense process rewards at training time from outcome-only supervision. This is the property that makes online PRM updates scalable: when the policy generates new rollouts, we only need the final-answer correctness (which the outcome verifier provides automatically) to update the PRM, and the updated PRM then provides fresh token-level rewards for the next policy iteration. No step labels, no extra rollouts — just the data already collected for the policy update.

A critical detail: the reference model $\pi_{\text{ref}}$ is typically set to the initial SFT model and kept frozen throughout training (Section B.1 investigates an alternative where the policy's old log-probabilities serve as reference, with similar results). This provides a stable baseline: a token that the PRM $\pi_\phi$ assigns higher probability than the initial SFT model did is considered "rewarded," while one assigned lower probability is "penalized." The $\beta = 0.05$ scaling keeps the rewards numerically well-behaved — without it, log-ratios could span large ranges, causing instability in the advantage computation.

Training the Implicit PRM Online: The Cross-Entropy Objective

The Implicit PRM $\pi_\phi$ is updated at each RL iteration using only the outcome-labeled rollouts collected for the policy update. The training objective is standard binary cross-entropy applied to the full-trajectory reward:

$L_{\text{CE}}(\phi) = -\mathbb{E}_{(x, y, r_o(y)) \sim \mathcal{T}} \left[ r_o(y) \cdot \log \sigma(r_\phi(y)) + (1 - r_o(y)) \cdot \log(1 - \sigma(r_\phi(y))) \right]$

where $(x, y, r_o(y))$ is a triple of prompt, generated response, and outcome label sampled from the rollout buffer $\mathcal{T}$, $r_o(y) \in \{0, 1\}$ is the binary outcome from the rule-based verifier (1 if the final answer matches ground truth, 0 otherwise), $r_\phi(y) = \beta \log \frac{\pi_\phi(y)}{\pi_{\text{ref}}(y)}$ is the PRM's predicted scalar reward for the full sequence, $\sigma(\cdot)$ is the sigmoid function that squashes the reward into a $[0,1]$ probability, and $\phi$ represents the trainable parameters of the PRM.

What it computes: the standard binary cross-entropy between the PRM's predicted correctness probability $\sigma(r_\phi(y))$ and the ground-truth binary outcome $r_o(y)$. When the response is correct ($r_o(y) = 1$), the loss is $-\log \sigma(r_\phi(y))$, which pushes the PRM to assign high reward to this response (i.e., make $\pi_\phi(y)$ much larger than $\pi_{\text{ref}}(y)$). When the response is incorrect ($r_o(y) = 0$), the loss is $-\log(1 - \sigma(r_\phi(y)))$, pushing the PRM to assign low reward (making $\pi_\phi(y)$ much smaller than $\pi_{\text{ref}}(y)$).

Why this form: the cross-entropy loss is the maximum-likelihood objective for binary classification and is the standard choice for training reward models. However, a subtlety arises because the Implicit PRM parameterization $r_\phi(y) = \beta \log \frac{\pi_\phi(y)}{\pi_{\text{ref}}(y)}$ uses a self-normalized probability ratio. The paper notes (Appendix C.2) that with cross-entropy loss, the optimal solution satisfies $\beta \log \frac{\pi^*_\phi(y)}{\pi_{\text{ref}}(y)} = r_o + c$ where $c$ is an arbitrary constant shift, rather than $= r_o$ exactly. This is because language model probabilities are inherently normalized, so the cross-entropy loss cannot push the ratio to match the raw outcome label exactly — there is always a constant offset.

Why this offset does not matter: PRIME uses relative rewards in its advantage computation. Specifically, the process reward at each token is computed as $r_\phi(y^i_t) - \frac{1}{K-1}\sum_{j \neq i} r_\phi(y^j)$, where the baseline is the mean reward across the $K-1$ other responses for the same prompt. Any constant offset $c$ that depends only on the prompt (not on the response) cancels out in this difference, because $(r_\phi(y^i) + c) - \frac{1}{K-1}\sum_{j \neq i}(r_\phi(y^j) + c) = r_\phi(y^i) - \frac{1}{K-1}\sum_{j \neq i}r_\phi(y^j)$. This means the PRM does not need to learn the exact outcome label value — it only needs to learn the relative ordering of correct vs. incorrect responses, which is a strictly easier optimization problem.

The paper also experimented with using DPO loss instead of cross-entropy (Appendix C.2, Table 8) and found "DPO and CE achieve similar results," with CE chosen for memory efficiency. This is a practically important finding: the simpler loss works just as well, reducing implementation complexity.

The PRM update uses a separate optimizer from the policy, with learning rate $10^{-6}$ for the PRM compared to $5 \times 10^{-7}$ for the policy, both using AdamW. The higher learning rate for the PRM makes sense: the PRM needs to adapt quickly to the shifting policy distribution to provide meaningful reward signals, while the policy needs more conservative updates to avoid catastrophic forgetting of the SFT-initialized reasoning patterns.

Advantage Estimation: Combining Process and Outcome Rewards

After obtaining token-level process rewards from the updated Implicit PRM, the next step is to convert these into advantages — per-timestep signals that tell the policy how much better or worse each action was compared to what was expected. PRIME's advantage formula (Equation 5) is a two-part construction:

$A^i_t = \underbrace{\sum_{s=t}^{|y^i|} \gamma^{s-t} \cdot \left[ r_\phi(y^i_s) - \frac{1}{K-1} \sum_{j \neq i} \overline{r_\phi(y^j)} \right]}_{\text{RLOO with implicit process rewards}} + \underbrace{\left[ r_o(y^i) - \frac{1}{K-1} \sum_{j \neq i} r_o(y^j) \right]}_{\text{RLOO with outcome rewards}}$

where $A^i_t$ is the advantage of the $t$-th token in the $i$-th response (out of $K$ responses sampled for the same prompt), $r_\phi(y^i_s)$ is the implicit process reward at token $s$ of response $i$, $\overline{r_\phi(y^j)}$ is the average process reward over all tokens in response $j$ (used as the baseline rather than per-token baselines), $r_o(y^i) \in \{0,1\}$ is the outcome reward for response $i$, $\gamma$ is a discount factor (set to 1.0 in all experiments unless otherwise noted), and $K$ is the number of responses sampled per prompt (default 4).

What it computes operationally: for each response $i$ and each timestep $t$, the advantage is the sum of two terms:

1. Process reward term: Starting from the current token $t$ and going to the end of the response, accumulate the deviation of each token's process reward from the average process reward of the $K-1$ other responses. Specifically, for each future token $s \geq t$, compute $r_\phi(y^i_s)$ (the reward for that specific token in this response), subtract the mean process reward computed over the other $K-1$ responses (each averaged over all its tokens), and discount by $\gamma^{s-t}$. This term answers: "how much better or worse were the tokens in this response from this point onward, compared to the alternative responses the model generated?"

2. Outcome reward term: Simply compute $r_o(y^i)$ (1 if this response's final answer is correct, 0 otherwise), subtract the leave-one-out mean outcome reward of the other $K-1$ responses. This term answers: "how much better or worse was the final outcome of this response compared to the alternatives?"

The two terms are added to produce the final advantage at each timestep.

Why the process reward baseline uses the average over entire other responses ($\overline{r_\phi(y^j)}$) rather than a per-token baseline: the paper does this for computational and statistical stability. Computing a per-token leave-one-out baseline (where each token is compared against the corresponding token in other responses) would require aligning tokens across responses, which is ill-defined since different responses have different lengths and token sequences. Averaging over the full response gives a single scalar baseline per alternative response that captures the overall "quality" signal.

Why process and outcome rewards are computed separately before summing, rather than mixed at the token level: the paper states this is to avoid "numerical instability" (Section 3.2). The outcome reward is binary (0 or 1) while process rewards are continuous real numbers that can vary in magnitude. Computing their statistics separately (separate baselines, separate summations) and then adding the final terms keeps each component well-behaved. This approach was also used in DeepSeekMath (Shao et al., 2024), which the paper cites as precedent.

Why the discount factor $\gamma$ is set to 1.0: in reasoning tasks with a verifiable final answer, there is no natural time-preference — a correct step at position 5 is just as valuable as a correct step at position 5,000, because both contribute to the single binary outcome. Setting $\gamma = 1$ means future process rewards are not discounted, which is appropriate when the only thing that ultimately matters is the final answer. The paper notes that this also allows the formulation to be viewed through the lens of potential-based reward shaping (Appendix C.3), where the Q-value acts as a potential function and the process reward $r_\phi(y_t) = q^\phi_t - q^\phi_{t-1}$ is a shaping reward that does not change the optimal policy but speeds up learning.

The choice of leave-one-out baseline (RLOO; Ahmadian et al., 2024) over alternatives (REINFORCE's simple reward, GRPO's group normalization, PPO's learned value function) is empirically motivated by the results in Section 5.3 (Figure 7 and Table 4). Among all advantage estimators tested, RLOO with PRIME achieved the highest average accuracy (41.0% vs. 37.8% for REINFORCE+PRIME and GRPO+PRIME, and 39.4% for PPO+PRIME). The leave-one-out baseline has the property of being unbiased (the expected value of the baseline equals the expected reward) while reducing variance relative to using raw rewards, without requiring a separately trained value model.

Policy Update: PPO-Clipped Surrogate Loss

The policy model $\pi_\theta$ is updated using a PPO-style clipped surrogate objective (Schulman et al., 2017):

$L^{\text{CLIP}}(\theta) = \mathbb{E}_t \left[ \min \left( \frac{\pi_\theta(y_t | y_{<t})}{\pi_{\theta_{\text{old}}}(y_t | y_{<t})} A_t, \text{clip}\left( \frac{\pi_\theta(y_t | y_{<t})}{\pi_{\theta_{\text{old}}}(y_t | y_{<t})}, 1 - \epsilon, 1 + \epsilon \right) A_t \right) \right]$

where $\pi_\theta(y_t | y_{<t})$ is the current policy's probability of token $y_t$ given prefix $y_{<t}$, $\pi_{\theta_{\text{old}}}(y_t | y_{<t})$ is the old policy's probability (from before the current update step) for the same token and prefix, $A_t$ is the advantage computed from Equation 5, $\epsilon$ is the clipping parameter (typically 0.2 in standard PPO; the paper does not specify a different value, suggesting the default), and $\frac{\pi_\theta(y_t|y_{<t})}{\pi_{\theta_{\text{old}}}(y_t|y_{<t})}$ is the importance sampling ratio that measures how much more or less likely the current policy is to generate this token compared to the old policy.

What it computes: for each token in each generated response, compute the ratio between the new policy's probability and the old policy's probability. Multiply this ratio by the advantage $A_t$ — this is the standard policy gradient objective that increases the probability of tokens with positive advantage and decreases the probability of tokens with negative advantage. However, the ratio is clipped to the interval $[1-\epsilon, 1+\epsilon]$, and the loss takes the minimum of the clipped and unclipped versions.

Specifically: if $A_t > 0$ (the token was better than expected), the loss encourages increasing $\pi_\theta(y_t|y_{<t})$, but the increase is capped at $1+\epsilon$ times the old probability — beyond that, the objective stops providing gradient, preventing the policy from moving too aggressively. If $A_t < 0$ (the token was worse than expected), the loss encourages decreasing $\pi_\theta(y_t|y_{<t})$, but the decrease is capped at $1-\epsilon$ times the old probability, preventing the policy from completely abandoning tokens that were only slightly worse than expected.

Why this form: the clipping mechanism addresses a fundamental problem with vanilla policy gradient: importance sampling from an off-policy distribution (the old policy $\pi_{\theta_{\text{old}}}$) is only valid when the new policy $\pi_\theta$ is not too different from the old policy. If the policy changes too much in a single update, the ratio $\frac{\pi_\theta}{\pi_{\theta_{\text{old}}}}$ can become very large (for tokens the new policy likes much more) or very small (for tokens the new policy dislikes much more), leading to high-variance, unstable updates. The clipping ensures that the policy moves in the right direction (signaled by $A_t$) but limits the step size, allowing multiple epochs of optimization on the same batch of rollouts without catastrophic divergence.

The paper uses one PPO epoch per rollout batch and sets the KL penalty coefficient to 0 (Section 4.1). This is a significant design choice that departs from standard RLHF PPO implementations, which typically add an explicit KL penalty to keep the policy close to a reference model. The paper finds that the implicit regularization from (1) the clipping objective and (2) initializing the PRM and policy from the same checkpoint is sufficient to maintain stability without an explicit KL term. Removing the KL penalty simplifies the implementation and reduces hyperparameter tuning burden.

The policy uses the AdamW optimizer with learning rate $5 \times 10^{-7}$ (constant, no schedule mentioned), batch size 256 (meaning 256 prompts per training step), and micro batch size 8 (for gradient accumulation). The rollout stage samples $K = 4$ responses per prompt by default, so each training step processes $256 \times 4 = 1024$ generated responses.

Initialization Strategy: SFT Model as Both Policy and PRM

A critical design choice that the paper validates extensively is the initialization of the Implicit PRM from the same SFT model that initializes the policy (Section 3.3 and Section 5.1). In conventional RLHF pipelines, the reward model is a separately trained component — often with different architecture, different training data, and sometimes different scale — that must be prepared before RL begins. PRIME eliminates this phase entirely.

The procedure is:

1. Take the base model (Qwen2.5-Math-7B-Base) and perform supervised finetuning (SFT) on 230K reasoning examples to obtain Eurus-2-7B-SFT.
2. Initialize the policy $\pi_\theta$ from Eurus-2-7B-SFT.
3. Initialize the Implicit PRM $\pi_\phi$ from the same Eurus-2-7B-SFT checkpoint.
4. Initialize the reference model $\pi_{\text{ref}}$ from Eurus-2-7B-SFT and keep it frozen throughout training.
5. Since $\pi_\phi = \pi_{\text{ref}}$ at initialization, the initial process rewards are zero for all tokens (because $\beta \log \frac{\pi_\phi(y_t|y_{<t})}{\pi_{\text{ref}}(y_t|y_{<t})} = \beta \log 1 = 0$).
6. As training proceeds and the PRM is updated on outcome-labeled policy rollouts, the PRM diverges from the reference model, and non-zero token-level rewards emerge.

Why this outperforms a dedicatedly trained PRM: Section 5.1 (Figure 4) compares this strategy against initializing the PRM from EurusPRM, a reward model trained on 500K additional samples from diverse model families (Llama-3.1, Qwen2.5). The SFT-initialized PRM substantially outperforms the dedicated EurusPRM. The paper hypothesizes the reason is distribution shift: EurusPRM was trained on solutions from multiple model families, not just the policy model's own outputs, so it is miscalibrated for the specific distribution of rollouts generated by Eurus-2-7B-SFT during RL. In contrast, initializing from the same checkpoint means the PRM starts perfectly calibrated to the policy's output distribution (trivially, since it is the same model), and subsequent online updates keep it aligned as the policy shifts.

This finding is practically transformative: it means the entire pipeline — pretraining → SFT → RL — can be executed with a single model lineage, no separate reward model training stage, no curation of reward model training data, no compute spent on a dedicated RM. The "time to first useful RL signal" is essentially zero: the PRM starts giving zero rewards (which is neutral) and improves as it sees outcome labels.

Online Prompt Filtering: Maintaining Training Stability

PRIME introduces an online prompt filtering mechanism (Section 3.3) that selectively removes prompts from the training batch based on the accuracy of the policy's responses. The procedure operates at each RL iteration before updating the policy and PRM:

1. For each prompt $x$ in the current batch $\mathcal{B}$, the policy generates $K = 4$ responses.
2. The outcome verifier grades each response, producing $\{r_o(y^1), ..., r_o(y^K)\}$ for that prompt.
3. Compute the accuracy for this prompt: $\text{acc}(x) = \frac{1}{K} \sum_{i=1}^K r_o(y^i)$.
4. Filter the batch to retain only prompts where $\text{acc}(x)$ falls within a certain range (the paper does not specify the exact thresholds, but describes it as "median-level difficulty").

What this accomplishes: prompts filtered into the training set are those of moderate difficulty — the policy gets some responses right and some wrong, providing informative contrast for both the PRM and the policy. Prompts where the policy gets all responses correct (too easy) or all responses wrong (too hard) are removed, as they provide no useful gradient signal: if all responses are correct, there is no negative example for the PRM to learn from or for the policy to improve upon; if all are wrong, there is no positive example.

Why this matters: Figure 2 demonstrates that online prompt filtering "largely lowers the variance of RL training." Without filtering, the training outcome reward curve shows high variance (large oscillations from step to step), while with filtering, the curve is smoother and rises more consistently. This makes intuitive sense: if the batch contains many prompts where all responses are already correct, the policy has no room to improve on those prompts and the updates are essentially noise. If it contains many prompts where all responses are wrong, the PRM receives only negative labels and cannot learn to distinguish good intermediate steps from bad ones.

The filtering also helps balance the data distribution for PRM training. The Implicit PRM is trained via binary cross-entropy on the filtered batch, meaning it sees roughly equal numbers of correct and incorrect responses. This prevents class imbalance from biasing the PRM toward always predicting "incorrect" (if most rollouts are wrong) or "correct" (if most rollouts are right).

This technique is inspired by Qwen2.5-Math (Yang et al., 2024b), which also filtered training data by difficulty, but adapted here to an online setting where difficulty is assessed dynamically based on the current policy's performance rather than statically from a fixed dataset.

The SFT Warmup Stage: Action-Centric Chain-of-Thought

Before RL training begins, the base model undergoes a supervised finetuning (SFT) warmup stage (Appendix D) that serves two purposes: (1) it imbues the model with a specific reasoning format that makes process rewards more interpretable, and (2) it provides a non-random initialization for both the policy and the PRM.

The SFT stage uses an action-centric chain-of-thought reasoning framework where the model is trained to structure its solutions as sequences of explicit reasoning actions. The seven actions defined in Table 9 are:

• ASSESS: Analyze the current situation, identify key elements and goals.
• ADVANCE: Move forward with reasoning — calculate, conclude, or form a hypothesis.
• VERIFY: Check the accuracy of the current approach and look for errors.
• SIMPLIFY: Break complex problems into simpler parts.
• SYNTHESIZE: Combine multiple pieces of information into a complete solution.
• PIVOT: Change strategy when the current approach is not working.
• OUTPUT: Summarize the thought process and present the final answer.

The response always begins with ASSESS and ends with OUTPUT, providing a structured template that the model learns to follow. The SFT data consists of 230K examples spanning math, coding, and biomedicine (Table 10 provides the breakdown by source dataset), with responses generated by Llama-3.1-70B-Instruct prompted to follow the action-centric format. The average response length is 1,390 tokens.

Key hyperparameters for SFT: full parameter fine-tuning (not LoRA), learning rate $1 \times 10^{-5}$, AdamW optimizer with cosine annealing schedule and 0.1 warmup ratio, batch size 96, 3 epochs over the 230K dataset.

Why action-centric reasoning: the paper does not explicitly ablate this format against alternatives, but the motivation is clear: explicit reasoning actions create natural segmentation points in the solution where the process reward can meaningfully distinguish between productive steps (e.g., ADVANCE with a correct calculation) and unproductive ones (e.g., repeated VERIFY without progress). Without such structure, token-level rewards might be noisier and harder for the policy to interpret. Additionally, the explicit actions serve as behavioral scaffolding that helps the model learn systematic reasoning patterns before RL fine-tunes them.

The paper makes an important data strategy decision: datasets used for SFT are disjoint from those used for RL. The SFT uses 230K examples from MathInstruct, OpenMathInstruct-2, Numina, Reasoning-001, Code-Feedback, Magicoder, and UltraMedical — all of which provide instructions but not necessarily ground-truth answers. The RL stage uses 457K math problems and 27K coding problems from NuminaMath-CoT, APPS, CodeContests, TACO, and Codeforces — all of which have verifiable ground-truth answers. This separation ensures that the RL stage sees novel problems not memorized during SFT, forcing genuine generalization rather than pattern matching.

Reference Model Configuration: Flexible but Consequential

The paper explores two strategies for the reference model $\pi_{\text{ref}}$ used in computing implicit process rewards (Appendix B.1, Figures 9 and 10):

1. SFT reference ($\pi_{\text{ref}} = \text{SFT model}$): The reference model is frozen at the initial SFT checkpoint and never updated. This is the default configuration in the main experiments.

2. Policy reference ($\pi_{\text{ref}} = \pi_{\theta_{\text{old}}}$): The reference model is the policy model's parameters from the previous iteration (the old log-probabilities used in the PPO clipping ratio). This means the reference model changes every iteration as the policy improves.

Tradeoffs: The SFT reference provides a stable baseline — tokens are rewarded if the PRM assigns higher probability than the initial SFT model did, which is a fixed target. This makes the reward signal interpretable and consistent across training. The policy reference means the baseline shifts every iteration, which could provide a more adaptive signal (rewarding tokens that the PRM considers good relative to the immediately previous policy, not relative to the distant SFT initialization) but also introduces non-stationarity in the reward definition.

Figure 10 shows that the two strategies produce "close" training reward curves, suggesting the choice is not critical for performance. The default choice of SFT reference is motivated by simplicity (no need to track old policy parameters for the PRM computation, which is separate from the PPO old parameters) and by the fact that keeping the reference frozen allows the PRM's $r_\phi(y)$ values to be interpreted as the PRM's assessment of solution quality without confounding from the moving target.

Putting It All Together: The Complete Algorithm

Algorithm 1 in the paper provides the complete pseudocode. Walking through one iteration of the loop:

Inputs: The SFT model $\pi_{\theta_{\text{init}}}$, the outcome verifier function $r_o$, the RL training dataset $\mathcal{D}$, number of responses per prompt $K = 4$, total number of iterations $N$.

Step 1 — Initialization (lines 1–2): The policy $\pi_\theta$ and old policy $\pi_{\theta_{\text{old}}}$ are both set to the SFT model. The Implicit PRM $\pi_\phi$ and reference model $\pi_{\text{ref}}$ are both set to the same SFT model. All four objects are copies of the same checkpoint at the start. $\pi_{\text{ref}}$ will remain frozen; the other three will evolve.

Step 2 — Rollout (lines 3–4): Sample a batch of prompts $\mathcal{B}$ from the training dataset (size 256). For each prompt $x \in \mathcal{B}$, the current policy $\pi_\theta$ generates $K = 4$ complete responses $\{y^1, ..., y^4\}$ by autoregressive sampling.

Step 3 — Outcome labeling (lines 5–6): The rule-based verifier $r_o$ checks each response against the ground-truth answer, producing binary labels $\{0, 1\}$. Apply the online prompt filter: for each prompt, compute accuracy across its 4 responses. Retain only prompts where this accuracy falls within the acceptable range (not all correct, not all incorrect). Let $\mathcal{T}$ be the set of $(x, y, r_o(y))$ triples that survive filtering.

Step 4 — PRM forward pass (line 7): Run $\pi_\phi$ and $\pi_{\text{ref}}$ in forward mode on each $(x, y) \in \mathcal{T}$ to obtain per-token log-probabilities. Compute the implicit process reward for each token via Equation 3: $r_\phi(y_t) = \beta \log \frac{\pi_\phi(y_t|y_{<t})}{\pi_{\text{ref}}(y_t|y_{<t})}$.

Step 5 — PRM update (line 8): Optimize $\pi_\phi$ using the cross-entropy loss over the filtered triples $\mathcal{T}$. This updates the PRM's parameters $\phi$ to better predict outcome correctness from the full-trajectory reward $r_\phi(y)$. Note: this step uses only the outcome labels $r_o(y)$, not any intermediate-step labels.

Step 6 — Advantage computation (line 9): Using the updated PRM (if double-forward is used; if single-forward, use the old PRM), compute token-level process rewards for all responses. Then compute per-timestep advantages via Equation 5, combining the leave-one-out-baselined process rewards and outcome rewards.

Step 7 — Policy update (line 10): Update the policy $\pi_\theta$ using the PPO clipped surrogate loss (Equation 6) with the computed advantages. This is a standard gradient step using the AdamW optimizer with learning rate $5 \times 10^{-7}$.

Step 8 — Synchronization (line 11): Update $\theta_{\text{old}} \leftarrow \theta$ for the next iteration's importance sampling ratio.

Repeat until iteration $N$ is reached.

The default experiment runs for 240 steps for the baseline RLOO comparison and 592 steps for the full PRIME training, with approximately 680.3 seconds per step for PRIME (vs. 530.7 seconds for RLOO without PRM updates), broken down in Table 2 as: rollout 281.7s, policy update 156.6s, PRM update 150.9s, other overhead 91.1s. The PRM update adds about 24% overhead per step, but the 2.5× sample efficiency gain means the total wall-clock time to reach a given performance is still roughly 2× faster.

The paper also experiments with double-forward (Appendix B.2, Figure 11), where the PRM is updated (step 5) and then the updated PRM is re-run to compute fresh process rewards for the advantage computation (step 6), rather than using the PRM's rewards from before the update (single-forward). While double-forward improves PRM classification accuracy on the training samples, the training reward curves are "close" between the two methods, suggesting the extra forward pass is not necessary. The default is single-forward, which saves computation.

4. Key Insights and Innovations

Innovation 1: Online PRM Updates Become Scalable When the Reward Model Is Parameterized to Need Only Outcome Labels

The intellectual pivot that makes PRIME possible is the recognition that two previously separate problems — obtaining dense process rewards and updating a reward model online — can be solved simultaneously by a single design choice about how the reward is parameterized. Prior work treated these as independent challenges: Lightman et al. (2023) solved dense rewards through expensive human annotation but could not update online; Wang et al. (2023) solved automated PRM training through Monte Carlo rollouts but made online updates computationally infeasible; DeepSeek-AI et al. (2025) simply abandoned PRMs in online RL because the combined cost of acquiring and maintaining process labels was too high. The field's implicit assumption was that dense rewards and online updates are separate problems requiring separate solutions.

PRIME's key insight is that if the reward model is expressed as the log-ratio between two language model probabilities, then the data required to update it online — outcome labels on policy rollouts — is exactly the data already collected for policy updates. The Implicit PRM parameterization r_φ(y_t) = β log π_φ(y_t|y_<t) / π_ref(y_t|y_<t) means that (a) per-token rewards are derivable from any model trained on whole-trajectory outcome labels, and (b) updating the model to stay calibrated to the policy distribution requires nothing beyond the (prompt, response, outcome_label) triples that every outcome-verifier-based RL pipeline already generates. No step labels, no Monte Carlo rollouts, no additional annotation — the PRM update piggybacks on data already flowing through the system.

This is a fundamental reframing, not an incremental improvement. It converts the question from "how do we afford to collect process labels for online updates?" to "how do we parameterize the reward so that outcome labels suffice for online updates?" — a question with a clean mathematical answer rather than an engineering compromise. The evidence that this reframing works is in Figure 5: offline PRMs (trained once, never updated) start with high classification accuracy on policy rollouts but steadily degrade as the policy distribution shifts, while online PRMs start with zero reward (since they are initialized from the same model as the policy, giving zero log-ratio) and steadily improve. The curves cross — the online PRM overtakes the offline one at roughly step 50 and maintains an increasing advantage thereafter. This is a direct demonstration that online update capability trumps initial PRM quality, reversing the prevailing wisdom that a well-trained offline PRM is a prerequisite for dense-reward RL.

The significance extends beyond math reasoning. The Implicit PRM parameterization is independent of task, model architecture, and RL algorithm — it is a mathematical identity that any causal language model trained on binary outcome labels can be decomposed into token-level rewards. This means the insight is potentially portable to any domain where outcome labels exist (code execution, factual correctness, safety compliance), and the paper's demonstration that it works online — not just at inference time as in Yuan et al. (2024b) — opens a path to dense-reward RL in domains where step labels are even harder to define than in math.

---

Innovation 2: The SFT Model Itself Is a Sufficient PRM Initialization — Eliminating the Dedicated Reward Model Training Phase

The paper's most practically disruptive finding is that the entire dedicated reward model training pipeline — data collection, annotation, model training, validation — can be eliminated without any performance penalty and with significant gains. Prior to PRIME, the standard RLHF and RL-for-reasoning workflow included a distinct reward model training phase: collect preference or correctness data, train a separate model (often larger than the policy, as with DeepSeekMath's 72B RM in Shao et al., 2024), validate its calibration, and only then begin RL. This phase consumed substantial compute and engineering effort, created a dependency on the quality and coverage of the RM training data, and introduced distribution shift between the RM's training distribution and the policy's eventual rollout distribution.

PRIME's finding — that initializing the Implicit PRM from the same SFT checkpoint used for the policy outperforms a PRM trained on 500K additional samples from diverse model families (EurusPRM; Figure 4) — is both surprising and consequential. The mechanism is distribution alignment: EurusPRM was exposed to solutions from Llama-3.1 and Qwen2.5 models that may have different reasoning styles, error patterns, and output distributions than Eurus-2-7B-SFT. When RL begins and the policy generates rollouts in its own distinctive style, EurusPRM is miscalibrated — it assigns incorrect reward signals because it was optimized for a different solution distribution. The SFT-initialized PRM, by contrast, starts perfectly calibrated to the policy's distribution because it is the same model. At initialization, all process rewards are zero (since π_φ = π_ref, giving log(1) = 0), and the rewards become informative only as the PRM is updated on the policy's own rollouts. This means the PRM's reward signal is always on-policy — it learns to distinguish good from bad tokens specifically within the policy's current behavioral regime.

The practical implications of this finding are substantial. Table 3 quantifies the resource differential: Qwen2.5-Math-7B-Instruct required 618K in-house RM training data and a separate 72B reward model; PRIME requires zero RM data and zero dedicated RM training. The conceptual implication is even larger: it means that any model that has been SFT'd is already a latent PRM, requiring only outcome-labeled rollouts and online updating to become functional. This collapses the "time to first RL signal" from weeks (data collection + RM training + validation) to essentially zero (start RL immediately after SFT). It also eliminates a major source of experimental fragility — if the RM training data has poor coverage or the RM architecture is poorly chosen, the entire downstream RL pipeline is compromised. PRIME's approach makes RM quality a function of RL training progress rather than a prerequisite for it.

This finding also connects to the paper's "Zero" experiments (Appendix B.4, Figure 13), where RL is started directly from the base model without any SFT warmup. The fact that PRIME-Zero converges faster than PRIME-from-SFT and surpasses Qwen2.5-Math-7B-Instruct within 32 steps suggests that the model's own pretrained representations contain sufficient signal to bootstrap reward modeling, and the SFT stage — while helpful for structuring reasoning — is not strictly necessary for the PRM to become informative. This is a fundamentally different relationship between pretraining and reward modeling than the field has assumed: the reward model is not a separate artifact that must be built from scratch; it is latent in the policy's weights and can be surfaced through online outcome-labeled training.

---

Innovation 3: Process Rewards and Outcome Rewards Have Complementary Roles in Advantage Estimation and Should Be Combined Additively

While prior work has used either process rewards or outcome rewards for policy gradient estimation — but rarely both — PRIME demonstrates that the two reward sources provide orthogonal, complementary signals that can be added together in the advantage function to produce strictly better policy updates than either alone. This is not obvious a priori: one might expect that adding token-level process rewards to an outcome-based advantage would be redundant (if the process rewards are accurate, they should already predict the outcome) or destabilizing (if the process rewards are noisy, they might corrupt the cleaner outcome signal). The paper's empirical results reject both concerns.

The additive combination in Equation 5 — where the process reward advantage (leave-one-out baselined, summed over future tokens) and the outcome reward advantage (leave-one-out baselined, constant per trajectory) are computed separately and summed — is justified on numerical stability grounds, but the conceptual insight runs deeper. The outcome reward provides a high-precision but sparse signal: it is measured against ground truth with zero error, but it provides no information about which tokens in a correct solution were essential versus incidental, nor which tokens in an incorrect solution were on the right track before a late error. The process reward provides a dense but approximate signal: it estimates token quality from statistical associations learned by the PRM, which is never perfectly calibrated, but it provides per-token gradient information that can accelerate learning and improve credit assignment. Together, they compensate for each other's weaknesses: the outcome reward anchors the overall direction (correct vs. incorrect), while the process reward sculpts the gradient within trajectories (which tokens to reinforce or suppress).

The evidence for complementarity appears in multiple results. Table 4 shows that RLOO with PRIME (41.0% average) substantially outperforms RLOO without PRIME (36.9%) — a 4.1 percentage point gain from adding process rewards on top of outcome rewards. Figure 7 generalizes this: every RL algorithm tested (REINFORCE, GRPO, PPO, RLOO) improves when PRIME's process rewards are added, with gains ranging from 1.2 to 4.1 percentage points. The fact that the gain is consistent across algorithms with fundamentally different baseline estimation strategies (leave-one-out, group normalization, value-function-based) suggests the process reward signal is providing information orthogonal to whatever the outcome-based advantage already captures.

A subtlety: the paper's analysis in Section 5.4 (Figure 8) shows that using the Implicit PRM as a value model (where it predicts expected returns at each state, serving as a baseline for outcome rewards) performs worse than using it as a reward model (where its token-level outputs are directly added to the advantage). Specifically, "Implicit PRM as value" achieves 36.6% average accuracy while "Implicit PRM as reward" (REINFORCE w/ PRIME) achieves 37.8%. This suggests that the Implicit PRM's token-level outputs are more informative as direct reward signals (indicating which actions were good) than as state-value estimates (indicating the expected return from this point). The paper frames this as "PRMs work better than value models in RL for LLMs," which is an important architectural insight: if you have a model that can produce token-level quality estimates, use them as additive rewards in the advantage rather than as subtractive baselines.

---

Innovation 4: Reward Overoptimization in Online RL Is Diagnosable Through the Trajectory of PRM Classification Accuracy

A recurring challenge in RLHF and RL-for-reasoning is detecting when reward overoptimization (also called reward hacking) has begun — the point at which the policy exploits the reward model's blind spots rather than genuinely improving. Standard diagnostics like monitoring training reward curves are insufficient because overoptimization manifests as rising reward model scores with stagnant or declining true performance, and the true performance metric (e.g., downstream benchmark accuracy) is typically evaluated only periodically, not at every training step.

PRIME provides a continuous, per-step diagnostic for overoptimization: the PRM's classification accuracy on policy rollouts. Because the Implicit PRM is trained on outcome labels (the ground-truth verifier), its ability to distinguish correct from incorrect responses on the policy's current outputs can be measured at every iteration — simply pair correct and incorrect rollouts and check whether the PRM's predicted reward r_φ(y) ranks the correct one higher. Figure 5 plots this metric over training and reveals two distinct trajectories:

• Offline PRM: starts at ~70% accuracy (high, because it was trained on diverse data) but steadily declines throughout training as the policy distribution shifts away from the PRM's training distribution. By step 240, accuracy has dropped to approximately 58%. This declining curve is a direct indicator of overoptimization: the PRM is becoming less able to distinguish good from bad responses, meaning the policy can exploit its blind spots.

• Online PRM: starts at 50% (random chance, because at initialization π_φ = π_ref, giving zero reward for all responses) and steadily improves to approximately 75% by step 200. The PRM is becoming more accurate over time because it is being trained on the policy's latest outputs, staying calibrated to the shifting distribution.

The diagnostic value of this metric is that it anticipates downstream performance degradation. In Figure 4, the offline PRM run shows declining training rewards and lower test accuracy compared to the online runs. An RL practitioner monitoring PRM accuracy could detect the declining trend early and intervene — e.g., by increasing the PRM update frequency, adjusting the learning rate, or collecting fresh PRM training data — before the policy's downstream performance suffers. This is a practical contribution beyond the algorithmic innovation: it gives operators a real-time signal of reward model health that has been missing from prior RLHF and RL-for-reasoning pipelines.

The broader conceptual contribution is the framing of reward model calibration as a dynamic, adversarial process rather than a static property. Prior work (Gao et al., 2022) established that reward model overoptimization is a function of the KL divergence between the policy and the reward model's training distribution, but this formulation implies a one-way relationship: the policy moves away from the RM's training distribution, and overoptimization results. PRIME's online PRM creates a two-player dynamic: the policy moves to exploit the PRM, and the PRM moves to stay calibrated to the policy. The PRM accuracy trajectory in Figure 5 shows this dynamic playing out in real time, with the online PRM winning the calibration battle (accuracy increases) while the offline PRM loses it (accuracy decreases). This dynamic framing suggests that the effectiveness of dense-reward RL is fundamentally limited by how quickly the PRM can adapt relative to the policy — a rate-of-learning competition that the paper quantifies through the separate learning rates for policy (5 × 10⁻⁷) and PRM (10⁻⁶), with the PRM learning twice as fast to stay ahead of distribution shift.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The RL training stage uses a curated set of 457K math problems (sourced from NuminaMath-CoT) and 27K coding problems (sourced from APPS, CodeContests, TACO, and Codeforces), all with verifiable ground-truth answers — LaTeX boxed expressions for math, test cases for code (Section 4.1, Appendix E.1). The SFT warmup stage uses a disjoint set of 230K examples spanning math, coding, and biomedicine, drawn from MathInstruct, OpenMathInstruct-2, Numina, Reasoning-001, Code-Feedback, Magicoder, and UltraMedical (Appendix D, Table 10). Evaluation uses seven benchmarks: AIME 2024, AMC, MATH-500, Minerva Math, OlympiadBench, LeetCode, and LiveCodeBench (v2) (Section 4.1). The AMC and AIME benchmarks are from Li et al. (2024); MATH-500 is from Hendrycks et al. (2021b); Minerva Math is from Lewkowycz et al. (2022); OlympiadBench is from He et al. (2024); LeetCode is from Guo et al. (2024); LiveCodeBench is from Jain et al. (2024).

• Base model. All experiments start from Qwen2.5-Math-7B-Base (Yang et al., 2024b), chosen for its "great mathematical capabilities" (Appendix D). The SFT warmup produces Eurus-2-7B-SFT as the starting point for RL. A Qwen2.5-32B-Base variant is explored in the "Zero" experiments (Appendix B.4, Figure 14). For the DeepScaleR comparison (Appendix B.7), DeepSeek-R1-Distill-Qwen-1.5B is used as the base.

• Metrics. The primary training metric is outcome training reward — the fraction of policy-generated responses whose final answer matches the ground truth, averaged over the training batch, plotted as a 10-step moving average (e.g., Figure 3a, Figure 4a). The primary evaluation metric is pass@1 accuracy (single greedy or sampled generation) on each of the seven downstream benchmarks, reported as percentages (Table 1). For AIME 2024 and AMC, avg@16 results (16 sampled generations with majority voting) are also reported (Table 12). The PRM diagnostic metric is PRM classification accuracy on policy rollouts — the fraction of correct-vs-incorrect response pairs where the PRM's predicted reward r_φ(y) ranks the correct response higher (Figure 5).

• Baselines. The primary baseline is RLOO with outcome verifier only (RLOO w/ OV Only) — the same RL algorithm and training setup but without implicit process rewards, computing advantages solely from the rule-based outcome verifier via Equation 4 (Section 4.2, Table 1). Additional algorithmic baselines in Section 5.3 and Table 4 include: REINFORCE (Williams, 1992) without process rewards; GRPO (Shao et al., 2024) without process rewards; PPO (Schulman et al., 2017) without process rewards — each with their respective advantage estimation formulas adapted to the compound reward structure (Equations 7–8 in Appendix B.3). External model baselines in Table 1 include GPT-4o, Llama-3.1-70B-Instruct, Qwen2.5-Math-7B-Instruct, and Eurus-2-7B-SFT (the SFT model before RL). For the DeepScaleR comparison (Appendix B.7), the baselines are DeepScaleR-1.5B-Preview and DeepSeek-R1-Distill-Qwen-1.5B.

• Generation budget / compute accounting. The compute budget is measured in RL training steps and wall-clock time. One "step" in RL consists of sampling a batch of 256 prompts, generating K=4 responses per prompt (so 1024 total generations), performing one PRM update, and performing one policy update with a single PPO epoch (Section 4.1). Step-wise timing for PRIME (680.3s per step) versus RLOO (530.7s per step) is reported in Table 2, broken down into rollout, policy update, PRM update, and other overhead. Sample efficiency comparisons (e.g., "2.5× sample efficiency") refer to the number of training steps needed to reach a given outcome training reward or test accuracy threshold. For the scaling experiments (Section 5.2), the number of rollout steps is extended to 800, and the number of responses per prompt is increased from 4 to 16. For the DeepScaleR comparison (Appendix B.7), GPU hours are reported: PRIME consumed 446.7 A800 GPU hours, while DeepScaleR stage 1 consumed approximately 600 GPU hours (and 3800 total).

• Cross-validation / statistical protocol. The paper does not employ formal cross-validation for RL experiments. The main PRIME run (592 steps) and the RLOO baseline (240 steps) are compared at multiple training step checkpoints (80, 160, 240, 320, 592 steps in Table 1), providing a view of performance trajectories rather than single-point estimates. The "Zero" experiments (Appendix B.4) report training curves across multiple random seeds implicitly through the moving-average smoothing. The ablation experiments in Section 5 are typically run for fewer steps than the main experiment (the paper notes in Appendix A: "besides the main experiments of PRIME, we ran fewer steps for other ablation experiments, while we conduct comparison under the same step number for fairness"). No confidence intervals, standard errors, or significance tests are reported. The 500-question MATH-500 test set and other benchmarks (AIME 2024 with 30 questions, AMC with 83 questions) are relatively small for percent-accuracy metrics, meaning small absolute changes can correspond to only 1–2 questions, though the paper does not discuss statistical power.

Main Quantitative Results

Dense Rewards vs. Sparse Rewards: PRIME Improves Both Sample Efficiency and Final Performance

The paper's central training dynamic comparison appears in Figure 3 and Table 1. PRIME (592 steps) and RLOO with outcome verifier only (240 steps) are compared on both training reward trajectories and downstream benchmark performance.

Training rewards (Figure 3a): PRIME's outcome training reward curve rises faster and reaches a higher asymptote than RLOO with outcome verifier only. The paper reports that PRIME achieves the same training reward as RLOO in 40% of the training steps, translating to a 2.5× sample efficiency gain when measured by training steps. At convergence, PRIME's training reward is 6.9% higher than RLOO's (Section 4.3, text preceding Figure 3). The PRIME curve also exhibits lower variance (smoother trajectory) compared to RLOO, which the paper attributes to the dense reward signal reducing the noise in advantage estimates. The wall-clock time comparison (Figure 17) shows a qualitatively similar efficiency advantage: the x-axis is wall-clock time rather than steps, and PRIME still substantially outperforms, though the paper does not quantify the exact multiplicative factor in wall-clock terms.

Downstream performance (Figure 3b, Table 1): Across seven reasoning benchmarks evaluated at multiple training checkpoints, PRIME consistently outperforms RLOO. At 240 steps (the final checkpoint for RLOO), PRIME achieves an average of 41.0% across the seven benchmarks compared to RLOO's 36.9% — a 4.1 percentage point absolute improvement (Table 4). This advantage persists and grows at later checkpoints: at 592 steps, PRIME reaches 43.9% average, a 15.1% relative improvement over Eurus-2-7B-SFT's baseline of 28.8% (Figure 12 caption, Table 1). The largest gains appear on competition-level math: AIME 2024 improves from 3.3% (SFT) to 26.7% (PRIME at 592 steps) — an 8.1× relative improvement — and AMC improves from 30.1% to 57.8% — a 92% relative improvement.

Benchmark-by-benchmark at the 240-step checkpoint (Table 1, comparing PRIME at 240 vs. RLOO at 240): AIME 2024: 20.0% vs. 20.0% (no difference); AMC: 50.6% vs. 47.0% (+3.6); MATH-500: 78.2% vs. 73.2% (+5.0); Minerva Math: 39.3% vs. 36.4% (+2.9); OlympiadBench: 40.3% vs. 35.4% (+4.9); LeetCode: 31.1% vs. 28.3% (+2.8); LiveCodeBench: 27.5% vs. 26.7% (+0.8). The gains are largest on math reasoning (MATH-500, OlympiadBench, AMC) and smaller or absent on coding (LeetCode: +2.8; LiveCodeBench: +0.8; AIME 2024: 0.0 at 240 steps). At later checkpoints (592 steps), PRIME opens a lead on AIME 2024 (26.7% vs. 20.0%), suggesting that process rewards provide greater benefit as training extends and problems become more complex.

Training efficiency breakdown (Table 2): PRIME requires 680.3 seconds per step compared to RLOO's 530.7 seconds — a 24% overhead from the PRM update (150.9s of the 680.3s total). However, because PRIME achieves equivalent training reward in 40% of the steps, the total wall-clock time to reach a given performance threshold is approximately 2× less for PRIME (Section 4.3: "PRIME would still be 2× more efficient than RLOO when estimated by training time").

Comparison to external models (Figure 12, Table 1): Eurus-2-7B-PRIME at 592 steps outperforms Qwen2.5-Math-7B-Instruct on five of seven benchmarks: AIME 2024 (26.7% vs. 13.3%), AMC (57.8% vs. 50.6%), OlympiadBench (42.1% vs. 40.7%), LeetCode (33.3% vs. 11.7%), and LiveCodeBench (28.6% vs. 11.3%). It falls short on MATH-500 (79.2% vs. 79.8%, essentially tied) and Minerva Math (38.6% vs. 34.6%, PRIME leads). Notably, Eurus-2-7B-PRIME exceeds GPT-4o on AIME 2024 (26.7% vs. 9.3%) and AMC (57.8% vs. 45.8%), and exceeds Llama-3.1-70B-Instruct on AIME 2024 (26.7% vs. 20.0%), AMC (57.8% vs. 37.3%), and most other benchmarks.

Resource efficiency (Table 3): The paper emphasizes that Eurus-2-7B-PRIME achieves these results with dramatically fewer resources than Qwen2.5-Math-7B-Instruct: 230K SFT data vs. 2.5M, zero RM data vs. 618K, no dedicated RM vs. a 72B Qwen2.5-Math-RM, and 150K RL queries × 4 samples vs. 66K queries × 32 samples. The phrase "with only 10% of the data used by Qwen-Math" (Section 1) refers to the SFT data (230K vs. 2.5M), not the total pipeline.

Online PRM Update Is Essential; Offline PRMs Degrade Under Distribution Shift

Figure 4 and Figure 5 present the core evidence for the necessity of online PRM updates. Three PRM configurations are compared:

1. Online PRM initialized from SFT model (the default PRIME configuration)
2. Online PRM initialized from EurusPRM (a PRM trained offline on 500K additional samples from Llama-3.1 and Qwen2.5 model families, then updated online during RL)
3. Offline EurusPRM (the same EurusPRM, kept frozen and never updated during RL)

Training rewards (Figure 4a): The online SFT-initialized PRM achieves the highest training rewards, with a smooth, steadily rising curve that reaches approximately 0.50 at step 240. The online EurusPRM variant rises more slowly initially — starting from a lower reward because the EurusPRM is miscalibrated to the policy's output distribution — but catches up somewhat by step 200. The offline EurusPRM variant shows the worst performance, with training rewards plateauing around 0.44–0.46 and showing higher variance.

Downstream performance (Figure 4b): This pattern is amplified on test accuracy. The online SFT-initialized PRM reaches approximately 40% average accuracy by step 224, while the online EurusPRM variant reaches roughly 37–38%, and the offline EurusPRM plateaus around 35%. The gap between online and offline EurusPRM demonstrates that online updating alone provides substantial benefit even when starting from a suboptimal initialization: the online EurusPRM run outperforms the offline EurusPRM run by approximately 2–3 percentage points. However, the best results come from starting with the SFT model — online EurusPRM never catches up to online SFT-PRM, suggesting that the initial distribution alignment advantage persists throughout training.

PRM classification accuracy (Figure 5): This figure provides the mechanistic explanation. The offline EurusPRM starts at approximately 70% accuracy (it was well-trained on its original data) but steadily declines to about 58% by step 200 — a clear signature of distribution shift and impending overoptimization. The online EurusPRM starts lower (approximately 62% — it is somewhat miscalibrated to begin with) but improves over training, reaching approximately 72% by step 200. The online SFT-PRM starts at 50% (chance level, since initial rewards are all zero) and rapidly improves to approximately 75% by step 200, overtaking both EurusPRM variants by step 50–100. The figure demonstrates that online updating reverses the direction of PRM calibration drift: offline PRMs get worse over time, online PRMs get better. And the SFT initialization, despite starting at chance-level accuracy, learns fastest because it is perfectly aligned with the policy's output distribution from the start.

PRIME Generalizes Across RL Algorithms

Figure 7 and Table 4 demonstrate that PRIME's process rewards provide consistent benefits regardless of the underlying policy gradient algorithm. Four algorithms are tested: REINFORCE (Williams, 1992), GRPO (Shao et al., 2024), PPO (Schulman et al., 2017), and RLOO (Ahmadian et al., 2024) — each with and without PRIME's implicit process rewards added to their advantage estimates.

Training rewards (Figure 7): All algorithms show higher training rewards when augmented with PRIME. The curves for PRIME-augmented variants (solid lines) are consistently above their outcome-only counterparts (dashed lines). The ordering among algorithms is preserved — RLOO+PRIME achieves the highest training rewards (~0.48), followed by GRPO+PRIME (~0.47), REINFORCE+PRIME (~0.46), and PPO+PRIME (~0.45) — but the PRIME boost is visible for each. Notably, PPO without PRIME shows the lowest training rewards (~0.42) but catches up substantially with PRIME (+~0.03). The spread between algorithms is smaller with PRIME than without it, suggesting that dense rewards partially compensate for weaknesses in the underlying advantage estimator.

Downstream performance (Table 4): All comparisons are at 240 steps.

• RLOO: 36.9% without PRIME → 41.0% with PRIME (+4.1)
• REINFORCE: 36.0% → 37.8% (+1.8)
• GRPO: 36.1% → 37.8% (+1.7)
• PPO: 35.8% → 39.4% (+3.6)

The gains are statistically meaningful (1.7–4.1 percentage points) and consistent in direction (all positive). RLOO benefits most, which the paper attributes to its leave-one-out baseline being particularly well-suited to the additive structure of process + outcome rewards. The paper also tests using the Implicit PRM as a value model (row "PRIME as Value Model" in Table 4) — where the PRM's cumulative log-ratio v_φ(y_<t+1) is used as a state-value baseline for outcome rewards, analogous to PPO's critic — and finds it achieves only 36.6%, worse than all process-reward-as-additive-reward variants. This supports the claim in Section 5.4 that "PRMs work better than value models in RL for LLMs."

The PPO+PRIME variant (39.4%) is notable because standard PPO already includes a learned value model, yet adding PRIME's process rewards still provides a 3.6-point gain. This suggests that the learned value model and the implicit process rewards capture different aspects of trajectory quality — the former estimates expected returns, the latter provides direct per-token quality signals — and they are complementary rather than redundant.

PRIME Scales with More Compute: Extended Training and Larger Rollout Budgets

Figure 6 presents two scaling experiments:

Extended training (Figure 6a): Training PRIME and RLOO for 800 rollout steps (3200 gradient steps, since each rollout step includes one policy update), substantially beyond the default 240–592 steps. PRIME's test accuracy rises from approximately 30% (at step 0, the SFT baseline) to approximately 39% at step 800, while RLOO rises from 30% to approximately 35.3%. The gap between PRIME and RLOO widens over training: at step 240, PRIME leads by roughly 2.5 points; at step 800, PRIME leads by roughly 3.7 points. PRIME's curve continues to rise throughout the 800 steps without evidence of plateauing, while RLOO's curve flattens after step 400. The paper reports a 3.7% improvement for PRIME over RLOO at 800 steps (Section 5.2).

Larger rollout budgets (Figure 6b): The number of responses per prompt is increased from the default K=4 to K=16, keeping other hyperparameters constant. Training proceeds for 600 rollout steps. PRIME's test accuracy rises to approximately 38.5% vs. RLOO's approximately 34.1% — a 4.4% improvement (Section 5.2). Compared to the K=4 setting in Figure 6a, the absolute accuracies are slightly lower at comparable step counts (roughly 35% at step 200 for K=16 vs. roughly 37% for K=4), which the paper does not comment on — this may reflect the challenge of optimizing with more diverse rollouts per prompt, or it may simply be that the larger batch of 16 responses requires more steps to converge.

Both experiments confirm that PRIME's benefits do not saturate at the default training budget, and that the gap between dense and sparse rewards grows with compute, suggesting that dense rewards are increasingly valuable as the policy explores more diverse reasoning paths.

"Zero" RL from Base Model Converges Faster but Saturates Earlier

Appendix B.4, Figures 13 and 14 explore PRIME-Zero: starting RL directly from Qwen2.5-Math-7B-Base or Qwen2.5-32B-Base without any SFT warmup.

7B model (Figure 13): PRIME-Zero achieves rapid initial gains — training rewards rise from approximately 0.37 to approximately 0.51 within 20 steps, and test accuracy surpasses Qwen2.5-Math-7B-Instruct (43.8%) within 32 steps, reaching approximately 45%. However, the training reward curve saturates quickly and shows oscillatory behavior after step 50, and test accuracy plateaus around 44–46% for the remaining 170 steps. In contrast, PRIME from the SFT model (Eurus-2-7B-SFT) starts lower but rises more steadily and surpasses PRIME-Zero by step 100, reaching approximately 48% by step 224.

32B model (Figure 14): PRIME-Zero from Qwen2.5-32B-Base shows even more dramatic initial gains: training rewards rise from approximately 0.28 to approximately 0.56 within 20 steps, and test accuracy surpasses Qwen2.5-32B-Instruct within 16 steps, reaching approximately 51% by step 96. The curve shows less saturation than the 7B case, continuing to rise gently from step 50 to step 96.

The paper interprets these results as (1) confirming that larger models benefit more from RL (consistent with DeepSeek-AI et al., 2025); (2) demonstrating that the SFT warmup is not strictly necessary — the base model's pretrained representations contain sufficient signal for the PRM to bootstrap; and (3) identifying a saturation problem in the "Zero" setting that limits further improvement, "possibly attributed to the decrease of response diversity" (Appendix B.4). The PRIME-from-SFT approach avoids this saturation by providing a more stable starting distribution through SFT warmup.

Comparison to VinePPO and DeepScaleR

VinePPO comparison (Appendix B.6, Table 6, Figure 15): Using RhoMath-1.1B as the base model and MATH as the training dataset, PRIME is compared to VinePPO (Kazemnejad et al., 2024), which uses average return across trajectories to estimate values. At 96 training steps, PRIME achieves 18.8% validation accuracy on MATH-500 vs. VinePPO's 18.4% — a small absolute improvement. However, the dramatic difference is in wall-clock time: PRIME completes 96 steps in 1.22 hours vs. VinePPO's 13.94 hours — an 11× efficiency advantage. This is because VinePPO's Monte Carlo value estimation requires generating and evaluating many completions from intermediate steps (the ~10× overhead the paper identifies in Section 1), while PRIME's Implicit PRM computes per-token rewards from two forward passes on the already-generated rollout. The accuracy advantage of PRIME over VinePPO is modest (0.4 percentage points at 96 steps), but the efficiency advantage is transformative.

DeepScaleR comparison (Appendix B.7, Figure 16, Table 7): Following the DeepScaleR training pipeline (Luo et al., 2025) using DeepSeek-R1-Distill-Qwen-1.5B as the base model, PRIME is run for only the first stage (8K context length) for 330 steps. DeepScaleR's first stage required 1040 steps (~600 GPU hours on A100); PRIME achieves comparable training rewards in 330 steps (446.7 A800 GPU hours), approximately 1/3 the steps and estimated 25% less GPU time (accounting for hardware differences). On downstream benchmarks, PRIME-DeepScaleR-1.5B-Stage1 achieves 52.0% average (Table 7) versus DeepScaleR-1.5B-Stage1's reported AIME 2024 score of 33.9% and DeepScaleR-1.5B-Preview's 57.0% (at 1750 total steps across all three stages). Against DeepSeek-R1-Distill-Qwen-1.5B (no RL), PRIME improves from 48.9% to 52.0% (+3.1 points). The paper notes that the overhead of PRM updates becomes relatively smaller for long-context training (18% overhead vs. 24% for the main experiments) because the rollout phase dominates the total time.

Ablation Studies and Robustness Checks

PRM loss function: CE vs. DPO (Appendix C.2, Table 8, Figure 18): At 96 training steps, PRIME with DPO loss achieves 32.4% average test accuracy vs. PRIME with CE loss at 32.3% — essentially identical. Training reward curves for the two losses are nearly overlapping (Figure 18a), and test accuracy trajectories track closely (Figure 18b). The paper selects CE for "memory efficiency" (since it avoids computing the reference model's log-probabilities for the rejected response in the DPO pair), indicating that the loss choice is an implementation convenience rather than a performance-critical decision.

Reference model configuration: SFT-ref vs. policy-ref (Appendix B.1, Figures 9, 10): Two strategies for the reference model π_ref are compared: using the frozen initial SFT model (SFT ref) versus using the running policy's old log-probabilities (policy ref, where π_ref = π_θold). Training reward curves for the two variants "are close" (Figure 10), with minor differences at specific steps but no systematic advantage for either. The SFT-ref strategy is chosen as the default because it provides a stable, interpretable baseline and because retaining the initial SFT model naturally also provides the reference for any KL divergence computation if needed.

Single-forward vs. double-forward (Appendix B.2, Figure 11): Double-forward updates the PRM on the current rollout batch and then re-runs the updated PRM to compute process rewards for the advantage estimation, rather than using the old PRM's rewards (single-forward). Figure 11a shows that double-forward improves PRM classification accuracy on training samples (the "after-double forward" curve is consistently higher than "before-double forward"). However, Figure 11b shows that the training reward curves for single-forward and double-forward are "close," with no systematic advantage for either. The paper adopts single-forward as the default to save computation.

PRM initialization: SFT model vs. dedicated EurusPRM (Section 5.1, Figure 4): As discussed in the main results, the SFT-initialized PRM substantially outperforms EurusPRM (trained on 500K additional samples) in both training rewards and downstream accuracy. This ablation is particularly strong because EurusPRM represents a realistic, well-resourced alternative — it was trained on diverse data from multiple model families, which would be considered a best practice in reward model development. The finding that the SFT model without any RM training outperforms it is genuinely surprising and validates the paper's claim that distribution alignment dominates initial reward model quality.

Reward model size (Appendix B.5, Table 5): With Qwen2.5-7B-Base as the fixed policy model, PRIME is tested with Implicit PRMs of varying size: Qwen2.5-3B, Qwen2.5-7B, and Qwen2.5-14B. The 7B PRM achieves the best average performance (32.6% across six benchmarks), followed by 3B (32.0%) and 14B (31.8%). The differences are small (0.6–0.8 percentage points), leading the paper to conclude that "reward model size has limited influence." This is practically important: it means the reward model can be the same size as the policy without penalty, simplifying the system architecture.

Online prompt filtering (Section 3.3, Figure 2): Using RLOO with outcome rewards only, training with and without online prompt filtering is compared. Without filtering, the training reward curve exhibits large oscillations (variance approximately 0.02–0.03 in reward units); with filtering, the curve is substantially smoother and rises more consistently, reaching approximately 0.48 vs. 0.46 at step 200. The paper attributes this to filtering removing prompts where the policy achieves 0% or 100% accuracy (no learning signal) and balancing the correctness distribution for PRM training. The exact filtering thresholds are not specified numerically.

ReST^EM revision model (referenced in the executive summary but not from this paper): This is an analysis from a different paper and is not evaluated in PRIME. The PRIME paper does not investigate revision models of the kind studied in the reference example paper.

Critical Assessment

The Central Claim: PRIME Enables Scalable Online Dense-Reward RL

The paper's headline claim is that PRIME provides a scalable framework for dense process rewards in online RL, achieving 2.5× sample efficiency and 6.9% better performance than outcome-reward-only RL (Section 4.3). The experiments do support this claim for the specific setting tested: 7B-parameter math-specialized model, competition-level math and coding benchmarks, rule-based outcome verification, and approximately 600 training steps. The evidence is multi-faceted — training reward curves (Figure 3a), downstream benchmark trajectories (Figure 3b, Table 1), algorithm-generality experiments (Figure 7, Table 4), and scaling experiments (Figure 6) — and the consistency across metrics and checkpoints strengthens credibility.

However, the claim is narrower than the paper's framing suggests in several important respects:

1. Single model family, single scale. All main experiments use Qwen2.5-Math-7B-Base. The "Zero" experiments extend to Qwen2.5-32B-Base (Appendix B.4) and Qwen2.5-7B-Base (Appendix B.4), and the DeepScaleR comparison uses DeepSeek-R1-Distill-Qwen-1.5B (Appendix B.7), but these are all from the Qwen family. Whether the findings transfer to models with different pretraining distributions, tokenizers, or architectural choices (e.g., Llama, Mistral, Gemma) is untested. The Implicit PRM formulation depends on token-level log-probability ratios between two language models, and its behavior may differ if the reference and PRM models have different calibration properties or if the tokenizer merges tokens in ways that blur reasoning steps. This is not a fatal gap — the paper never claims to have tested multiple model families — but it means the "scalability" claim currently rests on evidence from a narrow slice of model space.

2. The outcome verifier is oracle-level. Both math and coding tasks use rule-based verifiers that provide perfect outcome labels — exact string matching against ground truth answers for math, fraction of test cases passed for code. This is the ideal case for PRIME because the PRM is trained on noise-free binary labels. In domains where outcome labels are noisy (e.g., learned reward models for open-ended generation, human preference labels with annotator disagreement), the PRM's training signal would be less reliable, and the implicit process rewards might amplify noise rather than signal. The paper does not address this limitation, and the "PRIME generalizes to any domain with outcome labels" framing implicitly assumes the outcome labels are as clean as mathematical ground truth.

3. The 2.5× sample efficiency figure requires careful interpretation. The paper states PRIME achieves the same training reward as RLOO in 40% of the steps (Section 4.3). However, the training reward metric is the outcome verifier's accuracy on training rollouts — the same signal used to train the PRM. Since PRIME's PRM is trained to predict this exact outcome, and the policy gradient incorporates both the PRM's predictions and the outcome itself, there is a potential for circularity: the training reward might overstate PRIME's true advantage if the PRM is overfitting to the training distribution. The paper partially addresses this by evaluating on held-out benchmarks (Table 1), where PRIME also outperforms RLOO, but the gap on held-out benchmarks (4.1 points at 240 steps) is smaller than the training reward gap (6.9%). Additionally, the 2.5× figure is measured in training steps, not wall-clock time. Table 2 shows PRIME takes 24% more time per step, so the wall-clock efficiency gain is closer to 2× — still impressive, but the paper's abstract and introduction emphasize the 2.5× figure without the wall-clock caveat.

4. The 4th "Zero" finding raises an unanswered question. PRIME-Zero converges faster than PRIME-from-SFT and then saturates (Appendix B.4). The paper attributes this to "decrease of response diversity" but provides no evidence (no entropy measurements, no diversity metrics). If PRIME's benefit is primarily from accelerating early-stage learning (where the policy is exploring broadly and dense rewards help navigate), but the asymptotic performance is similar, then the practical value of PRIME depends on whether one is compute-constrained (train as fast as possible) or performance-constrained (reach the highest possible accuracy). The paper does not disentangle these regimes, and the saturation in PRIME-Zero — which hits a ceiling around 45% for the 7B model while PRIME-from-SFT reaches 48% — suggests that the SFT warmup provides benefits beyond what PRIME alone can achieve.

The Claim: PRIME Eliminates the Dedicated Reward Model Training Phase

The paper demonstrates that initializing the Implicit PRM from the SFT model outperforms a dedicatedly trained EurusPRM (Figure 4, Section 5.1). This finding is strong and well-supported by the experiment. The EurusPRM comparison is a fair baseline: 500K additional training samples from diverse model families, which represents a significant investment in reward model development. The fact that the SFT-initialized PRM wins is genuinely surprising and practically valuable.

However, there is a nuance: the SFT-initialized PRM starts with zero reward signal for all tokens (since π_φ = π_ref at initialization). This means PRIME's early training steps are effectively doing outcome-reward-only RL (the process reward term in Equation 5 is zero) until the PRM has been updated on enough outcome-labeled rollouts to produce non-zero token-level rewards. The paper does not analyze how many steps this takes or whether the early outcome-only phase is necessary for stability. The rapid rise in PRM accuracy (Figure 5, from 50% to 70% in approximately 50 steps) suggests the warmup is brief, but it means PRIME is not providing dense rewards from the very first step — it is bootstrapping from sparse rewards to dense rewards, which is different from having dense rewards available from the start.

Additionally, the claim that PRIME "eliminates" the dedicated reward model training phase is true for the default configuration, but the paper also tests an online EurusPRM variant (Figure 4) where a dedicatedly trained PRM is updated online. This variant underperforms the SFT initialization but still improves over the offline EurusPRM. This suggests that the core mechanism — online update — is more important than the specific initialization. A practitioner with an already-trained PRM could use PRIME's online update mechanism and still benefit, even if starting from the SFT model is optimal. The paper's framing of "eliminating the reward model training phase" might understate the value of the online update mechanism itself, which is the real innovation.

The Claim: PRIME Is Compatible with Diverse RL Algorithms

The experiments in Section 5.3 (Figure 7, Table 4) convincingly demonstrate that PRIME's process rewards improve REINFORCE, GRPO, PPO, and RLOO. The gains are consistent (all positive) and nontrivial (1.7–4.1 percentage points). This is well-supported with the caveat that all algorithms are tested in a specific configuration (same base model, same SFT warmup, same hyperparameters for the policy optimizer).

A genuine limitation: the paper adapts each algorithm's advantage formula to accommodate process rewards (Equations 7–8 in Appendix B.3), but these adaptations involve design choices — for GRPO, the process reward baseline uses the mean and standard deviation of response-averaged process rewards, while for RLOO it uses leave-one-out. These choices could interact with the algorithms in ways that favor one over another. The paper does not ablate the adaptation strategies, so it is possible that GRPO+PRIME's smaller gain (+1.7) reflects suboptimal adaptation rather than an inherent limitation of GRPO.

The inclusion of the "Implicit PRM as value model" variant (Table 4, row "PRIME as Value Model") is an important negative result: using the Implicit PRM's cumulative log-ratio as a state-value baseline for outcome rewards (equivalent to replacing PPO's critic) achieves only 36.6%, worse than any process-reward-as-additive-reward variant. This sharpens the contribution: the Implicit PRM matters not just because it produces token-level signals, but because those signals are used as additive rewards rather than subtractive baselines in the advantage function. The paper's analysis in Section 5.4 is brief but this finding deserves more emphasis — it is a non-obvious architectural insight that could guide future work.

The Claim About Scaling: PRIME Benefits Grow with Compute

The scaling experiments (Figure 6) show PRIME maintaining or widening its lead over a longer training horizon (800 steps) and with larger rollout budgets (K=16). The claim that "PRIME consistently exhibits stable growth" (Section 5.2) is supported for the tested range, but the range is limited: 800 rollout steps for the 7B model, 96 steps for the 32B model, 330 steps for the 1.5B DeepScaleR comparison. These are relatively modest scales compared to the thousands of steps reported in DeepSeek-R1 (DeepSeek-AI et al., 2025) or the ~1750 steps for DeepScaleR. The PRIME curves do not show obvious saturation (except in the "Zero" setting, where saturation happens quickly), but whether the advantage persists to 2000+ steps is unknown.

The paper does not explore scaling with model size as an independent dimension: all comparisons of PRIME vs. RLOO are within the same model size, and the only cross-model-size PRIME runs are the "Zero" experiments comparing 7B and 32B (which are not directly comparable because the 32B run had different hyperparameters and a shorter training horizon). A critical missing experiment would be: does PRIME's advantage over RLOO increase or decrease as model size grows? If the advantage shrinks, PRIME is most useful for smaller models; if it grows, PRIME is essential infrastructure for large-scale RL.

Missing Experiments and Unexplored Dimensions

No direct comparison to step-level PRM methods. The paper argues that step-level PRM training methods like Math-Shepherd (Wang et al., 2023) and VinePPO (Kazemnejad et al., 2024) are too expensive for online RL, and provides a comparison to VinePPO (Appendix B.6) showing 11× wall-clock advantage. However, there is no comparison where a step-level PRM is trained offline (with the ~10× rollout cost) and then used as a frozen reward model in online RL. This would isolate whether PRIME's advantage comes from the online update mechanism, the implicit reward formulation, or both. The offline EurusPRM experiment (Figure 4) partially addresses this, but EurusPRM is an outcome-trained PRM used as an Implicit PRM, not a step-level PRM used traditionally — so it doesn't answer the question of whether implicit rewards are inherently better than explicit step-level rewards when both are kept frozen.

No analysis of token-level reward quality. The paper relies on PRM classification accuracy (Figure 5) and downstream task performance to validate the PRM's quality, but provides no direct analysis of the token-level rewards themselves. Do tokens corresponding to correct mathematical operations receive positive rewards? Do tokens in incorrect reasoning paths receive negative rewards? Are the rewards well-calibrated (does a reward of +0.5 at a token genuinely indicate a 50% higher chance of eventual correctness)? A qualitative analysis — sampling a few reasoning trajectories and overlaying the per-token rewards — would substantially strengthen the paper's claim that the implicit process rewards are "meaningful" and would help practitioners understand what the PRM is learning.

No reward hacking analysis in the RL setting. The paper's introduction identifies reward hacking as a key motivation for online PRM updates, and Figure 5 shows that offline PRMs degrade in accuracy, which is a precondition for reward hacking. However, the paper does not directly demonstrate that offline PRMs lead to reward hacking behavior — solutions that score highly under the PRM but are incorrect, or degenerate outputs that exploit PRM blind spots. The inference-time scaling literature (referenced in the executive summary of the prior paper) provides clear examples of verifier over-optimization in beam search; similar qualitative analysis for PRIME's RL setting would close the loop between the diagnostic (Figure 5) and the consequence (policy degradation).

No experiments on coding-only or math-only ablation. The training data mixes math (457K problems) and coding (27K problems), and the benchmarks include both (AIME, AMC, MATH-500, Minerva Math, OlympiadBench for math; LeetCode, LiveCodeBench for code). The coding benchmarks show smaller gains from PRIME than the math benchmarks (e.g., LiveCodeBench: +0.8 at 240 steps vs. MATH-500: +5.0 at 240 steps, Table 1). The paper does not ablate whether PRIME's benefits on coding are from the coding training data, transfer from math training, or some combination. A math-only training run evaluated on code benchmarks (and vice versa) would reveal the degree of cross-task transfer and whether process rewards are more valuable for math than for code.

Small sample sizes for some benchmarks. AIME 2024 has only 30 questions (implied by the 26.7% pass@1 corresponding to approximately 8 correct answers at 592 steps, Table 1). A 3.3 percentage point improvement on AIME 2024 corresponds to a single additional correctly-answered question. The paper does not report confidence intervals or statistical tests, and for the smaller benchmarks, the percent-accuracy metric is coarse. This is standard practice in the LLM reasoning literature but worth noting when interpreting the precise magnitude of gains, especially on AIME where small absolute changes can appear large in percentage terms.

Summary Assessment

The experimental evidence supports PRIME's core claims — online-updated implicit process rewards improve sample efficiency and downstream performance across multiple RL algorithms — within the specific regime tested (7B math-specialized model, competition-level math and code, rule-based outcome verification, modest training horizons). The key findings (Figure 3: 2.5× efficiency, 6.9% better reward; Table 1: 4.1-point downstream gain at 240 steps; Figure 4: SFT initialization outperforms dedicated PRM; Figure 5: online PRM reverses calibration drift; Figure 7: benefits generalize across algorithms) are robust and internally consistent. The paper's strongest contributions are the demonstrations that (1) online PRM updates are feasible and essential, (2) the SFT model itself is a sufficient PRM initialization, and (3) the Implicit PRM's token-level outputs are more effective as additive rewards than as value-function baselines. The main weaknesses are the single model family, the reliance on oracle-level outcome labels, and the absence of direct process reward quality analysis — all of which limit the certainty with which one can claim "scalability" beyond the tested setting.

6. Limitations and Trade-offs

1. Difficulty Estimation Cost Is Not Accounted For

The assumption or constraint. The paper's compute-optimal policy depends critically on estimating each prompt's difficulty before allocating test-time compute. The method for doing so — generating 2048 samples per question and averaging PRM final-answer scores to bin questions into difficulty quintiles (Section 3.2) — is extraordinarily expensive. The authors acknowledge this explicitly:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. The reported 4× efficiency gains over best-of-N (Figures 4 and 8) are computed after difficulty is known, without amortizing the cost of learning it. The difficulty estimation step (2048 generations per question) consumes more compute than the largest test-time compute budgets studied (256–512 generations). In a realistic deployment, the total cost would be difficulty estimation + strategy execution, and the former could dominate the latter. The 4× figure should therefore be understood as an upper bound on achievable efficiency that is only realizable if difficulty can be estimated cheaply — a capability the paper does not demonstrate.

What evidence exists in the paper. The paper explicitly flags this concern in Section 3.2 but provides no experiments measuring the total cost (including difficulty estimation) or evaluating cheaper difficulty estimation approaches beyond the oracle vs. predicted comparison, which still uses 2048 samples per question. The predicted-bin curves (using PRM scores instead of ground-truth labels) largely overlap with the oracle-bin curves (Figures 4 and 8), but both require the same number of samples — the prediction simply removes the need for ground-truth answers, not the computational expense.

Mitigation status. The paper suggests future work on "pretraining or finetuning models to directly predict difficulty of a question" (Section 8) and frames the current approach as an exploration-exploitation tradeoff. No cheaper difficulty estimator is developed or evaluated. The mitigation is entirely deferred to future work.

---

2. The Framework Is Validated on a Single Benchmark with a Single Model Family

The assumption or constraint. All experiments — search against PRMs, iterative revisions, compute-optimal allocation, and the FLOPs-matched comparison — use only the MATH benchmark (Hendrycks et al., 2021) with 500 test questions, and only PaLM 2-S* as the base model. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is unverified.

The consequence. Several key findings could be model- or benchmark-specific. The PRM's quality and over-optimization behavior depend on PaLM 2-S*'s output distribution — a model with different calibration properties might exhibit different difficulty-dependent scaling curves. The revision model's ability to learn from in-context incorrect examples depends on the base model's in-context learning capabilities, which vary substantially across model families. The MATH benchmark consists exclusively of competition-level math problems requiring symbolic reasoning; it is unknown whether the difficulty-dependent patterns (beam search hurting easy problems, revisions helping easy problems) generalize to code generation, logical reasoning, or scientific QA. The test set of 500 questions, split into five difficulty quintiles of ~100 questions each, means the compute-optimal policy is selected based on only ~50 questions per fold per bin (with two-fold cross-validation) — a small sample that raises questions about statistical reliability. No confidence intervals are reported.

What evidence exists in the paper. The paper acknowledges the single-benchmark limitation implicitly through its scope but provides no cross-benchmark or cross-model replication. No sensitivity analysis of the number of difficulty bins, the test set size, or the cross-validation procedure is reported.

Mitigation status. The authors acknowledge the model scope but frame it as a deliberate choice to study a "representative" model. No extension to other benchmarks or model families is provided, and no such extension is suggested in the future work section — the suggested future directions focus on combining search with revisions, cheaper difficulty estimation, and self-improvement loops, not on broader validation.

---

3. The ~14× Larger Model Baseline for the FLOPs-Matched Comparison Is Not Compute-Optimally Trained

The assumption or constraint. The FLOPs-matched comparison in Section 7 scales model parameters while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023) rather than the Chinchilla-optimal approach of scaling both parameters and data equally (Hoffmann et al., 2022). The authors acknowledge this:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

Additionally, the larger model uses only greedy decoding — no majority voting, best-of-N, or search — while the smaller model with test-time compute uses sophisticated, compute-optimal allocation strategies. This creates an asymmetric comparison: the smaller model gets the benefit of the paper's best inference-time methods, while the larger model gets no inference-time optimization at all.

The consequence. The ~14× larger model is likely weaker than a properly compute-optimal model trained with the same total FLOPs but optimally allocated between parameters and data. This makes the pretraining baseline weaker than it needs to be, and the reported advantages of test-time compute over pretraining — e.g., +27.8% relative improvement on easy questions at R ≪ 1 (Section 7, Figure 9) — may shrink or reverse against a stronger baseline. The paper's central claim that "test-time compute can substitute for pretraining, with boundaries" is therefore contingent on the specific (non-optimal) pretraining recipe used for the larger model. Additionally, giving the larger model even a modest test-time compute budget (e.g., best-of-8 or majority voting) would create a much fairer comparison. The paper never tests how the larger model would perform with any inference-time augmentation.

What evidence exists in the paper. Section 7 presents the FLOPs accounting and the three R values, all computed relative to the parameter-only-scaled larger model. The results in Figure 9 and the bar charts in Figure 1 show the comparison as described. No sensitivity analysis to the pretraining scaling strategy is provided.

Mitigation status. The authors explicitly acknowledge the limitation in the text and defer compute-optimal pretraining comparisons to future work. The mitigation is fully deferred.

---

4. The Revision Model Has a Structural Correct-to-Incorrect Reversion Problem

The assumption or constraint. The revision model is trained on sequences where all in-context answers are incorrect followed by a correct target (Section 6.1). This creates a training distribution mismatch at test time: when the revision chain produces a correct answer, the model may encounter it in context and — because it was never trained to see correct answers as input — incorrectly "revise" it into a wrong answer. The paper reports:

"approximately 38% of correct answers get converted back to incorrect ones using a naive approach"

The consequence. This reversion rate fundamentally limits the effectiveness of sequential revisions. Without mitigation, extending the revision chain beyond a certain length becomes counterproductive — correct answers produced at step t get undone at step t+1. This means the sequential revision strategy, which the compute-optimal policy favors for easy problems (Figure 7, right), has a built-in reliability ceiling. The paper mitigates this with within-chain selection (majority voting or verifier-based selection across the chain), but these are patches that select the best answer post-hoc rather than preventing the model from making destructive revisions. A model that could recognize when no revision is needed — by being trained on "already correct, do not change" examples — would be strictly more capable but was not developed.

What evidence exists in the paper. The 38% reversion rate is reported in Section 6.1. The paper demonstrates that within-chain selection mitigates the problem to some extent (sequential revisions still marginally outperform parallel sampling in Figure 6, right), but does not measure the residual reversion rate with mitigation in place or quantify how much performance is lost relative to a hypothetical model that never regresses correct answers.

Mitigation status. Partially mitigated via majority voting or verifier-based selection across the chain (Section 6.1). The paper does not propose training the revision model on mixed trajectories (some requiring revision, some already correct) as a more principled solution, nor does it explore inference-time strategies to detect and halt unnecessary revisions. The problem is acknowledged but not solved.

---

5. Hard Problems Remain Essentially Unsolved — Test-Time Compute Cannot Compensate for Missing Capability

The assumption or constraint. The paper's findings on difficulty-dependent strategy effectiveness (Figures 3 and 7) reveal a hard boundary: on the hardest questions (difficulty bin 5), no method makes meaningful progress regardless of compute budget. Section 5.3 reports bin 5 accuracy hovering at 1–3% for all search methods and budgets. Section 6.2 reports bin 5 accuracy at roughly 2–3% irrespective of the sequential-to-parallel revision ratio. Section 7 reports that the hardest problems show a -52.9% relative disadvantage from using test-time compute instead of a larger model at R ≫ 1.

The consequence. Test-time compute amplifies existing capability but does not create it from nothing. If the base model's pass@1 is near zero on a problem class (meaning it produces correct solutions only by accident, if at all), no amount of search or revision will help — there are no correct solutions in the proposal distribution to find or refine. This means the 4× efficiency gains and the FLOPs-matched advantages over larger models are restricted to problems where the base model already has non-trivial competence. For genuinely novel, out-of-distribution, or extremely challenging reasoning tasks that exceed the base model's training distribution, pretraining remains the only viable path. The paper's framework provides no guidance for how to handle such problems, and the compute-optimal policy would correctly allocate zero test-time compute to them (since no strategy helps), but this leaves the core challenge unsolved.

What evidence exists in the paper. The bin 5 results appear consistently across all experiments: Figure 3 (right, bottom panel: near-zero accuracy for all methods), Figure 7 (right, bin 5 line at ~2–3% across all ratios), Figure 9 (bin 5 scaling line flat near 0–5% for revisions, essentially zero for PRM search). The paper is transparent about this finding, including a takeaway box in Section 7 stating that test-time compute is most effective on easy-to-medium problems and that pretraining is "almost always more effective" on hard ones.

Mitigation status. The paper does not attempt to solve the hard-problem limitation and does not propose it as future work. It is treated as a fundamental boundary condition of the approach, which is honest but leaves the most important class of problems — the ones where reasoning improvement is most needed — unaddressed.

---

6. Sequential Revision Strategies Impose Latency That Is Not Analyzed

The assumption or constraint. The paper measures compute in "generations" (number of complete solutions sampled), which is a reasonable proxy for total FLOPs but ignores wall-clock latency. Sequential revisions are inherently serial — each revision depends on the output of the previous one — while parallel best-of-N can be executed simultaneously with sufficient hardware. The compute-optimal policy found in Section 6.2 favors sequential-heavy strategies on easy problems (fully sequential is optimal at lower budgets; at 256 generations the optimal ratio is ~2:1 to 8:1 sequential-to-parallel).

The consequence. A strategy that allocates 128 generations as 64 sequential × 2 parallel takes roughly 64× longer wall-clock time than one that runs 128 parallel samples simultaneously (assuming generation latency dominates and sufficient parallel capacity exists). For latency-sensitive applications — interactive assistants, real-time decision-making, user-facing systems — the sequential-heavy strategies favored by the compute-optimal policy on easy-to-medium problems may be impractical regardless of their FLOPs efficiency advantages. The paper's 4× efficiency claims, measured in generations, may not translate to 4× latency reductions — and in some regimes, the compute-optimal strategy could be slower in wall-clock time than a simpler but more parallel strategy.

What evidence exists in the paper. The paper never discusses latency. Table 2 in the reference example paper provides step-wise timing for RL training, but for the inference-time scaling analysis, only generation counts are reported. The tradeoff between sequential depth and parallel width is analyzed purely in terms of accuracy vs. generation budget (Figure 7), with no consideration of how long each configuration takes to execute. The paper does not report the wall-clock time of any inference-time strategy.

Mitigation status. Not addressed. The paper does not acknowledge the latency-throughput tradeoff as a limitation, and does not suggest latency-aware allocation policies as future work. A practitioner deploying these methods would need to independently evaluate whether the accuracy gains from sequential revisions justify the latency cost for their specific use case.

7. Implications and Future Directions

How This Work Changes the Landscape

PRIME changes the RL-for-reasoning landscape by demonstrating that dense, token-level process rewards can be made practical at scale — a capability that the field's leading research groups had explicitly abandoned as infeasible. The DeepSeek-R1 technical report (DeepSeek-AI et al., 2025) concluded that incorporating PRMs into large-scale RL training was not viable due to the cost of acquiring and maintaining step-level labels. PRIME shows this conclusion was contingent on a specific implementation choice — explicit step-level reward modeling — and that an alternative parameterization (implicit process rewards from outcome-labeled training) sidesteps the bottleneck entirely. This does not make PRIME a paradigm shift in the sense of introducing a new class of algorithms; it is better understood as a re-framing that converts an engineering impasse into a mathematical identity. The Implicit PRM parameterization r_φ(y_t) = β log π_φ(y_t|y_<t) / π_ref(y_t|y_<t) means that any language model trained on binary outcome labels contains latent token-level reward information that can be extracted through a single forward pass. The conceptual shift is from "process rewards require process labels" to "process rewards are a byproduct of outcome-labeled training when the reward model has the right parameterization."

This re-framing has several concrete consequences for how the field thinks about reward modeling:

Reward models become dynamic rather than static artifacts. Prior work treated reward model development as a distinct phase: collect data, train a model, validate its calibration, then deploy it as a fixed critic during RL. PRIME demonstrates that this separation is not just unnecessary but actively harmful — Figure 5 shows that offline PRMs steadily degrade in classification accuracy as the policy distribution shifts (from ~70% to ~58% over 200 steps), while online PRMs improve (from 50% to ~75%). This suggests that reward model quality is fundamentally a function of how recently it has been updated on the policy's outputs, not of how much data it was originally trained on. The practical implication is that reward model training should be continuous and co-located with policy training, not segmented into a preliminary phase.

The SFT model is revealed as a latent PRM. The finding that initializing the Implicit PRM from the SFT checkpoint outperforms a dedicated PRM trained on 500K additional samples (EurusPRM, Figure 4) redefines the relationship between supervised finetuning and reward modeling. It implies that the SFT model already contains representations that distinguish good reasoning from bad, and that these representations can be surfaced through online outcome-labeled training without any dedicated reward model phase. This collapses the development pipeline — pretraining → SFT → RL becomes a single model lineage rather than requiring a separate reward model branch. Table 3 quantifies the practical impact: Qwen2.5-Math-7B-Instruct required 618K in-house reward model training data and a separate 72B reward model; PRIME requires zero RM data and no dedicated RM.

Dense rewards are shown to be complementary to outcome rewards, not redundant with them. The additive advantage formula (Equation 5) demonstrates that process rewards and outcome rewards contribute orthogonal information to the policy gradient. Process rewards sculpt the gradient within trajectories (which tokens to reinforce or suppress), while outcome rewards anchor the direction (correct vs. incorrect). The consistent 1.7–4.1 percentage point gains from adding process rewards across four different RL algorithms (Table 4) — including PPO, which already has a learned value model — indicates that the two signals are not substitutes. This resolves a latent question in the field about whether dense rewards would add value when a clean outcome verifier is available: they do, because they provide per-token credit assignment that outcome-only signals cannot.

The reward-vs-value distinction is sharpened for LLM contexts. Section 5.4 (Figure 8) shows that using the Implicit PRM's token-level outputs as additive rewards (summing them into the return) substantially outperforms using them as value baselines (subtracting them from the outcome reward). This is a non-obvious architectural finding: the Implicit PRM's per-token log-ratios are more informative as direct quality signals than as expected-return estimates. The paper frames this as "PRMs work better than value models in RL for LLMs," which has implications for how actor-critic architectures should be designed for language model training: if a model can produce token-level quality estimates, use them in the reward channel, not the baseline channel.

Sample efficiency becomes a primary axis of comparison for RL algorithms. The paper's emphasis on the 2.5× sample efficiency gain (Figure 3) and 11× wall-clock advantage over VinePPO (Appendix B.6) reflects a maturing of the field: as RL training runs become more expensive (thousands of steps, billions of tokens), the rate of learning becomes as important as the asymptotic performance. PRIME's efficiency gains come from two sources: (1) dense rewards provide more gradient information per trajectory, reducing the number of trajectories needed to learn, and (2) the Implicit PRM's token-level rewards are computed from two forward passes on already-generated rollouts, avoiding the ~10× generation overhead of Monte Carlo step-label estimation methods. This positions implicit process rewards as the most computationally efficient known approach to dense-reward RL for LLMs.

Follow-Up Research This Work Enables

Verifying whether implicit process rewards transfer to non-math reasoning domains. The paper's experiments are exclusively on competition-level math and coding, where outcome labels come from oracle-level rule-based verifiers (string matching on LaTeX expressions, test case execution). The Implicit PRM formulation is task-agnostic — any domain with binary outcome labels could, in principle, use it — but whether the token-level rewards remain meaningful when outcome labels are noisier (e.g., human preference annotations, learned reward models, factual correctness with ambiguous answers) is completely untested. A strong follow-up would apply PRIME to a domain like long-form question answering (where correctness is partial and multi-dimensional), medical diagnosis (where ground truth exists but is probabilistic), or code generation with real-world requirements (where test cases are incomplete). The key measurement would be whether the PRM's token-level accuracy (analogous to Figure 5) rises above chance when trained on noisier outcome labels, and whether the process rewards still improve sample efficiency over outcome-only RL. A negative result here — showing that implicit process rewards amplify label noise rather than signal in domains without oracle verifiers — would establish a critical boundary condition on PRIME's applicability.

Testing whether implicit process rewards scale with model size and training horizon. The paper demonstrates PRIME on 7B and 32B models (the latter only in the "Zero" setting), and for training horizons up to 800 rollout steps. DeepSeek-R1 (DeepSeek-AI et al., 2025) trained models at scales where 7B is a fraction of a single expert, and for thousands of steps. Whether PRIME's advantage over outcome-only RL grows, shrinks, or stays constant as model size increases is an open question with immediate practical relevance for teams planning large-scale RL runs. If the advantage grows with scale — because larger models produce more diverse rollouts, making credit assignment harder and dense rewards more valuable — then PRIME becomes essential infrastructure for frontier model training. If the advantage shrinks — because larger models already have better internal representations that make outcome rewards sufficient — then PRIME is most valuable for smaller, resource-constrained settings. A concrete experiment: replicate the PRIME vs. RLOO comparison (Figure 3) at model sizes spanning 1.5B, 7B, 32B, and (compute permitting) 70B+, holding training data and hyperparameters constant, and measure the multiplicative efficiency gain at each scale. Extend training to 2000+ steps to check for saturation or divergence.

Combining implicit process rewards with structured reasoning formats beyond action-centric CoT. The paper's SFT warmup teaches the model an action-centric chain-of-thought format (ASSESS, ADVANCE, VERIFY, SIMPLIFY, SYNTHESIZE, PIVOT, OUTPUT; Table 9). This format creates natural segmentation points where the PRM might learn more interpretable token-level rewards. But modern reasoning models increasingly use less structured formats — free-form CoT, "thinking" blocks, interleaved code and natural language — and it is unknown whether implicit process rewards are equally effective without explicit action markers. A follow-up study would ablate the action-centric format: train PRIME on the same math dataset with (a) action-centric CoT, (b) standard free-form CoT, (c) code-interleaved reasoning (where solutions mix natural language and executable Python), and (d) minimal-format responses (direct answers with implicit reasoning). Measure whether the PRM's classification accuracy (Figure 5 diagnostic) degrades without explicit structure, and whether the sample efficiency gain over outcome-only RL persists. A negative result — showing that implicit process rewards require explicit reasoning steps to be meaningful — would constrain the approach to structured-format settings and motivate research into format-agnostic process reward extraction.

Using PRM classification accuracy as an online early-stopping signal for RL training. Figure 5 demonstrates that the PRM's accuracy on policy rollouts traces a characteristic curve: rising for online PRMs, falling for offline PRMs. This suggests a practical diagnostic: if PRM classification accuracy begins declining during training, reward overoptimization is imminent, and the policy update should be paused or the PRM update frequency increased. A follow-up study would implement this as an automated controller: at each RL step, compute PRM accuracy on the current rollout batch; if accuracy drops below a threshold (e.g., 60%) or declines for N consecutive steps, increase the PRM learning rate, add a KL penalty to the policy, or pause policy updates while the PRM catches up. The experiment would compare this adaptive controller against fixed-schedule training on downstream benchmark trajectories, measuring whether it prevents the performance degradation that offline PRMs exhibit (Figure 4b) and whether it allows longer training without collapse.

Investigating whether the process reward signal can be distilled into the policy for deployment without a PRM. PRIME requires maintaining a separate PRM during training, which adds ~24% computational overhead per step (Table 2). At inference time, however, the PRM is not needed — only the trained policy is deployed. This raises a natural question: can the information the PRM provides during training be distilled into the policy weights, such that a policy trained with PRIME at a given budget matches or exceeds a policy trained with outcome-only RL at a larger budget, even though neither uses a PRM at inference? The paper already demonstrates this implicitly (Eurus-2-7B-PRIME is evaluated without any PRM at inference; Table 1), but a systematic study would train matched pairs of policies — one with PRIME, one without — at varying compute budgets, then compare their inference-time performance to quantify how much "PRM value" is retained in the policy weights. If the PRIME-trained policy retains most of the advantage even without its PRM, it means the process rewards serve as a training accelerator whose benefits persist into deployment, strengthening the case for adoption.

Practical Applications and Downstream Use Cases

Cost-efficient fine-tuning of open-weight reasoning models. The resource comparison in Table 3 — PRIME requiring 230K SFT examples, zero RM data, and 150K RL queries versus Qwen2.5-Math-7B-Instruct's 2.5M SFT examples, 618K RM data, and a separate 72B reward model — has immediate implications for teams fine-tuning open-weight models on domain-specific reasoning tasks. A medical AI lab wanting to adapt Qwen2.5-7B for differential diagnosis reasoning, or a legal tech company adapting it for statutory analysis, can follow the PRIME pipeline: collect a modest SFT dataset of domain reasoning examples (with or without ground-truth answers, since SFT uses different data than RL), curate a set of problems with verifiable outcomes, and run PRIME directly from the SFT checkpoint. The elimination of the reward model training phase reduces both the computational budget (no need to train a separate 72B model) and the data collection burden (no need to gather RM training examples with preference labels). The paper's finding that the SFT model itself serves as the best PRM initialization means the pipeline has only two stages (SFT then RL), not three (SFT then RM training then RL), reducing the engineering complexity and the number of failure points.

Scaling inference-time compute for deployed models without architecture changes. While PRIME focuses on training-time RL, the Implicit PRM formulation originated in inference-time verification (Yuan et al., 2024b), and the paper's analysis of implicit process rewards at the token level suggests a potential dual-use scenario. A deployed Eurus-2-7B-PRIME model could, at inference time, generate multiple candidate solutions and use a frozen Implicit PRM (the final checkpoint from training) to score tokens within each solution for best-of-N selection — analogous to how the executive summary paper used PRM-guided search for test-time compute allocation, but with the PRM derived from the RL training process rather than separately trained. The advantage is that the inference-time PRM is guaranteed to be in-distribution for the deployed model's outputs, since it was trained on the policy's rollouts throughout RL. This could mitigate the verifier over-optimization problem that the executive summary paper identified for inference-time search. The practical scenario: a math tutoring system using Eurus-2-7B-PRIME generates 8 candidate solutions per student query, scores them with the Implicit PRM, and returns the highest-scoring one, achieving higher accuracy than a single greedy decode with latency acceptable for interactive use (since the 8 generations can be batched).

Bootstrapping reasoning capabilities in resource-constrained settings. The PRIME-Zero experiments (Appendix B.4, Figure 13) demonstrate that starting RL directly from Qwen2.5-Math-7B-Base — skipping SFT entirely — achieves rapid initial gains, surpassing the instruct version of the model within 32 training steps. While the Zero setting saturates quickly compared to the full SFT→PRIME pipeline, the fact that it works at all has implications for settings where SFT data is scarce or expensive. A research group working on a low-resource language or a specialized reasoning domain where curated SFT examples are unavailable could start RL directly from a pretrained base model using PRIME with outcome verifiers (which are often easier to construct than SFT demonstrations — writing test cases for code or answer keys for math problems requires less effort than writing full solution trajectories). The rapid initial improvement could produce a useful model quickly, and the saturation could potentially be addressed by iteratively collecting more diverse prompts or by resetting the PRM periodically, though the paper does not explore these mitigations.

When to Prefer This Method

The paper explicitly positions PRIME against two alternatives: outcome-reward-only RL (RLOO, REINFORCE, GRPO, PPO without process rewards) and offline PRM approaches (training a PRM once and using it as a frozen reward model). The decision logic implied by the paper's results is:

• Prefer PRIME over outcome-reward-only RL when the task requires long, multi-step reasoning where credit assignment is difficult (competition math, algorithmic coding, scientific problem-solving), outcome labels are available via rule-based verification or ground truth, and training compute is the bottleneck rather than wall-clock time. The 2.5× sample efficiency gain (Figure 3) and 4.1-point average downstream improvement at 240 steps (Table 4) make the 24% per-step overhead worthwhile. The evidence for this preference is strongest for 7B-scale models on math reasoning; for coding tasks, the gains are smaller (LiveCodeBench: +0.8 at 240 steps, Table 1), making the tradeoff less clear.

• Prefer PRIME over offline PRM approaches when the RL training horizon exceeds ~50 steps. Figure 5 shows that offline PRM classification accuracy begins declining immediately as the policy distribution shifts, while online PRM accuracy rises and overtakes the offline PRM by step 50–100. For short RL runs (fewer than ~50 steps), an offline PRM may provide better initial rewards and require no per-step PRM update computation. For runs of any meaningful length, online update dominates. The finding that the SFT model itself outperforms a dedicated PRM (Figure 4) means there is no scenario where training a separate offline PRM for PRIME-style RL is beneficial — the SFT initialization is both cheaper and better.

• Prefer the SFT→PRIME pipeline over PRIME-Zero when asymptotic performance matters more than rapid initial gains. PRIME-Zero converges faster and surpasses the instruct model within 32 steps (Figure 13), but saturates around 45% average accuracy while the SFT→PRIME pipeline continues improving to ~48% (Figure 13b) and beyond (43.9% at 592 steps on the 7-benchmark average, Table 1). The SFT warmup provides behavioral scaffolding (the action-centric CoT format) and a more stable initial policy distribution that prevents the early saturation observed in the Zero setting. If the use case prioritizes time-to-deployment over final capability, PRIME-Zero may be preferable; if it prioritizes capability, SFT→PRIME is the better choice.

• Prefer outcome-reward-only RL when the per-step computational overhead of PRIME (24%, Table 2) is unacceptable due to hardware constraints, or when the reasoning task is simple enough that credit assignment is not a bottleneck (e.g., single-step factual QA, short-form translation). The paper does not test PRIME on such tasks, so this preference is extrapolation rather than direct evidence, but the mechanism — dense rewards provide per-token credit assignment — is most valuable when trajectories are long and errors can occur at any step.