RLPR: Extrapolating RLVR to General Domains without Verifiers

The model is expected to generate reasoning content after the ` thinking` marker, then output `</think>`, then the final answer, then `</answer>`. This structured format allows reliable extraction of the generated reasoning `$z$` (everything between ` thinking` and ` response`) and the generated answer `$y$` (everything between ` response` and `</answer>`) for computing the probability reward.

**Temperature and generation settings:**
- **Training:** Temperature 1.0, maximum generation length 3072 tokens. The paper notes "minimal truncation observed" at this length, meaning most responses fit within the budget.
- **Default model (Qwen2.5-7B):** The above template and temperature apply.
- **Llama and Gemma models:** Training and evaluation temperature reduced to 0.6, and the ` thinking` part of the template removed. The paper notes this is done "to prevent generation degradation," suggesting these models were less stable with the full template or higher temperature.

**Batch composition and update schedule.** Each rollout step processes 768 prompts, generating 8 responses per prompt (6,144 total responses). These responses are used for 4 policy updates before the next rollout. This balances exploration (8 samples provides sufficient reward variance estimation) with computational efficiency (processing 768 prompts per step enables distributed training across 32 NVIDIA A100 GPUs, as noted in Appendix A.1).

**Optimizer and hyperparameters (Section 3.1, Appendix A.1):**
- **RL algorithm:** GRPO (Group Relative Policy Optimization) by default.
- **Learning rate:** `$5 \times 10^{-7}$` for the policy model.
- **Entropy coefficient:** `$1 \times 10^{-3}$` — a small entropy bonus is added to the loss to prevent premature convergence to deterministic outputs.
- **KL penalty coefficient:** 0 — the KL divergence penalty (which would constrain policy updates to stay close to a reference model) is explicitly removed. The paper relies on the PPO clipping alone to control policy drift.
- **PPO clip range:** (0.8, 1.27) — the lower bound of 0.8 prevents the policy from rapidly reducing probability of previously sampled responses, while the upper bound of 1.27 prevents rapid probability increase that could lead to entropy collapse.
- **Filtering scale `$\beta$`:** 0.5 — this is the scaling factor applied to the EMA of reward standard deviations for prompt filtering (Section 2.4 and Table 7).

**Evaluation protocol (Section 3.1).** During evaluation, the rollout temperature is set to 1 (matching training). To reduce evaluation variance from sampling, the model is evaluated multiple times per benchmark and the average accuracy is reported as Avg@k, where k is the number of samples per prompt (e.g., Avg@2 or Avg@4 for most benchmarks, Avg@16 for AIME24 in Table 1). For answer grading, the paper uses two verifier models:

1.  **Qwen2.5-7B-Inst server:** For benchmarks with standard multiple-choice or short-answer formats, a Qwen2.5-7B-Instruct model is deployed as an evaluation server to judge answer correctness. The paper notes this is done because "rule-based scoring scripts introduce errors in benchmarks containing question formats beyond multiple-choice."
2.  **GPT-4.1:** For more complex benchmarks (TheoremQA, Minerva), GPT-4.1 is additionally used for evaluation, presumably because these benchmarks involve free-form mathematical expressions or nuanced answers that require stronger judgment capabilities.

**Monitoring metrics (Figure 6 in Appendix A.1.2).** During training, three key metrics are tracked:
- **Mean response length:** Steadily increases throughout training (Figure 6a), indicating the model is developing longer, more detailed reasoning chains. There is "no sign of degeneration" — the length growth is controlled rather than exploding.
- **Format reward:** Measures how well the model adheres to the structured output format (thinking/reasoning within tags). This "quickly learns to follow the response structure" (Figure 6b), reaching near-perfect format compliance early in training.
- **Token entropy:** The average entropy of the model's token-level output distribution. The paper specifically notes that entropy "exhibits neither collapses as a result of the clip-high trick, nor abrupt increases" (Figure 6c), demonstrating that the chosen clip range successfully balances exploration and exploitation.

---

#### RLPR on Verifiable Domains (Combining PR with Rule-Based Rewards)

Section 3.5 explores whether PR can be useful even in domains where rule-based verifiers already work well (mathematics). The insight is that binary verifiers, while reliable for labeling correctness, "lack fine-grained discrimination capability on different responses sharing the same correctness." For example, given the reference answer "200," a generated answer of "199" is qualitatively better than "1" — both are incorrect, but one is much closer to correct. A binary verifier gives both a reward of 0, providing no gradient to prefer "199" over "1."

The paper combines the rule-based verifier score `$f_{\text{verifier}}$` with the probability reward `$\hat{r}$` using a simple summation (details not fully specified, but Table 4 headers indicate "We combine rule-based reward and PR by summarizing advantages"). The results in Table 4 show that training on mathematical data with the combined reward outperforms training with the rule-based reward alone, suggesting that the continuous PR signal provides useful fine-grained guidance even when the binary signal already works well.

This is an important finding because it demonstrates that PR is not merely a fallback for domains without verifiers — it captures complementary information about response quality that binary verifiers miss, and the two reward sources can be productively combined. The authors do not specify the exact combination mechanism (weighted sum, normalized sum, etc.), which is a minor omission, but the qualitative result is clear.

---

#### Summary of Design Choices and Their Justifications

- **Arithmetic mean over product for aggregation:** Robustness to low-probability tokens; avoids the length-constraint limitation of VeriFree; demonstrated via both quantitative AUC comparisons (Figure 4) and the concrete example showing a 0.01→0.05 change producing massive variance under product but minimal variance under mean.
- **Subtraction-based debiasing over direct reward usage:** Isolates reasoning-specific probability improvement from baseline answerability; subtraction directly measures additive contribution with a natural zero point, unlike division which would amplify noise for small baselines.
- **EMA-based adaptive filtering over fixed threshold:** Automatically tracks shifting reward distributions during training; prevents the filter from being too aggressive early or too lenient late; creates an implicit curriculum where prompts enter training as the model becomes capable of producing diverse-quality responses on them.
- **Group-normalized advantages (GRPO) over raw reward gradients:** Normalizes rewards within each prompt group, making the absolute scale of PR less important than relative ranking within the group; eliminates need for reward normalization parameters.
- **Clip range (0.8, 1.27) over standard PPO clip (0.8, 1.2):** The asymmetric upper bound (1.27 > 1.2) is specifically designed to prevent entropy collapse — the tendency of RL-trained language models to become overconfident and deterministic. The higher upper bound allows the model to increase probability of good responses while the lower bound of 0.8 prevents dramatic probability decreases.
- **Zero KL penalty:** Relies solely on PPO clipping for policy constraint; removes the need for a reference model and reduces computational overhead. The moderate learning rate (`$5 \times 10^{-7}$`) and 4-update reuse further constrain policy drift.
- **Qwen2.5-7B-Inst as evaluation verifier rather than string matching:** Acknowledges that rule-based scoring introduces errors on benchmarks with non-standard formats; using a capable instruction-tuned model as the evaluator provides more reliable correctness judgments, though at the cost of introducing a dependency on a secondary model for evaluation (not training).

## 4. Key Insights and Innovations

### Innovation 1: Reframing the Reward Problem from External Verification to Intrinsic Self-Evaluation

The dominant assumption in reinforcement learning for language model reasoning has been that reward signals must be *external* — either a handcrafted rule (for math and code) or a separately trained verifier model (for domains that lack clean correctness criteria). This assumption has constrained the RLVR paradigm to a small set of domains where such external signals can be built, leaving the vast space of general reasoning tasks (scientific explanation, legal analysis, strategic decision-making) untouched. RLPR challenges this assumption at its root.

The paper's fundamental conceptual move is to recognize that the language model *already possesses* a graded reward signal — the token-level probability it assigns to a reference answer given its own reasoning. This is not a trained reward model bolted onto the system; it is an intrinsic property of the policy itself. The key insight (Section 1, paragraph 3) is that $p(y^* | z, Q)$ directly reflects "the LLM's own evaluation of the reasoning reward (i.e., how well the reasoning process leads to the correct answer)." A reasoning trace that genuinely supports the correct answer will make that answer more probable under the model's own distribution; a reasoning trace that is confused, irrelevant, or incorrect will make the correct answer less probable.

This reframing is more than a practical convenience. It changes the nature of the reward from a **binary, external, domain-engineered** signal to a **continuous, intrinsic, model-native** one. The implications are substantial:

- **Coverage.** The probability reward works on any text where a reference answer exists, regardless of domain, format, or length. There is no need for answer extraction, string matching, sandbox execution, or format-specific parsing. The paper demonstrates this breadth: RLPR improves performance across general domains (MMLU-Pro, GPQA, TheoremQA, WebInstruct) and mathematical domains (MATH-500, Minerva) simultaneously, while rule-based RLVR methods (SimpleRL-Zoo, PRIME) improve only on math and show negligible or negative transfer to general benchmarks (Table 1).

- **Granularity.** Binary verifiers collapse all errors into a single "incorrect" category — a generated answer of "199" when the reference is "200" receives the same zero reward as a completely nonsensical answer. The probability reward is fundamentally continuous: a near-correct answer may receive a mean probability of 0.65 while a completely wrong answer receives 0.05. This preserves a *gradient of correctness* that binary verifiers discard. The paper provides empirical evidence for this in Section 3.5, showing that combining PR with rule-based verifier rewards on mathematical data improves over the binary reward alone — the continuous signal captures useful information that the binary signal misses even in domains where the binary signal is reliable.

- **Robustness to answer variation.** The rule-based verifier in Figure 2 fails on answers that are semantically correct but differ in phrasing from the expected format (wrongly labeling both "A, B" and "A" as equally incorrect, despite one being a near-miss and the other being incomplete). The probability reward naturally handles this: if the model generates "Option B, then Option A" when the reference says "B, A," the probability will still be high because the model's own language understanding recognizes these as equivalent. The token-level probability visualization in Figure 3 makes this concrete: the probabilities precisely drop at positions where the generated answer diverges from the reference in a meaningful way.

The crucial diagnostic evidence for this reframing is **Figure 4**, which compares the discriminative quality (ROC-AUC) of different reward types against human judgments. On general-domain prompts, the rule-based verifier achieves only 0.61 AUC — barely better than random for distinguishing correct from incorrect responses. The trained General-Verifier improves modestly to 0.69 but still makes errors on ~31% of judgments. The probability reward achieves 0.83+ AUC across model sizes, with even the smallest Qwen2.5-0.5B model outperforming the trained verifier. This is not a marginal improvement — it is a qualitative shift in reward reliability that makes reinforcement learning on general-domain data feasible for the first time without external verifiers.

This reframing is a **fundamental shift** rather than an incremental improvement. It does not propose a better verifier architecture or a more clever rule-based extraction method — approaches that would still be domain-specific. Instead, it eliminates the verifier concept entirely, replacing it with a signal that emerges from the model's own pre-trained knowledge. The parallel to self-supervised learning in pretraining is instructive: just as masked language modeling showed that supervision signals can be extracted from the data itself rather than from human labels, RLPR shows that reasoning reward signals can be extracted from the model itself rather than from external verifiers.

---

### Innovation 2: Identifying Variance and Bias as the Blockers, Not the Absence of Signal

The observation that LLM probabilities correlate with answer quality is not entirely novel — prior work has used sequence likelihood as a scoring metric, and VeriFree (Zhou et al., 2025) used likelihood as a reinforcement learning reward. What distinguishes RLPR is its **diagnosis of why these prior attempts underperform**: the probability signal exists and is qualitatively useful, but it is corrupted by two specific pathologies (high variance from product aggregation, and bias from baseline answerability) that require specific remedies, not generic penalty terms or data filtering heuristics.

**The variance problem: product vs. mean aggregation.** Prior work using sequence likelihood (the product of per-token probabilities) as a reward implicitly assumed that likelihood is the natural probabilistic quantity to optimize. This assumption is mathematically natural — likelihood is what maximum likelihood estimation maximizes, and it has clean connections to perplexity and cross-entropy. But the paper demonstrates that likelihood is "overly sensitive to minor variations" (Section 2.2). The concrete example in Section 3.4 makes this visceral: probabilities of `$1 \times 10^{-4}$` versus `$1 \times 10^{-5}$` on a single token produce a 10x difference in the product, despite representing nearly identical absolute probabilities. For longer reference answers, the probability of containing at least one low-probability token approaches certainty, making the product reward dominated by the *worst* token rather than reflecting overall answer quality.

The conceptual innovation is recognizing that **what matters for a reward is discriminability within a batch, not probabilistic interpretation**. The arithmetic mean is a deliberately "incorrect" aggregation from a likelihood perspective (it no longer corresponds to sequence probability), but it produces a reward that more reliably ranks responses by quality because it is robust to outliers. This is a practical insight: the ideal mathematical formulation (likelihood) is worse for the actual optimization task than a simpler, less principled alternative (mean). Figure 4 confirms this quantitatively — the mean-based PR achieves higher AUC than the likelihood-based reward for both domain categories.

**The bias problem: reasoning vs. answerability.** Even with mean aggregation, the probability reward conflates two signals: (1) how much the reasoning helps produce the correct answer, and (2) how inherently probable the reference answer is given the question alone. A reference answer of "42" might be highly probable for a math question regardless of reasoning because the model has seen similar answers in its training data; a reference answer of a rare technical term might be low-probability regardless of reasoning quality. The paper formalizes this decomposition in Section 2.3 as $U_r = U_z + U_{\text{others}}$, where $U_z$ is the reasoning contribution and $U_{\text{others}}$ captures question and answer characteristics.

The debiasing solution — subtracting the probability of the reference answer *without* any reasoning context — is technically simple but conceptually non-obvious. It requires computing a second forward pass for each prompt (the "base score" $r'$), adding computational cost. But it isolates the specific quantity that matters for reinforcement learning: **the improvement in probability attributable to the reasoning process**. The clipping to [0, 1] further ensures the reward remains well-behaved when reasoning actually *reduces* the probability of the reference answer (which can happen when the model generates reasoning that contradicts the correct answer).

The ablation in Table 3 quantifies the importance: removing debiasing (using raw $r$ instead of $\hat{r}$) reduces TheoremQA performance from 55.4 to lower values, demonstrating that the bias is not just a theoretical concern but a practical impediment to learning.

**Why this diagnosis matters beyond this paper.** The variance/bias decomposition identifies the *specific failure modes* of intrinsic probability rewards, which is more valuable than a blanket claim that "the probability signal works." It tells future researchers *where to invest effort*: improving aggregation methods for variance reduction, developing more sophisticated debiasing approaches, rather than abandoning intrinsic rewards when they initially underperform. It also explains why VeriFree's likelihood-based approach underperforms RLPR: VeriFree addresses variance through a crude 7-token length filter (discarding most training data) rather than through an aggregation method that is inherently robust to length, and VeriFree does not implement debiasing at all, leaving both failure modes partially addressed.

This is a **diagnostic contribution** — the paper identifies the specific mechanisms that prevent intrinsic probability rewards from working, and designs targeted solutions, rather than proposing an entirely new reward paradigm. It transforms the problem from "how do we get any reward signal in general domains?" to "how do we extract a clean reward signal from a noisy but information-rich source?" — a much more tractable framing.

---

### Innovation 3: Standard Deviation Filtering as a Continuous Analog of Accuracy Filtering

Existing RLVR methods use **accuracy filtering** to remove prompts where all sampled responses are correct or all are incorrect — these prompts provide no discriminative signal because all responses receive the same binary reward. This is straightforward when rewards are binary: you simply check whether all 8 responses in a group pass or all fail the verifier. But with continuous rewards, there is no natural "correctness" threshold — a reward of 0.42 could be high for one prompt and low for another, and the distribution of rewards shifts throughout training as the model improves.

The paper's solution — filter prompts whose reward standard deviation falls below a threshold — is conceptually elegant because it **unifies accuracy filtering and reward variance filtering under a single principle**. Both remove prompts where the model's responses are insufficiently diverse in quality. In the binary case, all-correct or all-incorrect prompts have zero reward variance (all rewards are 1 or all are 0), so standard deviation filtering automatically handles this case. In the continuous case, prompts where all responses achieve roughly the same probability reward (whether high or low) are removed because they provide no learning signal — the model cannot distinguish good from bad responses within that prompt.

The adaptive threshold mechanism (EMA of past reward standard deviations, scaled by $\beta = 0.5$) adds another layer of insight. During early training, when the model's responses on most prompts are similar in quality, the average standard deviation is low, and the threshold is correspondingly low — preventing the filter from removing too many prompts prematurely. As training progresses and the model begins producing more varied responses (some reasoning chains are genuinely better than others), the standard deviation distribution shifts upward, and the threshold rises with it, removing a larger fraction of prompts. This creates an **implicit curriculum**: prompts enter the training distribution when the model becomes capable of producing diverse-quality responses on them. Prompts that are trivially easy (the model always gets near-maximum probability reward) or fundamentally impossible (the model always gets near-zero reward) are excluded regardless of training stage.

The ablation in Table 3 shows that removing standard deviation filtering reduces TheoremQA performance from 55.4 to lower values, confirming it is not merely cosmetic. The paper's framing of this as an "adaptive curriculum learning mechanism" (Section 2.4) accurately captures its function: it is not just about removing noise; it dynamically shapes which prompts contribute to the gradient at each training stage.

This innovation is **incremental but structurally necessary**. It does not change the fundamental reward formulation, but it addresses a genuine challenge that arises specifically from moving from binary to continuous rewards. Without it, the continuous reward RL training would be destabilized by prompts that produce no meaningful reward variation. It is a case study in how a seemingly minor implementation detail (filtering) requires fundamental redesign when the underlying signal type changes.

---

### Innovation 4: Probability Reward as a Complement to, Not Just Replacement for, Rule-Based Verifiers

A natural reading of the paper is that PR replaces verifiers — you use PR in general domains where verifiers don't exist, and you keep using verifiers in math and code where they work well. But Section 3.5 demonstrates something more interesting: **PR captures information that binary verifiers systematically discard**, and combining both signals improves performance even in domains where verifiers are already effective.

The conceptual insight is that binary correctness labels, while reliable, are **coarse-grained**. Given reference answer "200," a generated answer of "199" and "1" receive identical zero reward, even though the former is much closer to correct and reflects a more competent reasoning process (probably a minor arithmetic error rather than a fundamental misunderstanding). The probability reward naturally distinguishes these: "199" will have higher probability under the model's distribution than "1" when conditioned on reasoning that nearly gets the answer right.

This is significant because it suggests the **limits of verifier-based RLVR are partly self-imposed**. The community has converged on binary rewards because they are clean and unambiguous, but this cleanness comes at the cost of discarding the continuous signal that the model's own probability distribution provides. The paper's finding that PR+rule combination outperforms rule alone on mathematical data (Table 4) implies that even in mathematics — the showcase domain for RLVR — the standard approach is leaving performance on the table by using only binary rewards.

The practical upshot is that PR is not merely a fallback for domains without verifiers; it is a **general-purpose reward augmentation** that can be added to any RLVR training pipeline. The authors do not deeply explore the combination mechanism (the paper says they "summarize advantages" without specifying exact weighting), but the qualitative result opens a clear research direction: continuous, model-native rewards as a complement to discrete, external rewards, with the combination potentially outperforming either alone.

This innovation is **incremental in scope** (it does not propose a new mechanism, just a new use for an existing one) but **potentially broad in impact** because it applies to every domain where RLVR is currently used. It transforms PR from a domain-extension technique into a universal performance booster.

---

### Innovation 5: Empirical Demonstration That Intrinsic Rewards Generalize Across Model Families and Prompt Templates

A legitimate concern about intrinsic probability rewards is that they might be **brittle** — tied to specific model architectures, training distributions, or prompt formats in ways that external verifiers are not. A rule-based verifier for math works identically whether the underlying model is Qwen, Llama, or Gemma. Would a probability-based reward trained on one model's probabilities transfer to another? Would different prompt templates (which shift the model's output distribution) cause the reward signal to degrade?

The paper addresses this concern through two empirical demonstrations that, while not a theoretical contribution, constitute an important **robustness finding**:

**Cross-model transfer (Table 1).** RLPR is successfully applied to three distinct model families — Qwen2.5-7B, Llama3.1-8B, and Gemma2-2B — with consistent improvements over both base models and RLVR baselines. On Llama, RLPR achieves 48.5 general reasoning average vs. 44.4 for RLVR; on Gemma, 33.8 vs. 32.4; on Qwen, 56.1 vs. 54.7. The improvements are not uniform (Llama benefits most, +4.1 points; Qwen benefits least, +1.4 points), but the direction is consistent. This matters because it demonstrates that the probability reward does not depend on idiosyncratic properties of a single model's calibration — the underlying capability to assign higher probabilities to higher-quality answers appears robust across model families.

**Template robustness (Figure 5).** The paper tests three prompt templates (from VeriFree, DeepSeek-R1, and a custom variant) and compares RLPR against VeriFree. RLPR maintains consistent performance across all three templates, while VeriFree shows "high sensitivity, with a notable performance drop of 8.0 at step-400 when using $p_1$." The middle and right panels of Figure 5 show that RLPR's response length and training entropy also converge to similar levels across templates, while VeriFree's diverge significantly. The paper attributes this to the mean aggregation resisting the variance amplification that affects VeriFree's product-based likelihood when prompt templates shift the model's token-level probability distribution.

This robustness finding is **empirical rather than theoretical**, but it addresses a practical concern that could otherwise discourage adoption. If intrinsic rewards required careful template engineering or model-specific tuning, they would merely replace one form of domain engineering (verifier construction) with another (template/prompt engineering for reward calibration). The evidence that RLPR works out-of-the-box across models and templates makes it a genuinely practical alternative to verifier-based approaches, not just a proof of concept.

## 5. Experimental Analysis

### Evaluation Methodology

- **Dataset.** RLPR is evaluated on seven benchmarks spanning general-domain and mathematical reasoning. For general domains: MMLU-Pro (1,000 randomly sampled prompts), GPQA-diamond subset, TheoremQA (747 questions after removing 53 multimodal instructions), and a held-out validation split from WebInstruct (638 questions after 10-gram deduplication). For mathematical reasoning: MATH-500, Minerva, and AIME24. The training data consists of 77k non-mathematical prompts from the WebInstruct collection, filtered by GPT-4.1 for difficulty (Section 3.1).

- **Base model(s).** Experiments primarily use Qwen2.5-7B-Base with additional evaluations on Llama3.1-8B-Inst and Gemma2-2B-it. Qwen2.5-7B is chosen as the main model for "fair comparison with most existing methods and thorough evaluation" (Section 3.1, Models). The multi-family evaluation tests whether the probability reward generalizes across architectures with different pretraining distributions and calibration properties.

- **Metrics.** The primary metric is **accuracy** (% of questions answered correctly), reported as Avg@k where k is the number of samples per prompt during evaluation. Most benchmarks use Avg@2 or Avg@4; AIME24 uses Avg@16. Answers are extracted from structured output tags and graded using Qwen2.5-7B-Instruct as an evaluation server, with GPT-4.1 assisting on complex benchmarks (TheoremQA, Minerva). The paper shifts away from rule-based scoring scripts, noting they "introduce errors in benchmarks containing question formats beyond multiple-choice" (Section 3.1, Implementation Details).

- **Baselines.** The paper compares against eight baselines organized into categories:
  - **Base and Instruct models:** Qwen2.5-7B, Qwen2.5-7B-Instruct, Gemma2-2B-it, Llama3.1-8B-Inst.
  - **Math-only RLVR methods:** PRIME (Cui et al., 2025a), SimpleRL-Zoo (Zeng et al., 2025) with both Qwen2.5-Math and Qwen2.5-7B bases, Oat-Zero (Liu et al., 2025b).
  - **Self-reward methods:** TTRL (Zuo et al., 2025) which uses majority voting to assign pseudo-labels.
  - **Verifier-model approaches:** General Reasoner (Ma et al., 2025) which trains a separate 1.5B verifier model distilled from Gemini 2.0.
  - **Verifier-free concurrent work:** VeriFree (Zhou et al., 2025) which uses sequence likelihood as reward with a 7-token length constraint.
  - **RLVR baseline:** The authors train their own RLVR baseline using the same data and setup as RLPR but with rule-based verifiers and accuracy filtering instead of probability reward (Table 1, "RLVR" rows).

- **Generation budget / compute accounting.** The paper does not use FLOPs accounting. Instead, fairness is maintained through identical training configurations: all methods using the same base model train with the same batch size (768 prompts, 8 responses each), learning rate (`$5 \times 10^{-7}$`), and number of policy updates (4 per rollout). Compute is implicitly measured in training steps and GPU-hours (32 NVIDIA A100 GPUs per experiment, Appendix A.1). For inference-time evaluation, temperature is set to 1 (or 0.6 for Llama/Gemma) with max generation length 3072. The paper does not compare methods at equivalent inference budget — all baselines use their default generation settings, and evaluation uses the same sampling budget.

- **Cross-validation / statistical protocol.** There is no k-fold cross-validation or statistical significance testing reported. The paper uses repeated evaluation (multiple samples per prompt, averaged into Avg@k) to reduce variance from stochastic sampling but does not report confidence intervals, error bars, or statistical tests comparing methods. The ablation studies (Table 3) and robustness analysis (Figure 5) compare single training runs across conditions without replication. This is a notable omission for a paper making claims about consistent improvements.

### Main Quantitative Results

#### Overall Performance Across Seven Benchmarks (Table 1)

The headline result is that RLPR, despite using no external verifier, achieves the highest average performance across all seven benchmarks among methods trained on general-domain data. On Qwen2.5-7B, RLPR achieves a general-domain average of 56.1 (MMLU-Pro: 56.0, GPQA: 37.6, TheoremQA: 55.4, WebInstruct: 75.5) and an all-benchmark average of 53.6. This surpasses:

- **RLVR (same data, rule-based verifier):** 54.7 general average, 52.6 all average — RLPR improves by +1.4 on general domains and +1.0 overall. This is the most direct comparison since both use identical training data and setup, differing only in the reward mechanism.
- **General Reasoner (verifier-model approach, 1.5B trained verifier):** 54.8 general average, 52.0 all average — RLPR improves by +1.3 on general domains and +1.6 overall, despite General Reasoner using a separately trained verifier model distilled from Gemini 2.0, while RLPR uses no external verifier at all.
- **VeriFree (concurrent verifier-free approach):** 52.6 general average, 49.4 all average — RLPR improves by +3.5 on general domains and +4.2 overall. The largest single-benchmark gaps are +7.6 on TheoremQA (55.4 vs. 47.6) and +7.5 on Minerva (56.5 vs. 49.0).
- **Best math-only RLVR method (SimpleRL-Zoo with Qwen2.5-7B base):** 52.6 general average, 50.1 all average — RLPR improves by +3.5 on general domains and +3.5 overall. Notably, RLPR achieves 56.5 on Minerva vs. SimpleRL-Zoo's 49.2, despite SimpleRL-Zoo being trained specifically on math data with rule-based verifiers.

On mathematical benchmarks, RLPR remains competitive with math-specialized methods. RLPR achieves 78.0 on MATH-500 (vs. 80.8 for Oat-Zero and 82.1 for TTRL — the two highest). On Minerva, RLPR's 56.5 exceeds Oat-Zero's 52.1, SimpleRL-Zoo's 49.2 (Qwen2.5-7B base), and even SimpleRL-Zoo's 51.0 (Qwen2.5-Math base). On AIME24, RLPR's 16.3 is lower than Oat-Zero's 29.8 and SimpleRL-Zoo's 26.5 (Qwen2.5-Math), suggesting that for the hardest competition-level math problems, specialized math training with binary verifiers retains an advantage.

#### RLPR Across Model Families (Table 1, Gemma and Llama rows)

RLPR's gains are consistent but vary in magnitude across model families:

- **Gemma2-2B-it:** RLPR achieves 33.8 general average vs. 32.4 for RLVR (+1.4) and 24.3 for the base Instruct model (+9.5). The improvement is concentrated in MMLU-Pro (+1.9 over RLVR, +5.6 over base), GPQA (+2.7, +9.2), and MATH-500 (+3.8, +3.8). TheoremQA and WebInstruct show negligible change vs. RLVR.
- **Llama3.1-8B-Inst:** RLPR achieves 48.5 general average vs. 44.4 for RLVR (+4.1) and 40.5 for the base Instruct model (+8.0). The largest improvements over RLVR are on WebInstruct (+8.3), MMLU-Pro (+4.3), and MATH-500 (+2.2). AIME24 nearly doubles from 4.6 to 8.8 — the largest relative improvement on the hardest math benchmark.
- **Qwen2.5-7B-Base:** RLPR achieves 56.1 general average vs. 54.7 for RLVR (+1.4) and 44.9 for the untrained base model (+11.2). The Qwen base model starts from a much stronger position (40.9 all average vs. Gemma2-2B-it's 19.9 and Llama3.1-8B-Inst's 35.6), suggesting the probability reward provides diminishing incremental benefit as the base capability rises.

The Qwen RLVR baseline (55.1 MMLU-Pro, 52.2 TheoremQA, 54.9 Minerva) already achieves strong performance — RLPR's +1.4 general improvement is meaningful but smaller than on Llama (+4.1). This pattern is consistent with the paper's claim that PR "reflects the policy by measuring how likely the LLM is to take the correct action" (Section 1): weaker base models have more room for the probability signal to provide discriminative guidance, while stronger base models already assign high probability to correct answers, compressing the useful reward range.

#### Reward Quality Analysis (Figure 4)

The paper evaluates reward quality using ROC-AUC against human correctness judgments, sampled from WebInstruct and DeepScale datasets with 50 prompts each filtered to include both correct and incorrect responses. Key findings:

- **Rule-based verifier on general data: 0.61 AUC.** This is only slightly better than random (0.50) and confirms the paper's central motivation: "The primary flaw of the rule-based verifier in general domains is that it overlooks correct responses due to its limited capability of processing natural language complexity" (Section 3.3).
- **Rule-based verifier on math data: 0.95 AUC.** The verifier works well where answers are extractable and matchable — exactly the domains where RLVR has succeeded.
- **General-Verifier (trained 1.5B model):** 0.69 on general data (+0.08 over rule-based) but 0.92 on math data (−0.03 vs. rule-based). The improvement on general data is modest; the degradation on math data suggests the fine-tuned verifier loses the precision that rule-based checking provides in domains with clean answer formats.
- **PR (mean probability) on general data:** Ranges from approximately 0.80 (Qwen2.5-0.5B) to 0.86 (Qwen2.5-7B). All model sizes substantially outperform both the rule-based verifier (0.61) and the trained General-Verifier (0.69).
- **PR on math data:** Ranges from approximately 0.96 (Qwen2.5-0.5B) to >0.97 (larger models). Even the smallest model outperforms the General-Verifier on math (0.96 vs. 0.92) and matches the rule-based verifier (0.95).
- **Likelihood reward (VeriFree's approach):** Lower AUC than PR on both general data (approximately 0.76–0.80) and math data (approximately 0.93–0.95), with the gap widening for smaller models. This quantitatively confirms that mean aggregation produces a more discriminative reward than product aggregation.

The critical takeaway from Figure 4 is that **PR achieves high discriminability (0.83+ AUC) on general-domain data where rule-based verifiers effectively fail (0.61 AUC)**. This is the evidentiary foundation for the paper's claim that probability reward makes RLVR viable on general domains.

#### Training Data and Reward Type Analysis (Table 2)

Table 2 compares models trained on mathematical prompts (from PRIME, using rule-based verifiers; Yu et al., 2025) versus general-domain prompts (WebInstruct, using either rule-based verifiers or PR). The results demonstrate two points:

- **General-domain data improves mathematical reasoning when rewarded appropriately.** Training on general-domain data with PR yields +1.9 on TheoremQA and +4.3 on Minerva compared to training on math-only data with rule-based verifiers. This supports the paper's claim that "general-domain data enhances the performance on both benchmarks."
- **Rule-based verifiers fail on general-domain data.** Training on general-domain data with rule-based verifiers yields *lower* performance than training on math-only data with rule-based verifiers — "diminished performance" as the paper states. This quantifies the verifier scalability problem: you cannot simply apply RLVR with rule-based verifiers to general-domain data and expect improvements; the reward quality is too low due to the verifier's inability to handle free-form answers.

#### RLPR on Verifiable Domains (Table 4)

When combining PR with rule-based verifier rewards on mathematical data, the combined reward achieves higher performance than rule-based reward alone across MATH-500, Minerva, and AIME24. Specific numbers are not provided in the main text (only in Table 4), but the paper states the improvement demonstrates that "our proposed probability reward can also improve the utilization of data from verifiable domains like mathematics." This is a secondary finding but conceptually important: PR captures fine-grained quality distinctions (e.g., "199" vs. "1" when the reference is "200") that binary verifiers discard, and the combination of both signals outperforms either alone.

### Ablation Studies and Robustness Checks

**Per-token probability (mean) vs. sequence likelihood (product):** The ablation in Table 3 compares replacing the mean probability reward with sequence likelihood. Likelihood as reward underperforms PR, which the paper attributes to instability from low-probability tokens: "probabilities of `$1 \times 10^{-4}$` versus `$1 \times 10^{-5}$` can lead to a tenfold difference in reward, despite their small absolute difference." VeriFree addresses this by filtering out reference answers longer than 7 tokens, but this "significantly limits the data diversity." PR's mean aggregation avoids this constraint entirely and outperforms likelihood in both reward quality (Figure 4) and downstream task performance (Table 3).

**Reward debiasing:** Table 3 shows that removing the debiasing operation (using the raw probability reward $r$ from Equation 2 instead of the debiased $\hat{r}$ from Equation 4) reduces performance on TheoremQA and MMLU-Pro. The paper does not provide specific numbers in the ablation table within the main text, but states that "performance on both benchmarks is worse with original reward, demonstrating the effectiveness of the debiasing operation." The debiasing step subtracts the baseline probability $r'$ (probability of the reference answer without reasoning context) to isolate the reasoning-specific contribution from the question-answer baseline difficulty. Without this, the reward conflates "this reasoning is helpful" with "this answer is inherently probable given the question."

**Standard deviation filtering:** Removing the standard deviation filtering mechanism (training on all prompts without filtering) reduces performance (Table 3). The paper states this "is important for the final performance of models by removing prompts that do not get diverse responses." Without filtering, prompts where all sampled responses receive similar probability rewards (whether uniformly high or uniformly low) contribute gradient updates with near-zero advantage — the model cannot distinguish good from bad responses within those prompts, and these updates add noise without useful learning signal.

**Template robustness (Figure 5):** RLPR is tested with three different prompt templates ($p_1$ from VeriFree, $p_2$ from DeepSeek-R1, $p_3$ with format requirements moved to the user prompt) using Qwen2.5-3B as the base model. RLPR maintains consistent performance across templates, while VeriFree shows high sensitivity — dropping by 8.0 points at step 400 under template $p_1$. RLPR's response length converges to similar levels across all three templates, and training entropy "remains within a reasonable range with no signs of entropy collapse." The paper attributes RLPR's robustness to mean aggregation, which is less sensitive to template-induced shifts in token-level probability distributions than product-based likelihood.

**Entropy and length correlation with PR:** The paper reports Spearman correlation between PR values and response length (average coefficient −0.060) and between PR values and response entropy (average coefficient 0.059), with only 8% of prompts showing a statistically significant correlation (p < 0.05). This is presented as evidence that the probability reward is not confounded by the model preferring longer or more deterministic responses — it measures response quality, not stylistic properties.

**Cross-model family transfer (Table 1, not a formal ablation but a robustness check):** The successful application of RLPR to Qwen, Llama, and Gemma without architecture-specific modifications demonstrates that the probability reward does not depend on model-specific calibration characteristics. The training hyperparameters are adjusted between families (temperature, template structure, $\beta$ scale for filtering — see Table 7), but the core reward mechanism is identical.

### Critical Assessment

#### Claim: RLPR "extrapolates RLVR to general domains without verifiers" and "consistently improves reasoning capabilities"

The evidence in Table 1 strongly supports that RLPR improves over base models across all seven benchmarks and all three model families. On Qwen2.5-7B, the improvement over the untrained base model is +11.2 points on the general-domain average (from 44.9 to 56.1). On Llama3.1-8B-Inst, it is +8.0 points (40.5 to 48.5). On Gemma2-2B-it, it is +9.5 points (24.3 to 33.8). These are substantial and consistent gains.

However, the claim that RLPR "extrapolates RLVR to general domains" requires closer examination. The RLVR baseline trained on identical data with rule-based verifiers already achieves 54.7 general average on Qwen (vs. RLPR's 56.1). The gap is real (+1.4) but not enormous — RLVR with the WebInstruct data and a rule-based verifier, despite the verifier's documented 0.61 AUC on general data, still produces meaningful improvements over the base model (+9.8 points). This suggests that even a noisy verifier provides a usable learning signal when the training setup (GRPO with group normalization, accuracy filtering, multiple samples per prompt) is robust to reward noise. The paper's framing implies that verifier-based approaches *cannot* work on general data, but the data shows they can work — just not as well as PR. The contribution is better described as *improving* general-domain RLVR rather than *enabling* it.

The stronger quantitative claim is RLPR's advantage over VeriFree (+4.2 all-average, +7.6 on TheoremQA, +7.5 on Minerva). This demonstrates that the specific design choices in RLPR (mean aggregation over product, debiasing over raw reward, standard deviation filtering over length-constrained filtering) produce substantially better results than the closest verifier-free alternative. This is a clean, well-controlled comparison since both methods use only the model's own probabilities.

#### Claim: PR "achieves better reward quality than naive likelihood as a reward"

Figure 4 provides direct quantitative evidence: mean-based PR achieves higher AUC than likelihood-based reward on both general and mathematical data, with the gap larger on general data and for smaller models. The concrete example in the ablation (Section 3.4) of `$1 \times 10^{-4}$` vs. `$1 \times 10^{-5}$` producing 10x reward difference under product is compelling. However, the AUC comparison is based on 50 prompts per dataset with human annotations — a relatively small sample. The paper does not report confidence intervals on AUC values, making it impossible to assess whether the observed differences are statistically significant or could arise from sampling variance. Given that AUC differences of 0.02–0.05 separate methods in Figure 4, and the sample size is 50 prompts, this is a non-trivial concern.

#### Claim: "RLPR even surpasses strong verifier-model-dependent approaches General-Reasoner by 1.6 average points"

This claim from the abstract is supported by Table 1: RLPR achieves 53.6 all-average vs. General Reasoner's 52.0 (+1.6). Breaking this down by benchmark, RLPR leads on MMLU-Pro (56.0 vs. 55.4, +0.6), TheoremQA (55.4 vs. 52.1, +3.3), WebInstruct (75.5 vs. 74.5, +1.0), MATH-500 (78.0 vs. 77.0, +1.0), and Minerva (56.5 vs. 51.7, +4.8). General Reasoner leads on GPQA (37.4 vs. 37.6, −0.2) and AIME24 (16.0 vs. 16.3, −0.3).

The comparison carries several caveats that the paper acknowledges only partially:

1.  **Different training data.** General Reasoner was trained on its own data pipeline with a verifier model distilled from Gemini 2.0. RLPR uses WebInstruct data. The comparison is between two complete systems (data + reward + training), not a controlled ablation of reward mechanisms. It is possible that RLPR's advantage comes partly from the WebInstruct data being higher quality or better curated than General Reasoner's training data.

2.  **Verifier model quality.** The General Reasoner verifier achieves only 0.69 AUC on general data (Figure 4). This is a relatively weak verifier — substantially worse than even the smallest PR model (Qwen2.5-0.5B at ~0.80). The fact that RLPR outperforms a system with a weak verifier is less surprising than the abstract's framing suggests. A stronger verifier (e.g., one achieving 0.85+ AUC) might close or reverse the gap. The paper is not comparing against a verifier-model ceiling; it is comparing against a specific implementation that happens to have a modest-quality verifier.

3.  **No statistical testing.** The +1.6 average margin is small relative to the variance expected from stochastic training. Without error bars, confidence intervals, or multiple training runs, it is unclear whether RLPR would consistently outperform General Reasoner or whether the observed margin is within the noise floor of RL training variance.

#### Claim: "RLPR maintains consistent performance regardless of prompt choice"

Figure 5 supports this for the tested templates, but the claim is based on Qwen2.5-3B with reduced batch size (128) and single update per step — a significantly cheaper training setup than the main experiments. The paper does not demonstrate template robustness at the 7B scale or with full training budgets. Additionally, only three templates are tested, all of which follow a similar structured format (thinking/reasoning tags). The robustness to fundamentally different prompt structures (e.g., few-shot prompting, role-based instructions, chain-of-thought zero-shot prompts) is untested. The claim should be scoped to "within the three tested structured templates" rather than the broader "regardless of prompt choice."

#### Missing experiments and analyses

Several experiments would have strengthened the paper's claims:

- **Multiple training runs with variance reporting.** RL training is known to be high-variance (Sutton and Barto, 2018). Reporting single-run results without confidence intervals makes it difficult to assess whether observed differences between methods are reliable. A minimum of 3–5 runs with mean and standard deviation would substantially strengthen all quantitative claims.

- **Scaling analysis.** The paper tests RLPR at 2B, 7B, and 8B scales but does not systematically vary model size to understand how PR reward quality and downstream performance scale. The Figure 4 analysis shows that larger models produce higher reward AUC, but does not connect this to downstream task performance. Does a 14B model trained with RLPR gain more or less relative improvement than a 7B model? The paper's own framework (Section 3.3: "PR is effective with even small-scale models") suggests diminishing returns with scale, but this is not systematically tested.

- **Comparison against supervised fine-tuning on the same data.** The paper forgoes SFT and directly applies RL to the base model, following "most RLVR practices." But a natural question is whether simply fine-tuning on the WebInstruct reference answers (treating them as supervised targets) would achieve comparable or better performance than RLPR, without the complexity of RL training. This ablation is missing and would clarify whether the RL component is necessary or whether the gains come primarily from exposure to high-quality reasoning data.

- **Analysis of what RLPR actually learns.** The paper monitors response length and entropy during training but provides no qualitative analysis of how reasoning patterns change. Does the model learn to produce more structured reasoning? Does it learn to self-correct? Does it develop verification behaviors? The paper's Figure 7 (pass@k curves in Appendix A.2) shows that RLPR maintains or improves pass@k relative to baselines, but this is a quantitative result without mechanistic insight.

- **Hardest-benchmark ceiling.** AIME24 performance (16.3 for RLPR vs. 29.8 for Oat-Zero) suggests RLPR is not competitive on the hardest competition math problems. The paper acknowledges this implicitly (math-specialized methods lead on AIME24) but does not analyze why. One hypothesis: for problems where the base model's probability distribution over the reference answer is near-zero regardless of reasoning quality (because the model fundamentally cannot solve the problem), the probability reward provides no signal. This would parallel the finding from the analyzed paper in the prompt that test-time compute cannot help on difficulty-bin-5 problems where the base model's pass@1 is near zero. This limitation is important but underexplored in Section 3.5.

- **Ablation on the debiasing computation cost.** The debiasing step requires a second forward pass for each prompt (to compute $r'$ without reasoning context). The paper does not report the computational overhead of this step relative to the main training loop. If debiasing doubles the per-step computation, the effective cost per unit of training progress is higher than the FLOPs comparison in Table 1 suggests.

#### Conditional nature of the contribution

RLPR's strongest contribution is **practical**: it provides a drop-in replacement for verifiers that works across domains without domain-specific engineering. For practitioners who want to apply RLVR to custom reasoning tasks (scientific QA, legal analysis, medical diagnosis), RLPR removes the largest barrier — the need to build a verifier for each domain. The cost is a modest increase in computational complexity (debiasing forward pass, standard deviation tracking) and a reliance on having reference answers available for training data.

The theoretical contribution is more qualified. The paper demonstrates that intrinsic probabilities can serve as rewards, but does not provide a principled understanding of *when* they work and when they fail. The 0.61 AUC of rule-based verifiers on general data establishes that external verifiers are insufficient, and the 0.83+ AUC of PR establishes that intrinsic rewards are better. But the paper does not characterize the failure modes of PR — on what types of prompts does the probability reward produce misleading signals? Under what conditions does the debiasing fail? The qualitative example in Figure 2 shows PR working correctly, but no examples of PR failure are provided.

The claim that RLPR "extrapolates RLVR to general domains" should be understood as demonstrating **feasibility** rather than **solved problem**. The approach works across the tested benchmarks and model families, but the gains over RLVR baselines are modest on Qwen (+1.4), and the approach has not been tested on truly open-ended generation tasks (essay writing, dialogue, creative tasks) where reference answers are either unavailable or fundamentally ambiguous. The paper's title — "extrapolating RLVR to general domains without verifiers" — accurately describes the ambition, but the experimental evidence supports this claim only for structured reasoning benchmarks with clear reference answers, not for "general domains" in the broadest sense.

## 6. Limitations and Trade-offs

### Capability Ceiling on the Hardest Problems: Probability Reward Fails When the Model Cannot Generate the Reference Answer

**The assumption or constraint.** The probability reward relies on the model's token-level probabilities of the reference answer $y^*$ given the reasoning $z$ and question $Q$. This signal is only meaningful when the model can assign non-negligible probability to tokens in $y^*$ — that is, when the reference answer lies within the model's plausible output distribution. For problems where the model fundamentally cannot produce (or even approximate) the correct answer, the probability reward degenerates to a near-zero signal with low variance across all sampled reasoning chains, providing no useful gradient for RL. The paper acknowledges this implicitly when discussing the standard deviation filtering mechanism (Section 2.4): prompts with "consistently all high or all low scores exhibit low standard deviation" and are removed from training.

**The consequence.** The model cannot improve on problems that exceed its base capability ceiling. If the model's pre-trained knowledge is insufficient to assign meaningful probability to the correct answer tokens regardless of reasoning quality, RLPR provides no pathway to acquiring new knowledge or discovering novel solution strategies — the reward is uniformly near-zero, and these prompts are filtered out by standard deviation filtering anyway. This creates a **hard boundary** on the approach: RLPR can amplify existing reasoning capability but cannot create capability where none exists.

**What evidence exists in the paper.** The AIME24 results in Table 1 provide the clearest evidence. On AIME24 (the hardest math benchmark, consisting of competition-level problems requiring non-trivial mathematical insight), RLPR achieves 16.3 — substantially below math-specialized RLVR methods like Oat-Zero (29.8) and SimpleRL-Zoo (26.5 using Qwen2.5-Math base). This gap (13.5 points behind Oat-Zero) suggests that for genuinely difficult problems, the probability reward is less effective than domain-specific binary verifiers with specialized training data. The paper does not break out performance by difficulty bins (unlike the analyzed RLVR scaling paper), so we cannot directly observe whether RLPR's gains concentrate on easier problems and vanish on the hardest ones. However, the standard deviation filtering mechanism (Section 2.4) — which explicitly removes prompts where "the overall standard deviation distribution continuously shifts during training" — is, by design, *removing* the hardest prompts from training as they fail to produce reward variance, which means the model never learns to solve them.

**Mitigation status.** Not addressed. The paper frames RLPR as an extrapolation of RLVR to general domains, not as a method for pushing the absolute capability frontier on the hardest problems. Section 5 acknowledges this in passing: "we will explore more domains, including multimodal understanding and scaling RLPR to larger models," but does not propose a mechanism for RLPR to operate on problems where the base model's probability mass on the correct answer is near zero. Future work could explore hybrid approaches — using binary verifiers or stronger models (e.g., GPT-4.1, which is already used for evaluation) to provide reward signals on hard problems while using PR on easier ones — but this is not developed in the current paper.

---

### Difficulty Estimation Cost Is Unaccounted for in the Standard Deviation Filtering Mechanism

**The assumption or constraint.** The standard deviation filtering mechanism (Section 2.4) requires computing the reward standard deviation across `$8$` sampled responses per prompt at each training step, then maintaining an exponential moving average of these standard deviations, and finally filtering prompts below `$\beta \times \text{EMA}$` (with `$\beta = 0.5$`). While this computation is integrated into the training loop (the 8 responses are generated anyway for GRPO), the *adaptive* nature of the threshold introduces an implicit data efficiency cost: at each training step, a fraction of the 768 prompts in the batch are filtered out after sampling but before gradient computation. The paper never reports what fraction of prompts are filtered at different training stages, making it impossible to estimate how many generated responses are effectively wasted.

**The consequence.** The effective number of training samples used for gradient updates is lower than the reported batch size. If, for example, 30% of prompts are filtered at a given stage, then the per-step effective batch size is `$768 \times 0.7 \times 8 \approx 4,300$` responses instead of the nominal `$6,144$`. This means the computational cost per *useful* gradient update is higher than the headline training FLOPs would suggest. Without reporting the filtering rate over time, the paper's training efficiency claims cannot be assessed. Additionally, the adaptive threshold depends on the EMA of past standard deviations, introducing a hyperparameter (the EMA decay rate) that is not reported and could affect training dynamics across different model families and datasets. The paper reports different `$\beta$` values for different model families in Table 7 (0.5 for Qwen, 0.9 for Llama, 1.0 for Gemma), confirming that the mechanism requires model-specific tuning.

**What evidence exists in the paper.** Table 3 (ablation) shows that removing standard deviation filtering reduces performance, but provides no breakdown of the filtering rate, the number of prompts filtered per step, or the computational overhead. Table 7 shows different `$\beta$` values across model families without explaining how these values were selected or how sensitive performance is to `$\beta$`. Appendix A.1.2 (training logs) does not include a plot of the filtering rate over time. The paper states that the filtering "introduce[s] an adaptive curriculum learning mechanism to improve both the training stability and final performance" (Section 2.4) but provides no curriculum-level analysis — does the filter remove easy or hard prompts predominantly? Does the composition of filtered prompts change over training? None of this is reported.

**Mitigation status.** Not addressed. The paper acknowledges the filtering cost qualitatively (Section 2.4 explains the mechanism) but does not account for it in any efficiency metric. The comparison in Table 1 between RLPR and baselines does not control for effective training samples — if RLPR filters 20% of prompts and RLVR filters 5% (because accuracy filtering uses a fixed threshold that may be less aggressive), then RLPR is being trained on fewer effective samples for the same nominal compute budget, making the comparison potentially unfair in RLPR's *disfavor*. Future work should report filtering rates and ideally compare methods at matched effective-update counts rather than matched nominal batch sizes.

---

### The Debiasing Step Adds Unaccounted Computational Overhead and Its Necessity Is Not Fully Ablated

**The assumption or constraint.** The reward debiasing step (Section 2.3) requires computing the probability reward $r$ for the full response (reasoning + reference answer) and also computing a baseline probability $r'$ by feeding only the reference answer $y^*$ to the model (without reasoning). This second forward pass for $r'$ is a separate invocation of the policy model, processing the same reference answer appended directly to the question prompt. For each prompt in the training batch, this adds one additional forward pass beyond the main generation and probability computation — approximately a 10–20% increase in forward-pass computation per training step (one extra pass for $r'$ compared to ~8 passes for generation and `$r$`-computation on sampled responses, depending on implementation details).

**The consequence.** The debiasing step improves the quality of the reward signal (as shown in the ablation, Table 3), but its computational cost is not tracked, reported, or factored into any efficiency comparison. The paper's comparisons in Table 1 ensure that RLPR and RLVR are trained with identical batch sizes, learning rates, and update schedules, which means RLPR is using *more* total FLOPs per training step than RLVR (which does not require a debiasing forward pass). If the debiasing forward pass adds 10–20% overhead, then for the same wall-clock training time, RLPR would complete ~10–20% fewer training steps than RLVR. The paper reports results at a fixed number of training steps (the x-axis of Figure 5 goes to step 400), not at matched compute budgets, so RLPR may have an unfair compute advantage that inflates its reported performance relative to baselines.

**What evidence exists in the paper.** The paper does not report the computational cost of the debiasing step anywhere. Section 2.3 describes the mechanism mathematically but does not discuss implementation details (e.g., whether `$r'$` is computed in parallel with the main forward pass, whether it is cached across training steps if reference answers are fixed, whether it reuses the same batch or requires a separate mini-batch). Appendix A.1 states that each experiment runs on 32 NVIDIA A100 GPUs but does not report total training time, throughput, or FLOPs. Table 3 (ablation) shows that removing debiasing reduces performance — the core claim that debiasing is necessary is supported — but the efficiency claim (that debiasing is a net win when accounting for its cost) is not evaluated. An ablation comparing RLPR with debiasing vs. RLPR without debiasing but with more training steps (to match total FLOPs) would test whether the debiasing improvement is more than what could be achieved by simply training longer without it. This ablation is absent.

**Mitigation status.** Not addressed. The paper does not discuss the computational cost of debiasing, does not propose efficiency improvements (e.g., caching `$r'$` since the reference answer does not change across training steps for the same prompt, amortizing the cost over multiple training epochs), and does not report total FLOPs or wall-clock time for any experiment. The paper's abstract and introduction frame RLPR as a "simple" framework that "eliminates the need for external verifiers," but simplicity in terms of removal of external components does not necessarily imply computational simplicity — the debiasing step replaces an external verifier forward pass with an internal policy forward pass, which may be a wash or a net increase in computation. This limitation is not critical for the paper's main contribution (showing that intrinsic rewards work), but it matters for practitioners deciding whether to adopt RLPR over alternatives at scale.

---

### Single Training Dataset (WebInstruct) with Unknown Generalization to Open-Ended Generation Tasks

**The assumption or constraint.** The paper's entire training pipeline uses prompts from the WebInstruct dataset (Ma et al., 2025), filtered to remove math-related questions and further filtered by GPT-4.1 to retain only challenging samples (score ≥ 3 on a 1–4 reasoning complexity scale). This dataset consists of structured reasoning questions (multiple-choice, short-answer, extractable-reference-answer formats) drawn from educational domains. The paper's evaluation benchmarks (MMLU-Pro, GPQA, TheoremQA, WebInstruct holdout) all share this structure — they are reasoning-intensive but have clear correct answers that can serve as reference answers $y^*$. The paper never tests on tasks where reference answers are **absent, ambiguous, or multi-dimensional** (e.g., essay writing, dialogue generation, creative story completion, open-ended code generation where multiple implementations are valid, scientific hypothesis generation).

**The consequence.** The paper's claim to "extrapolate RLVR to general domains" (title, abstract, Section 1) overstates the demonstrated scope. The term "general domains" in the paper's usage means "reasoning benchmarks beyond math and code" — not "any natural language task." This is a meaningful extension (MMLU-Pro, GPQA, and TheoremQA cover physics, chemistry, biology, economics, law, and other disciplines), but the core requirement — that a ground-truth reference answer exists and can be evaluated via probability — limits the approach to tasks with **verifiable answers**, which is a subset of "general domains," not the whole space. For tasks where answer quality is defined by coherence, informativeness, style, factuality, or alignment with human preferences rather than correctness, the probability reward cannot be computed because there is no single reference $y^*$ to score against.

**What evidence exists in the paper.** The evaluation benchmarks are all standard multiple-choice or short-answer reasoning benchmarks (Section 3.1). The training data filtering description (Appendix A.3) explicitly targets "highly challenging samples" from educational domains that have reference answers. The paper never discusses open-ended generation tasks, does not propose a variant of PR for tasks without reference answers, and does not discuss the limitation. The mathematical benchmarks (MATH-500, Minerva, AIME24) further reinforce that even the "general-domain" evaluation is restricted to answer-verifiable tasks — it just happens that the answers are free-form natural language rather than mathematical expressions.

**Mitigation status.** Not addressed. The paper's future work section (Section 5) mentions exploring "multimodal understanding and scaling RLPR to larger models" but does not mention extending PR to tasks without reference answers. One could imagine using PR with **sampled pseudo-reference answers** (e.g., majority-voted answers from multiple generations, as in TTRL) or with **multiple acceptable reference answers** (computing max probability across several valid phrasings), but these extensions are not developed. The limitation is not fatal — many important reasoning tasks do have reference answers — but the paper's framing as "general domains" without qualification is misleading relative to the demonstrated scope.

---

### No Statistical Significance Testing and Single Training Runs for All Primary Results

**The assumption or constraint.** The paper reports all main results (Table 1, Figures 4–5, Tables 2–4) from single training runs without replication. There are no error bars on any accuracy number, no confidence intervals, no standard deviations across multiple seeds, and no statistical tests comparing methods. The evaluation protocol uses repeated sampling (Avg@k with `$k = 2$` or `$k = 4$` for most benchmarks) to reduce within-evaluation variance, but this only addresses sampling noise during inference, not training noise from random initialization, data order, or RL stochasticity.

**The consequence.** The reported performance differences between methods may be within the noise floor of RL training variance, which is known to be substantial for language model RL (due to high-variance gradient estimates, sensitivity to random seeds, and non-convex optimization dynamics). The paper's key quantitative claims — RLPR surpassing General Reasoner by 1.6 average points, RLPR improving over RLVR by 1.4 points on Qwen, RLPR outperforming VeriFree by 7.6 on TheoremQA — are all based on point estimates from single runs. If the standard deviation across training runs is, say, ±2 points (a plausible magnitude given reported variance in prior RLVR work like Oat-Zero and SimpleRL-Zoo), then several of these comparisons would not be statistically significant. The ablation study (Table 3) and robustness analysis (Figure 5) similarly compare single runs, making it impossible to distinguish genuine improvements from run-to-run variation.

**What evidence exists in the paper.** None. The paper does not report any measure of statistical uncertainty anywhere. Table 1 provides precise integers and one-decimal-point averages with no indication of variance. Figure 4 (reward quality) uses 50 prompts per dataset but does not report AUC confidence intervals. Figure 5 (robustness analysis) shows performance trajectories for single training runs without shaded regions indicating variance. The training logs in Appendix A.1.2 (Figure 6) show smooth curves for response length, format reward, and entropy — but these are metrics for a single run and do not demonstrate reproducibility.

**Mitigation status.** Not addressed. The paper does not discuss the issue of training variance, does not report multiple runs, and does not qualify its claims with uncertainty estimates. This is a significant omission for a paper making comparative claims across methods with small performance margins (1–4 points on several benchmarks). The code and model weights are released (links in the paper header), which enables reproducibility by other researchers, but the paper itself does not establish that the reported differences are reliable. The practical consequence is that a practitioner adopting RLPR cannot know whether the 1.4-point improvement over RLVR on Qwen is a robust effect or a lucky training run — and whether it is worth the additional implementation complexity of PR computation, debiasing, and standard deviation filtering compared to simply using a rule-based verifier with the same data.

---

### Reliance on Structured Output Format with Template Dependence for Reward Computation

**The assumption or constraint.** The probability reward computation (Section 2.2) requires extracting the generated answer `$y$` from the full response `$o$`, then constructing a modified sequence `$o'$` by replacing `$y$` with the reference answer `$y^*$`, and finally computing token probabilities for this modified sequence. This extraction depends on the model reliably adhering to a structured output format (the `thinking...response<answer>...</answer>` template from DeepSeek-R1, shown in Table 5) where the final answer appears inside explicitly delimited tags. If the model fails to follow this format — generating answers outside the tags, producing malformed XML, or generating content after `</answer>` — the probability computation breaks down because (a) `$y$` cannot be reliably extracted, and (b) the modified sequence `$o'$` may not be a valid input to the model.

**The consequence.** The approach is brittle with respect to the output format. The paper monitors "format reward" during training (Figure 6b) and notes that "the policy model quickly learns to follow the response structure," achieving near-perfect format compliance. However, this means RLPR is essentially training the model to both (a) reason better and (b) strictly adhere to the output template. These two objectives may conflict: the model may learn to produce reasoning that fits the template at the cost of reasoning quality, or the template enforcement may suppress useful reasoning patterns that do not fit the structured format. Moreover, if RLPR were applied to a model family or domain where the template is not naturally followed (the paper had to modify templates for Llama and Gemma, removing the `thinking` part to "prevent generation degradation"), the reward signal would degrade significantly.

**What evidence exists in the paper.** The robustness analysis in Figure 5 tests three prompt templates but all three enforce structured output (thinking/reasoning + answer extraction via tags). The paper does not test a template that allows free-form reasoning without answer extraction — for example, the standard chain-of-thought format used in base model evaluations where the answer may appear anywhere in the response. The Llama and Gemma experiments (Table 7) required template modifications (removing `thinking`) that are not systematically studied — the paper notes this was done "to prevent generation degradation" without analyzing why these model families required different templates or whether the template change affected reward quality. The format reward metric in Figure 6b shows the model reaching near-perfect compliance, but the paper does not report how this varies across templates or model families.

**Mitigation status.** Partially addressed but not resolved. The structured template is a practical engineering choice that works well for Qwen2.5 models, but the paper does not propose a template-independent method for extracting answers for probability computation. An alternative approach — using the probability of the entire response `$o$` rather than just the answer `$y$` — would eliminate the need for answer extraction but would confound reasoning quality with reasoning style and length. The paper does not discuss this tradeoff. The reliance on templates is not a fatal limitation for research purposes (most RLVR work uses similar structured formats), but it limits the approach's applicability to domains where answers cannot be cleanly extracted (e.g., multi-turn dialogue, open-ended generation) and introduces a hyperparameter (template design) that affects reward quality in ways that are not fully characterized.

## 7. Implications and Future Directions

### How This Work Changes the Landscape

RLPR does not propose a new reinforcement learning algorithm, a new model architecture, or a new training objective. It proposes something more fundamental: a **reframing of where reward signals come from** in post-training. The dominant assumption in RLVR—that rewards must be external, whether from handcrafted rules or separately trained verifier models—has been so pervasive that the field has implicitly accepted that RL-based post-training is only viable in domains where such external signals can be engineered. RLPR challenges this assumption by demonstrating that the language model's own token-level probability distribution over reference answers is not just a coarse proxy for correctness but a **high-quality, discriminative reward signal** that exceeds the quality of trained verifier models in general domains (PR achieves 0.83+ AUC vs. General-Verifier's 0.69 AUC in Figure 4) and matches or exceeds rule-based verifiers in mathematical domains.

This is not a paradigm shift on the scale of the Transformer architecture or reinforcement learning from human feedback. It is better characterized as a **diagnostic reframing that removes a structural bottleneck**. The bottleneck was not that general-domain reasoning is harder in principle than math reasoning; it was that the standard reward infrastructure (rule-based verifiers) could not handle free-form natural language answers. RLPR shows that the reward infrastructure was the binding constraint, not the reasoning capability of the models or the quality of available training data. By removing that constraint—literally eliminating the verifier from the RLVR pipeline and replacing it with a computation the model already performs—RLPR makes the entire WebInstruct dataset (230,000+ prompts spanning economics, physics, chemistry, biology, law, and many other domains) usable for reinforcement learning for the first time without domain-specific engineering.

The shift in research priorities that this work implies is significant:

**Toward intrinsic reward signals over external verifiers.** Before RLPR, the natural research direction for extending RLVR was "how do we build better verifiers for more domains?"—either through more sophisticated rule engineering (which the paper shows fails on general data, with rule-based verifiers achieving only 0.61 AUC) or through training larger, better verifier models (which the paper shows plateau at 0.69 AUC and degrade on math). After RLPR, the question becomes "how do we extract and refine the reward signal that already exists inside the model?" This is a fundamentally different research program: instead of building external oracles, we study the calibration, biases, and failure modes of the model's own probability distribution as a reward source. The paper's detailed analysis of variance (product vs. mean aggregation), bias (reasoning-independent answerability), and training dynamics (standard deviation filtering) provides the initial diagnostic toolkit for this program.

**Toward domain unification.** A striking pattern in Table 1 is that RLPR improves both general-domain and mathematical reasoning simultaneously, while math-specialized RLVR methods (Oat-Zero, SimpleRL-Zoo, PRIME) show no improvement on general-domain benchmarks and sometimes regress (PRIME achieves only 39.5 on MMLU-Pro vs. 45.3 for the untrained base model). This suggests that training on diverse general-domain data with an intrinsic reward produces **transferable reasoning capabilities** in a way that training on narrow math data with rule-based verifiers does not. This challenges the implicit assumption that math reasoning, code reasoning, and general reasoning are separate skills requiring separate training pipelines. RLPR demonstrates that a single reward mechanism, applied to diverse data, can improve reasoning across domains—a finding that aligns with the pretraining paradigm where diverse data produces general capabilities, but now extended to the reinforcement learning phase.

**Reconciling the tension between self-reward and reference-based reward.** The paper situates itself between two extremes in the recent literature: self-reward methods (TTRL, Zhao et al., 2025) that use majority voting to assign pseudo-labels without any external reference, and verifier-dependent methods (General Reasoner, PRIME) that rely on trained external models. Self-reward methods work by entropy minimization—collapsing the output distribution onto the majority answer—which the paper characterizes as "problematic for restricting exploration" (Section 4). RLPR avoids this critique because it does not reward conformity to the majority; it rewards *alignment with a reference answer*, which preserves diversity of reasoning paths as long as they converge to the correct answer. The training entropy plot (Figure 6c) showing "neither collapses...nor abrupt increases" is evidence for this claim. At the same time, RLPR avoids the cost and fragility of external verifier models. It occupies a **middle ground**: using external reference answers (which are cheap to obtain for many reasoning tasks) combined with internal probability scoring (which requires no external model), sidestepping both the exploration restriction of self-reward and the domain-dependence of external verifiers.

**What becomes less attractive as a research direction.** The paper's evidence suggests that **training general-domain verifier models**—the approach taken by General Reasoner (Ma et al., 2025) with its 1.5B-parameter verifier distilled from Gemini 2.0—may be an inefficient use of resources. General Reasoner's verifier achieves only 0.69 AUC on general data (Figure 4), which is worse than even the smallest Qwen2.5-0.5B model's PR (approximately 0.80 AUC). Training a separate verifier model requires distilling from a larger model, curating training data, implementing a two-model RL pipeline, and accepting the verifier's domain-specific degradation (General-Verifier drops from 0.95 to 0.92 AUC on math vs. rule-based verifiers). RLPR achieves better results with no external verifier at all. Unless future verifier models can substantially exceed the 0.80–0.86 AUC range that PR achieves natively, the verifier-model approach to general-domain RLVR is difficult to justify on either performance or cost grounds.

The paper also indirectly suggests that **naive likelihood-based rewards** (the approach of VeriFree) are a dead end without the specific variance-reduction techniques RLPR introduces. VeriFree's length constraint (7-token limit on reference answers) discards large portions of training data, and its sensitivity to prompt templates (8.0-point performance drop in Figure 5) makes it fragile in practice. RLPR's design choices—mean aggregation, debiasing, standard deviation filtering—are not arbitrary improvements; they are **necessary conditions** for intrinsic probability rewards to work reliably. Future work that uses model probabilities as rewards without addressing these specific pathologies (variance, bias, training instability) will likely underperform.

---

### Follow-Up Research This Work Enables

**Characterizing when probability rewards fail: difficulty-dependent reward quality analysis.** The paper's AIME24 results (RLPR achieves 16.3 vs. Oat-Zero's 29.8) suggest that PR underperforms binary verifiers on the hardest problems. A critical follow-up would replicate the difficulty-bin analysis from the RLVR scaling paper (see Section 3.2 of that paper) but for reward quality: bin questions by the base model's pass@1 rate, compute PR's AUC within each bin, and measure whether PR's discriminability degrades as problem difficulty increases. The hypothesis is that on problems where the base model assigns near-zero probability to the correct answer across all reasoning chains, PR's variance collapses and the signal becomes uninformative—exactly the regime where binary verifiers (which can still say "wrong" vs. "right") might provide a stronger learning signal. A concrete experiment: evaluate PR's AUC on MATH-500 questions binned by Qwen2.5-7B's pass@1 into quintiles, compare with rule-based verifier AUC in each bin, and measure whether downstream RL training with PR vs. rule-based reward shows difficulty-dependent performance gaps matching the AUC differences. This would establish the **applicability envelope** for PR and inform whether hybrid reward strategies should route hard problems to rule-based verifiers while using PR for easy/medium problems.

**Information-theoretic analysis of debiasing: is the reasoning-independent baseline $r'$ the optimal control variate?** The paper's debiasing method (Section 2.3) computes the baseline probability $r'$ by feeding only the reference answer to the model (no reasoning). The reward is then $\hat{r} = \text{clip}(0, 1, r - r')$. This subtracts one specific source of bias (the inherent answerability of the reference answer given the question), but there may be other latent factors in $U_{\text{others}}$ (Equation 3) that are not addressed—for example, the semantic similarity between the question and reference answer, the frequency of answer tokens in the model's training data, or the syntactic complexity of the reference answer. A thorough follow-up would: (1) decompose the probability reward variance into components attributable to reasoning quality vs. answerability vs. question-answer similarity vs. token frequency using a linear mixed-effects model on a large sample of (question, reasoning, answer) triples; (2) test alternative debiasing strategies such as subtracting the probability conditioned on a *random* reasoning chain, using the mean probability across multiple answer paraphrases rather than a single reference, or using learned baseline functions that take question embeddings as input; (3) measure whether improved debiasing (lower correlation between $\hat{r}$ and $U_{\text{others}}$ components) translates to improved downstream RL performance. The paper's current ablation (Table 3) shows debiasing helps, but doesn't establish that subtraction of $r'$ is the optimal form of debiasing or quantify how much residual bias remains.

**Combining PR with process reward models (PRMs) for step-level credit assignment in general domains.** RLPR provides a sequence-level reward (the average probability of the reference answer tokens). This gives the model feedback on whether its *final answer* is correct, but provides no intermediate feedback on the quality of individual reasoning steps. In mathematical domains, process reward models (PRMs) trained with Monte Carlo rollout supervision (Wang et al., 2023; Lightman et al., 2023) have shown that step-level feedback enables more efficient search and credit assignment than outcome-level rewards alone. The natural extension is to train a PRM using PR as the outcome signal: for each intermediate step in a reasoning chain, compute the probability reward for completing the chain from that step onward (using Monte Carlo rollouts), and train a step-level verifier on these soft PR values. This would produce a **general-domain PRM** without requiring any human step-level annotations—the PR signal serves as the automatic supervision. A concrete experiment: generate 16 reasoning chains per WebInstruct prompt, for each step in each chain sample 16 completions, compute the mean PR for each completion, use these as soft labels to fine-tune the base model as a step-level value predictor, then compare beam search guided by this PR-trained PRM against best-of-N PR-only search on TheoremQA and GPQA. If the PR-trained PRM can discriminate good intermediate steps from bad ones, beam search should outperform best-of-N, especially at higher generation budgets.

**Adversarial robustness of probability rewards: can models learn to game their own probabilities?** A fundamental concern with any self-referential reward signal is that the model may learn to *manipulate* the reward rather than improve its reasoning. In RLPR, this would manifest as the model learning to generate reasoning that increases the probability of the reference answer *without actually reasoning correctly*—for example, by producing reasoning that coincidentally makes the reference answer tokens more probable through surface-level associations (mentioning keywords from the reference answer, producing reasoning that ends with a phrase the reference answer commonly follows) rather than through genuine logical derivation. This is analogous to the reward hacking documented in RLHF (where models learn to produce responses that score highly under a learned reward model without being genuinely helpful) and the verifier over-optimization documented in the RLVR scaling paper (where beam search against a PRM produces degenerate solutions that score highly but are incorrect). A stress test would: (1) train a model with RLPR on a dataset where reference answers are deliberately chosen to be surface-level predictable from question keywords (e.g., multiple-choice questions where the answer can be guessed from lexical overlap between question and answer options); (2) evaluate whether the trained model's reasoning chains actually engage with the problem or merely route to high-probability answer tokens; (3) test whether the model's accuracy transfers to a distribution-shifted test set where the lexical shortcut is removed. If the model learns to exploit surface-level probability correlations, it would demonstrate that PR requires auxiliary constraints (e.g., reasoning length penalties, diversity bonuses, or an independent factuality verifier) to prevent reward gaming.

**Scaling laws for intrinsic probability reward quality and downstream RL performance.** Figure 4 shows that PR's AUC increases with model size (Qwen2.5-0.5B ~0.80 → Qwen2.5-7B ~0.86 on general data), but the improvement from 3B to 7B appears marginal compared to the jump from 0.5B to 1.5B. This raises questions about scaling: does PR reward quality saturate at some model size, or does it continue to improve? Does the downstream performance gain from RLPR scale proportionally with PR quality, or does it saturate earlier? A scaling study would train RLPR with Qwen2.5 models at multiple scales (0.5B, 1.5B, 3B, 7B, 14B, 32B) on identical data, measure both PR AUC (against human judgments) and downstream benchmark accuracy, and fit power laws relating model size to reward quality and downstream performance. This would establish whether larger models benefit *more* from RLPR (because their probability distributions are better calibrated, providing higher-quality rewards that drive more learning) or *less* (because larger models already have high pass@1, compressing the useful reward range, and the improvement over base models diminishes—as hinted by the Qwen (+1.4) vs. Llama (+4.1) vs. Gemma (+1.4) pattern in Table 1). The practical implication: if reward quality saturates at 7B, there is little benefit to running RLPR on larger models; if it continues to improve, scaling PR could be a competitive alternative to scaling verifier model capacity.

**Extending PR to tasks without reference answers via constrained decoding or multiple reference sampling.** The paper's current formulation requires a reference answer $y^*$ to compute the probability reward. This limits applicability to tasks where a correct answer exists and is known during training. However, many important reasoning tasks—essay writing, open-ended code generation where multiple implementations are valid, strategic planning, dialogue—do not have a single reference answer. Two extensions are immediately suggested by the PR framework: (1) **Multiple reference answers**: instead of computing $p(y^* | z, Q)$ for a single reference, compute $\max_{y^* \in \mathcal{Y}} p(y^* | z, Q)$ over a set of acceptable answers (e.g., multiple correct code implementations, multiple valid essay theses). This tests whether PR can handle tasks where correctness is multi-faceted. (2) **Constraint-based PR without explicit references**: for tasks where "correctness" is defined by constraints rather than matching a reference (e.g., "the code must pass these test cases," "the essay must cite at least three sources," "the plan must satisfy preconditions A, B, C"), compute the probability of the generated output satisfying each constraint using the model's own probability assessment (e.g., for code, generate test case assertions and compute their probability; for essays, extract cited sources and compute their probability of being real and relevant). A concrete experiment: on HumanEval, replace the reference answer with a set of unit tests, compute PR as the mean probability of the test assertions given the generated code, and compare RLPR-trained models against standard RLVR (sandbox execution reward) and supervised fine-tuning baselines. If constraint-based PR achieves comparable performance to execution-based rewards, it opens the door to RLPR on any task where success criteria can be expressed as verifiable constraints, even without reference answers.

---

### Practical Applications and Downstream Use Cases

**Custom domain-specific reasoning models without verifier engineering.** An organization with a proprietary dataset of domain-specific reasoning questions (e.g., a legal firm with case law analysis questions, a hospital with diagnostic reasoning cases, a financial services company with regulatory compliance scenarios) currently faces a barrier: they cannot apply RLVR to improve their models on this data because building rule-based verifiers for free-text legal or medical answers is infeasible, and training a specialized verifier model requires annotating tens of thousands of response-quality judgments. RLPR removes this barrier entirely. The organization's existing question-answer pairs—which they already have as part of their domain documentation, training materials, or historical records—can be directly used as training data. The probability reward is computed automatically from the model's own forward pass, requiring no annotation, no verifier training, and no domain-specific engineering. Based on Table 1, a 7B model trained with RLPR on such data can expect ~11-point accuracy improvement on domain-specific reasoning tasks (the gap between Qwen2.5-7B base at 44.9 general average and RLPR at 56.1). For a legal tech company with a base model achieving 55% on bar exam questions, this could translate to ~66%—a practically meaningful improvement without any human annotation cost beyond collecting the reference answers (which typically already exist).

**Data-efficient self-improvement for medium-sized models in resource-constrained deployments.** Table 1 shows that RLPR on Llama3.1-8B-Inst improves the all-benchmark average from 35.6 to 42.3 (+6.7 points) and on Gemma2-2B-it from 19.9 to 26.0 (+6.1 points). These models are small enough to run on consumer GPUs or even edge devices. The practical scenario: a company deploys a 2B–8B model for customer support reasoning tasks (e.g., diagnosing technical issues from user descriptions, routing inquiries based on policy rules). As new edge cases emerge (new products, policy changes, novel failure modes), the company collects question-answer pairs from successful human resolutions. RLPR enables continuous, low-cost fine-tuning: each week, the collected Q&A pairs are used to run RLPR training on the deployed model, improving its handling of recent edge cases without any verifier annotation. The compute cost (32 A100 GPUs for the main experiments, likely reducible for fine-tuning on smaller domain-specific datasets) is modest compared to the alternatives (hiring annotators to judge model responses, building and maintaining rule-based verifiers for each policy domain).

**Improving general-domain reasoning in open-source model releases.** The paper demonstrates RLPR across three model families (Qwen, Llama, Gemma) with consistent improvements, and releases all code, data, and model weights. For open-source model teams (e.g., the Llama, Qwen, and Gemma teams themselves, or community fine-tuning efforts like OpenAssistant), RLPR provides a plug-and-play post-training step that improves the model's reasoning across diverse benchmarks without requiring domain-specific reward engineering. A concrete pipeline: take a pre-trained base model, apply RLPR on the 77k WebInstruct prompts (or the full 230k dataset if math prompts are included), and release the resulting model as a "reasoning-enhanced" variant. Based on Table 1, the Qwen2.5-7B base model gains +11.2 general-domain points, and the resulting model (56.1 average) approaches the Instruct model (52.2 average) on general reasoning while exceeding it on math reasoning—all without supervised instruction tuning or human preference data. For model teams that already have SFT pipelines, RLPR can be added as an additional RL phase after SFT, potentially combining the instruction-following benefits of SFT with the reasoning benefits of PR-based RL.

---

### When to Prefer This Method

The paper positions RLPR against three categories of alternatives: rule-based RLVR (e.g., Oat-Zero, SimpleRL-Zoo), verifier-model-based RLVR (e.g., General Reasoner), and verifier-free likelihood-based RL (e.g., VeriFree). The tradeoffs are implicitly defined by the experimental results rather than stated as explicit decision rules, but they can be extracted:

- **Prefer RLPR over rule-based RLVR when** the target domain involves free-form natural language answers that rule-based verifiers cannot reliably grade. The paper's Figure 4 quantifies this boundary: rule-based verifiers achieve only 0.61 AUC on general-domain data, making them worse than random for reward assignment on ~39% of judgments. In such domains, the rule-based verifier is not just suboptimal—it is actively harmful (Table 2 shows general-domain RLVR underperforms even math-only RLVR when rule-based verifiers are applied to general data). The counter-case: on purely mathematical or code-execution tasks where rule-based verifiers exceed 0.95 AUC, RLPR still works but the cost-benefit calculation shifts—PR provides fine-grained signals that can complement binary verifiers (Section 3.5) but the absolute performance gain is smaller than the jump from unusable verifier to PR in general domains.

- **Prefer RLPR over verifier-model approaches when** the cost of training and deploying a separate verifier model is prohibitive, or when the available verifier training data is insufficient to achieve high discriminability. General Reasoner's verifier achieves 0.69 AUC after distillation from Gemini 2.0, which is a non-trivial engineering effort requiring access to a large frontier model as teacher. RLPR achieves 0.80–0.86 AUC with no verifier training at all. Unless a verifier model can substantially exceed the native PR AUC (which would require a different training methodology than the distillation approach in General Reasoner), RLPR offers better reward quality at lower cost. The counter-case: if a high-quality verifier model already exists for the target domain and inference cost for the verifier is not a concern, combining PR with the verifier (as in Section 3.5) may outperform either alone, so the decision is not "use PR or use a verifier" but "add PR to the existing verifier."

- **Prefer RLPR over VeriFree when** the training data includes reference answers longer than 7 tokens, or when robustness to prompt template variation is important. Figure 5 shows VeriFree dropping 8.0 points under template changes; RLPR maintains consistent performance. Table 3 and Section 3.4 demonstrate that VeriFree's likelihood-based reward is fundamentally limited by the 7-token constraint (most reference answers in general domains exceed this length) and the high variance of product aggregation. RLPR should be preferred in any setting where data diversity matters or where the prompt template cannot be carefully controlled (e.g., multi-lingual deployments, user-facing systems where prompts vary by use case).

- **Do not prefer RLPR when** the base model fundamentally cannot produce or assign meaningful probability to the correct answer (difficulty bin 5 problems in the RLVR scaling paper's taxonomy). On AIME24, RLPR achieves 16.3 vs. Oat-Zero's 29.8 and SimpleRL-Zoo's 26.5. For the hardest problems where the model's pass@1 is near zero regardless of reasoning, the probability reward provides no discriminative signal (all responses get near-zero probability), and binary verifiers—which can still distinguish between a completely wrong answer and a near-miss—retain an advantage. This is not a domain distinction (math vs. general) but a difficulty distinction: RLPR is preferred on problems within the base model's capability neighborhood; specialized verifier-based RLVR with curated training data is preferred on problems at the capability frontier. The practical decision rule: estimate the base model's pass@1 on the target problem distribution; if it is below ~5–10%, RLPR's reward signal may be too weak to drive improvement, and alternative approaches (stronger base model, specialized verifier, or human feedback) should be considered.