How Far Can Unsupervised RLVR Scale LLM Training?

ArXiv: 2603.08660

🎯 Pitch

All intrinsic reward methods for unsupervised LLM reinforcement learning inevitably collapse, not due to engineering flaws but because they mathematically sharpen the model's initial beliefs—amplifying correct predictions early and wrong ones later. External reward methods exploiting compute asymmetries (easy to verify, hard to generate) show they can escape this ceiling entirely. The new Model Collapse Step metric predicts RL trainability 5.6× faster than supervised training, identifying when collapse will strike before you commit the GPU hours.

---

1. Executive Summary

This paper systematically analyzes the scalability of intrinsic Unsupervised RLVR (URLVR) methods—training LLMs with rewards derived from the model's own outputs rather than ground-truth labels—across the Qwen and LLaMA model families on math reasoning benchmarks. The authors establish that all intrinsic reward methods, despite diverse designs (certainty-based rewards such as token-level entropy versus ensemble-based rewards such as majority voting), converge toward a sharpening mechanism that amplifies the model's initial probability distribution, producing a universal rise-then-fall performance pattern where early gains are inevitably followed by model collapse. The paper further introduces Model Collapse Step as a practical indicator that predicts RL trainability using 5.6× fewer tokens than full supervised RL training, while also demonstrating that external reward methods grounded in generation-verification asymmetries (self-verification on Countdown arithmetic puzzles) sustain improvement without collapse, establishing that intrinsic URLVR can scale safely only when applied to small, domain-specific datasets in test-time training and cannot create capabilities beyond what the model's prior distribution already encodes.

2. Context and Motivation

The Core Problem: The Supervision Bottleneck in Reinforcement Learning for LLMs

The paper addresses a fundamental scalability challenge facing one of the most successful paradigms in LLM post-training: reinforcement learning with verifiable rewards (RLVR). In standard RLVR, models receive rewards based on whether their outputs match ground-truth answers — for instance, a math problem gets a reward of 1 if the final answer is correct, and 0 otherwise. This approach has driven remarkable breakthroughs in reasoning capability, most visibly in models like DeepSeek-R1 (Guo et al., 2025), Gemini 2.5 (Comanici et al., 2025), and the Qwen3 series (Team, 2025; Yang et al., 2025), which achieved state-of-the-art performance on mathematics, coding, and science benchmarks by scaling supervised RLVR.

The problem is that scaling supervised RLVR hits a hard ceiling. As the authors frame it (Section 1):

"scaling supervision requires prohibitively high human costs, and as models reach or surpass human expertise in specialized domains, obtaining reliable ground truth supervision becomes increasingly infeasible"

This is not a hypothetical concern. Consider what happens when an LLM is being trained to solve frontier mathematical problems, verify complex proofs, or write sophisticated software. At some point, the model's outputs surpass what human annotators can reliably evaluate, either because the problems are too hard for the annotators themselves, or because the volume of required labels grows faster than annotation budgets. The RLVR pipeline — which has been the engine of recent reasoning breakthroughs — therefore has a built-in expiration date: it works brilliantly until the model gets too good for human supervisors.

This is the supervision bottleneck: as models approach or exceed human expert performance, the supply of verified ground-truth labels dries up, capping further improvement. The paper positions this as a fundamental roadblock on the path to superintelligence (citing Burns et al., 2023; Silver and Sutton, 2025), and one that motivates a search for reward signals that do not depend on human-provided labels.

Why This Problem Matters: The Promise and Peril of Unsupervised RLVR

The paper sees the supervision bottleneck not just as a practical inconvenience, but as a conceptual motivator for a new paradigm: Unsupervised RLVR (URLVR). The analogy the authors draw (Section 1) is to pretraining scaling laws. Just as the discovery that language modeling loss scales predictably with model size and data volume (Brown et al., 2020; Raffel et al., 2020) unlocked the era of large-scale pretraining on unlabeled text, the hope behind URLVR is that something analogous happens for post-training: that reinforcement learning can be decoupled from human labels and made to scale with computation and unlabeled data instead.

The paper defines its problem setting explicitly in Section 2:

"We investigate reinforcement learning for verifiable tasks where ground-truth labels are difficult to obtain. In Unsupervised RLVR, models must learn from proxy reward signals derived without relying on human efforts."

The distinction from general "self-rewarding" methods (Huang et al., 2024; Yuan et al., 2024) is important. The paper focuses on tasks with verifiable properties — math problems with deterministic answers, code with executable test cases, puzzles with checkable solutions — where there exists some notion of correctness even if the human label is unavailable. This scoping is deliberate: it separates the paper's analysis from the much broader (and less well-defined) space of open-ended self-improvement, keeping the investigation anchored in domains where success and failure can be measured cleanly.

The stakes are high because URLVR promises to extend the RLVR scaling curve beyond the human annotation frontier. If it works, the same reinforcement learning machinery that produced DeepSeek-R1 and Qwen3 could continue improving models on problems that no human can solve, using rewards derived from the model itself, from unlabeled data, or from computational verification procedures. If it doesn't work — if URLVR hits its own, earlier ceiling — then the field needs to know that now, not after wasting enormous compute on an unscalable approach.

The Landscape of Existing Intrinsic URLVR Methods

Prior to this paper, a significant body of work had already explored intrinsic reward methods — approaches where the reward signal is computed entirely from the model's own outputs, without reference to any external ground truth. The paper organizes these into two categories (Section 2.1, Tables 1–2):

Certainty-based methods derive rewards from the model's confidence in its own predictions. These include:

• Self-Certainty (RLIF; Zhao et al., 2025b): rewards outputs where the model's next-token distribution has high KL divergence from a uniform distribution — essentially rewarding "confident" (peaked) predictions.
• Token-Level Entropy (EM-RL; Agarwal et al., 2025; RENT; Prabhudesai et al., 2025): directly penalizes high entropy at each token position, encouraging low-uncertainty generations.
• Trajectory-Level Entropy (EM-RL; Agarwal et al., 2025): aggregates log-probability across the entire output sequence, rewarding sequences the model assigns high overall probability.
• Probability (RLSC; Li et al., 2025a): uses the raw product of token probabilities as the reward, which is simply the exponentiated trajectory-level entropy.
• Probability Disparity (RLSF; van Niekerk et al., 2025): rewards the gap between the top-1 and top-2 token probabilities, capturing distribution sharpness.

All of these are, at their core, different mathematical formalizations of the same intuition: models that are "confident" (producing peaked, low-entropy distributions) are more likely to be correct, so rewarding confidence should improve correctness.

Ensemble-based methods derive rewards from agreement across multiple independent rollouts:

• Majority Voting (TTRL; Zuo et al., 2025; SRT; Shafayat et al., 2025; ETTRL; Liu et al., 2025, and others): generates N independent solutions, computes the most common final answer, and rewards all outputs that match this majority answer.
• Semantic Clustering (EMPO; Zhang et al., 2025b): uses soft majority voting based on semantic similarity rather than exact answer match.
• Trajectory Consistency (CoVo; Zhang et al., 2025a): derives rewards from whether intermediate reasoning steps are consistent across rollouts.
• Multi-model and paraphrased variants: Co-Reward (Zhang et al., 2025d) augments majority voting by also sampling on rephrased questions; RLCCF (Yuan et al., 2025) incorporates multiple model agents; SeRL (Fang et al., 2025) and R-Zero (Huang et al., 2025) build asymmetric proposer-solver architectures.

The common assumption across all ensemble methods is that consensus correlates with correctness — if many independent rollouts agree on an answer, that answer is probably right.

The Conflicting Signals: Early Gains and Growing Concerns

The critical context for this paper is that the existing literature on intrinsic URLVR painted a confused and contradictory picture. On one hand, multiple papers reported encouraging early training gains. TTRL (Zuo et al., 2025) showed that majority-voting rewards could improve math reasoning in test-time settings. EM-RL (Agarwal et al., 2025) and RENT (Prabhudesai et al., 2025) demonstrated that entropy minimization alone — without any correctness signal — could boost reasoning performance. These results suggested that intrinsic rewards might be a viable path to scaling post-training without labels.

On the other hand, red flags were accumulating. The same studies that reported gains also documented worrying failure modes:

• Reward hacking: Agarwal et al. (2025), Shafayat et al. (2025), and Zhang et al. (2025c) all observed that models trained with intrinsic rewards eventually learned to maximize the proxy reward signal without improving actual correctness — the classic pattern where the agent exploits the reward function rather than learning the intended behavior.
• Model collapse: Training dynamics showed abrupt performance degradation after an initial improvement phase, with the model's outputs becoming increasingly deterministic but decreasingly correct.
• Methodological fragmentation: Different studies used different model families (GPT-4, Qwen, LLaMA), different reward formulations, different datasets, and different evaluation protocols, making it impossible to compare results or extract general principles.

The field was thus in a state of genuine uncertainty. Were intrinsic rewards a viable scaling path that just needed better engineering, or were they fundamentally limited in a way that no amount of tuning could fix? The paper frames this as the central question (Section 1):

"This raises a fundamental question for the field: Can intrinsic rewards truly scale LLM training?"

The structure of the paper — taxonomy → theory → systematic experiments → boundary characterization → alternatives — is designed to answer this question definitively, distinguishing between what is fixable and what is fundamental.

Where Prior Work Falls Short: The Missing Pieces

The paper identifies several specific gaps that prevent the existing literature from resolving the uncertainty around URLVR scalability:

No unified theoretical framework. Despite the proliferation of intrinsic reward methods, there was no analysis of why different methods produce similar patterns (early gains, later collapse), or what mechanism underlies the observed dynamics. Each paper proposed its own reward and evaluated it in isolation. The community lacked a theoretical lens for understanding what all these methods have in common, why they sometimes help and sometimes hurt, and under what conditions they can be expected to work.

No systematic comparison across methods and models. The diversity of experimental settings made it impossible to determine whether failure was caused by the reward formulation, the model architecture, the dataset, or the hyperparameters. The paper notes that "diverse methodologies have been applied across different model families and evaluation settings without systematic comparison or consensus on what constitutes reliable unsupervised rewards" (Section 1). Without controlled experiments, the field couldn't distinguish between engineering problems (tunable) and fundamental limitations (untunable).

No characterization of boundary conditions. Even the positive results in prior work were reported without clear boundaries. When does URLVR work and when does it fail? Does it depend on model size? Training stage? Dataset difficulty? These questions were underexplored, making it hard for practitioners to decide when (or whether) to use intrinsic rewards.

No practical metrics for anticipating success or failure. The standard approach for selecting a base model for RL training is to run full supervised RLVR on multiple candidates and pick the best — expensive and slow. The community lacked a cheap, reliable indicator of whether a given model would benefit from RL training.

Limited exploration of alternatives to intrinsic rewards. While some prior work had begun exploring external reward sources — self-verification (Shao et al., 2025b), execution-based code rewards (Zhao et al., 2025a), puzzle-rule verifiers (Simonds et al., 2025) — these were studied independently and not systematically compared against intrinsic methods. The relative scaling properties of intrinsic versus external URLVR were unknown.

How This Paper Positions Itself

The paper positions itself as a comprehensive analysis and synthesis rather than a new method proposal. Its explicit goal (Section 1) is to "conduct a comprehensive study of URLVR, spanning taxonomy, theory and extensive experiments," with the aim of resolving the field's uncertainty about whether intrinsic rewards can scale.

The structure reflects this analytical ambition:

1. Taxonomy (Section 2): Classify all existing URLVR methods into intrinsic versus external based on reward source, providing a unified vocabulary and organizing framework.
2. Theory (Section 3): Derive the sharpening mechanism that the authors argue is the common underlying dynamic of all intrinsic methods, showing mathematically that the model converges toward amplifying its initial distribution regardless of reward formulation specifics.
3. Empirical validation of theory (Section 4): Demonstrate through controlled experiments across multiple methods, hyperparameters, and models that the rise-then-fall pattern is universal and that failure is a matter of when, not if. This directly answers the "can intrinsic rewards scale?" question with a negative.
4. Boundary characterization (Sections 5–6): Show where intrinsic rewards can still be useful (small datasets, test-time training) and introduce Model Collapse Step as a practical diagnostic tool.
5. Path forward (Section 7): Contrast intrinsic methods with external reward approaches, providing preliminary evidence that generation-verification asymmetries may escape the confidence-correctness ceiling.

The paper's central theoretical claim — that all intrinsic methods converge toward sharpening the model's initial distribution — is what unifies the taxonomy and explains the empirical patterns. If confidence in a particular answer is already high and that answer happens to be correct, sharpening improves performance (early gains). If confidence is high but the answer is wrong, sharpening amplifies error. And if the model's distribution becomes nearly deterministic on incorrect answers, performance collapses. This mechanism explains not only that collapse occurs, but why it is inevitable: the model cannot, by consulting only its own output distribution, determine whether its confidence is justified. The reward signal is fundamentally bounded by what the model already "knows" (or believes).

By distinguishing between the fixable (hyperparameter sensitivity, dataset size effects, training duration) and the fundamental (the sharpening mechanism, the confidence-correctness alignment ceiling), the paper aims to redirect the field's energy. The message is not that URLVR is hopeless, but that intrinsic rewards have well-defined boundaries beyond which they should not be pushed, and that external verification mechanisms — grounded in computation rather than model confidence — represent the more promising direction for long-term scaling.

This framing also explains the paper's choice to include the self-verification experiments on Countdown (Section 7). These serve not as a full-blown alternative method but as a proof of concept that external rewards exhibit qualitatively different scaling behavior — sustained improvement without collapse — validating the theoretical distinction drawn in the taxonomy and providing a concrete direction for future work.

3. Technical Approach

3.1 Reader Orientation (Approachable Technical Breakdown)

This paper is primarily an analytical framework and empirical investigation, not a new algorithm — it builds a unified mathematical lens for understanding why all intrinsic URLVR methods behave similarly (early gains, then collapse), and then systematically validates that this behavior is universal across methods, hyperparameters, models, and datasets. The system being analyzed is the reinforcement learning training loop with intrinsic (model-derived) rewards instead of ground-truth labels, and the paper's core technical contribution is formalizing the sharpening mechanism — a geometric convergence process that amplifies the model's initial output distribution — which explains both the strengths and the fundamental limitations of intrinsic URLVR.

3.2 Big-Picture Architecture (Diagram in Words)

The paper's technical apparatus has four major components that interact in a specific analytical flow:

1. Taxonomy and Unified Reward Framework (Section 2, Appendix A.3) — a classification system that maps diverse intrinsic reward formulations (Self-Certainty, Token-Level Entropy, Probability, Majority Voting, etc.) onto a single mathematical template parameterized by anchor distributions, model distributions, and monotonic transformations. This is the theoretical scaffolding that enables the sharpening analysis to apply uniformly.

2. Sharpening Mechanism Theory (Section 3, Appendix A) — the core mathematical derivation showing that repeated KL-regularized policy updates with intrinsic rewards drive the policy's probability mass to concentrate geometrically on an initial majority or high-confidence region. This component takes the unified reward framework as input and produces convergence rate predictions and asymptotic policy characterizations as output.

3. Controlled Training Experiments (Section 4, Appendix B) — a systematic empirical campaign that instantiates five intrinsic reward formulations (Majority Voting, Self-Certainty, Token-Level Entropy, Trajectory-Level Entropy, Probability) under the GRPO algorithm using the veRL framework, sweeps four hyperparameter dimensions (temperature, mini-batch size, KL regularization, rollout count), and monitors training dynamics (Reward Accuracy, Label Accuracy, Actor Entropy, Ground Truth Reward, Majority Voting Reward) to verify the rise-then-fall prediction.

4. Model Collapse Step Diagnostic (Section 6) — a practical tool derived from the sharpening theory that uses the training step at which Reward Accuracy drops below 1% as a proxy for model prior, enabling cheap assessment of RL trainability without running full supervised RL training.

Information flows as follows: the unified reward framework parameterizes any intrinsic reward → the sharpening theory predicts that iterative policy updates under this reward will geometrically concentrate probability mass → the controlled experiments test whether this prediction holds across methods, hyperparameters, and models → the Model Collapse Step extracts a practical diagnostic from the observed collapse dynamics.

3.3 Roadmap for the Deep Dive

• First, the unified reward framework (Appendix A.3), because it is the mathematical abstraction that makes the sharpening theory applicable across all intrinsic methods — without it, each reward formulation would require separate analysis.
• Second, the sharpening mechanism derivation (Section 3), starting with the one-step update dynamics for majority voting as the canonical case, then extending to the generalized convergence result and optimal policy characterizations for other intrinsic rewards. This is the paper's central theoretical contribution.
• Third, the experimental infrastructure (Appendix B), including the GRPO training setup, the metrics used to track training dynamics, the hyperparameter sweep methodology, and the five intrinsic reward implementations, because the empirical validation depends on these operational details.
• Fourth, the failure mode taxonomy (Section 4.1.2 and Appendix B.3), which documents three distinct collapse patterns (gradual degradation, length collapse, repetition collapse) and explains how different reward formulations lead to different failure behaviors even though all share the sharpening mechanism.
• Fifth, the Model Collapse Step methodology (Section 6), including the operational definition, the hyperparameter tuning strategy for rapid assessment, and the computation-cost comparison against gold-standard supervised RL training.
• Sixth, the self-verification external reward setup (Section 7), as a contrast case demonstrating that rewards grounded in computational asymmetries rather than model confidence exhibit qualitatively different scaling behavior.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical analysis paper with a substantial theoretical component. Its core idea is that all intrinsic URLVR methods — despite surface-level diversity in how rewards are computed — converge toward the same fundamental dynamic: geometrically amplifying the model's initial output distribution, which succeeds when the initial distribution aligns with correctness but inevitably collapses when it does not. The unified reward framework (Appendix A.3) provides the mathematical abstraction that makes this claim provable, and the controlled experiments (Section 4, Appendix B) provide the empirical validation.

---

The Unified Reward Framework

The paper observes that despite the proliferation of intrinsic reward formulations in prior work (Tables 1–2), every method can be expressed as a specific instantiation of a single parametric template. This unification is not merely taxonomic — it is the mathematical foundation that enables the sharpening analysis to apply uniformly rather than requiring separate proofs for each method.

Why unification is necessary. Without a shared mathematical representation, comparing Majority Voting to Token-Level Entropy to Self-Certainty is like comparing apples to oranges to bicycles. Each method has its own formula, its own scale, and its own optimization dynamics. Any theoretical claim about "intrinsic rewards in general" would be untestable without a way to map these diverse formulations onto a common space. The unified reward framework provides exactly that mapping by parameterizing three key degrees of freedom that capture all the variation across methods.

The template. The paper proposes that most intrinsic rewards can be written as:

$r_{\text{uni}}(x, y) = \psi\left(\sigma \cdot \frac{1}{|\mathcal{I}|} \sum_{i \in \mathcal{I}} \mathbb{H}(q_i, \pi^i_{\theta})\right)$

where $\psi$ is a monotonic transformation (identity or exponential), $\sigma \in {+1, -1}$ is a sign factor controlling optimization direction, $\mathcal{I}$ is the aggregation granularity (token-level or answer-level), $q_i$ is an anchor distribution at granularity $i$, $\pi^i_{\theta}$ is the model's output distribution at granularity $i$, and $\mathbb{H}(q_i, \pi^i_{\theta}) = -\sum_{v \in \mathcal{V}_i} q_i(v) \log \pi^i_{\theta}(v)$ is the cross-entropy between the anchor and model distributions.

What this template computes. For a given prompt $x$ and generated output $y$, the reward is determined by computing, at each element of the aggregation granularity $\mathcal{I}$, the cross-entropy between a fixed anchor distribution $q_i$ (which does not depend on the current policy) and the model's own token-level or answer-level distribution $\pi^i_{\theta}$. These cross-entropies are averaged across the granularity, multiplied by the sign $\sigma$ to determine whether high or low cross-entropy is rewarded, and then passed through the monotonic transformation $\psi$ which reshapes the reward magnitude without changing the ordering. The result is a scalar reward per generated sequence.

Why this form. Cross-entropy is the natural metric for measuring divergence between probability distributions in the context of maximum-likelihood training, which is what RL with KL regularization implicitly performs. The separation into anchor distribution $q$ and model distribution $\pi_{\theta}$ captures the essential structure: the reward is always a comparison between what the model currently outputs and some reference distribution. The sign $\sigma$ determines whether the model is pushed toward or away from the anchor. The monotonic transformation $\psi$ controls how aggressively the reward amplifies small differences. Every intrinsic method in Tables 1 and 2 can be recovered by making specific choices for $\mathcal{I}$, $q$, $\sigma$, and $\psi$, as demonstrated in Table 6.

The taxonomy of instantiations. The paper walks through how each intrinsic method maps onto this template (Table 6, Appendix A.3.2):

Certainty-based methods all use token-level granularity $\mathcal{I} = {1, \dots, |y|}$ but differ in their anchor distributions and transformations:

• Self-Certainty (RLIF): anchor is the uniform distribution $U_{|V|}$ over the vocabulary; sign $\sigma = +1$ rewards divergence from uniformity (i.e., confidence); transformation $\psi(z) = z + \log|V|$ (identity plus constant).
• Token-Level Entropy (EM-RL, RENT): anchor is the model's own distribution $\pi^t_{\theta}$ (self-anchored); sign $\sigma = -1$ rewards low entropy; transformation $\psi(z) = z$ (identity).
• Trajectory-Level Entropy (EM-RL): anchor is the one-hot $\delta_t$ centered on the actually-sampled token; sign $\sigma = -1$ rewards alignment with the sampled token; transformation $\psi(z) = z$ (identity).
• Probability (RLSC): same as Trajectory-Level Entropy but with exponential transformation $\psi(z) = \exp(|\mathcal{I}| \cdot z)$, which amplifies the sharpening effect by exponentiation.

Ensemble-based methods use answer-level granularity $\mathcal{I} = {A}$:

• Majority Voting (TTRL, SRT, ETTRL): anchor is the one-hot $\delta_A$ centered on the semantic majority answer class; sign $\sigma = -1$ rewards concentration on the majority; in the limit of infinite rollouts, this reduces to rewarding the highest-probability answer under the current policy.
• Semantic Entropy (EMPO): same structure but with a soft anchor derived from semantic clustering rather than exact answer matching.

The critical structural insight. The unified framework reveals why all intrinsic methods converge toward similar dynamics despite their differences. In every case, the reward is monotonically aligned with the model's own confidence in some sense: methods with $\sigma = -1$ directly reward the model for assigning high probability to tokens or answers it already favors; methods with $\sigma = +1$ reward confidence by pushing the distribution away from uniformity. Both lead to the same outcome — progressive sharpening, or increasing determinism — because they both create a gradient that increases the probability of high-confidence outputs relative to low-confidence ones. The framework thus demonstrates that sharpening is not a bug of specific reward designs but a structural consequence of deriving rewards from model-internal signals.

---

The Sharpening Mechanism: One-Step Update Dynamics for Majority Voting

The paper begins its theoretical analysis with majority voting as the canonical intrinsic reward because it has the simplest structure (binary reward: 1 if the output matches the most common answer, 0 otherwise), making the derivation transparent before generalizing.

The RL objective. The training uses the standard KL-regularized reinforcement learning objective (Equation 1):

$\max_{\pi_{\theta}} \mathbb{E}_{y \sim \pi_{\theta}(\cdot|x)}\left[r(x, y)\right] - \beta D_{\text{KL}}\left[\pi_{\theta}(\cdot|x) \parallel \pi_{\text{ref}}(\cdot|x)\right]$

where $\pi_{\theta}$ is the current policy (the LLM being trained), $\pi_{\text{ref}}$ is the reference policy (typically the initial model before training), $r(x, y)$ is the reward for output $y$ given prompt $x$, $\beta > 0$ is the KL penalty coefficient controlling how far the policy can diverge from the reference, and $D_{\text{KL}}[\cdot \parallel \cdot]$ is the Kullback-Leibler divergence.

What this objective does. It maximizes expected reward while penalizing the policy for straying too far from the reference distribution. The $\beta$ parameter acts as a temperature: large $\beta$ means heavy penalty and small policy changes; small $\beta$ means the policy moves aggressively toward high-reward outputs. This is the standard formulation from RLHF literature (Rafailov et al., 2023) and is what makes the optimal policy have a clean closed-form solution.

The majority voting reward at iteration $k$ (Equation 3):

$r_k(x, y) = \mathbf{1}[\text{ans}(y) = \text{maj}_k(Y_k)]$

where $Y_k = {y^{(1)}, \dots, y^{(N)}}$ is a set of $N$ rollouts sampled from the current policy $\pi^{(k)}_{\theta}$, $\text{ans}(y)$ extracts the final answer from output $y$, and $\text{maj}_k(Y_k) = \arg\max_a |{i \in [N] : \text{ans}(y^{(i)}) = a}|$ is the most frequent answer among the rollouts. The reward is 1 for any output whose answer matches the majority, and 0 otherwise.

What this reward does. For each training prompt, the model generates N candidate solutions, counts which final answer appears most often, and assigns a reward of 1 to all outputs with that answer and 0 to the rest. The majority answer serves as a pseudo-ground-truth — the assumption is that "what most rollouts agree on is probably correct." The model is then updated to increase the probability of outputs matching the majority and decrease the probability of all others.

The closed-form optimal policy for fixed reward. A key property of the KL-regularized objective is that if the reward function $r_k$ is held fixed and infinite gradient steps are taken from the reference policy $\pi^{(k)}_{\theta}$, the optimal policy has a closed form (Equation 2, adapted from Rafailov et al., 2023):

$\pi^{*,(k+1)}_{\theta}(y|x) = \frac{1}{Z_k(x)} \pi^{(k)}_{\theta}(y|x) \exp\left(\frac{1}{\beta} r_k(x, y)\right)$

where $Z_k(x) = \sum_{y} \pi^{(k)}_{\theta}(y|x) \exp\left(\frac{1}{\beta} r_k(x, y)\right)$ is the partition function that ensures proper normalization.

What this policy represents. For each possible output $y$, the optimal policy after training multiplies the reference policy's probability $\pi^{(k)}_{\theta}(y|x)$ by a reward-dependent weight $\exp(r_k(x,y)/\beta)$ and then renormalizes. Since $r_k$ is binary (0 or 1), the weight takes only two values: $\exp(1/\beta) = e^{1/\beta} > 1$ for majority-matching outputs, and $\exp(0) = 1$ for all others. This means the optimal policy simply amplifies the relative probability of majority outputs by a factor of $e^{1/\beta}$ while leaving non-majority outputs unchanged, then renormalizes.

The explicit update rule (Equation 5):

$\pi^{*,(k+1)}_{\theta}(y|x) = \begin{cases} \frac{\pi^{(k)}_{\theta}(y|x) \cdot e^{1/\beta}}{Z_k(x)} & \text{if } \text{ans}(y) = \text{maj}_k(Y_k) \\ \frac{\pi^{(k)}_{\theta}(y|x)}{Z_k(x)} & \text{otherwise} \end{cases}$

The probability mass shift (Equation 8). Define $p^{(k)}_{\text{maj}} = \sum_{y: \text{ans}(y) = \text{maj}_k(Y_k)} \pi^{(k)}_{\theta}(y|x)$ as the current policy's total probability mass on all trajectories leading to the majority answer. After the optimal update, this mass becomes:

$p^{*,(k+1)}_{\text{maj}} = \frac{p^{(k)}_{\text{maj}} \cdot e^{1/\beta}}{p^{(k)}_{\text{maj}} \cdot e^{1/\beta} + (1 - p^{(k)}_{\text{maj}})}$

What this computes. This is a simple hyperbolic transformation: the new majority probability is the old majority probability multiplied by the amplification factor $e^{1/\beta}$ and then renormalized against the unchanged non-majority mass. Since $e^{1/\beta} > 1$ for any finite $\beta$, this transformation always increases the majority probability: $p^{*,(k+1)}_{\text{maj}} > p^{(k)}_{\text{maj}}$ for any $p^{(k)}_{\text{maj}} \in (0, 1)$.

The ordering of actual one-step updates (Equation 9). In practice, training performs only one gradient update per iteration rather than converging to the optimal policy. The actual probability mass after one update $p^{(k+1)}_{\text{maj}}$ satisfies:

$p^{*,(k+1)}_{\text{maj}} \geq p^{(k+1)}_{\text{maj}} \geq p^{(k)}_{\text{maj}}$

Why this ordering holds. The lower bound follows from the policy gradient: the gradient $\nabla_{\theta} J = \mathbb{E}_{\pi_{\theta}}[r_k(x,y) \nabla_{\theta} \log \pi_{\theta}(y|x)]$ increases log-probability only for majority trajectories (where $r_k = 1$), so after one update, their probability must be at least as large as before. The upper bound follows because $\pi^{*}$ maximizes the objective, so no single update can exceed it. This establishes that $p_{\text{maj}}$ is monotonically non-decreasing across iterations.

Empirical validation of the ordering. Appendix A.1 provides two validations. First, training on individual MATH-500 problems with $N = 1024$ rollouts (to reduce majority vote randomness) for 50 steps shows strict monotonic increase in $p_{\text{maj}}$ at every single step (Table 4), with convergence from initial values toward 98.5%–99.8% by step 50 (Table 5). Second, an extreme off-policy experiment with fixed rewards (1024 gradient updates using rewards computed once from the initial rollout majority, with no reward recomputation) drives the Majority Voting Reward to 1.0 while validation accuracy drops to zero, confirming that the closed-form optimum is achievable and represents a degenerate deterministic policy.

---

The Sharpening Mechanism: Geometric Convergence to a Deterministic Policy

The one-step analysis shows that each update increases the probability mass on the majority answer. The paper's central theoretical result (Theorem 1) shows that iterating this process causes the policy to converge geometrically to a deterministic policy focused entirely on the initial majority answer, regardless of whether that answer is correct.

Theorem 1 statement. Consider the training process where at each iteration $k$: (1) sample $N$ rollouts $Y_k$ from $\pi^{(k)}_{\theta}$, (2) compute majority $\text{maj}_k(Y_k)$, (3) perform one gradient update with reward $r_k(x, y) = \mathbf{1}[\text{ans}(y) = \text{maj}_k(Y_k)]$ to obtain $\pi^{(k+1)}_{\theta}$. Under two assumptions validated empirically in Appendix A.1:

Assumption A1 (Majority stability): $\text{maj}_k(Y_k) = \text{maj}_0(Y_0)$ for all $k$ — the majority answer class does not flip across iterations. This holds for sufficiently large $N$ by the Law of Large Numbers: as $N \to \infty$, the empirical majority converges to $\arg\max_a \pi^{(k)}_{\theta}(a|x)$, and since $p^{(k)}_{\text{maj}}$ increases monotonically, the argmax remains the initial majority throughout training.

Assumption A2 (Effective learning): $\eta_k \geq \eta_{\min} > 0$ for all $k$ — each gradient update makes non-trivial progress. This is a standard assumption in policy gradient convergence.

Then $p^{(k)}_{\text{maj}}$ converges geometrically to 1 with rate $\rho = e^{-1/\beta}$, and the limiting policy is:

$\lim_{k \to \infty} \pi^{(k)}_{\theta}(y|x) = \begin{cases} \frac{\pi_{\text{ref}}(y|x)}{\sum_{y': \text{ans}(y') = \text{maj}_0(Y_0)} \pi_{\text{ref}}(y'|x)} & \text{if } \text{ans}(y) = \text{maj}_0(Y_0) \\ 0 & \text{otherwise} \end{cases}$

What this limit represents. At convergence, the policy assigns zero probability to any output whose answer differs from the initial majority. All probability mass is concentrated on trajectories that produce the initial majority answer, distributed proportionally to the reference policy's relative probabilities within that answer class. The model has become deterministic with respect to the answer: it will never produce any answer other than the one that was most common in the first batch of rollouts. The specific reasoning chain or wording within that answer class still follows the reference policy's relative preferences, but the answer itself is locked in.

The proof structure (Appendix A.2). The proof proceeds in six steps:

Step 1 — Effective update rule. The actual one-step update is modeled as moving the probability mass a fraction $\eta_k$ of the way toward the optimum:

$p^{(k+1)}_{\text{maj}} = p^{(k)}_{\text{maj}} + \eta_k \cdot (p^{*,(k+1)}_{\text{maj}} - p^{(k)}_{\text{maj}})$

where $\eta_k \in (0, 1]$ is the step efficiency (how much progress one gradient step makes toward the optimum). Substituting the optimal mass from Equation (8):

$p^{(k+1)}_{\text{maj}} = p^{(k)}_{\text{maj}} + \eta_k \cdot \frac{(\alpha - 1)(1 - p^{(k)}_{\text{maj}}) p^{(k)}_{\text{maj}}}{1 + (\alpha - 1) p^{(k)}_{\text{maj}}}$

where $\alpha = e^{1/\beta} > 1$ is the reward amplification factor.

What this equation says. The increase in majority probability at each step is proportional to three factors: the learning rate $\eta_k$, the reward amplification $(\alpha - 1)$, and a term $(1 - p^{(k)}_{\text{maj}}) p^{(k)}_{\text{maj}}$ that represents the product of current majority mass and current non-majority mass. This last term captures a key dynamic: the increase is largest when $p^{(k)}_{\text{maj}} \approx 0.5$ (maximum uncertainty) and approaches zero as $p^{(k)}_{\text{maj}}$ approaches either 0 or 1 — a sigmoidal convergence pattern.

Step 2 — Error dynamics. Define the error $\epsilon^{(k)} = 1 - p^{(k)}_{\text{maj}}$ as the distance from the fixed point at 1:

$\epsilon^{(k+1)} = \epsilon^{(k)} \left(1 - \eta_k \cdot \frac{(\alpha - 1)(1 - \epsilon^{(k)})}{\alpha - (\alpha - 1) \epsilon^{(k)}}\right)$

Step 3 — Monotonic decrease. Since $\alpha > 1$, $\epsilon^{(k)} \in (0, 1)$, and $\eta_k \in (0, 1]$, the multiplier is strictly between 0 and 1, so $\epsilon^{(k+1)} < \epsilon^{(k)}$ — the error strictly decreases at each step.

Step 4 — Convergence to zero. If the limit $\ell = \lim_{k \to \infty} \epsilon^{(k)}$ were positive, then for large $k$ the multiplier would be bounded below 1 (since $\eta_k \geq \eta_{\min}$ and the fraction is bounded above), causing continued decay — contradiction. Hence $\ell = 0$ and $p^{(k)}_{\text{maj}} \to 1$.

Step 5 — Geometric convergence rate. For large $k$ when $\epsilon^{(k)}$ is small, the multiplier simplifies to approximately $1 - \eta_k \cdot (\alpha - 1)/\alpha$. With $\eta_k \geq \eta_{\min}$, this gives geometric convergence:

$\epsilon^{(k+1)} \leq \left(1 - \eta_{\min} \cdot \frac{\alpha - 1}{\alpha}\right) \epsilon^{(k)}$

Step 6 — Limiting policy. Given majority stability (A1), the initial majority class $\text{maj}_0(Y_0)$ remains the majority throughout, and as all probability mass concentrates on this class, the limiting distribution is proportional to the reference policy restricted to that class.

What the convergence rate $\rho = e^{-1/\beta}$ means concretely. In the ideal case where $\eta_k = 1$ (each update reaches the optimum exactly), the error halves every $\ln(2) / (1 - e^{-1/\beta})$ iterations. For typical RLVR settings with $\beta$ around 0.04 (which appears implicitly in the GRPO experiments, though the paper doesn't state the exact GRPO-implicit $\beta$), $e^{-1/\beta} \approx e^{-25} \approx 1.4 \times 10^{-11}$, meaning the convergence is extremely rapid — the policy would become essentially deterministic within a handful of iterations if full convergence were achieved each step. In practice, $\eta_k$ is much smaller than 1, which slows convergence but does not change the qualitative result: determinism is inevitable.

Why this is a "rich-get-richer" dynamic. The key term in the update equation is $(1 - p^{(k)}_{\text{maj}}) p^{(k)}_{\text{maj}}$, which shows that the absolute increase in majority probability is proportional to the current majority probability. A majority that starts at 0.6 will grow faster (in absolute terms) than one that starts at 0.3. This positive feedback loop — the majority amplifies itself — is what drives geometric convergence and what makes the process irreversible once the majority is established.

The profound implication. Whether this convergence is beneficial or catastrophic depends entirely on whether the initial majority answer is correct. If 60% of initial rollouts produce the correct answer, the sharpening mechanism will drive the model toward reliably producing that correct answer (the "early gains" phase). If 60% of initial rollouts produce a wrong answer — which can easily happen on difficult math problems where common misconceptions lead to consistent errors — the same mechanism will drive the model toward deterministically producing that wrong answer (the "collapse" phase). The model has no way to distinguish between these two cases because the reward signal is purely self-referential: it rewards agreement with the model's own outputs, not agreement with truth.

---

Generalization to Other Intrinsic Rewards (Appendix A.3–A.5)

The paper extends the sharpening analysis beyond majority voting using the unified reward framework. The key structural property identified is Reward-Confidence Monotonicity for methods with $\sigma = -1$:

$\pi_{\theta}(y_a|x) > \pi_{\theta}(y_b|x) \implies r_{\text{uni}}(x, y_a) > r_{\text{uni}}(x, y_b)$

What this property means. For any two outputs $y_a$ and $y_b$, if the model assigns higher probability to $y_a$ than to $y_b$, then $y_a$ also receives a higher intrinsic reward than $y_b$. The reward function is monotonically aligned with the model's own probability ranking. This property holds for:

• Probability-based rewards ($\sigma = -1$, anchor $\delta$): $r(y) = \psi(\log \pi(y))$, and since $\psi$ is strictly increasing, higher probability directly implies higher reward.
• Answer-level ensemble rewards (majority voting, semantic entropy): all outputs in the majority answer class receive reward 1 and all others receive reward 0, so within-class outputs have higher reward than cross-class outputs.

Why this creates the same sharpening dynamic. The proof sketch (Proposition 1, Appendix A.4) shows that for any dominant trajectory $y^*$ (the one with highest probability under the current policy) and any competitor $y'$, the reward gap $\Delta_r = r(y^*) - r(y') > 0$ is strictly positive. The optimal policy update for the KL-regularized objective produces a target ratio:

$\frac{\pi^*(y^*)}{\pi^*(y')} = \frac{\pi^{(k)}(y^*)}{\pi^{(k)}(y')} \cdot \exp\left(\frac{\Delta_r}{\beta}\right)$

Since $\Delta_r > 0$, the target ratio strictly exceeds the current ratio, creating pressure to increase $\pi(y^*)$ relative to $\pi(y')$. As the policy moves in this direction, the reward gap $\Delta_r$ either stays the same (for answer-level rewards) or increases (for self-reinforcing rewards where $r(y) \propto \log \pi(y)$), creating a positive feedback loop that ensures the sharpening pressure persists until the dominant trajectory captures all probability mass.

For $\sigma = +1$ methods (Self-Certainty), the analysis is more subtle. Self-Certainty rewards the KL divergence from a uniform distribution, which means that a high-probability output and a very-low-probability output could both have high reward if both have sharply peaked per-token distributions. Confidence — not raw probability — drives the reward. However, the paper argues (Appendix A.4 Remark) that Self-Certainty still induces sharpening because maximizing distance from uniformity naturally favors peaked, low-entropy policies. The optimal policy for Self-Certainty (Appendix A.5) shows that $\pi_{\theta}(y|x) \propto \pi_{\text{ref}}(y|x) \exp\left(-\frac{1}{\beta|y||V|} \sum_{t=1}^{|y|} \sum_{v=1}^{|V|} \log \pi^t_{\theta}(y_t = v)\right)$, which amplifies sequences where the per-token distributions are concentrated — again reinforcing the model's existing preferred outputs.

Optimal policies for each intrinsic reward. Appendix A.5 derives the closed-form optimal policy after one KL-regularized update for each representative intrinsic reward, showing that all converge to amplifying high-probability sequences under the reference policy:

• Token-Level Entropy: $\pi_{\theta}(y|x) \propto \pi_{\text{ref}}(y|x) \exp\left(-\frac{1}{\beta|y|} \sum_{t=1}^{|y|} \sum_{v=1}^{|V|} \pi^t_{\theta}(y_t = v) \log \pi^t_{\theta}(y_t = v)\right)$ — amplifies low-entropy sequences.
• Trajectory-Level Entropy: $\pi_{\theta}(y|x) \propto \pi_{\text{ref}}(y|x) \cdot \left(\pi_{\theta}(y|x)\right)^{1/(\beta|y|)}$ — amplifies high-likelihood sequences by a power law.
• Probability: $\pi_{\theta}(y|x) \propto \pi_{\text{ref}}(y|x) \exp\left(\frac{1}{\beta} \pi_{\theta}(y|x)\right)$ — amplifies high-probability sequences exponentially.
• EMPO: $\pi_{\theta}(y|x) \propto \pi_{\text{ref}}(y|x) \exp\left(\frac{\pi_{\theta}(\text{ans}(y)|x)}{\beta}\right)$ — amplifies all sequences whose answer class has high probability.

In every case, the optimal policy after one update increases the relative probability of outputs the model already favors. Iterating this process necessarily drives the distribution toward determinism on the initially favored outputs.

---

Experimental Infrastructure: GRPO Training and Metrics

The GRPO algorithm. All experiments use the Group Relative Policy Optimization (GRPO) algorithm implemented in the veRL framework (Sheng et al., 2025), with the default hyperparameters specified in Table 7. GRPO is a variant of PPO (Proximal Policy Optimization) adapted for language model training, where the advantage for each rollout within a group is computed relative to the mean reward of the group rather than using a learned value function. This removes the need for a separate critic model, reducing memory and computational overhead. The advantage estimator, training temperature, batch sizes, and other settings are:

• Advantage estimator: GRPO
• Training temperature: 1.0 (default; swept in hyperparameter tuning)
• Global batch size: 64
• Mini-batch size: 64 (default; swept in tuning)
• Rollout number: 8 per prompt (default; swept in tuning)
• KL/Entropy regularization: None (default; KL regularization tested separately)
• Max prompt length: 1024 tokens
• Max response length: 7168 tokens
• Learning rate: $1 \times 10^{-6}$
• Epoch: 1

What these hyperparameters mean operationally. For each training step, the model generates 8 independent solutions per prompt (rollout number 8) across a batch of prompts (global batch size 64, so 8 × 64 = 512 total generated responses per step). All 512 responses are used in a single gradient update (mini-batch size 64 matches global batch size, meaning pure on-policy training with no stale samples). Training runs for 1 epoch over the dataset, which for DAPO-17k (~17,000 problems with batch size 64) corresponds to approximately 265 training steps. The low learning rate ($1 \times 10^{-6}$) and single epoch reflect the standard practice in RLVR where models are fine-tuned from a pretrained checkpoint rather than trained from scratch.

The five intrinsic reward implementations. The paper implements five reward functions by customizing the RewardManager module of veRL, following the formulas in Tables 1 and 2:

1. Majority Voting (ensemble-based): For each prompt, 8 rollouts are generated, the most common extracted answer is determined, and all rollouts matching that answer receive reward 1, others receive 0. The extraction uses the \boxed{} format standard in math reasoning.

2. Self-Certainty (certainty-based): $r_{\text{SC}}(x, y) = \frac{1}{|y|} \sum_{t=1}^{|y|} D_{\text{KL}}(U_{|V|} \parallel \pi_{\theta}(\cdot|x, y_{<t}))$ — the average KL divergence from a uniform distribution over the vocabulary to the model's next-token distribution at each position. High values mean peaked, confident distributions.

3. Token-Level Entropy (certainty-based): $r_{\text{H}}(x, y) = -\frac{1}{|y|} \sum_{t=1}^{|y|} H(\pi_{\theta}(\cdot|x, y_{<t}))$ — the negative average entropy across token positions. High values mean low entropy (confident) predictions.

4. Trajectory-Level Entropy (certainty-based): $r_{\text{Traj}}(x, y) = \frac{1}{|y|} \sum_{t=1}^{|y|} \log \pi_{\theta}(y_t|x, y_{<t})$ — the average log-probability of the actually-generated tokens, i.e., the sequence's log-likelihood normalized by length.

5. Probability (certainty-based): $r_{\text{Prob}}(x, y) = \prod_{t=1}^{|y|} \pi_{\theta}(y_t|x, y_{<t})$ — the raw product of token probabilities, equivalent to $\exp(|y| \cdot r_{\text{Traj}})$, which amplifies the sharpening effect exponentially.

Training dynamics metrics. To monitor reward hacking and validate theoretical predictions, the paper tracks six metrics throughout training (Section B.1, Appendix B.2):

1. Label Accuracy (ensemble methods): Prompt-level accuracy of majority-voted answers against ground truth. Computed as $\frac{1}{M} \sum_{i=1}^{M} \mathbf{1}[\text{maj}(x_i) = a^*_i]$, where $M$ is the number of prompts, $\text{maj}(x_i)$ is the majority-voted answer for prompt $i$, and $a^*_i$ is the ground-truth answer. This measures whether the pseudo-label (the majority) is correct, independent of individual rollout rewards.

2. Reward Accuracy (ensemble methods): Sample-level agreement between pseudo-rewards and oracle rewards. Computed as $\frac{1}{M \cdot N} \sum_{i=1}^{M} \sum_{j=1}^{N} \mathbf{1}[r_{\text{mv}}(y_{i,j}) = r_{\text{gt}}(y_{i,j})]$, where $r_{\text{mv}}$ is the majority-voting pseudo-reward and $r_{\text{gt}}$ is the ground-truth reward. This captures "lucky hits" where individual rewards are correct even when the majority vote is wrong — for example, if the majority is wrong but a minority response is correct, that minority response still gets the appropriate zero reward since it doesn't match the (incorrect) majority.

3. Ground Truth Reward (ensemble methods): $\frac{1}{M \cdot N} \sum_{i=1}^{M} \sum_{j=1}^{N} r_{\text{gt}}(y_{i,j})$ — the average oracle reward, representing what the reward would be if ground truth were available. This is the supervised baseline.

4. Majority Voting Reward (ensemble methods): $\frac{1}{M \cdot N} \sum_{i=1}^{M} \sum_{j=1}^{N} r_{\text{mv}}(y_{i,j})$ — the average pseudo-reward from majority voting. The divergence between this metric and Ground Truth Reward is the primary diagnostic for reward hacking: when Majority Voting Reward increases while Ground Truth Reward stays flat or decreases, the model is learning to maximize the proxy signal without improving actual correctness.

5. Label Accuracy (certainty methods): $\frac{1}{M} \sum_{i=1}^{M} \mathbf{1}[\text{ans}(y_{i, j^*_i}) = a^*_i]$ where $j^*_i = \arg\max_j r_{\text{cert}}(y_{i,j})$ — the ground-truth accuracy of the highest-certainty response per prompt. This tests whether maximum certainty implies correctness.

6. Actor Entropy: The average entropy of the policy's output distribution, tracked to measure distribution sharpening directly. Lower entropy corresponds to more peaked, deterministic outputs.

Validation benchmarks. The paper evaluates on three math reasoning benchmarks:

• AIME 2024 (Li et al., 2024): 30 competition-level math problems.
• AIME 2025 (Balunović et al., 2025): 30 problems, used to test generalization to a different year.
• AMC 2023 (Li et al., 2024): Additional competition problems.

For evaluation, the model generates 32 solutions per problem at temperature 0.6 with top-p 0.95, reporting average accuracy (avg@32) — the fraction of problems where at least one of the 32 solutions is correct, averaged over all problems.

Training dataset. The primary training dataset is DAPO-17k (Yu et al., 2025), consisting of approximately 17,000 math problems. Additional datasets tested for the dataset-size analysis (Section 5.1) and dataset-type analysis (Appendix C.3) include MATH-8k (Hendrycks et al., 2021), DeepScaleR-40k (Luo et al., 2025), and ORZ-56k (Hu et al., 2025). The base model is Qwen3-1.7B-Base unless otherwise specified.

---

The Three Collapse Patterns (Section 4.1.2, Appendix B.3)

The paper identifies that while all intrinsic methods eventually collapse, the specific failure mode depends on the reward formulation. Figure 3 documents three distinct patterns:

Pattern 1: Gradual degradation (Self-Certainty, Majority Voting). These methods degrade most slowly, maintaining higher validation performance and higher Label Accuracy without collapsing within one epoch (roughly 265 steps). Self-Certainty sharpens against a uniform distribution at each token position, making it less aggressive than direct probability maximization — it penalizes uniform distributions regardless of whether the model is confident in the correct or incorrect token. Majority Voting operates at the answer level rather than the token level, avoiding token-level artifacts (like repetition or length bias) that accelerate collapse in other methods. The collapse, when it comes, is in the quality of the majority vote itself: the model becomes so deterministic that all rollouts produce the same (possibly incorrect) answer, making the reward signal perfectly consistent but perfectly wrong.

Pattern 2: Length collapse (Probability). The Probability reward — the raw product of token probabilities — inherently favors shorter sequences because each additional token multiplies by a probability less than 1, making the product exponentially smaller with length. The model learns to produce confident but overly brief answers, driving Mean Response Length downward (Figure 3, fourth panel from left) while still reducing Actor Entropy (the model is confident in its brief outputs). The paper notes that length normalization (using geometric mean or average log-probability) would likely mitigate this bias, but this is not explored.

Pattern 3: Repetition collapse (Token-Level Entropy, Trajectory-Level Entropy). Both entropy-based methods average entropy across tokens. Minimizing average entropy can be achieved not only by making confident predictions but also by padding sequences with repeated high-probability tokens. Unlike Probability (which rewards brevity because each extra token reduces the product), averaging across sequences does not intrinsically penalize length, so the model can minimize average entropy by generating long sequences of repetitive, high-confidence tokens. The Mean Response Length increases while Actor Entropy decreases, indicating the model is producing long but low-information outputs.

Why these patterns matter. The fact that different intrinsic rewards produce different failure modes — despite sharing the same sharpening mechanism — demonstrates that the sharpening mechanism is the necessary condition for collapse, but the specific reward formulation determines the sufficient condition (how collapse manifests). This is important because it means that mitigating one failure mode (e.g., by length-normalizing Probability) would not prevent collapse — it would just change how collapse manifests. The fundamental issue is the self-referential nature of the reward, not the specific computational form.

---

Model Collapse Step: Operational Definition and Methodology (Section 6)

The paper leverages the inevitability of collapse to create a practical diagnostic: the Model Collapse Step, defined as the training step at which Reward Accuracy drops below 1% during intrinsic URLVR training with majority voting reward and aggressive hyperparameters.

Operational definition. Formally:

$\text{Model Collapse Step} = \min \{ k \in \mathbb{N} : \text{Reward Accuracy at step } k < 0.01 \}$

where $k$ counts training steps in a diagnostic run using majority voting reward with tuned aggressive hyperparameters ($MBS = 1$, $N = 8$). Training continues until Reward Accuracy falls below the 1% threshold, at which point the step count is recorded.

Why this definition works. Recall that Reward Accuracy measures the fraction of individual rewards that match ground-truth correctness (Appendix B.2, Equation 40): $\frac{1}{M \cdot N} \sum_{i=1}^{M} \sum_{j=1}^{N} \mathbf{1}[r_{\text{mv}}(y_{i,j}) = r_{\text{gt}}(y_{i,j})]$. When the model's distribution is diverse (high entropy), whether a given rollout's majority-vote reward matches ground truth is a meaningful signal: correct rollouts sometimes get reward 1 (if they're in the majority), sometimes 0 (if they're in the minority); incorrect rollouts similarly fluctuate. When the model collapses to determinism, two things happen: (1) all rollouts produce the same answer, so the majority-voting reward is always 1 for all rollouts, and (2) if that answer is wrong, the ground-truth reward is always 0, making Reward Accuracy exactly 0. Even before complete determinism, Reward Accuracy provides a sensitive measure of the divergence between the proxy reward signal and actual correctness. The 1% threshold captures the point where the model has essentially stopped receiving useful learning signal.

Aggressive hyperparameters for rapid assessment (Section 6.3). To make Model Collapse Step computationally cheap, the paper tunes hyperparameters to accelerate convergence without sacrificing ranking consistency across models. The key findings from the hyperparameter sweep (Appendix B.3) guide this:

• Smaller mini-batch size accelerates collapse (Figure 17): mini-batch size 1 collapses within ~20 steps; size 64 maintains stability longest. Small mini-batches create stale reward signals (rewards computed under old policies applied to samples from newer policies), which the theory predicts should amplify the sharpening effect.
• Larger rollout count accelerates collapse (Figure 19): $N = 32$ collapses within 180 steps; $N = 8$ within the full epoch. More rollouts provide a more statistically reliable majority, which the theory predicts creates stronger reward signal and faster convergence.

The tuned aggressive configuration for rapid assessment uses $MBS = 1$ and $N = 8$. The paper validates (Figure 12) that while these aggressive settings accelerate collapse in absolute step counts, the relative ranking of models by Collapse Step is preserved across settings — a model that collapses at step 200 under default settings might collapse at step 100 under aggressive settings, but its rank order relative to other models remains stable.

Computation cost comparison. Table 3 quantifies the efficiency advantage. Computing Model Collapse Step across 7 models requires $7k \times 8 \times 662 \times 32 = 1.19B$ total tokens (response length × rollouts × total collapse steps across all models × batch size). The gold-standard supervised RL training (GT Gain) requires $7k \times 8 \times 17k \times 7 = 6.66B$ tokens (response length × rollouts × training problems × models). The ratio is $6.66 / 1.19 \approx 5.6\times$ fewer tokens, and critically, Model Collapse Step requires no ground-truth labels — only the proxy reward signal, which is computed from the model itself.

What Model Collapse Step captures that $\text{pass}@k$ misses. The widely-used $\text{pass}@k$ metric measures the fraction of problems where at least one of $k$ samples is correct. This is a static measure of the model's distribution quality. Model Collapse Step measures a dynamic property: how long the model can sustain useful learning from its own outputs before the sharpening mechanism drives the reward signal to zero. The paper argues (Section 6.2) that this dynamic property is more predictive of actual RL trainability because real RL training involves feedback between the policy and the reward signal — a feedback loop that $\text{pass}@k$ ignores but that Model Collapse Step explicitly captures by running the actual intrinsic RL process.

Figure 11 shows the correlation: Model Collapse Step correlates strongly with GT Gain (the improvement from running a full epoch of supervised RL with ground-truth rewards), matching or exceeding $\text{pass}@k$'s predictive power. The paper also notes that Model Collapse Step cannot be gamed by random guessing on multiple-choice questions, whereas $\text{pass}@k \to 1$ for large $k$ on multiple-choice regardless of model quality.

---

Self-Verification External Reward Setup (Section 7)

As a contrast case demonstrating that external rewards can escape the sharpening collapse, the paper implements a self-verification system on the Countdown arithmetic puzzle task.

The Countdown task. Given a set of numbers and a target value, the model must produce an arithmetic expression using each number exactly once that evaluates to the target. For example, with numbers {3, 4, 5} and target 23, the model might generate (3 × 4) + 5 = 17 (incorrect) or 3 + (4 × 5) = 23 (correct). The key property that makes Countdown suitable for external URLVR is the generation-verification asymmetry: finding a valid expression requires search over the large combinatorial space of operator arrangements and orderings, but checking whether a candidate expression correctly uses all numbers and evaluates to the target is a deterministic, constant-time arithmetic computation. The verification procedure is objective and cannot be gamed.

Training setup. The model (Qwen3-1.7B-Base or Qwen3-4B-Base) is trained on 4,000 randomly sampled Countdown problems from the Countdown-Tasks-3to4 dataset, with 1,000 held-out problems for validation. For each problem, the model generates a solution containing a proposed arithmetic expression and then invokes the self-verification step: the model uses a verification prompt (Prompt 1 or Prompt 2 from Appendix C.1) to check whether the expression is valid, receiving a binary reward (1 if the verifier says correct, 0 otherwise). The ground-truth scoring function (deterministic arithmetic evaluation) is used only for evaluation, not for training — the training reward comes entirely from the model's own verification judgment.

Verification prompts. The paper tests two prompts to measure robustness (Section C.1):

• Prompt 1 (adapted from RLSR): A four-step checklist checking that the expression uses only allowed numbers, each number appears exactly once, the expression is valid arithmetic, and it equals the target. All checks must pass for a True verdict.
• Prompt 2 (custom): A five-step checklist with additional checks for missing/empty expressions, valid operators, and a numerical evaluation tolerance of $10^{-6}$. This prompt is more structured and explicit.

Metrics tracked. The experiments track:

• Validation accuracy (avg@16): the fraction of validation problems where at least one of 16 generated solutions is correct.
• Ground Truth Reward: the fraction of training solutions that are actually correct according to deterministic arithmetic evaluation.
• Reward Accuracy: the fraction of solutions where the model's self-verification judgment matches ground-truth correctness.
• Self-Verify Reward: the fraction of solutions the model's verifier judges as correct (the proxy reward used for training).

Comparison baselines. The self-verification approach is compared against:

• Trajectory-Level Entropy (intrinsic reward from Table 1) as a representative intrinsic method.
• Oracle Supervision (training with ground-truth rewards) as the supervised upper bound.

The paper also tests prompt sensitivity by comparing the base model (Qwen3-1.7B-Base) against its instruction-tuned variant (Qwen3-1.7B) with both verification prompts, to assess whether self-verification's success depends on instruction-following capability.

Why this setup tests the sharpening hypothesis. The critical theoretical distinction is that self-verification's reward is grounded in an external procedure (arithmetic evaluation, even if performed by the model itself through a verification prompt) rather than in the model's confidence or internal distribution. The sharpening mechanism predicts that intrinsic rewards inevitably collapse because the reward signal is self-referential: it rewards agreement with the model's own outputs, creating a feedback loop that amplifies whatever the model initially favors. Self-verification, in contrast, provides a reward that depends on whether the generated expression actually satisfies the puzzle constraints — a property that is independent of whether the model is confident or uncertain. If the sharpening mechanism is the correct explanation for intrinsic URLVR collapse, then self-verification should exhibit qualitatively different training dynamics: sustained improvement (or at least stable performance) without the rise-then-fall pattern. Figure 13 shows exactly this: both Qwen3-1.7B-Base and Qwen3-4B-Base show increasing or stable validation accuracy and Ground Truth Reward with self-verification, while Trajectory-Level Entropy shows the familiar collapse pattern.

4. Key Insights and Innovations

Innovation 1: The Sharpening Mechanism as a Unifying Causal Explanation for All Intrinsic URLVR Behavior

The paper’s deepest intellectual contribution is not the observation that intrinsic URLVR methods fail — prior work had already documented reward hacking and model collapse in individual methods (Agarwal et al., 2025; Shafayat et al., 2025; Zhang et al., 2025c). What is genuinely novel is the identification of a single causal mechanism — distribution sharpening — that explains why every intrinsic method, despite surface-level design diversity, converges toward the same rise-then-fall fate. This transforms the URLVR conversation from "some methods fail sometimes" to "all intrinsic methods must eventually fail because they share a structural dynamic."

What makes this shift distinctive. Before this paper, the field treated the proliferation of intrinsic reward formulations as evidence of a rich design space. Self-Certainty, Token-Level Entropy, Probability, and Majority Voting were framed as fundamentally different approaches — one rewards divergence from uniformity, another penalizes entropy, a third rewards answer consensus. The implicit assumption was that these differences mattered, that careful reward engineering could find a formulation that avoided the collapse observed in earlier attempts. The paper demolishes this assumption by demonstrating that all these formulations are instances of a single parametric family (the unified reward framework of Appendix A.3) and that this family has a single convergent dynamic: amplifying the model’s initial probability distribution until it becomes deterministic.

The intellectual move is reclassifying surface variation as parametric variation within a single convergent class. The unified reward framework is not merely a taxonomy — it is a proof of non-distinctiveness. By showing that Self-Certainty differs from Majority Voting only in the choice of anchor distribution and aggregation granularity, the paper makes the case that no amount of reward engineering within the intrinsic paradigm can escape the sharpening dynamic. Changing the reward formulation changes how fast collapse occurs and what failure mode it takes (length collapse versus repetition collapse versus gradual degradation, as documented in Figure 3 and analyzed in Section 4.1.2), but does not change whether it occurs. This is a fundamentally different claim from prior work, which typically attributed failure to specific reward design flaws and proposed alternative rewards as solutions.

The theoretical architecture is what makes this claim credible. Rather than arguing by empirical induction ("we tested five methods and all five collapsed, so intrinsic methods probably collapse"), the paper provides a deductive argument. The sharpening mechanism is derived mathematically from the KL-regularized RL objective and the structure of self-referential rewards (Section 3, Theorem 1). The unified reward framework then shows that all intrinsic methods instantiate this structure. The empirical results (Section 4) serve as validation of the theory, not as the primary evidence. This is a higher standard of argument than the field typically demands, and it enables conclusions ("intrinsic URLVR cannot scale") that empirical induction alone could never justify.

Comparison to prior theoretical work. The field had existing theory for KL-regularized RL with fixed rewards — the closed-form optimal policy (Rafailov et al., 2023) was well-known — but no one had analyzed what happens when the reward itself is a function of the policy’s own distribution. The paper identifies that this self-referential structure creates a feedback loop: the policy update increases probability on high-reward outputs, which changes the reward (since the reward depends on the policy’s distribution), which further increases probability on those outputs. This feedback loop is what drives geometric convergence — it is not present when rewards are fixed (as in supervised RLVR), which is why supervised RLVR does not exhibit the same collapse pattern. The theoretical insight is not the KL-optimization math itself, but the identification that making rewards policy-dependent transforms a stable optimization into an unstable positive-feedback system.

Significance beyond performance. This insight is primarily a negative result with reframing power: it tells the field to stop trying to fix intrinsic URLVR through reward engineering and look elsewhere. Without this reframing, the field might have spent years iterating on reward formulations, each paper claiming modest gains before eventual collapse, each failure attributed to specific design flaws rather than to the paradigm itself. The paper’s contribution is to provide a principled stopping condition: intrinsic rewards are bounded by what the model already knows, and no amount of tuning can push them beyond that boundary. This is analogous to how the No Free Lunch theorems in optimization (Wolpert and Macready, 1997) reframed algorithm design by showing that no optimizer universally dominates — it changed what questions researchers asked, even if it didn’t provide a better optimizer. The sharpening mechanism plays the same role for URLVR: it changes the question from "how do we design better intrinsic rewards?" to "what reward sources can escape the confidence-correctness ceiling?"

Evidence anchor. Figure 3 provides the empirical signature: five different reward formulations, all exhibiting the same rise-then-fall pattern but with different failure modes. Figure 19 and the associated Appendix B.3 sweeps show that hyperparameter tuning shifts when collapse occurs but does not prevent it. The extreme hyperparameter experiment (Section 4.1.1, extended training to 1,000 steps with optimized settings, where "collapse still occurs around 1,000 steps, roughly 4 epochs") demonstrates that the pattern persists even under the most favorable engineering conditions the authors could construct.

---

Innovation 2: Model Collapse Step as a Cheap, Theory-Grounded Diagnostic for RL Trainability

The paper introduces a new kind of evaluation metric: one that measures not static model capability but dynamic trainability under self-referential feedback. The Model Collapse Step — the training step at which Reward Accuracy drops below 1% during a short intrinsic URLVR diagnostic run — is conceptually novel because it exploits the sharpening mechanism rather than fighting it. Instead of trying to prevent collapse, the paper uses collapse timing as an informative signal about the model.

Why this is not just another metric. The standard approach for assessing whether a model will benefit from RL training is to run the full RL training and measure the gain — expensive (6.66B tokens for 7 models in the paper’s calculation, Table 3) and requiring ground-truth labels. The widely-used alternative, pass@k, measures how many problems the model can solve with k attempts. This is a static capability measure: it tells you what the model can do now, not how it will respond to training. The critical conceptual gap is that RL training involves a feedback loop between the policy and the reward signal — the model’s outputs affect the rewards it receives, which affect future outputs. A static metric cannot capture this feedback dynamic. Model Collapse Step captures exactly this dynamic by running the actual intrinsic RL process and measuring when the feedback loop degrades from beneficial (reinforcing correct answers) to harmful (reinforcing incorrect answers).

The theory-grounding is what gives the metric its power. Model Collapse Step is not an arbitrary heuristic — it is a direct operationalization of Theorem 1. The theorem predicts that the policy converges geometrically toward determinism on its initial majority, with convergence rate determined by the model’s initial confidence-correctness alignment. A model whose initial majority answers are mostly correct (strong prior) will maintain Reward Accuracy for longer because the sharpening mechanism is reinforcing correct behavior. A model whose initial majority answers are frequently wrong (weak prior) will see Reward Accuracy drop rapidly as the sharpening mechanism amplifies errors. The Collapse Step is therefore a revealed-preference measure of model prior: it measures not what the model claims (confidence scores can be miscalibrated) but what actually happens when the model is forced to learn from its own outputs. This is a deeper notion of prior than pass@k, which measures only the model’s surface accuracy without probing the stability of its internal preferences.

Comparison to pass@k. The paper shows (Figure 11, right panel) that pass@k Gain (the difference between pass@256 and pass@1) correlates less reliably with actual RL gains than Model Collapse Step does. The conceptual reason is that pass@k measures whether correct solutions exist somewhere in the model’s distribution, but RL training requires that correct solutions receive high enough probability to serve as effective learning targets. A model might have pass@256 = 0.8 (80% of problems have at least one correct solution in 256 samples) but assign very low probability to those correct solutions, meaning the sharpening mechanism will amplify the higher-probability incorrect solutions instead. Model Collapse Step indirectly measures this probability-mass alignment: if the model assigns high probability to correct answers, the majority will be correct, sharpening will be beneficial, and Reward Accuracy will stay high longer.

The efficiency argument is conceptual as well as practical. The 5.6× token reduction (Table 3) matters, but the deeper point is that Model Collapse Step requires no ground-truth labels — only the majority-voting proxy reward, which is computed from the model’s own outputs. This means the diagnostic can be run on problems where ground-truth labels are unavailable or expensive, which is exactly the regime URLVR is supposed to address. A diagnostic that requires ground-truth labels to assess whether a model is ready for training without ground-truth labels would be circular. Model Collapse Step escapes this circularity by using the same self-referential signal that drives training as the diagnostic signal — it is a self-contained assessment.

Evidence anchor. Figure 11 shows the correlation between Model Collapse Step (left), GT Gain (middle), and pass@k Gain (right) across 7 models from 3 families. The Collapse Step ranking (OLMo-2-1124-7B at 34 steps through Qwen3-8B-Base at 383 steps) strongly predicts the GT Gain ranking. Figure 12 demonstrates that aggressive hyperparameters (mini-batch size 1, 32 rollouts) accelerate collapse but preserve model rankings, enabling rapid assessment.

---

Innovation 3: The Confidence-Correctness Alignment Ceiling as a Fundamental Scalability Boundary

The paper’s most practically significant conceptual contribution is the identification and formalization of the confidence-correctness alignment ceiling — the boundary beyond which intrinsic URLVR cannot improve performance, regardless of compute budget, because the reward signal is structurally incapable of distinguishing between justified and unjustified confidence.

Why this is a fundamental boundary rather than an engineering limitation. The sharpening mechanism reveals that intrinsic rewards are monotonically aligned with the model’s own probability ranking (Reward-Confidence Monotonicity, Appendix A.4). This means the reward function is, in a precise mathematical sense, a coarsening of the model’s own belief distribution — it preserves the ordering of outputs by probability but cannot introduce new information about which outputs are correct. The optimization process can make the model more confident in its existing preferences but cannot change which outputs it prefers. The confidence-correctness alignment ceiling is the maximum achievable accuracy when the model’s initial preference ordering over outputs is taken as fixed and training can only sharpen that ordering.

This is a stronger claim than "intrinsic rewards sometimes fail." Prior work documented failure cases without characterizing the boundary between success and failure. The paper shows that the boundary is determined by a specific, measurable property: whether the model’s initial highest-probability output is correct, and more generally, whether the probability mass assigned to correct outputs exceeds the mass assigned to incorrect outputs. Section 4.2.1 demonstrates this cleanly with per-problem training: of 25 problems trained individually, 22 simply sharpened the model’s initial preference regardless of correctness (Figure 4). The 3 cases where correctness flipped were precisely the cases where the highest-reward sample was mostly correct during training — meaning the model’s probability ordering already favored correct outputs, and sharpening merely made that preference more decisive.

The practical implication: intrinsic URLVR can only exploit existing capability, never create it. This distinguishes URLVR from supervised RLVR, where ground-truth rewards can tell the model that its high-confidence outputs are wrong and redirect probability mass toward initially low-confidence but correct outputs. In supervised RLVR, the reward function is independent of the model’s probability distribution, so the optimization can move the model toward outputs it initially assigned low probability. In intrinsic URLVR, the reward function depends on the model’s distribution, creating a fixed point: the model’s initial high-probability outputs remain high-probability indefinitely because they generate the highest rewards, which further increases their probability. The only way an output can gain probability is if it already had enough probability to influence the reward signal — a catch-22 for correct-but-initially-unlikely outputs.

Comparison to the "rich-get-richer" narrative in other domains. The sharpening dynamic is structurally similar to preferential attachment in network science (Barabási and Albert, 1999), where nodes with high degree attract more new connections and thus further increase their degree. In both cases, the feedback loop creates a power-law or winner-take-all distribution from initially small advantages. The conceptual contribution here is recognizing that this dynamic applies to LLM self-training and that it creates a fundamental scalability ceiling — the model’s initial distribution determines its asymptotic distribution, modulo the sharpening transformation. This is not incremental: it provides a principled explanation for why test-time training works (the initial distribution on a small, domain-specific dataset is likely well-aligned with correctness) while large-scale training fails (the initial distribution on a large, diverse dataset contains many pockets of high-confidence error).

Evidence anchor. The dataset-size experiment (Section 5.1, Figure 6) provides the cleanest demonstration. Training on 32 or 128 problems — where the model’s initial majority answers are mostly correct because the problems are in-distribution for the base model — maintains stable performance without collapse. Training on 512 or more problems — where the dataset contains problems on which the model’s initial preferences are wrong — exhibits sharp reward hacking. The KL divergence analysis (Figure 7) shows that small datasets induce localized parameter updates (DAPO-32 reaches only 0.057 KL from reference after 600 steps) while large datasets induce global policy shift (DAPO-512 reaches ~2× higher KL). The ceiling is crossed when the training distribution contains enough misaligned problems that the aggregate reward signal becomes dominated by sharpening incorrect preferences rather than correct ones.

---

Innovation 4: External Rewards as a Paradigm Separation, Not Just an Alternative Method

The paper introduces a conceptual taxonomy (Section 2) that divides URLVR into intrinsic versus external methods based on whether the reward signal derives from the model’s internal state or from an independent verification procedure. This is not merely a classification exercise — it is a predicate for a theoretical claim about scalability. The paper argues that the distinction between intrinsic and external rewards is the boundary between unscalable and scalable URLVR, because external rewards are (by construction) independent of the model’s probability distribution and therefore immune to the sharpening feedback loop.

Why this taxonomy matters intellectually. Before this paper, the URLVR literature treated all unsupervised reward methods as variations on a theme. Self-verification (Shao et al., 2025b), execution-based code rewards (Zhao et al., 2025a), and majority voting (Zuo et al., 2025) were lumped together as "unsupervised rewards" without distinguishing their structural properties. The paper’s taxonomy introduces a sharp criterion — does the reward depend on the model’s own output distribution? — that predicts which methods will hit the confidence-correctness ceiling and which will not. This is a falsifiable theoretical claim: if an external reward method were found to exhibit the same rise-then-fall collapse pattern as intrinsic methods, the taxonomy’s predictive power would be undermined. The paper provides preliminary evidence against this null hypothesis with the self-verification experiments (Section 7, Figure 13).

The self-verification experiments are a proof of concept for the taxonomy, not a method proposal. The paper does not claim that self-verification on Countdown is a scalable solution to URLVR — the task is too narrow, the verification prompt engineering is fragile (the base model fails with Prompt 1 but works with Prompt 2; Figure 14, left). Rather, the experiments demonstrate a qualitative difference in training dynamics: self-verification shows sustained improvement or stability without collapse (Figure 13, top), while Trajectory-Level Entropy — an intrinsic baseline — shows a decline. The Reward Accuracy curve for self-verification (Figure 13, bottom) reveals an interesting dynamic: it initially drops around step 200 as the policy explores and tries to exploit the verifier, then recovers and stabilizes above 0.5, suggesting the model is genuinely learning to produce verifiably correct outputs rather than learning to fool its own verifier. This recovery pattern is categorically different from intrinsic methods, where Reward Accuracy declines monotonically toward zero (Figure 2, Figure 6).

The generation-verification asymmetry principle is the conceptual engine. The paper identifies (Section 2.2) that many reasoning domains have the property that generating a correct solution is hard (requiring search over a large space) while verifying a candidate solution is easy (a deterministic, constant-time procedure). This asymmetry is not domain-specific — it appears in math (numerical evaluation), code (test execution), formal proof (proof checking), puzzles (rule evaluation), and potentially many scientific domains (simulation-based verification). The paper argues that this asymmetry is precisely what URLVR should exploit: the verification procedure provides an objective, scalable reward signal that is independent of the model’s confidence. The failure of intrinsic methods is not a failure of URLVR per se, but a failure of using model confidence as a proxy for correctness when a genuine verification procedure is available.

Comparison to the "self-play" and "self-improvement" narratives. Prior work on self-play for LLMs (SeRL, R-Zero, SQLM; cited in Section 2.1) often conflates two distinct ideas: (1) generating training data without human labels, and (2) using model-internal signals as rewards. The paper’s taxonomy separates these: external reward methods generate rewards without human labels but ground them in independent verification, while intrinsic methods generate rewards from the model’s own distribution. The generation-verification asymmetry provides a way to do (1) without (2) — to be "unsupervised" in the sense of not requiring human labels, but still "supervised" in the sense of having a ground-truth reward signal (the verifier). This reframing recasts the URLVR problem from "how do we learn without labels?" to "how do we construct verifiers that are cheaper than labelers?"

Evidence anchor. Figure 13 demonstrates the contrasting dynamics: self-verification (Qwen3-1.7B-Base) reaches approximately 0.55 validation accuracy (avg@16) and maintains or improves Ground Truth Reward, while Trajectory-Level Entropy peaks around 0.3 and declines. Figure 14 shows that instruction alignment (Qwen3-1.7B instruction-tuned) enables robust self-verification across both prompts, reaching over 80% accuracy, suggesting that the bottleneck for external URLVR is not the reward structure but the model’s ability to follow verification instructions — a capability that improves with model scale and alignment training.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary training dataset is DAPO-17k (Yu et al., 2025), consisting of approximately 17,000 math problems. Additional datasets used for specific analyses include MATH-8k (Hendrycks et al., 2021), DeepScaleR-40k (Luo et al., 2025), and ORZ-56k (Hu et al., 2025). For test-time training experiments, AMC23 (40 problems) is used. For the self-verification external reward experiments, Countdown-Tasks-3to4 is used with 4,000 training problems and 1,000 validation problems. For per-problem analysis, 25 individual problems are sampled from MATH500. The dataset size scaling experiment uses randomly sampled subsets of DAPO-17k with sizes {32, 128, 512, 2048, 8192, 16384}.

• Base model(s). The primary base model is Qwen3-1.7B-Base (Yang et al., 2025), a 1.7B-parameter model from the Qwen family, chosen as representative of contemporary open-source LLMs. For the Model Collapse Step analysis (Section 6), 7 models from 3 families are evaluated: OLMo-2-1124-7B, Meta-Llama-3.1-8B, Qwen2.5-Math-1.5B, Qwen2.5-1.5B, Qwen2.5-7B, Qwen3-1.7B-Base, and Qwen3-8B-Base. For the backbone model analysis (Appendix C.2), an expanded set of 11 models from Qwen and Llama families is used, including base, math-specialized, SFT, and instruction-tuned variants. For self-verification, Qwen3-1.7B-Base and Qwen3-4B-Base are used.

• Metrics. The paper tracks multiple training dynamics metrics and evaluation metrics. Training dynamics metrics include: (1) Label Accuracy — for ensemble methods, the prompt-level accuracy of majority-voted answers against ground truth ($\frac{1}{M} \sum_{i=1}^M \mathbf{1}[\text{maj}(x_i) = a^*_i]$); for certainty methods, the ground-truth accuracy of the highest-certainty response per prompt. (2) Reward Accuracy — sample-level agreement between pseudo-rewards and oracle rewards ($\frac{1}{M \cdot N} \sum_{i=1}^M \sum_{j=1}^N \mathbf{1}[r_{\text{mv}}(y_{i,j}) = r_{\text{gt}}(y_{i,j})]$). (3) Ground Truth Reward — average oracle reward across all generated responses. (4) Majority Voting Reward — average pseudo-reward from majority voting, where divergence from Ground Truth Reward indicates reward hacking. (5) Actor Entropy — average entropy of the policy's output distribution, tracking distribution sharpening directly. (6) Mean Response Length — tracked for certainty-based methods to detect length or repetition collapse. Evaluation metrics: Validation performance is measured using avg@32 — the fraction of problems where at least one of 32 generated solutions is correct, averaged over all problems, with generation at temperature 0.6 and top-p 0.95. For the Model Collapse Step analysis, GT Gain measures the improvement from running one epoch of supervised RLVR with ground-truth rewards. For the self-verification experiments, avg@16 is used.

• Baselines. The paper compares intrinsic URLVR methods against: (1) Ground Truth (Oracle) Training — standard supervised RLVR using ground-truth labels as rewards, representing the gold-standard upper bound. (2) Majority Voting — the TTRL (Zuo et al., 2025) method, which uses majority voting across 8 rollouts to create pseudo-labels as rewards. This is the primary intrinsic baseline against which all other intrinsic methods and hyperparameter variations are compared. (3) Trajectory-Level Entropy — a representative certainty-based intrinsic reward (Agarwal et al., 2025), used as the primary certainty baseline in method comparison experiments and as the intrinsic comparison for self-verification. Additional intrinsic methods compared include Self-Certainty, Token-Level Entropy, and Probability (Table 1). For the self-verification external reward experiments, baselines are Trajectory-Level Entropy and Oracle Supervision.

• Generation budget / compute accounting. For intrinsic URLVR experiments, the standard generation budget is 8 rollouts per prompt with a global batch size of 64, producing 512 generated responses per training step. Training runs for 1 epoch over DAPO-17k, corresponding to approximately 265 steps at batch size 64. For evaluation, models generate 32 solutions per problem (avg@32). For the Model Collapse Step computation cost comparison, token counts are computed as response_length × rollouts × total_steps_or_problems × batch_size, with the gold standard (GT Gain) requiring 6.66B tokens across 7 models and Model Collapse Step requiring 1.19B tokens — a 5.6× reduction (Table 3). For the self-verification experiments, 4,000 training problems and 1,000 validation problems are used, with generation budget implicit in the avg@16 evaluation metric.

• Cross-validation / statistical protocol. For the dataset-size experiments (Section 5.1), results are verified across 3 random seeds for subset sizes {32, 128, 512}. For the per-problem analysis (Section 4.2), 25 individually sampled problems from MATH500 are used to ensure findings are representative. For the Model Collapse Step analysis, results are compared across varying hyperparameters (mini-batch size ∈ {1, 8, 64} and rollout count ∈ {8, 16, 32}) to verify ranking consistency. No formal cross-validation over prompt splits is used, as the training set consists of distinct math problems and the evaluation benchmarks (AIME 2024, AIME 2025, AMC 2023) are separate held-out datasets.

Main Quantitative Results

The Rise and Fall of Intrinsic URLVR (Section 4.1)

Headline result: Intrinsic URLVR universally exhibits a rise-then-fall pattern across all methods, hyperparameters, and models. Collapse is a matter of when, not whether.

Majority voting vs. ground-truth training (Figure 2). Training Qwen3-1.7B-Base on DAPO-17k with majority voting reward shows that intrinsic rewards initially match or exceed ground-truth training across all three evaluation benchmarks during the early phase. On AIME 2024, both majority voting and ground-truth training reach approximately 0.06–0.08 avg@32 during early steps. On AMC 2023, majority voting matches ground-truth training at approximately 0.32–0.36 avg@32. However, while the proxy Majority Voting Reward continues rising (reaching approximately 0.48–0.56 by step 260), both Reward Accuracy and validation performance decline: AIME 2024 avg@32 drops to approximately 0.02 by step 260, AIME 2025 stays near 0.015, and AMC 2023 declines to 0.24–0.28. Actor Entropy is driven down faster under majority voting than under ground-truth training (Figure 2, third panel), connecting reduced uncertainty to eventual collapse.

Hyperparameter sweep results (Appendix B.3). The paper thoroughly sweeps four hyperparameters across five intrinsic reward methods:

• Training temperature (Figure 16, 20–23): For majority voting, temperature 1.0 provides the best balance, with low temperatures (0.6–0.8) causing rapid sharpening and unstable Label Accuracy, while higher temperature (1.2) maintains stability longer but reduces peak performance. For certainty-based methods, higher temperature (1.2) significantly delays collapse for Token-Level Entropy, Trajectory-Level Entropy, and Probability by maintaining elevated Actor Entropy. Self-Certainty shows contrasting behavior, where temperature 1.0 provides the most stable performance and temperature 1.2 leads to excessive exploration without convergence.

• Mini-batch size (Figure 17, 24–27): For majority voting, mini-batch size 1 drives rapid collapse within 20 steps, while pure on-policy training (mini-batch 64, matching global batch) provides maximum stability. Self-Certainty exhibits exceptional robustness to mini-batch size variations, with minimal sensitivity across the full range, while Token-Level Entropy, Trajectory-Level Entropy, and Probability all benefit from larger mini-batch sizes.

• KL regularization (Figure 18, 28–31): Adding KL regularization (coefficient 0.005) yields only marginal benefits for majority voting — small early gains in Label Accuracy but increased training variance and minimal delay in collapse. The same pattern holds across all certainty-based methods: KL regularization neither prevents eventual collapse nor significantly improves validation scores. Self-Certainty with KL regularization maintains stability slightly better but still does not escape the fundamental pattern.

• Rollout count (Figure 19, 32–35): For majority voting, larger rollout counts accelerate collapse: N=32 collapses within 180 steps, N=16 within 220 steps, while N≤8 remains stable over the full epoch. The paper recommends N=8 for best statistical reliability while maintaining convergence control. For certainty-based methods (Token-Level Entropy, Trajectory-Level Entropy, Probability), the same pattern holds: larger rollout counts accelerate convergence and premature collapse. Self-Certainty again shows unique robustness, maintaining consistent performance across all rollout configurations without collapse.

Critically, no hyperparameter setting prevents collapse — even the most stable configurations (large mini-batch, low rollout count, moderate temperature) only delay failure. The paper's extended training experiment (Section 4.1.1) with all stabilizing insights combined still shows collapse around 1,000 steps (~4 epochs), demonstrating this is "a fundamental limitation rather than an engineering problem."

The three failure modes (Figure 3). Training Qwen3-1.7B-Base on DAPO-17k with five intrinsic rewards reveals distinct collapse patterns:

• Gradual degradation (Self-Certainty, Majority Voting): These methods degrade most slowly, maintaining Label Accuracy above 0.20 for Self-Certainty and above 0.16 for Majority Voting at step 260. Validation performance on AIME 2024 reaches approximately 0.06–0.07 for both methods at peak before declining. Actor Entropy decreases gradually for Self-Certainty (from 1.5 to ~1.2) and more sharply for Majority Voting (from 1.5 to ~0.8). Self-Certainty's robustness stems from sharpening against a uniform distribution at each token position, which is less aggressive than direct probability maximization. Majority Voting operates at the answer level, avoiding token-level artifacts.

• Length collapse (Probability): The Probability reward causes Mean Response Length to drop from ~3,500 to near 0 by step 260, while Actor Entropy decreases from 2.0 to near 0. Validation performance stays near zero throughout (AIME 2024 avg@32 ~0.01, AMC 2023 ~0.06). The model learns to produce confident but overly brief answers because multiplying token probabilities inherently favors shorter sequences.

• Repetition collapse (Token-Level Entropy, Trajectory-Level Entropy): Both entropy methods show Mean Response Length initially around 3,000–4,000, with Token-Level Entropy increasing to approximately 4,500 by step 260 while Actor Entropy drops from 1.5 to ~0.5. Validation performance peaks early (Token-Level Entropy: AIME 2024 ~0.08 at step ~20, then drops to near 0; Trajectory-Level Entropy: AIME 2024 ~0.05 at peak, then declines). The model pads sequences with repetitive high-probability tokens to minimize average entropy without improving correctness.

Per-Problem Sharpening Analysis (Section 4.2.1)

Headline result: Training amplifies initial preferences rather than correcting errors. Of 25 individually trained problems, only 3 (12%) flipped correctness; the remaining 22 simply sharpened the model's initial preference regardless of correctness.

Four distinct patterns (Figure 4, Figure 15). Training Qwen3-1.7B-Base on 25 individual MATH500 problems with Trajectory-Level Entropy as reward for 100 epochs reveals:

• Amplifying success (ID 262, 146, 258): These problems start with correct greedy decoding (blue heatmap at epoch 0) and the highest-reward sample remains correct throughout (green wave at 1). Training deepens the blue color, increasing confidence in the already-correct solution.

• Amplifying failure (ID 222, 422): The highest-reward sample is almost always wrong (green wave near 0), and training deepens the red color, amplifying incorrect solutions.

• Wrong → Correct (ID 76, 131): Greedy decoding starts wrong (red) but the highest-reward sample is usually correct (green wave mostly at 1), guiding the model from wrong to correct (red → blue transition).

• Correct → Wrong (ID 420): The model starts correct (blue) but the highest-reward sample fluctuates inconsistently between correct and wrong, causing gradual degradation (blue → red).

Among the 3 problems where correctness flipped, the color still deepens over training, showing sharpening occurs regardless of whether correctness changes. The key factor is whether the highest-reward sample is mostly correct — which is determined by the model's initial probability distribution, not by the training process.

Out-of-Distribution Cross-Problem Generalization (Section 4.2.2)

Headline result: Even when training amplifies errors on in-distribution problems, the sharpening effect can generalize to improve performance on unseen problems where the model's initial confidence aligns with correctness.

Cross-problem generalization experiment (Figure 5). Training on 6 MATH500 problems where the highest-reward sample is mostly wrong (Train Label Accuracy near 0.2–0.6, blue curve) for 100 epochs, the model's performance on two unseen test problems (ID 76 and ID 131) transitions from wrong to correct. Test Label Accuracy increases steadily from 0 to 1 on both problems (orange and green curves). This demonstrates that even when all training problems have wrong initial answers and intrinsic URLVR amplifies their failures, the sharpening can still generalize to OOD problems where confidence aligns with correctness — the model learns the general skill of making more decisive predictions, which helps on problems where its initial preferences happen to be right.

Small Datasets Prevent Model Collapse (Section 5.1)

Headline result: Training with ≤128 samples maintains stable performance without collapse, while larger datasets (≥512) consistently exhibit reward hacking across 3 random seeds.

Dataset size scaling (Figure 6). Training Qwen3-1.7B-Base on randomly sampled DAPO-17k subsets with sizes {32, 128, 512, 2048, 8192, 16384}, fixing global batch size to 32 and adjusting epochs to complete 600 optimization steps:

• DAPO-32 and DAPO-128 maintain stable performance. DAPO-32 achieves rapid consensus (Majority Voting Reward → 1.0 by step ~100) while preserving high Ground Truth Reward (~0.40–0.45), and Reward Accuracy remains near 0.8. DAPO-128 shows similarly stable behavior with only slight degradation in Reward Accuracy toward the end of training.

• DAPO-512 and larger exhibit clear reward hacking. By step 600, Majority Voting Reward rises to 0.8–1.0 for all settings DAPO-512 and above, while Reward Accuracy drops to ~0.2 for DAPO-512 and near 0 for DAPO-16384. Ground Truth Reward declines from initial peaks across all larger datasets.

• Seed stability: Verified across 3 random seeds for sizes {32, 128, 512}, DAPO-32 never collapses while DAPO-512 always does, confirming the effect is not due to random sampling variation.

KL divergence analysis (Figure 7). Measuring KL divergence from the reference model at each training step ($D^{(t)}_{\text{KL}}(\pi^{(t)}_{\theta} \parallel \pi_{\text{ref}})$) reveals that smaller datasets induce far smaller distributional shifts. DAPO-32 reaches only 0.057 KL after 600 steps, while DAPO-512 reaches approximately 0.12 — roughly 2× higher. This supports the hypothesis that small datasets induce localized overfitting (learning isolated facts on specific problems through localized parameter updates) rather than systematic policy shift (altering the model's global output distribution). The limited drift preserves general reasoning capability on OOD benchmarks.

Test-Time Training as a Safe Application (Section 5.2)

Headline result: Intrinsic URLVR can be safely applied in test-time training, where small domain-specific datasets prevent collapse while enabling consistent performance gains.

Test-time vs. train-time comparison (Figure 8). Training on AMC23 (40 problems, test-time) versus DAPO-17k (~17,000 problems, train-time), both with batch size 40:

• AMC23 (test-time): Ground Truth Reward rises from ~0.15 to ~0.45 and stabilizes without decline. Majority Voting Reward rises and stabilizes around 0.8–0.9. Performance on AMC23 increases from ~0.18 avg@32 to ~0.38 by step 600. Performance on AIME 2024 (OOD) also improves from ~0.02 to ~0.06, demonstrating test-time training can produce gains that generalize.

• DAPO-17k (train-time): Shows the familiar rise-then-fall: Ground Truth Reward peaks at ~0.25 around step 100 then declines to ~0.15. Majority Voting Reward rises to 0.6+ but validation accuracy on both AIME 2024 and AMC 2023 peaks early and then falls.

This validates why many recent works using intrinsic rewards focus on test-time rather than train-time settings (Prabhudesai et al., 2025; Zuo et al., 2025).

Incorrect Majority Votes Still Improve Reasoning (Section 5.3)

Headline result: Even when almost all training samples have incorrect initial majority votes, small-dataset training avoids catastrophic collapse and yields gains on OOD benchmarks.

Extreme incorrect-majority experiment (Figure 9). Filtering DAPO-17k to select 32 samples where initial maj@64 is incorrect in a non-negligible proportion (controlling for majority ratios >40%), then training with majority voting reward (maj@8 with temperature 1.0):

• Label Accuracy (whether maj@8 during training matches ground truth) shows non-zero values at only 1–2 early steps, then consistently drops to zero by step 100. The Majority Voting Reward converges to ~0.85.

• Despite essentially all training samples having incorrect pseudo-labels, validation performance improves: AIME 2024 avg@32 rises from ~0.042 to ~0.058 by step 200, and AMC 2023 avg@32 rises from ~0.33 to ~0.37. No catastrophic collapse occurs.

This extreme case demonstrates that small-scale training operates under fundamentally different dynamics than large-scale training. Even when training amplifies errors on the specific training samples, the localized parameter updates do not cause global policy collapse, and the sharpening skill can transfer to OOD problems where the model's initial preferences happen to be correct.

Model Collapse Step Predicts RL Trainability (Section 6)

Headline result: Model Collapse Step strongly correlates with actual RL gains (GT Gain), matches or exceeds pass@k's predictive power, and requires 5.6× fewer tokens with no ground-truth labels.

Pilot study across model families and training stages (Figure 10). Training four models on DAPO-17k with majority voting reward reveals contrasting stability:

• Qwen family: DeepSeek-R1-Distill-Qwen-1.5B (SFT variant) maintains Reward Accuracy above 0.8 throughout 260 steps, while Qwen2.5-1.5B (base) drops to near zero by step 200. Despite starting with higher Actor Entropy (3.6 vs. 1.5), the base model collapses much faster. Validation performance reflects this: DS-R1-1.5B reaches AIME 2025 avg@32 of ~0.18 at peak, while Qwen2.5-1.5B peaks at ~0.04.

• LLaMA family: Both Meta-Llama-3.1-8B (base) and Llama-3.1-Tulu-3-8B-SFT eventually collapse, but at different rates. The base model fails by step 40, while the SFT variant shows an initial performance rise (AMC 2023 avg@32 from ~0.01 to ~0.08) before collapsing later. Both variants show similar peak AIME 2025 performance (~0.003–0.005), far below Qwen family models.

This architectural difference highlights that Qwen models have fundamentally more stable intrinsic RL dynamics, and that higher initial entropy is a consequence of the sharpening process, not a predictor of successful RL training.

Correlation with GT Gain (Figure 11, Table 3). Across 7 models from 3 families evaluated on AIME24:

• Model Collapse Step values: OLMo-2-1124-7B (34 steps), Meta-Llama-3.1-8B (40), Qwen2.5-Math-1.5B (160), Qwen2.5-1.5B (221), Qwen2.5-7B (245), Qwen3-1.7B-Base (280), Qwen3-8B-Base (383).

• GT Gain values: +0.42, +1.01, +3.96, +3.96, +6.67, +7.08, +17.08 (percentage points on AIME24 avg@32).

• Correlation: The ranking by Collapse Step closely matches the ranking by GT Gain. Models that survive longer during intrinsic URLVR training (larger Collapse Step) consistently yield better results in standard supervised RL training. This validates that Collapse Step captures a meaningful measure of model prior relevant to RL trainability.

• Comparison to pass@k Gain: pass@k Gain (pass@256 − pass@1) values: +6.67, +3.33, +30.00, +20.00, +60.00, +36.67, +56.67. The correlation with GT Gain is weaker — for example, Qwen2.5-7B shows pass@k Gain of 60.00 but GT Gain of only 6.67, while Qwen3-8B-Base shows pass@k Gain of 56.67 but GT Gain of 17.08. pass@k overestimates trainability for some models.

Rapid assessment with aggressive hyperparameters (Figure 12). Varying mini-batch size and rollout count to accelerate collapse preserves model rankings:

• Varying rollout count N (Figure 12, left): Across N ∈ {8, 16, 32}, the relative ordering of models by Collapse Step remains stable. With N=32 (most aggressive), Qwen3-8B-Base collapses at ~150 steps vs. OLMo-2-1124-7B at ~20 steps — preserving the approximately 7–8× ratio between best and worst models.

• Varying mini-batch size (Figure 12, right): Across MBS ∈ {1, 4, 64}, rankings remain consistent. With MBS=1 (most aggressive), Qwen3-8B-Base collapses at ~195 steps vs. OLMo-2-1124-7B at ~22 steps.

• Computation cost (Table 3): Under aggressive hyperparameters (MBS=1, N=8), total collapse steps across 7 models are [22, 14, 19, 112, 128, 172, 195] = 662 steps. Total tokens: 7k × 8 × 662 × 32 = 1.19B. Gold standard GT Gain: 7k × 8 × 17k × 7 = 6.66B. Ratio: 5.6× faster.

Self-Verification External Rewards Escape Collapse (Section 7)

Headline result: Self-verification on Countdown arithmetic puzzles shows sustained improvement without the collapse patterns inherent to intrinsic methods, though success depends on instruction-following capability.

Self-verification vs. intrinsic vs. oracle (Figure 13). Training Qwen3-1.7B-Base and Qwen3-4B-Base on 4,000 Countdown problems:

• Validation accuracy (avg@16): For Qwen3-1.7B-Base, self-verification reaches approximately 0.55 by step 600, while Trajectory-Level Entropy peaks around 0.30 and then declines toward 0.25. Oracle Supervision (upper bound) reaches approximately 0.72. For Qwen3-4B-Base, self-verification reaches approximately 0.75 by step 500, while Trajectory-Level Entropy peaks around 0.30 and declines.

• Training dynamics (Figure 13, bottom): Self-verification's Reward Accuracy initially drops from ~0.75 to ~0.55 around step 200 as the policy explores and tries to exploit the verifier, then recovers and stabilizes above 0.55. Ground Truth Reward continues rising throughout training for both models. This recovery pattern is categorically different from intrinsic methods, where Reward Accuracy declines monotonically toward zero.

Instruction alignment and prompt sensitivity (Figure 14). Comparing Qwen3-1.7B-Base vs. Qwen3-1.7B (instruction-tuned) with two verification prompts:

• Qwen3-1.7B (instruction-tuned): Starts above 60% accuracy with both prompts and improves to over 80% by step 600. Reward Accuracy remains stable near 0.7–0.75 for both prompts. Self-Verify Reward tracks closely with Ground Truth Reward.

• Qwen3-1.7B-Base: Only works with Prompt 2 (the more structured, explicit checklist). With Prompt 1, validation accuracy stays near 0.2 and Self-Verify Reward collapses to near 0. With Prompt 2, reaches approximately 0.55 but with lower Reward Accuracy (~0.55) and more prompt sensitivity than the instruction-tuned model.

This demonstrates that self-verification's success depends on the model's ability to follow verification instructions — a capability that improves with instruction alignment training — and that the verification prompt must be carefully designed for base models.

Ablation Studies and Robustness Checks

Hyperparameter tuning across five intrinsic methods (Appendix B.3): The paper conducts an extensive sweep of four hyperparameters (training temperature, mini-batch size, KL regularization, rollout count) for each of five intrinsic reward methods (Majority Voting, Self-Certainty, Token-Level Entropy, Trajectory-Level Entropy, Probability), totaling approximately 20 distinct experimental configurations. All methods eventually degrade across all settings, confirming the universality of collapse (see Section 4.1 and Appendix B.3 for detailed results per hyperparameter).

Extended training with best hyperparameters (Section 4.1.1): Combining all stabilizing insights from hyperparameter tuning (optimal temperature, large mini-batch, low rollout count) and extending training to 1,000 steps still results in collapse — demonstrating that the rise-then-fall pattern is a fundamental limitation, not an engineering problem.

Dataset type scaling (Appendix C.3, Figure 41): Training on MATH-8k, DAPO-17k, DeepScaleR-40k, and ORZ-56k shows that larger datasets consistently exhibit reward hacking, while smaller datasets (MATH-8k at ~8,000 problems, DAPO-17k at ~17,000 problems) show early-stage stability or rise patterns. The paper observes that current intrinsic methods show "short-sighted incremental improvements at the early stage, while extending to much larger training corpora inevitably encounter reward hacking." This confirms dataset size as a causal factor rather than dataset source.

Backbone model architecture and training stage (Appendix C.2, Figures 37–40): Testing 11 models from Qwen and Llama families reveals that (1) Qwen models exhibit fundamentally greater stability than Llama models, with math-specialized and SFT variants showing superior stability within each family, (2) smaller models within the same architecture generation (Qwen3-1.7B vs. Qwen3-4B; Octo-3B vs. Octo-8B) consistently outperform larger variants in stability, with larger models' increased capacity amplifying sensitivity to noisy pseudo-rewards and accelerating convergence toward degenerate solutions, and (3) newer architecture generations (Qwen3 vs. Qwen2.5) show improved stability, likely due to better-calibrated uncertainty estimates.

Does intrinsic URLVR truly improve capabilities or merely self-consistency? (Appendix C.4, Table 9): Applying TTRL (Majority Voting-based URLVR) to Qwen2.5-Math-1.5B and Qwen2.5-Math-7B on AIME 2024, the TTRL-trained models' avg@32 significantly exceeds the base models' majority-vote performance at large sample sizes. For Qwen2.5-Math-1.5B, maj@1024 achieves 37.30 accuracy, while the TTRL-trained model's avg@32 reaches 48.90 — a gain of +11.60 points beyond the base model's majority-vote ceiling. For Qwen2.5-Math-7B: maj@1024 achieves 50.79, TTRL avg@32 reaches 68.10 — a gain of +17.31 points. This demonstrates that TTRL genuinely enhances the model's ability to generate accurate predictions, not merely aligning outputs with existing majority preferences.

Different intrinsic methods, different failure modes (Figure 3, Section 4.1.2): As detailed in the Main Quantitative Results, the five intrinsic methods exhibit three distinct collapse patterns (gradual degradation, length collapse, repetition collapse), confirming that while all methods share the sharpening mechanism, the specific reward formulation determines how collapse manifests operationally.

Per-problem sharpening tracking (Figure 4, Figure 15, Section 4.2.1): Training on 25 individual problems with Trajectory-Level Entropy reward for 100 epochs shows that only 3 of 25 problems (12%) change correctness, while the remaining 22 simply sharpen the initial preference — confirming the sharpening mechanism at fine granularity.

OOD cross-problem generalization (Figure 5, Section 4.2.2): Training on 6 problems where the highest-reward sample is mostly wrong still produces Test Label Accuracy improvements from 0 to 1 on two unseen problems, demonstrating that the sharpening skill can generalize across problem boundaries.

Dataset size and seed robustness (Figures 6–7, Section 5.1): Across 3 random seeds, DAPO-32 never collapses while DAPO-512 always does. The KL divergence analysis (Figure 7) quantifies the mechanism: DAPO-32 reaches 0.057 KL vs. DAPO-512 at ~0.12 KL, supporting the localized-overfitting explanation.

Incorrect majority votes on small datasets (Figure 9, Section 5.3): Filtering to 32 samples with deliberately incorrect initial majority votes, the model still avoids catastrophic collapse and produces OOD gains — AIME 2024 from 0.042 to 0.058, AMC 2023 from 0.33 to 0.37. This demonstrates robustness to even the most adverse initial conditions when dataset size is small.

Model Collapse Step hyperparameter robustness (Figure 12, Section 6.3): Varying mini-batch size ∈ {1, 4, 64} and rollout count ∈ {8, 16, 32} preserves model rankings, enabling aggressive hyperparameters for rapid assessment.

Self-verification prompt sensitivity (Figure 14, Section 7): Testing two verification prompts across base and instruction-tuned models reveals that instruction alignment provides robustness to prompt choice, while base models require carefully structured verification prompts (Prompt 2 works; Prompt 1 fails). This identifies instruction-following capability as a key bottleneck for external URLVR methods using self-verification.

Critical Assessment

Does the paper demonstrate that all intrinsic URLVR methods converge toward a sharpening mechanism that causes inevitable collapse?

The theoretical derivation (Section 3, Theorem 1) provides a mathematically rigorous argument that under the standard KL-regularized RL objective with majority voting reward, the policy converges geometrically to determinism on the initial majority answer. The empirical validation (Section 4) demonstrates that this pattern — early gains followed by collapse — holds across five different reward formulations, multiple hyperparameter configurations (Appendix B.3), multiple models (Appendix C.2), multiple datasets (Appendix C.3), and three evaluation benchmarks. The per-problem analysis (Section 4.2.1) directly observes the sharpening mechanism at work: on 22 of 25 problems, training amplifies existing preferences without changing correctness.

However, several caveats limit the strength of this conclusion:

(1) The theoretical analysis only rigorously proves convergence for majority voting (binary reward case). The extension to other intrinsic rewards (Appendix A.3–A.5) is described as a "proof sketch" by the authors themselves (Appendix A.4: "A fully rigorous treatment requires additional technical conditions that we validate empirically"). For Self-Certainty ($\sigma = +1$), the analysis is acknowledged as requiring "separate analysis" because it does not satisfy Reward-Confidence Monotonicity in the same way. The paper relies on empirical evidence to argue that these methods still sharpen distributions, which is a weaker claim than a unified mathematical proof. The unified reward framework demonstrates structural similarity but not necessarily identical convergence properties for all instantiations.

(2) The empirical evidence for "all methods collapse" relies heavily on a single training configuration (Qwen3-1.7B-Base on DAPO-17k with 1 epoch). While the paper tests multiple methods, hyperparameters, and models (Appendix C.2), the number of fully independent training runs is difficult to assess. The hyperparameter sweeps vary one parameter at a time from a single base configuration. The model family comparison tests 11 models but only with majority voting reward and default hyperparameters. This is broader than most studies in the field, but a factorial design (all methods × all hyperparameter settings × all models × all datasets) would provide much stronger evidence for universality. The current design leaves open the possibility that some untested combination (e.g., a specific model family with a specific reward formulation and specific hyperparameters) might avoid collapse — though the paper's argument is that this is unlikely because the sharpening mechanism is structural.

(3) The definition of "collapse" varies across experiments. For majority voting, collapse is defined by Reward Accuracy dropping below 1% (the Model Collapse Step definition) or, more qualitatively, by Ground Truth Reward diverging from Majority Voting Reward. For certainty-based methods, collapse manifests as length collapse, repetition collapse, or gradual degradation — but the paper does not provide a uniform quantitative threshold for declaring collapse across all methods. This makes cross-method comparison of "when collapse occurs" somewhat subjective, though the qualitative patterns in Figure 3 are visually clear.

(4) The paper does not explore whether the sharpening mechanism's rate can be controlled through adaptive reward design. The theoretical analysis assumes a fixed reward function. If the reward were designed to explicitly penalize overconfidence (e.g., by adding an entropy bonus, which is common in RL and which the GRPO algorithm can implement natively), the sharpening dynamic might be slowed or even reversed. The paper tests KL regularization (Appendix B.3) and finds it insufficient, but does not test entropy regularization (which would directly counter sharpening by rewarding high-entropy policies) or dynamic temperature adjustment. The claim that "all intrinsic methods collapse" might be more precisely stated as "all intrinsic methods collapse under standard RLVR training configurations that do not actively penalize distribution sharpening."

Does Model Collapse Step genuinely predict RL trainability better than pass@k?

The evidence in Figure 11 shows strong visual correlation between Model Collapse Step ranking and GT Gain ranking across 7 models. The Collapse Step values span a wide range (34 to 383 steps) and the ordering closely matches GT Gain. The pass@k Gain values are less well-ordered with respect to GT Gain, as noted in the results. However:

(1) N = 7 models is a small sample for assessing predictive accuracy. With only 7 data points, the correlation (whether measured by Spearman rank correlation or visual inspection) is suggestive but not statistically robust. A single outlier could significantly change the apparent correlation. The paper does not report a quantitative correlation coefficient or confidence interval.

(2) The comparison to pass@k uses only one specific formulation (pass@256 − pass@1 on AIME24). pass@k is a family of metrics parameterized by k, and the choice of k=1 and k=256 may not be optimal. Additionally, alternative static metrics (e.g., the model's average entropy on evaluation prompts, its calibration error, or its performance on related benchmarks) are not compared. The claim that Model Collapse Step "matches and even surpasses pass@k's predictive power" (Section 6.2) is based on a single instantiation of pass@k and a small sample size.

(3) Model Collapse Step requires running actual RL training with majority voting reward. While this is 5.6× cheaper than full supervised RL training (Table 3), it still requires ~1.19B tokens of computation across 7 models and access to a training dataset (DAPO-17k). pass@k requires only inference on the evaluation benchmark, which may be cheaper or more expensive depending on the evaluation set size and the value of k. The practical advantage depends on the specific deployment context — if the evaluation benchmark is large (thousands of problems), pass@k might be more expensive; if it is small (dozens), pass@k might be cheaper.

(4) The paper does not demonstrate that Model Collapse Step generalizes across training datasets. All measurements use DAPO-17k as the training set. If a practitioner wants to assess whether a model is suitable for RL training on a different dataset (e.g., a code generation dataset), it is unclear whether Model Collapse Step measured on DAPO-17k would transfer, or whether it would need to be re-measured on the target dataset.

Does the self-verification experiment demonstrate that external rewards escape the confidence-correctness ceiling?

The Countdown experiments (Section 7, Figures 13–14) provide suggestive but preliminary evidence. Self-verification shows sustained improvement or stability where Trajectory-Level Entropy declines, and the Reward Accuracy recovery pattern (dropping then recovering) is qualitatively different from intrinsic methods' monotonic decline. However:

(1) The comparison is to only one intrinsic method (Trajectory-Level Entropy). The paper does not compare self-verification against Majority Voting, Self-Certainty, or the other intrinsic methods on Countdown. It is possible that some intrinsic methods would also avoid collapse on Countdown due to task-specific properties (e.g., the discrete, verifiable answer structure might make majority voting more reliable than on open-ended math problems). A full comparison of self-verification against all five intrinsic methods on Countdown would be needed to confidently attribute the different dynamics to the reward source rather than the task.

(2) The Countdown task is significantly simpler than the MATH benchmark used for intrinsic method evaluation. Countdown involves arithmetic puzzle-solving with a small fixed set of operations, while MATH problems require multi-step algebraic reasoning, geometric insight, and sometimes proof construction. The generation-verification asymmetry is starker on Countdown (verification is a handful of arithmetic operations) than on MATH (verification requires checking the logical validity of a multi-step reasoning chain, which is itself non-trivial). The paper's claim that external rewards "may escape the confidence-correctness ceiling" (Section 7 takeaway) might be specific to tasks with simple, deterministic verification procedures, and may not generalize to domains where verification itself is hard or ambiguous.

(3) Self-verification is not purely "external" in the same sense as execution-based code verification or Lean proof checking. The verification is performed by the model itself through a verification prompt, which means the verifier's accuracy depends on the model's capability (as demonstrated by the prompt sensitivity in Figure 14). If the model's verifier degrades in quality as the model's generation policy shifts — for instance, if the model learns to generate expressions that are subtly incorrect but its verifier fails to catch the error — the reward signal could become corrupted. The paper observes the initial Reward Accuracy drop (Figure 13, bottom, step ~200) as the policy explores ways to exploit the verifier, but the subsequent recovery suggests the verifier remained sufficiently robust. Whether this robustness holds at larger scales or on harder tasks is not tested.

(4) The experiment uses only 4,000 training problems and 600 training steps. The intrinsic URLVR experiments typically use 17,000 problems and ~260 steps. The longer training horizon for self-verification (600 steps) is promising but still relatively short compared to production RL training runs that can span thousands of steps. It is possible that self-verification would eventually exhibit reward hacking at longer training horizons if the policy discovers adversarial examples that fool the verifier.

What experiments would strengthen the paper's claims?

Several experiments would address the limitations identified above:

(1) A factorial experiment crossing all five intrinsic methods with all major hyperparameter settings (temperature, mini-batch size, rollout count) on at least two model families (Qwen and Llama) and two datasets (DAPO-17k and MATH-8k). This would provide much stronger evidence for the universality claim by demonstrating that no combination in the Cartesian product avoids collapse. The current evidence is broad but not fully crossed.

(2) An entropy-regularized version of intrinsic URLVR, where the RL objective explicitly includes an entropy bonus (in addition to or instead of KL regularization). This would test whether the sharpening mechanism can be counterbalanced by directly rewarding diversity, potentially extending the useful training horizon. The GRPO algorithm supports entropy regularization natively; the paper's choice to use only KL regularization (and then only as an ablation) leaves this important countermeasure unexplored.

(3) Comparison of self-verification against all five intrinsic methods on Countdown, to determine whether the superior performance of self-verification is due to the external reward structure or to task-specific properties of Countdown that might also allow certain intrinsic methods to avoid collapse.

(4) A scaling experiment for self-verification — training for more steps (e.g., 2,000–5,000) on a larger dataset to determine whether the apparent stability is asymptotic or whether a delayed form of reward hacking eventually appears.

(5) A larger-scale evaluation of Model Collapse Step — measuring it on 20–30 models with documented RL training outcomes to provide statistically robust correlation estimates and to identify potential failure modes where Collapse Step fails to predict trainability.

(6) Cross-dataset validation of Model Collapse Step — measuring Collapse Step on one training dataset and testing whether it predicts RL gains on a different training dataset, to assess transferability.

Overall assessment. The paper's central empirical claims — that intrinsic URLVR methods universally exhibit rise-then-fall dynamics, that collapse timing is determined by model prior rather than engineering choices, and that small-dataset test-time training avoids collapse — are well-supported by the experiments presented, within the acknowledged limitations of single model family focus and the specific hyperparameter ranges tested. The Model Collapse Step metric shows promise as a practical diagnostic but requires larger-scale validation. The self-verification experiments are best viewed as a proof of concept motivating further work on external rewards rather than as a definitive demonstration that external methods scale arbitrarily. The paper's most robust contribution is the documentation of the sharpening mechanism's empirical signatures (the rise-then-fall pattern, the three failure modes, the dataset-size threshold for collapse) across a wide range of conditions, providing a clear set of empirical predictions that future work can test — and potentially falsify — on new model families, tasks, and reward formulations.

6. Limitations and Trade-offs

6.1 The Difficulty Estimation Cost Is Not Accounted For

The assumption or constraint. The paper's compute-optimal framework explicitly conditions on estimated prompt difficulty, computed by generating 2048 samples from the base model and averaging either ground-truth correctness (oracle) or PRM final-answer scores (predicted). Section 3.2 acknowledges this cost without resolving it:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. The headline efficiency gains — up to 4× over best-of-N (Figures 4 and 8) — are computed after difficulty is known, without amortizing the cost of learning it. Generating 2048 samples per question for difficulty estimation alone consumes as much or more computation than the largest test-time budgets studied (256–512 generations). The reported 4× figure is therefore an upper bound on achievable efficiency in a realistic deployment, not a realized gain. If difficulty estimation costs were amortized, the actual total compute would be difficulty estimation + strategy execution, and the former could dominate the latter. The paper does not provide any analysis of what total efficiency (estimation + solving) looks like at any budget level.

What evidence exists in the paper. The paper's cross-validation protocol (Section 3.2) and the difficulty-bin analyses (Figures 3 right, 4, 7 right, 8) all treat difficulty as a given input to the allocation policy. The predicted-difficulty results (using PRM scores instead of ground-truth labels) show that the method works without oracle access, but still require the 2048-sample generation step. Figure 4 shows that predicted-difficulty compute-optimal curves largely overlap with oracle-difficulty curves for search, and Figure 8 shows a small gap at high budgets for revisions (roughly 41% vs. 44% at 256 generations). The text in Section 3.2 explicitly frames this as an "exploration-exploitation tradeoff" but does not quantify it.

Mitigation status. The paper does not attempt to mitigate this limitation. Section 8 suggests future work on "pretraining or finetuning models to directly predict difficulty of a question" and on "adaptive difficulty estimation" that amortizes difficulty assessment into the problem-solving process, but no such model is developed or evaluated. Until difficulty can be predicted cheaply (e.g., from the question text alone, or from a small number of initial samples as part of a bandit-style allocation), the 4× figure should be understood as an aspirational upper bound rather than a realized deployment gain. This is potentially the most consequential limitation for practitioners because it means the paper's central efficiency claim does not reflect what a deployed system would actually achieve.

---

6.2 Hard Problems Remain Essentially Unsolved — Test-Time Compute Cannot Substitute for Missing Capability

The assumption or constraint. The paper's FLOPs-matched analysis (Section 7) demonstrates that test-time compute with a smaller model can outperform a ~14× larger model — but only on problems where the base model's pass@1 is non-trivially above zero. The paper is transparent about this boundary. For difficulty bin 5 (the hardest problems), the results show that no amount of test-time compute helps.

The consequence. Test-time compute can amplify existing capability but cannot create it from nothing. If the base model has near-zero probability of generating a correct solution on a problem class, search and revisions cannot find or refine what is not present in the proposal distribution. The FLOPs-matched comparison reveals sharp boundaries:

In the revision model comparison (Figure 9, left; Figure 1 bar chart), hard problems (bins 4–5) show a −37.2% relative disadvantage for test-time compute at $R \gg 1$ (high inference volume) compared to simply using the larger pretrained model. For PRM search (Figure 9, right), hard problems show a −52.9% relative disadvantage at $R \gg 1$, and even at $R \ll 1$ (low inference volume, where test-time compute gets the largest budget advantage), PRM search shows a −3.6% disadvantage.

This means that for deployment scenarios where the problem distribution skews toward genuinely hard questions — problems that are outside the base model's capability range — scaling pretraining (training a larger model) is not just preferable but is the only viable path. Test-time compute provides zero or negative benefit in this regime, and the compute-optimal policy correctly identifies that no allocation strategy helps.

What evidence exists in the paper. Figure 3 (right, bottom row) shows bin 5 accuracy hovering at 1–3% for all search methods and all budgets. Figure 7 (right) shows bin 5 at roughly 2–3% accuracy regardless of the sequential-to-parallel ratio. Figure 9 shows the bin 5 scaling line essentially flat near 0–5% for both revisions and search in the FLOPs-matched comparison. The paper's Section 7 takeaway box explicitly states: "Hard questions (bins 4–5): Pretraining is almost always more effective."

Mitigation status. The paper does not attempt to solve this limitation — and it is not clear that any test-time technique could, since the problem is the absence of correct solutions in the model's output distribution. Section 7 handles this limitation transparently by presenting the failure case alongside the success case, which is a strength of the analysis rather than a weakness. However, practitioners need to understand that the 4× efficiency gains and the ~14× pretraining substitution only apply within the base model's capability envelope. For frontier problems that genuinely exceed the model's training-distribution coverage, no inference-time budget reallocation helps. This boundary condition is empirically precise but practically significant, especially for organizations deploying models on problems of unknown difficulty.

---

6.3 Single Benchmark, Single Model Family, Small Test Set

The assumption or constraint. All experiments use the MATH benchmark (500 test questions from Hendrycks et al., 2021; Lightman et al., 2022 split) with PaLM 2-S* as the base model. The authors acknowledge this in Section 4:

"We believe this model is representative of the capabilities of many contemporary LLMs"

but provide no evidence for representativeness beyond stating the belief.

The consequence. Several aspects of the findings could be model-specific or task-specific in ways that limit generalization:

• Model specificity of over-optimization: The PRM's quality and over-optimization behavior depend on PaLM 2-S*'s output distribution. A model with different calibration properties might have a PRM that degrades at different rates or at different budget thresholds. The observation that beam search over-optimizes on easy problems and helps on medium problems (Figure 3, right) might shift to different difficulty bins for different model families.

• Task specificity of the revision model's effectiveness: MATH problems require symbolic multi-step reasoning with verifiable final answers. The revision model's success on easy problems (where editing an almost-correct solution is useful) and its failure on hard problems (where the initial solution is fundamentally wrong) might not transfer to code generation (where syntax errors might be easier to fix than logic errors), open-ended reasoning (where there is no single correct answer), or factual QA (where errors are knowledge gaps, not reasoning mistakes).

• Small test set limits statistical reliability: The 500-question test set is split into five difficulty quintiles of ~100 each, then further split by two-fold cross-validation. This means the compute-optimal policy is selected based on ~50 questions per fold per bin. The paper does not report confidence intervals on the compute-optimal scaling curves, making it impossible to assess whether the observed differences between strategies (e.g., beam search vs. best-of-N at a given budget) are statistically significant or could be reversed with a different random split.

What evidence exists in the paper. All figures report only point estimates (accuracy percentages) without error bars, confidence intervals, or statistical tests. The cross-validation protocol (Section 3.2) addresses overfitting the strategy selection to the test set but does not address the variance of the selected strategy's estimated performance. The difficulty prediction method (PRM-based scoring) is evaluated only on the same test set, not on an independent dataset. The paper provides no experiments on any benchmark other than MATH.

Mitigation status. The paper does not mitigate this limitation. The authors acknowledge it implicitly by framing their claims about PaLM 2-S* specifically, but the broader implications — that compute-optimal test-time scaling is a general principle, that 4× efficiency gains are achievable, that PRM search and revision models are complementary — are extrapolated beyond the single-benchmark, single-model evidence. A replication on even one additional benchmark (e.g., GSM8K for easier problems, MATH with a different split, or a code generation benchmark) and one additional model family would substantially strengthen generalizability. The small test set (500 questions) is a constraint inherited from the MATH benchmark's standard split; the PRM800k training dataset (12,000 questions) is adequate for training but the test set is small for the fine-grained difficulty-bin analysis.

---

6.4 The ~14× Larger Model Baseline Is Not Compute-Optimally Trained

The assumption or constraint. The FLOPs-matched comparison in Section 7 scales model parameters while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023) rather than Chinchilla-optimal training (Hoffmann et al., 2022). The paper explicitly acknowledges this:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

Additionally, the larger model is evaluated using only greedy decoding with no test-time compute augmentation — no majority voting, no best-of-N, no search.

The consequence. A Chinchilla-optimal model trained with 14× more total FLOPs (scaling both parameters and data) would likely outperform a parameter-only-scaled model. The reported advantages of test-time compute over pretraining (e.g., +27.8% on easy/medium questions at $R \ll 1$ for revisions, Figure 1 bar chart) are measured against a suboptimal pretraining baseline. The 14× parameter scaling is an engineering choice that reflects common practice (the LLaMA family is trained this way), but it is not the strongest possible pretraining baseline.

The greedy-decoding-only baseline for the larger model is a separate concern. The paper's argument is that test-time compute with a small model can substitute for pretraining a large model, but it compares a small model with sophisticated inference-time strategies (beam search, revision chains, adaptive allocation) against a large model without any inference-time strategies. A fairer comparison would give the larger model some test-time compute budget — for example, best-of-8 or best-of-16 — which might close some of the gap. The FLOPs accounting would need to include the larger model's inference-time computation, but since the larger model's per-token cost is 14× higher, a small inference budget for the larger model might still leave room for test-time compute advantages.

What evidence exists in the paper. Figure 9 shows the FLOPs-matched comparison with the ~14× larger model's greedy performance marked as stars. The paper does not ablate the choice of pretraining scaling paradigm (parameter-only vs. Chinchilla-optimal) or the choice of greedy-only evaluation for the larger model. Section 7 states the parameter-only assumption and briefly notes the Chinchilla-optimal alternative but does not test it or bound its effect.

Mitigation status. The paper acknowledges the limitation explicitly (Section 7) and frames it as future work. The parameter-only scaling choice is defensible as representative of common practice (LLaMA, Qwen, and many other model families scale parameters faster than data), and the paper's contribution in this section is primarily to demonstrate that a regime exists where test-time compute is competitive with pretraining, not to claim universal superiority. However, until a Chinchilla-optimal comparison is performed, the quantitative advantage of test-time compute over pretraining (the +27.8%, +19.1%, etc.) should be treated as favorable-to-test-time-compute upper bounds rather than neutral estimates.

---

6.5 Revisions and Search Are Studied Independently — The Combined Potential Is Unexplored

The assumption or constraint. The paper studies two complementary axes of test-time compute — PRM-guided search (Section 5) and iterative revisions (Section 6) — but never combines them. Section 8 explicitly acknowledges:

"we did not experiment with PRM tree-search techniques in combination with revisions"

The consequence. The two mechanisms have complementary, difficulty-dependent strengths: revisions improve the proposal distribution (generating better candidates through iterative refinement, most effective on easy problems), while PRM search improves candidate selection (finding the best among generated candidates, most effective on medium problems). The paper demonstrates that both individually yield 4× efficiency gains over best-of-N (Figures 4 and 8), but combining them — using the revision model as the proposal distribution within beam search, or using the PRM to guide which revisions to pursue — could yield gains beyond either method alone. The current results therefore represent a lower bound on what a fully integrated system could achieve.

More importantly, the paper's central framework (Section 2) positions revisions and verifier-guided search as two axes of a unified test-time compute paradigm, with the compute-optimal policy selecting between them per prompt. But the paper only selects between search algorithms or between sequential/parallel revision ratios — it never selects between search and revisions as alternative strategies for the same prompt, and it never combines them in a hybrid approach where, for example, a revision chain is generated and then beam search is applied to select among revisions.

What evidence exists in the paper. Sections 5 and 6 are presented as independent analyses with separate compute-optimal curves, separate difficulty-bin breakdowns, and separate FLOPs-matched comparisons. There is no figure showing a combined search+revision policy or comparing search and revisions head-to-head on the same prompts. The paper does not report the difficulty-bin overlap — for instance, whether the problems that benefit most from beam search (bin 3–4) are the same problems that benefit most from revisions, or whether they are different problems where a combined strategy would be additive.

Mitigation status. The paper explicitly identifies this gap as future work in Section 8, but does not provide any analysis of what combined gains might look like. This is a significant omission because it means the paper's central claim — that compute-optimal scaling is about adaptively allocating between different strategies — is tested only within strategy families (within search algorithms, within revision ratios), not across families. The adaptive allocation that the framework promises (Section 3.1, Equation 1) is more general than what the experiments implement. For practitioners, this means the paper provides evidence that search helps on some problems and revisions help on others, but does not tell them whether deploying both in a unified system with a meta-policy that routes each problem to the right mechanism would provide additional gains, or whether the gains from each mechanism are largely overlapping.

---

6.6 Latency and Wall-Clock Time Are Not Considered — Only Total Generation Count

The assumption or constraint. The paper measures compute exclusively in generations — the number of complete solutions sampled, with adjustments for lookahead search's extra rollout costs (Section 5.3). This is a reasonable proxy for total FLOPs but ignores the distinction between parallel and sequential computation.

The consequence. Sequential revisions are inherently serial — each revision depends on the previous one, so a chain of length $L$ takes $L$ times the wall-clock time of a single generation. Parallel best-of-N can be executed simultaneously with sufficient hardware. A compute-optimal strategy that allocates, say, $N = 256$ generations as $S = 64$ sequential revisions across $P = 4$ parallel chains uses the same total FLOPs as $N = 256$ parallel best-of-N, but requires approximately $64\times$ longer wall-clock time to complete the sequential dependencies.

This matters critically for the paper's difficulty-dependent findings. The revision model results (Figure 7) show that easy problems perform best with purely sequential revisions (a long chain of refinements), while hard problems benefit from a balanced sequential-to-parallel ratio. For latency-sensitive applications — interactive assistants, real-time decision-making, on-device inference where the user is waiting — a strategy that allocates 64 sequential revisions (even if it achieves the best accuracy-per-FLOP) may be unacceptable regardless of its efficiency advantage. The paper's 4× efficiency gains are in FLOPs, not in latency; the latency cost of sequential allocation could easily exceed the FLOPs savings by orders of magnitude.

Similarly, the beam search and lookahead search algorithms (Section 5.2) involve step-by-step generation where each step depends on the PRM's scoring of the previous step — this is an inherently sequential process whose wall-clock time is proportional to the number of expansion rounds (up to 40 in the paper's configuration). Best-of-N weighted, in contrast, can execute all $N$ generations in parallel and then aggregate scores in a single post-processing step.

What evidence exists in the paper. The paper does not report wall-clock time, latency, or any measure of serial dependency for any method. The FLOPs-matched comparison (Section 7) uses the standard approximations $X = 6ND_{\text{pretrain}}$ and $Y = 2ND_{\text{inference}}$, which count total multiply-add operations independent of parallelism. The sequential-to-parallel ratio sweep (Figure 7) reports only accuracy as a function of the ratio, not latency or throughput. The paper acknowledges that search involves step-by-step PRM scoring (Section 5.2) but does not translate this into latency implications.

Mitigation status. The paper does not discuss this tradeoff at all. This is a significant omission for practitioners because many real-world deployments are latency-constrained. The compute-optimal framework optimizes accuracy subject to a total-FLOPs budget, but a more practically relevant optimization might be accuracy subject to a latency budget or a combined FLOPs + latency constraint. The framework could in principle incorporate a latency model (e.g., penalizing sequential steps or rewarding parallelism), but the paper does not take this step. For now, practitioners should treat the paper's recommendations — favoring sequential revisions on easy problems, beam search on medium problems — as FLOPs-optimal but potentially latency-suboptimal, and should measure the wall-clock implications in their specific deployment context.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper functions primarily as a course correction for the URLVR research agenda. Its central contribution is not a new method but a reframing of what the field should consider the right problem: the question shifts from "how do we design better intrinsic rewards?" to "what reward sources can escape the confidence-correctness ceiling?" The sharpening mechanism analysis (Theorem 1, Appendix A.4) provides the theoretical warrant for this reframing — it shows that intrinsic rewards are not just sometimes unreliable but structurally bounded by the model's initial belief distribution, and that no amount of reward engineering within the intrinsic paradigm can escape this bound.

The magnitude of this shift is diagnostic rather than paradigmatic. The paper does not introduce a new training regime or demonstrate that a previously impossible capability is now achievable. Instead, it provides a principled stopping condition for a line of research that was generating confusing, contradictory results. Prior to this work, the URLVR literature was accumulating both positive reports (TTRL's test-time gains, EM-RL's entropy-minimization improvements) and negative reports (reward hacking in SRT, model collapse in Zhang et al., 2025c) without a framework for resolving the contradiction. The paper's explanatory contribution is to show that these are not contradictions at all — they are the same dynamic (sharpening) observed at different points on the confidence-correctness alignment curve. Intrinsic rewards produce early gains when the model's initial preferences happen to align with correctness, and inevitable collapse when the training distribution contains enough misaligned problems to flip the aggregate reward signal from beneficial to harmful.

The reconciliation of prior contradictions is probably the paper's most immediately useful contribution for the broader field. The finding that small-dataset test-time training avoids collapse (Section 5.1, Figure 6) explains why TTRL (Zuo et al., 2025) and RENT (Prabhudesai et al., 2025) reported success in test-time settings — their problem sets were small enough to stay below the collapse threshold. The finding that larger datasets trigger collapse (Section 5.1, Figure 6) explains why SRT (Shafayat et al., 2025) and Zhang et al. (2025c) observed reward hacking — they were training on datasets large enough to cross the boundary. The finding that model prior determines collapse timing (Section 6, Figure 10) explains why Qwen models appeared more stable than LLaMA models in prior studies. None of these prior results were wrong; they were just measuring different points in a parameter space that the paper now maps systematically.

Concretely, this work makes several research directions more attractive:

• External reward methods grounded in generation-verification asymmetries. The self-verification experiments (Section 7, Figure 13) provide preliminary evidence that rewards derived from computational verification procedures exhibit qualitatively different training dynamics — sustained improvement without the rise-then-fall collapse pattern. This directs attention toward tasks with native verification asymmetries (code execution, proof checking, puzzle evaluation, numerical simulation) and toward techniques for constructing verifiable rewards from unlabeled data (the next-token prediction paradigm of RPT, the dual reconstruction of DuPO).

• Test-time training as a safe deployment of intrinsic methods. The paper's evidence that small, domain-specific datasets prevent collapse (Section 5.1) and that even incorrect majority votes can yield OOD gains on small datasets (Section 5.3) provides a clear operational envelope for intrinsic URLVR: use it at test time on the specific problem distribution you care about, with the understanding that gains come from sharpening existing knowledge rather than acquiring new capability. This is a constrained but real use case, and the paper provides concrete thresholds (≤128 samples, domain-specific) for practitioners.

• Model prior assessment through diagnostic training runs. Model Collapse Step (Section 6) offers a cheap, label-free alternative to full supervised RL training for base model selection. The 5.6× token reduction (Table 3) and the fact that it requires no ground-truth labels make it immediately practical for organizations evaluating multiple model candidates. The finding that collapse timing preserves model rankings under aggressive hyperparameters (Figure 12) means the diagnostic can be run even faster with tuned settings.

Equally important, the paper makes several research directions less attractive:

• Iterating on intrinsic reward formulations. The unified reward framework (Appendix A.3) demonstrates that Self-Certainty, Token-Level Entropy, Trajectory-Level Entropy, Probability, and Majority Voting are all instances of the same parametric family, and that this family has a single convergent dynamic. A new intrinsic reward that is also a member of this family — say, a weighted combination of entropy and probability, or a smoothed version of majority voting — would be predicted to exhibit the same sharpening behavior. The paper's empirical demonstration that five different formulations all collapse, and that no hyperparameter setting prevents collapse, strongly suggests that further intrinsic reward engineering is unlikely to yield breakthroughs unless the new formulation structurally breaks the Reward-Confidence Monotonicity property.

• Large-scale intrinsic URLVR training. The dataset-size scaling experiment (Section 5.1, Figure 6) and the extended training experiment (Section 4.1.1, collapse at ~1,000 steps even with optimized hyperparameters) establish that intrinsic URLVR does not scale with data quantity — larger training sets accelerate collapse rather than deferring it. Organizations planning to invest compute in large-scale unsupervised post-training should be directed toward external reward methods or toward supervised RLVR, not toward scaling up intrinsic URLVR.

• Using entropy as a general-purpose training signal. The failure mode analysis (Section 4.1.2, Figure 3) shows that entropy-based rewards are particularly prone to pathological collapse patterns — repetition collapse for token-level and trajectory-level entropy, length collapse for probability. While entropy minimization has legitimate applications in test-time adaptation (TENT; Wang et al., 2020) and in narrow domains where confidence-correctness alignment is strong, the paper's evidence suggests it should not be used as a general-purpose RL reward for large-scale LLM training.

---

Follow-Up Research This Work Enables

Adaptive difficulty estimation that amortizes into the solving process. The most immediate bottleneck the paper identifies (Section 5.1 of the prior summary; the current paper does not have a difficulty estimation setup, so I'll reframe for this paper) is that diagnosing whether intrinsic URLVR will work requires running actual training. Model Collapse Step addresses this partially by providing a cheap diagnostic, but it still requires running RL training on the target dataset. A natural extension is to predict Model Collapse Step from static model properties: can the collapse step be estimated from the model's initial entropy distribution on the training set, from its calibration error, or from the agreement between pass@1 and majority-vote accuracy? If such a predictor existed and was reliable, practitioners could assess RL trainability without running any training at all, enabling rapid screening of model candidates and dataset combinations. A strong follow-up would collect Model Collapse Step measurements for 50+ model-dataset pairs, train a lightweight predictor (possibly a small classifier or a regression model using simple features like average entropy, top-k probability concentration, and pass@k vs. majority-vote gap), and evaluate whether the predictor transfers to unseen model families and task domains. The paper's evidence that collapse timing is determined by model prior rather than engineering choices (Section 6.1, Figure 10; Appendix B.3) provides the conceptual foundation for believing such a predictor could exist.

Entropy-regularized intrinsic URLVR to test whether sharpening can be counterbalanced. The paper's theoretical analysis (Section 3) shows that the sharpening mechanism is driven by the KL-regularized RL objective with self-referential rewards. A critical control experiment that the paper does not perform is to add an explicit entropy bonus to the RL objective — rewarding the model for maintaining diverse outputs — and test whether this can counterbalance the sharpening pressure and extend the useful training horizon. The GRPO algorithm (used in all experiments) supports entropy regularization natively, so this is a low-implementation-barrier experiment. A strong follow-up would sweep entropy bonus coefficients across multiple intrinsic reward methods (Majority Voting, Token-Level Entropy, Probability) on DAPO-17k, measuring whether there exists a coefficient that prevents collapse (Reward Accuracy not dropping below 50%) while still yielding performance improvements over the base model. The paper's finding that KL regularization does not prevent collapse (Appendix B.3, Figures 18, 28–31) sets an important baseline — entropy regularization is distinct from KL regularization because it directly penalizes the low-entropy distributions that the sharpening mechanism produces, rather than penalizing divergence from a reference policy. If entropy regularization also fails to prevent collapse, that would strengthen the paper's fundamental-limitation claim. If it succeeds, it would define a safe operating regime for intrinsic URLVR at larger scales.

Cross-domain replication of the dataset-size collapse threshold. The paper's finding that training with ≤128 samples avoids collapse while ≥512 consistently triggers it (Section 5.1, Figure 6) is established only on DAPO-17k math problems with Qwen3-1.7B-Base. A critical open question is whether this threshold is universal or domain-specific and model-specific. Math reasoning problems have structured, verifiable answers, and the Qwen family has specific architectural properties that the paper shows contribute to stability (Appendix C.2, Qwen models are substantially more stable than LLaMA models). A strong replication study would measure the collapse threshold across: (1) different domains (code generation with HumanEval-style problems, multiple-choice QA, summarization with structured evaluation), (2) different model families (LLaMA, OLMo, DeepSeek, Gemma), and (3) different model sizes within each family. The prediction from the paper's theory is that the threshold depends on the proportion of training problems where the model's initial confidence-correctness alignment is wrong — so harder benchmarks should have lower collapse thresholds, and weaker models should have lower collapse thresholds. Quantifying this relationship would provide a practical rule of thumb for practitioners: "for model family X at scale Y on task Z, intrinsic URLVR is safe up to approximately N training samples."

Combined intrinsic-external reward training with an explicit boundary. The paper's taxonomy (Section 2) draws a sharp line between intrinsic and external rewards, but a natural extension is to deploy both in a single training loop with a explicit routing mechanism. For example, during training on a mixed dataset, problems where the model's majority vote is correct (high confidence-correctness alignment) could use intrinsic majority-voting rewards to sharpen existing knowledge, while problems where the majority vote is wrong (low alignment) could use external verification rewards (self-verification, execution, or ground-truth when available) to provide corrective signal. The paper's per-problem analysis (Section 4.2.1, Figure 4) shows that intrinsic rewards succeed precisely when the highest-reward sample is mostly correct, and fail otherwise. This suggests that a simple heuristic — checking whether the majority-vote pseudo-label matches an external verifier's judgment, and routing accordingly — could combine the efficiency of intrinsic rewards (no external verification cost on easy problems) with the scalability of external rewards (correction on hard problems). A strong follow-up would implement this on the Countdown task (where external verification is cheap), comparing pure intrinsic, pure self-verification, and the hybrid routing approach across varying dataset sizes and difficulty distributions. The key measurement would be whether the hybrid avoids both the collapse of intrinsic methods (on hard problems) and the increased computational cost of external verification (on easy problems where it's unnecessary).

Stress-testing self-verification at scale to identify failure modes. The self-verification experiments (Section 7, Figures 13–14) are preliminary — 4,000 training problems, 600 steps, one model family, one task. Before the field commits to external URLVR as the scalable alternative, the failure mode of external rewards needs to be characterized as carefully as the paper characterizes the failure modes of intrinsic rewards. Concrete stress tests include: (1) training horizon scaling: run self-verification on Countdown for 5,000+ steps to determine whether the Reward Accuracy recovery (Figure 13, bottom) is permanent or whether a delayed form of verifier exploitation eventually appears; (2) verifier-model co-evolution: as the generator policy improves and produces more sophisticated solutions, does the verifier's accuracy degrade? This would manifest as a gradual decline in Reward Accuracy not captured in 600-step experiments; (3) adversarial testing: intentionally introduce problems where the correct answer is ambiguous or where a subtle error (e.g., operator precedence, floating-point precision) could fool a naive verifier, and test whether self-verification amplifies these errors; (4) task complexity scaling: move from Countdown (a constrained puzzle with trivial verification) to tasks where verification is harder — math word problems requiring multi-step reasoning verification, code generation requiring test-case construction, or open-ended reasoning where verification requires human-like judgment. The paper's theoretical claim — that external rewards escape the confidence-correctness ceiling — predicts that self-verification should continue to improve at scale, but this prediction needs rigorous testing before it is operationalized.

Model Collapse Step as a meta-learning objective for pre-RL model selection. The paper shows that Model Collapse Step predicts supervised RL gains (Figure 11) and can be computed 5.6× faster than running full training (Table 3). A practical extension is to use Model Collapse Step as a selection criterion in a model development pipeline: train multiple model variants (different sizes, training stages, architectures, or fine-tuning recipes), compute Model Collapse Step for each, and select the variant with the latest collapse step for expensive supervised RL training. This is already implied by the paper but not validated — the critical experiment would be to compare the downstream performance of models selected by Model Collapse Step against models selected by pass@k, validation loss, or random selection, across multiple rounds of model development (e.g., selecting the best checkpoint to continue training, or selecting the best fine-tuning recipe from a set of candidates). If Model Collapse Step consistently selects models that yield better supervised RL outcomes than alternative selection criteria, it would become a standard tool in the RLHF/RLVR workflow, analogous to how validation perplexity is used to select pretraining checkpoints. The paper's evidence that Model Collapse Step requires no ground-truth labels is particularly relevant here, because in a real development pipeline the candidate models may be evaluated on problems where labels are expensive or unavailable.

---

Practical Applications and Downstream Use Cases

Cheap model prior assessment before committing to expensive RL training. The most immediately actionable application of this work is using Model Collapse Step (Section 6) as a diagnostic filter in RLVR pipelines. Organizations training reasoning models with supervised RLVR typically evaluate multiple base model candidates (different pretraining checkpoints, different fine-tuning recipes, different model families) by running full RL training on each and comparing the gains — an expensive process that the paper quantifies at 6.66B tokens for 7 models (Table 3). Model Collapse Step reduces this to 1.19B tokens (a 5.6× reduction) and requires no ground-truth labels, meaning the diagnostic can be run on problems where labels are unavailable. The workflow would be: (1) for each candidate model, run intrinsic URLVR training with majority voting reward and aggressive hyperparameters (MBS=1, N=8) on the target training dataset, (2) record the training step where Reward Accuracy drops below 1%, (3) rank candidates by Collapse Step, and (4) run full supervised RL training only on the top-ranked candidates. The paper's evidence that Collapse Step correlates with GT Gain across 7 models from 3 families (Figure 11) and that aggressive hyperparameters preserve rankings (Figure 12) provides the empirical foundation for this workflow.

Safe deployment of test-time training on small, domain-specific problem sets. The paper's finding that training on ≤128 samples avoids collapse (Section 5.1, Figure 6) provides a clear operational envelope for intrinsic URLVR in production: when a model is deployed to handle a specific class of problems (e.g., a customer support system handling a particular product's FAQ, a tutoring system focused on a specific math curriculum, or a code review system for a particular codebase), training on the target problem distribution with majority voting reward can improve performance without risking model collapse. The test-time training experiments (Section 5.2, Figure 8) demonstrate this concretely: training on 40 AMC problems improves AMC23 accuracy from ~18% to ~38% avg@32 while also improving OOD performance on AIME24 (from ~2% to ~6%), without any collapse. The practical recipe is: collect the target problem set (≤128 problems), run intrinsic URLVR with majority voting reward, monitor that the training set is small enough to avoid collapse (described as KL divergence from reference staying low, Figure 7), and deploy the adapted model. The paper's evidence that even training on problems where all initial majority votes are incorrect can yield OOD gains (Section 5.3, Figure 9) provides a safety margin: the method degrades gracefully even when applied to problem sets where the model initially gets most answers wrong.

Prioritization of verifier development over reward function engineering for URLVR scaling. For research teams and organizations investing in unsupervised post-training, the paper's central finding — that the sharpening mechanism is structural, not an artifact of specific reward designs — has a clear resource-allocation implication: invest in building better verifiers, not better intrinsic rewards. The failure mode analysis (Section 4.1.2, Figure 3) shows that different intrinsic reward formulations produce different collapse patterns but all eventually collapse. The self-verification experiments (Section 7, Figure 13) show that external verification can produce qualitatively different dynamics. The practical implication is that effort spent designing a new intrinsic reward (e.g., a more sophisticated entropy metric, a weighted majority voting scheme, or an ensemble of certainty measures) is unlikely to pay off at scale, because any reward derived from model-internal signals will be subject to the sharpening dynamic. Effort should instead go into: (1) identifying domains with native generation-verification asymmetries and building verifiable environments for them (structured math, executable code, proof assistants, puzzle engines, physics simulators), (2) improving self-verification capability through instruction tuning and prompt engineering (as suggested by Figure 14, where instruction-tuned models show dramatically better self-verification robustness), and (3) developing techniques for constructing verifiable rewards from unlabeled data (the next-token prediction paradigm of RPT, the dual reconstruction of DuPO). The paper does not prove that external methods will scale indefinitely, but it provides a principled argument that they are not subject to the same structural ceiling as intrinsic methods.

---

When to Prefer This Method

The paper does not propose a single method to be preferred over alternatives — it is an analysis paper that characterizes the boundary conditions for when intrinsic URLVR works and when it fails, and provides a diagnostic (Model Collapse Step) rather than a new training algorithm. The "method" in question is the framework itself: the claim that intrinsic URLVR is safe and beneficial within a specific envelope, and that practitioners should assess whether their use case falls within that envelope before deploying intrinsic rewards.

Prefer intrinsic URLVR (majority voting reward) when:

• The training dataset is small (≤128 problems) and domain-specific, matching the test-time training setting validated in Sections 5.1–5.3. In this regime, the model avoids collapse (Figure 6, DAPO-32 and DAPO-128 maintain stable Ground Truth Reward and high Reward Accuracy) and can achieve performance improvements even when initial majority votes are incorrect (Section 5.3, Figure 9).
• Ground-truth labels are unavailable or prohibitively expensive, making supervised RLVR infeasible. Intrinsic URLVR requires no labels — only the model's own outputs.
• The base model has reasonable confidence-correctness alignment on the target problem distribution, as assessed by Model Collapse Step (Section 6). A late collapse step (e.g., >200 steps under default hyperparameters for Qwen models) indicates the model's prior is strong enough that sharpening will be mostly beneficial. A very early collapse step (e.g., <50 steps for LLaMA base models, Figure 10) indicates the model's prior is too weak for intrinsic URLVR to help, and either supervised RLVR or external URLVR should be used instead.

Prefer external URLVR (self-verification or execution-based rewards) when:

• The task has a native generation-verification asymmetry — verification is computationally cheap and deterministic (code execution, arithmetic evaluation, puzzle checking, proof verification), making external rewards scalable and objective. The self-verification experiments (Section 7, Figure 13) show sustained improvement without collapse on Countdown, where verification is trivial arithmetic.
• The training dataset is large (≥512 problems), where intrinsic URLVR is predicted to collapse (Section 5.1, Figure 6). External rewards are not subject to the sharpening feedback loop that causes collapse at scale.
• The base model has sufficient instruction-following capability to perform reliable self-verification. The paper shows (Figure 14) that instruction-tuned models (Qwen3-1.7B) maintain high Reward Accuracy (>0.7) with both verification prompts, while base models are highly prompt-sensitive and only work with carefully structured prompts. For base models, self-verification may require prompt engineering or may not work at all.

Prefer supervised RLVR (ground-truth rewards) when:

• Ground-truth labels are available and affordable. The paper consistently shows that supervised RLVR (Oracle Supervision in Figure 2, Oracle Supervision in Figure 13) achieves the highest performance and never exhibits collapse. Intrinsic and external URLVR are alternatives for when supervised RLVR is infeasible, not replacements for it when labels are available.
• The model's prior is very weak (Model Collapse Step <50 steps), indicating that sharpening will amplify errors rather than correct solutions. In this regime, even external verification may struggle because the model cannot produce verifiably correct solutions at sufficient frequency to create a learning signal.

Use Model Collapse Step as a diagnostic before any RL training when:

• Evaluating multiple base model candidates for RL training. The paper demonstrates (Section 6.2, Figure 11) that Collapse Step ranking predicts supervised RL gains across 7 models from 3 families, and the computation cost is 5.6× lower than running full supervised RL training (Table 3).
• Assessing whether a specific model-dataset combination is suitable for intrinsic URLVR. A Collapse Step measured on the target dataset with aggressive hyperparameters (MBS=1, N=8, Section 6.3) provides a rapid assessment: if collapse occurs very early (<20 steps under aggressive settings), intrinsic URLVR should not be used on that combination; if collapse occurs late (>100 steps), intrinsic URLVR is likely safe and beneficial.