The Invisible Leash: Why RLVR May or May Not Escape Its Origin

🎯 Pitch

RLVR does not expand a model's reasoning frontier—it shrinks it. By disproportionately killing off rare but correct solutions that the base model could occasionally find, RLVR systematically lowers the ceiling on what’s achievable even with unlimited retries, while the base model’s broader search often outperforms the RLVR-tuned model at scale.

1. Executive Summary

This paper empirically investigates whether Reinforcement Learning with Verifiable Rewards (RLVR) expands a model’s reasoning capabilities or primarily sharpens probability mass around solutions the base model already knows. Using DeepSeek-R1-Distill-Qwen base models (1.5B–14B parameters) evaluated across math and non-math benchmarks (MATH500, AIME, OlympiadBench, SimpleQA, LiveBench, SciBench, Reasoning Gym), the authors introduce the concept of empirical support to quantify which correct completions are accessible under finite sampling, categorizing post-RLVR changes into support preservation, shrinkage (correct solutions lost), and expansion (new solutions gained) — for example, ProRL-1.5B-v2 preserves 2,388 correct completions while gaining only 48 and losing 175, yielding a shrinkage-to-expansion ratio of ~3.6:1. Across all models and domains, support retention rates exceed 0.93 while net discovery rates remain below 0.04, establishing that RLVR primarily acts as a support-constrained optimizer that improves precision by concentrating mass on known high-reward trajectories but systematically narrows solution diversity, with the base model’s broader support often outperforming the RLVR model at high sampling budgets (e.g., AIME 2024 base pass@8192 = 93.3% vs. ProRL-1.5B’s 83.3%) — the “invisible leash” holding that RLVR cannot discover solutions the base model’s initial distribution assigns negligible probability to, even while it sometimes increases token-level entropy only to converge onto fewer distinct final answers.

2. Context and Motivation

The Core Problem: Does RLVR Actually Expand Reasoning Capabilities?

The fundamental question driving this paper is one that has split the research community: when we apply Reinforcement Learning with Verifiable Rewards (RLVR) to a language model, are we teaching it genuinely new reasoning patterns, or are we simply making it better at retrieving solutions it already "knows" but couldn't reliably produce before?

This distinction matters enormously for how we think about scaling AI systems. If RLVR truly expands a model's reasoning boundary, then RLVR alone — given enough compute and data — might eventually push even modestly-sized base models to superhuman reasoning performance, with each generation of training building on the last. If, instead, RLVR merely sharpens existing capabilities, then improvements from RLVR will eventually plateau, bounded by what the base model's initial pretraining already encoded, and genuine advances in reasoning will require either architectural innovations or better pretraining.

The paper opens with a striking observation that makes this question immediately concrete (Section 1):

"While models trained with RLVR consistently outperform base models on a single attempt, base models often perform better with multiple attempts."

This pattern is paradoxical under the assumption that RLVR makes the model "smarter." If RLVR genuinely improves reasoning, the trained model should dominate the base model regardless of how many attempts each gets. The fact that the base model's pass@k catches up to or exceeds the RLVR model's pass@k at larger k suggests something subtler: RLVR may be making the model more precise (it gets the right answer more often on a single try) but less diverse (it can no longer access certain correct solutions that the base model could occasionally produce if given enough chances).

Why This Gap Matters

The practical stakes are high because RLVR has become the dominant post-training recipe for state-of-the-art reasoning models. DeepSeek-R1 (Guo et al., 2025), OpenAI-o1 (Jaech et al., 2024), and numerous open-source efforts all rely on RLVR to boost reasoning performance on mathematics, programming, and scientific problem-solving. If RLVR's gains are primarily from concentrating probability mass rather than expanding capabilities, it means:

Deployment asymmetry. The RLVR-trained model is better when you can afford exactly one generation (pass@1), but the base model is potentially better when you can afford many (pass@k). For latency-critical applications (real-time chatbots, on-device inference), the RLVR model's improved pass@1 is genuinely valuable. But for batch inference or self-improvement pipelines where massive sampling is feasible, sticking with the base model and using compute-optimal test-time strategies (as analyzed in prior work) might actually yield higher accuracy.

Self-improvement ceilings. If RLVR only sharpens the base model's existing distribution, an iterative loop of "train with RLVR, use the improved model to generate data, train again" will hit a ceiling determined by the original base model's support. The model can never create reasoning patterns it didn't already have in its long tail — it can only amplify them. This is precisely the "invisible leash" metaphor of the title: the model remains tethered to its initialization, no matter how many rounds of RLVR it undergoes.

Benchmark misinterpretation. The paper points out that the field's standard evaluation metrics — especially pass@1 — may systematically obscure the limitations of RLVR. A model that achieves 80% pass@1 but has collapsed onto a single solution strategy may look "better" than a model with 70% pass@1 but access to multiple diverse solution paths, even though the latter might be more robust to distribution shift or more useful for downstream applications requiring diverse outputs. The paper cites Wen et al. (2025), who argue that "standard evaluation metrics, such as Pass@K, can overstate RLVR gains by ignoring the correctness of intermediate reasoning."

Conflicting Narratives in Prior Work

The paper enters a landscape where researchers have reached diametrically opposed conclusions about what RLVR does. Understanding why this confusion exists is essential for appreciating the paper's contribution.

The conservative-optimization view. Several recent studies have interpreted the pass@k crossover pattern — where the base model wins at high k — as evidence that RLVR performs conservative optimization within the base model's existing capabilities. Yue et al. (2025a) frame RLVR as primarily redistributing probability mass among solutions the base model already knows, not generating novel reasoning. Zhao et al. (2025) describe RLVR as an "echo chamber" that amplifies behaviors learned during pretraining. Shah et al. (2025) argue that reflection capabilities in post-trained models are already present in pretrained models. Ma et al. (2025) show that RLVR struggles specifically on the hardest questions where the base model's initial support is weakest, and He et al. (2025) demonstrate that even noisy or spurious reward signals can produce apparent gains under RLVR — further suggesting that the mechanism is sharpening rather than learning.

The capability-expansion view. On the other side, Liu et al. (2025a) argue that RLVR can substantially expand reasoning when applied to domains where the base model wasn't already saturated. Their ProRL work specifically claims to "expand reasoning boundaries" through prolonged RL training, demonstrating pass@k improvements even at large k on challenging tasks the base model initially struggled with. The paper acknowledges this debate directly:

"Seeking a definitive answer to this debate remains an open challenge."

Why prior analyses fall short. The paper identifies specific methodological gaps that prevent prior work from resolving this debate convincingly (Section 1 and related work):

Snapshot vs. trajectory analysis. Most prior work examines RLVR through "before/after" snapshots — comparing base and trained models at fixed checkpoints — without tracking how the model's solution support evolves throughout training. This leaves open the question of whether RLVR might temporarily expand support (discovering new solutions) before later collapsing onto a narrower set, or whether the narrowing is monotonic from the start. The paper addresses this with a temporal analysis in Section 4.
No unified framework for solution accessibility. Prior work lacks a formal definition of what it means for a solution to be "accessible" to a model. Without this, claims about "expansion" vs. "sharpening" remain vague: does a solution that the base model assigns probability 10⁻⁶ to count as "already known"? What about 10⁻²⁰? The paper introduces the concept of empirical support with a principled statistical threshold derived from sampling confidence bounds (Section 3, Appendix C.4), providing a rigorous foundation for these claims.
Single-domain focus. Many prior studies examine RLVR on a single benchmark or domain, making it impossible to separate domain-specific effects from general properties of RLVR. The paper evaluates across mathematics, logical reasoning, factual QA, code generation, and even visual reasoning, demonstrating that the support-preservation pattern holds broadly — this cross-domain consistency is key evidence that the "invisible leash" is a fundamental property of RLVR rather than an artifact of, say, math-specific reward structures.
Missing theoretical grounding. The paper identifies a gap in theoretical understanding: why should RLVR be support-bounded? Prior empirical work documented the phenomenon but didn't explain it. The paper provides formal proofs (Theorem C.1, Theorem C.3, Proposition C.4) showing that on-policy RLVR updates cannot assign non-zero probability to completions that the base model assigns zero probability to, and that under finite sampling, even completions with negligible-but-nonzero base model probability remain effectively unreachable. This theoretical scaffolding transforms the empirical observation into a principled limitation.

How This Paper Positions Itself

The paper frames itself not as advocating for one side of the debate or the other, but rather as providing the analytical machinery needed to resolve it. Its methodological contributions are:

1. The empirical support framework. The paper defines a principled way to categorize what happens to correct solutions under RLVR (Section 3). Rather than vaguely asking whether RLVR "improves reasoning," it quantifies support preservation (solutions both models can find), support shrinkage (solutions the base model could find but the RLVR model cannot), and support expansion (solutions the RLVR model can find that the base model could not). This framework is domain-agnostic, model-agnostic, and computationally well-defined via the statistical threshold ϵ ≈ 3.66 × 10⁻⁴ for MATH tasks at 95% confidence with 8,192 samples (Appendix C.4).

2. A unified theoretical account of why RLVR is support-bounded. Rather than just documenting the phenomenon, the paper proves two complementary results: (a) In the infinite-sample limit, on-policy RLVR updates preserve the base model's support exactly — the trained model can never assign positive probability to any output the base model assigns zero probability to (Theorem C.1); (b) Under finite sampling, even solutions that are technically in the base model's support but have probability below the detectability threshold are effectively invisible to RLVR, and gradient updates cannot lift them above that threshold without deliberate exploration mechanisms (Theorem C.3). The variational interpretation (Proposition C.4) further shows that RLVR solves a KL-constrained projection problem — it finds the distribution closest to the base model that satisfies reward constraints — which naturally explains why it preserves the base model's structure while concentrating mass.

3. The precision-diversity trade-off formalized at the entropy level. The paper extends the support analysis to entropy, showing that RLVR systematically reduces answer-level entropy (the diversity of final outputs) while sometimes increasing token-level entropy (stochasticity during generation). This decoupling — local uncertainty increasing while global diversity decreases — is a novel empirical finding that challenges the common assumption that higher token-level entropy indicates exploration. The paper calls this "local stochasticity without global exploration" (Section 5.2).

4. Cross-domain, cross-scale empirical validation. The paper's evidence isn't from a single model or benchmark. It evaluates eight base-model/RLVR pairs spanning 1.5B to 14B parameters (ProRL, Nemotron, Skywork, AceReason, Phi4, DAPO), across math (MATH500, AIME, AMC, Minerva, OlympiadBench) and non-math (SimpleQA, LiveBench, SciBench, LiveCodeBench, Reasoning Gym) domains, and includes a vision-language model (Kangheng-OVR-7B) for visual math reasoning. This breadth makes the support-preservation conclusion substantially more robust than any single-experiment finding.

5. A comparison with supervised fine-tuning. To test whether support shrinkage is specific to RLVR or an artifact of any post-training, the paper runs a controlled experiment: same base model (Qwen2.5-Math-7B), same training data (DeepMath-103K), same hyperparameters, varying only the objective (SFT vs. DAPO). SFT produces moderate support expansion with positive Net Support Change Rate; DAPO produces sharp concentration with negative NSCR. This is direct evidence that support-constrained behavior "emerges from the objective itself" (Section 4.2) rather than from data, scale, or implementation details.

The Analogy That Organizes the Paper

The paper's title — "The Invisible Leash" — is more than a catchy metaphor. It captures the paper's central theoretical and empirical claim: RLVR-trained models are tethered to their base model's initial distribution in a way that is invisible when looking only at pass@1 improvements but becomes apparent under careful support analysis. The "leash" is invisible because:

The RLVR model's pass@1 improves, creating the appearance of new capability.
Token-level entropy sometimes increases, creating the appearance of exploration.
Rare expansions (composing subskills the base model already has, correcting format misalignment) create the appearance of genuine discovery.

But the leash is real: across all experiments, support shrinkage consistently outweighs expansion, net discovery rates never exceed 4%, and the base model's broader support yields higher pass@k at large k. The paper's goal is to make this leash visible and measured, enabling the community to design training methods that intentionally break it — by seeding probability mass into underrepresented solution regions — rather than being surprised by its constraints.

3. Technical Approach

3.1 Reader Orientation

This is an empirical measurement and analysis paper — it does not propose a new training algorithm but rather introduces a systematic framework for measuring what existing RLVR algorithms do to a model's solution space, treating training as a black-box transformation and analyzing its effects on the set of correct completions the model can access. The core idea is that RLVR's apparent "improvements" in reasoning can be decomposed into three measurable components — support preservation (solutions both base and trained models can reach), support shrinkage (solutions only the base model can reach), and support expansion (solutions only the trained model can reach) — and that by quantifying these components across models, domains, and training trajectories, we can determine whether RLVR genuinely expands reasoning boundaries or merely sharpens probability mass within the base model's existing capabilities.

The system being studied is the RLVR training pipeline (base model → RL training with verifiable rewards → trained model), and the system being built by this paper is the empirical support measurement apparatus that captures, quantifies, and explains the changes in solution accessibility induced by this pipeline.

3.2 Big-Picture Architecture (Diagram in Words)

The measurement framework has four major components:

Base Model and RLVR Model Pair — a pretrained language model $q$ and its RLVR-fine-tuned counterpart $\pi_\theta$ , both producing probability distributions over completions given a prompt. This is the object of study, not something the paper builds; the paper samples from these models at scale to construct their empirical support sets.
Reward Function $R(x, y) \in \{0, 1\}$ — a verifiable, binary correctness signal for each completion-prompt pair. This is task-specific (e.g., MATH500 answer matching, LiveBench grading scripts, SimpleQA GPT-4.1 judging) and is used to partition all completions into correct ( $\mathcal{C}$ ) and incorrect sets. The reward function defines what "support" means by identifying which completions are correct.
Empirical Support Extractor — the core measurement apparatus. For each prompt, this component draws $k$ completions from the model, filters to correct ones (using $R$ ), and records the set of distinct correct completions observed. The choice of $k$ is critical: it must be large enough that any correct completion the model assigns probability below the detectability threshold $\epsilon$ is effectively invisible. The paper derives $\epsilon \approx 3.66 \times 10^{-4}$ for $k=8192$ at 95% confidence (Appendix C.4), meaning that with 8,192 samples, any correct completion with probability above this threshold will almost certainly be observed, while those below it will almost certainly be missed — defining the operational boundary of "accessible."
Support Dynamics Classifier and Metric Engine — takes the sets of correct completions from the base model and the RLVR model for each prompt, classifies each correct completion into one of four categories (Preservation: both models find it; Expansion: only RLVR finds it; Shrinkage: only base finds it; Out-of-Support: neither finds it), and aggregates these counts into per-domain, per-model metrics (SRR, NDR, SDS, NSCR). This is the analytical layer that transforms raw completion data into quantitative claims about RLVR's behavior.

Information flows as follows: a prompt enters the system → the base model is sampled $k$ times and correct completions are extracted → the RLVR model is sampled $k$ times and correct completions are extracted → the two correct-completion sets are compared to classify each completion into P/E/S/O → counts are aggregated across all prompts in a benchmark → metrics are computed to characterize support dynamics → the process is repeated across models, domains, and training checkpoints to build the full empirical picture.

3.3 Roadmap for the Deep Dive

First, the empirical support formalization — what counts as "accessible" to a model, why a principled threshold $\epsilon$ is necessary, and how it's derived from sampling confidence bounds (Section 3.1 of the paper). This is the foundation because all subsequent categorization depends on a clean definition of accessibility.
Second, the four-category support dynamics taxonomy (Preservation, Expansion, Shrinkage, Out-of-Support) and the derived metrics (SRR, NDR, SDS, NSCR) that quantify RLVR's behavior (Section 3.2 of the paper). This is the analytical lens through which the paper interprets its experiments.
Third, the experimental measurement protocol — how prompts are selected, how many samples are drawn, how correct completions are identified, and how the classification is operationalized across math and non-math domains (drawing from Section 4.1 and Appendix B). This makes the abstract framework concrete.
Fourth, the theoretical scaffolding that explains why RLVR is support-bounded — the support preservation theorems (C.1, C.3), the variational interpretation as KL projection (C.4), and the entropy-reward trade-off (C.6). This provides the mechanistic understanding behind the empirical patterns.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical measurement paper whose core idea is that RLVR's effects on solution accessibility can be precisely quantified using a statistical support framework, and that this quantification reveals RLVR to be fundamentally support-bounded — it concentrates probability mass within the base model's existing support rather than expanding that support to include genuinely novel solutions.

The Empirical Support Framework

Why "support" needs a rigorous definition. The paper's central analytical construct is the empirical support of a model — the set of correct completions that the model can realistically produce under finite sampling. This is not the same as the model's mathematical support (the set of outputs with strictly positive probability), because modern language models use softmax outputs that assign non-zero probability to every possible token sequence, making the mathematical support of any model the entire output space $\mathcal{Y}$ . The paper explicitly addresses this:

"The effective support assumes $q = 0$ , which rarely holds. Softmax layers yield strictly positive probabilities across all tokens, making the nominal support of $q$ span the entire space $\mathcal{Y}$ . This factor, along with sampling noise or temperature scaling, contributes to what we refer to as empirical support diffusion: over time, the model may assign growing probability mass to completions that initially had negligible but still nonzero probability under the base model."

The practical consequence is stark: without an empirical threshold, every possible completion is technically "in the support" of every model, making support-based analysis meaningless. The paper must define a principled cutoff that separates completions the model can realistically produce from those that exist only as mathematical artifacts of the softmax.

Definition of empirical support. The paper defines, for a prompt $x$ , a model distribution $p(y \mid x)$ , and a correctness set $\mathcal{C} = \{y \in \mathcal{Y} \mid R(x, y) = 1\}$ :

$\text{supp}_\epsilon(p) := \{y \in \mathcal{C} \mid p(y \mid x) > \epsilon\}$

where $\epsilon > 0$ is a minimal cutoff that separates completions with practically observable likelihood from those that are statistically negligible.

What this definition means operationally: for a given prompt, collect all completions that are both correct (by the reward function) and assigned probability above threshold $\epsilon$ by the model. These are the correct completions the model can "realistically discover" — if you sampled from the model, you would eventually see them. Completions below $\epsilon$ are "effectively invisible" because they are so unlikely that standard on-policy RL with finite rollouts will never encounter them, meaning no gradient signal will ever reinforce or suppress them.

Deriving $\epsilon$ from sampling confidence bounds. The paper provides a statistical justification for $\epsilon$ in Appendix C.4, rather than choosing it arbitrarily. The logic: if you sample $k$ completions from a model and do not observe a particular correct completion $y^*$ , what is an upper bound on its true probability $p = \pi(y^* \mid x)$ ?

The probability of missing $y^*$ in all $k$ independent trials is $(1 - p)^k$ . To ensure this "miss" event occurs with probability at most $\zeta$ (i.e., with confidence $1 - \zeta$ , we would have observed $y^*$ if it were truly probable), we solve:

$(1 - p)^k \leq \zeta$

Taking logs and using the inequality $\log(1 - p) \leq -p$ for $p \in (0, 1)$ :

$k \cdot (-p) \geq \log \zeta \quad \Rightarrow \quad p \leq -\frac{\log \zeta}{k}$

Consequently, if $y^*$ is not observed in $k$ samples, then with confidence $1 - \zeta$ , its true probability satisfies $p \leq -\log \zeta / k$ . The paper uses this to set $\epsilon$ :

"If $k = 8192$ in the math reasoning tasks and we desire 95% confidence (i.e., $\zeta = 0.05$ ), then $\pi(y^* \mid x) \leq -\log(0.05) / 8192 \approx 2.996 / 8192 \approx 3.66 \times 10^{-4}$ ."

What this means in practice: for MATH tasks with $k = 8192$ samples, any correct completion that the base model assigns probability above $3.66 \times 10^{-4}$ will be observed at least once with 95% confidence. If a completion is not observed, it is either genuinely below this threshold (truly negligible) or we have encountered a rare sampling failure. The threshold thus defines the operational boundary: the empirical support is the set of correct completions with probability above the detectability floor determined by the sampling budget.

Why this form matters: without a principled $\epsilon$ , the paper's central claims about shrinkage vs. expansion would be definitionally ambiguous. A researcher arguing that RLVR "expands" capabilities could claim that any completion the RLVR model assigns probability $10^{-20}$ to counts as "accessible," while a skeptic could set an arbitrarily high threshold. The statistical derivation ties $\epsilon$ to observable quantities (sample count, confidence level), making the definition reproducible and dispute-resolution-friendly. Different sampling budgets yield different $\epsilon$ values, which is consistent with the paper's approach of using large $k$ (4,096–16,384 for math) to push $\epsilon$ low enough that genuinely accessible completions are captured while statistical noise is bounded.

The four support categories. Given the empirical support sets of the base model $q$ and the RLVR model $\pi_\theta$ for a prompt $x$ , the paper classifies each correct completion $y^* \in \mathcal{C}$ into exactly one of four categories (Definition 3.2, illustrated in Figure 1):

Support Preservation ( $P$ ): $q(y^* \mid x) > \epsilon$ and $\pi_\theta(y^* \mid x) > \epsilon$ . The correct completion is accessible to both the base model and the RLVR model. This is the "retained" category.
Support Expansion ( $E$ ): $q(y^* \mid x) \leq \epsilon$ but $\pi_\theta(y^* \mid x) > \epsilon$ . The base model assigned negligible probability to this correct completion (below detectability), but the RLVR model now assigns non-negligible probability (above detectability). This is the "genuine discovery" category — the RLVR model can reach a correct solution the base model couldn't.
Support Shrinkage ( $S$ ): $q(y^* \mid x) > \epsilon$ but $\pi_\theta(y^* \mid x) \leq \epsilon$ . The base model could access this correct completion, but the RLVR model has reduced its probability below the detectability threshold. This is the "lost solution" category — RLVR has concentrated mass elsewhere, abandoning a previously accessible correct path.
Out of Support ( $O$ ): $q(y^* \mid x) \leq \epsilon$ and $\pi_\theta(y^* \mid x) \leq \epsilon$ . Neither model can access this correct completion. It may technically have nonzero probability (softmax guarantees this) but is empirically invisible to both.

Operationalizing the classification: for each prompt, the paper draws $k$ samples from the base model and $k$ from the RLVR model, collects all distinct correct completions from each, and compares the sets. If a correct completion appears in both sets, it's Preservation. If it appears only in the RLVR set, it's Expansion. If it appears only in the base set, it's Shrinkage. If it appears in neither but is known to be correct (from the task's ground truth), it's Out of Support. The paper explicitly handles the case where multiple reasoning traces produce the same final answer (e.g., different solution methods arriving at the same numerical result):

"We define empirical support over answer-level correct completions, where multiple reasoning traces that yield the same verified outcome are treated as a single equivalence class."

This means that two distinct reasoning traces both producing "42" are counted as a single completion in the support set, not as two separate entries. This prevents overcounting of semantically identical outputs and focuses the analysis on answer-level diversity rather than surface-form variation.

The Support Dynamics Metrics

The raw counts $P$ , $E$ , $S$ , $O$ are informative but difficult to compare across benchmarks with different numbers of prompts and different baseline pass rates. The paper defines four derived metrics (Definition 3.3) that normalize these counts into interpretable scalar values.

Support Retention Rate (SRR):

$\text{SRR} = \frac{P}{P + S}$

where $P$ is the number of preserved completions and $S$ is the number of shrunken completions.

What it computes: the fraction of the base model's accessible correct completions that the RLVR model also retains access to. SRR answers the question: "Of the solutions the base model could find, what proportion can the RLVR model still find?"

Why this form: the denominator $P + S$ is exactly the number of correct completions accessible to the base model (those in the base model's empirical support). The numerator $P$ is the subset of those that survive RLVR training. SRR ranges from 0 (complete collapse — every base-accessible solution is lost) to 1 (perfect retention — no base-accessible solution is lost). A high SRR means RLVR is conservative with respect to the base model's solution set.

Net Discovery Rate (NDR):

$\text{NDR} = \frac{E}{P + E}$

where $E$ is the number of expanded completions and $P$ is (as before) the number of preserved completions.

What it computes: the fraction of the RLVR model's accessible correct completions that represent genuine new discoveries (completions the base model couldn't access). NDR answers: "Of the solutions the RLVR model can find, what proportion are genuinely novel?"

Why this form: the denominator $P + E$ is the number of correct completions in the RLVR model's empirical support (preserved + expanded). The numerator $E$ is the subset of those that the base model couldn't reach. NDR ranges from 0 (no novelty — all RLVR-accessible solutions were already accessible to the base model) to 1 (complete novelty — all RLVR-accessible solutions are new discoveries, though this would require $P = 0$ , i.e., zero preservation, which would be pathological). A near-zero NDR is the paper's key evidence for the "invisible leash": RLVR rarely discovers solutions that weren't already in the base model's reach.

Support Dynamic Score (SDS):

$\text{SDS} = \frac{2 \cdot \text{SRR} \cdot \text{NDR}}{\text{SRR} + \text{NDR}} = \frac{2PE}{P^2 + 2PE + ES}$

What it computes: the harmonic mean of SRR and NDR, providing a balanced measure that penalizes models that achieve high retention but no discovery (SRR near 1, NDR near 0 → SDS near 0) and models that achieve high discovery but poor retention (SRR low, NDR high → SDS near 0). SDS answers: "How well does RLVR balance preserving existing solutions and discovering new ones?"

Why this form: the harmonic mean $2ab/(a+b)$ has the property that it is low whenever either component is low, regardless of how high the other is. This is exactly what the paper wants: a single number that captures whether RLVR achieves both goals simultaneously. An arithmetic mean $(a+b)/2$ would allow high SRR to "compensate" for zero NDR, producing middling scores that mask the imbalance. The harmonic mean produces SDS near zero whenever either SRR or NDR is near zero, correctly flagging the support-constrained optimization regime (high SRR, near-zero NDR → SDS ≈ 0.01–0.07 in the paper's results).

Net Support Change Rate (NSCR):

$\text{NSCR} = \frac{E - S}{P + E + S}$

What it computes: the net change in the size of the empirical support, normalized by total accessible completions. Positive NSCR means RLVR expanded the support (gained more correct completions than it lost); negative NSCR means RLVR shrank the support (lost more than it gained); zero means balanced change. NSCR answers: "Does the RLVR model's accessible solution set grow or shrink relative to the base model's?"

Why this form: the denominator $P + E + S$ is the total number of correct completions accessible to either model (the union of their supports). The numerator $E - S$ is the net change. NSCR ∈ [-1, 1], with negative values indicating net shrinkage — the paper's consistent finding across models and domains (overall NSCR ≈ -0.01 to -0.06).

How these metrics complement each other. The paper uses all four together because each reveals a different aspect of RLVR's behavior that could be hidden by the others:

High SRR, low NDR, low SDS, negative NSCR characterizes the support-constrained optimization regime: RLVR preserves most base-model solutions but discovers few new ones, and the few discoveries are outweighed by losses, producing net shrinkage. This is the dominant pattern across all 1.5B–14B models in the paper.
Low SRR, high NDR, moderate SDS, positive NSCR would characterize aggressive exploration at the cost of forgetting: RLVR discovers many new solutions but abandons many existing ones. This pattern does not appear in the paper's results.
High SRR, high NDR, high SDS, positive NSCR would characterize genuine capability expansion: RLVR preserves existing solutions while discovering new ones, expanding the overall reachable set. This is the ideal case that the paper's results suggest RLVR currently fails to achieve.

The paper acknowledges a limitation: these metrics are sensitive to the choice of $\epsilon$ and $k$ . The same model pair evaluated with different sampling budgets could yield different support counts (higher $k$ pushes $\epsilon$ lower, potentially revealing completions that were previously below detectability). The paper mitigates this by using large, consistent $k$ across experiments (4,096–8,192 for math, 1,024–16,384 for non-math) and explicitly computing $\epsilon$ from the confidence bound.

The Experimental Measurement Protocol

Model selection and rationale. The paper evaluates eight base-model/RLVR pairs spanning four base model families and scales from 1.5B to 14B parameters (Table 1):

DeepSeek-R1-Distill-Qwen-1.5B → ProRL-1.5B-v1 and ProRL-1.5B-v2 (two checkpoints of the same training run at different steps)
DeepSeek-R1-Distill-Qwen-7B → AceReason-Nemotron-7B and Skywork-OR1-7B
DeepSeek-R1-Distill-Qwen-14B → Nemotron-1-14B and Phi4-Reason-Plus-14B
Qwen2.5-Math-7B → DAPO (evaluated separately for SFT vs. RLVR comparison)
Qwen2.5-32B → DAPO-32B (entropy analysis only)
OLMo-2-0425-1B → RLVR (Appendix comparison)
OVR-7B-ColdStart (vision-language model) → Kangheng-OVR-7B

The choice of DeepSeek-R1-Distill-Qwen as the primary base model family is strategic: these are strong but not saturated reasoning models, with non-trivial but far-from-perfect pass@1 on MATH benchmarks. This places them in the regime where both preservation and expansion are measurable — a model that already achieves 100% pass@1 leaves no room for expansion to occur, while a model with near-zero pass@1 makes shrinkage essentially meaningless because there is nothing to lose.

Sampling budgets and justification. The paper uses aggressive sampling budgets, far larger than typical evaluation settings, precisely because the $\epsilon$ threshold shrinks as $k$ grows:

Math benchmarks: $k \in \{4096, 8192\}$ for most models, yielding $\epsilon \approx 7.32 \times 10^{-4}$ (at $k=4096$ , 95% confidence) or $\epsilon \approx 3.66 \times 10^{-4}$ (at $k=8192$ ). The AIME, AMC, MATH500, Minerva, and OlympiadBench evaluations all use these budgets.
Non-math benchmarks: $k \in \{1024, 2048\}$ for SimpleQA, LiveBench, SciBench, and LiveCodeBench, yielding correspondingly higher $\epsilon$ thresholds (e.g., $\epsilon \approx 2.93 \times 10^{-3}$ at $k=1024$ ). The paper acknowledges this means "the empirical support threshold of the base model" is higher for non-math tasks — completions with very low probability may be classified as Out of Support when they might fall into Shrinkage or Expansion under larger budgets. However, the consistent pattern of dominant preservation across all $k$ values suggests this is not distorting the qualitative conclusions.
Reasoning Gym: $k \in \{1024, 2048, 4096, 8192, 16384\}$ , providing the widest range and allowing the paper to trace pass@k curves across orders of magnitude of sampling budget (Figures 2–4). This is where the pass@k crossover phenomenon — base model outperforming RLVR at large $k$ — is most visible.

Sampling parameters. All generation uses temperature 0.6, top-p 0.95, and maximum response length of 32,768 tokens (Appendix B.1). The relatively high temperature (0.6, compared to the typical 0.0–0.2 used for greedy/beam-search evaluation) is deliberate: it ensures sufficient diversity for the support analysis to be meaningful. A temperature of 0 would collapse sampling to a single deterministic output per prompt, making support measurement impossible. The high maximum length accommodates the long reasoning chains characteristic of DeepSeek-R1-style models.

Inference infrastructure. The paper uses vLLM (Kwon et al., 2023) as the inference backend, chosen for its efficient memory management and throughput — necessary given the massive sampling budgets (8,192 samples × 500 prompts × multiple models = millions of generations).

Correctness determination. The reward function $R(x, y)$ is task-specific:

Math benchmarks: the grading function from Lighteval (Habib et al., 2023) extracts final answers from completions and compares them to ground truth. For MATH500, Minerva, OlympiadBench, AIME, and AMC, this uses exact string matching after normalization.
SimpleQA: GPT-4.1 serves as the judge, evaluating whether the model's answer matches the ground-truth answer for factual questions. The paper explicitly notes this: "For SimpleQA, we employ GPT-4.1 as the judge" (Section 4.1).
LiveBench: the paper uses the 2024-11-25 version and "focused exclusively on tasks with binary correct/incorrect judgments" to avoid ambiguity. This includes web of lies v2 and spatial subsets for reasoning, typos subset for language, and all coding tasks. Tasks with "intermediate floating-point judgments" are excluded because they "lack clear correctness criteria" (Appendix B.2).
SciBench: complete dataset evaluation with standard grading.
LiveCodeBench: v5 and v6 versions, evaluated only on 1.5B and 7B models due to computational constraints.
Reasoning Gym: the paper uses the "easy set" from the repository version after commit 17a8431 as the default configuration, with the "hard set" for challenging evaluation. A critical detail in Appendix B.3: the paper enhanced the answer extraction protocol specifically because the base model frequently produced correct answers in non-standard formats (e.g., LATEX formatting when the reward function expected plain text, scientific notation vs. standard notation mismatches). The authors describe a "hierarchical, priority-based extraction mechanism" with multiple fallback levels to ensure both base and RLVR models are evaluated fairly despite format differences. This is methodologically important because format incompatibility could artificially inflate the appearance of support expansion (the base model "couldn't" find the solution because its correct answer was parsed incorrectly, making it look like RLVR discovered something new when it really just learned to format correctly).

The SFT vs. RLVR controlled comparison. A critical experiment in Section 4.2 fixes "the base model (Qwen2.5-Math-7B), dataset (DeepMath-103K), sampling protocol, and optimization hyperparameters, varying only the training objective (SFT vs. DAPO)." This is designed to isolate whether support shrinkage is inherent to RLVR's objective or an artifact of other factors:

Under SFT, the model sees correct completions from the training data and is trained to maximize their likelihood via standard cross-entropy. This pushes probability mass toward observed correct completions but does not explicitly push mass away from anything — incorrect completions are simply not reinforced.
Under DAPO (an RLVR variant with GRPO-style updates), the model receives positive reward for correct completions and negative signal for incorrect ones, creating an explicit pressure to reduce probability on incorrect outputs while increasing it on correct ones.

The result — SFT shows positive NSCR (net expansion) while DAPO shows negative NSCR (net shrinkage) — supports the paper's theoretical claim that the support-constrained behavior "emerges from the objective itself" rather than from data, scale, or implementation. SFT's likelihood-based objective naturally preserves probability mass in regions where the base model already assigned it (since SFT starts from the base model and adds mass on correct completions without subtracting from anything), while RLVR's reward-maximizing objective explicitly redistributes mass, concentrating it on a subset of correct completions at the expense of others.

Temporal analysis of support dynamics. The paper tracks support metrics at 30-step intervals during DAPO training of DeepSeek-1.5B on math benchmarks (Table 4, Appendix). Rather than taking a single before/after snapshot, this reveals the trajectory of support evolution:

AIME24: SRR starts at 0.960 (step 30), briefly reaches 1.000 (step 90, zero shrinkage), then declines to 0.880 (steps 240–300) as shrinkage accumulates. NSCR starts at -0.040, briefly hits 0.000, then declines to -0.120. The model initially preserves support well but progressively loses correct completions, and never achieves a single expansion (E = 0 throughout all 300 steps).
AIME25: NDR briefly rises to 0.136 (step 60) with 3 expansions, but this is transient — by step 180, expansions drop to 0 and NSCR becomes negative. The model briefly explores and finds new solutions, then loses them.
MATH500: Near-perfect SRR (>0.98) throughout, but slow accumulation of shrinkage: S rises from 2 (step 30) to 8 (step 240) to eventually 8 (step 300). Even on "easy" benchmarks where most solutions are preserved, RLVR slowly loses access to correct completions that the base model could reach.
Olympiad: The most dramatic decline: S rises from 22 (step 30) to 61 (step 300), with NSCR declining from -0.015 to -0.093. SDS drops from 0.043 to 0.018. This is the paper's clearest evidence that RLVR training is not monotonically beneficial — longer training progressively narrows the solution set, even as pass@256 remains relatively stable or declines only slightly.

The perplexity analysis as a complementary probe. Table 2 reports an additional measurement that does not rely on the support framework: the perplexity of the base model and the RLVR model when evaluated against external reasoning traces from DeepSeek-R1 and Claude Sonnet 4. This measures how compatible each model is with diverse reasoning styles:

For shrinkage cases (correct under base, incorrect under ProRL): ProRL shows higher perplexity on both external references. On AIME24 against Claude Sonnet 4, base perplexity is 1.70, ProRL perplexity is 2.12 — a 24.7% increase. This suggests that ProRL has concentrated probability mass onto its own reasoning style, making it less able to "understand" (assign high probability to) alternative correct reasoning paths.
For expansion cases (incorrect under base, correct under ProRL): the pattern reverses. On Olympiad, base perplexity against DeepSeek-R1 is 1.41, ProRL perplexity is 1.28 — a decrease. The "new" solution the RLVR model can access is one that looks more like the external reference.
For cases where both models fail: ProRL shows uniformly higher perplexity against both references, with the largest gap on Claude Sonnet 4 (8.76 → 14.91 on AIME24, a 70.2% increase). This is strong evidence that RLVR doesn't just drop probability on incorrect completions — it also reduces probability on correct-but-different-reasoning-style completions, making the model overall less compatible with diverse solution approaches.

The paper interprets this as evidence that RLVR concentrates probability "around narrower solution trajectories" and that "the dominant effect is structural: RLVR concentrates probability around narrower solution trajectories." This complements the support-count analysis with a continuous measure of distributional narrowing.

The Theoretical Foundations of Support-Bounded Behavior

The paper provides formal proofs explaining why RLVR cannot escape the base model's support, grounding the empirical findings in the mathematics of policy gradient updates.

Theorem C.1: Support Preservation Under RLVR (Infinite-Sample Version). The theorem states that for any RLVR-trained distribution $\pi_\theta$ obtained via standard on-policy gradient updates with verifiable rewards:

$\text{supp}(\pi_\theta(\cdot \mid x)) \subseteq \text{supp}(q(\cdot \mid x))$

where $\text{supp}(p) = \{y \in \mathcal{C} \mid p(y \mid x) > 0\}$ is the mathematical support (not the empirical support — this theorem uses strict positivity, not the $\epsilon$ -threshold version).

Proof structure (simplified). The proof proceeds by induction:

Base case: Initialize $\pi_{\theta_0} = q$ , so $\text{supp}(\pi_{\theta_0}) = \text{supp}(q)$ trivially.
Inductive step: Assume at training step $t$ , the policy $\pi_{\theta_t}$ assigns zero probability to some completion $y^*$ , i.e., $\pi_{\theta_t}(y^* \mid x) = 0$ . All standard policy gradient updates (REINFORCE, PPO, GRPO, DAPO) take the form:

$\theta_{t+1} = \theta_t + \eta \nabla_\theta \mathbb{E}_{y \sim \pi_{\theta_t}(\cdot \mid x)}\left[R(x, y) - \beta^{-1} \log \frac{\pi_\theta(y \mid x)}{q(y \mid x)}\right]$

The expectation is over $y \sim \pi_{\theta_t}$ — the current policy's own distribution. Since $\pi_{\theta_t}(y^* \mid x) = 0$ , $y^*$ is never sampled, and therefore contributes zero to the gradient. Consequently, $\pi_{\theta_{t+1}}(y^* \mid x) = 0$ as well.

Conclusion: By induction, no update can ever introduce positive probability mass on any completion $y^*$ for which $q(y^* \mid x) = 0$ , and no update can increase probability of any completion with initially zero probability in the current policy.

What this theorem establishes: In the mathematical limit (infinite samples, exact gradients), RLVR cannot create new reasoning capabilities — it can only redistribute probability mass among outputs the base model already assigns non-zero probability to. This is the formal statement of the "invisible leash."

Why this form matters — and its limitations. The theorem uses mathematical support (strict positivity) rather than empirical support ( $> \epsilon$ ). This makes the proof clean (the inductive step is exact) but potentially overstates the practical constraint: since softmax assigns $>0$ probability to every completion, the mathematical support of $q$ is the entire output space, meaning the theorem's conclusion $\text{supp}(\pi_\theta) \subseteq \text{supp}(q)$ is vacuously true and places no practical restriction on RLVR. The theorem is valuable as a conceptual anchor — it proves that RLVR cannot create probability mass where none existed — but it doesn't explain why RLVR fails to amplify the long tail of the base model's distribution. That's what Theorem C.3 addresses.

Corollary C.2: Asymptotic Sampling Upper Bound. Under the support preservation condition and independent sampling:

$\limsup_{k \to \infty} \text{pass@k}_{\pi_\theta}(x) \leq \limsup_{k \to \infty} \text{pass@k}_q(x)$

What it computes: the asymptotic pass@k (as the number of samples goes to infinity) for the RLVR model cannot exceed that of the base model. In the infinite-sample limit, the RLVR model will eventually find at most as many correct solutions as the base model, and potentially fewer.

Why this form: as $k \to \infty$ , pass@k approaches 1 if the model assigns any non-zero probability to at least one correct completion, and 0 otherwise. Since the RLVR model's support is a subset of the base model's support, any correct completion accessible to the RLVR model is also accessible to the base model, but the converse need not hold. Therefore, the base model's asymptotic pass@k is an upper bound on the RLVR model's.

This corollary directly predicts the pass@k crossover phenomenon: if the RLVR model has undergone support shrinkage (some correct completions that the base model could access are no longer accessible), then for large enough $k$ , the base model will find those "lost" completions while the RLVR model cannot, producing higher pass@k for the base model. This is exactly what the paper observes empirically (e.g., AIME 2024 base pass@8192 = 93.3% vs. ProRL-1.5B's 83.3%).

Theorem C.3: Empirical Support Preservation (Finite-Sample Version). The mathematical support theorem is too weak to explain the paper's empirical results, because softmax ensures every completion has non-zero probability. The practical question is whether RLVR can amplify extremely low-probability completions above the detectability threshold. Theorem C.3 addresses this:

$\text{supp}_\epsilon(\pi_\theta(\cdot \mid x)) \subseteq \text{supp}_\epsilon(q(\cdot \mid x))$

under standard sampling and update procedures with finite sample budgets. That is, even the $\epsilon$ -thresholded empirical support of the RLVR model is contained within that of the base model. A completion that the base model assigns probability below the detectability threshold cannot be lifted above that threshold by RLVR alone, for any finite number of training steps.

Proof structure (simplified). The proof decomposes the RLVR gradient update into two components (following Zhu et al., 2025):

$\nabla \mathcal{L}_{\text{total}} = \nabla \mathcal{L}_{\text{PSR}} + \nabla \mathcal{L}_{\text{NSR}}$

where PSR (Positive Sample Reinforcement) increases probability on correct completions that are actually sampled, and NSR (Negative Sample Reinforcement) decreases probability on incorrect completions while redistributing that mass proportionally to the current policy. For a completion $y^*$ with $q(y^* \mid x) \leq \epsilon$ :

$\nabla \mathcal{L}_{\text{PSR}}$ contributes nothing to $y^*$ because $y^*$ is never sampled as a positive example (its probability is below detectability, so it virtually never appears in the finite rollout set).
$\nabla \mathcal{L}_{\text{NSR}}$ can only increase $y^*$ 's probability via redistribution from incorrect completions, but this increase is bounded. The proof shows that after $K$ NSR updates, the probability of any completion $y^*$ can grow by at most a factor of $\exp(2\eta T K)$ , where $\eta$ is the learning rate, $T$ is the sequence length, and $K$ is the number of update steps.

Consequently, $\pi^{(K)}(y^* \mid x) \leq \epsilon \cdot \exp(2\eta T K)$ . For finite $\eta$ , $T$ , and $K$ , this product is still a finite multiple of $\epsilon$ , and as $\epsilon \to 0$ , it remains vanishingly small. Therefore, $\pi_\theta(y^* \mid x) > \epsilon$ can only hold if $q(y^* \mid x) > \epsilon$ in the first place — the empirical support cannot expand.

Why this form matters: this theorem explains why the paper observes near-zero NDR across all experiments. Even though the base model technically has non-zero probability on every completion (softmax), completions with very low probability cannot be "surfaced" by RLVR because the gradient updates that would increase their probability (positive sample reinforcement) require sampling them first, which requires them to already have non-negligible probability. This is a catch-22: to be amplified, a completion must already be sufficiently probable to be sampled; if it is sufficiently probable to be sampled, it was already in the empirical support.

The theorem also explains the rare expansion cases the paper observes. When expansion does occur, it happens via two mechanisms that circumvent the catch-22:

Subskill recombination: The base model assigns non-negligible probability to fragments of the correct solution individually, and RLVR amplifies these fragments and helps the model assemble them coherently. The expansion doesn't create new probability mass from nothing — it rearranges existing mass from sub-solution regions into the full correct combination.
Format correction: The base model could produce the correct answer but in a format that the reward function couldn't parse, making it appear "incorrect" during training. RLVR reshapes the distribution to follow instructions, making the same underlying reasoning now recognized as correct. The solution itself was always in the support; the barrier was format compatibility, not reasoning capability.

Neither mechanism constitutes genuine discovery of reasoning patterns absent from the base model — they are reorganization or surfacing of already-present capabilities. This directly reinforces the paper's central claim: RLVR can amplify and reorganize what the base model already knows, but it cannot create new knowledge.

Proposition C.4: Variational Interpretation as KL Projection. The paper provides a complementary theoretical perspective: RLVR can be understood as solving a KL-constrained optimization problem. Rather than viewing RLVR as directly maximizing reward, it can be framed as finding the distribution closest to the base model (in KL divergence) that satisfies a reward constraint.

The solution to the following variational problem:

$\min_{\pi \in \mathcal{P}_\rho} \text{KL}(\pi \parallel q) \quad \text{where} \quad \mathcal{P}_\rho = \{\pi \in \Delta(\mathcal{Y}) \mid \mathbb{E}_\pi[R(x, y)] \geq \rho\}$

takes the form:

$\pi^*(y \mid x) \propto q(y \mid x) \cdot \exp(\beta R(x, y))$

where $\beta \geq 0$ is the Lagrange multiplier (dual variable) associated with the reward constraint $\rho$ , $\Delta(\mathcal{Y})$ is the probability simplex over the output space, and $\mathcal{P}_\rho$ is the set of distributions achieving at least expected reward $\rho$ .

What this computes: given a target expected reward $\rho$ , find the distribution that achieves at least that reward while minimizing divergence from the base model. The optimal distribution is an exponentially tilted version of the base model: each completion's probability is multiplied by $\exp(\beta)$ if it's correct (where $\beta$ depends on how tight the reward constraint is) and unchanged if it's incorrect, then the whole distribution is renormalized.

Why this form: this reveals that RLVR is fundamentally a conservative update. The KL penalty term (whether explicit in the objective or implicit in the optimization dynamics) ensures that the trained distribution stays as close as possible to the base model while meeting the reward constraint. This is precisely the "invisible leash" — the optimization prefers to concentrate mass on high-reward completions that the base model already assigns substantial probability to, rather than venturing into regions where the base model assigns negligible probability, because KL divergence penalizes putting mass where $q$ puts none.

The paper also notes the KL-free limit (Corollary C.5): as $\beta \to \infty$ (when explicit KL regularization is removed or becomes negligible), the RLVR update converges to:

$\lim_{\beta \to \infty} \pi_\beta(y \mid x) = \frac{q(y \mid x) \cdot \mathbf{1}\{y \in \mathcal{C}\}}{\sum_{y' \in \mathcal{C}} q(y' \mid x)}$

That is, the RLVR model becomes the base model's distribution restricted to correct completions and renormalized. Even in this extreme, the relative probabilities among correct completions are preserved from the base model — the model never creates new correct completions, only filters out incorrect ones and reweights. This explains why even aggressive RLVR variants without explicit KL penalties (like DAPO) still show the support-preservation pattern: the training dynamics themselves, not just explicit regularization, keep the model tethered to its base distribution.

The Entropy-Reward Trade-off Formalization

Theorem C.6: Entropy Reduction and Precision-Coverage Trade-off. The paper proves a complementary structural property of RLVR: it systematically reduces the entropy of the output distribution.

Part (a): Entropy reduction. For any RLVR update $\pi_\theta$ of the exponentially tilted form $\pi_\theta(y \mid x) = q(y \mid x) \exp(\beta R(x, y)) / Z$ :

$H[\pi_\theta] \leq H[q]$

with equality if and only if the reward is constant on the support of $q$ .

Proof sketch: By properties of KL divergence, $\text{KL}(\pi_\theta \parallel q) \geq 0$ . Expanding:

$\text{KL}(\pi_\theta \parallel q) = \sum_y \pi_\theta(y) \log \frac{\pi_\theta(y)}{q(y)} = -H[\pi_\theta] + H[q, \pi_\theta]$

where $H[q, \pi_\theta]$ is the cross-entropy. Rearranging gives $H[\pi_\theta] = H[q] - \text{KL}(\pi_\theta \parallel q) \leq H[q]$ . The inequality is strict unless the reward is constant (making $\pi_\theta = q$ , KL divergence zero, and entropy unchanged).

What this means: RLVR mathematically cannot increase the entropy of the output distribution relative to the base model. This isn't a contingent empirical finding — it's a theorem following from the exponential tilting form of the RLVR update. Any apparent increase in "uncertainty" (which the paper observes at the token level) must be a compositional phenomenon (longer sequences have more tokens, each with its own entropy, and the sum can increase even as per-token entropy on corresponding positions decreases) rather than evidence of increased answer-level diversity.

Part (b): Trade-off with coverage. Lower entropy improves precision at small sampling budgets but reduces diversity at large budgets. Formally:

Small $k$ (e.g., $k = 1$ ): The increased concentration on high-reward outputs improves pass@1, since $\text{pass@1}_{\pi_\theta}(x) = \sum_{y: R(x,y)=1} \pi_\theta(y \mid x) > \sum_{y: R(x,y)=1} q(y \mid x) = \text{pass@1}_q(x)$ whenever the reweighting boosts correct completions relative to incorrect ones.
Large $k$ : Reduced entropy leads to concentration on fewer modes. If there exist rare but correct completions that are further downweighted by the tilting, the total mass on correct completions can actually decrease: $\pi_\theta(\mathcal{C}) < q(\mathcal{C})$ , meaning the RLVR model assigns lower total probability to all correct completions combined than the base model does.

The token-level vs. answer-level entropy decoupling. The paper's entropy experiments (Section 5) reveal a subtlety not captured by the theorem directly: RLVR sometimes increases token-level entropy while decreasing answer-level entropy. The paper calls this "local stochasticity without global exploration" (Section 5.2).

Token-level entropy is computed as the average entropy of the model's output distribution at each generation step:

$\text{TokenEntropy} = \frac{1}{N} \sum_{i=1}^N \left(\frac{1}{T^{(i)}} \sum_{t=1}^{T^{(i)}} H^{(i)}_t\right)$

where $H^{(i)}_t = -\sum_{v \in \mathcal{V}} p^{(i)}_t(v) \log p^{(i)}_t(v)$ is the entropy of the vocabulary-level distribution at step $t$ of sequence $i$ . Answer-level entropy is the entropy over distinct final answers:

$\text{AnswerEntropy} = -\sum_{j=1}^M p_j \log p_j$

where $M$ is the number of unique answers among $N$ completions and $p_j = f_j / N$ is the empirical frequency of answer $j$ .

In the paper's results (Table 5), ProRL-1.5B shows increased token-level entropy (0.44 → 0.52) but decreased answer-level entropy (1.30 → 0.66) relative to DeepSeek-1.5B. DAPO-32B shows an even more dramatic effect: token-level entropy rises from 0.18 to 0.29, while answer-level entropy plummets from 1.61 to 0.61.

Why this decoupling occurs: the paper suggests that increased token-level entropy may arise from longer or more elaborated reasoning chains that introduce additional decision points ("forking" tokens, as described in Wang et al., 2025). The model appears more "uncertain" at each step because it's making more fine-grained decisions about reasoning structure, but all those paths ultimately converge to a small set of final answers — the entropy is being "spent" on surface-level variation (wording, formatting, intermediate steps) rather than on genuinely different solution strategies.

This finding is methodologically important because it warns against using token-level entropy as a proxy for exploration. A model with high per-step entropy may look like it's exploring broadly but in fact be converging to the same narrow set of outputs. The paper's answer-level entropy metric captures the diversity that actually matters for support analysis.

The Role of Difficulty and Task-Specific Dynamics

When expansion occurs: two identified mechanisms. The paper doesn't just document that expansion is rare — it analyzes when it does occur to understand the conditions under which RLVR can partially escape the invisible leash (Section 4.3).

Mechanism 1: Subskill recombination. Tasks like graph coloring, arc 1d, and boxnet share a common property: they are compositional, requiring the model to assemble local operations (e.g., coloring a single vertex, updating a single grid cell, manipulating a JSON key-value pair) into a globally correct solution. The base model already assigns non-negligible probability to the components individually but fails to assemble them correctly. RLVR amplifies these weakly-represented components and helps the model learn to compose them — not by creating new subskills, but by learning which combinations of existing subskills lead to reward.

The paper operationalizes this through the pass@k curve shape: "all observed expansions occur in tasks where the base model's pass@1 is low, but its pass@k curve rises steadily (Fig. 3), indicating that all fragments needed for correct reasoning already lie in the base model's long tail." The key empirical signature is a base model with low pass@1 (the model rarely gets it right on a single try) but high pass@k at large $k$ (if you sample enough, you eventually see correct solutions). This means the components are present — they're just not being assembled correctly most of the time.

Mechanism 2: Format correction. In tasks like dice and boxnet, the base model demonstrates partial competence but fails to follow the response format required by the reward function. The paper's enhanced answer extraction protocol (Appendix B.3) was designed specifically to catch these cases for fair evaluation, but during training, the reward function receives the raw model output, not the enhanced extraction. If the base model produces "3/4" when the reward expects "\frac{3}{4}", it receives zero reward despite producing the correct answer. RLVR can reshape the output distribution to match the expected format, "unlocking capabilities that were previously present but suppressed."

The paper supports this interpretation with the perplexity analysis: "the perplexity gaps in Tab. 2 remain modest in expansion cases, showing that the 'new' completions are stylistic or formatting variants of reasoning patterns already accessible to the base model." If RLVR were genuinely discovering new reasoning approaches, the perplexity of those completions against external references would be lower for the base model (which can't produce them) and higher for RLVR (which learned a novel pattern). Instead, the small perplexity gaps suggest the reasoning content was already present.

Limits of expansion. Even these two mechanisms produce only modest expansion. Across all 1.5B–14B models, NDR never exceeds 0.04, and NSCR remains non-positive for every model-domain pair. The paper is explicit: "RLVR's gains arise from amplifying low-probability but existing solution fragments or format-correct variants, but never from discovering solutions truly absent from the base distribution."

This is the paper's strongest empirical claim about the "invisible leash": even in the most favorable cases (compositional tasks with clear subskill structure, format-sensitive tasks where the base model's competence is masked), RLVR cannot create genuinely new reasoning capabilities. It can only reorganize, amplify, and surface what already exists in the base model's long tail. Breaking this constraint, the paper argues, "requires future innovations that seed probability mass into underrepresented solution regions" — explicit exploration mechanisms, off-policy data infusion, or architectural changes that go beyond the current RLVR paradigm of on-policy reward maximization starting from a frozen base distribution.

4. Key Insights and Innovations

This section identifies the conceptual moves that distinguish the paper — not the mechanisms (which Section 3 details at length), but the intellectual contributions that change how a researcher thinks about RLVR after reading this work.

Innovation 1: Empirical Support as a Diagnostic Framework for Reasoning Capability, Not Just Performance

What’s distinctive at the idea level. The paper introduces empirical support — the set of correct completions a model can realistically discover under finite sampling, defined with a statistically principled threshold derived from sampling confidence bounds — as the central analytical construct for evaluating RLVR. This shifts the unit of analysis from how often does the model get the right answer? (pass@k accuracy) to which correct answers can the model access at all, and what happens to that access set during training?

This is not an incremental refinement of existing metrics. It is a fundamental reframing of what it means to "improve" a reasoning model. Under the standard evaluation paradigm, a model that achieves 90% pass@1 by reliably producing a single correct solution strategy is strictly "better" than a model with 70% pass@1 that can access five different correct strategies. The empirical support framework reveals that the first model has actually lost capability along an important dimension — solution diversity — even as its measured accuracy improved. This is a genuinely diagnostic concept: it separates the signal (genuine expansion of reasoning capacity) from the confound (improved precision via probability concentration on known modes).

Comparison to prior work. Before this paper, the debate about whether RLVR expands or sharpens reasoning was conducted through pass@k comparisons and qualitative argument. Yue et al. (2025a) observed that pass@k crossover (base model winning at large k) suggested conservative optimization, but lacked a formal framework for quantifying how much solution space was lost. Zhao et al. (2025) described RLVR as an "echo chamber" but didn't measure the size or composition of the chamber. Liu et al. (2025a) claimed that ProRL "expands reasoning boundaries" using pass@k improvements as evidence, but couldn't distinguish between genuinely new solutions and amplification of low-probability existing ones.

The paper's empirical support framework resolves this impasse by making "expansion" and "shrinkage" operational: you count completions, you set a threshold, you classify, and you compute metrics. This transforms a philosophical debate into a measurement problem. The four support categories (Preservation, Expansion, Shrinkage, Out-of-Support) and four derived metrics (SRR, NDR, SDS, NSCR) provide a shared vocabulary and quantitative standard that neither side of the prior debate had.

Significance beyond raw performance. The framework's value is theoretical and methodological, not just empirical. It provides a falsifiable criterion for claims about capability expansion: if you assert that your training method expands reasoning, you must demonstrate positive NSCR and non-trivial NDR. The paper shows that RLVR systematically fails this test across models, domains, and scales (Table 1: NSCR negative for all model-domain pairs, NDR ≤ 0.04) — a finding that would be impossible to establish cleanly without the framework.

Moreover, the framework reveals a qualitative structure to RLVR's behavior — the precision-diversity trade-off is not just an empirical correlation but a mathematical consequence of the KL-projection form of RLVR updates (Proposition C.4), which the paper proves. The entropy-reward trade-off (Theorem C.6) formalizes why this happens: lower entropy necessarily follows from exponential tilting, and lower entropy implies fewer modes. The framework thus connects empirical measurement (support counts) to theoretical mechanism (KL-constrained optimization) in a tight loop that prior work lacked.

Evidence anchor. The framework's diagnostic power is visible in the SFT vs. DAPO controlled comparison (Table 3, Section 4.2): SFT shows positive NSCR (net expansion) while DAPO shows negative NSCR (net shrinkage), despite identical data, model, and hyperparameters. This directly demonstrates that support-constrained behavior "emerges from the objective itself" — a claim that would be merely hand-wavy without the empirical support metrics to quantify it.

Innovation 2: The Proof That RLVR Is Inherently Support-Bounded — And Precisely How

What's distinctive at the idea level. The paper provides formal proofs establishing that RLVR's inability to escape the base model's support is not an implementation flaw, not a hyperparameter problem, and not a domain-specific quirk — it is a mathematical consequence of on-policy gradient optimization starting from a frozen base distribution. This transforms the "invisible leash" from a metaphor into a theorem.

The key theoretical move is the distinction between mathematical support and empirical support, and the proof that RLVR is bounded by both. Theorem C.1 establishes the clean infinite-sample result: on-policy gradient updates from initialization $q$ can never assign positive probability to any completion that $q$ assigns zero probability to. But this is insufficient to explain the empirical results, because softmax means $q$ assigns positive probability to everything. The genuinely novel theoretical contribution is Theorem C.3, which proves that even under finite sampling, completions below a detectability threshold $\epsilon$ cannot be lifted above $\epsilon$ by RLVR alone — the gradient updates that would amplify them require sampling them first, which requires them to already be above $\epsilon$ , forming a catch-22.

Comparison to prior work. Prior theoretical work on RLVR (Zhao et al., 2024; Zhu et al., 2025) analyzed sample complexity and convergence properties under coverage assumptions, but did not prove that the base model's support serves as an absolute ceiling. The variational interpretation of RLVR as KL projection (Proposition C.4) is not entirely new — it echoes the RLHF analysis of Zhao et al. (2024) and the exponential tilting connections in contextual bandits — but the paper extends this insight in two distinctive ways. First, it proves the finite-sample empirical support bound (Theorem C.3), which is the practically relevant result given that all real training uses finite rollouts. Second, it connects the variational interpretation directly to the support-dynamics metrics, showing that the KL-projection view predicts high SRR and low NDR rather than merely being consistent with them.

The KL-free limit analysis (Corollary C.5) is a subtle but theoretically important contribution. Recent open-source RLVR recipes (DAPO, GRPO without explicit KL penalties) have trended toward removing KL regularization, under the implicit assumption that this would enable more aggressive exploration and capability expansion. The paper proves that even in the β → ∞ limit — with no explicit KL penalty whatsoever — the optimal policy is simply the base model's distribution renormalized over correct completions, preserving relative probabilities within the correct set. This means removing KL regularization does not solve the exploration problem; it only sharpens the distribution faster.

Significance beyond raw performance. This is a theoretical advance with direct practical implications for research prioritization. By proving that the leash is inherent to the on-policy RLVR objective rather than a contingent implementation issue, the paper redirects the research agenda from "let's tune RLVR better" toward "let's break the leash with mechanisms that explicitly seed probability mass into unexplored regions." The proof tells you what kind of intervention is needed: anything that relies on sampling from the current policy to discover new solutions (standard on-policy RL) cannot escape the base model's support; you need off-policy data, explicit exploration bonuses, architectural modifications that change the probability landscape without gradient updates, or hybrid training paradigms (like SFT + RLVR interleaving) to create probability mass where none existed.

The finite-sample bound (Theorem C.3) also provides a quantitative limit: a completion with base model probability $\epsilon$ cannot exceed $\epsilon \cdot \exp(2\eta T K)$ after $K$ RLVR updates, where $\eta$ is the learning rate and $T$ is sequence length. For typical values ( $\eta \approx 10^{-5}$ , $T \approx 10^3$ , $K \approx 10^3$ ), the amplification factor is $\exp(2 \cdot 10^{-5} \cdot 10^3 \cdot 10^3) = \exp(20) \approx 4.85 \times 10^8$ , which sounds large but means a completion with $q = 10^{-9}$ is amplified to at most $4.85 \times 10^{-1}$ — potentially above threshold! However, this bound is for the maximum possible amplification under perfect conditions; the proof also shows that in practice, the PSR (positive sample reinforcement) component contributes nothing to low-probability completions, and NSR-based redistribution is far weaker. The practical ceiling is much lower than the worst-case bound suggests, consistent with the paper's empirical finding of near-zero NDR.

Evidence anchor. The SFT vs. DAPO comparison (Table 3) is the cleanest experimental confirmation of the theoretical claim: same data produces expansion under SFT (which is not support-bounded in the same way, since SFT can assign probability mass to completions it observes even if the base model assigned them negligible probability) and shrinkage under DAPO (which inherits the support constraint via on-policy sampling). This directly validates that the objective form — not data, not scale, not implementation — is the root cause.

Innovation 3: The Decoupling of Token-Level and Answer-Level Entropy — Local Stochasticity Without Global Exploration

What's distinctive at the idea level. The paper documents and names a phenomenon that challenges a widely-held assumption in the RL for language models community: increased token-level entropy during generation does not imply increased exploration of the output space. The paper calls this "local stochasticity without global exploration" — the model appears more uncertain at each generation step (higher per-token entropy, more "forking" tokens, more diverse intermediate reasoning paths) while simultaneously converging onto a smaller set of final answers (lower answer-level entropy).

This decoupling is not predicted by standard views of entropy in RL. The typical intuition — inherited from RL applications in continuous control and game-playing — is that higher policy entropy indicates broader exploration, which should lead to more diverse outcomes. The paper shows this intuition fails for language model RLVR: token-level entropy and answer-level entropy can move in opposite directions, and it is the answer-level entropy that actually reflects support coverage.

Comparison to prior work. Prior entropy analyses in RLVR (Cui et al., 2025) documented entropy collapse during training but did not distinguish between token-level and answer-level effects, potentially missing cases where token-level entropy remained high or even increased while answer diversity collapsed. Wang et al. (2025) identified "high-entropy minority tokens" as important for effective RL, but focused on their role in learning dynamics rather than their relationship to global output diversity. The paper's contribution is to separate these two entropy measures and show their divergence, providing both a diagnostic tool (answer-level entropy as a support-coverage proxy) and a conceptual correction (token-level entropy is not a reliable exploration metric).

The finding that ProRL-1.5B increases token-level entropy from 0.44 to 0.52 while halving answer-level entropy from 1.30 to 0.66 (Table 5) is striking because it means the model is more stochastic in its generation process but produces less diverse final outputs. The paper's interpretation — that longer reasoning chains introduce more decision points whose variation is surface-level (wording, intermediate step ordering, formatting) while the final answer set narrows — provides a mechanistic explanation. The entropy is being "spent" on variation that doesn't translate to different solutions, analogous to a Markov chain that mixes slowly at the token level but converges rapidly to a small set of absorbing states (final answers).

Significance beyond raw performance. This is a diagnostic and conceptual contribution rather than a numerical gain. It addresses a methodological blind spot: researchers monitoring token-level entropy to assess whether their model is "exploring" may be misled into thinking exploration is healthy when answer-level diversity is actually collapsing. The paper's answer-level entropy metric is cheap to compute (it requires only extracting final answers from completions, not running a separate evaluation), making it a practical addition to training dashboards.

The decoupling also has implications for how we design exploration strategies. Standard approaches like entropy bonuses (adding a term proportional to $-H[\pi_\theta]$ to the reward) operate at the token level, encouraging higher per-step uncertainty. If token-level entropy doesn't translate to answer-level diversity — and the paper provides evidence it doesn't — then token-level entropy bonuses may be ineffective at preventing support shrinkage. Exploration mechanisms need to operate at the answer level, incentivizing the model to produce diverse final outputs rather than diverse token sequences, which may require fundamentally different approaches (e.g., embedding-space diversity penalties, explicit multi-modal objectives, or architectural modifications that decouple reasoning path variation from answer convergence).

Evidence anchor. Table 5 is the key exhibit, but the phenomenon is not limited to one model: DAPO-32B shows an even more dramatic decoupling (token entropy 0.18 → 0.29, answer entropy 1.61 → 0.61), while AceReason-7B and Skywork-OR1-7B show the opposite pattern (token entropy decreasing alongside answer entropy). The variation across models — built on the same base architecture — confirms that the decoupling is not a universal law but an important possible regime that standard monitoring would miss.

Innovation 4: The Cross-Domain, Cross-Scale Empirical Demonstration That RLVR's Behavior Is Objective-Driven, Not Artifactual

What's distinctive at the idea level. The paper's empirical scope is not merely breadth for breadth's sake — it serves a specific argumentative purpose: to establish that RLVR's support-constrained behavior is inherent to the RLVR objective itself rather than being an artifact of model scale, domain idiosyncrasy, data composition, or implementation detail. By demonstrating the same pattern (high SRR, near-zero NDR, negative NSCR) across eight model pairs spanning 1.5B to 14B parameters, across math and non-math domains with qualitatively different reward structures, and across multiple RLVR algorithm variants (GRPO-based ProRL, DAPO, and the undisclosed algorithms behind Skywork, AceReason, Nemotron, and Phi4), the paper rules out a wide range of alternative explanations.

This is fundamentally an argument from invariance: if a pattern persists across large variations in the experimental conditions, it is likely driven by a common underlying cause — in this case, the on-policy RLVR objective structure identified in the theoretical analysis. The SFT comparator experiment provides the most direct causal evidence: hold everything constant except the objective function, and the support-constrained behavior disappears. This is the empirical complement to the theoretical proof, and together they make a case that would be substantially weaker with either component alone.

Comparison to prior work. Most prior work on RLVR's limitations studied a single model on a single benchmark, or at most a single domain. Yue et al. (2025a) focused on math reasoning with a specific model family. Zhao et al. (2025) examined a particular RLVR recipe. Liu et al. (2025a) argued for expansion based on their ProRL model's performance on a curated set of benchmarks. None of these studies could rule out the possibility that their findings were domain-specific, model-specific, or algorithm-specific — a critic could always respond "that's just how that particular model/domain/algorithm behaves."

The paper's inclusion of a vision-language model (Kangheng-OVR-7B), non-math reasoning tasks (SimpleQA, LiveBench, SciBench), and the Reasoning Gym (with its diverse task formats from graph coloring to palindrome generation to boxnet) makes the invariance argument substantially stronger. The inclusion of OLMo-2-0425-1B (Table 12, Appendix) — a base model not from the DeepSeek-R1-Distill-Qwen family — further extends the claim beyond a single pretraining distribution.

Significance beyond raw performance. This is a negative result with broad implications, establishing a baseline against which future methods can be compared. Any future work claiming that a new training method "expands reasoning beyond the base model" must now demonstrate not just improved pass@1 or pass@k, but positive NSCR and non-trivial NDR — and must do so across a range of domains and model scales, or explicitly argue why the domain/model restriction matters. The paper's empirical methodology (large-k sampling, support classification, metric computation) provides the template for such evaluations.

The cross-scale finding — that 14B models show the same support-constrained pattern as 1.5B models — is particularly important because it addresses the objection that the "invisible leash" might be a small-model phenomenon that disappears with scale. Larger models might be expected to have broader base support (making shrinkage harder to detect) or more capacity to learn genuinely new patterns (making expansion more likely). The fact that Nemotron-14B and Phi4-14B show SRR = 0.99, NDR ≤ 0.01, and negative NSCR (Table 1) suggests that scaling model size within the current RLVR paradigm does not break the leash — it only makes the leash longer (more solutions preserved) without detaching it (still no meaningful expansion).

Evidence anchor. Table 1 is the central exhibit, but the temporal training analysis (Table 4) adds crucial nuance: even on benchmarks where the final model shows near-perfect preservation, there are transient expansions (e.g., AIME25 at step 60 shows NDR = 0.136 with 3 expansions, later lost) that suggest the training process sometimes briefly explores new solutions before collapsing back. This temporal pattern would be invisible in a single-snapshot comparison, and it complicates the picture: RLVR is not incapable of producing expansions, but it is incapable of sustaining them — the optimization dynamics eventually prune them. This reframes the problem from "how do we get RLVR to discover new solutions?" to "how do we get RLVR to keep the new solutions it discovers?" — a more tractable and specific research question.

Innovation 5 (Minor): The Problem Specification of When Expansion Can Occur — Subskill Recombination and Format Correction

What's distinctive at the idea level. Rather than just documenting that expansion is rare, the paper characterizes the structural conditions under which it does occur, providing a positive (if modest) account of RLVR's capabilities that complements the primarily negative main finding. The paper identifies two specific mechanisms — subskill recombination and format correction — and argues that they exhaust the observed expansion cases, explaining both why expansion happens when it does and why it remains sharply bounded.

This is an explanatory contribution rather than a method or metric. It gives researchers a principled way to predict which tasks will benefit from RLVR beyond simple precision improvement: tasks where the base model demonstrates competence in components but fails at assembly (rising pass@k curves at low pass@1) or tasks where the base model's underlying reasoning is correct but its output format fails the verifier. Conversely, tasks where the base model's pass@k is flat and low (no components present to assemble) or where the failure is genuinely in the reasoning rather than the formatting should not be expected to benefit from RLVR alone — a prediction the paper's data supports (difficulty bin 5 in the prior work's analysis, or the hardest Reasoning Gym tasks where neither model succeeds).

Comparison to prior work. The distinction between "subskill recombination" and "genuinely novel reasoning" is not entirely unprecedented — it echoes the compositional generalization literature and the distinction between "in-distribution" and "out-of-distribution" generalization. However, the paper operationalizes this distinction in the context of RLVR specifically, connecting it to the empirical support framework via the pass@k curve shape (rising vs. flat) and the perplexity analysis (small vs. large gaps on expansion cases). Prior work either didn't distinguish between types of expansion or treated all expansions as equivalent evidence of capability improvement. The paper's analysis shows that expansion cases are qualitatively different from preservation cases — they arise from reorganizing existing capabilities, not from creating new ones — which limits how excited we should be about them.

Significance beyond raw performance. This is an incremental but clarifying contribution that sharpens the research agenda. If RLVR can only help by recombining subskills and correcting formats, then the path to genuine reasoning expansion requires either (a) improving the base model so that it acquires new subskills (via better pretraining, curriculum learning, or data infusion), (b) designing training objectives that can create new solution patterns rather than merely reweighting existing ones (e.g., methods that allow gradient updates to flow to regions with zero current probability mass), or (c) hybrid approaches that interleave RLVR (to sharpen existing skills) with SFT or other data-driven stages (to introduce new solution candidates that RLVR can then amplify). The paper provides negative evidence for option (c) in its current form via the ReST^EM experiment mentioned in the prior work analysis, but frames the two-mechanism account as a constraint on what kinds of expansions are even possible under the current paradigm — a useful prior for researchers designing new methods.

Evidence anchor. The pass@k curves in Figures 2–4 and the perplexity analysis in Table 2 provide the empirical basis: expansion tasks (dice, arc 1d, boxnet, graph color vertex20) show the characteristic "low pass@1, rising pass@k" pattern for the base model, and the perplexity gaps between base and RLVR on expansion cases are modest (Table 2), consistent with the claim that RLVR is surfacing existing reasoning patterns rather than creating new ones. The answer processing enhancements described in Appendix B.3 — where the authors had to add format-normalization steps specifically to ensure the base model's correct-but-malformatted answers were counted correctly — provide practical evidence for the format-correction mechanism: the base model often had the right answer but failed the verifier due to surface formatting, and RLVR's primary contribution was fixing that alignment.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates across two broad categories. Math reasoning tasks: MATH500 (Hendrycks et al., 2021; 500 test questions), Minerva (Lewkowycz et al., 2022), OlympiadBench (He et al., 2024), AIME 2024, AIME 2025, and AMC 2023. Non-math reasoning tasks: SimpleQA (Wei et al., 2024; factuality, evaluated on a 10% uniform sample of 433 questions with GPT-4.1 as judge), LiveBench (White et al., 2025; 2024-11-25 version, using only tasks with binary correct/incorrect judgments — web of lies v2 and spatial for reasoning, typos for language, all coding tasks), SciBench (Wang et al., 2023; complete dataset), LiveCodeBench v5 and v6 (Jain et al., 2024; evaluated only on 1.5B and 7B models due to computational constraints), and Reasoning Gym (Stojanovski et al., 2025; "easy set" from commit 17a8431 as default, with "hard set" for challenging evaluation, focused on tasks ProRL highlighted as challenging). The paper also includes a vision-language benchmark: MathVista (Lu et al., 2023) and MathVision (Wang et al., 2024) for the Kangheng-OVR-7B VLM evaluation.
Base model(s). The primary base models are DeepSeek-R1-Distill-Qwen variants at 1.5B, 7B, and 14B parameter scales. These are strong-but-not-saturated reasoning models, chosen deliberately because they have non-trivial but far-from-perfect pass@1, placing them in the regime where both preservation and expansion are measurable. Additional base models include Qwen2.5-Math-7B (for the SFT vs. DAPO controlled comparison), Qwen2.5-32B (entropy analysis), OLMo-2-0425-1B (Appendix comparison in a different model family), and OVR-7B-ColdStart (vision-language model). The paper argues that DeepSeek-R1-Distill-Qwen is "representative of the capabilities of many contemporary LLMs" in the reasoning domain.
Metrics. Four primary support dynamics metrics are computed from raw counts of correct completions classified into Preservation (P), Expansion (E), Shrinkage (S), and Out-of-Support (O): Support Retention Rate (SRR = P/(P+S)) measures fraction of base-accessible solutions retained; Net Discovery Rate (NDR = E/(P+E)) measures fraction of RLVR-accessible solutions that are genuinely new; Support Dynamic Score (SDS = 2·SRR·NDR/(SRR+NDR)) is the harmonic mean balancing retention and discovery; Net Support Change Rate (NSCR = (E−S)/(P+E+S)) captures net expansion or shrinkage. Standard pass@k accuracy is also reported. For entropy analysis, token-level entropy is the average Shannon entropy of the model's per-timestep vocabulary distribution, and answer-level entropy is the Shannon entropy over the distribution of distinct final answers across completions. Perplexity is computed by evaluating model likelihood against external reference traces from DeepSeek-R1 and Claude Sonnet 4.
Baselines. The primary comparison is always base model vs. RLVR-trained model on the same architecture. For the controlled objective comparison (Section 4.2, Table 3): same base model (Qwen2.5-Math-7B), same dataset (DeepMath-103K), same hyperparameters, varying only the training objective — SFT (supervised fine-tuning via cross-entropy on correct completions) vs. DAPO (Yu et al., 2025; an RLVR variant with GRPO-style updates). For the temporal training analysis (Table 4): the base model at step 0 serves as the baseline against intermediate RLVR checkpoints at 30-step intervals up to step 300.
Generation budget / compute accounting. The paper measures compute in terms of sampling budget k — the number of completions generated per prompt. For math benchmarks (MATH500, Minerva, OlympiadBench, AIME, AMC), budgets are k ∈ {4096, 8192}, with 8192 used for the primary support counts (Table 1, detailed in Appendix A). For non-math benchmarks (SimpleQA, LiveBench, SciBench, LiveCodeBench), budgets are k ∈ {1024, 2048}. For Reasoning Gym, budgets span k ∈ {1024, 2048, 4096, 8192, 16384} to trace pass@k curves across orders of magnitude. The paper derives that at k=8192 with 95% confidence, the empirical support threshold is ϵ ≈ 3.66 × 10⁻⁴ (Appendix C.4). All generation uses temperature 0.6, top-p 0.95, maximum response length 32,768 tokens, with vLLM as the inference backend. The massive sampling budgets are necessary because the ϵ threshold shrinks as 1/k — smaller k would raise ϵ, making it impossible to detect low-probability completions and artificially inflating the appearance of expansion.
Cross-validation / statistical protocol. No cross-validation is used; this is a measurement paper, not a method paper, so the concern is measurement accuracy rather than generalization of a trained policy. The key statistical protocol is the derivation of ϵ from sampling confidence bounds (Appendix C.4): with k samples and desired confidence 1−ζ, any completion not observed has true probability bounded above by −log(ζ)/k. The paper uses ζ = 0.05 (95% confidence), yielding ϵ ≈ 3.66 × 10⁻⁴ for k=8192. For the SFT vs. DAPO controlled comparison, training seeds, data ordering, and hyperparameters are held fixed, isolating the objective as the only varying factor. For evaluation, the enhanced answer extraction protocol (Appendix B.3) applies hierarchical, priority-based extraction with multiple fallback levels to ensure both base and RLVR models are graded fairly despite format differences, preventing format incompatibility from artificially inflating expansion counts.

Main Quantitative Results

Aggregate Support Dynamics Across Models and Domains

The paper's central empirical finding is presented in Table 1: across all evaluated base-model/RLVR pairs spanning 1.5B to 14B parameters, RLVR predominantly acts as a support-constrained optimizer with near-perfect preservation and negligible expansion.

ProRL-1.5B-v1 (overall): P = 2,400 preserved completions, E = 36 expansions, S = 163 shrinkage cases, O = 805 out-of-support. This yields SRR = 0.94, NDR = 0.02, SDS = 0.03, NSCR = −0.05. The shrinkage-to-expansion ratio is 163/36 ≈ 4.5:1 — for every new correct solution RLVR can access that the base model couldn't, it loses access to roughly 4.5 previously accessible solutions.

ProRL-1.5B-v2 (overall): P = 2,388, E = 48, S = 175, O = 793. SRR = 0.93, NDR = 0.02, SDS = 0.04, NSCR = −0.05. The shrinkage-to-expansion ratio is 175/48 ≈ 3.6:1, cited explicitly in the executive summary. The v2 checkpoint (different training step) shows a very similar pattern to v1, suggesting the support dynamics are stable across training — this is not a transient effect.

Nemotron-1-7B (overall): P = 2,715, E = 28, S = 56, O = 605. SRR = 0.98, NDR = 0.01, SDS = 0.02, NSCR = −0.01. The higher SRR relative to ProRL-1.5B reflects the larger model's broader base support (more to preserve) and better training stability. But even with SRR near 1, NDR remains negligible (0.01) and NSCR is still negative — larger scale does not break the pattern.

Skywork-OR1-7B (overall): P = 2,685, E = 26, S = 86, O = 607. SRR = 0.97, NDR = 0.01, SDS = 0.02, NSCR = −0.02. Notable for having the highest shrinkage count (86) among the 7B models, despite similar expansion.

Nemotron-1-14B and Phi4-Reason-Plus-14B: Both near-perfect preservation (SRR = 0.99 for both, overall) with NDR = 0.00–0.01, SDS = 0.01, and NSCR ≈ −0.01. The 14B models discover 8 and 16 new completions respectively while losing 23 and 23 — net shrinkage even at the largest scale tested.

Domain-level patterns (Table 1): Math reasoning consistently shows SRR = 0.96–0.99 and NDR = 0.00–0.01 across all models. Non-math shows slightly higher NDR (up to 0.04 for ProRL-1.5B-v2) but still far below 0.05, and SRR is slightly lower (0.90–0.99). The non-math domain's higher NDR likely reflects the format-correction mechanism — non-math tasks have more diverse output formats (JSON, code, natural language, structured answers), making format misalignment a larger factor for the base model and thus creating more opportunities for RLVR to "discover" solutions that were actually present but malformatted.

Vision-language model (Kangheng-OVR-7B, Table 11): P = 781, E = 3, S = 4, O = 516. SRR = 1.00 (rounding from 0.995), NDR = 0.00, NSCR ≈ 0.00. The pattern holds even for visual reasoning, with expansion essentially zero.

OLMo-2-0425-1B (Table 12): P = 761, E = 83, S = 104, O = 599. SRR = 0.88, NDR = 0.10, SDS = 0.18, NSCR = −0.02. This model — not from the DeepSeek-R1-Distill-Qwen family — shows the highest NDR (0.10) and NSCR still negative, suggesting that even when RLVR discovers more (possibly because the base model's support is sparser), shrinkage still outweighs expansion.

The SFT vs. DAPO controlled comparison (Table 3) is the paper's most direct causal evidence. On Qwen2.5-Math-7B with identical training data and hyperparameters:

MATH500: SFT achieves NSCR = +0.026 (net expansion), P = 480, E = 15, S = 2. DAPO achieves NSCR = −0.012 (net shrinkage), P = 470, E = 6, S = 12. Same data, same model, same procedure — only the objective differs — and the sign of NSCR flips.
Minerva: SFT NSCR = +0.050, E = 28, S = 18. DAPO NSCR = −0.120, E = 12, S = 34. SFT gains 28 new completions while losing only 18; DAPO gains 12 while losing 34.
Olympiad: SFT NSCR = +0.042, E = 49, S = 25. DAPO NSCR = −0.065, E = 25, S = 61. DAPO's shrinkage is more than double its expansion.
AIME24: SFT NSCR = 0.000 (balanced), P = 15, E = 4, S = 4. DAPO NSCR = +0.045, P = 17, E = 3, S = 2. One of the few benchmarks where DAPO shows net expansion, but the absolute numbers are tiny (3 expansions, 2 losses) and SFT also expands.
AMC23: The most dramatic contrast. SFT maintains perfect support (P = 40, E = 0, S = 0, NSCR = 0.000). DAPO loses 3 solutions with zero expansion: P = 37, S = 3, E = 0, NSCR = −0.075.

The consistent pattern is that SFT produces moderate support expansion with positive NSCR, while DAPO produces sharp concentration with mixed or negative NSCR. Since all other factors are held constant, this directly demonstrates that "support-constrained behavior is not an artifact of scaling, data mixture, or procedure, but emerges from the objective itself" (Section 4.2).

Temporal Evolution of Support During RLVR Training

Table 4 tracks support dynamics at 30-step intervals during DAPO training of DeepSeek-1.5B on six math benchmarks, revealing that RLVR reshapes the solution distribution throughout training rather than monotonically improving it.

AIME24: Pass@256 fluctuates between 73.33% and 83.33% across steps, with no clear trend. SRR declines from 0.960 (step 30) to 0.880 (steps 240–300), as shrinkage accumulates from S=1 to S=3. NDR is 0.000 at every single step — zero expansions for the entire 300-step training run. NSCR declines from −0.040 to −0.120. The model never discovers a single new correct solution on AIME24; all observed pass@256 variation comes from redistribution of existing solutions.

AIME25: Shows the most interesting temporal pattern. At step 30, E=2 (NDR=0.095). At step 60, E=3 (NDR=0.136). But by step 180, E=0 and NDR=0.000, and by step 300, E=2 again (NDR=0.095). The model transiently discovers solutions and then loses them. SRR remains around 0.90–0.91 throughout, with S oscillating between 2–3. NSCR starts positive (+0.042 at step 60) and ends negative (−0.042 at step 300, with E=2, S=3). This is direct evidence that RLVR is not incapable of producing expansions — it finds new solutions in early/mid training — but cannot sustain them; the optimization dynamics eventually prune them.

MATH500: Near-perfect SRR (0.984–0.998) throughout 300 steps, but slow monotonic shrinkage accumulation: S rises from 2 (step 30) to 4 (step 90–150) to 8 (steps 240–300). E fluctuates between 0–1 across all steps. NSCR slowly declines from near-zero to −0.014. Even on a benchmark where nearly everything is preserved, RLVR gradually loses access to a small number of correct solutions that the base model could reach.

Minerva: The most dramatic progressive decline. S rises from 11 (step 30) to 16 (step 180) to 15 (step 300). E starts at 10 (step 30) and declines to 4 (step 300). SRR declines from 0.935 to 0.912. NDR declines from 0.059 to 0.025. NSCR becomes increasingly negative: −0.006 → −0.063. SDS drops from 0.111 to 0.049. Pass@256 also declines: 62.13% → 58.46%. This is the paper's clearest case of net-negative impact from extended RLVR training — the model gets worse at high-k evaluation while losing access to correct solutions.

Olympiad: Similar progressive decline to Minerva. S rises from 22 (step 30) to 61 (step 300), nearly tripling. E fluctuates but trends down (13 → 5). SRR drops from 0.963 to 0.897. NSCR drops from −0.015 to −0.093. Pass@256 declines from 86.67% to 79.70%. This is a 7-percentage-point drop in high-k accuracy over 300 training steps, accompanied by the loss of 61 previously accessible correct completions.

AMC23: An outlier — perfect preservation (SRR = 1.000, S = 0–1, E = 0) throughout all 300 steps, with pass@256 stable at 97.50%. This is the benchmark where the base model is already near-saturated (97.50% pass@256), leaving no room for measurable shrinkage or expansion. It serves as a control: when the base model already accesses nearly all correct solutions, RLVR has nothing to lose or gain, and support dynamics are flat.

The temporal analysis reveals that RLVR "reshapes rather than monotonically improves the reasoning distribution. It reinforces a narrow set of stable trajectories over time, which explains both the early-stage gains and the late-stage degradation observed on more diverse math tasks" (Section 4.2). The transient expansions on AIME25 and the progressive shrinkage on Minerva and Olympiad are particularly important because they would be invisible in a single before/after snapshot — a researcher comparing only step 0 and step 300 might conclude expansion never occurs, when in fact it occurs and is then reversed.

Pass@k Crossover and Empirical Support in Reasoning Gym

Figures 2–4 provide the most visually compelling evidence for the support preservation/shrinkage phenomenon, showing pass@k curves for specific Reasoning Gym tasks where the base model and RLVR model are compared across sampling budgets spanning several orders of magnitude.

Support preservation cases (Figure 2): In graph coloring, palindrome generation, and advanced geometry tasks, the RLVR model (ProRL) achieves much higher pass@k at low k — often reaching near-perfect accuracy at k = 8–64 where the base model is still far below ceiling — but the base model eventually catches up or surpasses at very large k. For example:

Graph coloring (prime_factorization): Base pass@1 = 0.140, ProRL pass@1 = 1.000. Base pass@1024 = 1.000, ProRL pass@1024 = 1.000. Both eventually saturate, but ProRL gets there with vastly fewer samples. This is the best case for RLVR: it accelerates convergence to solutions the base model could eventually find.
Graph coloring hard: Base pass@1 = 0.120, ProRL pass@1 = 0.960. But at k = 2048, base pass@k = 1.000, ProRL pass@k = 1.000. The base model requires ~200× more samples to achieve the same ceiling.
Palindrome generation: Base pass@1 = 0.154, ProRL pass@1 = 0.923. Base pass@512 = 1.000, ProRL pass@512 = 1.000. Similar pattern — ProRL dramatically accelerates convergence.
Advanced geometry: Base pass@1 = 0.611, ProRL pass@1 = 0.865. Base pass@2048 = 1.000, ProRL pass@2048 = 1.000. The gap is narrower because the base model already had decent pass@1.

Support shrinkage cases (Figure 4): In leg counting, family relationships, and power function tasks, the ProRL model's pass@k curves plateau at levels below the base model's eventual ceiling:

Leg counting: Base pass@k rises steadily from 0.040 at k=1 to 0.400 at k=1024, showing a rising curve. ProRL pass@k is flat at 0.280–0.300 across all k from 1 to 1024. The base model continues improving with more samples; ProRL does not. This is direct evidence of support shrinkage — the base model can access correct solutions at high k that ProRL cannot access at any k.
Family relationships: Base pass@k rises from 0.140 to 1.000 at k=1024. ProRL pass@k starts higher (0.840 at k=1) but plateaus at 0.860–0.880 from k=4 onward, never reaching 1.000. The base model finds the remaining ~14% of solutions at higher k; ProRL's narrowed distribution excludes them entirely.
Power function: Base pass@k rises from 0.465 to 1.000 at k=1024. ProRL starts at 0.624 but plateaus at 0.901 from k=8 onward. The base model eventually finds all correct solutions; ProRL caps at ~90%.

These cases vividly illustrate the precision-diversity trade-off: ProRL dramatically improves low-k performance (making it appear "better" in standard evaluations) but at the cost of losing access to a subset of correct solutions that the base model's broader distribution could eventually reach.

Support expansion cases (Figure 3): In boxnet, dice, arc 1d, and graph color vertex20, ProRL shows genuine expansion:

Boxnet: Base pass@1 = 0.010, ProRL pass@1 = 0.040. But the key pattern: base pass@k rises gradually to 0.570 at k=2048, while ProRL pass@k rises to 0.680 — ProRL's ceiling is higher. This is the signature of expansion: the base model cannot eventually catch up, even at very large k.
Dice: Base pass@1 = 0.000, ProRL pass@1 = 0.180. Base pass@k rises to 0.540 at k=8192, ProRL rises to 1.000 at k=4096 and saturates. ProRL's ceiling (1.000) exceeds the base model's ceiling (0.540), indicating that ProRL can access correct solutions the base model cannot reach at any budget.
Arc 1d: Base pass@1 = 0.024, ProRL pass@1 = 0.097. Base pass@k rises to 0.345 at k=16384, ProRL rises to 0.492. A clear but modest ceiling improvement — ProRL can reach ~49% of solutions vs. the base model's ~34%.
Graph color vertex20: Base pass@1 = 0.000, pass@k rises slowly to 0.406 at k=4096. ProRL pass@k jumps from 0.762 at k=64 to 1.000 at k=256. Dramatic acceleration combined with a higher ceiling (1.000 vs. ~0.406). This is an expansion case where the base model had the components but couldn't assemble them, and RLVR bridged the assembly gap.

The paper emphasizes that these expansion cases share a common structural property: the base model's pass@k curve rises steadily with k (indicating that solution fragments are present in the base distribution's long tail), and the tasks are compositional (graph coloring = local vertex assignments, boxnet = JSON key-value manipulations, arc 1d = grid cell updates, dice = fractional probability computation). This supports the "subskill recombination" mechanism — RLVR amplifies existing components and learns to assemble them, rather than creating new reasoning capabilities ex nihilo.

Perplexity Analysis: Structural Narrowing of Solution Trajectories

Table 2 reports perplexity where base and ProRL models are evaluated against external reasoning traces from DeepSeek-R1 and Claude Sonnet 4, segmented by correctness patterns and reference types.

Shrinkage cases (correct under base, incorrect under ProRL): ProRL consistently shows higher perplexity. On AIME24 against Claude Sonnet 4, base perplexity = 1.70, ProRL perplexity = 2.12 (+24.7%). On AIME25, base = 1.54, ProRL = 1.98 (+28.6%). On Olympiad, base = 1.51, ProRL = 1.83 (+21.2%). Against DeepSeek-R1, the gaps are smaller but consistent: AIME24 base = 1.24, ProRL = 1.39 (+12.1%); AIME25 base = 1.39, ProRL = 1.70 (+22.3%). These increases indicate that ProRL has concentrated probability mass onto its own learned reasoning style, reducing its ability to assign high probability to alternative correct reasoning paths produced by other models. This is direct evidence that RLVR is not just dropping probability on incorrect completions — it is also reducing probability on correct-but-different completions.

Expansion cases (incorrect under base, correct under ProRL): The pattern reverses — ProRL shows slightly lower perplexity. On Olympiad against DeepSeek-R1, base = 1.41, ProRL = 1.28 (−9.2%). Against Claude Sonnet 4, base = 1.65, ProRL = 1.38 (−16.4%). The "new" correct completions that ProRL can access are more compatible with the external reference's reasoning style than the completions the base model could access — consistent with the format-correction mechanism, where ProRL learns to produce answers in a format and style that matches the expected norms (which external models also follow).

Cases where both models fail: ProRL shows uniformly higher perplexity, with the largest gaps against Claude Sonnet 4: AIME24 base = 8.76, ProRL = 14.91 (+70.2%); AIME25 base = 6.05, ProRL = 9.76 (+61.3%); Olympiad base = 5.98, ProRL = 9.55 (+59.7%). These large gaps suggest that RLVR concentrates probability so aggressively around its own (incorrect) preferred trajectories that it loses the ability to even recognize correct reasoning when presented with it from an external source — the broader distributional compatibility of the base model is lost.

The paper interprets these results as showing that "RLVR concentrates probability around narrower solution trajectories" and that "the dominant effect is structural: RLVR concentrates probability around narrower solution trajectories" (Section 4.2). The lower perplexity for ProRL in expansion cases (against external references) further supports that those expansions involve learning to match expected formats and reasoning styles — not discovering genuinely novel solution approaches.

Entropy Analysis: Token-Level vs. Answer-Level Divergence

Table 5 reports avg@32 accuracy, response length, token-level entropy, and answer-level entropy for nine base-model/RLVR pairs across five math benchmarks.

Accuracy: RLVR consistently improves avg@32 across all models. DeepSeek-1.5B → ProRL-1.5B: 54.5% → 65.4% (+10.9 percentage points overall average). DeepSeek-7B → AceReason-7B: 69.2% → 75.2%. DeepSeek-7B → Skywork-OR1-7B: 69.2% → 74.7%. DeepSeek-14B → AceReason-14B: 75.4% → 79.6%. Qwen2.5-32B → DAPO-32B: 43.0% → 61.3%. These are substantial precision gains at k=32 — exactly what RLVR is designed to achieve.

Response length: RLVR consistently reduces response length. DeepSeek-1.5B average length = 10,422 tokens → ProRL-1.5B = 6,479 (−37.8%). DeepSeek-7B = 7,745 → AceReason-7B = 6,995 (−9.7%), Skywork-OR1-7B = 10,096 (+30.3% — an exception, possibly reflecting the specific training recipe). Qwen2.5-32B = 1,426 → DAPO-32B = 4,989 (+250% — the base 32B model produced very short responses, and DAPO training dramatically increased reasoning chain length). The general trend toward shorter responses (with DAPO-32B as a notable exception) is consistent with the concentration interpretation: RLVR learns to produce more efficient reasoning chains that reach the answer faster, pruning unnecessary exploration.

Token-level entropy: This is where models diverge. DeepSeek-1.5B → ProRL-1.5B: 0.44 → 0.52 (▲ increase). Qwen2.5-32B → DAPO-32B: 0.18 → 0.29 (▲ increase). But DeepSeek-7B → AceReason-7B: 0.37 → 0.23 (▼ decrease); → Skywork-OR1-7B: 0.37 → 0.16 (▼ decrease). DeepSeek-14B → AceReason-14B: 0.33 → 0.14 (▼ decrease). Some models (ProRL, DAPO) show increased per-step uncertainty, while others (AceReason, Skywork) show sharp decreases.

Answer-level entropy: Every single model shows a decrease, and the decreases are substantial. DeepSeek-1.5B → ProRL-1.5B: 1.30 → 0.66 (−49.2%). DeepSeek-7B → AceReason-7B: 0.75 → 0.50 (−33.3%); → Skywork-OR1-7B: 0.75 → 0.54 (−28.0%). DeepSeek-14B → AceReason-14B: 0.54 → 0.38 (−29.6%). Qwen2.5-32B → DAPO-32B: 1.61 → 0.61 (−62.1%). This is the universal finding: regardless of what happens to token-level entropy, answer-level entropy consistently and substantially decreases.

The decoupling phenomenon: ProRL-1.5B increases token-level entropy (+18.2%: 0.44 → 0.52) while nearly halving answer-level entropy (−49.2%: 1.30 → 0.66). DAPO-32B increases token-level entropy (+61.1%: 0.18 → 0.29) while cutting answer-level entropy by nearly two-thirds (−62.1%: 1.61 → 0.61). These are the two models where token-level and answer-level entropy move in opposite directions — what the paper calls "local stochasticity without global exploration." The model appears more uncertain at each generation step (perhaps due to longer reasoning chains with more decision points) but ultimately collapses onto a much narrower set of final answers.

The consistent-decrease case: AceReason-14B decreases both token-level entropy (−57.6%: 0.33 → 0.14) and answer-level entropy (−29.6%: 0.54 → 0.38). This is the more intuitive pattern — the model becomes more deterministic both locally and globally. But the paper's key point is that even when token-level entropy increases, answer-level entropy still decreases — the local uncertainty is misleading.

Benchmark-level variation: Harder tasks (AIME24, Minerva) show higher absolute answer-level entropy for both base and RLVR models across all model pairs. For DeepSeek-14B → AceReason-14B: AIME24 answer entropy drops from 1.01 to 0.66, AMC23 drops from 0.14 to 0.06, MATH500 drops from 0.13 to 0.07. Harder problems inherently foster broader solution spaces (more valid approaches), but RLVR narrows them regardless. The narrowing is proportionally similar across difficulty levels — it's a systematic effect, not one that differentially affects easy vs. hard problems.

Cross-model comparison at same scale: AceReason-7B and Skywork-OR1-7B are both trained from DeepSeek-7B but with different RLVR recipes. AceReason-7B has lower token-level entropy (0.23 vs. 0.16 for Skywork) but higher answer-level entropy (0.50 vs. 0.54 for Skywork). The relationship between token-level and answer-level entropy is not even monotonic across training recipes — lower per-step stochasticity does not necessarily mean lower output diversity. This reinforces the paper's warning that "token-level entropy should not be conflated with genuine exploratory behavior" (Section 5.2).

FLOPs-Matched Comparison: SFT vs. RLVR Objective

Table 3 (and the detailed pass@k curves in Table 14, Appendix) provides the most causally clean comparison in the paper. On Qwen2.5-Math-7B with identical data, sampling protocol, and optimization hyperparameters:

AIME24 (Table 14): DAPO achieves higher pass@k at middle budgets (pass@8: 40.0% vs. 33.3% for SFT; pass@16: 46.7% vs. 43.3%) but SFT catches up at large budgets (pass@256: SFT = 63.33%, DAPO = 66.67%). The support dynamics tell the full story: SFT generates 4 expansions with 4 losses (balanced, NSCR = 0.000); DAPO generates 3 expansions with 2 losses (NSCR = +0.045, one of the rare DAPO-positive cases).

AIME25: SFT dominates at large k: pass@256 = 63.33% vs. DAPO = 56.67%. Support counts: SFT P=14, E=5, S=1 (NSCR = +0.200); DAPO P=15, E=2, S=0 (NSCR = +0.118). SFT discovers 5 new solutions while losing only 1; DAPO discovers 2 while losing none. SFT's NSCR is nearly double DAPO's.

AMC23: SFT maintains perfect support: P=40, E=0, S=0, pass@256 = 100.0%. DAPO loses ground: P=37, E=0, S=3, pass@256 = 92.5%. This is the clearest case of pure shrinkage with zero expansion — DAPO simply loses access to 3 correct solutions that both the base model and the SFT variant could reach.

MATH500: SFT pass@256 = 99.0%, DAPO pass@256 = 95.2%. SFT: P=480, E=15, S=2 (NSCR = +0.026). DAPO: P=470, E=6, S=12 (NSCR = −0.012). SFT discovers 15 new solutions while losing 2; DAPO discovers 6 while losing 12 — DAPO's losses are double its gains.

Minerva: The largest gap. SFT pass@256 = 66.91%, DAPO pass@256 = 55.15%. SFT: P=154, E=28, S=18 (NSCR = +0.050). DAPO: P=138, E=12, S=34 (NSCR = −0.120). DAPO loses nearly three times as many solutions as it gains; SFT gains 1.5× as many as it loses.

Olympiad: SFT pass@256 = 81.78%, DAPO pass@256 = 72.89%. SFT: P=503, E=49, S=25 (NSCR = +0.042). DAPO: P=467, E=25, S=61 (NSCR = −0.065). DAPO loses 61 previously accessible correct completions — more than double its expansions. This is a 9-percentage-point pass@256 deficit relative to SFT.

The paper interprets these results as direct evidence that "support-constrained behavior is not an artifact of scaling, data mixture, or procedure, but emerges from the objective itself" (Section 4.2). SFT's cross-entropy objective naturally preserves probability mass from the base model while adding mass on observed correct completions, producing net expansion. DAPO's reward-maximizing objective explicitly redistributes mass, concentrating it on a subset of high-reward completions at the expense of others, producing net shrinkage. This is the empirical complement to the theoretical proofs in Appendix C — the formal support-preservation proofs predict exactly this pattern, and the controlled experiment delivers it.

Ablation Studies and Robustness Checks

Objective function (SFT vs. DAPO): Fixing the base model (Qwen2.5-Math-7B), dataset (DeepMath-103K), sampling protocol, and optimization hyperparameters while varying only the training objective demonstrates that support-constrained behavior is objective-driven. SFT produces moderate support expansion with positive NSCR values across all benchmarks; DAPO produces sharply concentrated distributions with mixed or negative NSCR. On MATH500, SFT NSCR = +0.026 vs. DAPO NSCR = −0.012. On Minerva, SFT NSCR = +0.050 vs. DAPO NSCR = −0.120. On Olympiad, SFT NSCR = +0.042 vs. DAPO NSCR = −0.065. Table 3 and Table 14. This is the paper's most causally rigorous ablation.

Model scale (1.5B vs. 7B vs. 14B): The support-preservation pattern holds across all scales, with SRR consistently above 0.93 and NDR consistently below 0.04 (Table 1). Larger models show higher SRR (0.99 for 14B models vs. 0.93–0.94 for 1.5B models) — the "leash gets longer" with scale because the base model's broader support means more correct solutions to preserve — but NDR remains near zero even at 14B. Scaling model size within the current RLVR paradigm does not break the leash.

RLVR algorithm variant (ProRL vs. DAPO vs. AceReason vs. Skywork vs. Nemotron vs. Phi4): The support dynamics pattern is consistent across all RLVR algorithm variants evaluated, spanning GRPO-based (ProRL), DAPO, and the undisclosed algorithms behind the other models (Table 1). The SFT vs. DAPO controlled comparison provides the strongest evidence that the objective class (reward-maximizing RL vs. likelihood-based SFT) rather than the specific algorithm variant or implementation determines the behavior.

Base model family (DeepSeek-R1-Distill-Qwen vs. OLMo-2 vs. Qwen2.5-Math): The pattern holds across all three base model families tested. OLMo-2-0425-1B (Table 12) shows SRR = 0.88, NDR = 0.10, NSCR = −0.02 — higher NDR than the DeepSeek-based models but still net shrinkage. Qwen2.5-Math-7B (Table 3) shows the same objective-dependent pattern: SFT expands, DAPO shrinks.

Domain (math vs. non-math vs. visual reasoning): Math domains show SRR = 0.96–0.99 and NDR ≈ 0.00–0.01; non-math shows slightly higher NDR (up to 0.04 for ProRL-1.5B-v2) and similar SRR (0.90–0.99). Visual reasoning (Kangheng-OVR-7B, Table 11) shows the same pattern: SRR ≈ 1.00, NDR ≈ 0.00, NSCR ≈ 0.00. The domain-consistency supports the claim that the support constraint is a general property of RLVR rather than a math-specific or text-specific quirk.

Training duration (temporal trajectory): Table 4 tracks support metrics at 30-step intervals from step 0 to step 300, revealing that support shrinkage is not monotonic — there are transient expansions (AIME25 at step 60: NDR = 0.136) that are later pruned — but the overall trend is progressive narrowing. Pass@256 on Olympiad declines from 86.67% to 79.70% over 300 steps, accompanied by S rising from 22 to 61. This demonstrates that extended RLVR training can be net-harmful for support coverage even as it maintains or slightly improves pass@1.

Perplexity against external references: Table 2. In shrinkage cases, ProRL shows higher perplexity against both DeepSeek-R1 and Claude Sonnet 4 references, confirming that RLVR concentrates probability onto narrower solution trajectories and reduces compatibility with diverse reasoning styles. In expansion cases, ProRL shows modestly lower perplexity — consistent with learning format/stylistic norms rather than novel reasoning. The 70.2% perplexity increase on AIME24 against Claude Sonnet 4 in cases where both models fail (8.76 → 14.91) is the strongest single-number evidence for structural narrowing.

Answer extraction fairness: Appendix B.3 describes enhanced answer processing specifically designed to prevent format incompatibility from biasing the support classification — the base model often produces correct answers in non-standard formats (LATEX where plain text is expected, a/b fractions where \frac is expected, scientific notation variants), and without the enhanced extraction, these would be misclassified as "incorrect," artificially inflating the appearance of expansion (RLVR learning to format correctly would look like discovering new solutions). The paper's extraction protocol processes both models' outputs identically, ensuring that the shrinkage and expansion counts reflect genuine differences in solution accessibility rather than parsing artifacts.

Token-level vs. answer-level entropy: Table 5. Across all nine model pairs, answer-level entropy consistently decreases (range: −28% to −62%), while token-level entropy shows mixed behavior (ProRL: +18%, DAPO: +61%, AceReason-7B: −38%, Skywork-OR1-7B: −57%). This decoupling — token-level entropy can increase while answer-level entropy decreases — is a robustness check on the claim that entropy reduction is fundamental: even when the model appears more stochastic at the step level, it converges to fewer final answers.

Aggregation level for support definition: The paper defines empirical support over answer-level correct completions, treating multiple reasoning traces yielding the same verified outcome as a single equivalence class. This prevents overcounting of semantically identical outputs and focuses the analysis on answer diversity rather than surface-form variation. The rationale is that support is about which answers the model can reach, not how many different ways it can express the same answer.

Negative result — ReST^EM training: Mentioned in the prior work summary but not a direct experiment in this paper: the ReST^EM-trained revision model from the prior study showed degraded performance with sequential revisions (Appendix K, Figure 16 of the prior paper), suggesting that on-policy data collection amplified spurious correlations. This negative result contextualizes the fragility of RLVR training — not all RL variants produce the same support dynamics, and some can be counterproductive.

Critical Assessment

The paper makes several central claims, each with varying degrees of empirical support from the experiments reported. I examine them individually.

1. "RLVR predominantly preserves rather than expands the base model's solution coverage."

This claim is well-supported by the experiments presented. Table 1 shows SRR ≥ 0.93 for all model-domain pairs, with most above 0.96. The support counts directly demonstrate that the number of preserved completions (P) dwarfs both expansions (E) and shrinkage (S). For ProRL-1.5B-v2, P = 2,388 vs. E = 48 — preserved completions exceed expansions by a factor of ~50×.

However, the claim depends on the sampling budget k defining the empirical support threshold. At k = 8,192 for math, the threshold is ϵ ≈ 3.66 × 10⁻⁴. If the RLVR model were discovering correct completions that the base model assigned probability between, say, 10⁻⁵ and 3.66 × 10⁻⁴, these completions would be classified as "expansion" under the paper's definitions (base below threshold, RLVR above threshold) but would represent amplification of existing probability mass rather than genuine creation of new mass. The paper acknowledges this in the discussion of Theorem C.3 — the finite-sample bound allows for amplification of very-low-probability completions — but the theoretical analysis suggests that such amplification is sharply bounded by the exp(2ηTK) factor. The empirical question is whether the observed expansions (E = 48 for ProRL-1.5B-v2) fall into this "amplified long-tail" category or represent something else. The perplexity analysis (Table 2) provides evidence that expansion cases involve format/stylistic correction rather than novel reasoning, but this evidence is limited to the few benchmarks where perplexity was evaluated (AIME, Olympiad) and relies on external references (DeepSeek-R1, Claude) whose own reasoning styles may not be representative of the base model's options.

A stronger test would be: for each expansion case, trace back what the base model assigned probability to the same answer (perhaps in a different format or with different intermediate steps) and determine whether the expansion represents genuinely new reasoning or merely reformatted existing reasoning. The paper's enhanced answer extraction protocol (Appendix B.3) partially addresses this by ensuring both models are evaluated on equal format-parsing footing, but it doesn't close the loop — a base model that produces the correct answer in LATEX format that gets parsed correctly by the enhanced extractor still might show lower raw probability on that LATEX-formatted completion than the RLVR model shows on the plain-text completion, making it appear as expansion when the underlying reasoning is identical. The paper is aware of this limitation (the format-correction mechanism is explicitly described as one of the two expansion types), but the support counts in Table 1 do not distinguish between "genuine reasoning expansion" and "format-correction expansion" — all expansions are counted equally.

2. "The precision-diversity trade-off is fundamental, not domain-specific."

This claim is strongly supported by the breadth of evaluation. The same support-dynamics pattern (high SRR, near-zero NDR, negative NSCR) appears across math (AIME, MATH500, Minerva, Olympiad, AMC), non-math (SimpleQA, LiveBench, SciBench, LiveCodeBench, Reasoning Gym), and visual (MathVista, MathVision) domains (Tables 1, 6–12). The domain coverage is genuinely impressive for a paper of this type — most RLVR analyses focus on 1–2 benchmarks.

A limitation is that the non-math benchmarks use lower sampling budgets (k = 1,024–2,048) than the math benchmarks (k = 4,096–8,192), meaning the empirical support thresholds are higher (ϵ ≈ 2.93 × 10⁻³ at k=1,024 vs. 3.66 × 10⁻⁴ at k=8,192). This means completions with base model probability between ~4 × 10⁻⁴ and ~3 × 10⁻³ would be classified as "below threshold" for non-math but "above threshold" for math — potentially making expansion appear rarer in non-math domains than it actually is. The paper acknowledges this limitation (Section 4.1: "ensuring that any unreachable solution y* ∈ C remains below the empirical support threshold of the base model"), but a reader should note that the non-math results are measured on a coarser scale.

Another domain limitation: all tasks used have verifiable, binary reward functions. This is necessary for the RLVR paradigm and for computing the correctness set C, but it means the findings don't speak to domains where correctness is ambiguous, multi-dimensional, or subjective (dialogue, creative writing, open-ended generation). The paper doesn't claim to cover such domains, but the word "fundamental" in the claim should be understood as "fundamental across the domains tested," not "universal across all possible tasks."

3. "Local uncertainty and global diversity can diverge — RLVR sometimes increases token-level entropy while reducing answer-level entropy."

This is a well-supported empirical finding with specific, compelling evidence. Table 5 shows ProRL-1.5B increasing token-level entropy from 0.44 to 0.52 (+18.2%) while answer-level entropy drops from 1.30 to 0.66 (−49.2%). DAPO-32B shows an even more dramatic decoupling: token entropy +61.1%, answer entropy −62.1%. These are large effects measured across all five math benchmarks, not just selected examples.

However, the claim that this "sometimes" happens is carefully hedged: AceReason-7B and Skywork-OR1-7B show decreases in both token-level and answer-level entropy, with no decoupling. The paper doesn't establish when or why the decoupling occurs vs. when both entropies decrease together — it documents the phenomenon and warns against using token-level entropy as a proxy for exploration, but doesn't provide a predictive theory of which models or training recipes will exhibit decoupling. This is a descriptive finding, not an explanatory one, and the paper appropriately presents it as such.

A technical concern: token-level entropy is computed via teacher forcing — the generated completions are fed back through the model in a forward pass to obtain probability distributions (Appendix B.4). This means the entropy values reflect the model's probability distribution given the specific generated tokens, not the distribution during sampling. The distinction matters because during sampling, the model sees its own generated tokens (which may diverge from the most likely continuations), while during teacher forcing, it sees the exact sequence that was generated. This is standard practice for entropy computation but means the entropy values reflect the model's evaluation of generated sequences rather than its generation-time uncertainty. The decoupling might be partially attributable to this methodological choice, though the consistent answer-level entropy decreases across all models suggest the qualitative pattern is robust.

4. "RLVR cannot discover solutions the base model assigns zero probability to."

This claim is supported by Theorem C.1, which proves that on-policy RLVR gradient updates preserve the mathematical support exactly. The proof is mathematically sound and relies on a straightforward inductive argument: initialization ensures inclusion, and gradient updates from the current policy's samples cannot create probability mass for completions never sampled.

However, the practical significance of this theorem is limited by the softmax property: the base model assigns strictly positive probability to every completion, so the mathematical support is the entire output space, making the constraint vacuous. The paper acknowledges this and provides Theorem C.3 (empirical support bound) as the practically relevant result, but Theorem C.3 has the limitation that it assumes a particular gradient decomposition (PSR + NSR from Zhu et al., 2025) and provides a worst-case amplification bound (exp(2ηTK)) that may not be tight for all RLVR algorithms. Some RLVR variants (e.g., those with off-policy replay buffers, those that mix SFT data with RL updates) might achieve larger amplification factors. The paper's SFT vs. DAPO experiment (Table 3) circumvents this theoretical limitation by providing direct empirical evidence: when the same data is used for SFT (which doesn't have the support constraint) and RLVR, SFT expands while RLVR shrinks. This is stronger evidence than the theory alone because it demonstrates the phenomenon in practice, under realistic training conditions, for a specific model and dataset.

The paper would be strengthened by a third comparator in the objective-function experiment: an RLVR variant that explicitly includes an exploration bonus or off-policy component. If such a variant showed positive NSCR, it would demonstrate that the support constraint is specific to standard on-policy RLVR, not inherent to all reward-maximizing training. Without this comparator, the paper's claim that "breaking the leash requires future innovations that seed probability mass into underrepresented solution regions" is a reasonable inference from the theory and experiments but is not directly tested.

5. "Support shrinkage outweighs expansion across all models and domains."

This is the paper's most quantitative and well-supported claim. Table 1 shows NSCR negative for every model-domain pair (overall NSCR ranging from −0.01 to −0.06). The detailed per-model tables (Tables 6–12) confirm this: shrinkage counts (S) consistently exceed expansion counts (E) for every model on every benchmark where the difference is statistically meaningful. The most extreme case is ProRL-1.5B-v2 overall: 175 losses vs. 48 gains, ratio ~3.6:1. The SFT vs. DAPO comparison confirms that this is objective-specific: SFT shows positive NSCR on 4 of 5 benchmarks while DAPO shows negative or mixed NSCR.

One nuance: the shrinkage-to-expansion ratio depends on the sampling budget. As k increases (and ϵ decreases), some completions currently classified as "Out of Support" (neither model can reach them) might shift to "Expansion" (RLVR can reach them, base cannot) or "Shrinkage" (base can reach them, RLVR cannot). The direction of this shift is not obvious a priori — it depends on the relative shapes of the two models' probability density functions in the low-probability tail. The paper uses large k (8,192 for math) to push ϵ low, but a reader should be aware that the exact shrinkage-to-expansion ratios are threshold-dependent. However, the consistent direction of the imbalance (S > E) across all thresholds tested (from k = 1,024 to k = 16,384) suggests this is a robust qualitative finding.

Missing experiments that would strengthen the paper:

An exploration-incentivized RLVR variant. The paper's theoretical analysis (Theorem C.3) shows that standard on-policy RLVR cannot lift below-threshold completions above threshold. Could an RLVR variant with explicit exploration bonuses (e.g., entropy bonus at the answer level, intrinsic motivation for diverse outputs, count-based exploration in embedding space) produce positive NSCR? This is the most direct test of whether the leash is specifically about current RLVR practice or about reward-maximizing optimization more broadly.
Interleaved SFT + RLVR. The paper shows that SFT expands and RLVR shrinks. What about a training pipeline that alternates between SFT (to introduce new solution candidates) and RLVR (to sharpen and filter)? This is the natural hybrid approach suggested by the paper's own findings, but it is not evaluated.
Difficulty-stratified support dynamics. The paper's companion prior work examined difficulty-dependent behavior extensively. Stratifying the support metrics by difficulty bin would reveal whether shrinkage primarily affects easy problems (where there are many solutions to lose), hard problems (where the base model's support is already narrow), or both. The Reasoning Gym figures (Figures 2–4) provide some task-level resolution, but a systematic difficulty analysis would strengthen the paper's contribution.
Confidence intervals on support metrics. The paper reports point estimates for SRR, NDR, SDS, and NSCR without any measure of uncertainty. Given that the test sets have modest sizes (e.g., AIME 30 questions, MATH500 500 questions, SimpleQA 433 questions), the support counts could have substantial sampling variance. Bootstrap confidence intervals would help readers assess whether differences between models (e.g., ProRL-1.5B-v1 SRR = 0.94 vs. ProRL-1.5B-v2 SRR = 0.93) are statistically meaningful or within noise.
Temperature sensitivity of support dynamics. All experiments use temperature 0.6. Would the support dynamics patterns change at lower temperatures (which would reduce stochasticity and potentially increase the appearance of shrinkage by narrowing the sampled distribution further) or higher temperatures (which would increase exploration and potentially reveal more of the base model's long tail, changing the expansion/shrinkage balance)? The paper's entropy analysis already shows temperature interacts with the entropy measures, but no temperature sweep is performed for the support metrics.
Oracle vs. learned verifier. The paper uses ground-truth verifiers (MATH grading scripts, GPT-4.1 for SimpleQA, etc.) to determine correctness. In practice, RLVR models are trained with verifiers that may have errors (both false positives and false negatives). Would support dynamics look different if the reward function were noisy — could a noisy verifier actually increase support expansion by allowing the model to explore completions that a perfect verifier would incorrectly reject? This is a relevant practical question given the paper's discussion of spurious rewards (citing Shao et al., 2025 in Section 1).

Overall, the experiments provide strong support for the paper's central narrative — that RLVR primarily concentrates probability mass within the base model's existing support rather than expanding it — and the most robust finding is the consistent negativity of NSCR across all models, domains, and scales. The SFT vs. DAPO controlled experiment is the paper's most compelling single piece of evidence. The entropy decoupling is a well-documented novel phenomenon with practical implications for training monitoring. The theoretical proofs provide a mechanistic explanation that complements the empirical measurements. The main limitations are the threshold-dependence of the support metrics (acknowledged), the absence of exploration-incentivized RLVR baselines (a missing experiment that would directly test the paper's prescriptive claims), and the lack of confidence intervals on the support counts (which makes it difficult to assess the statistical reliability of the precise SRR/NDR/NSCR values, even though the qualitative pattern of S > E is clear from the raw counts).

6. Limitations and Trade-offs

The Empirical Support Framework Depends on a Sampling Threshold That Is Both Computationally Expensive and Methodologically Loaded

The assumption or constraint. The entire support-dynamics analysis — the classification into Preservation, Expansion, Shrinkage, and Out-of-Support — rests on the choice of the empirical support threshold $\epsilon$ , which is derived from the sampling budget $k$ via a confidence-bound argument (Appendix C.4). The paper acknowledges the cost directly but does not fully absorb its implications:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

While this quote originally refers to difficulty estimation in the companion work, the same structural issue applies here: the paper's primary measurement apparatus requires generating 4,096 to 16,384 completions per prompt per model to establish empirical support sets. For the full evaluation across 8 model pairs, 11 benchmarks, and temporal checkpoints, the total generation cost is enormous — the paper never quantifies it, but a back-of-the-envelope estimate (e.g., MATH500 alone: 500 prompts × 8,192 samples × multiple models = tens of millions of generations at 32K max tokens each) suggests the measurement costs might exceed the training costs of the models being analyzed.

The consequence. The paper's framework is a measurement and diagnostic tool, not an efficiency-optimized deployment protocol, and the computational cost of applying it means that it cannot be used as a lightweight training monitor or a cheap evaluation step in an iterative development cycle. A practitioner wanting to check whether their RLVR training run is experiencing support shrinkage would need to invest compute comparable to or exceeding the training run itself just to take a single measurement. This makes the framework impractical for routine use and limits its impact to one-off diagnostic studies rather than continuous integration into training pipelines.

Additionally, the $\epsilon$ threshold varies with $k$ , and different benchmarks use different $k$ values: math benchmarks use $k = 4,096$ or $8,192$ ( $\epsilon \approx 3.66 \times 10^{-4}$ at 95% confidence), while non-math benchmarks use $k = 1,024$ or $2,048$ ( $\epsilon \approx 2.93 \times 10^{-3}$ ). This means completions classified as "Expansion" or "Shrinkage" on non-math tasks operate at a coarser resolution — a completion with base-model probability of $5 \times 10^{-4}$ would be classified as "below threshold" (potentially Expansion if RLVR lifts it) on math tasks but "above threshold" (Preservation if both find it) on non-math tasks. The paper acknowledges this indirectly (Section 4.1: "ensuring that any unreachable solution $y^* \in \mathcal{C}$ remains below the empirical support threshold of the base model"), but does not discuss how the threshold mismatch affects cross-domain comparisons of SRR, NDR, or NSCR. A reader comparing math SRR = 0.96 to non-math SRR = 0.91 (ProRL-1.5B-v1, Table 1) cannot determine how much of the difference is genuine domain variation vs. an artifact of different detection thresholds.

What evidence exists in the paper. The $\epsilon$ derivation is provided in Appendix C.4, the sampling budgets per benchmark are specified in Section 4.1, and the support metrics are reported in Table 1 and Appendix A. The paper does not conduct any sensitivity analysis varying $k$ (and thus $\epsilon$ ) on the same benchmark to assess how the support classification changes — this would directly reveal the threshold-dependence of the P/E/S/O breakdown. The paper does not report the computational cost of the measurement apparatus in FLOPs, GPU-hours, or any other unit that would allow practitioners to assess feasibility.

Mitigation status. Not addressed. The paper suggests future work on "pretraining or finetuning models to directly predict difficulty" in the companion work's Section 8, but this applies to difficulty estimation, not support measurement. No cheaper proxy for empirical support classification is proposed — the paper provides no way to approximate SRR or NDR without the massive sampling budgets used in the evaluation. The framework is presented as-is, with the computational cost acknowledged but not solved.

The Absence of Exploration-Incentivized RLVR Baselines Leaves the "Leash" Claim Partially Untested

The assumption or constraint. The paper's central prescriptive claim is that "breaking this invisible leash requires future innovations that seed probability mass into underrepresented solution regions" (Section 6). This implies that the support-constrained behavior is inherent to the specific RLVR recipes evaluated (on-policy GRPO, DAPO, and the undisclosed algorithms behind Skywork, AceReason, Nemotron, Phi4) and that different training paradigms — those with explicit exploration mechanisms — could circumvent it. However, the paper evaluates no such exploration-incentivized RLVR variant.

The consequence. The paper demonstrates that current popular RLVR recipes exhibit support shrinkage, but it does not demonstrate that support shrinkage is unavoidable for all reward-maximizing training. A critic could reasonably argue that the "invisible leash" is not a fundamental property of RLVR but an artifact of insufficient exploration in the specific implementations tested. For instance, an RLVR variant with an explicit answer-level entropy bonus (penalizing concentration on few outputs), with count-based exploration rewards (incentivizing sampling of rarely-produced correct completions), or with off-policy data mixing (injecting SFT-generated correct completions into the RL training buffer) might show positive NSCR and non-trivial NDR. The paper's theoretical proofs (Theorems C.1, C.3) establish that standard on-policy policy-gradient updates from initialization $q$ are support-bounded, but these proofs assume the gradient update form $\nabla_\theta \mathbb{E}_{y \sim \pi_\theta}[\cdot]$ — adding an exploration term that breaks this expectation (e.g., by sampling from a mixture distribution $\pi_\theta + \text{noise}$ ) would invalidate the inductive step.

The SFT vs. DAPO comparison (Table 3) is the paper's strongest evidence that support-constrained behavior is objective-driven, but SFT is not an RLVR variant — it is a fundamentally different training paradigm (likelihood maximization vs. reward maximization). The missing experiment is: same base model, same data, same RLVR framework (e.g., DAPO), but with an added exploration incentive (e.g., entropy bonus). If this variant also showed negative NSCR, the paper's claim would be substantially strengthened. If it showed positive NSCR, the paper's prescriptive conclusion ("breaking the leash requires seeding probability mass into underrepresented regions") would be empirically validated within the RLVR paradigm rather than only by switching to SFT.

What evidence exists in the paper. The SFT vs. DAPO comparison (Section 4.2, Tables 3 and 14) demonstrates that the objective function matters and that SFT expands while DAPO shrinks. The theoretical analysis (Appendix C, Theorems C.1, C.3, C.4) establishes bounds for on-policy RLVR. The temporal training analysis (Table 4) shows that transient expansions can occur (AIME25 at step 60) before being pruned, hinting that the training dynamics can discover new solutions but cannot sustain them — this is consistent with an exploration-insufficient narrative but does not test whether explicit exploration incentives would change the outcome.

Mitigation status. Not addressed experimentally. The paper acknowledges the limitation implicitly through its future-work suggestion: "RLVR must be paired with explicit exploration strategies or off-policy mechanisms that allocate probability mass to underrepresented regions" (Section 6). No such strategies are evaluated. The theoretical bounds (Theorem C.3) provide a quantitative ceiling on amplification without exploration but do not characterize what happens when exploration is added.

The Paper Provides No Confidence Intervals or Statistical Significance Tests for the Support Metrics, Making Cross-Model and Cross-Domain Comparisons Difficult to Evaluate

The assumption or constraint. All support-dynamics metrics (SRR, NDR, SDS, NSCR) and support counts (P, E, S, O) are reported as point estimates without any measure of uncertainty — no confidence intervals, no standard errors, no bootstrapped bounds, no hypothesis tests for differences between models or domains.

The consequence. The paper draws a number of comparative conclusions that depend on the precision of these point estimates:

"ProRL-1.5B-v2 loses 175 completions while gaining only 48 (ratio ≈3.6:1)" — is this ratio significantly different from 1:1? From the 4.5:1 ratio reported for ProRL-1.5B-v1?
"Non-math datasets exhibit the highest relative discovery (NDR ≤0.04)" — is NDR = 0.04 for non-math statistically distinguishable from NDR = 0.02 for math?
"All 1.5B–14B models achieve very high support retention (overall SRR ≈0.93–0.99)" — is SRR = 0.93 meaningfully different from SRR = 0.99, or is this within sampling noise given the finite test-set sizes?

The small test-set sizes compound this problem. AIME has only 30 questions, AMC has 40, SimpleQA uses 433 after downsampling, and LiveBench subsets are often even smaller (e.g., LiveBench-L has 50 questions). With 30 questions and thousands of samples per question, the support counts P, E, S, O are estimated from a finite number of prompts, and the variance across prompts could be substantial — a single anomalous prompt where the base model produces 10 correct completions only one of which is preserved could materially affect SRR on a 30-question benchmark. The paper's temporal analysis (Table 4) shows SRR fluctuating by 0.02–0.04 between consecutive 30-step checkpoints (e.g., MATH500 SRR oscillates between 0.984 and 0.998), which suggests that some of the variation is noise rather than signal.

What evidence exists in the paper. All support metrics are point estimates without error bars. The per-benchmark breakdown in Appendix A (Tables 6–12) provides the raw counts, but without per-question variances or bootstrap-based confidence intervals. The pass@k curves (Figures 2–4) provide visual evidence that sampling noise is present (curves are not perfectly monotonic), but the support metrics are not visualized with uncertainty.

Mitigation status. Not addressed. The paper makes no mention of statistical uncertainty in the support metrics and provides no mechanism for a reader to assess whether, say, ProRL-1.5B-v2's NSCR = −0.05 is "significantly" different from Nemotron-7B's NSCR = −0.01. The consistency of the qualitative pattern (NSCR negative for all models) is visible from the raw counts, but the precise numeric values — which the paper reports to two or three decimal places — carry a false sense of precision. A practitioner trying to determine whether their own model's NSCR of −0.03 represents a meaningful improvement over a baseline's NSCR of −0.05 has no guidance from this paper on what magnitude of difference is reliable.

The Findings Are Specific to Verifiable Binary-Reward Tasks, Leaving Open-Ended and Subjective-Evaluation Domains Uncharacterized

The assumption or constraint. Every benchmark used in the paper has a binary, verifiable reward function: math problems have unique correct answers, SimpleQA uses GPT-4.1 as a binary judge, LiveBench uses only tasks with "binary correct/incorrect judgments" (Appendix B.2) and explicitly excludes "tasks involving intermediate floating-point judgments" because they "lack clear correctness criteria," SciBench and LiveCodeBench use test-case-based verification, and Reasoning Gym tasks all have ground-truth answers that can be checked algorithmically. The paper's own definition of the correctness set $\mathcal{C}$ (Definition 3.1) and the entire support-classification framework assume $R(x, y) \in \{0, 1\}$ .

The paper acknowledges this constraint indirectly — the exclusion of non-binary LiveBench tasks and the focus on "binary correct/incorrect judgments" is explicit — but does not discuss how the findings would extend (or fail to extend) to domains where correctness is non-binary, multi-dimensional, or subjective.

The consequence. Many important applications of RLVR — and of language model post-training generally — operate in domains without clean binary rewards:

Dialogue and instruction-following: RLHF is typically used because human preferences are non-binary and multi-attribute (helpfulness, harmlessness, honesty), making a single verifiable reward unavailable.
Creative generation: Story writing, poetry, brainstorming, and code generation with stylistic constraints have multiple valid outputs with no ground-truth "correct answer."
Open-ended reasoning: Scientific hypothesis generation, strategic planning, and essay writing involve outputs that are partially correct, better or worse along multiple dimensions, or evaluable only by expert humans.

In these domains, the concept of "empirical support" as the set of completions with $R(x, y) = 1$ breaks down — there is no binary $R$ . One could attempt to define a threshold on a continuous reward function (e.g., $\mathcal{C}_\tau = \{y \mid R(x, y) \geq \tau\}$ ), but the choice of $\tau$ would be arbitrary and the support-classification framework would inherit that arbitrariness. Moreover, the RLVR objective itself changes: instead of maximizing the probability of hitting a binary reward, the model maximizes expected continuous reward, which may lead to different distributional dynamics (e.g., the optimal policy under continuous rewards may not be the exponentially tilted form of Proposition C.4, which assumes binary $R$ ).

The paper's key insight — that RLVR concentrates probability mass on high-reward outputs at the expense of diversity — may be more consequential in continuous-reward domains where diversity is inherently valuable (e.g., creative generation, where producing multiple stylistically distinct valid outputs is desirable) or less consequential (e.g., if continuous rewards create smoother optimization landscapes that don't collapse as aggressively). The paper provides no evidence either way.

What evidence exists in the paper. None. All experiments use binary-reward tasks. The paper does not discuss continuous-reward settings, does not propose a generalization of empirical support for non-binary rewards, and does not speculate about whether the support-shrinkage pattern would hold, reverse, or disappear in such settings. The related work section mentions RLHF briefly as an alternative paradigm (Section 2: "RLVR has emerged as a scalable alternative to RLHF"), and the impact statement mentions "creative reasoning" as a domain where the precision-diversity trade-off is "especially consequential" (Section 5.2), but no experiments span this gap.

Mitigation status. Not addressed. The paper's scope is explicitly on RLVR with verifiable rewards, and the authors are transparent about this in the title and framing. The limitation is not a flaw in the paper's execution but a constraint on the generalizability of its conclusions. A practitioner working in continuous-reward or subjective-evaluation domains cannot directly apply this paper's framework or assume its findings transfer.

The Measured Support Dynamics Reflect a Static Before/After Snapshot (Except for One Model's Temporal Analysis), and the Paper Does Not Characterize How Training Hyperparameters Modulate the Shrinkage-Expansion Balance

The assumption or constraint. The paper's primary results (Table 1, the per-model tables in Appendix A) compare a single base-model checkpoint against a single RLVR checkpoint — a static before/after measurement. The temporal analysis in Table 4 (DAPO training of DeepSeek-1.5B) provides the only dynamic view, tracking support metrics at 30-step intervals for one specific model, one specific RLVR algorithm, and one specific training configuration.

The paper does not vary training hyperparameters to assess their impact on support dynamics. No experiments sweep the KL penalty coefficient $\beta$ (Proposition C.4), the learning rate $\eta$ (which appears in the Theorem C.3 amplification bound as $\exp(2\eta T K)$ ), the number of training steps (beyond the single temporal run), the temperature used during training rollouts, the batch size, or the mix of prompts in the training distribution.

The consequence. The paper's prescriptive claim — that "breaking the leash requires explicit exploration strategies" (Section 6) — implicitly assumes that hyperparameter tuning within the existing RLVR paradigm cannot mitigate the support-shrinkage problem. But the paper's own theory suggests potential levers:

KL penalty coefficient $\beta$ (Proposition C.4): Smaller $\beta$ (stronger KL regularization) keeps the policy closer to the base model, which should increase SRR (preserve more base-model solutions) but potentially reduce pass@1 gains (less concentration on high-reward outputs). Larger $\beta$ (weaker KL, approaching the KL-free limit of Corollary C.5) should sharpen the distribution more aggressively, potentially increasing both pass@1 gains and shrinkage. The paper does not test this trade-off — all RLVR models evaluated use some $\beta$ setting, but the specific values are not reported or compared.
Learning rate $\eta$ and training duration (Theorem C.3): The amplification bound $\pi^{(K)}(y^*) \leq \epsilon \cdot \exp(2\eta T K)$ suggests that larger $\eta$ or longer training ( $K$ ) could amplify low-probability completions more, potentially converting some "below-threshold" completions into "above-threshold" expansions. But larger $\eta$ also accelerates shrinkage (mass leaves non-rewarded completions faster). The single temporal analysis (Table 4) shows that on Minerva and Olympiad, longer training increases shrinkage — but this is at one fixed $\eta$ setting. A sweep of $\eta \times K$ combinations could reveal whether there exists a "sweet spot" where expansion transiently exceeds shrinkage, or whether the ratio E/S is monotonic in training compute.
Sampling temperature during training rollouts: All training was presumably done with some temperature >0 (the paper uses temperature 0.6 for evaluation, but training temperature is not reported). Higher training temperature would increase the diversity of sampled rollouts, potentially exposing the model to more correct completions from the base model's long tail and providing gradient signal for those completions. This could increase E (by surfacing more long-tail completions for PSR updates). Conversely, higher temperature also increases noise in the gradient estimates and might destabilize training. The paper does not explore this.

Without hyperparameter sweeps, the paper cannot distinguish between "support shrinkage is an inevitable consequence of any practical RLVR training" and "support shrinkage is a consequence of the specific hyperparameter configurations used by these open-source models." A practitioner reading the paper might reasonably ask: "If I use a smaller learning rate, stronger KL regularization, and stop training earlier, can I avoid the shrinkage problem while still getting meaningful pass@1 improvements?" The paper provides no evidence to answer this question.

What evidence exists in the paper. Table 4 (temporal analysis) provides one training duration sweep for one model, showing that longer training generally increases shrinkage. Table 1 provides across-model comparisons where different models use different RLVR recipes (and thus different hyperparameters), but the hyperparameters are not reported, so the cross-model variation in SRR/NDR/NSCR cannot be attributed to specific hyperparameter choices vs. differences in base model quality, training data, or algorithm design. The SFT vs. DAPO comparison (Table 3) shows that the objective function matters, but does not test whether different DAPO hyperparameters would produce different outcomes.

Mitigation status. Not addressed. The paper does not report training hyperparameters for the evaluated models (except to note that ProRL uses "GRPO enhanced with decoupled clipping, dynamic sampling, KL divergence regularization, and periodic reference resets" in Section 4.1 — a qualitative description without quantitative values). The paper does not include hyperparameter sensitivity analysis as a limitation or suggest it as future work. The temporal analysis (Table 4) partially addresses the training-duration dimension but for only one model, one algorithm, and one hyperparameter configuration.

The Paper Does Not Account for Latency or Wall-Clock Constraints, Limiting the Practical Applicability of Its Recommendations for Deployment

The assumption or constraint. The paper's entire analysis — support dynamics, pass@k curves, entropy metrics — treats number of generations $k$ as the sole unit of compute, implicitly assuming that all $k$ samples can be generated with equal cost and, critically, that the only relevant cost is total FLOPs rather than wall-clock time. The paper's discussion of the precision-diversity trade-off (Section 5.2) and its recommendations for evaluation practice (Section 6) assume this FLOPs-centric view.

The paper does not discuss latency — the time required to produce a single answer — or the distinction between parallel sampling ( $k$ completions generated simultaneously across $k$ hardware units) and sequential sampling ( $k$ completions generated one after another on a single unit).

The consequence. The paper's key empirical pattern — base model outperforms RLVR model at large $k$ — has very different practical implications depending on the deployment scenario:

Batch inference with abundant parallelism: If you can run $k = 8,192$ completions in parallel across many GPUs, the total wall-clock time is roughly the latency of a single generation (plus minor overhead), and the base model's pass@k advantage at large $k$ is directly actionable — you should use the base model for high-k batch evaluation.
Latency-constrained interactive deployment: If you must produce an answer within, say, 2 seconds, you cannot generate 8,192 completions regardless of your total FLOPs budget. In this regime, the RLVR model's dramatically better pass@1 (e.g., ProRL pass@1 = 0.96 on graph coloring vs. base pass@1 = 0.12; Figure 2) is overwhelmingly more important than the base model's eventual pass@8192 = 1.0. The "invisible leash" is irrelevant if you're constrained to k=1 by latency requirements.
Moderate parallelism: If you can afford, say, k = 16 completions in parallel, the relevant comparison is RLVR pass@16 vs. base pass@16 — and in many of the paper's examples (Figure 2), RLVR dominates at this budget. The crossover point where base model surpasses RLVR varies by task and is typically at much larger $k$ (e.g., k > 64 in palindrome generation, k > 256 in advanced geometry).

The paper's framing — that support shrinkage is a problem because the base model "often perform[s] better with multiple attempts" (Section 1, quoted from Shao et al., 2025) — is true for sufficiently large k but potentially misleading for practitioners who cannot afford that k in their latency budget. The paper never specifies at what k the crossover occurs per task, instead showing pass@k curves (Figures 2–4) that allow a reader to assess this visually but without explicit discussion of the latency dimension.

Additionally, the sequential/parallel distinction matters for the entropy analysis (Section 5). The paper computes answer-level entropy from $k = 32$ completions generated in parallel. If those 32 completions were instead generated sequentially with access to previous completions (as in the revision model setting of the companion work), the answer-level entropy might be different — the model could condition its diversity on what it has already produced. The paper's entropy measurements assume independent parallel sampling, which is one specific deployment mode.

What evidence exists in the paper. The pass@k curves (Figures 2–4) show accuracy as a function of k, which implicitly captures the accuracy-latency trade-off (larger k = more latency under fixed parallelism). The paper reports avg@32 accuracy (Table 5) as a single-point metric. The paper does not discuss latency, throughput, or parallelism assumptions anywhere.

Mitigation status. Not addressed. The paper's recommendations — "the base model's broader support yields higher pass@k at large k" (Section 4.2), "the observed divergence between token- and answer-level entropy highlights the need for a more nuanced interpretation of stochasticity" (Section 5.2) — are framed without reference to deployment constraints. A practitioner would need to overlay their own latency requirements onto the paper's pass@k curves to determine whether the shrinkage phenomenon actually matters for their use case. The paper could have strengthened its practical relevance by reporting the crossover $k$ (the smallest $k$ at which base pass@k exceeds RLVR pass@k) for each task, or by discussing the latency regimes under which RLVR shrinkage is vs. is not a practical concern.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not propose a new training algorithm or a better model — it introduces a measurement and diagnostic framework that changes what counts as evidence in debates about RLVR's capabilities. Before this work, the central question "Does RLVR expand reasoning or sharpen existing capabilities?" was contested through pass@k comparisons and qualitative argument, with both sides able to point to selected benchmarks and models supporting their view. After this work, the question is operationalized: you count completions, you set a threshold derived from sampling confidence bounds, you classify into Preservation/Expansion/Shrinkage/Out-of-Support, and you compute SRR, NDR, SDS, and NSCR. The debate shifts from "we disagree about what RLVR does" to "we disagree about what the support metrics mean or whether they capture what we care about," which is a more tractable and scientifically productive disagreement.

This is not a paradigm shift — the paper doesn't overthrow the RLVR paradigm or demonstrate that it fails. It is a diagnostic reframing with immediate practical consequences: any future paper claiming that a new training method "expands reasoning beyond the base model" must now either (a) demonstrate positive NSCR and nontrivial NDR using the paper's framework, (b) use an alternative measurement methodology and argue why it's superior, or (c) explicitly limit its claims to pass@1 improvement rather than capability expansion. The paper establishes a falsifiable standard for a claim that was previously unmeasurable.

The paper reconciles the central contradiction in prior work — why some studies find RLVR "expands reasoning" (Liu et al., 2025a) while others find it "cannot self-correct" or "amplifies existing behaviors" (Yue et al., 2025a; Zhao et al., 2025; Huang et al., 2023) — through a single, consistent mechanism. The mechanism is not that one side is wrong and the other right. Rather, both observations are manifestations of the same underlying phenomenon viewed at different sampling budgets: RLVR concentrates probability mass onto a subset of the base model's correct completions (improving pass@1 — the "expansion" camp's evidence) while simultaneously dropping probability on other correct completions that the more diverse base model could eventually reach (enabling the base model's pass@k crossover — the "conservative optimization" camp's evidence). The pass@k crossover is not evidence against pass@1 improvement; it is the cost of pass@1 improvement, measurable via NSCR < 0. The paper's unified account transforms a contradiction into a trade-off.

Research directions that become more attractive as a result of this work:

Verifier robustness and over-optimization. The paper's documentation of support shrinkage — where RLVR loses access to correct solutions the base model could reach — is a form of verifier over-optimization at the answer-set level. The model optimizes reward so aggressively that it collapses onto a narrow subset of correct completions, excluding others that also satisfy the verifier. This frames verifier design as not just about reward accuracy (does the verifier correctly label correct vs. incorrect completions?) but about reward coverage (does the verifier's optimization landscape preserve access to all correct solutions, or does it create local maxima that exclude valid completions?). Improving verifier robustness to prevent support shrinkage — perhaps through ensemble verification, adversarial training against mode collapse, or reward shaping that explicitly penalizes concentration — becomes a higher-priority research direction than improving search algorithms or tuning RL hyperparameters.
Metrics beyond pass@1. The paper demonstrates that pass@1 improvements can mask support shrinkage — a model that goes from 70% to 80% pass@1 while losing 20% of its accessible correct solutions looks "better" under standard evaluation. This strengthens the case for multi-budget evaluation as standard practice (reporting pass@1, pass@64, and pass@1024 together, or providing pass@k curves) and for diversity-aware metrics (answer-level entropy, distinct-correct-answers count, or the support metrics themselves) alongside accuracy. The paper provides a concrete methodology for the diversity component.
Hybrid SFT + RLVR training pipelines. The SFT vs. DAPO comparison (Table 3) is perhaps the paper's most actionable single result: same data, same model, same hyperparameters, but SFT expands support while DAPO shrinks it. This directly motivates training pipelines that interleave SFT (to introduce new solution candidates and expand the support) with RLVR (to sharpen and filter, improving precision on the expanded set). The temporal analysis (Table 4) showing that RLVR can transiently discover solutions before losing them further suggests that RLVR could be used as a discovery-then-lock-in mechanism: let RLVR explore and find new solutions for a short period, then "freeze" those discoveries via distillation or SFT before the optimization dynamics prune them.

Research directions that become less attractive:

Better on-policy RLVR algorithms within the current paradigm. The paper's theoretical analysis (Theorems C.1, C.3, C.4) and cross-algorithm empirical results (Table 1: ProRL, DAPO, AceReason, Skywork, Nemotron, Phi4 all show the same pattern) suggest that the support-constrained behavior is inherent to the on-policy RLVR objective class, not to specific implementation details. Tuning GRPO hyperparameters, designing better advantage estimators, or adding minor exploration heuristics within the on-policy framework is unlikely to meaningfully change the NSCR sign — the leash is in the mathematical structure of the update, not in the engineering. Research effort is better directed at breaking out of the on-policy support constraint entirely (via off-policy data, explicit exploration objectives, or hybrid training paradigms) rather than incrementally improving within it.
Scaling RLVR to larger models as a solution to the capability ceiling. The paper shows that SRR increases with model scale (0.93 for 1.5B models → 0.99 for 14B models) but NDR does not (remains ≤0.01 for all scales). Larger models preserve more of their base support — the leash gets longer — but they remain tethered. If the goal is genuinely new reasoning capabilities beyond what the base model encodes, scaling model size within the current RLVR paradigm does not appear to be the answer; it only provides a better base model (whose support is broader, so shrinking it is less harmful) but not a mechanism for support expansion. This redirects scaling efforts from "bigger model + RLVR" toward "better pretraining + RLVR for precision" or "bigger model + explicit exploration mechanisms + RLVR."

Follow-Up Research This Work Enables

Exploration-incentivized RLVR and the critical experiment: does an answer-level entropy bonus produce positive NSCR? The paper's central prescriptive claim — "breaking the leash requires explicit exploration strategies" — is not directly tested. The most important follow-up experiment is a controlled comparison: take the exact DAPO training setup from Table 3 (Qwen2.5-Math-7B, DeepMath-103K, fixed hyperparameters) and add an answer-level entropy bonus to the reward function — a term proportional to $+H[\text{answer distribution}]$ that penalizes the model for collapsing onto too few outputs. Then measure NSCR, SRR, and NDR on the same benchmarks. A positive result (NSCR > 0 with maintained or improved pass@1) would validate the paper's diagnosis and demonstrate that the leash can be broken within the RLVR paradigm by addressing the exploration deficit. A negative result (NSCR still negative) would suggest that the concentration force from reward optimization dominates even with explicit diversity incentives, implying that off-policy data infusion or architectural changes are necessary — a deeper and more consequential finding that would reshape the research agenda. The paper's entropy metrics (token-level vs. answer-level, Table 5) provide the measurement apparatus for monitoring whether the bonus operates at the right level, since token-level entropy bonuses may be ineffective if token-level and answer-level entropy decouple as the paper documents.

Off-policy data injection during RLVR training and its effect on support expansion. Theorem C.3 proves that on-policy RLVR cannot amplify completions below the detection threshold because they are never sampled and thus contribute zero gradient. A natural circumvention: inject SFT-generated correct completions (or completions from a different model, or human-written solutions) into the RLVR training buffer, providing positive gradient signal for completions the current policy assigns negligible probability to. The follow-up experiment would augment DAPO training with a "support injection" phase every N steps: generate a set of correct completions via SFT (which the paper shows expands support; Table 3) or via a diverse ensemble, and mix these into the RLVR training batch alongside on-policy rollouts. Measure whether NSCR becomes positive, whether the injected solutions are retained (preserved) after subsequent RLVR steps, and whether pass@1 continues to improve at the same rate as pure RLVR. The paper's support-classification framework makes this experiment straightforward to evaluate: you can track exactly which injected completions the RLVR model retains access to vs. which it prunes, providing fine-grained feedback on the injection schedule and mixing ratio.

Difficulty-stratified support dynamics and the "hard problem ceiling." The paper's companion work analyzed difficulty-dependent behavior extensively, showing that test-time compute cannot help on problems where the base model's pass@1 is near zero. The natural extension to this paper: stratify the support metrics (SRR, NDR, NSCR) by the base model's pass@1 on each prompt — does support shrinkage primarily affect easy prompts (where there are many correct solutions to lose), hard prompts (where the base model's support is already narrow, making each loss proportionally more damaging), or both? The paper already has the data to perform this stratification (the prompts are labeled by the base model's correctness patterns, as evidenced by the P/E/S/O classification), and the analysis would reveal whether the "invisible leash" binds most tightly where it matters most (hard problems that the base model barely knows how to solve). A finding that shrinkage disproportionately affects hard problems would strengthen the paper's argument that RLVR fails precisely where capability expansion is most needed, rather than merely being a benign precision-diversity trade-off on already-solvable tasks.

Training-hyperparameter sweep for the shrinkage-expansion trade-off. The paper's theoretical bounds (Theorem C.3: amplification ≤ ε · exp(2ηTK)) predict that larger learning rates η and longer training K increase both potential expansion (by amplifying very-low-probability completions more) and shrinkage (by more aggressively removing mass from non-rewarded completions). The single temporal analysis (Table 4) shows that at one fixed η, longer training increases shrinkage on Minerva and Olympiad, but doesn't explore whether a different η would change the outcome. A systematic sweep of η × K (learning rate × training steps) with NSCR as the response variable would reveal whether there exists an "expansion window" — a combination of η and K where transient expansions (visible in AIME25 at step 60; Table 4) are captured and frozen before being pruned — or whether the E/S ratio is monotonic in total update magnitude such that no hyperparameter setting can produce net expansion. This experiment doesn't require any algorithmic innovation, only a grid search over training hyperparameters with the paper's support-classification framework as the evaluation metric, making it immediately actionable.

Cross-architecture and cross-modality replication of the support-constrained pattern. The paper evaluates DeepSeek-R1-Distill-Qwen variants at 1.5B–14B and includes one non-Qwen model (OLMo-2-0425-1B) and one vision-language model (Kangheng-OVR-7B), all showing the same qualitative pattern. But all models share the decoder-only transformer architecture and the causal language modeling pretraining objective. Would the support-constrained pattern hold for encoder-decoder models (T5, UL2), for models pretrained with different objectives (diffusion LMs, discrete diffusion, masked LMs adapted for generation), for models with explicit memory or retrieval mechanisms, or for multi-modal models operating over different input modalities (speech, video, code)? A positive finding (NSCR negative across architectures) would strengthen the case that the constraint is inherent to on-policy reward maximization from a frozen initialization, independent of architecture. A negative finding (some architecture shows positive NSCR) would identify architectural properties that naturally promote exploration, providing a design target for future reasoning models.

Support dynamics under continuous and non-binary rewards. The paper's entire framework assumes binary verifiable rewards R(x, y) ∈ {0, 1}, and Proposition C.4's exponentially tilted optimal policy form depends on this binary structure. What happens when the reward is continuous (e.g., a score from 0 to 1 based on partial correctness, a learned reward model's output, or a multi-dimensional reward vector)? The support concept would need to be generalized — perhaps defining support at threshold τ as completions with R(x, y) ≥ τ, and tracking how the set of "good enough" completions changes under RLVR with continuous rewards. This is practically important because many real-world RLHF and RLVR setups use continuous or multi-dimensional rewards (helpfulness scores, safety classifiers, task-specific rubrics), and the paper's framework currently cannot speak to these settings. A follow-up experiment using a continuous reward function (e.g., a learned Bradley-Terry reward model on a preference dataset, or a rubric-based scoring function for open-ended generation) and a threshold-based generalization of the support metrics would test whether the concentration phenomenon intensifies (binary rewards are already the sharpest possible signal; continuous rewards might allow smoother optimization that preserves more diversity) or persists similarly.

Practical Applications and Downstream Use Cases

Evaluation protocol for RLVR model releases. When an organization releases an RLVR-trained reasoning model (e.g., a new DeepSeek-R1 variant, a new open-source math model), the current standard is to report pass@1 and sometimes pass@64 on standard benchmarks. This paper provides a concrete, replicable protocol for a more informative evaluation: for each benchmark, sample k completions per prompt from both the base model and the RLVR model (k = 8,192 for math, k = 2,048 for non-math), extract correct completions, classify into P/E/S/O, and report SRR, NDR, SDS, and NSCR alongside the standard pass@k curve. A model with 95% pass@1 and NSCR = −0.10 (net shrinkage) is fundamentally different from a model with 95% pass@1 and NSCR = 0.00 (no net change) or NSCR = +0.02 (net expansion) — the first model has paid for its precision with lost solution diversity, while the third has genuinely expanded the set of reachable correct answers. Model release papers that include these metrics would give downstream users much better information about the model's robustness under high-k evaluation and its suitability for diversity-sensitive applications. The paper's methodology makes this protocol directly adoptable: the ϵ derivation (Appendix C.4), sampling budgets (Section 4.1), and enhanced answer extraction for format fairness (Appendix B.3) are all specified in replicable detail.

RLVR training monitoring with answer-level entropy. The paper's finding that answer-level entropy consistently decreases under RLVR (Table 5: -28% to -62% across all models) while token-level entropy sometimes increases (+61% for DAPO-32B) has a direct operational use: training dashboards for RLVR runs should track answer-level entropy as a diversity health metric, alongside standard training loss and reward curves. A steady decline in answer-level entropy signals that the model is collapsing onto fewer distinct correct answers, even if reward and pass@1 are improving and token-level entropy looks healthy. A sharp drop in answer-level entropy combined with flat or declining reward might indicate that the model has entered the verifier over-optimization regime documented in the paper's companion work on test-time compute, where it finds a small set of solutions that satisfy the verifier but fails to maintain access to the full set of correct completions. The paper provides the computation method in Section 5.1: extract final answers from the training rollouts, compute the empirical distribution over unique answers, and compute Shannon entropy. This is cheap (it uses the same rollouts already generated for training), requires no additional inference, and can be logged as a scalar per training step. A practitioner could set a threshold — say, answer-level entropy should not drop below 50% of its base-model value — and trigger a training intervention (reduce learning rate, add exploration bonus, interleave SFT steps) when the threshold is crossed, before the support shrinkage documented in Table 4 (S rising from 22 to 61 on Olympiad over 300 steps) becomes irreversible.

Hybrid SFT + RLVR training pipelines for precision + diversity. The paper's SFT vs. DAPO controlled comparison (Table 3) provides direct evidence that SFT expands support (NSCR positive on 4 of 5 benchmarks) while DAPO achieves better pass@1 at moderate k but shrinks support (NSCR negative or mixed). This suggests a specific training recipe for practitioners who want both improved single-sample accuracy and preserved solution diversity: (1) Start with the base model and run SFT on a diverse set of correct completions to expand the support set (introduce new solution approaches the base model could not reliably produce). (2) Run a moderate amount of RLVR on the SFT-expanded model, monitoring answer-level entropy to stop before significant shrinkage sets in — the temporal analysis (Table 4) suggests early stopping around the point where NSCR crosses from positive to negative, or where S begins to consistently increase. (3) Optionally, run a final SFT phase on the RLVR model's own high-reward outputs mixed with the original SFT data to "lock in" the RLVR-discovered solutions and restore any lost diversity. This pipeline leverages RLVR's demonstrated strength (improving precision by concentrating mass on high-reward completions) while using SFT to counter its demonstrated weakness (shrinking support by dropping probability on correct-but-lower-reward completions). The paper provides the metrics (NSCR, SRR, NDR, answer-level entropy) to monitor each phase and to determine the optimal transition points between phases — a concrete improvement over current practice, where RLVR and SFT are typically treated as mutually exclusive post-training strategies rather than complementary stages.

Deployment decision-making based on inference budget. The paper's pass@k curves (Figures 2–4) and support-shrinkage documentation provide guidance for a practical decision that every deployment team faces: given a fixed inference budget, should I deploy the base model or the RLVR model? The answer depends on the per-query sampling budget the deployment can afford: if you can generate at most k = 1–4 completions per query (typical for latency-constrained interactive systems), deploy the RLVR model — its pass@1 advantage is substantial (e.g., ProRL pass@1 = 0.96 vs. base pass@1 = 0.12 on graph coloring hard; Figure 2). If you can afford k = 64–256 completions (typical for batched evaluation pipelines or offline scoring), check whether the crossover point (where base pass@k exceeds RLVR pass@k) has been reached for your specific tasks — the paper's Reasoning Gym pass@k curves show that crossover varies by task and can occur anywhere from k = 16 to k > 1,024. If you can afford k > 1,024 (large-scale batch inference, self-improvement data generation), the base model may be strictly better, as its broader support yields higher pass@k in cases where RLVR has undergone significant shrinkage (e.g., AIME 2024 base pass@8192 = 93.3% vs. ProRL's 83.3%; Section 4.2). The paper's per-task pass@k curves and the crossover pattern provide the data needed to make this decision on a per-benchmark basis rather than relying on aggregate metrics that may obscure task-level variation.