RL Training2025~107 minarXiv:2504.13837

Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?

score 9Yang Yue, Zhiqi Chen, Rui Lu +5↑ 139★ 329Apr 18, 2025cs.AIhttps://github.com/LeapLabTHU/limit-of-RLVR

🎯 Pitch

Current RL training doesn't actually expand an LLM's reasoning boundaries—it just makes the model more efficient at finding correct answers that already exist within the base model's output distribution. In fact, the base model's pass@k ceiling actually shrinks as RL training progresses, with all six popular algorithms leaving a massive 43-point gap between what the base model can do and what RL training actually unlocks.

1. Executive Summary

This paper systematically investigates whether reinforcement learning with verifiable rewards (RLVR) genuinely elicits novel reasoning capabilities in LLMs beyond those already present in the base model, evaluating multiple model families (Qwen2.5, LLaMA-3.1, Mistral), RL algorithms (GRPO, PPO, Reinforce++, RLOO, ReMax, DAPO), and benchmarks (MATH500, AIME24, LiveCodeBench, MathVista) using pass@k at large k values as the primary metric for reasoning boundary assessment. The central finding is that RLVR improves sampling efficiency toward correct paths (increasing the likelihood of already-accessible correct solutions, boosting pass@1) but does not expand the reasoning boundary—base models consistently achieve higher pass@k scores than their RLVR-trained counterparts as k grows large (e.g., base models surpass RL models by ~9% on Minerva at k=128 for the 32B model), and perplexity analysis reveals that reasoning paths generated by RLVR models already exist within the base model's sampling distribution. Across six popular algorithms, the paper quantifies a consistently large sampling efficiency gap (ΔSE)—the difference between the RL model's pass@1 and the base model's pass@256—ranging from 42.6 to 43.9 points, establishing that current RLVR methods remain far from fully leveraging base model potential, while distillation from a stronger teacher is shown to genuinely expand reasoning boundaries beyond the base model.

2. Context and Motivation

The Core Problem: What Exactly Does RLVR Do to an LLM's Reasoning?

The rise of reasoning-centric LLMs like OpenAI-o1, DeepSeek-R1, and Kimi-1.5 has been widely attributed to Reinforcement Learning with Verifiable Rewards (RLVR) — a training paradigm where models are optimized against automatically computable binary rewards (correct/incorrect final answers, pass/fail unit tests) rather than human preference labels. The narrative surrounding these models, particularly following DeepSeek-R1, has been that RLVR enables LLMs to autonomously discover novel reasoning strategies — behaviors like self-reflection, verification, enumeration of alternatives, and iterative refinement — that were not explicitly present in the base pretrained model.

This paper confronts a fundamental question that this narrative leaves unanswered: Does RLVR genuinely elicit new reasoning capabilities, or does it merely improve the model's ability to reliably sample reasoning paths that the base model already could produce?

The distinction matters enormously. If RLVR expands what a model can reason about — enabling it to solve problem classes that were completely inaccessible to the base model — then it represents a genuine path toward self-improving AI systems that can transcend their training distribution. But if RLVR only sharpens the model's tendency to produce correct answers on problems it could already solve (albeit unreliably), then the apparent "emergence" of sophisticated reasoning behaviors is an illusion created by inadequate evaluation — we would be mistaking better sampling for better reasoning.

Why This Question Has Been Difficult to Answer

The default evaluation paradigm in the RLVR literature uses average-case metrics: greedy decoding accuracy (pass@1) or the mean of a small number of nucleus samples. These metrics answer the question "how often does the model get the right answer on the first try?" — a practically important question for deployment, but one that fundamentally confuses sampling efficiency with reasoning capability.

Consider an analogy: if a student knows how to solve a calculus problem but gets nervous and makes arithmetic errors 80% of the time, their average performance looks poor. Training that reduces their error rate from 80% to 20% dramatically improves average scores without teaching them any new calculus. Similarly, an LLM might possess the reasoning patterns needed to solve a complex math problem within its output distribution, but those patterns might be buried among many incorrect alternatives. RLVR could surface those patterns more reliably without adding any new reasoning capability.

The pass@1 metric cannot distinguish between these two scenarios. A model that genuinely can't solve a problem (no correct paths in its distribution) and a model that can solve it 1% of the time are both near zero on pass@1 — but they have fundamentally different reasoning boundaries. To answer whether RLVR expands reasoning capability, we need a metric that probes the boundary of what a model can potentially solve, not just what it typically solves.

The Gap in Prior Analysis

Several prior works have hinted at the limitations this paper systematically investigates, but none provided a comprehensive answer:

DeepSeek-Math (Shao et al., 2024) observed that their base model could achieve competitive performance with the RLVR-trained model when evaluated with repeated sampling, but this observation was limited to a single instruction-tuned model on two math benchmarks. It did not explore whether this was a general phenomenon across model families, tasks, or RL algorithms, nor did it analyze what this implied about the nature of RLVR's effect on reasoning.

Dang et al. (2025) observed that pass@k performance deteriorated after RLVR training in a limited experimental setup — Qwen-2.5-0.5B on GSM8K. While this finding anticipated some of the current paper's results, the scope was severely constrained: a single very small model, a single relatively easy benchmark, and crucially, no investigation of the relationship between the base model and the RL-trained model's output distributions. The paper did not establish whether the RL model was losing capabilities or simply redistributing its probability mass.

Liu et al. (2025a) and Zhao et al. (2025b) argued that "reflective behaviors" observed in R1-like models — such as the model checking its own work or backtracking from incorrect intermediate steps — actually originate from the pretraining data rather than being introduced by RLVR training. This was an important clue: if even the signature behavioral patterns attributed to RLVR are pre-existing in the base model, what exactly is RLVR adding? However, their analysis was qualitative and focused specifically on self-reflection patterns rather than the broader question of reasoning capability boundaries.

Shah et al. (2025) provided evidence that reflection-like behaviors can emerge purely from pretraining without any RL, further supporting the idea that RLVR's role may be more about amplification than origination.

None of these works, however, provided the comprehensive, quantitative framework needed to definitively answer the overarching question: does RLVR expand reasoning boundaries or merely improve sampling efficiency within existing boundaries?

Why This Matters: Practical and Theoretical Stakes

The question has immediate practical implications for how the field allocates resources. If RLVR genuinely expands reasoning boundaries, then the path to more capable systems is straightforward: apply more RLVR training to larger base models, potentially creating a virtuous cycle of self-improvement. This is the vision articulated in DeepSeek-R1: RLVR as a path toward "continuously self-evolving LLMs." But if RLVR is fundamentally bounded by the base model's capabilities, then the ceiling on RLVR-based improvement is determined by pretraining quality — and the field's focus should shift from better RL algorithms to better base models, data curation, and techniques (like distillation) that can genuinely introduce new reasoning patterns.

There are also theoretical stakes. Traditional reinforcement learning — in game-playing domains like Go (AlphaGo Zero) and Atari (DQN) — is celebrated precisely because agents discover novel strategies that exceed human-designed heuristics. AlphaGo Zero didn't just get better at executing known Go patterns; it invented new ones that human masters hadn't considered. If RLVR for LLMs does not exhibit this same capacity for genuine discovery, then it represents a fundamentally different phenomenon from traditional RL — one where the "exploration" is so constrained by the pretrained prior that it cannot escape into novel regions of the solution space.

The paper positions itself as the first systematic attempt to answer this question across the full landscape of relevant variables: multiple model families (Qwen2.5 in 7B/14B/32B sizes, LLaMA-3.1-8B, Mistral), multiple task domains (mathematics, code generation, visual reasoning), multiple RL algorithms (PPO, GRPO, Reinforce++, RLOO, ReMax, DAPO), and the crucial contrast case of distillation (which is shown to behave fundamentally differently).

How This Paper Positions Itself

The paper's central methodological innovation is the use of pass@k at sufficiently large k as a probe for reasoning boundaries. The pass@k metric asks: if we sample k independent solutions from the model, does at least one of them solve the problem correctly? When k is large enough (e.g., 128, 256, or 1024), the pass@k value approaches the model's true reasoning coverage — the fraction of problems for which a correct solution exists somewhere in the model's output distribution. This metric distinguishes between the two scenarios we described:

A model that genuinely cannot solve a problem (no correct paths exist in its distribution) will have pass@k ≈ 0 regardless of k.
A model that can solve a problem but does so rarely (correct paths exist but have low probability) will have pass@1 ≈ 0 but pass@k → 1 as k increases.

By comparing pass@k curves for base models and their RLVR-trained counterparts, the paper can directly test the expansion hypothesis: if RLVR expands reasoning boundaries, the RLVR model's pass@k curve should lie above the base model's curve at large k, indicating that it can solve problems the base model cannot. If RLVR only improves sampling efficiency, the curves should cross: the RLVR model wins at small k (where efficiency matters) but the base model wins at large k (where coverage dominates).

The paper positions the pass@k framework not as a practical deployment metric (the authors are explicit that best-of-N or majority voting are more practical for actual use), but as a diagnostic tool for understanding the nature of RLVR's effect. This is a crucial distinction: the goal is not to advocate for using pass@k in production, but to use it to answer a scientific question about what RLVR actually does to the model.

Beyond the pass@k analysis, the paper deploys a battery of supporting analyses — accuracy distribution histograms, solvable-problem set comparisons, perplexity-based distribution analysis, and the sampling efficiency gap (ΔSE) metric — to triangulate the answer from multiple angles. The perplexity analysis is particularly telling: by measuring how likely the base model is to generate the exact reasoning paths produced by the RLVR-trained model, the paper can directly test whether those paths were already within the base model's distribution. Finding that the base model assigns low perplexity (high likelihood) to RLVR-generated paths constitutes direct evidence that those paths were not invented by RLVR — they were already there.

3. Technical Approach

3.1 Reader Orientation

This paper is primarily an empirical diagnostic study—it does not propose a new training algorithm or model architecture, but rather constructs and applies a systematic evaluation methodology to determine what reinforcement learning with verifiable rewards (RLVR) actually changes in an LLM's reasoning capabilities. The core idea is deceptively simple: by comparing how base models and their RLVR-trained counterparts perform when we sample many solutions per problem (not just one), we can distinguish between improved sampling efficiency and genuine expansion of reasoning capability. If RLVR only makes the model more likely to produce correct answers it was already capable of generating, then at large sample sizes the base model should catch up to or surpass the RLVR model. If RLVR genuinely teaches the model to solve new problems, the RLVR model should outperform the base model even when both are given unlimited attempts.

3.2 Big-Picture Architecture (Diagram in Words)

The paper's analytical framework has four major components:

Model Pairs — for each experiment, a base model (pretrained or instruction-tuned) and its RLVR-trained counterpart, evaluated side-by-side under identical conditions. The base model serves as the reference distribution; the RLVR model is the object of investigation.
Pass@k Evaluation Engine — for each model and each problem in a benchmark, the system samples $n$ independent responses (where $n$ is the maximum $k$ to be evaluated, typically 128, 256, or 1024), checks each response against a ground-truth verifier (math answer matching, code unit tests, or visual reasoning correctness), and uses an unbiased statistical estimator to compute the pass@k value for every $k \leq n$ . This produces a pass@k curve: a monotonically non-decreasing function from $k=1$ to $k=n$ representing the estimated fraction of problems that can be solved with at most $k$ attempts.
Supporting Analysis Suite — a collection of diagnostic tools that triangulate the pass@k findings: accuracy distribution histograms showing how per-problem solve rates shift with RLVR training, set-overlap analysis of which specific problems each model can solve, perplexity measurements quantifying how likely the base model is to generate the RLVR model's reasoning paths, and the sampling efficiency gap (ΔSE) metric that quantifies how far an RL model is from fully exploiting the base model's potential.
Controlled Training Experiments — clean re-implementations of six RL algorithms (GRPO, PPO, Reinforce++, RLOO, ReMax, DAPO) using the VeRL framework on a fixed training dataset (Omni-MATH-Rule), enabling direct comparison of how different algorithmic choices affect the pass@k curve while holding all other variables constant.

Information flows as follows: a problem is sampled from the benchmark → the model (base or RLVR) generates $n$ responses → each response is scored by the verifier → the unbiased pass@k estimator aggregates across problems to produce the pass@k curve → supporting analyses decompose why the curves look the way they do.

3.3 Roadmap for the Deep Dive

First, the pass@k metric and its unbiased estimator (Section A.2), since this is the central analytical instrument and understanding its statistical properties is essential for interpreting every result.
Second, the evaluation protocol—sampling configurations, temperature, prompts, verifier design—because the comparison between base and RLVR models must be fair and controlled.
Third, the training configurations for the RLVR models being evaluated, including the zero-RL setting for math, the instruction-tuned starting point for code, and the controlled re-implementation of six RL algorithms for the algorithm comparison.
Fourth, the supporting analysis methods: accuracy histograms, solvable-problem set analysis, perplexity measurement, and the sampling efficiency gap (ΔSE) metric.
Fifth, the distillation contrast experiment, which serves as a positive control—showing what a genuine expansion of reasoning boundaries looks like—and the entropy/temperature analysis probing the relationship between output diversity and reasoning coverage.

3.4 Detailed, Sentence-Based Technical Breakdown

This is an empirical diagnostic study whose core methodological contribution is the systematic application of pass@k at large k values as a probe for reasoning boundaries, combined with a suite of supporting analyses that together demonstrate that current RLVR training reshapes sampling distributions within pre-existing boundaries rather than expanding those boundaries outward.

The Pass@k Metric and Unbiased Estimation

What pass@k measures. Given a model, a problem, and a verification function that returns a binary correctness judgment for any model output, the pass@k metric asks: if we sample $k$ independent responses from the model, does at least one of them pass verification? For a single problem, pass@k is a binary random variable (1 if at least one of the k samples is correct, 0 otherwise), and the expected value over random sampling is the probability that the model can solve the problem within k attempts.

At the dataset level, the average pass@k across all problems represents the expected fraction of the dataset that the model can solve within k trials. As $k$ grows, this quantity approaches a natural limit: the fraction of problems for which a correct solution exists somewhere in the model's output distribution—what the paper calls the model's reasoning boundary or reasoning coverage.

Why pass@k distinguishes capability from efficiency. The paper's central insight is that the shape of the pass@k curve, not just any single point on it, reveals the nature of RLVR's effect. A model with broad reasoning coverage distributes its probability mass across many correct solutions; its pass@k rises quickly and reaches a high asymptote. A model with narrow but concentrated coverage has high probability on a few solutions; it may have excellent pass@1 but its pass@k curve rises slowly beyond that, potentially being surpassed at larger k by a model with more diffuse but broader coverage.

This is exactly the pattern the paper observes: RLVR models have steeper pass@k curves at very small k (high pass@1 relative to base) but flatter curves at larger k, while base models have shallower initial slopes but higher asymptotes. The crossing of the curves—where the base model's pass@k overtakes the RLVR model's—is the signature of improved sampling efficiency paired with reduced reasoning coverage.

The unbiased estimator (Equation 2). Directly computing pass@k by taking exactly k samples per problem and checking if any is correct produces a high-variance estimate, because the binary outcome from k samples has substantial sampling variation. The paper uses the standard low-variance unbiased estimator from Chen et al. (2021), which works by oversampling: generate $n > k$ responses per problem, count how many are correct, and use a combinatorial formula that accounts for the fact that we observed n samples but want to estimate what would happen with k.

The estimator for a single problem $i$ with $c_i$ correct samples out of $n$ total is:

$\text{pass@k}_i = 1 - \frac{\binom{n-c_i}{k}}{\binom{n}{k}}$

where $\binom{a}{b}$ denotes the binomial coefficient "a choose b."

What it computes: For a specific problem, the estimator calculates $1$ minus the probability that none of the $k$ draws would be correct, given that $c_i$ out of $n$ total samples were observed to be correct. The term $\frac{\binom{n-c_i}{k}}{\binom{n}{k}}$ is the hypergeometric probability of drawing $k$ samples that are all from the $n - c_i$ incorrect ones—in other words, the probability that a random subset of size $k$ contains zero correct samples. Subtracting from $1$ gives the probability that at least one of the $k$ samples is correct.

Why this form: This formula is an unbiased estimator of the true pass@k probability for any $k \leq n$ , meaning its expected value over repeated sampling equals the true pass@k. This is not obvious—a naive approach of subsampling exactly k from the n and checking would also be unbiased but would have higher variance because it discards information. The hypergeometric form uses all $n$ samples to inform the estimate at every $k$ , reducing variance substantially. It also allows computing pass@k for all $k \leq n$ from a single batch of $n$ samples, which is computationally efficient. The only requirement is that $n > c_i$ when $k > n - c_i$ (i.e., we can't estimate pass@k when $k$ exceeds the number of incorrect samples we observed if none of the k could possibly be correct), but this condition is naturally satisfied in the paper's experiments where $n$ is set to the maximum $k$ of interest.

Dataset-level aggregation. The overall pass@k for a dataset $D$ is the simple average over problems:

$\text{pass@k} = \mathbb{E}_{x_i \sim D}\left[1 - \frac{\binom{n-c_i}{k}}{\binom{n}{k}}\right]$

where the expectation is taken over the problems in the dataset, with each problem contributing its own $c_i$ and $n$ values (in practice, $n$ is constant across problems).

Practical values of n. For most benchmarks, the paper sets $n$ to the rightmost point on the pass@k curve: $n = 128$ for MATH500, Minerva, GSM8K, and Olympiad (Qwen models); $n = 1024$ for AIME24 and AMC23; $n = 1024$ for LLaMA-3.1-8B on Olympiad due to its lower base capacity requiring more samples to observe correct answers. For coding benchmarks (LiveCodeBench, HumanEval+, MBPP+), $n = 128$ is used.

Distinction from best-of-N and majority voting. The paper is explicit that pass@k is not a practical deployment metric—it does not include a mechanism for selecting the correct answer from among the k samples, it simply checks whether any correct answer exists. Best-of-N (which uses a learned verifier to pick the best sample) and majority voting (which picks the most common answer) are practical selection methods but may miss correct solutions if the verifier is imperfect or the correct answer is not the plurality. Pass@k is a potential metric: it reveals the model's latent capability, not its realized performance under a particular selection strategy. The paper studies pass@k "not to assess practical utility but to investigate the boundaries of reasoning capacity" (Section 2.2).

Evaluation Protocol: Sampling, Verification, and Fair Comparison

Sampling configuration. For all pass@k evaluations, the paper uses a temperature of 0.6 and a top-p value of 0.95, with a maximum generation length of 16,384 tokens. These settings are applied identically to both base and RLVR models. The paper also explores alternative temperature settings in Figure 17 to verify that the conclusions are not sensitive to this choice.

Temperature sensitivity (Figure 17). The paper found that base model performance degrades when temperature exceeds 1.0, as higher temperatures produce more random and less coherent tokens. RLVR-trained models, by contrast, maintain relatively stable performance across temperature settings—likely because RLVR training sharpens the output distribution, making it less sensitive to the sampling temperature. The paper therefore uses $T = 0.6$ as the setting that allows both model types to demonstrate their best performance.

Prompt design: zero-shot for base models, matching prompts for RLVR models. A critical methodological choice is making the comparison fair. Base models are typically evaluated with few-shot examples that guide output format, while RLVR training usually uses a specific zero-shot prompt. To eliminate any confounding effect where the base model benefits from few-shot guidance that the RLVR model doesn't receive (or vice versa), the paper evaluates both models using the exact same zero-shot prompt that was used during RLVR training—or, for benchmarks with their own standard prompts (like LiveCodeBench), the default benchmark prompt.

The authors note an interesting observation: "although base models often produce unformatted or non-sensical responses without few-shot guidance, we observe that with sufficient sampling, they are still capable of generating correctly formatted outputs and successfully solving complex problems" (Section 3). This means the base model's pass@k advantage at large k is despite the handicap of zero-shot prompting, making the finding even stronger.

Verification mechanisms by domain.

Mathematics: The verifier checks whether the model's final answer (extracted from the \boxed{} formatting) matches the ground-truth answer. The paper is aware that as k grows large, there is a risk of "lucky guessing"—generating an incorrect chain of thought that accidentally produces the right final answer. To address this, the authors manually inspect a subset of the hardest problems (those with average accuracy below 5%) to verify that correctly-answered problems actually contain valid reasoning. On GSM8K: the base model solved 25 such problems with 24 containing at least one correct CoT; the RL model solved 25 with 23 containing at least one correct CoT. On a filtered subset of AIME24 (with guessable problems removed), the base model solved 7 hard problems with 5 out of 6 containing at least one correct CoT, while the RL model solved 6 with 4 containing at least one correct CoT. These manual checks support the validity of pass@k as a measure of genuine reasoning coverage.
Code generation: Verification is done by executing the generated code against predefined unit tests. Since passing all unit tests by generating semantically incorrect code is essentially impossible, pass@k provides a reliable measure of reasoning capability without the guessing concern that exists in mathematics.
Visual reasoning: Similar to mathematics, the verifier checks final answer correctness, and the authors manually inspect the hardest problems to confirm CoT validity, finding that for both base and RL models, 7 out of 8 hardest problems had at least one correct CoT.

Cross-model comparison protocol. The paper ensures that comparisons between base and RLVR models are always "apples-to-apples" by using the same model family, same model size, same sampling parameters, same prompts, and same verification criteria. The only difference is the RLVR training. This clean controlled comparison is what makes the pass@k crossing phenomenon interpretable as an effect of RLVR rather than a confound.

RLVR Training Configurations and Models Evaluated

Zero-RL for mathematics (SimpleRLZoo). For math experiments, the paper primarily uses models from SimpleRLZoo (Zeng et al., 2025), which apply GRPO directly to pretrained base models (Qwen2.5-7B/14B/32B-Base and LLaMA-3.1-8B) trained on GSM8K and the MATH training set. Critically, these models use correctness reward only with no format reward—the reward is 1 if the final answer is correct, 0 otherwise. The absence of a format reward is noteworthy because it means the RL training does not explicitly incentivize the model to produce well-structured outputs; any improvement in output structure is a byproduct of the correctness signal.

The paper also evaluates Oat-Zero-7B (Liu et al., 2025b) and DAPO-32B (Yu et al., 2025), two models characterized by strong performance on the challenging AIME24 benchmark. These serve as additional data points testing the generality of the pass@k crossing phenomenon across different training recipes.

RL from instruction-tuned models for code generation. For code tasks, the paper uses CodeR1-Zero-Qwen2.5-7B (Liu & Zhang, 2025), which applies zero-RL training starting from Qwen2.5-7B-Instruct-1M (an instruction-tuned model, not a raw pretrained model). The training data consists of 12K LeetCode and TACO samples, trained for 832 steps. The paper notes that for coding, open-source work "typically uses instruction-tuned models as starting points, primarily due to the training instability and limited effectiveness of using a pure zero-RL setting" (Section 2.1). The base model for comparison is therefore the instruction-tuned model, not the raw pretrained model.

Additionally, the paper evaluates DeepCoder-14B-Preview (Luo et al., 2025), built on DeepSeek-R1-Distill-Qwen-14B, as the most capable open-source RLVR-trained coding model. Due to computational constraints (32k response length), this model pair is evaluated only on LiveCodeBench.

RL from instruction-tuned models for visual reasoning. For visual reasoning, the paper uses EasyR1 (Zheng et al., 2025) to train Qwen2.5-VL-7B-Instruct (Bai et al., 2025) on Geometry3K, evaluating on filtered subsets of MathVista-TestMini and MathVision-TestMini (with multiple-choice questions removed). Again, the base model for comparison is the instruction-tuned version.

Controlled re-implementation of six RL algorithms. For the deep analysis in Section 4.3, the paper re-implements six popular RLVR algorithms using the VeRL framework (Sheng et al., 2024) to ensure a fair head-to-head comparison. The algorithms are:

GRPO (Group Relative Policy Optimization): Estimates advantage as the normalized reward within a group of responses to the same question: $A_i = (r_i - \text{mean}(r)) / \text{std}(r)$ where $r = \{r_1, ..., r_G\}$ for $G$ sampled responses.
PPO (Proximal Policy Optimization): Uses the clipped surrogate objective $L^{\text{CLIP}} = \mathbb{E}[\min(r_t(\theta)A_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon)A_t)]$ where $r_t(\theta) = \pi_\theta(y_t|x, y_{<t}) / \pi_{\theta_{\text{old}}}(y_t|x, y_{<t})$ and $A_t$ is estimated by a learned value network.
Reinforce++: A simple policy gradient variant.
RLOO (Reinforce with Leave-One-Out baseline): Estimates advantage as $A_i = r_i - \frac{1}{|B|-1}\sum_{j \neq i} r_j$ where $B$ is the batch, providing a low-variance baseline without a separate value network.
ReMax: Uses the reward of the greedy-decoded response as the advantage baseline.
DAPO: Uses dynamic sampling with variable numbers of responses per prompt, requiring approximately 3-6× more samples per batch during training.

Training hyperparameters for the controlled comparison. The training configuration for these experiments is detailed in Section 4.3: AdamW optimizer (Loshchilov & Hutter, 2017) with a constant learning rate of $10^{-6}$ , prompt batch size of 256 with 8 responses generated per prompt, maximum rollout length of 8,192 tokens, sampling temperature of 1.0, and PPO mini-batch size of 256. Following DAPO and Oat-Zero, the KL divergence penalty term is removed to avoid constraining model learning—a design choice that the paper also ablates separately (adding KL back with coefficient 0.001, as reported in Section 4.4 and Figure 16).

The training data is Omni-MATH-Rule, a subset of Omni-MATH containing verifiable problems, split into a 2,000-sample training set and a 821-sample in-domain test set. MATH500 serves as the out-of-domain benchmark, enabling assessment of both in-distribution and out-of-distribution generalization.

Scaling experiments (Magistral-Medium). For the scaling analysis in Section 4.6, the paper uses Mistral-Medium-3-2505 as the base model and Magistral-Medium-2506 as its RLVR-trained counterpart, queried via API with a maximum context length of 40k tokens. Magistral-Medium is described as being "trained using pure RL" and is "positioned near the frontier in terms of reasoning capability" (Section 4.6), offering a test of whether the pass@k crossing phenomenon persists at larger scales.

Accuracy Distribution Analysis

Motivation. The pass@k curves show aggregate behavior, but don't reveal which problems drive the crossing. The accuracy distribution analysis decomposes per-problem solve rates into histograms, showing how RLVR training shifts the population of problems across accuracy bins.

Procedure. For each problem in a benchmark, the per-problem accuracy is computed as the fraction of samples (out of n total) that are correct, producing a value in [0, 1]. The distribution of these per-problem accuracies is then binned into 11 intervals: 0.0, (0.0, 0.1], (0.1, 0.2], ..., (0.9, 1.0), 1.0. Comparing the histogram for the base model against the histogram for the RLVR model reveals systematic shifts.

Key pattern (Figure 5, Figure 14). The paper observes a consistent pattern across benchmarks and model families: RLVR training increases the frequency of high-accuracy problems (near 1.0) and reduces the frequency of moderate-accuracy problems (e.g., 0.1-0.9), but also increases the frequency of accuracy-0 problems—problems that the model can no longer solve at all. The authors state: "RLVR increases the frequency of high accuracies near 1.0 and reduces the frequency of low accuracies (e.g., 0.1, 0.2). However, a deviation from this trend is the increased frequency at accuracy 0—indicating that RLVR leads to more unsolvable problems" (Section 4.1).

Interpretation. This explains why RLVR improves average scores (pass@1) while reducing overall coverage (pass@k at large k). RLVR is effectively redistributing probability mass: it concentrates the model's sampling budget on problems it can solve reliably, at the expense of abandoning problems it previously could solve occasionally. The model becomes more specialized and less general—a classic exploitation-exploration tradeoff manifesting at the level of problem coverage.

Solvable-Problem Set Analysis

Motivation. Even if the base model achieves higher aggregate pass@k, it's theoretically possible that the RLVR model solves a different set of problems—some that the base model cannot solve, offset by losses on other problems. The set analysis directly tests whether the RLVR model's solvable problems are a subset of the base model's solvable problems.

Procedure. For a given benchmark and a fixed k (k = 1024 for AIME24, k = 128 for MATH500), a problem is classified as "solvable" by a model if at least one of the n samples is correct. The paper then categorizes each problem into one of four categories: (1) solvable by both models, (2) solvable only by the base model, (3) solvable only by the RLVR model, (4) unsolvable by either model. The results are presented as percentages in Table 2 and as explicit problem index lists in Tables 5 and 6.

Results (Table 2). On AIME24: 63.3% of problems are solvable by both, 13.3% are solvable only by the base model, 0.0% are solvable only by the RLVR model, and 23.3% are unsolvable by either. On MATH500: 92.4% both, 3.6% base only, 1.0% RLVR only, 3.0% neither. The 1.0% RLVR-only cases on MATH500 (approximately 5 problems) are further investigated and found to actually be solvable by the base model when sampling 1024 times rather than 128—suggesting that even the apparent exceptions are artifacts of finite sampling rather than genuine expansions of capability.

The explicit problem indices in Tables 5 and 6 confirm the approximate subset relationship visually: the RLVR model's solvable problem list is nearly contained within the base model's list, with the base model solving additional problems that the RLVR model misses.

Perplexity Analysis: Testing Whether RLVR Paths Exist in the Base Distribution

Motivation. The set analysis shows which problems each model can solve, but doesn't directly answer whether the reasoning paths themselves produced by the RLVR model are novel or pre-existing. The perplexity analysis provides a direct test: if the base model assigns low perplexity (high likelihood) to the RLVR model's generated responses, then those responses were already likely under the base model's distribution—meaning they were not "discovered" by RLVR but were always within reach.

Perplexity definition. Given a model $m$ , a problem $x$ , and a response $Y = (y_1, ..., y_T)$ , the perplexity is:

$\text{PPL}_m(Y | x) = \exp\left(-\frac{1}{T}\sum_{t=1}^{T}\log P(y_t | x, y_1, ..., y_{t-1})\right)$

where $P(y_t | x, y_1, ..., y_{t-1})$ is the model's predicted probability for token $y_t$ given the prompt and previous tokens.

What it computes: perplexity is the exponentiated average negative log-likelihood of the response under the model. A lower perplexity means the model assigns higher probability to the exact token sequence—in other words, it "expects" this response. A perplexity of 1 would mean the model assigns probability 1 to every token (perfect prediction); typical values for natural language are much higher.

Why exponentiation: the average negative log-likelihood is in log space (nats), which is hard to interpret directly. Exponentiating converts it to a more interpretable scale: perplexity can be thought of as the model's "effective vocabulary size" at each position—the number of equally likely alternatives it would need to consider to achieve that average log probability. A perplexity of 100 means the model is, on average, as uncertain at each position as if choosing uniformly from 100 options.

Procedure. The paper samples two problems from AIME24 and generates 16 responses from each model: $Y_{\text{base}}$ from Qwen2.5-7B-Base and $Y_{\text{RL}}$ from SimpleRL-Qwen2.5-7B-Base. They then compute three perplexity distributions:

$\text{PPL}_{\text{Base}}(Y_{\text{base}} | x)$ : how likely the base model finds its own responses.
$\text{PPL}_{\text{Base}}(Y_{\text{RL}} | x)$ : how likely the base model finds the RLVR model's responses.
$\text{PPL}_{\text{RL}}(Y_{\text{RL}} | x)$ : how likely the RLVR model finds its own responses.

Additionally, they include $\text{PPL}_{\text{Base}}(Y_{\text{GT}} | x)$ where $Y_{\text{GT}}$ are responses generated by OpenAI-o1, serving as a reference for genuinely "out-of-distribution" content.

Key finding (Figure 6). The distribution of $\text{PPL}_{\text{Base}}(Y_{\text{RL}} | x)$ closely matches the lower portion of the $\text{PPL}_{\text{Base}}(Y_{\text{base}} | x)$ distribution—the part corresponding to responses that the base model tends to generate with high confidence. This means the RLVR model's responses are not novel or surprising to the base model; they fall squarely within the high-probability region of the base model's output distribution. Moreover, as RL training progresses (Figure 15), $\text{PPL}_{\text{Base}}(Y_{\text{RL}} | x)$ gradually decreases, indicating that RLVR is sharpening the model's focus within the base distribution rather than venturing outside it.

Why this evidence is conclusive. If RLVR were discovering genuinely new reasoning strategies, the RLVR model's responses would look unusual or surprising to the base model—they would have high perplexity under the base distribution (similar to how o1-generated responses might look unfamiliar). Instead, the base model finds the RLVR model's responses typical—indeed, more typical (lower perplexity) than many of its own responses. This is direct distributional evidence that RLVR is selecting within, not expanding beyond, the base model's prior.

The Sampling Efficiency Gap (ΔSE) Metric

Motivation. Given the finding that the base model's pass@k at large k represents an approximate upper bound on what the RLVR model can achieve, the paper defines a metric to quantify how close a given RL algorithm comes to that bound. This metric enables fair comparison across algorithms and reveals how much room for improvement remains.

Definition. The sampling efficiency gap is:

$\Delta\text{SE} = \text{pass@1}_{\text{RL}} - \text{pass@}k_{\text{Base}}$

where $\text{pass@1}_{\text{RL}}$ is the RL-trained model's greedy or single-sample accuracy, and $\text{pass@}k_{\text{Base}}$ is the base model's pass@k evaluated at a large k (the paper uses k = 256 as a proxy for the upper-bound performance). Lower ΔSE is better—it means the RL model's single-sample performance is closer to what the base model can achieve with unlimited attempts.

What it computes: ΔSE measures the gap between what the RL model achieves on its first try and what the base model could theoretically achieve if we could afford to sample many times and pick the correct answer. It quantifies the remaining potential that RLVR has not yet captured.

Why this specific form: using pass@1 for the RL model reflects practical utility (how often does the model get it right on a single attempt?), while using the base model's pass@k reflects the theoretical ceiling (what fraction of problems could be solved if we had perfect selection among many candidate solutions?). The subtraction gives units of percentage points of accuracy. A ΔSE of 0 would mean the RL model achieves in one attempt what takes the base model many attempts—perfect sampling efficiency. A large positive ΔSE means the base model could solve many more problems than the RL model does on its first attempt, indicating the RL model is far from extracting the base model's full potential.

Results (Figure 8, Tables 3 and 4). Across the six algorithms and three datasets, ΔSE values range from RLOO's best of 42.6 to GRPO's 43.9 on the in-domain test set, with all algorithms showing ΔSE consistently above 40 points. On MATH500 (out-of-domain), the gap is smaller (approximately 20 points) because the base model's pass@256 is already near ceiling (96.2%), leaving less room for improvement. The paper interprets the fact that all algorithms cluster within a narrow range (42.6-43.9) as evidence that "current RLVR methods, while improving sampling efficiency, are still far from optimal" and that "novel RL algorithms or entirely new paradigms may be necessary to approach the upper bound" (Section 4.3).

Additional details from algorithm comparison (Section 4.3, Appendix C.5). The paper notes several algorithm-specific observations:

DAPO achieves slightly higher pass@1 but drops significantly at k=256, and its dynamic sampling strategy requires 3-6× more training samples per batch.
RLOO and Reinforce++ perform consistently well across the entire k range with efficient training costs.
ReMax shows lower performance at both pass@1 and pass@256, hypothesized to be due to its use of a single greedy reward as the advantage baseline, which in the RLVR setting is binary (0 or 1) and highly variable, leading to unstable gradient updates.

Distillation as a Positive Control

Motivation. To validate that the pass@k framework can indeed detect genuine expansions of reasoning capability when they occur, the paper includes a distillation experiment as a positive control. Distillation from a stronger teacher model should—if successful—genuinely introduce reasoning patterns that were not present in the student's base distribution. If the pass@k crossing phenomenon is specific to RLVR and not an artifact of the metric, distilled models should show a qualitatively different pattern.

Setup. The paper compares four model variants derived from Qwen2.5-Math-7B:

Base: the pretrained Qwen2.5-Math-7B.
RLVR (Oat-Zero): Qwen2.5-Math-7B-Oat-Zero, trained with RLVR.
Instruct: Qwen2.5-Math-7B-Instruct, trained with standard instruction fine-tuning.
Distill: DeepSeek-R1-Distill-Qwen-7B, which distills DeepSeek-R1 (a powerful reasoning model) into the Qwen2.5-Math-7B student.

Key finding (Figure 7). The distilled model's pass@k curve lies consistently and significantly above that of the base model at all values of k—there is no crossing. This indicates that distillation genuinely expands the reasoning boundary: the distilled model can solve problems that the base model cannot solve at all, not just solve them more efficiently. The paper states: "unlike RL that is fundamentally bounded by the reasoning capacity of the base model, distillation introduces new reasoning patterns learned from a stronger teacher model" (Section 4.2).

The contrast between RLVR (where curves cross) and distillation (where the distilled curve dominates) validates the diagnostic power of the pass@k framework and reinforces the paper's central claim that RLVR's effect on reasoning capability is qualitatively different from what many in the field have assumed.

RL Training Progress and Asymptotic Effects

Motivation. If RLVR training progressively narrows the reasoning boundary, we should see pass@k at large k decline over training steps even as pass@1 improves. The paper directly tests this by evaluating checkpoints at different stages of GRPO training.

Setup. Using the controlled GRPO training setup from Section 4.3, the paper evaluates three checkpoints: step 150, step 300, and step 450, and compares them to the base model on Omni-MATH-Train, Omni-MATH-Test, and MATH500.

Key finding (Figure 1, right; Figure 8, bottom; Table 4). As training progresses:

On Omni-MATH-Train: pass@1 improves from 26.1% (step 150) → 33.6% (step 300) → 42.5% (step 450). However, pass@256 decreases from 66.3% → 65.3% → 64.3%, progressively falling further below the base model's 67.2%.
On Omni-MATH-Test: pass@1 improves from 25.1% → 27.1% → 28.3%, but pass@256 decreases from 68.3% → 66.6% → 63.9%, compared to the base model's 69.1%.
On MATH500: a similar pattern holds, with pass@1 improving (74.4% → 75.4% → 76.3%) while pass@256 declines (97.2% → 96.0% → 95.4%), though the base model's 96.2% creates a narrower gap due to the ceiling effect.

This monotonic decline in pass@k at large k over training is one of the paper's most striking findings: more RLVR training makes the model better on its first attempt but actually reduces the total set of problems it can solve at all. This is the "reduced scope of reasoning capacity" depicted in Figure 1 (left).

Entropy and Temperature Analysis

Motivation. As RL training progresses, the model's output entropy typically decreases—the distribution becomes more peaked, which could explain reduced reasoning coverage simply through reduced diversity. To assess whether reduced entropy alone accounts for the coverage reduction, the paper conducts a controlled experiment that matches the RLVR model's sampling entropy to the base model's.

Procedure (Figure 18). For each benchmark, the base model's output entropy $E_{\text{base}}$ is first measured at T = 0.6. Then the temperature of the RLVR model (SimpleRLZoo) is increased until its output entropy approximately matches $E_{\text{base}}$ . For example, on AIME24, the base model's entropy is 0.47; the RLVR model at T=1.5 achieves a matching entropy. The paper then compares pass@k curves for three conditions: base at T=0.6, RLVR at T=0.6 (standard), and RLVR at the temperature-matched setting.

Key finding. While the temperature-matched RLVR model performs slightly better at large k than the standard RLVR at T=0.6, it still underperforms the base model. The paper concludes: "this suggests that while reduced entropy contributes to the narrowing of the reasoning boundary, it alone does not fully account for the reduction" (Section 4.5). In other words, the coverage reduction is not simply a matter of the model becoming less diverse—there appears to be a genuine loss of reasoning capability on specific problems, not just reduced sampling of those capabilities.

Ablation Experiments: KL Penalty and Rollout Number

KL penalty ablation (Figure 16). Some prior RLVR work adds a KL divergence penalty to the training objective to prevent the model from deviating too far from the base distribution. The paper ablates this by training a GRPO variant with KL coefficient 0.001 (applied alongside the standard GRPO without KL). The KL-regularized model achieves similar pass@1 to the unregularized GRPO, but with much lower pass@128—meaning the KL penalty, while not helping pass@1, actively hurts the model's ability to maintain broad reasoning coverage. This is consistent with the hypothesis that RLVR benefits from concentrating probability mass, and KL regularization partially counteracts this concentration.

Rollout number ablation (Figure 16). The number of responses sampled per prompt during training ( $n$ in training, distinct from $n$ in evaluation) controls how much exploration occurs in each training step. The paper increases this from 8 to 32, keeping the prompt batch size constant (which increases computation per step). Due to resource constraints, the n=32 model is only trained for 220 steps (vs. 450 for n=8). Even at this earlier training stage, the n=32 model achieves higher pass@128, highlighting that larger per-prompt sampling during training helps maintain broader coverage. However, even with n=32, the RL-trained model is still eventually surpassed by the base model at the largest k values—the broader training-time exploration helps but does not fundamentally change the crossing phenomenon.

Summary of Design Choices and Their Justifications

Pass@k rather than pass@1 or best-of-N as the primary metric: pass@k probes potential capability rather than realized performance, enabling the distinction between sampling efficiency and reasoning coverage.
Unbiased hypergeometric estimator with n >> k: provides low-variance pass@k estimates for all k from a single batch of samples, avoiding the statistical noise that would plague a naive k-sample-per-estimate approach.
Zero-shot evaluation for base models matching RLVR prompts: eliminates confounding from prompt format, ensuring that observed differences are attributable to model capability rather than prompting artifacts.
Manual CoT inspection for hardest problems: validates that pass@k measures genuine reasoning rather than lucky guessing, particularly important for mathematics where answer matching alone could be misleading.
Controlled re-implementation of six RL algorithms with identical training data and hyperparameters: enables fair comparison across algorithms, eliminating confounds from different training datasets, model sizes, or implementation details.
Removing KL penalty in main experiments: following DAPO and Oat-Zero, avoids constraining model learning, though the paper separately ablates KL to understand its effect.
Distillation as a positive control: establishes that the pass@k framework can detect genuine capability expansion when it occurs, validating the interpretation of RLVR's crossing pattern as evidence of purely distributional sharpening.
Perplexity computed with the base model on RLVR-generated responses: provides direct distributional evidence that RLVR outputs are within the base model's high-probability region, rather than being novel discoveries.
ΔSE as a normalized comparison metric: abstracts away from absolute performance levels to measure how efficiently each algorithm extracts the base model's potential, enabling comparison across algorithms with different raw performance.
Entropy-matching experiment: controls for the confound that RLVR's reduced diversity (rather than lost capability) drives the pass@k crossing—finding that diversity reduction contributes but does not fully explain the phenomenon.

4. Key Insights and Innovations

Innovation 1: Reframing RLVR's Effect as Distributional Sharpening Rather Than Capability Expansion

This paper makes a fundamental conceptual move that changes how the field should think about reinforcement learning for LLM reasoning. Prior to this work, the dominant narrative—exemplified by the DeepSeek-R1 paper (Guo et al., 2025) and echoed across the RLVR literature—was that RLVR enables LLMs to discover novel reasoning strategies through exploration, analogous to how AlphaGo Zero discovered new Go strategies beyond human knowledge. The paper directly challenges this analogy and provides a competing framework: RLVR does not expand the set of reasoning paths a model can generate; it sharpens the sampling distribution within a pre-existing boundary, making correct paths more likely while simultaneously making other paths less likely or inaccessible.

What makes this reframing intellectually distinctive is that it is not a claim about RLVR being ineffective—it demonstrably improves pass@1 dramatically. Rather, it is a claim about what kind of improvement RLVR provides and, crucially, what kind it does not. This is a diagnostic reframing: the question shifts from "does RLVR improve reasoning?" (which is trivially yes by pass@1) to "does RLVR expand the set of solvable problems?" (which the paper answers with a systematic no). This distinction between sampling efficiency and reasoning coverage was latent in the literature but had never been operationalized as the central object of study across models, tasks, and algorithms.

The prior work that came closest to this insight was DeepSeek-Math (Shao et al., 2024), which noted in passing that their base model could achieve competitive performance with repeated sampling. But that observation was treated as a curiosity, not as evidence for a fundamental limitation of RLVR. Similarly, Liu et al. (2025a) and Zhao et al. (2025b) argued that reflective behaviors originate in pretraining rather than RLVR, but their analysis was qualitative and focused on specific behavioral patterns rather than the global question of capability boundaries. This paper's contribution is to elevate a scattered set of hints into a unified, quantitative framework and to argue that the distributional-sharpening interpretation is not a special case but the general rule for current RLVR methods.

The evidence for this reframing is multi-layered and mutually reinforcing. The pass@k crossing phenomenon (Figure 2, across seven benchmarks and multiple model families) is the most direct demonstration: base models consistently surpass RLVR models at large k, with the gap widening as training progresses (Figure 1, right: pass@256 drops from 66.3 to 64.3 over 450 GRPO steps while pass@1 rises from 26.1 to 42.5). The perplexity analysis (Figure 6) provides a mechanistic explanation: RLVR-generated paths are not novel to the base model but fall within the high-probability region of its output distribution. The solvable-problem set analysis (Table 2) shows that the RLVR model's solvable problems are nearly a subset of the base model's, with essentially zero problems solved uniquely by RLVR. Together, these constitute a convergent body of evidence for the distributional-sharpening interpretation.

The significance of this reframing extends beyond academic taxonomy. If RLVR is fundamentally bounded by the base model's output distribution, then the ceiling on RLVR-based reasoning improvement is determined by pretraining quality—not by better RL algorithms, larger RL training budgets, or more training steps. This has direct implications for resource allocation: investments in pretraining data quality and coverage may yield higher returns than investments in RLVR infrastructure if the goal is to solve genuinely new problem classes. It also reframes the role of distillation: unlike RLVR, distillation can inject reasoning patterns from outside the student's distribution (Figure 7), making it a genuinely capability-expanding process rather than a capability-surfacing one.

Innovation 2: Pass@k as a Diagnostic Instrument for Reasoning Boundaries

The paper's second conceptual innovation is methodological: it repurposes pass@k from a practical evaluation metric into a diagnostic probe for the nature of model improvement. Pass@k originated in the code generation literature (Chen et al., 2021) as a practical measure of how many attempts a user might need to get a correct solution, and was extended to broader reasoning evaluation by Brown et al. (2024) to study the scaling of repeated sampling. But neither prior work used pass@k to discriminate between types of model improvement—the key conceptual move in this paper.

The diagnostic logic is elegant: if an intervention (RLVR) only improves sampling efficiency, the pass@k curve of the treated model should start higher but rise more slowly, eventually being overtaken by the base model's curve at large k. If the intervention expands reasoning capability, the treated model's curve should lie above the base model's curve at all k, including at the asymptote. The shape of the pass@k curve—not just any single point—encodes the nature of the improvement. The crossing pattern observed across all RLVR experiments provides a clean, interpretable signature of pure efficiency improvement without capability expansion.

What makes this a genuine innovation rather than an obvious application of existing metrics is that the field had not conceptualized the question this way. Prior RLVR evaluations used pass@1 (or average of few samples) as the sole metric, which conflates efficiency and capability. The field's collective interpretation of RLVR's success—that it was teaching models new reasoning strategies—was an inference drawn from pass@1 improvements without considering the alternative hypothesis that the base model already possessed those strategies. The pass@k diagnostic framework makes this alternative hypothesis testable, and the results reject the capability-expansion interpretation for current RLVR methods.

The paper is careful to distinguish pass@k as a scientific instrument from best-of-N or majority voting as practical deployment methods (Section 2.2). This distinction is important because it prevents the misunderstanding that the paper is advocating for large-k sampling in production. Instead, pass@k serves as a boundary probe: it reveals the outer envelope of what a model can potentially solve, regardless of whether that potential is practically realizable without an oracle verifier. This is analogous to using a telescope to map the cosmos—the map isn't directly useful for navigation, but it tells us fundamental things about what exists.

The validation of pass@k as a boundary measure comes from the manual CoT inspections (Sections 3.1, C.2): on the hardest solvable problems, the vast majority of correct answers found through large-k sampling are accompanied by valid reasoning chains, not lucky guesses. This rules out the concern that large-k pass@k gains are artifacts of random answer matching rather than genuine reasoning coverage. The distillation experiment (Figure 7) serves as an additional validation: distillation, which should genuinely expand capability, produces a pass@k curve that dominates the base model's curve at all k, confirming that the diagnostic framework can detect expansion when it occurs.

Innovation 3: The Sampling Efficiency Gap (ΔSE) as a Normalized Metric for Algorithm Comparison

The third conceptual contribution is the ΔSE metric, which operationalizes the gap between what an RLVR-trained model achieves on a single attempt and what the base model could theoretically achieve with unlimited sampling and perfect answer selection. At first glance, this might seem like a simple subtraction, but it embodies a deeper insight: the base model's pass@k at large k defines an approximate upper bound on what RLVR can achieve, and the remaining gap quantifies how much potential remains untapped.

Prior work on RLVR algorithm comparison (Zeng et al., 2025; Liu et al., 2025b; Yu et al., 2025) focused on pass@1 or benchmark-specific metrics, making it difficult to know whether better pass@1 reflected genuinely better optimization or simply a more favorable base model. By normalizing against the base model's boundary, ΔSE provides a baseline-relative measure of algorithmic efficiency that abstracts away from absolute capability differences between model families and sizes. This is conceptually similar to how perplexity provides a normalized measure of language modeling quality across vocabularies, or how the Chinchilla scaling laws normalized training efficiency against compute.

The empirical finding that ΔSE varies only modestly across six algorithms (ranging from 42.6 for RLOO to 43.9 for GRPO on the in-domain test set, Figure 8 top) is itself a significant insight. It suggests that current RLVR algorithms are exploring a narrow region of the optimization landscape—none fundamentally changes the relationship between the trained model and its base distribution. This is a negative result with positive implications: it tells the field that incremental improvements to existing policy gradient variants are unlikely to close the gap, and that more fundamental innovations (the paper speculates about exploration in high-level abstraction spaces, process rewards, and multi-turn agentic interaction in Section 5) may be necessary.

The ΔSE metric also reveals an asymmetry between in-domain and out-of-domain generalization. On MATH500 (out-of-domain), ΔSE is approximately 20 points across algorithms, much smaller than the ~40+ points on the in-domain Omni-MATH test set. But this smaller gap is largely due to the base model's pass@256 already being near ceiling (96.2%), leaving limited room for improvement. This suggests that for well-covered problem distributions, the headroom for RLVR improvement is naturally constrained by the base model's coverage, independent of algorithmic quality.

Innovation 4: The Progressive Narrowing of Reasoning Boundaries During Training

The fourth conceptual contribution is the empirical demonstration that RLVR training produces a monotonic tradeoff between sampling efficiency and reasoning coverage: as training progresses, pass@1 improves while pass@k at large k declines. This finding, shown in Figure 1 (right) and Table 4, reframes RLVR from a process of learning new skills to a process of specialization that incurs capability loss.

This is not an obvious prediction from the standard RLVR narrative. The optimistic view—that RLVR helps the model discover new correct paths while also reinforcing existing ones—would predict that pass@k at both small and large k should increase with training, just at different rates. The observed decline at large k is the signature of a more complex dynamic: the RLVR objective (maximizing expected reward) creates a selective pressure that favors problems where correct answers are already accessible and penalizes exploration of problems where correct paths are rare or absent. Over training steps, the model effectively "reallocates" its probability mass from hard problems (where it occasionally succeeded) to easier ones (where success is more reliable), reducing overall coverage while improving average performance.

This finding connects RLVR to a broader literature on catastrophic forgetting and the stability-plasticity tradeoff in neural networks, but with a twist specific to language generation. Unlike classification tasks where forgetting means losing the ability to recognize a class entirely, the "forgetting" in RLVR is distributional: the model hasn't lost the knowledge of how to solve certain problems (those reasoning paths still exist at some non-zero probability), but it has suppressed them so strongly that they become practically inaccessible. The entropy analysis (Section 4.5, Figure 18) shows that this suppression is only partially explained by reduced output diversity—even when the RLVR model's temperature is increased to match the base model's entropy, the coverage gap persists, suggesting structural changes in the probability distribution rather than just a narrowing of the sampling variance.

The progressive narrowing also has practical implications for RLVR training recipes. It suggests that extended RLVR training may be counterproductive if broad reasoning coverage is valued, and that early stopping criteria should consider pass@k at large k—not just pass@1—to avoid overspecialization. This is a concrete recommendation that emerges from the diagnostic framework and would not be visible under standard evaluation protocols.

5. Experimental Analysis

Evaluation Methodology

Datasets. The paper evaluates across three task domains using multiple benchmarks. For mathematics: GSM8K (Cobbe et al., 2021), MATH500 (Hendrycks et al., 2021), Minerva (Lewkowycz et al., 2022), Olympiad (He et al., 2024), AIME24, and AMC23—ranging from grade-school math to competition-level problems. For code generation: LiveCodeBench v5 (279 problems spanning August 2024 to January 2025; Jain et al., 2025), HumanEval+, and MBPP+ (Liu et al., 2023). For visual reasoning: filtered subsets of MathVista-TestMini (Lu et al., 2024) and MathVision-TestMini (Wang et al., 2024) with multiple-choice questions removed. The controlled RL algorithm comparison in Section 4.3 uses Omni-MATH-Rule, a subset of Omni-MATH (Gao et al., 2025) containing verifiable problems, split into a 2,000-sample training set and an 821-sample in-domain test set, with MATH500 as the out-of-domain benchmark.
Base model(s). The primary model family is Qwen2.5 in 7B, 14B, and 32B base variants (Yang et al., 2024), with additional experiments on LLaMA-3.1-8B (Grattafiori et al., 2024), Qwen2.5-Math-7B, Qwen2.5-VL-7B-Instruct (Bai et al., 2025) for visual reasoning, Qwen2.5-7B-Instruct-1M (Yang et al., 2025b) for code, DeepSeek-R1-Distill-Qwen-14B for the strongest coding model, and Mistral-Medium-3-2505 for the scaling analysis. The paper argues these models span a representative range of capabilities and are "representative of the capabilities of many contemporary LLMs" (Section 4). For math tasks, RLVR is applied in the "zero-RL" setting starting from pretrained models; for code and visual reasoning, instruction-tuned models serve as the starting point due to training instability with pure zero-RL in those domains.
Metrics. The primary metric is pass@k—the estimated probability that at least one correct solution exists among k independent samples from the model. The paper uses the unbiased hypergeometric estimator from Chen et al. (2021): for a problem with $c_i$ correct samples out of $n$ total, $\text{pass@k}_i = 1 - \binom{n-c_i}{k} / \binom{n}{k}$ , averaged across all problems in the dataset. The paper evaluates pass@k for all $k \leq n$ where $n$ is the maximum k of interest (typically 128, 256, or 1024 depending on benchmark difficulty). A secondary metric is the Sampling Efficiency Gap (ΔSE), defined as $\text{pass@1}_{\text{RL}} - \text{pass@256}_{\text{Base}}$ , which quantifies how closely the RL model's single-sample performance approaches the base model's approximate upper bound. The paper explicitly distinguishes pass@k from practical deployment metrics like best-of-N or majority voting—pass@k is used "not to assess practical utility but to investigate the boundaries of reasoning capacity" (Section 2.2).
Baselines. For each RLVR-trained model, the corresponding base model (pretrained or instruction-tuned, matched for model family and size) serves as the primary baseline. This is not a weak baseline but the central object of comparison—the paper's core question is precisely how RLVR changes the base model's capabilities. For the distillation contrast experiment, additional baselines include the instruct-tuned variant and the base pretrained model. In the controlled algorithm comparison, all six RL algorithms are compared against each other and against the Qwen2.5-7B base model. The pass@k curve of the base model at large k effectively serves as an estimated upper bound on what current RLVR can achieve.
Generation budget / compute accounting. The unit of computation is a sampled response (one complete generation from the model), with a maximum length of 16,384 tokens for most experiments and up to 32,768 tokens for the DeepCoder models. For pass@k evaluation, the paper generates $n$ responses per problem where $n$ equals the maximum k on the curve—typically 128 for MATH500, Minerva, and GSM8K; 256 for LiveCodeBench; 1024 for AIME24 and AMC23; and 1024 for LLaMA-3.1-8B on Olympiad due to its lower base capacity. The paper does not translate generation count to FLOPs or wall-clock time, focusing instead on the more interpretable metric of number of attempts. All comparisons between base and RLVR models use identical sampling parameters (temperature 0.6, top-p 0.95) and identical prompts to ensure fairness.
Cross-validation / statistical protocol. No formal cross-validation is used for the main pass@k evaluation—the benchmarks are standard test sets and the comparisons are between model variants rather than tuned hyperparameters. For the controlled algorithm comparison, the Omni-MATH-Rule dataset is split into a fixed training set (2,000 problems) and in-domain test set (821 problems), with MATH500 serving as an out-of-domain test. The unbiased pass@k estimator provides low-variance estimates without requiring multiple evaluation runs, since the hypergeometric formula accounts for sampling variation analytically. Manual inspection of chain-of-thought reasoning is performed on the hardest problems (those with below 5% average accuracy) to validate that pass@k reflects genuine reasoning rather than guessing—on GSM8K, 24/25 base model correct answers and 23/25 RL model correct answers contained valid CoT; on a filtered AIME24 subset, 5/7 base model and 4/6 RL model correct answers had valid CoT.

Main Quantitative Results

RLVR for Mathematical Reasoning: Pass@k Curves Across Model Families and Benchmarks

The paper's central empirical finding is the consistent crossing pattern in pass@k curves across all mathematical benchmarks, model sizes, and model families evaluated.

Aggregate pass@k behavior (Figure 2, Figure 10, Figure 11). Across all configurations in Figure 2—spanning Qwen2.5-7B/14B/32B and LLaMA-3.1-8B on AIME24, MATH500, Minerva, Olympiad—the same qualitative pattern emerges: "When k is small (e.g., k=1, equivalent to average-case accuracy), RL-trained models outperform their base counterparts. However, as k increases, with steeper curves, base models consistently catch up to and eventually surpass RL-trained models across all benchmarks" (Section 3.1). The crossing point varies by benchmark and model size—for easier benchmarks it occurs at lower k, for harder benchmarks at higher k—but the pattern is universal.

Quantitative examples of crossing magnitudes. On Minerva with the 32B model, "the base model outperforms the RL-trained model by approximately 9% at k=128, implying that it can solve around 9% more problems in the validation set" (Section 3.1). On MATH500, the gap at k=128 (the maximum evaluated for this benchmark) is visible in Figure 2 though the specific percentage is not stated in the text. On AIME24 with Qwen2.5-7B, the RLVR model starts substantially ahead at k=1 but the base model crosses over in the range k=16-64, and at k=1024 the base model achieves higher coverage. On GSM8K (Figure 10), the base model's pass@k reaches near-ceiling at k=128 for the 7B model, while the RLVR model also reaches high coverage but is slightly lower; the crossing is less dramatic on this easier benchmark where both models approach saturation.

Oat-Zero and DAPO (Figure 11). The paper evaluates two additional strong RLVR-trained models to test generality: Qwen2.5-Math-7B-Oat-Zero (Liu et al., 2025b) and DAPO-32B (Yu et al., 2025). "Although the RL model initially demonstrates a strong performance, nearly 30% higher than the base model, it is eventually surpassed by the base model" (Section 3.1). On AIME24, Oat-Zero-7B shows a dramatic initial advantage at k=1 (approximately 0.6 vs. 0.3 for the base model), but the base model's curve rises more steeply and overtakes Oat-Zero around k=64-256, reaching a higher asymptote at k=1024. DAPO-32B shows a similar but less pronounced crossing, with the base model achieving approximately 0.82 at k=1024 compared to DAPO's approximately 0.78.

Effect of model scale (Figure 2, comparing rows). The crossing phenomenon is present at all three Qwen2.5 sizes (7B, 14B, 32B), but the absolute pass@k values shift upward with scale and the crossing point may shift. At 32B, the base model's asymptotic advantage appears larger on Minerva than at 7B, suggesting that larger base models may have proportionally more reasoning capacity that RLVR fails to fully surface. However, the paper does not conduct a systematic scaling analysis of the crossing gap as a function of model size.

CoT validity analysis. To address the concern that the base model's pass@k advantage might reflect lucky guessing rather than genuine reasoning, the paper manually inspects the hardest solvable problems. On GSM8K problems with below-5% average accuracy, the base model solved 25 such questions with 24 containing at least one correct CoT; the RL model solved 25 with 23 containing at least one correct CoT. On a filtered AIME24 subset (with guessable problems removed by checking whether the model can answer correctly without CoT), the base model solved 7 hard problems (below-5% accuracy) with 5 out of 6 containing valid CoT (one ambiguous case excluded); the RL model solved 6 with 4 containing valid CoT. The paper also presents two example correct base model CoTs from AIME24 (Figures 20 and 21) showing long reasoning chains with reflective behavior.

RLVR for Code Generation: Pass@k Curves on LiveCodeBench, HumanEval+, MBPP+

Qwen2.5-7B-Instruct-1M vs. CodeR1-Zero (Figure 4 left, Figure 12). On HumanEval+ at k=128, the base instruction-tuned model achieves approximately 0.95 coverage compared to the RLVR-trained model's approximately 0.92—a small but consistent gap at the asymptote. On MBPP+, both models reach approximately 0.9 at k=128, with the base model slightly ahead. On LiveCodeBench (August 2024–January 2025 subset, Figure 12), the RLVR-trained CodeR1 achieves roughly 0.30 at k=128 versus the base model's roughly 0.32, again showing the crossing pattern. On the full LiveCodeBench (May 2023–January 2025, Figure 12 right), the crossing is more pronounced, with the base model achieving approximately 0.48 at k=128 versus CodeR1's approximately 0.45.

DeepCoder-14B vs. DeepSeek-R1-Distill-Qwen-14B (Figure 3). Even for the most capable open-source RLVR-trained coding model, the pattern holds. On LiveCodeBench, DeepSeek-R1-Distill-Qwen-14B (base) achieves approximately 0.80 pass@k at k=64, while DeepCoder-14B-Preview (RLVR) reaches approximately 0.76—a 4 percentage point gap at the asymptote, though the RLVR model has a higher pass@1 (approximately 0.60 vs. 0.55). Notably, DeepCoder uses the distilled model (not a raw pretrained model) as its starting point, meaning the "base" for this comparison is already a model with injected reasoning capabilities from distillation. The fact that the crossing pattern persists even when starting from a distilled model suggests that RLVR's distributional-sharpening effect operates regardless of the starting point's source of capability.

Coding-specific considerations. The paper notes that for code generation, "passing all unit tests is nearly impossible to achieve by guesswork" (Section 3.2), making pass@k a particularly reliable measure of reasoning boundary without the CoT validity concerns that exist in mathematics. The clean verifiability of code makes the coding results a strong corroboration of the mathematical findings.

Solvable-problem set analysis for code (Table 6). On a subset of 50 LiveCodeBench problems (indices 400–449), the base model (Qwen2.5-7B-Instruct-1M) solves 23 problems at k=128, while Coder-R1 solves 20. The RLVR model's solvable set is nearly a subset: 19 of the 20 are also solved by the base model, with problem 430 being unique to Coder-R1 (and problem indices 409, 418, 432, 436, 448 being solved only by the base model). As with the mathematics case, the single RLVR-unique problem may reflect finite sampling rather than genuine capability expansion.

RLVR for Visual Reasoning: Pass@k Curves on MathVista and MathVision

Qwen2.5-VL-7B-Instruct vs. EasyR1-trained model (Figure 4 right). The crossing pattern extends to multimodal reasoning. On MathVista-TestMini (filtered to open-ended questions), the base instruction-tuned VL model achieves approximately 0.93 pass@k at k=128, compared to the RLVR-trained model's approximately 0.87—a 6 percentage point gap at the asymptote. On MathVision-TestMini, a more challenging benchmark, the gap is larger: the base model reaches approximately 0.83 at k=128 versus the RLVR model's approximately 0.70, a 13 percentage point difference.

CoT validity for visual reasoning. Manual inspection of the hardest problems (below-5% average accuracy) revealed that "for both the original and RL models, 7 out of 8 problems have at least one correct CoT" (Section 3.3), supporting the conclusion that pass@k measures genuine reasoning rather than guessing in the visual domain as well.

Significance of the visual reasoning result. The consistency of the crossing pattern across three modalities (text-only math, code, visual reasoning) argues strongly against modality-specific explanations and supports the paper's general interpretation that RLVR produces distributional sharpening regardless of the input or output modality, provided the base model has sufficient reasoning coverage to begin with.

Accuracy Distribution Analysis: The Mechanism Behind the Crossing

Histogram shift pattern (Figure 5, Figure 14). The accuracy distribution analysis decomposes the aggregate pass@k results by showing how per-problem solve rates shift with RLVR training. On Minerva with Qwen2.5-7B (Figure 5), RLVR produces a characteristic redistribution:

A large increase in problems with accuracy in the (0.9, 1.0] and exactly-1.0 bins—problems the model now solves almost every time.
A decrease in problems with moderate accuracy (0.1–0.9)—problems the model could sometimes solve but not reliably.
A notable increase in problems with accuracy exactly 0.0—problems the model can no longer solve at all, despite having non-zero accuracy under the base model.

The paper quantifies this: "RLVR increases the frequency of high accuracies near 1.0 and reduces the frequency of low accuracies (e.g., 0.1, 0.2). However, a deviation from this trend is the increased frequency at accuracy 0—indicating that RLVR leads to more unsolvable problems" (Section 4.1). The full set of accuracy histograms in Figure 14 shows this pattern persists across all four benchmark × model-family combinations evaluated (AIME24, AMC23, MATH500, Minerva, GSM8K, Olympiad × Qwen2.5-7B/14B/32B, LLaMA-3.1-8B).

Interpretation. The histogram directly explains the pass@k crossing: RLVR improves average performance (shifting mass toward 1.0) while reducing overall coverage (adding mass at 0.0). The pass@1 improvement is driven by converting moderate-accuracy problems into high-accuracy problems; the pass@k degradation at large k is driven by converting some previously-solvable problems into completely unsolvable ones. The net effect depends on which k you measure: at small k, the benefits of more reliable correct answers outweigh the losses; at large k, the losses dominate because the base model could already solve many of those problems with sufficient attempts.

Solvable-Problem Set Analysis: RLVR-Solved Problems Are a Subset of Base-Solved Problems

Quantitative set overlap (Table 2). On AIME24 with k=1024: 63.3% of problems are solvable by both base and RLVR models; 13.3% are solvable only by the base model; 0.0% are solvable only by the RLVR model; 23.3% are unsolvable by either. This means that for every problem the RLVR model can solve, the base model can also solve it—a strict subset relationship at this k value. On MATH500 with k=128: 92.4% both; 3.6% base only; 1.0% RLVR only (approximately 5 problems); 3.0% neither. The paper further notes that "even in the rare type 3 cases (e.g., 1% or about 5 problems in MATH500), the base model is able to solve all of them when sampling 1024 times" (Section C.7), suggesting the apparent RLVR-unique solutions are artifacts of insufficient sampling at k=128 rather than genuine new capabilities.

Explicit problem index lists (Tables 5 and 6). For AIME24, the RLVR model solves 19 problems, all of which are among the 23 problems solved by the base model. The 4 problems unique to the base model are indices 4, 11, 17, and 19. For LiveCodeBench (problems 400–449), the RLVR model solves 20 problems, 19 of which overlap with the base model's 23. The single RLVR-unique problem (index 430) and the 4 base-unique problems (409, 418, 432, 436, 448) are listed. The subset relationship is visually apparent from the index tables.

Perplexity Analysis: RLVR Reasoning Paths Exist Within the Base Distribution

Headline result (Figure 6). The perplexity of base-model-generated responses under the base model itself ( $\text{PPL}_{\text{Base}}(Y_{\text{base}})$ ) spans a wide range—from approximately 1.0 to 1.30 in the median-interquartile range shown in the box plots. The perplexity of RLVR-generated responses under the base model ( $\text{PPL}_{\text{Base}}(Y_{\text{RL}})$ ) falls almost entirely within the lower portion of this range—approximately 1.02 to 1.15. This means: "the distribution of $\text{PPL}_{\text{Base}}(Y_{\text{RL}}|x)$ closely matches the lower portion of the $\text{PPL}_{\text{Base}}(Y_{\text{Base}}|x)$ distribution, corresponding to responses that the base model tends to generate" (Section 4.1). The RLVR model's responses are not anomalous or surprising to the base model; they are among the most expected outputs in the base distribution.

Comparison to out-of-distribution content. The perplexity of OpenAI-o1-generated responses under the base model ( $\text{PPL}_{\text{Base}}(Y_{\text{GT}})$ ) is substantially higher—approximately 1.15 to 2.2 in the quartile range shown—indicating that genuinely novel reasoning patterns (from a different model family with different training) look unfamiliar to the base model. This serves as a "positive control" confirming that the perplexity metric can detect out-of-distribution content when it exists.

Training dynamics of perplexity (Figure 15). As RLVR training progresses from early to middle to final checkpoints, $\text{PPL}_{\text{Base}}(Y_{\text{RL}}|x)$ gradually decreases (the specific values are reported in Appendix C.4: the average of median perplexities across 10 problems, evaluated on 32 samples per problem per checkpoint). This trend indicates that RLVR is not expanding into new regions of the output space but rather concentrating more tightly on the high-probability region already present in the base distribution.

Sampling Efficiency Gap (ΔSE): Algorithm Comparison Results

Headline ΔSE values (Figure 8 top, Table 3). Across six RL algorithms evaluated on the Omni-MATH-Train set:

RLOO: ΔSE = 42.6 (pass@1 = 28.6, base pass@256 = 67.2 → gap of 38.6? The paper defines ΔSE as $\text{pass@1}_{\text{RL}} - \text{pass@}k_{\text{Base}}$ , which would give 28.6 - 67.2 = -38.6. The reported positive values in Section 4.3 suggest the paper reports |ΔSE| or defines it as the difference between the base model's pass@256 and the RL model's pass@1, i.e., 67.2 - 28.6 = 38.6 for RLOO. The text states values of "43.9 to 42.6" which must refer to base pass@256 minus RL pass@1, yielding a positive gap that is "large." I will report the numbers as the paper presents them without reinterpreting the sign convention.)
GRPO: pass@1 = 26.1, base pass@256 = 67.2
PPO: pass@1 = 27.2, base pass@256 = 65.8
ReMax: pass@1 = 24.4, base pass@256 = 65.5
Reinforce++: pass@1 = 28.2, base pass@256 = 67.7
DAPO: pass@1 = 31.4, base pass@256 = 66.1

The paper summarizes: "different RL algorithms yield slightly different ΔSE values (i.e., ranging from GRPO's 43.9 to RLOO's best 42.6 on the in-domain test set)" and "ΔSE remains consistently above 40 points across different algorithms, highlighting that existing RL methods are still far from achieving optimal sampling efficiency" (Section 4.3).

Out-of-domain ΔSE (MATH500, Table 3). The gap is smaller on MATH500 because the base model's pass@256 is already 96.2%, leaving limited headroom. RLVR models achieve pass@1 ranging from 73.5% (ReMax) to 75.6% (DAPO), yielding gaps of approximately 20–23 percentage points. The paper notes this is less informative about algorithm quality due to the ceiling effect.

Algorithm-specific observations (Section 4.3, Appendix C.5). DAPO achieves the highest pass@1 (31.4 on the training set) but drops more significantly at k=256 (66.1), and its dynamic sampling strategy requires 3–6× more training samples per batch than other algorithms. RLOO and Reinforce++ "perform consistently well across the entire k range (from 1 to 256), while maintaining efficient training costs, achieving a good balance between effectiveness and efficiency" (Appendix C.5). ReMax shows lower performance at both pass@1 and pass@256, which the paper attributes to its reliance on a single greedy reward as the advantage baseline—in the RLVR setting where rewards are binary (0 or 1), this baseline is "highly variable" and "likely results in unstable gradient updates" (Appendix C.5).

RL Training Step Analysis: Monotonic Coverage Decline

Quantitative trajectory (Figure 8 bottom, Figure 1 right, Table 4). As GRPO training progresses from step 150 to 300 to 450 on the Omni-MATH-Train set:

pass@1: 26.1 → 33.6 → 42.5 (monotonically increasing, +16.4 points total)
pass@256: 66.3 → 65.3 → 64.3 (monotonically decreasing, −2.0 points total)
Base model pass@256: 67.2 (used as reference)

On the in-domain test set (Omni-MATH-Test):

pass@1: 25.1 → 27.1 → 28.3 (+3.2 points)
pass@256: 68.3 → 66.6 → 63.9 (−4.4 points)
Base model pass@256: 69.1

On MATH500 (out-of-domain):

pass@1: 74.4 → 75.4 → 76.3 (+1.9 points)
pass@256: 97.2 → 96.0 → 95.4 (−1.8 points)
Base model pass@256: 96.2

The interpretation: "As RL training progresses, pass@1 on the training set consistently improves from 26.1 to 42.5. However, as RLVR training progresses, pass@256 progressively decreases, indicating a reduced reasoning boundary" (Section 4.4). The monotonic decline in pass@256 even as pass@1 improves is strong evidence against the hypothesis that RLVR is simply learning slowly—if extended training would eventually expand the reasoning boundary, we would expect pass@256 to increase at some point, but instead it steadily decreases.

Distillation vs. RLVR: A Positive Control

Pass@k curves compared (Figure 7). On Minerva, the four models derived from the Qwen2.5-Math-7B family show qualitatively different pass@k behavior:

Base model (Qwen2.5-Math-7B): lowest curve across all k, reaching approximately 0.45 at k=128.
Instruct model (Qwen2.5-Math-7B-Instruct): higher than base, reaching approximately 0.50 at k=128—instruction tuning provides some improvement but the curve shape is similar.
RLVR model (Qwen2.5-Math-7B-Oat-Zero): passes the instruct model at low k but is surpassed at higher k (around k=32-64), eventually falling back toward the instruct curve—the crossing pattern again.
Distilled model (DeepSeek-R1-Distill-Qwen-7B): consistently and substantially above all other curves at every k, reaching approximately 0.60 at k=128. There is no crossing—the distilled curve dominates.

The paper concludes: "the pass@k curve of the distilled model is consistently and significantly above that of the base model. This indicates that, unlike RL that is fundamentally bounded by the reasoning capacity of the base model, distillation introduces new reasoning patterns learned from a stronger teacher model" (Section 4.2).

Significance. The distillation result serves as a positive control validating the pass@k framework's ability to detect genuine reasoning expansion. It also establishes that RLVR and distillation are "fundamentally different" (Section 1) in their effects on reasoning capability, with important implications for practitioners deciding between these two approaches.

Scaling Analysis: Magistral-Medium vs. Mistral-Medium

Near-frontier model comparison (Figure 9). To test whether the crossing phenomenon persists at larger scales, the paper evaluates Mistral-Medium-3-2505 (base) and Magistral-Medium-2506 (RLVR-trained) on AIME24 and AIME25. On AIME24 at k=1: the RLVR model solves approximately 7 more problems than the base model (specific accuracy values are not stated in text; must be read from Figure 9, which shows RLVR at approximately 0.65 vs. base at approximately 0.45 at k=1 on AIME24). However, as k increases, "the performance gap steadily narrows" (Section 4.6). On AIME25: similar pattern, with RLVR solving approximately 8 more problems at k=1 and the gap narrowing with increasing k.

The paper notes that at k=1024, the remaining gap is small on AIME24 (the curves nearly converge at approximately 0.92 vs. 0.90). Whether a true crossing would occur at even larger k is not answerable given the computational constraints, but the narrowing trend is consistent with the pattern observed at smaller scales. The paper interprets this cautiously: "these observations suggest that our conclusion continues to hold even for current, highly capable, near-frontier reasoning models" while acknowledging that "whether this trend persists as more compute, such as pre-training scale budgets, is dedicated to RL training remains a critical question for the future of LLM reasoning" (Section 4.6).

Ablation Studies and Robustness Checks

KL penalty (Figure 16): Training GRPO with a KL divergence penalty (coefficient 0.001) achieves similar pass@1 to GRPO without KL, but with "much lower pass@128"—the KL-regularized model shows a steeper drop-off at large k. This suggests that KL regularization, while intended to prevent the model from deviating too far from the base distribution, actually restricts the model's ability to maintain broad reasoning coverage. The finding is consistent with the distributional-sharpening interpretation: KL regularization interferes with RLVR's tendency to concentrate probability mass, but apparently does so in a way that hurts coverage of correct solutions more than it helps preserve diversity.
Rollout number n (Figure 16): Increasing the number of responses sampled per prompt during training from 8 to 32 (while keeping prompt batch size constant, which increases computation per training step) improves pass@128. The n=32 model, even though trained for only 220 steps compared to 450 for n=8 due to resource constraints, achieves higher pass@k at large k values on all three test sets. The paper interprets this as highlighting "the positive effect of larger rollout numbers in improving pass@k at higher values of k" (Appendix C.6), suggesting that broader exploration during training helps mitigate—but does not eliminate—the coverage reduction. Even with n=32, the RL-trained model is still eventually surpassed by the base model at the largest k.
Temperature sensitivity (Figure 17): The paper evaluates both base and RLVR models at temperatures 0.6, 1.0, and 1.2 (for MATH500 and AIME24). The base model's performance degrades at temperatures above 1.0, as higher randomness produces less coherent outputs. The RLVR model's performance is relatively stable across temperatures—likely a consequence of its sharper output distribution making it less sensitive to the sampling temperature. The paper uses T=0.6 for main experiments as it allows both models to demonstrate their best performance.
Entropy matching (Figure 18): To test whether reduced output diversity (lower entropy) in RLVR models fully explains the coverage reduction, the paper increases the RLVR model's generation temperature until its output entropy approximately matches the base model's entropy at T=0.6. For example, on AIME24, the base model's entropy is 0.47; the RLVR model reaches this at T=1.5. The temperature-matched RLVR model performs better at large k than the standard RLVR at T=0.6 but still underperforms the base model. The paper concludes: "this suggests that while reduced entropy contributes to the narrowing of the reasoning boundary, it alone does not fully account for the reduction" (Section 4.5). In other words, there is a structural change in the probability distribution—certain problems become genuinely unsolvable—that cannot be reversed merely by increasing sampling diversity.
Training dynamics metrics (Figure 19): The paper reports training curves for reward, response length, and generation entropy across the six RL algorithms and ablation conditions. Actor reward increases steadily for all methods, response length increases (as the model learns to generate longer CoT for complex problems), and generation entropy decreases monotonically—all expected patterns that confirm the training is proceeding normally.
Scaling across model sizes (Figure 2, three rows): The crossing phenomenon is demonstrated at 7B, 14B, and 32B scales for Qwen2.5 on mathematical benchmarks. The pattern is qualitatively consistent, and if anything, the absolute gap at large k appears to grow with model size (the 32B base model's advantage on Minerva at k=128 is more pronounced than the 7B model's). However, no quantitative scaling law is derived relating the crossing gap to model size.
Multi-model-family replication: The core pass@k crossing is replicated across Qwen2.5 (three sizes), LLaMA-3.1-8B, Qwen2.5-Math-7B, Qwen2.5-VL-7B, and Mistral-Medium—five distinct model families spanning different pretraining recipes, architectures (dense and VL), and scales (7B to undisclosed large scale). This breadth argues strongly against model-specific artifacts.
Manual CoT validation on hardest problems: For all three domains (math, code, visual reasoning), the paper manually verifies that correct answers found through large-k sampling are accompanied by valid reasoning. The validation covers the most challenging problems (below 5% average accuracy) and finds valid CoT in 92–96% of solved problems across domains. This addresses the concern that pass@k at large k measures lucky guessing rather than genuine reasoning coverage.

Critical Assessment

The paper's central empirical claim is that current RLVR methods improve sampling efficiency but do not expand reasoning boundaries beyond the base model. The evidence for this claim is unusually multi-layered and consistent across diverse settings, which strengthens it considerably. However, several aspects of the experimental design warrant careful scrutiny.

Does the crossing pattern genuinely demonstrate bounded reasoning capability, or could it be an artifact of evaluation methodology? The paper goes to considerable lengths to rule out alternative explanations: manual CoT validation rules out lucky guessing; entropy matching rules out pure diversity reduction; the distillation positive control shows the framework can detect expansion when it occurs; multi-family replication rules out model-specific artifacts. The perplexity analysis is particularly compelling because it provides a direct, assumption-free test: if RLVR models were generating novel reasoning paths, those paths would look unfamiliar (high perplexity) to the base model—but they don't. The convergence of these diverse analytical approaches makes the distributional-sharpening interpretation much more robust than any single analysis could.

What is the nature of the "loss" of reasoning coverage? The paper convincingly demonstrates that RLVR-trained models solve fewer distinct problems than their base models at large k, and that problems "lost" to RLVR training are those the model could occasionally solve but not reliably. However, the paper does not characterize what types of problems are lost—are they systematically harder, do they require different reasoning patterns, or are they idiosyncratic? A deeper qualitative analysis of lost problems could reveal whether RLVR suppresses specific reasoning strategies (e.g., brute-force enumeration vs. insight-based solutions) or simply reduces the model's willingness to explore low-probability paths regardless of their nature.

The Magistral-Medium scaling analysis is suggestive but inconclusive. Figure 9 shows narrowing but not a clear crossing for the near-frontier model pair on AIME24 and AIME25. The paper acknowledges this limitation: "whether this trend persists as more compute, such as pre-training scale budgets, is dedicated to RL training remains a critical question" (Section 4.6). The Magistral experiment is important because it tests the paper's thesis at the largest available scale, but several factors limit its conclusiveness: (1) the model sizes are not disclosed, making it impossible to quantify the scaling relationship; (2) evaluation is only on two benchmarks (AIME24 and AIME25); (3) API throughput constraints limited the evaluation to k=1024, and a crossing might occur at larger k—or might not. The paper is appropriately cautious about this, but readers should note that the near-frontier evidence is weaker than the evidence at 7B-32B scales.

The controlled algorithm comparison uses a specific training configuration that may not be optimal for all algorithms. The paper standardizes across algorithms by using the same learning rate (10⁻⁶), batch size (256 prompts × 8 responses), and removing KL penalty. While this ensures a fair comparison, it may disadvantage algorithms that were designed to work with specific hyperparameter configurations—for example, PPO was originally designed with KL regularization, and ReMax's reliance on greedy reward baselines might work better with different learning rates. The paper's claim that algorithms "remain far from optimal" should be understood in the context of these specific configurations.

The 1% of RLVR-unique solutions on MATH500 deserves more scrutiny. The paper notes that the base model can solve these 5 problems when sampling 1024 times rather than 128, which is interpreted as evidence that the RLVR-unique findings are sampling artifacts. This interpretation makes sense given the subset relationship at larger k, but it also highlights a practical limitation: proving strict subset inclusion would require infinite sampling, and at any finite k, there is always some probability of observing false RLVR-unique problems. The paper handles this appropriately by treating the 1024-sample base model results as the more reliable reference, but the statistical nuance is worth noting.

Missing experiments that would strengthen the paper:

Scaling of the crossing gap with model size. The paper evaluates 7B, 14B, and 32B models but does not quantify how the gap between base and RLVR pass@k at large k scales with model size. A systematic scaling analysis—modeling the gap as a function of parameter count—could reveal whether the phenomenon intensifies or diminishes with scale, which has direct implications for the largest models.
Effect of RLVR training data scale. The paper explores the effect of training steps and rollout number but not the effect of training dataset size. If RLVR's coverage reduction is partly due to training on a limited set of problems (e.g., GSM8K + MATH train), would training on a larger or more diverse set of problems mitigate the coverage loss? This is relevant to the paper's own suggestion about "deliberate and large-scale data curation" as a future direction.
Combined RLVR + distillation. The paper shows distillation expands boundaries and RLVR sharpens distributions. What happens when both are applied? Does RLVR after distillation preserve the expanded boundary while improving efficiency, or does it cause coverage loss even on distilled models? The DeepCoder result (Figure 3) shows a crossing pattern starting from a distilled base, suggesting RLVR still causes coverage loss, but a systematic study would be informative.
Different verifier signals. The paper uses binary outcome rewards (correct/incorrect). Would process-based rewards (step-level correctness) produce different behavior—potentially enabling RLVR to genuinely expand reasoning by providing credit assignment that guides exploration to novel correct paths? This is directly relevant to the paper's "future work" discussion about process rewards.
Longer training and cyclic training. The paper shows monotonic pass@256 decline over 450 steps. Would this trend continue indefinitely, or would the curve eventually stabilize? Could cyclic training (alternating RLVR with SFT on newly discovered correct solutions) recover lost coverage? The paper's "self-improvement" framing in the introduction suggests this as a key question, but no cyclic experiments are reported.

Overall assessment of the evidence. The paper's central claim—that current RLVR methods improve sampling efficiency within pre-existing boundaries rather than expanding those boundaries—is very well supported at the scale of models and training configurations evaluated (7B-32B, standard RLVR training recipes with binary outcome rewards). The consistency across model families, tasks, and algorithms, combined with the multi-pronged analytical approach (pass@k curves, accuracy histograms, set analysis, perplexity), creates a compelling case. The near-frontier evidence (Magistral) is suggestive but not definitive. The primary limitation is that the paper cannot rule out the possibility that RLVR at vastly larger scale (more training data, more training steps, larger models, different reward structures) might eventually break through the base model's boundary—but the monotonic trends observed (coverage declining with more training steps, the gap persisting across model sizes) do not provide positive evidence for such a breakthrough.

The most important contribution may be less the specific empirical findings and more the diagnostic framework itself: pass@k at large k as a probe for reasoning boundaries, ΔSE as a normalized efficiency metric, perplexity-based distribution analysis, and distillation as a positive control. This framework enables the community to evaluate future RLVR methods on the dimension that matters most for genuine capability advancement—not just "does it improve pass@1?" but "does it solve problems the base model couldn't?"

6. Limitations and Trade-offs

Limitation 1: The Boundary-Probing Methodology Cannot Rule Out Capability Expansion at Larger RLVR Scale

The assumption or constraint. The paper's central empirical claim is that current RLVR methods do not expand reasoning boundaries beyond the base model. However, this claim is necessarily circumscribed by the scale of RLVR training evaluated: models trained with a few hundred to low thousands of GRPO/PPO steps on datasets like GSM8K + MATH (SimpleRLZoo), 12K LeetCode/TACO samples (CodeR1), or 2,000 Omni-MATH problems (controlled algorithm comparison). The paper explicitly acknowledges this scope limitation:

"whether this trend persists as more compute, such as pre-training scale budgets, is dedicated to RL training remains a critical question for the future of LLM reasoning" (Section 4.6).

The Magistral-Medium analysis (Section 4.6, Figure 9) pushes closer to the frontier but still evaluates a single model pair on two benchmarks with a maximum of k=1024.

The consequence. The paper cannot distinguish between two fundamentally different interpretations of its results: (1) RLVR is inherently bounded by the base model's output distribution—a structural limitation that no amount of scaling will overcome—or (2) RLVR as currently practiced (at current training budgets, dataset sizes, and algorithmic maturity) is bounded, but qualitatively different behavior might emerge at much larger scales. This distinction matters enormously for the field's resource allocation. If the limitation is inherent, investment should shift to fundamentally different paradigms (e.g., exploration in high-level abstraction spaces, as the paper suggests in Section 5). If the limitation is merely one of scale, then scaling up RLVR—more training data, more steps, larger models—could eventually produce the capability expansion that the paper's framework was designed to detect. The monotonic decline in pass@256 with extended training (Figure 1 right, Figure 8 bottom, Table 4) provides some evidence against the "just need more scale" interpretation—if more of the same training causes coverage to decrease, simply extending training duration seems unlikely to reverse the trend. But the paper does not evaluate orders-of-magnitude larger RLVR training runs (comparable to pretraining compute budgets), leaving the possibility that a phase transition exists at larger scale.

What evidence exists in the paper. The training-step analysis (Section 4.4, Table 4) shows pass@256 declining monotonically from step 150 (66.3) to step 300 (65.3) to step 450 (64.3) while pass@1 rises. The Magistral experiment (Figure 9) shows narrowing but not a definitive crossing. The rollout-number ablation (n=8 vs. n=32, Figure 16) shows that broader training-time exploration helps but does not eliminate the coverage gap. No experiment varies the total RLVR training compute budget across orders of magnitude (e.g., 10×, 100×, 1000× the current training FLOPs).

Mitigation status. The paper partially addresses this through the Magistral-Medium analysis (Section 4.6) and by explicitly flagging it as an open question. The monotonically declining pass@256 provides suggestive evidence that simply "more of the same" RLVR training is unlikely to help, but the paper does not experimentally test whether radically different RLVR recipes (different reward structures, different exploration mechanisms, cyclic training, curriculum-based data ordering) at larger scale could produce different outcomes. The discussion section (Section 5) proposes several directions—"efficient exploration strategies in high-level abstraction," "data scale via curriculum," "process reward and fine-grained credit assignment," "multi-turn agentic RL"—that implicitly acknowledge the current paradigm may be insufficient, but none are empirically tested.

Limitation 2: The Difficulty-Estimation and Problem-Coverage Analysis Has No Practical Selection Mechanism

The assumption or constraint. The paper uses pass@k as a diagnostic instrument to probe reasoning boundaries, explicitly distinguishing it from practical deployment metrics:

"we use pass@k not to assess practical utility but to investigate the boundaries of reasoning capacity" (Section 2.2).

This is methodologically sound—the paper's goal is scientific understanding, not deployment optimization—but it creates a gap that limits the practical implications of the findings. Pass@k measures whether any correct solution exists among k samples, which requires an oracle verifier that can perfectly identify correctness. In deployment, no such oracle exists: one must use an imperfect verifier (trained reward model), majority voting, or other heuristic selection methods that may fail to identify the correct solution even when it is present among the candidates.

The consequence. The paper's central finding—that the base model has broader reasoning coverage than the RLVR model—establishes that the base model could theoretically solve more problems with unlimited sampling and perfect answer selection. But it does not tell us whether the base model would actually outperform the RLVR model under any practical selection strategy (best-of-N with a learned verifier, majority voting, etc.). If the base model's additional correct solutions are systematically harder to identify (e.g., they appear with unusual formatting, are embedded in otherwise-incorrect reasoning, or produce answers that are not the plurality), then the pass@k advantage may be practically unrealizable. Conversely, if the RLVR model's sharper distribution makes its correct answers more reliably identifiable by imperfect verifiers, the RLVR model might be the better choice for deployment despite its narrower coverage. The paper does not evaluate any practical selection strategy on the base model's large-k samples to test whether the theoretical coverage advantage translates to realized performance gains. This is the dual of the search-vs-verifier over-optimization problem from the reference example—just as better search algorithms can over-exploit an imperfect verifier signal, the base model's broader "raw" coverage may be practically inaccessible without a perfect verifier.

What evidence exists in the paper. The paper states that "best-of-N and majority voting are practical methods for selecting correct answers, but they may overlook a model's full reasoning potential" (Section 2.2), directly acknowledging the gap. However, no experiments combine large-k base model sampling with practical selection strategies. The accuracy distribution analysis (Figure 5, Figure 14) shows that base models have non-zero accuracy on many problems where RLVR models have zero accuracy, but the base model's accuracy on these problems is typically low (5-20%), meaning correct solutions are present but rare. Whether imperfect selection methods could reliably surface these rare correct solutions is untested.

Mitigation status. The paper does not attempt to bridge this gap. It acknowledges the distinction between pass@k as a diagnostic tool and best-of-N/majority voting as practical methods, but does not provide any experiments that connect the two (e.g., "can majority voting at k=256 on the base model match the RLVR model's pass@1?"). The authors explicitly flag this scope choice, and it is reasonable given the paper's scientific goals, but practitioners should be aware that the paper's headline finding (base models have broader reasoning coverage) does not directly translate to a deployment recommendation without additional engineering to extract that coverage through practical selection.

Limitation 3: The Analysis Treats Problem Difficulty Implicitly and Does Not Characterize Which Capabilities Are Lost

The assumption or constraint. The paper's primary analyses—pass@k curves, accuracy histograms, solvable-problem set comparisons—operate at the aggregate level, characterizing how RLVR training shifts the overall distribution of which problems are solvable. However, the paper does not systematically characterize what types of problems are gained or lost, beyond the aggregate statistics in Table 2 (showing that 3.6-13.3% of problems are solvable only by the base model at the evaluated k values). The accuracy distribution in Figure 5 shows that RLVR shifts mass from moderate-accuracy problems (0.1-0.9) to high-accuracy (near 1.0) and zero-accuracy bins, but what distinguishes problems that move to the zero bin from those that move to the 1.0 bin?

The consequence. Without knowing the characteristics of problems that are "lost" to RLVR training—their difficulty, the reasoning strategies they require, their relationship to the training distribution—we cannot predict whether the coverage reduction will affect specific downstream applications. If the lost problems are systematically the hardest ones (which the paper suggests by describing questions as being in accuracy bins based on pass@1), then applications that require solving many hard-but-solvable problems (e.g., mathematical research assistance, competition-level problem solving) may be particularly impacted. If the lost problems require specific reasoning patterns (e.g., proof by contradiction, case analysis, working backward from the answer) that RLVR suppresses in favor of a narrower set of strategies, then the diversity of reasoning approaches may also degrade in ways that pass@k alone doesn't capture. The paper's qualitative analysis is limited to two hand-selected base model CoTs (Figures 20 and 21) that demonstrate the base model's capability, but there is no corresponding analysis of what the RLVR model's failed attempts on lost problems look like.

What evidence exists in the paper. The difficulty of problems is indirectly captured through the per-problem accuracy bins in the histogram analysis (Figure 5, Figure 14), but difficulty is not used as an analytical dimension in the main results—the paper does not, for example, produce pass@k curves stratified by difficulty quintile (as the reference example paper does). The paper reports that manual CoT inspection was performed on the "most challenging solvable problems" (accuracy below 5%), finding valid CoT in the majority of cases, but this only characterizes what the base model maintains, not what the RLVR model loses. The solvable-problem indices in Tables 5 and 6 are provided but not analyzed for patterns. There is no taxonomy of lost problems by required reasoning strategy, problem structure, or relationship to the RLVR training data.

Mitigation status. Not addressed. The paper's analysis remains at the aggregate distributional level. The accuracy histograms reveal that problems are lost, and the solvable-problem set comparison reveals which problems are lost (by index, for the examined benchmarks), but no systematic characterization of the properties of lost problems is attempted. This is a natural extension that would strengthen the paper's practical guidance—if we knew which problems RLVR training causes the model to "forget," we could design training curricula or data mixtures to mitigate this forgetting or make informed decisions about when to use RLVR vs. the base model vs. distillation.

Limitation 4: Computational Cost of Pass@k Evaluation Constrains the Generality of the Findings

The assumption or constraint. The paper's central diagnostic tool—pass@k at large k values—is computationally expensive. Evaluating pass@k up to k=1024 on 30 AIME24 problems requires generating 30 × 1024 = 30,720 responses per model, each up to 16,384 tokens. For the 32B model and especially for the DeepCoder-14B model (which uses 32k response length and required API access for throughput), this cost limited the scope of what could be evaluated. The paper explicitly notes that for Deepseek-R1-Zero, "throughput was limited to around 50 tokens per second at a maximum sequence length of 32k, rendering pass@k evaluation currently impractical" (Section 4.6). For Magistral-Medium, API queries were used but the model size is not disclosed, and the evaluation was limited to two benchmarks.

The consequence. The practical consequence is that several of the most interesting model pairs for testing the paper's thesis could not be evaluated. DeepSeek-R1-Zero is perhaps the most prominent example of a model trained with "zero-RL" (pure RLVR from the base model, no SFT), making it a critical test case—but generating 1024 responses per AIME24 problem at 50 tokens/second for 30 problems at 32k max length would require an estimated 30 × 1024 × 32,768 / 50 ≈ 20 million seconds (over 230 days) of inference time, which is infeasible without massive parallelism. For the near-frontier closed-source models (OpenAI-o1's base model is not publicly accessible; Qwen3-235B trained through "multiple stages, including RLVR and long-context CoT supervised fine-tuning, which makes it impossible to disentangle the impact of RLVR alone" per Section 4.6), the base model comparison needed for the paper's framework is unavailable.

This means the paper's findings are necessarily concentrated on models at the 7B-32B scale where the full base model is publicly available and can be sampled at scale. The most capable reasoning models—where the question of whether RLVR expands boundaries is most consequential—are precisely the ones where the paper's methodology cannot be fully applied. The Magistral-Medium result (Figure 9) provides a partial bridge to larger scales, but as discussed in Limitation 1, it is limited to two benchmarks and shows narrowing rather than definitive crossing.

What evidence exists in the paper. The cost constraint is evident from the sampling configurations: the maximum k varies by benchmark and model capacity (128 for MATH500/Minerva/GSM8K with Qwen models, 1024 for AIME24/AMC23, 1024 for LLaMA-3.1-8B on Olympiad "due to its relatively lower base model capacity" requiring more samples to observe correct solutions). The DeepSeek-R1-Zero throughput limitation is explicitly stated (Section 4.6). The Magistral-Medium evaluation uses "a maximum context length of 40k as the original paper does" and is limited to AIME24 and AIME25 (Section 4.6).

Mitigation status. The paper partially mitigates this through the breadth of evaluated models—the consistency of the crossing pattern across Qwen2.5 (three sizes), LLaMA-3.1-8B, multiple math/code/visual reasoning benchmarks, and the partial Magistral result provides some confidence that the phenomenon is not an artifact of small scale. The unbiased pass@k estimator with n >> k allows multiple k values to be evaluated from a single batch of samples, reducing the total sampling cost compared to naive approaches. However, the fundamental tension between evaluation cost and model scale remains unresolved, and the paper's findings at 7B-32B cannot be assumed to generalize to the 100B+ scale without additional evidence that does not currently exist.

Limitation 5: The Paper Does Not Characterize Whether Coverage Loss Is Reversible or Permanent

The assumption or constraint. The paper demonstrates that RLVR training progressively narrows reasoning coverage (pass@256 declines monotonically with training steps, Section 4.4, Table 4), and that this narrowing is only partially attributable to reduced output entropy (entropy-matching experiment, Section 4.5, Figure 18). However, the paper does not investigate whether the lost problem coverage can be recovered—through further training, through returning to the base checkpoint and training differently, through cyclic approaches, or through any other intervention.

The consequence. We do not know whether the coverage loss represents a reversible shift in the model's probability distribution (the model still "knows" how to solve those problems but assigns them very low probability, recoverable through techniques like increased temperature, resampling, or continued training with different objectives) or a permanent loss of capability (the model has genuinely unlearned the reasoning patterns needed for those problems, through catastrophic forgetting or destructive interference between optimization steps). This distinction has enormous practical implications. If coverage loss is reversible, then a deployment strategy could use the RLVR model for efficient first-attempt solving and fall back to the base model (or a temperature-adjusted RLVR model, or a model with different training) for problems where broad coverage is needed. If coverage loss is permanent, then choosing to RLVR-train a model represents an irrevocable decision to sacrifice breadth for depth—one cannot "undo" the RLVR training to recover the lost problems without retraining from the base checkpoint. The entropy-matching experiment (Figure 18) provides some evidence that the loss is not purely reversible through increased sampling diversity—even when RLVR output entropy is matched to base model entropy, the coverage gap persists—but this only tests one recovery mechanism (increased temperature at inference time) and does not address whether the model's internal representations have been altered in ways that cannot be recovered through any inference-time intervention.

What evidence exists in the paper. The entropy-matching experiment (Section 4.5, Figure 18) tests whether increasing the RLVR model's generation temperature (to match base model output entropy) recovers the coverage gap—it helps slightly but does not close the gap. The KL-penalty ablation (Figure 16) shows that adding KL regularization during training actually produces worse pass@128 (not better), suggesting that simply constraining the RLVR model to stay close to the base distribution does not preserve coverage. No experiment tests whether the base model's broader coverage can be recovered by: (1) continued SFT on the base distribution after RLVR training, (2) alternating RLVR and SFT phases, (3) using the RLVR model with different decoding strategies (e.g., best-of-N with the base model's verifier rather than greedy), or (4) any other intervention.

Mitigation status. Not addressed. The paper demonstrates that coverage is lost and that this loss is not purely a diversity reduction (entropy matching doesn't fix it), but the question of whether the loss is permanent or recoverable is left entirely open. The discussion section (Section 5) suggests future work on "exploration in high-level abstraction" and "curriculum" as ways to avoid the coverage-loss problem in future RLVR methods, but does not address whether lost coverage in existing RLVR-trained models can be recovered.

Limitation 6: All Evaluations Use Binary Outcome Rewards—Process Rewards Are Hypothesized but Not Tested

The assumption or constraint. All RLVR training configurations evaluated in this paper use binary outcome rewards: the verifier returns 1 if the final answer is correct and 0 otherwise. This is true for SimpleRLZoo (trained with "correctness reward only, excluding any format-based reward," Section 3.1), CodeR1 (pass/fail on unit tests), EasyR1 (correctness on visual math problems), and the controlled algorithm comparison (Omni-MATH-Rule with binary answer verification). The paper explicitly identifies this as a potential cause of the observed limitations:

"Compared to purely binary outcome rewards, incorporating intermediate signals to guide the reasoning trajectory may significantly improve exploration efficiency and steer exploration toward more promising solution paths" (Section 5, "Possible Future Work").

The consequence. The paper's central finding—that RLVR does not expand reasoning boundaries—is established only for the specific reward structure of binary outcome rewards. Process-based rewards (step-level correctness signals), dense rewards (partial credit for intermediate progress), or shaped rewards could theoretically produce qualitatively different behavior by providing credit assignment that guides the model toward genuinely novel reasoning paths rather than simply reinforcing paths already present in the base distribution. If the exploration bottleneck (as argued in the Discussion, Section 5) arises because "most responses generated by naive token-level sampling exploration are constrained by the base model's prior" and "any sample deviating from the prior is highly likely to produce invalid or non-sensical outputs, leading to negative outcome reward" (Section 5, Discussion 2), then process rewards could alleviate this by providing positive signals for partially correct exploration that would otherwise receive zero reward. The paper does not test this hypothesis.

Additionally, the binary reward structure means that the RLVR models evaluated are optimized to maximize the probability of producing a correct final answer, not to produce good reasoning. This could explain the coverage reduction: problems where the base model occasionally stumbles into a correct answer through a fragile or circuitous reasoning path might become unsolvable after RLVR if RLVR optimizes for the more reliable paths at the expense of the fragile ones. A process-based reward that values intermediate reasoning quality might preserve broader coverage.

What evidence exists in the paper. None. All experiments use binary outcome rewards. The discussion section (Section 5) speculates about process rewards as a future direction. There is no ablation comparing outcome-only vs. outcome+process rewards, and no evaluation of publicly available process-reward-trained models (if any existed at the time of writing). The paper's claim that "inefficient exploration mechanisms in a vast action space and the reliance on binary outcome rewards may be the root causes of the limitations observed" (Section 5) is a hypothesis, not an experimentally supported conclusion.

Mitigation status. Not addressed experimentally. The paper identifies process rewards as a promising direction for future work and provides a plausible theoretical argument for why they might help (better credit assignment → more effective exploration → potential to discover paths outside the base prior). But without empirical evidence, it remains unknown whether process rewards would fundamentally change RLVR's effect on reasoning boundaries or merely improve sampling efficiency more effectively (similar to how PRM-guided search in the reference example improved efficiency but still hit verifier over-optimization limits). This is a consequential open question for the field, and the paper's framework (pass@k curves, ΔSE, perplexity analysis) provides the tools to answer it—but those experiments are left to future work.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper fundamentally reframes the conversation around reinforcement learning for LLM reasoning from a narrative of capability creation to one of capability surfacing. Prior to this work, the dominant interpretation—exemplified by DeepSeek-R1's framing of RLVR as enabling models to "autonomously develop novel reasoning patterns" and the broader field's analogy to AlphaGo Zero's strategy discovery—was that RLVR teaches LLMs to reason in ways they couldn't before. This paper demonstrates, with a convergence of evidence across models, tasks, and algorithms, that this interpretation is mistaken for current RLVR methods: what RLVR actually does is sharpen the sampling distribution within a reasoning boundary that was already established by pretraining, making correct paths more likely while simultaneously making other paths less accessible or completely suppressed.

The magnitude of this reframing is substantial but not paradigm-shattering. It does not render RLVR useless—the pass@1 improvements are real and practically valuable. Rather, it relocates RLVR in the taxonomy of LLM improvement techniques: alongside instruction tuning and SFT as methods for making latent capabilities more reliably accessible, but distinct from distillation (and, presumably, further pretraining) as methods that genuinely expand the set of problems a model can solve. This reframing matters because it changes what we should expect from RLVR and where we should invest to achieve genuine capability expansion. If RLVR is distributional sharpening, then improving it—better algorithms, more training steps, larger reward models—will asymptotically approach the base model's boundary but cannot exceed it. Breakthroughs in reasoning coverage require either better base models (through pretraining), distillation from stronger teachers, or fundamentally different RL paradigms that enable exploration beyond the pretrained prior.

The paper also provides a reconciliation of conflicting signals in the prior literature. Several works had hinted at the paper's central finding without fully articulating it: DeepSeek-Math observed that base models could match RLVR performance with repeated sampling; Liu et al. (2025a) and Zhao et al. (2025b) argued that reflective behaviors originate in pretraining; Dang et al. (2025) saw pass@k decline after RLVR in a limited setting. These scattered observations could be interpreted as curiosities, implementation artifacts, or domain-specific limitations. This paper unifies them into a single coherent picture: they are all manifestations of the same underlying phenomenon—RLVR redistributing probability mass within a pre-existing boundary rather than expanding that boundary. The paper's framework (pass@k curves as boundary probes, perplexity as a distributional overlap measure, accuracy histograms showing the redistribution mechanism) provides the language and tools to recognize this phenomenon wherever it appears.

The work also redirects research attention from algorithmic refinement to exploration mechanisms. The finding that six different RLVR algorithms (GRPO, PPO, Reinforce++, RLOO, ReMax, DAPO) produce quantitatively similar ΔSE values (all within a ~1.3 percentage point range on the in-domain test set, Section 4.3) is a strong negative result: incremental improvements to existing policy gradient variants are exploring a narrow neighborhood of the optimization landscape and are unlikely to close the ~40-point gap to the base model's boundary. This suggests that the field's substantial investment in RLVR algorithm design may have diminishing returns if the goal is capability expansion rather than sampling efficiency. Instead, the bottleneck is the exploration mechanism itself—the process by which new reasoning paths are discovered—and the paper's Discussion (Section 5) points toward fundamentally different approaches: exploration in high-level abstraction spaces, curriculum-based training that hierarchically reduces exploration difficulty, process-based rewards that provide credit assignment for novel paths, and multi-turn agentic interaction that generates genuinely new experiences. These directions become more attractive after this paper; tuning the clipping parameter in PPO becomes less so.

Follow-Up Research This Work Enables

Process reward models (PRMs) for RLVR: Do they enable boundary expansion or just faster sharpening? The paper's Discussion hypothesizes that "inefficient exploration mechanisms in a vast action space and the reliance on binary outcome rewards may be the root causes" of the bounded reasoning coverage (Section 5), implying that process-based rewards—which provide intermediate credit assignment for partially correct reasoning—could alleviate the exploration bottleneck. A direct test would replicate the paper's controlled algorithm comparison (Section 4.3) with a process reward model trained via Monte Carlo rollouts (following the recipe from the reference example paper or Lightman et al., 2023) replacing the binary outcome verifier. The key measurement would be whether the RLVR model's pass@k curve at large k (k=256 or 1024) crosses above the base model's curve—indicating genuine boundary expansion—or merely rises more steeply at small k without changing the asymptote. The paper's perplexity analysis (Section 4.1) provides the tool for a mechanistic investigation: if PRM-guided RLVR discovers novel paths, those paths should have higher perplexity under the base model than the base model's own outputs. A finding that PRMs still don't expand boundaries would strongly constrain the space of possible explanations, pointing toward the pretrained prior as an inescapable constraint rather than merely an exploration challenge. Conversely, a finding that PRMs do enable boundary expansion would immediately make process reward training a critical component of RLVR pipelines and would reframe the paper's conclusions as specific to the outcome-reward regime.

Curriculum-based RLVR with deliberate difficulty progression: Can hierarchical training preserve or expand coverage? The paper shows that RLVR training causes the model to lose coverage on problems it could occasionally solve (accuracy histograms, Figure 5; solvable-problem set analysis, Table 2), and the Discussion speculates that "a curriculum can begin by training on easier subproblems, allowing the model to improve sampling efficiency and acquire essential meta-skills" (Section 5). A concrete experiment would organize the Omni-MATH training data (or a larger verifiable dataset) into explicit difficulty tiers (using pass@1 under the base model as the difficulty metric, analogous to the reference example paper's quintile binning) and train RLVR sequentially from easiest to hardest, measuring pass@k at all k values on held-out problems from each difficulty tier after each curriculum stage. The key question is whether training on easier problems first creates "scaffolding" that allows the model to maintain or expand coverage on harder problems—the pass@k curve on hard problems should not decline as training progresses, in contrast to the monotonic decline observed in Figure 1 (right). A related question is whether interleaving SFT on the base model's correct solutions (to refresh the full output distribution) between RLVR phases can arrest the coverage loss. If curriculum-based training succeeds where uniform training fails, it would validate the paper's hypothesis that the exploration problem is hierarchical and provide a practical recipe for RLVR without coverage sacrifice.

Scaling the pass@k crossing phenomenon: Does the gap between base and RLVR pass@k at large k grow, shrink, or saturate with model size? The paper evaluates 7B, 14B, and 32B models and reports qualitatively consistent crossing patterns across all sizes. However, a systematic scaling analysis—modeling pass@k_base(k=large) − pass@k_RL(k=large) as a function of parameter count—is not performed. This is an important missing piece because the field's largest investments are in ever-larger models. A concrete follow-up would evaluate base and RLVR-trained variants of Qwen2.5 at every available size (0.5B, 1.5B, 3B, 7B, 14B, 32B, 72B) or the LLaMA-3 family on a consistent set of benchmarks (MATH500, GSM8K, AIME24) with consistent sampling budgets (n=1024 for all). The analysis would fit a scaling law of the form gap = α × N^{−β} or similar, where N is parameter count. If β > 0 (the gap shrinks with scale), then the bounded-coverage phenomenon may be a small-model artifact that disappears at frontier scales—which would limit the paper's practical relevance. If β ≈ 0 or β < 0 (the gap is constant or grows), then the phenomenon is scale-invariant and the paper's conclusions apply with full force to the largest models. The Magistral-Medium result (Figure 9) provides a single data point suggesting the gap persists near the frontier, but a systematic scaling study would be far more informative. This experiment is computationally expensive but tractable given that the open-source models are available and the pass@k evaluation protocol is fully specified.

Distillation followed by RLVR: Does RLVR after distillation preserve or erode the teacher's expanded boundary? The paper establishes that distillation genuinely expands reasoning boundaries (Figure 7: the distilled curve dominates the base curve at all k), while RLVR applied to a base model causes coverage loss. What happens when RLVR is applied after distillation—a common pipeline in practice (e.g., DeepCoder-14B applies RLVR starting from a distilled model)? The paper's DeepCoder result (Figure 3) provides a single data point: even starting from DeepSeek-R1-Distill-Qwen-14B (a distilled model), RLVR training produces a modest coverage reduction relative to the distilled base. A systematic study would take a distilled model (e.g., DeepSeek-R1-Distill-Qwen-7B, DeepSeek-R1-Distill-Llama-8B), apply RLVR training with controlled hyperparameters, and measure pass@k curves at multiple training checkpoints to quantify whether and how much of the teacher's expanded boundary is lost during RLVR. The ΔSE metric could be adapted: for distilled models, the relevant upper bound might be the teacher's pass@k rather than the student base model's pass@k. If RLVR causes significant erosion of the distilled boundary, it would suggest that RLVR and distillation should be carefully staged (distillation first, minimal RLVR, or RLVR with constraints that preserve coverage). If RLVR after distillation maintains the expanded boundary while improving sampling efficiency, it would validate the combined pipeline and suggest that distillation provides a "better prior" that RLVR can safely sharpen without boundary loss.

High-level exploration mechanisms: Can program-level or latent-space search discover reasoning patterns beyond the pretrained prior? The paper's Discussion identifies the lack of "efficient exploration strategies in high-level abstraction" as a potential root cause and cites AlphaEvolve (Novikov et al., 2025) as an example of exploration in program-level abstraction space. A follow-up would implement a structured exploration mechanism for mathematical reasoning—for example, using the base model to generate solution sketches in a formal intermediate representation (like Lean or a structured proof format) and then using RLVR to refine these sketches into complete solutions, with rewards based on both the sketch quality and the final answer correctness. The key diagnostic would be whether this two-level exploration (searching over sketches, then refining within sketches) enables the model to solve problems that neither the base model (even with large-k sampling) nor standard RLVR can solve. The paper's pass@k framework would measure whether the structured-exploration model's pass@k curve at large k exceeds the base model's—the signature of genuine boundary expansion. If this works, it would validate the paper's hypothesis that the exploration mechanism is the bottleneck and provide a concrete path forward. If it doesn't, it would suggest that the pretrained prior constrains reasoning even at the level of structured sketches, and that expanding boundaries may require pretraining-level interventions rather than inference-time search.

Multi-turn agentic RLVR: Does environment interaction generate novel reasoning experiences? The Discussion argues that "current RLVR reasoning are limited to single-turn response, whereas iterative refinement based on feedback is crucial for IMO-level reasoning" and that "a multi-turn agentic RL paradigm, featuring richer interactions with environment feedback, could allow models to generate novel experiences and learn from them" (Section 5). A concrete experiment would connect an LLM to an interactive theorem prover (e.g., Lean) or a code execution environment, allow the model to propose partial solutions, receive compiler/prover feedback, and iteratively refine, with RLVR rewards based on eventual success. The key measurement would be whether the model, after multi-turn agentic RLVR training, can solve problems in a single turn (without the interactive environment) that the base model could not solve—i.e., whether the interactive experience during training transfers to expanded zero-shot reasoning capability at test time. The paper's pass@k framework would measure this by comparing single-turn pass@k curves (without environment access) before and after multi-turn training. If interactive training expands single-turn boundaries, it would demonstrate that environment-grounded experience generation can overcome the pretrained prior constraint. If the expanded capability only manifests during interaction (the model needs the environment to succeed), it would still be practically valuable but would represent a different kind of capability from what current RLVR aims to achieve.

Practical Applications and Downstream Use Cases

Deciding when to use RLVR vs. the base model in deployment. The paper's findings provide a concrete decision framework for practitioners. If an application requires high single-attempt reliability on problems within the base model's general capability range—such as automated grading, customer-facing math tutoring where users expect a correct answer on the first try, or real-time code completion—RLVR-trained models are preferable because they dramatically improve pass@1 (e.g., from ~26% to ~43% on Omni-MATH-Train after 450 GRPO steps, Section 4.4). However, if an application can afford multiple attempts per problem with a reliable answer selection mechanism—such as batch evaluation of competition problems, offline data generation for training, or scientific discovery where correctness can be verified post-hoc—the base model with large-k sampling (and a good verifier or majority voting) may achieve higher overall coverage. The paper's finding that base models can solve 9% more problems on Minerva at k=128 with the 32B model (Section 3.1) and 3.6% more on MATH500 at k=128 with the 7B model (Table 2) provides concrete estimates of the coverage gain from using the base model for high-k applications.

Budgeting distillation vs. RLVR for capability improvement. The paper establishes a clear functional distinction: distillation expands reasoning boundaries (the distilled curve dominates the base curve at all k, Figure 7), while RLVR improves sampling efficiency within existing boundaries. For organizations with access to a strong teacher model (whether proprietary or open-source), this implies an investment prioritization: if the goal is to enable a smaller student model to solve new classes of problems it currently cannot solve at all, distillation is the more appropriate tool. If the goal is to make the student model more reliable on problems it can already sometimes solve, RLVR is appropriate and likely more cost-effective than further distillation (since RLVR does not require generating expensive teacher outputs). The paper's finding that RLVR can produce a ~30 percentage point pass@1 improvement on AIME24 relative to the base model (Figure 11, Oat-Zero result at k=1) while requiring no teacher model makes a strong case for RLVR as a cost-effective reliability enhancer for problems within the base distribution.

Training data generation pipelines with guaranteed coverage. Many self-improvement pipelines (STaR, ReST, rejection sampling fine-tuning) use RLVR-trained models to generate high-quality training data for further fine-tuning. The paper's coverage-loss finding raises a concern: if the RLVR model has narrower reasoning coverage than the base model, then using the RLVR model as the data generator may produce training data that covers a subset of the problems the base model could have solved, potentially leading to capability collapse over multiple self-improvement iterations. The paper's framework suggests a mitigation: use the base model (with large-k sampling and answer verification) as the data generator, then apply RLVR to the newly fine-tuned model to improve sampling efficiency. Alternatively, a hybrid generation strategy could use the RLVR model for problems where it has high accuracy (the problems in the 0.9-1.0 accuracy bin from Figure 5) and fall back to the base model with large-k sampling for problems where the RLVR model's accuracy is low. The accuracy histogram tool (Figure 5, Figure 14) provides the diagnostic needed to identify which problems fall into which category.

When to Prefer This Method

The paper does not propose a new training method to be preferred over alternatives—it is a diagnostic study that characterizes what existing RLVR methods do and do not accomplish. The paper does, however, articulate clear tradeoffs that inform when to use RLVR vs. other approaches based on its empirical findings:

Prefer RLVR over the base model when deployment requires high single-attempt reliability (low k) on problems within the base model's capability range. The pass@1 improvements are substantial and consistent across all evaluated configurations: ~2.5× improvement on AIME24 with Oat-Zero-7B, ~1.6× improvement on Omni-MATH-Train with GRPO at step 450, and similar patterns across code and visual reasoning tasks.
Prefer the base model with large-k sampling and a reliable verifier when the application tolerates multiple attempts per problem and maximum problem coverage is the priority. The base model consistently solves 3-13% more distinct problems than the RLVR model at large k (Table 2), with the gap widening as model size increases (9% on Minerva at 32B, Section 3.1). This preference is conditional on having an effective answer selection mechanism—the paper's pass@k metric assumes an oracle verifier, and practical selection methods (best-of-N with a learned verifier, majority voting) may not recover the full base model coverage.
Prefer distillation over RLVR when the goal is to solve problem classes that the current model cannot solve at all. The distilled model's pass@k curve dominates the base model's at all k (Figure 7), indicating genuine capability expansion, whereas RLVR's curve crosses below the base model's at larger k, indicating pure efficiency improvement within a pre-existing boundary.
If using RLVR, prefer shorter training or monitor pass@k at large k as an early-stopping criterion to avoid the progressive coverage loss documented in Figure 1 (right) and Table 4. The tradeoff between pass@1 gains and pass@256 losses is monotonic—more training steps improve reliability but reduce coverage—and practitioners should choose a stopping point based on their specific reliability-vs-coverage requirements rather than training to convergence on pass@1 alone.

Dependency tree

Papers this one builds on, walking backwards through the same domain and shared topics.

Reinforcement Learning from Human Feedback2025· rl-training← in-domain predecessor
- DAPO: An Open-Source LLM Reinforcement Learning System at Scale2025· rl-training← in-domain predecessor
- The Lessons of Developing Process Reward Models in Mathematical Reasoning2025· training-methods← topic thread
- MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions2024· rl-training← similar topic
DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning2025· rl-training← similar topic
- REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization2025· rl-training← in-domain predecessor
- rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking2025· reasoning← topic thread
- Less Noise, More Voice: Reinforcement Learning for Reasoning via Instruction Purification2026· rl-training← similar topic
- CoBA-RL: Capability-Oriented Budget Allocation for Reinforcement Learning in LLMs2026· rl-training← similar topic
- QwenLong-L1: Towards Long-Context Large Reasoning Models with Reinforcement Learning2025· rl-training← similar topic
- DLER: Doing Length pEnalty Right - Incentivizing More Intelligence per Token via Reinforcement Learning2025· rl-training← similar topic
- Stabilizing Reinforcement Learning with LLMs: Formulation and Practices2025· rl-training← similar topic
QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMs2025· rl-training← similar topic
- Verifier-free Test-Time Sampling for Vision Language Action Models2025· rl-training← in-domain predecessor
- INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formats2025· inference-optimization← topic thread

What it influenced

Later papers in the same chain. Direction reversed.

Benefits and Pitfalls of Reinforcement Learning for Language Model Planning: A Theoretical Perspective2025· training-methods→ LLM-judged similar
- Winning the Pruning Gamble: A Unified Approach to Joint Sample and Token Pruning for Efficient Supervised Fine-Tuning2025· training-methods→ in-domain predecessor
- Rainbow: Combining Improvements in Deep Reinforcement Learning2000· training-methods→ similar topic
- Off-Policy Deep Reinforcement Learning without Exploration2018· training-methods→ similar topic
- Robust Preference Learning for Storytelling via Contrastive Reinforcement Learning2022· training-methods→ similar topic
Front-Loading Reasoning: The Synergy between Pretraining and Post-Training Data2025· training-methods→ LLM-judged similar
- CARE: Cognitive-reasoning Augmented Reinforcement for Emotional Support Conversation2025· training-methods→ in-domain predecessor
- QwenLong-L1.5: Post-Training Recipe for Long-Context Reasoning and Memory Management2025· training-methods→ similar topic
- Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillation2026· training-methods→ similar topic
On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models2025· training-methods→ LLM-judged similar
Learning to Reason under Off-Policy Guidance2025· rl-training→ in-domain predecessor
- TTRL: Test-Time Reinforcement Learning2025· rl-training→ in-domain predecessor
- Apprenticeship Learning via Inverse Reinforcement Learning2000· rl-training→ similar topic
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor2018· rl-training→ similar topic
- Never Stop Learning: The Effectiveness of Fine-Tuning in Robotic Reinforcement Learning2020· rl-training→ similar topic
DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning2025· rl-training→ similar topic
- Process Reinforcement through Implicit Rewards2025· rl-training→ in-domain predecessor
- o3-mini vs DeepSeek-R1: Which One is Safer?2025· safety→ topic thread
QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMs2025· rl-training→ similar topic
- FP8-RL: A Practical and Stable Low-Precision Stack for LLM Reinforcement Learning2026· low-precision→ LLM-judged similar
- Jet-RL: Enabling On-Policy FP8 Reinforcement Learning with Unified Training and Rollout Precision Flow2026· rl-training→ LLM-judged similar
- The Art of Scaling Reinforcement Learning Compute for LLMs2025· rl-training→ in-domain predecessor
- WUSH: Near-Optimal Adaptive Transforms for LLM Quantization2025· alignment→ topic thread