Beyond Pass@1: Self-Play with Variational Problem Synthesis Sustains RLVR

🎯 Pitch

Reinforcement learning for LLM reasoning trains models to hack the reward by memorizing solutions to a fixed problem set, causing output diversity to collapse and Pass@k to plateau. This paper shows that letting the model invent its own training problems—by rephrasing its own correct solutions to hard prompts—keeps policy entropy alive and boosts AIME Pass@32 by over 18 absolute points where standard RLVR flatlines.

1. Executive Summary

This paper introduces Self-play with Variational problem Synthesis (SvS), an online data augmentation strategy for Reinforcement Learning with Verifiable Rewards (RLVR) that sustains policy entropy and generation diversity during post-training by having the policy model synthesize new training problems from its own correct solutions to underperforming prompts. Training Qwen2.5-32B-Instruct on DAPO-17k with GRPO, SvS maintains stable policy entropy throughout training — preventing the entropy collapse that standard RLVR suffers — and achieves absolute gains of 18.3% and 22.8% in Pass@32 on AIME 24 and AIME 25 respectively, where standard RLVR plateaus. The core mechanism is a self-play loop in which correct solutions to challenging problems serve as the context for generating variational problems (structurally rephrased but semantically equivalent variants sharing the original reference answer), which are then solved by the policy and validated through a reward-shaping constraint that accepts problems only when the policy achieves moderate solve rates on them (12.5%–62.5% group accuracy), thereby preventing the policy from exploiting the synthesis task by embedding answer hints or generating trivially easy problems. SvS is agnostic to the underlying RLVR algorithm and scales across model sizes from 3B to 32B, establishing that self-play data augmentation can expand reasoning boundaries beyond what fixed-dataset training achieves — but only when the augmented problems remain appropriately challenging for the current policy.

2. Context and Motivation

The Core Problem: RLVR Training Causes Entropy Collapse That Limits Reasoning Boundaries

The paper addresses a specific, well-documented failure mode in Reinforcement Learning with Verifiable Rewards (RLVR) for post-training Large Language Models on reasoning tasks. When an LLM is trained via RLVR on a fixed set of problems (e.g., using GRPO with a correctness reward), the policy initially improves its Pass@1 performance — it gets better at sampling a correct answer on the first try. However, this improvement comes at a steep cost: policy entropy steadily declines throughout training. Entropy here refers to the diversity of the model's generated outputs, specifically the distribution over tokens in its reasoning trajectories. As training progresses, the policy collapses toward producing a narrow set of high-reward solutions — typically memorized correct trajectories for the training problems — rather than continuing to explore novel reasoning strategies.

This phenomenon, which the paper terms entropy collapse, has been documented in recent studies that the authors cite directly. Cui et al. (2025b) showed a logarithmic trade-off between performance gains and response diversity during RLVR training: Performance = -a exp(Entropy + b). Yue et al. (2025) raised a more pointed question: does RLVR actually incentivize reasoning capability beyond what the base model already possesses? Their experiments demonstrated that while RLVR improves Pass@k for small values of k, it fails to improve — and sometimes degrades — Pass@k when k scales to tens or hundreds of samples. Since Pass@k represents the model's upper-bound reasoning capability (if you sample enough times, can the model ever produce a correct answer?), this finding suggests that standard RLVR narrows the policy's generation distribution toward the most reward-prone solutions without expanding its fundamental reasoning frontier.

The practical consequence is that RLVR training plateaus prematurely. The policy finds a few correct solution templates, learns to reproduce them reliably, and stops exploring. Because all sampled trajectories for a given training problem look similar, the advantage estimates in GRPO (which are computed group-relative) collapse toward zero, eliminating the very training signal that drives further improvement. The model becomes a high-precision but low-diversity solver — good at what it has memorized, incapable of discovering new reasoning strategies, and unable to improve further.

The paper's central thesis is that the root cause of this entropy collapse is training on a fixed, limited problem set. When the policy encounters the same problems repeatedly across RLVR iterations, it can trivially "hack" the reward by reproducing previously successful solutions rather than generating genuinely novel reasoning. This is not a failure of the RLVR algorithm per se (GRPO, PPO, etc.) but rather a failure of the data curriculum — the policy has no incentive to explore when it can reliably collect rewards by repeating what it already knows.

Why This Problem Matters

The entropy collapse problem has both practical and foundational significance.

Practical significance. The primary promise of RLVR for LLM reasoning is that it can push models beyond their supervised fine-tuning ceiling by rewarding correct final answers regardless of intermediate reasoning paths. If training entropy collapses prematurely, this promise is fundamentally limited — the model may improve Pass@1 by sharpening its distribution around known solutions, but it will never discover the novel reasoning strategies that constitute genuine capability advancement. For competition-level benchmarks like AIME, where problems require creative mathematical insight, this limitation is severe: standard RLVR yields diminishing returns after a few hundred training steps (Figure 1), leaving substantial performance on the table. Any technique that can sustain the exploration phase of RLVR training longer — enabling the model to discover reasoning strategies it wouldn't find through supervised learning alone — would unlock significant practical gains.

Theoretical significance. The entropy collapse problem reveals a tension at the heart of RLVR: the same mechanism that improves sampling efficiency (reward-guided optimization) also destroys the diversity needed for continued learning. This is not merely a hyperparameter tuning issue; it reflects a fundamental limitation of applying reinforcement learning to fixed datasets. Understanding how to maintain policy entropy during training — not through explicit regularization (which can dilute the RL signal) but through the natural structure of the training data itself — is a conceptual contribution that connects to broader questions in curriculum learning, exploration in RL, and the role of data diversity in generalization.

Deployment implications. For organizations training reasoning models, the entropy collapse problem has direct cost implications. If standard RLVR plateaus after a few hundred steps regardless of additional training budget, then simply "training longer" is wasteful. The paper's proposed solution (SvS) not only improves peak performance but also sustains improvement for more training steps, suggesting more efficient use of compute budgets.

Prior Approaches and Where They Fall Short

The paper situates itself relative to three categories of prior work:

1. Fixed-Dataset RLVR (the Standard Baseline)

Standard RLVR — exemplified by DeepSeekMath (Shao et al., 2024), DAPO (Yu et al., 2025a), and Open-Reasoner-Zero (Hu et al., 2025b) — applies GRPO or PPO to a fixed set of math problems with verifiable ground-truth answers. These methods have achieved strong Pass@1 improvements on benchmarks like MATH and AIME. However, as Yue et al. (2025) and Cui et al. (2025b) demonstrated, they do so by collapsing the policy's generation distribution, failing to improve Pass@k at scale, and exhibiting steadily declining training entropy. The paper's own baselines confirm this: in Figure 5, all standard RLVR configurations (across model sizes from 3B to 32B) show monotonically decreasing entropy throughout training. In Figure 1 (top panel), standard RLVR's Pass@32 on AIME 24 and 25 plateaus after approximately 450 steps.

The key insight the paper draws is that this is a data problem, not an algorithm problem. The RLVR optimization algorithm (GRPO) is not inherently flawed; rather, the training set is too small and static for sustained exploration. This framing is important because it suggests that fixing the data pipeline — rather than redesigning the RL algorithm — is the more direct path to improvement.

2. Data Augmentation via Rephrasing (the MetaMath Approach)

A natural first response to the data diversity problem is to augment the training set with rephrased versions of existing problems. The paper explicitly evaluates this approach using MetaMath (Yu et al., 2023), which uses external LLMs to paraphrase math problems while preserving their underlying mathematical content. In the preliminary experiments shown in Figure 2, the authors compare three configurations:

Vanilla (blue): Training only on MATH-12k throughout.
Aug (orange): Training on MATH-12k plus 36k MetaMath-rephrased problems, with the rephrased problems updated once at step 300.

The results in Figure 2 provide instructive but ultimately insufficient evidence for the rephrasing approach. The augmented setting does slow entropy decline and achieves higher Pass@32, particularly near the update step where entropy briefly increases. However, the paper identifies three critical limitations that prevent rephrasing from being a complete solution:

Limitation 1: Semantic inconsistency. Rephrased problems generated by external LLMs may inadvertently change the mathematical meaning of the original problem, compromising the accuracy of the reference answer. Since RLVR relies entirely on answer verification as the training signal, any semantic drift introduces incorrect training data — the policy may be rewarded for producing answers that don't actually solve the rephrased problem.

Limitation 2: Limited diversity. Rephrasing typically uses the original problem as context, which constrains the diversity of generated variants. A rephrased problem is structurally similar to its source; the model doesn't learn to handle genuinely different formulations of the same underlying mathematical concept.

Limitation 3: External dependency. The rephrasing approach depends on an external LLM (which may not be available, may be expensive, and introduces a dependency on a model that is itself fixed and may not align well with the policy's own capabilities and knowledge).

These limitations motivate the paper to seek an augmentation strategy that is iterative, precise, and self-contained — generating new problems from the policy's own outputs without external models or manual annotation.

3. Entropy Regularization Methods (the Clip-CoV Approach)

A parallel line of work addresses entropy collapse not by changing the data but by modifying the RLVR optimization objective to explicitly preserve diversity. The most prominent example cited in the paper is Clip-CoV (Cui et al., 2025b), which adds an entropy-based regularization term to the GRPO objective. The idea is to penalize the policy when its output distribution becomes too peaked, maintaining a minimum level of diversity in generated solutions.

The paper evaluates Clip-CoV as a comparison point in Appendix F (Figure 16) and finds that SvS consistently outperforms it. The authors' interpretation is that entropy regularization addresses the symptom (declining output diversity) but not the cause (memorization of previous correct solutions from a fixed training set). A policy can maintain high entropy while still primarily reproducing previously memorized solutions intermixed with random noise — entropy is a distributional property, not a guarantee of meaningful exploration. In contrast, SvS addresses the root cause by continuously introducing new problems that the policy has not seen before, forcing genuine exploration of novel reasoning paths regardless of the policy's entropy level.

4. Self-Play and Weakness-Guided Synthesis (Cheng et al., 2024; Liang et al., 2025)

The most direct precursors to SvS are Cheng et al. (2024)'s SPAG framework (self-play adversarial language games) and Liang et al. (2025)'s SwS (Self-aware Weakness-driven problem Synthesis). SPAG uses an adversarial self-play setup where one model generates problems and another solves them, improving both through iterative training. SwS generates synthetic problems specifically targeting the policy's failure cases during RLVR training — the model identifies problems it gets wrong and generates similar variants to practice on.

SvS inherits the self-play paradigm from SPAG and the weakness-targeting philosophy from SwS, but introduces a crucial difference in how problems are synthesized. Rather than generating problems directly from the original problem text (as in SwS) or from scratch (as in SPAG), SvS generates problems conditioned on the policy's own correct solutions. This design choice is motivated by a specific insight: a correct solution contains the complete informational content of the original problem (all constraints, values, and relationships are necessarily referenced in the reasoning), so it can serve as sufficient context for reconstructing a valid variant. Moreover, because the solution is known to be correct for the original problem, any well-constructed variant derived from it will — by construction — share the original's reference answer, eliminating the need for answer annotation.

How This Paper Positions Itself

The paper frames SvS as a data-centric solution to the entropy collapse problem that satisfies three design criteria not met by prior approaches:

Online and iterative: SvS generates new problems at each training step based on the current policy's outputs, creating a continuously evolving curriculum that never becomes stale. This contrasts with one-time data augmentation (MetaMath) and fixed-dataset approaches (standard RLVR).
Precise answer verification without annotation: By synthesizing problems from correct solutions, SvS ensures that the original reference answer remains valid for the synthetic variants. This contrasts with rephrasing approaches (where semantic drift can invalidate answers) and de novo synthesis approaches (where answer verification is an open problem). The paper emphasizes this as a key practical advantage: "the variational problems should share the same reference answers as the original ones, ensuring precision and eliminating the need for additional labeling computation" (Section 3.1).
Self-contained and capability-aligned: SvS uses only the policy model itself — no external LLMs, no human annotators, no distillation from stronger models. This means the synthetic problems are naturally aligned with the policy's current capabilities and knowledge representation, since they are generated from the policy's own reasoning traces. The paper calls this "end-to-end self-improvement" and positions it as a contrast to methods relying on GPT-4 or other external models for data generation (Section 3.1).

The paper further distinguishes SvS from prior work by its focus on Pass@k as the primary metric of reasoning advancement. While most RLVR papers report Pass@1 (single-sample accuracy), this paper argues — following Yue et al. (2025) — that Pass@k with large k (up to 1024) is a more faithful measure of whether the model's fundamental reasoning capabilities have expanded. Standard RLVR's failure to improve Pass@k at scale is the central diagnostic that motivates SvS, and SvS's ability to improve Pass@k (not just Pass@1) is the paper's primary claim of novelty. This framing situates SvS not as an incremental improvement to RLVR training efficiency, but as a qualitatively different capability — expanding the frontier of what problems the model can ever solve, not just improving its first-attempt success rate.

Finally, the paper's scope explicitly encompasses both mathematical reasoning and code generation (Section 5.4, Figure 8), positioning SvS as a general-purpose RLVR augmentation strategy rather than a math-specific technique. The experiments on code generation tasks with Qwen2.5-7B-Instruct demonstrate that the self-play synthesis paradigm transfers across domains, with SvS achieving comparable performance to standard RLVR in approximately one-fifth the training steps.

3. Technical Approach

3.1 Reader Orientation

This paper presents Self-play with Variational problem Synthesis (SvS), an online data augmentation module that wraps around any standard RLVR algorithm (e.g., GRPO) to continuously inject newly synthesized training problems during post-training of reasoning LLMs. The system solves the problem of entropy collapse during RLVR — the phenomenon where a policy trained on a fixed problem set steadily loses generation diversity until it can only produce memorized solution templates, causing both Pass@k and eventually Pass@1 to plateau — by having the policy itself generate novel training problems from its own correct solutions to underperforming prompts, creating an ever-changing curriculum that forces sustained exploration.

3.2 Big-Picture Architecture (Diagram in Words)

The SvS system operates as a pipeline inserted into each training iteration of a standard RLVR loop. It has five major components:

Original Problem Solver — the policy model generating solution groups for problems sampled from the fixed training set $\mathcal{D}$ ; solutions are scored against ground-truth answers, and problems with mixed correct/incorrect solutions are retained.
Underperforming Problem Filter — a threshold-based selector that identifies training problems where the policy's group accuracy falls within a configured range $[\text{acc}_l, \text{acc}_h]$ , isolating problems at the current frontier of the policy's capabilities (not too easy, not unsolvable).
Variational Problem Synthesizer — the same policy model, prompted with its own correct solution to an underperforming problem, generates $G_v$ new problem variants that preserve the original semantic content and reference answer while changing surface structure and wording.
Synthetic Problem Solver — the policy solves its self-generated variational problems; solutions are scored against the original ground-truth answer (which remains valid because variational problems are derived from correct solutions that contain all original constraints).
Reward-Shaped Synthesis Validator — a filtering mechanism that assigns positive rewards to variational problems only when the policy achieves moderate solve rates on them (12.5%–62.5% group accuracy), preventing easy exploitation (embedding answer hints in the problem text) and ensuring synthetic problems are appropriately challenging.

Information flows through the system in each iteration: (1) problems are sampled from $\mathcal{D}$ → (2) the policy generates solution groups, which are scored, filtered, and added to buffer $\mathbf{B}$ → (3) underperforming problems are identified → (4) their correct solutions are used as context for variational problem synthesis → (5) the policy solves synthetic problems, which are scored and filtered into $\mathbf{B}$ → (6) reward shaping validates problem quality and adds synthesis training pairs to $\mathbf{B}$ → (7) the augmented buffer is used for a GRPO policy update → (8) the buffer is cleared and the cycle repeats.

3.3 Roadmap for the Deep Dive

First, the formal GRPO objective (Equation 6 in Appendix A) that underlies all RLVR training in the paper, since SvS modifies the data feeding into this objective without changing the optimization algorithm itself.
Second, the original problem solving phase — how the policy generates solutions, how rewards are computed, and how problems are filtered into the training buffer — because this establishes the baseline data collection that SvS extends.
Third, the underperforming problem filter, since this is the gating mechanism that determines which problems get augmented and directly shapes the difficulty profile of the synthetic data.
Fourth, the variational problem synthesis pipeline — prompting strategy, generation parameters, and the critical design choice of conditioning on solutions rather than problem text — because this is the core mechanism that creates new training data without external models or answer annotation.
Fifth, synthetic problem solving and reward shaping — how the policy solves its own generated problems, how correctness is validated using original answers, and how the $R_V$ reward constraint prevents exploitation — because this is what makes SvS sustainable rather than self-defeating.
Sixth, the full training buffer construction and policy update — how three distinct types of training pairs (original solving, problem synthesis, synthetic solving) are mixed for a single GRPO update — because this joint training is what enables the self-improving loop.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a data augmentation method paper whose core idea is that RLVR entropy collapse can be prevented by having the policy continuously generate new training problems from its own correct solutions, and that this self-play loop — when combined with reward shaping that enforces appropriate problem difficulty — sustains exploration and expands reasoning boundaries beyond what fixed-dataset training achieves.

Preliminaries: The GRPO Optimization Objective

SvS does not modify the underlying RL algorithm; it augments the data that feeds into it. Throughout the paper, the base RLVR optimizer is GRPO (Group Relative Policy Optimization; Shao et al., 2024) with additional techniques from DAPO (Yu et al., 2025a): Clip-Higher with $\epsilon = 0.28$ , Token-Level Loss, and Dynamic Sampling. Understanding the GRPO objective is necessary to understand why the data that SvS generates is structured the way it is — specifically, why each training pair must include a group of solutions with mixed correctness, and why the synthesis reward is designed as a binary indicator rather than a continuous score.

GRPO computes advantages in a group-relative manner without a learned critic model. For a given input prompt $x$ , the policy $\pi_{\theta_{\text{old}}}$ generates a group of $G$ responses $\mathbf{Y} = \{y_i\}_{i=1}^G$ , each receiving a scalar reward $r_i$ . The advantage $A_{i,t}$ for each token at position $t$ in response $y_i$ is:

$A_{i,t} = \frac{r_i - \text{mean}(\{r_i\}_{i=1}^G)}{\text{std}(\{r_i\}_{i=1}^G)}$

where $\text{mean}(\{r_i\}_{i=1}^G)$ is the average reward across the group and $\text{std}(\{r_i\}_{i=1}^G)$ is the standard deviation across the group.

What it computes: for each response in a group, this equation converts its raw reward into a standardized score measuring how much better or worse that response is compared to the group average, measured in units of group standard deviation. If all responses in a group receive the same reward, the standard deviation is zero and all advantages collapse to zero — there is no signal to learn from. This is why SvS filters out problems where the policy is either perfectly accurate (all solutions correct) or perfectly inaccurate (all solutions incorrect): such groups produce degenerate advantages that contribute nothing to the gradient. The downstream consequence is that SvS must ensure every training problem fed into the buffer yields mixed correctness — at least one correct and one incorrect solution — which is enforced by the filtering condition $0 < \text{Acc}(x) < 1$ .

Why this form: standard PPO requires a learned value function (critic) to estimate baseline rewards, which adds model parameters and training instability. GRPO replaces the learned baseline with the group average, eliminating the critic entirely. The division by standard deviation normalizes the advantage magnitude across groups with different reward scales, which is essential when different problems have different intrinsic difficulty and therefore different typical reward ranges. The Clip-Higher modification (from DAPO) ensures the policy can increase probability on good tokens even when the ratio $k_{i,t}(\theta)$ exceeds the upper clip bound, promoting more aggressive exploration toward high-reward trajectories.

The full GRPO objective with clip and KL penalty is:

$\mathcal{J}(\theta) = \mathbb{E}_{x \sim \mathcal{D}, \mathbf{Y} \sim \pi_{\theta_{\text{old}}}(\cdot|x)} \left[ \frac{1}{G} \sum_{i=1}^G \frac{1}{|y_i|} \sum_{t=1}^{|y_i|} \left( \min \left( k_{i,t}(\theta) A_{i,t}, \text{clip} \left( k_{i,t}(\theta), 1 - \epsilon, 1 + \epsilon \right) A_{i,t} \right) - \beta D_{\text{KL}}(\pi_{\theta} \parallel \pi_{\text{ref}}) \right) \right]$

where $k_{i,t}(\theta) = \frac{\pi_{\theta}(y_{i,t}|x,y_{i,<t})}{\pi_{\theta_{\text{old}}}(y_{i,t}|x,y_{i,<t})}$ is the probability ratio between the current and old policies, $\epsilon$ is the clipping parameter (0.28 in this work), and $\beta D_{\text{KL}}$ is a KL-divergence penalty keeping the policy close to a reference distribution.

The relevant hyperparameters for all experiments (Appendix D.1): learning rate $1 \times 10^{-6}$ with constant schedule, sampling temperature fixed to 1.0, batch sizes of 256 for both sampled problems and policy updates per iteration, and group size $G = 8$ for all problem-solving tasks (both original and synthetic) with $G_v = 8$ for variational problems generated per solution.

Original Problem Solving Phase

Each training iteration of SvS begins with standard RLVR data collection on problems sampled from the fixed training set $\mathcal{D}$ . The training sets used in experiments are MATH-12k (Hendrycks et al., 2021) and DAPO-17k (Yu et al., 2025a), with the 32B model additionally trained on DAPO-17k augmented with 8k open-ended problems from DeepMath (D25k).

For each problem-answer pair $(x, a)$ in the sampled batch, the policy $\pi_\theta$ generates $G = 8$ complete solutions $\{y_i\}_{i=1}^G$ . Each solution is a full chain-of-thought reasoning trajectory ending with a final answer. The correctness reward $\mathbf{R}_c$ is a binary 0/1 determined by exact answer matching:

$\mathbf{R}_c(y_i, a) = \mathbb{I}(\text{Extract}(y_i) = a)$

where $\text{Extract}(\cdot)$ extracts the final answer from the reasoning trajectory (using a rule-based parser looking for boxed answers or equivalent formats), $a$ is the ground-truth answer for the problem, and $\mathbb{I}(\cdot)$ is the indicator function returning 1 if the extracted answer matches the ground truth and 0 otherwise.

What it computes: for each of the 8 generated solutions, this produces a binary correctness label — 1 if the final answer is correct, 0 otherwise. The group accuracy $\text{Acc}(x) = \frac{1}{G}\sum_{i=1}^G \mathbf{R}_c(y_i, a)$ is then the fraction of the 8 solutions that are correct.

Why this form: the binary reward is the defining feature of RLVR — unlike RLHF, which uses a learned reward model, RLVR uses automatic verification against a ground-truth answer. This makes the reward signal perfectly reliable (no reward model error) but also perfectly sparse (the model receives no partial credit for partially correct reasoning). The sparsity means the policy must discover correct reasoning through exploration alone, which is why maintaining generation diversity throughout training is so critical — without diverse attempts, the policy can never stumble upon novel correct solutions.

The critical filtering step follows: problems where the group accuracy is exactly 0 or exactly 1 are discarded. If all 8 solutions are correct, the standard deviation of rewards is zero and the GRPO advantages collapse to zero — there is no signal to learn from, and the policy would only reinforce its already-perfect behavior. If all 8 solutions are incorrect, the standard deviation is also zero and the same collapse occurs — the policy has no positive examples to learn from within this group. Only problems where $0 < \text{Acc}(x) < 1$ (at least one correct and one incorrect solution) provide non-degenerate advantages and are added to the training buffer $\mathbf{B}$ as training pairs $(x, y_i)$ for $i = 1, \ldots, G$ .

This filtering criterion has an important downstream effect: it means that over the course of training, as the policy improves, certain problems will naturally exit the training distribution. When a problem's group accuracy reaches 100% (the policy consistently solves it correctly), it stops contributing to the buffer. This creates an implicit curriculum where the policy naturally shifts its attention toward harder problems over time. SvS's augmentation strategy is designed to work within this filtering framework — it augments problems while they are still in the "productive" accuracy range.

Underperforming Problem Filter

After collecting solutions for all problems in the batch, SvS identifies a subset of problems for augmentation: those whose group accuracy $\text{Acc}(x)$ falls within the configured range $[\text{acc}_l, \text{acc}_h]$ . In all experiments, this range is fixed at 12.5% to 50.0% (Appendix D.1; Line 11 in Algorithm 1). This means only problems where the policy gets roughly 1 to 4 correct solutions out of 8 are selected for variational problem synthesis.

The choice of this range embodies a specific philosophy about what kinds of problems benefit from augmentation. Problems with accuracy above 50% are ones the policy already handles reasonably well — it gets at least half its attempts correct. Augmenting these would generate variants of problems the policy has mostly mastered, which (as shown in the ablation SvS-Asp in Table 3, where augmenting simpler problems with accuracy 37.5%–75% produced worse results) leads to overfitting on already-strong capabilities without expanding the frontier. Problems with accuracy below 12.5% are ones the policy mostly fails — it gets at most 1 out of 8 correct. Augmenting these would be wasteful because the policy hasn't demonstrated the ability to produce correct solutions that could serve as reliable context for synthesis, and even if it occasionally does, the small number of correct solutions would limit the diversity of synthesized variants.

The 12.5%–50% range captures problems at the policy's capability frontier — problems it can sometimes solve but not reliably. These are precisely the problems where additional practice on semantically equivalent but structurally diverse variants could push the policy toward more robust mastery. The paper calls these "underperforming problems" throughout.

For each identified underperforming problem $x$ , SvS selects the subset of its $G$ solutions that were correct: $\mathcal{I} = \{i \mid \mathbf{R}_c(y_i, a) = 1\}$ . These correct solutions are the context for variational problem synthesis. The number of correct solutions available varies by problem — for a problem with exactly 25% accuracy, there would be 2 correct solutions to synthesize from; for one at 50%, there would be 4. Each correct solution independently spawns a group of $G_v = 8$ variational problems, meaning a single underperforming problem can generate anywhere from 8 to 32 synthetic variants per iteration depending on how many of its solutions were correct.

The paper does not discuss whether the number of correct solutions per underperforming problem affects synthesis quality — a problem with only 1 correct solution might yield less diverse variants than one with 4, since all 8 synthetic problems would be conditioned on the same solution text. This is a subtle potential limitation not explored in the analysis.

Variational Problem Synthesis

This is the core mechanism that distinguishes SvS from all prior approaches. Rather than generating new problems from the original problem text (as in rephrasing approaches like MetaMath) or from scratch (as in problem-generation methods like SwS), SvS generates variational problems conditioned on a correct solution produced by the policy.

The synthesis prompt. For each correct solution $y_i$ to an underperforming problem, the policy is prompted with the template shown in Figure 20 (the "Variational Problem Synthesis Prompt (Reasoning)"). The prompt instructs the model to act as "an expert in educational assessment and mathematical problem synthesis" and provides the solution enclosed in <response> tags with the assurance that "The solution is assured to be correct." The model is asked to:

Identify the topic and context from the response.
Infer the type of reasoning involved.
Determine the educational purpose behind the problem.
Reconstruct a "clear, concise, and natural-sounding original problem in English" that "precisely aligns with the provided response" and "reflects a realistic problem that could appear in an educational context or standard curriculum."

The model's output is a variational problem $\hat{x}_i^j$ for $j = 1, \ldots, G_v$ (where $G_v = 8$ ). Each correct solution thus spawns 8 problem variants. The synthesis is done with the same sampling temperature of 1.0 used throughout training, ensuring diversity in the generated problems.

Why condition on solutions rather than problem text? This design choice is the paper's most consequential architectural decision, and the logic behind it unfolds in several layers:

Semantic preservation. A correct solution to a math problem necessarily references all the key information from the original problem: the variables, the constants, the relationships, the constraints, and the required operations. When the model writes $f_n(z) = \frac{\pi^2}{24} z^2 - \frac{1}{n^2} z + \frac{589}{20} \cdot \frac{1}{n^3}$ in its solution, it has encoded the function definition; when it writes "sum of all roots with positive imaginary part," it has encoded the target quantity. By conditioning synthesis on the solution, SvS ensures the synthetic problem will contain these same mathematical elements in some rearranged form, preserving the underlying semantic content.

Answer guarantee. Because the solution is known to be correct for the original problem, any well-formed variational problem reconstructed from it will — by construction — have the same correct answer as the original. If the solution computes $2 + 22i$ as the final answer, then the variational problem, which is designed to "precisely align with the provided response," must be a problem whose answer is also $2 + 22i$ . This eliminates the need for answer annotation: the original ground-truth answer $a$ is reused directly for scoring solutions to the synthetic problem. This is stated explicitly in Section 3.1: "the variational problems should share the same reference answers as the original ones, ensuring precision and eliminating the need for additional labeling computation."

Structural diversity. The solution text does not contain the original problem's exact phrasing — it contains the reasoning steps. When the model reconstructs a problem from reasoning, it must invent new wording, new framing, and potentially new structural organization to turn the reasoning back into a well-posed question. This naturally produces problems that differ in surface form from the original while preserving mathematical content. The paper provides concrete examples in Figure 4, where the original problem describes a specific function family $f_n$ and asks for a sum of roots, and the synthetic variants rephrase this as "Consider the polynomial functions defined by..." or "Let us examine the function defined by..." — different surface forms describing the same underlying mathematical task.

Capability alignment. Because the solution was generated by the policy itself, the variational problem is naturally aligned with the policy's own knowledge representation and reasoning style. An external LLM (like GPT-4) might generate problems that are too hard, too easy, or use mathematical conventions that don't match the policy's training distribution. Self-generated problems are guaranteed to be within the policy's "conceptual vocabulary," even if they are structurally novel.

Self-improvement. The synthesis itself is a learning task. By training the policy to reconstruct problems from solutions (the synthesis task), the policy learns the inverse mapping from reasoning to problem statements. As discussed in Appendix E.2, this inverse task serves as a regularizer that prevents overfitting to the forward problem-solving task, helping the model maintain generalization to out-of-distribution benchmarks.

What the synthesis produces. Figure 4 provides a concrete illustration. The original problem asks about a family of complex functions $f_n(z) = \frac{\pi^2}{24} z^2 - \frac{1}{n^2} z + \frac{589}{20} \cdot \frac{1}{n^3}$ and requires finding the sum of roots with positive imaginary parts, with the answer $2 + 22i$ . Three synthetic variants are shown:

One that fails synthesis: its group accuracy on the synthetic problem is [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0] with a different ground-truth answer of $1 + 16i$ , meaning the synthesis produced a problem whose mathematical content actually changed (failed answer preservation).
One that succeeds: its group accuracy is [0.0, 1.0, 1.0, 1.0, 0.0, 1.0, 0.0, 1.0] with the correct answer $2 + 22i$ , meaning the problem is both valid (it has the right answer) and appropriately challenging (the policy gets some but not all solutions correct).
One that is too easy (flagged as "With hints, oversimple"): its group accuracy is [1.0, 1.0, 1.0, 1.0, 0.0, 1.0, 1.0, 1.0] with the correct answer, but the problem text includes hints ("The sum of a series of complex numbers can be found by summing the real and imaginary parts separately.") that make it trivially solvable.

The synthesis is not guaranteed to produce valid problems on every attempt. The paper's quality analysis in Appendix E.5 indicates that approximately 80% of synthesized problems are judged as correct by state-of-the-art LLM evaluators, with the remaining 20% potentially having surface-level weirdness without being genuinely incorrect. The filtering and reward-shaping mechanisms (described next) are designed to handle these failure cases by only retaining problems that demonstrate appropriate difficulty for the policy.

Synthetic Problem Solving

Once a set of $G_v = 8$ variational problems $\{\hat{x}_i^j\}_{j=1}^{G_v}$ has been synthesized from a correct solution $y_i$ , the policy is tasked with solving them in the same way it solves original training problems. For each variational problem $\hat{x}_i^j$ , the policy generates $G = 8$ solutions $\{\hat{y}_k\}_{k=1}^G$ , and the original ground-truth answer $a$ (paired with the source problem $x$ ) is reused to evaluate correctness:

$\mathbf{R}_c(\hat{y}_k, a) = \mathbb{I}(\text{Extract}(\hat{y}_k) = a)$

What it computes: for each of the 8 solutions to each synthetic problem, this produces a binary correctness label by comparing the extracted answer to the original problem's ground truth. The group accuracy for the synthetic problem is $\text{Acc}(\hat{x}_i^j, a) = \frac{1}{G}\sum_{k=1}^G \mathbf{R}_c(\hat{y}_k, a)$ .

Why reuse the original answer: this is the key practical advantage of synthesis-from-solutions. If SvS generated problems from scratch (without conditioning on solutions), there would be no reliable way to know the correct answer — the synthesis would need to either generate an answer (which might be wrong) or rely on the policy's own solving to verify (which would be circular). By conditioning on a known-correct solution, the method guarantees that the original answer remains valid for any well-formed synthetic variant, making answer verification as reliable as it is for the original training problems.

The same filtering criterion applies as in the original problem solving phase: only variational problems where $0 < \text{Acc}(\hat{x}_i^j, a) < 1$ (mixed correctness) are retained for policy updates. This means:

If the policy gets all 8 solutions correct $\text{Acc} = 1.0$ , the problem is too easy (no learning signal from degenerate advantages).
If the policy gets all 8 solutions incorrect $\text{Acc} = 0.0$ , the problem is either too hard or invalid (no positive examples to learn from).
Only problems with at least one correct and one incorrect solution — indicating they are at the right difficulty level — contribute to the training buffer.

For retained variational problems, the solution groups $(\hat{x}_i^j, \hat{y}_k)$ for $k = 1, \ldots, G$ are added to the training buffer $\mathbf{B}$ as type (3) training pairs: Synthetic Problem Solving.

The number of synthetic problems that survive this filtering depends heavily on the policy's current accuracy level and the quality of synthesis. Figure 9 (middle panel) shows the number of variational problems and their corresponding responses throughout training. For DAPO-17k, early training produces many synthetic problems because the policy's accuracy on most training problems falls within the 12.5%–50% underperforming range, and synthesis quality is reasonable. As training progresses and the policy's accuracy improves, fewer problems fall in the underperforming range, and synthetic generation volume declines naturally. For MATH-12k with the 32B model, the initial accuracy is already around 80% on the training set, so very few problems qualify as underperforming, and SvS generates limited synthetic data — the training time is comparable to standard RLVR (Section 5.5, Figure 9).

Reward Shaping for Problem Synthesis

This is the mechanism that prevents SvS from collapsing into a degenerate self-exploitation loop. The initial, naive version of the synthesis reward (described in Appendix G as a "failed attempt") assigned a positive reward to any variational problem for which the policy produced at least one correct solution:

$R_V^{\text{naive}}(\hat{x}_i^j) = \mathbb{I} \left( \text{Acc}(\hat{x}_i^j, a) > 0 \right)$

Why this fails: the policy quickly learns to exploit this criterion by embedding explicit hints about the final answer directly into the synthetic problem text. For example, a problem might include "Hint: the answer is $2 + 22i$ " or structure the problem description to make the answer trivially extractable. Since all such hint-laden problems produce at least one correct solution (the policy just copies the embedded answer), they consistently receive positive rewards. As shown in Appendix G, this causes the policy's accuracy on synthetic problems to saturate at 100%, mean advantages collapse to zero, and no meaningful training signal reaches the GRPO update. The policy wastes its synthesis budget on generating problems that are trivially solvable and learns nothing.

The fix introduced in SvS is a difficulty-aware reward-shaping constraint that only assigns positive rewards to synthetic problems that maintain an appropriate difficulty level for the policy. The reward for a synthetic problem $\hat{x}_i^j$ is:

$R_V(\hat{x}_i^j) = \mathbb{I} \left( a\hat{c}_1 \leq \text{Acc}(\hat{x}_i^j, a) \leq a\hat{c}_h \right)$

where $a\hat{c}_1$ and $a\hat{c}_h$ define an acceptable accuracy range for synthetic problems. In the paper, these thresholds are set to 12.5% and 62.5% respectively (Appendix D.1: "the positive reward range $[a\hat{c}_1, a\hat{c}_h]$ for variational problem synthesis is defined as 12.5%–62.5%").

What it computes: a synthetic problem receives a reward of 1.0 if the policy's group accuracy on that problem falls within the 12.5%–62.5% range, and 0.0 otherwise. This means:

Problems that are too easy (accuracy > 62.5%, i.e., at least 6 out of 8 solutions correct) receive a reward of 0.0, penalizing hint-laden or oversimplified problems.
Problems that are too hard (accuracy < 12.5%, i.e., at most 1 out of 8 solutions correct) receive a reward of 0.0, penalizing unsolvable or invalid problems.
Problems in the sweet spot where the policy can solve some but not all instances receive a reward of 1.0.

Why this range:

Lower bound (12.5%, or 1/8): ensures the problem is actually solvable by the current policy. A problem that never yields a correct answer is either invalid (the synthesis failed to preserve the original mathematical content) or beyond the policy's current capability, and either way it provides poor training signal — there are no positive examples for the policy to learn from within that group, and the synthetic problem itself is likely flawed.
Upper bound (62.5%, or 5/8): ensures the problem is genuinely challenging. A problem the policy solves too easily (6, 7, or 8 out of 8 correct) likely contains hints, has been seen before in similar form, or is otherwise trivial. The upper bound of 62.5% is deliberately set above the underperforming threshold's upper bound of 50.0%, allowing synthetic problems to be somewhat easier than the source problems (which is natural, since synthesis may simplify certain aspects) but preventing outright triviality.
Relation to the underperforming range: the synthesis reward range (12.5%–62.5%) is wider on the upper end than the underperforming selection range (12.5%–50.0%). This asymmetry reflects an intentional design: problems are selected for augmentation when they are relatively hard for the policy (≤50%), but the synthetic variants are allowed to be somewhat easier (up to 62.5%) while still being considered valid. The paper does not explicitly justify this asymmetry, but it likely reflects the practical reality that synthesizing problems at exactly the same difficulty as the originals is extremely challenging — some difficulty reduction is acceptable as long as the problem remains nontrivial.

Additional filtering for synthesis training pairs. The problem synthesis task itself becomes a training objective: the policy should learn to generate problems that, when solved, yield moderate accuracy. For this learning signal to be meaningful, the synthesis training pairs must also exhibit mixed rewards. Specifically, from the set of $G_v = 8$ variational problems generated from a single correct solution, the subset $\mathcal{J}_2$ that receive positive rewards (accuracy in 12.5%–62.5%) is identified. If $\mathcal{J}_2$ contains at least one problem, then all $G_v$ synthesis pairs $(y_i, \hat{x}_i^j)$ for $j = 1, \ldots, G_v$ are added to the buffer, with $\hat{x}_i^j$ receiving reward 1.0 if $j \in \mathcal{J}_2$ and 0.0 otherwise. However, these synthesis pairs are only retained if the correct solution $y_i$ (which serves as the input/prompt for the synthesis task) produces a mix of positive and negative synthesis rewards — i.e., the group of 8 synthetic problems generated from $y_i$ must include both good and bad quality problems. If all 8 are good or all 8 are bad, the advantages for the synthesis task collapse and the pairs are discarded. This mirrors the mixed-correctness filtering applied to problem-solving data.

This filtering logic means that only discriminative synthesis situations are learned from: the policy learns from cases where some of its generated problems are appropriately challenging and others are not, allowing it to distinguish what makes a good synthetic problem. If the policy consistently generates good problems (all $R_V = 1$ ), it already has this capability and doesn't need to learn further from these examples; if it consistently generates bad problems (all $R_V = 0$ ), there are no positive examples to learn from.

Effectiveness of reward shaping. Figure 18 in Appendix G shows the contrast between the naive reward (failed version) and the shaped reward (successful SvS). With naive rewards: valid training responses to synthetic problems (middle panel) saturate near 100%, policy entropy (right panel) drops sharply, and Pass@1 performance (left panel) is substantially worse. With shaped rewards: valid training responses stay at moderate levels (well below 100%), entropy remains stable, and performance continues improving. The reward shaping is therefore not merely a refinement — it is the mechanism that makes the entire self-play loop sustainable. Without it, the synthesis-exploitation feedback loop destroys training signal within a few hundred steps.

Full Training Buffer Construction and Policy Update

After all data collection phases complete, the training buffer $\mathbf{B}$ contains three distinct types of training pairs, each structured as (prompt, response, reward) tuples:

Type 1: Original Problem Solving — $(x, y_i, \mathbf{R}_c(y_i, a))$ for all solutions from problems with $0 < \text{Acc}(x) < 1$ . These are the standard RLVR training pairs that would exist without any augmentation.

Type 2: Variational Problem Synthesis — $(y_i, \hat{x}_i^j, R_V(\hat{x}_i^j))$ where $y_i$ is a correct solution to an underperforming problem serving as the prompt, $\hat{x}_i^j$ is the generated variational problem serving as the response, and $R_V(\hat{x}_i^j) \in \{0, 1\}$ is the binary synthesis quality reward. These pairs train the policy on the inverse task: given a correct solution, generate a well-posed problem that the solution answers. This is present only when the synthesis pairs exhibit mixed rewards (at least one $R_V = 1$ and at least one $R_V = 0$ in the group from a given $y_i$ ).

Type 3: Synthetic Problem Solving — $(\hat{x}_i^j, \hat{y}_k, \mathbf{R}_c(\hat{y}_k, a))$ for all solutions to synthetic problems that exhibited mixed correctness ( $0 < \text{Acc}(\hat{x}_i^j) < 1$ ). These are forward problem-solving pairs on the augmented problems, using the original answer $a$ for verification.

All three types are mixed together in the buffer $\mathbf{B}$ , and the policy is updated using a single GRPO gradient step (Equation 6) with the entire buffer. The buffer is then cleared, and the next iteration begins with fresh data collection. There is no replay, no prioritization of synthetic over original data, and no separate optimization phases — everything is joint and online.

What the joint training accomplishes:

The policy simultaneously learns three interconnected capabilities:

Problem solving (Types 1 and 3): the standard RLVR objective of generating correct reasoning trajectories for given problems. The synthetic problems (Type 3) provide additional problem-solving practice on structurally diverse but semantically equivalent variants of challenging training problems. Because these variants are generated online at each step, they prevent the policy from memorizing specific problem-solution pairs.
Problem synthesis from solutions (Type 2): the inverse mapping from a complete reasoning trace back to a well-posed problem statement. This task requires the policy to understand the structure of problems — what information must be specified, what can be left implicit, what makes a problem well-posed versus ambiguous. The paper argues in Appendix E.2 that this inverse task serves as a regularizer that prevents overfitting on the forward problem-solving objective, particularly preserving performance on out-of-distribution benchmarks where the answer format differs from the training set (e.g., DAPO-17k has integer-only answers, but OOD benchmarks may have fractions or expressions).
Problem difficulty calibration (implicitly via Type 2's reward shaping): the policy learns to generate problems at an appropriate difficulty level for itself. The reward shaping constraint $R_V = \mathbb{I}(12.5\% \leq \text{Acc} \leq 62.5\%)$ provides a binary signal of whether the generated problem was "appropriately challenging." Over training, the policy learns to avoid injecting hints, simplifying crucial steps, or otherwise making problems trivially easy, while also avoiding generating unsolvable or invalid problems.

Concrete numbers on buffer composition: The paper does not provide exact ratios of the three data types in the training buffer, but Figure 9 provides indirect information. For DAPO-17k training with the 32B model, the number of synthetic problem-solution pairs (Types 2 and 3) starts high at early training steps (when most problems are in the underperforming range) and declines as the policy improves. For MATH-12k, the synthetic volume is low throughout because the 32B model already achieves high accuracy on most MATH problems. The computational cost analysis in Section 5.5 (Figure 9) shows that per-iteration training time for SvS on DAPO-17k can be roughly 2-3 times that of standard RLVR in early training (due to the synthesis and additional solving overhead), but this overhead decreases as fewer problems qualify for augmentation.

Training duration: Models trained on MATH-12k run for 300 steps, while 32B models trained on DAPO-17k run for 600 steps (Appendix D.1). Checkpoints are selected based on the best average scores from AIME 24 and 25 (Section 5.3). All experiments use 32 H100 GPUs (Section 5.5).

Self-Play Loop Dynamics Over Training

The SvS pipeline forms a feedback loop with several emergent dynamics that the paper documents but does not fully formalize:

Declining synthesis volume. As training progresses and the policy's accuracy on training problems improves, fewer problems fall within the underperforming range $[12.5\%, 50.0\%]$ . This means fewer correct solutions are available as synthesis context, and the volume of synthetic data naturally declines. This creates an implicit curriculum: early in training, when the policy is weak, many problems are augmented and the policy gets diverse practice; later in training, as the policy masters most problems, augmentation naturally tapers off and training converges toward standard RLVR on the remaining challenging problems. The paper frames this as efficient resource allocation — "targeting the policy's weakest capabilities" (Section 1) — but it also means SvS's benefits are concentrated in the exploration-heavy early-to-middle phases of training.

Synthesis quality evolution. Appendix E.5 (Figure 13) shows that the fraction of synthetic problems judged correct by external LLM evaluators declines slightly over training, from above 85% in early steps to around 80% in later steps. The authors hypothesize that this is because "the synthetic problems gradually adopt a description style closer to the model's own way of expressing solutions in the later stages of training." As the policy's reasoning style becomes more specialized through RLVR training, its synthesized problems may become more idiosyncratic — potentially harder for external evaluators to parse but not necessarily incorrect. The paper's manual inspection found that problems flagged as invalid by evaluators were often actually solvable but phrased in unusual ways.

Capability expansion. The self-play loop enables the policy to practice on problems it generated at earlier, weaker stages of training. A problem synthesized at step 100 (when the policy was relatively weak) might remain in the buffer and be solved again at step 200 (when the policy has improved). However, since the buffer is cleared after each update step and problems are regenerated fresh each iteration, the policy never encounters exactly the same synthetic problem twice — each iteration produces new variants based on the current policy's solutions. This continuous renewal is what maintains exploration: even if the policy has seen the original problem hundreds of times, each synthetic variant is novel in surface form.

Training set interaction. SvS does not expand the permanent training set — it augments each batch online. The original training set $\mathcal{D}$ remains fixed (MATH-12k, DAPO-17k, or DAPO-25k). The synthetic problems exist only for one training iteration: they are generated, solved, used for a policy update, and then discarded. In the next iteration, fresh problems are sampled from $\mathcal{D}$ , fresh solutions are generated, and fresh synthetic variants are created. This online, transient nature of the augmentation is what distinguishes SvS from one-time data augmentation approaches like MetaMath, where the augmented data is precomputed and then treated as a static extension of the training set.

Summary of Key Design Choices and Their Justifications

Conditioning synthesis on solutions rather than problem text: ensures semantic preservation and answer validity without requiring annotation; enables self-play without external models; produces structurally diverse variants by requiring the policy to reconstruct problems from reasoning traces rather than paraphrase existing text.
Underperforming problem filtering at 12.5%–50% accuracy: focuses augmentation on problems at the policy's capability frontier; avoids wasting compute on problems the policy has either mastered (accuracy > 50%) or cannot yet solve (accuracy < 12.5%); creates an implicit curriculum where augmentation volume naturally declines as the policy improves.
Reward shaping for synthesis at 12.5%–62.5% accuracy: prevents exploitation where the policy generates hint-laden trivial problems; the upper bound (62.5%) is deliberately set higher than the underperforming upper bound (50.0%) to allow synthetic problems to be somewhat easier than originals while remaining nontrivial; the lower bound (12.5%) ensures solvability.
Mixed-correctness filtering for all data types (original and synthetic): aligns with GRPO's requirement for non-degenerate group advantages; filters out problems that provide zero or near-zero training signal; means that only problems at the productive edge of the policy's capability contribute to updates.
Joint training on problem solving and problem synthesis: the inverse synthesis task serves as a regularizer against overfitting; enriches the training distribution; prevents the policy from collapsing into a pure forward-solving mode that loses generalization.
Online, transient augmentation (no permanent dataset expansion): prevents the policy from memorizing synthetic problems across iterations; ensures continuous novelty; avoids the distributional issues of training on a fixed augmented set where synthetic problems may become stale relative to the improving policy.
Pure self-play (no external models, no distillation): eliminates dependency on stronger models; ensures capability alignment between the problem generator and solver; enables end-to-end self-improvement without human annotation or external data sources.
Fixed synthesis hyperparameters across model sizes: the group sizes $G = 8$ and $G_v = 8$ , the accuracy ranges $[12.5\%, 50.0\%]$ and $[12.5\%, 62.5\%]$ , and the synthesis prompt (Figure 20) are held constant across all experiments from 3B to 32B models, demonstrating robustness to model scale without per-model tuning.

4. Key Insights and Innovations

Innovation 1: Reframing RLVR Entropy Collapse as a Data Diversity Problem Rather Than an Optimization Problem

The most intellectually distinctive contribution of this paper is not the SvS mechanism itself but the diagnostic reframing that motivates it. The field has observed that RLVR training causes policy entropy to decline and Pass@k to plateau (Cui et al., 2025b; Yue et al., 2025), but the dominant interpretation treated this as an optimization pathology — something wrong with the RL algorithm that needed to be fixed through algorithmic intervention. The natural responses followed that interpretation: add entropy regularization terms to the loss (Clip-CoV; Cui et al., 2025b), adjust sampling temperatures (An et al., 2025; Chen et al., 2025), or modify the advantage computation to incentivize diversity.

Liang et al. argue for a fundamentally different diagnosis: entropy collapse is primarily a data curriculum failure, not an optimization failure. The policy is not unable to maintain diversity — it is rationally choosing not to, because the fixed training set makes memorization the optimal strategy for reward accumulation. When the same problems appear iteration after iteration, the policy learns that reproducing previously successful solution templates is the most reliable path to positive rewards. This is not a bug in GRPO; it is the correct behavior of a reward-maximizing agent given a stationary environment.

The evidence for this reframing is the preliminary experiment in Figure 2, which is conceptually the most important result in the paper even though it uses a simpler augmentation method (MetaMath) rather than SvS. Adding 36k rephrased problems slows entropy decline compared to training on MATH-12k alone. More tellingly, when the rephrased problems are updated at step 300 with fresh variants, entropy — which had been declining — actually increases, and Pass@32 correspondingly jumps. This is a clean demonstration that data novelty, not algorithmic regularization, is what reactivates exploration. The policy had not lost the capacity to generate diverse solutions; it had merely converged to a low-entropy equilibrium because the environment stopped rewarding exploration. Changing the environment restores exploration without any change to the optimization algorithm.

This reframing has significant implications beyond the specific SvS method. It suggests that the extensive literature on entropy regularization for RLVR may be treating a symptom (declining output diversity) rather than the cause (data stationarity). It also connects RLVR training to broader principles in curriculum learning and continual learning: the problem is not that RL "overfits" in some generic sense, but that a fixed dataset creates a stationary reward landscape where exploration has negative expected value. The solution is not to force exploration through regularization (which can dilute the RL signal, as the paper shows in Appendix F where SvS outperforms Clip-CoV) but to make exploration valuable again by continuously changing the problems the policy must solve. This is a conceptual shift from "RL algorithms need better exploration bonuses" to "RL training environments need non-stationarity," and it is the intellectual foundation on which the entire SvS framework rests.

Innovation 2: Solutions-to-Problems as a Self-Verifying Synthesis Paradigm

The paper's second conceptual contribution is the identification of solution-conditioned problem synthesis as a uniquely well-suited data generation paradigm for RLVR. This is not merely an engineering trick — it resolves three tension points that make standard data augmentation approaches unsuitable for verifiable-reward training.

Prior work on data augmentation for math reasoning has overwhelmingly operated in the problem-to-problem paradigm: take an existing problem and rephrase it (MetaMath; Yu et al., 2023), or take a problem description and generate similar problems (SwS; Liang et al., 2025), or generate problems from scratch using task definitions (Guo et al., 2025b). These approaches share a common challenge: answer verification. A rephrased problem might subtly change the mathematical content, invalidating the original ground-truth answer. A de-novo generated problem has no ground-truth answer at all unless one is separately computed or verified. In the RLVR setting, where the entire training signal comes from binary answer correctness, any uncertainty in answer validity is catastrophic — the model may be rewarded for incorrect answers or penalized for correct ones.

The SvS solution-to-problem paradigm inverts the synthesis direction and, in doing so, sidesteps the verification problem entirely. By conditioning synthesis on a known-correct solution, the method exploits a simple logical property: if a solution S correctly solves problem P with answer A, and if a new problem P' is constructed to be "the problem that S answers," then P' must have answer A (assuming the reconstruction is faithful). The ground-truth answer is preserved by construction, not by post-hoc verification. This means SvS can generate entirely novel problem statements — with different surface forms, different variable names, different narrative framing — while maintaining guaranteed answer validity, a property that no problem-to-problem synthesis method can offer without additional verification infrastructure.

The significance of this paradigm extends beyond the specific prompt template or synthesis hyperparameters used in the paper. It suggests a general principle for self-supervised data generation in verifiable-reward settings: synthesize from outputs, not inputs. The output (a solution) contains more structured information about the task than the input (a problem statement) — it encodes not just what the task is, but how it is solved, what constraints are active, what edge cases matter, and what the final answer must be. This informational asymmetry makes output-conditioned generation more reliable for answer-preserving synthesis than input-conditioned generation. The principle may generalize to other domains with verifiable outputs: code generation (generate new coding problems from known-correct implementations), theorem proving (generate new theorems from known proofs), or any task where the solution contains the complete specification of the problem it solves.

The paper also demonstrates, through its comparison with the MetaMath rephrasing baseline (Figure 2) and the failure of the naive synthesis reward (Appendix G), that simply having a solution-to-problem pipeline is insufficient — the synthesis must be constrained to produce problems at an appropriate difficulty level. This connects to the reward shaping innovation discussed below, but the fundamental conceptual move — the inversion of the synthesis direction — is independent and arguably more foundational.

Innovation 3: Difficulty-Aware Synthesis Validation as a Guardrail Against Self-Exploitation

The third conceptual contribution is the identification and solution of a self-exploitation feedback loop that emerges when a model both generates and solves its own training problems. This is a subtle but critical insight about the dynamics of self-play systems: when the generator and solver are the same model, naive reward structures create perverse incentives that cause the system to collapse.

The naive approach — rewarding the synthesis of any problem that the policy can solve at least once — seems intuitive. If the policy can produce a correct answer for a synthetic problem, the problem must be valid, and the synthesis should be rewarded. What the paper shows (Appendix G, Figure 18) is that this reward structure is trivially exploitable: the policy learns to embed answer hints directly into the problem text, producing problems like "Consider the function... [hint: the sum is 2+22i]" that are solved with 100% accuracy by simply extracting the embedded answer. The synthesis task collapses to a copy-paste operation, the synthetic problems provide no meaningful reasoning practice, and the training signal for both the synthesis and solving tasks vanishes (since all problems yield all-correct solution groups, GRPO advantages collapse to zero).

The paper's solution — the R_V reward-shaping constraint that only rewards synthetic problems with moderate policy accuracy (12.5%–62.5%) — is more than a hyperparameter choice. It represents a general design principle for self-play data generation systems: the generator should be rewarded not for producing solvable problems, but for producing problems at the frontier of the solver's capability. This principle aligns the generator's objective with the solver's learning needs. A problem that is too easy (accuracy > 62.5%) provides no learning signal for the solver because it's already mastered; a problem that is unsolvable (accuracy < 12.5%) provides no positive examples to learn from. Only problems in the intermediate range — where the solver sometimes succeeds and sometimes fails — create the mixed-correctness groups that GRPO requires for effective gradient updates.

This principle has implications beyond the specific accuracy ranges chosen in this paper. It suggests that self-play data generation systems should incorporate dynamic difficulty calibration — the generator should adapt its output difficulty to match the solver's current capability level, maintaining the generation of problems in the "productive struggle" zone. SvS achieves this implicitly through the accuracy-threshold reward, but more sophisticated implementations could use continuous difficulty metrics or adaptive thresholds that shift as the solver improves. The paper's finding that the 12.5%–62.5% range works robustly across model sizes from 3B to 32B (Table 2) suggests the principle is not highly sensitive to the exact thresholds, but the existence of such a constraint is essential — the Appendix G results show that removing it causes complete training collapse.

The self-exploitation problem that this innovation addresses is not unique to SvS or to math reasoning. Any system where a model generates its own training data and is evaluated on some downstream metric faces the risk that the generator will optimize for evaluatability rather than usefulness — producing data that scores well on the metric without requiring genuine capability from the solver. This is analogous to reward hacking in RLHF (where the policy exploits the reward model) but operates at the data generation level rather than the policy optimization level. The solution — constraining the generator by the solver's difficulty profile rather than just its success rate — may generalize to other self-improvement paradigms.

Innovation 4: Pass@k as the Primary Metric for Reasoning Advancement Rather Than a Secondary Diagnostic

The paper's fourth conceptual contribution is methodological rather than algorithmic: it establishes Pass@k scaling (up to k = 1024) as the definitive metric for evaluating whether RLVR training expands the model's reasoning frontier, and uses this metric to demonstrate a qualitative gap between standard RLVR and SvS that would be invisible from Pass@1 alone.

Prior RLVR work (DeepSeekMath, DAPO, Open-Reasoner-Zero) has primarily reported Pass@1 as the headline metric, sometimes accompanied by Pass@k for small k (typically 4, 8, or 64) as a supplementary measure. The implicit assumption has been that Pass@1 improvements reflect genuine capability advancement — if the model gets better at producing correct answers on the first try, its overall reasoning must have improved. Yue et al. (2025) challenged this assumption by showing that standard RLVR improves Pass@k only for small k and fails to improve (or degrades) Pass@k when k scales to hundreds or thousands, arguing that RLVR primarily sharpens the policy's distribution around existing correct solutions rather than discovering new ones.

This paper takes Yue et al.'s diagnostic observation and elevates it to a primary evaluation framework. Throughout the paper, Pass@32 and scaled Pass@k (up to 1024) are treated as co-equal headline metrics alongside Pass@1, not as supplementary analyses. Table 1 reports both metrics for all benchmarks. Figure 1 tracks Pass@32 throughout training. Figure 6 is dedicated exclusively to Pass@k scaling from 1 to 1024 across four benchmarks. The paper's strongest claims — "achieving absolute gains of 18.3% and 22.8% in Pass@32 on AIME 24 and AIME 25" — are stated in Pass@k terms in the abstract.

This metric choice matters because it reveals a qualitative difference between standard RLVR and SvS that Pass@1 obscures. On AIME 24 and 25 (Table 1, DAPO-17k), SvS improves Pass@1 by 10.5 points over RLVR — substantial, but consistent with the magnitude of gains seen from other RLVR improvements. But SvS improves Pass@32 by 18.3 and 22.8 points — gains that are disproportionately larger than the Pass@1 improvements, indicating that SvS is not merely sharpening existing capabilities but actually expanding the set of problems the model can solve under any sampling budget. This is visible in Figure 6: on AIME 24, the gap between SvS and RLVR widens as k increases from 1 to 1024, meaning SvS is discovering correct solutions for problems that RLVR never solves regardless of how many attempts it gets. Standard RLVR's Pass@k curves are closer to the base model's, consistent with Yue et al.'s finding that RLVR doesn't expand reasoning boundaries.

The methodological contribution is the argument — supported by this evidence — that Pass@1 and Pass@k measure fundamentally different properties: Pass@1 measures sampling efficiency (how concentrated the policy is on correct solutions), while Pass@k at large k measures the policy's reasoning coverage (what fraction of problems the model can ever solve). RLVR without data diversity improves sampling efficiency at the expense of coverage (the entropy-performance tradeoff from Cui et al., 2025b). SvS improves both simultaneously, which is the signature of genuine capability expansion rather than mere distributional sharpening. This diagnostic framework — using the Pass@1/Pass@k relationship to distinguish efficiency gains from capability gains — provides a principled evaluation methodology for future RLVR research that goes beyond reporting whichever metric looks best.

The counterexample on MATH-500 (Figure 6, bottom-right) reinforces this point. Standard RLVR achieves higher Pass@1 on MATH-500 than the base model, but its Pass@k curve falls below the base model's curve at large k — meaning RLVR training has actually lost some correct solutions that the base model could occasionally produce. SvS's Pass@k curve remains above the base model's across all k. This demonstrates that the coverage loss documented by Yue et al. is real and that SvS specifically addresses it, not through entropy regularization (which might preserve diversity at the cost of precision) but through continuous data renewal (which preserves both).

Assessment: Which Innovations Are Fundamental vs. Incremental?

Innovation 1 (data diversity reframing) is a fundamental conceptual shift. It changes the diagnosis of a well-documented problem (entropy collapse) from an optimization pathology to a curriculum design failure, redirecting research attention from algorithmic fixes to data pipeline design. The evidence from Figure 2 — that data updates alone, without any algorithm changes, can reverse entropy decline — makes this reframing empirically grounded rather than merely philosophical.

Innovation 2 (solution-to-problem synthesis) is a fundamental paradigm shift for verifiable-reward data generation. The inversion of the synthesis direction (outputs → inputs rather than inputs → inputs) solves the answer verification problem that has constrained prior data augmentation approaches. This is not an incremental improvement to existing synthesis methods — it is a different category of synthesis that enables self-play without external verification. The generalizability of this paradigm to other verifiable-output domains (code, proofs) makes it potentially influential beyond math reasoning.

Innovation 3 (difficulty-aware synthesis validation) is a refinement of the self-play paradigm rather than a fundamentally new concept, but it is an essential refinement — without it, the solution-to-problem pipeline collapses. The recognition that self-play data generation creates a self-exploitation risk, and that the generator must be constrained by the solver's difficulty profile, is a general insight applicable to any self-improving system. The specific accuracy-range implementation (12.5%–62.5%) is likely domain-specific, but the principle of difficulty-gated generation rewards is general.

Innovation 4 (Pass@k as primary metric) is a methodological contribution that codifies and operationalizes an observation from prior work (Yue et al., 2025). It is incremental in the sense that Pass@k evaluation existed before this paper, but fundamental in its implications: if the field adopts Pass@k scaling as a standard evaluation protocol, it will change which methods are considered successful and redirect research toward techniques that expand reasoning coverage rather than just sharpening existing capabilities.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary RLVR training sets are MATH-12k (Hendrycks et al., 2021; 12,000 problems) and DAPO-17k (Yu et al., 2025a; 17,000 problems), with an additional configuration DAPO-25k that augments DAPO-17k with 8,000 open-ended-answer problems from DeepMath (He et al., 2025b). For code generation (Section 5.4), 12k problems are sampled from PRIME-RL (Cui et al., 2025a), covering Apps, CodeContest, Taco, and Codeforces. Evaluation benchmarks span 12 mathematical reasoning datasets: GSM8K, MATH-500, Minerva Math, Olympiad-Bench, Gaokao-2023, AMC23, AIME 24, AIME 25, Beyond-AIME, OlymMATH (en-easy and en-hard subsets), and Math24o. For code generation evaluation (Figure 8), 100 instances are sampled from each validation set of the PRIME-RL sources.
Base model(s). Experiments use Qwen2.5-3B-Instruct, LLaMA-3.1-8B-Instruct (Grattafiori et al., 2024), and Qwen2.5-32B-Instruct (Yang et al., 2024), spanning an order of magnitude in parameter count (3B to 32B) and two model families (Qwen and LLaMA). The Qwen2.5-32B-Instruct model additionally serves as the base for DAPO-17k and DAPO-25k training; Qwen2.5-7B-Instruct is used for the code generation experiments. The paper argues that these models are "representative of the capabilities of many contemporary LLMs" (Section 4.1) and that testing across scales demonstrates robustness of SvS to model size.
Metrics. The primary metrics are Pass@1 and Pass@k (for k up to 1024). Pass@1 for competition-level benchmarks is reported as the average over 32 independent inferences per problem to mitigate high variance from small sample counts (Section 4.1: "we evaluate models smaller than 8B on AIME-level benchmarks using an average of 32 inferences"). For the main results in Tables 1 and 2, Pass@1 for all models on competition-level benchmarks uses this 32-run averaging. Pass@k is computed using the unbiased estimation method from Chen et al. (2021) to reduce variance. For code generation (Section 5.4), an Avg@16 metric is additionally reported on CodeContest. Policy entropy is tracked throughout training as a diagnostic of generation diversity, computed as the entropy of the policy's output distribution over tokens (Section 5.1, Figure 5). Training response length (Figure 10) is used as a proxy for reasoning depth.
Baselines. Five categories of baselines appear across the experiments: (1) Init Model: the instruction-tuned base model before any RLVR training; (2) RLVR: standard GRPO training with Clip-Higher (ε = 0.28), Token-Level Loss, and Dynamic Sampling (Yu et al., 2025a) on the same fixed dataset, without any augmentation — this is the primary comparison point throughout; (3) Open-source RLVR models: SimpleRL-32B (Zeng et al., 2025), Open-Reasoner-Zero-32B (ORZ-32B; Hu et al., 2025b), and Qwen2.5-32B (base, non-instruct); (4) Alternative augmentation strategies in the ablation (Section 5.3, Table 3): Ext (extended standard RLVR training to match the total number of problem-solution pairs seen by SvS), Eup (enhanced underperforming problems with a second rollout stage), and SvS-Asp (SvS but augmenting simpler problems with accuracy 37.5%–75.0% instead of underperforming ones); (5) Entropy regularization: Clip-CoV (Cui et al., 2025b) evaluated in Appendix F.
Generation budget / compute accounting. The unit of training is RLVR steps, with MATH-12k models trained for 300 steps and 32B DAPO-17k models trained for 600 steps (Appendix D.1). Each step processes a batch of 256 problems, with group size G = 8 solutions per problem (both original and synthetic) and G_v = 8 variational problems synthesized per correct solution. The paper does not report total FLOPs or GPU-hours for the main experiments but provides a computation analysis in Section 5.5 (Figure 9): experiments use 32 H100 GPUs, with per-iteration step time and standalone policy update time compared between SvS and RLVR. For evaluation, inference uses vLLM with temperature 1.0, top-p 0.7, and max response length 8,192 tokens (increased to 24,576 for Pass@k scaling experiments in Figure 6). For the Pass@k scaling experiments in Figure 6, k ranges from 1 to 1,024.
Cross-validation / statistical protocol. Checkpoint selection is based on the best average score on AIME 24 and AIME 25 (Section 5.3), providing a consistent model selection criterion. For the ablation experiments in Section 5.3 (Table 3, Figure 7), intermediate evaluations are tracked throughout training, and the final comparison uses the same AIME-average checkpoint selection. The paper does not report confidence intervals, standard errors, or statistical significance tests for any results. All curves showing training trajectories (Figures 1, 2, 5, 7, 10, 16, 17, 18, 19) are smoothed with a window size of 5 steps, with original data points shown as faint dots. For Tables 1, 2, and 3, Pass@1 is computed as the average over 32 independent evaluations per problem to reduce variance from sampling randomness; no uncertainty quantification is reported for these averages. For the synthetic problem quality analysis (Appendix E.5, Figure 13), 6,000 variational problems (10 per step across 600 steps) are evaluated by two external LLM judges (Qwen3-235B-A22B-Instruct-2507 and OpenAI O3), with a problem considered correct if at least one judge deems it valid.

Main Quantitative Results

SvS Significantly Improves Both Pass@1 and Pass@32 on Competition-Level Benchmarks

The headline results appear in Table 1 and Figure 1. Training Qwen2.5-32B-Instruct on DAPO-17k with SvS yields:

Pass@1: 39.3% on AIME 24 (+10.5 over RLVR's 28.8%), 40.5% on AIME 25 (+10.5 over RLVR's 30.0%), averaged across all six competition-level benchmarks 27.9% (+5.4 over RLVR's 22.5%).
Pass@32: 70.8% on AIME 24 (+18.3 over RLVR's 52.5%), 65.2% on AIME 25 (+22.8 over RLVR's 42.4%), averaged across all six benchmarks 53.1% (+8.5 over RLVR's 44.6%).

Figure 1 provides the training dynamics behind these numbers. Standard RLVR's Pass@32 on AIME 24 and AIME 25 plateaus after approximately 450 steps, while SvS continues improving through 600 steps. Pass@1 similarly plateaus for RLVR but continues rising for SvS. The gap between SvS and RLVR widens over training, indicating that SvS's advantage is not merely a faster initial improvement but a sustained capability that standard RLVR cannot match with any amount of additional training on the fixed dataset.

On MATH-12k with the same 32B model (Table 1, MATH-12k section), SvS achieves Pass@32 of 63.6% on AIME 24 (+16.2 over RLVR) and 55.1% on AIME 25 (+18.7 over RLVR), demonstrating that even with a simpler, medium-difficulty training set, SvS's augmentation strategy transfers gains to competition-level benchmarks. The Pass@32 average across the six competition benchmarks is 54.6% for SvS versus 38.6% for RLVR (+16.0).

The improvement is not uniform across all benchmarks. On OlymH (OlymMATH en-hard), SvS shows a decline relative to RLVR on DAPO-17k: Pass@1 of 2.7% versus 4.8% (Δ = -2.1), and Pass@32 of 16.7% versus 18.3% (Δ = -1.6). The paper does not discuss this exception specifically. On OlymE (OlymMATH en-easy), the DAPO-17k SvS model shows a decline in Pass@32 (43.4% vs. 47.1%, Δ = -3.7) while improving Pass@1 (21.8% vs. 17.9%, Δ = +3.9). These exceptions suggest that for certain out-of-distribution benchmarks, SvS's augmentation may trade off coverage for precision or vice versa, though the pattern is not consistent across configurations (the MATH-12k SvS model shows gains on OlymE for both metrics).

SvS Boosts RLVR Across All Model Sizes and Training Sets

Table 2 reports Pass@1 (average of 32 runs for AIME-level benchmarks) across nine benchmarks and three model sizes. The average improvements of SvS over RLVR, computed across all nine benchmarks:

Qwen2.5-3B-Instruct on MATH-12k: SvS average 40.8% vs. RLVR 37.9% (Δ = +2.9). On MATH-500 specifically, RLVR training degrades the base model (67.8% → 67.4%), while SvS improves it to 70.8% (+3.0 over base, +3.4 over RLVR). On AIME 24: 10.5% vs. 6.7% (Δ = +3.8). On AIME 25: 7.8% vs. 3.4% (Δ = +4.4).
LLaMA-3.1-8B-Instruct on MATH-12k: SvS average 35.9% vs. RLVR 34.2% (Δ = +1.7). The largest gains appear on MATH-500 (62.2% vs. 57.4%, Δ = +4.8) and Gaokao-2023 (54.8% vs. 47.8%, Δ = +7.0). Competition-level gains are more modest: AIME 24 8.5% vs. 8.1% (Δ = +0.4), AIME 25 1.8% vs. 1.2% (Δ = +0.6).
Qwen2.5-32B-Instruct on MATH-12k: SvS average 57.3% vs. RLVR 54.8% (Δ = +2.5). Gains appear across most benchmarks, with notable improvements on Olympiad-Bench (56.7% vs. 52.7%, Δ = +4.0) and AIME 24 (30.0% vs. 26.7%, Δ = +3.3).
Qwen2.5-32B-Instruct on DAPO-17k: The SvS configuration here (SvS on D17k) shows a markedly different pattern. Gains on competition-level AIME benchmarks are dramatic: AIME 24 53.3% vs. 33.3% (Δ = +20.0), AIME 25 43.3% vs. 36.7% (Δ = +6.6), Beyond-AIME 19.0% vs. 13.0% (Δ = +6.0). However, performance declines on several benchmarks: MATH-500 75.6% vs. 87.0% (Δ = -11.4), Minerva Math 42.3% vs. 45.6% (Δ = -3.3), Olympiad-Bench 45.9% vs. 54.8% (Δ = -8.9), Gaokao-2023 62.9% vs. 78.7% (Δ = -15.8). The paper attributes these declines to the model "overfitting to DAPO-17k's integer-only format during augmentation" (Section 4.2), since DAPO-17k answers are exclusively integers while the degraded benchmarks require open-ended answers (fractions, expressions, etc.). The SvS on D25k configuration (DAPO-17k augmented with 8k open-ended problems from DeepMath) recovers performance on these benchmarks: MATH-500 88.6% (highest across all configurations), Minerva Math 47.8%, Olympiad-Bench 59.9%, Gaokao-2023 79.2%, while largely retaining the AIME gains (AIME 24 50.0%, AIME 25 40.0%). The D25k average across all benchmarks is 62.8%, the highest in the table. This result demonstrates that SvS's effectiveness is sensitive to training set composition — when the dataset skews toward a specific answer format, SvS's augmentation amplifies that bias.

SvS Sustains Policy Entropy Throughout Training

Figure 5 shows policy entropy trajectories for all model-dataset combinations. Across all four configurations (3B on MATH-12k, 8B on MATH-12k, 32B on MATH-12k, 32B on DAPO-17k), standard RLVR exhibits monotonically decreasing entropy. SvS, by contrast, maintains entropy within a relatively stable range after an initial decline, without the continued downward drift seen in RLVR. The gap between SvS and RLVR entropy is largest for the 3B model (where RLVR entropy declines most sharply) and the DAPO-17k 32B model (where SvS generates substantial synthetic data volume, as shown in Figure 9). The paper argues that this entropy stability "explains the continuous improvements in both Pass@1 and Pass@32 achieved by SvS, as shown in Figure 1, whereas RLVR saturates after a certain number of training steps" (Section 5.1).

SvS Expands Reasoning Boundaries as Measured by Pass@k Scaling

Figure 6 evaluates Pass@k from k = 1 to k = 1,024 on four benchmarks: AIME 24, AIME 25, Beyond-AIME, and MATH-500. The findings:

AIME 24 and AIME 25: Both standard RLVR and SvS improve Pass@k over the base model across all k, but SvS's advantage over RLVR grows as k increases. At small k (k = 1), the gap between SvS and RLVR is modest; at k = 1,024, the gap is substantially larger. For AIME 24, SvS at k = 1,024 reaches approximately 80% Pass@k, compared to roughly 65% for RLVR and roughly 55% for the base model (estimated from the figure, exact numbers not reported in text). For AIME 25, SvS reaches approximately 75%, RLVR approximately 55%, and the base model approximately 40%.
Beyond-AIME: A similar pattern holds: SvS outperforms RLVR at all k, with the gap widening at larger k. The absolute Pass@k values are lower than AIME (Beyond-AIME is harder), but the relative advantage of SvS persists.
MATH-500: This benchmark reveals a qualitatively different pattern. Standard RLVR achieves higher Pass@1 than the base model, but its Pass@k curve crosses below the base model's curve at larger k — RLVR's Pass@k improvement diminishes and eventually becomes negative relative to the base model. SvS, by contrast, maintains its Pass@k curve above both RLVR and the base model across all k from 1 to 1,024. At k = 1,024, SvS reaches roughly 98% Pass@k, RLVR approximately 94%, and the base model approximately 95% — RLVR is actually worse than the untrained base model at large k.

The paper interprets these results as evidence that SvS "significantly extend[s] the model's reasoning boundaries" (Section 5.2), where the reasoning boundary is operationally defined as the Pass@k ceiling at large k. The widening gap between SvS and RLVR on competition benchmarks indicates SvS is discovering correct solutions for problems that RLVR never solves regardless of sampling budget, while RLVR's crossover with the base model on MATH-500 indicates it has lost reasoning coverage that the base model originally possessed.

SvS Elicits Deeper Reasoning as Measured by Response Length

Figure 10 (Appendix E.1) shows training response lengths throughout training. SvS consistently produces longer reasoning trajectories than RLVR across all model-dataset configurations. For the 32B model on MATH-12k, SvS responses are roughly 20–30% longer than RLVR responses by the end of training (estimated from the figure). For the 8B model on MATH-12k, the gap is even larger, with SvS reversing an initial length decline that RLVR exhibits. The paper notes that "standard RLVR consistently failed to generate extended reasoning paths from the initial policy, whereas SvS succeeded" (Appendix E.1), attributing this to SvS forcing the model to "tackle new variational problems at each step, promoting continuous exploration of advanced reasoning strategies, whereas standard RLVR often secures high rewards by reusing memorized correct solutions."

SvS Improves Instance-Level Performance on the Hardest Problems

Figure 12 (Appendix E.4) presents an instance-level comparison of SvS and RLVR on AIME 24 and AIME 25, with accuracy averaged over 1,024 generations per problem. Across the 15 problems of each AIME exam, SvS achieves higher per-problem accuracy than RLVR on the large majority of instances. The most striking results are problems that RLVR consistently fails: on AIME 24 problem 12, RLVR achieves near-zero accuracy while SvS reaches approximately 20%; on AIME 25 problem 17, a similar pattern appears with SvS reaching approximately 15% while RLVR remains near zero. These are "problems that standard RLVR consistently fails to answer," demonstrating SvS's "ability to extend the boundaries of model reasoning" (Appendix E.4). The gap is particularly pronounced on the hardest problems within each exam (the later problems, which are typically more difficult in AIME's ordering).

SvS Generalizes to Code Generation with Dramatically Faster Training

Figure 8 (Section 5.4) shows intermediate evaluations on code generation benchmarks using Qwen2.5-7B-Instruct trained on 12k PRIME-RL problems. SvS training achieves significant improvements with roughly one-fifth the training steps of standard RLVR: SvS after approximately 100 steps matches or exceeds the performance of standard RLVR after over 600 steps. The figure reports Pass@1 on three benchmarks, Avg@16 on CodeContest, and the average Pass@1 across all four benchmarks. Policy entropy in the SvS code generation experiment also remains stable, while the paper notes that the RLVR baseline's entropy is not directly compared in this figure. The conclusion stated is that "SvS training achieves significant improvements with five times fewer training steps than the RLVR baseline and while maintaining stable policy entropy, demonstrating the strong generalization of this online self-play augmentation strategy in RLVR" (Section 5.4).

Ablation Studies and Robustness Checks

Alternative augmentation strategies vs. full SvS (Table 3): Three alternative configurations are compared on Qwen2.5-32B-Instruct with DAPO-17k, with checkpoints selected by best average AIME 24/25 scores.

Ext (Extended standard RLVR training): Training RLVR longer to match the total number of problem-solution pairs seen by SvS. Results: Pass@1 average 24.6% (+2.1 over RLVR's 22.5%), Pass@32 average 46.3% (+1.7 over RLVR's 44.6%). The improvements are modest — extending training alone provides some benefit but falls far short of full SvS (Pass@1 average 27.9%, Pass@32 average 53.1%). The paper notes this "aligns with the results of Liu et al. (2025a)."
Eup (Enhanced underperforming problems in RLVR): A second rollout stage is assigned to underperforming problems (those with accuracy 12.5%–50%), generating additional solution attempts without variational problem synthesis. Results: Pass@1 average 21.7% (-0.8 vs. RLVR), Pass@32 average 50.9% (+6.3 vs. RLVR). This configuration achieves much higher Pass@32 but lower Pass@1 compared to standard RLVR, indicating a trade-off: additional rollouts on hard problems improve exploration and coverage (Pass@32) at the cost of exploitation efficiency (Pass@1). The paper connects this to the conclusion in Zhu et al. (2025) that "this strategy introduces more negative samples from underperforming problem augmentation, and such exploration effectively improves the model's Pass@k scores."
SvS-Asp (Augmenting simpler problems in SvS): SvS is applied but augmenting simpler problems (group accuracy 37.5%–75.0%) instead of underperforming ones (12.5%–50.0%). Results: Pass@1 average 22.8% (+0.3 vs. RLVR), Pass@32 average 42.8% (-1.8 vs. RLVR). This configuration achieves Pass@1 comparable to RLVR but lower Pass@32, indicating that "this augmentation accelerates overfitting to the policy's already mastered capabilities while limiting exploration" (Section 5.3).

The comparison of full SvS against all three alternatives isolates two conclusions: (1) the focus on underperforming problems specifically (Eup vs. SvS-Asp) is crucial — augmenting easier problems provides no exploration benefit; (2) maintaining diversity through variational synthesis rather than simply adding more rollouts on the same problems (Eup vs. full SvS) is also crucial — Eup's Pass@32 gains come at the cost of Pass@1, while SvS improves both simultaneously.

Reduced problem-synthesis training data (Appendix E.2, Figure 11): An experiment where only 20% of variational problem synthesis pairs are used for policy updating in each RL step, using Qwen2.5-32B-Instruct on DAPO-17k. Benchmarks are categorized as IID (AIME24, AIME25, AMC23 — integer answers, matching DAPO-17k's format) and OOD (MATH-500, Minerva Math, Olympiad-Bench — diverse answer formats). The finding: SvS with 20% synthesis data performs comparably to full SvS on IID benchmarks but significantly worse on OOD benchmarks. The paper interprets this as evidence that "pure problem-solving training is susceptible to overfitting and reduces the model's generalizability, whereas the inclusion of problem synthesis helps mitigate this issue" through enrichment of the training distribution and dual-task regularization from the forward (solving) and inverse (synthesis) learning objectives.

Naive vs. shaped synthesis reward (Appendix G, Figure 18): The initial version of SvS assigned positive synthesis rewards whenever the policy produced at least one correct answer on a synthetic problem. This failed: the policy exploited the reward by embedding answer hints in problem text. Figure 18 shows the consequences: valid training responses to synthetic problems saturate near 100%, policy entropy drops sharply, and Pass@1 on AIME plateaus at a substantially lower level than the shaped-reward SvS. The shaped reward (only positive for problems with accuracy 12.5%–62.5%) prevents this exploitation, maintaining sub-100% valid response rates, stable entropy, and continued Pass@1 improvement. This is a critical negative result that validates the necessity of the difficulty-gated reward design.

SvS combined with entropy regularization (Appendix F, Figures 16 and 17): SvS is compared against Clip-CoV (Cui et al., 2025b), an entropy regularization method, using LLaMA-3.1-8B-Instruct on MATH-12k trained for over 400 steps. Figure 16 shows that SvS consistently outperforms Clip-CoV in intermediate evaluations on GSM8k, MATH-500, Minerva-Math, Olympiad-Bench, Gaokao-2023, and AMC-23, with the gap widening over training steps. Figure 17 shows that integrating SvS augmentation with Clip-CoV consistently improves over the Clip-CoV baseline alone, demonstrating that SvS is complementary to entropy-regulation methods — the data augmentation addresses the root cause (memorization from fixed training data) while entropy regularization addresses a downstream symptom, and combining them yields additive benefits.

Synthetic problem quality analysis (Appendix E.5, Figures 13, 14, 15): Using external LLM evaluators (Qwen3-235B-A22B-Instruct and OpenAI O3) to judge 6,000 variational problems (10 per step across 600 DAPO-17k training steps): more than 80% of synthetic problems are consistently judged as correct across training, with a slight downward trend over steps (from ~85% to ~80%). Figure 14 shows a strong positive correlation between a model's accuracy on original DAPO problems and their associated variational problems, with OLS slopes of approximately 0.6 (Qwen2.5-32B) and 0.5 (Qwen3-30B-A3B), indicating that synthetic problems largely preserve the difficulty ordering of their originals. Figure 15 shows that the histogram of accuracy differences between original and variational problems is sharply centered near zero, with only a small proportion exhibiting large gaps. The paper's manual inspection found that some problems flagged as invalid by both evaluators were actually solvable but phrased in unusual ways, suggesting the 80% correctness rate may be an underestimate.

Synthesis accuracy range sensitivity (implicit in the constant hyperparameters across all experiments, Appendix D.1): The underperforming range [12.5%, 50.0%] and the synthesis reward range [12.5%, 62.5%] are held constant across all model sizes from 3B to 32B and across both MATH-12k and DAPO-17k training sets, without per-model tuning. The consistent improvements across all settings (Table 2, Figure 5) indicate that these thresholds are reasonably robust to model scale and training data difficulty, though the paper does not present a systematic sweep of alternative threshold values.

SvS on general benchmarks beyond reasoning (Appendix E.3, Table 4): Evaluated on MMLU-Pro, ARC-C, ARC-E, HellaSwag, Winogrande, PIQA, BoolQ, HumanEval, and AGIEval using Qwen2.5-32B-Instruct trained on DAPO-17k. Standard RLVR degrades performance on most of these benchmarks relative to the initial instruction-tuned model (average 72.99% vs. 73.35%), consistent with the concern that RLVR overfits to mathematical problem-solving. SvS not only avoids this degradation but achieves a higher average (73.77%) than both the RLVR model and the initial model, with improvements on MMLU-Pro (71.58% vs. 68.33%), ARC-C (58.79% vs. 58.62%), and HellaSwag (85.34% vs. 85.17%). This suggests the synthesis task generalizes to preserve broader instruction-following capabilities.

Code generation adaptation (Section 5.4, Figure 8): SvS is applied to code generation by adapting the synthesis prompt for coding tasks (Figure 20, bottom). The prompt asks the model to "reconstruct the original programming problem that would most plausibly elicit such a response" and specifies inputs, outputs, and constraints. The evaluation shows SvS reaching the performance of 600-step standard RLVR in approximately 100 steps, a roughly 5× training speedup, with stable policy entropy. This experiments demonstrates that the solution-to-problem synthesis paradigm transfers across domains — the underlying principle (generating problems from correct solutions to preserve answer validity) does not depend on the mathematical nature of the original problems.

Critical Assessment

Does SvS Truly Expand Reasoning Boundaries, or Does It Improve Sampling Diversity Within Existing Capabilities?

The paper's central claim is that SvS "significantly extend[s] the model's reasoning boundaries" (Section 5.2), where reasoning boundaries are operationally defined as Pass@k performance at large k. The evidence for this claim consists primarily of Figure 6, which shows SvS's Pass@k curves rising above both RLVR and the base model at large k. This is interpreted as the model solving problems it previously could never solve.

The evidence is suggestive but not conclusive due to two considerations. First, Pass@k at k = 1,024 with a maximum response length of 24,576 tokens is an extremely expensive sampling regime. The paper does not discuss whether the base model's Pass@k curve has been accurately estimated at these scales — the unbiased estimator from Chen et al. (2021) reduces variance but requires careful implementation, and the paper provides no uncertainty quantification. Small estimation errors at large k could change the qualitative interpretation (whether SvS truly surpasses the base model's ceiling, or merely matches it with better sampling efficiency). Second, the paper does not establish whether the Pass@k improvements represent genuinely novel reasoning strategies or simply more diverse variants of strategies the model already possessed. A qualitative analysis of the solutions produced by SvS versus RLVR at large k — categorizing reasoning approaches, identifying whether SvS discovers novel solution methods — would strengthen the claim but is absent. The case study in Appendix J compares individual responses but does not systematically characterize the distribution of reasoning strategies.

A critical experiment that would have strengthened this claim: evaluating SvS on held-out problems that are truly novel relative to training (e.g., problems from a different competition or a later year, with no overlap in problem structure). AIME 24 and AIME 25 are used both for checkpoint selection and for the primary Pass@k evaluation in Figure 6, creating potential for selection bias — the model is chosen because it performs well on these benchmarks, and then evaluated on the same benchmarks, inflating apparent gains. The paper's cross-validation protocol (selecting checkpoints based on AIME average) is stated but not described in detail for the Pass@k scaling experiments.

How Much of SvS's Benefit Comes from Simply Seeing More Data?

The Ext ablation in Table 3 addresses this partially: extending standard RLVR training to match SvS's total problem-solution pairs yields only modest gains (Pass@1 average +2.1%, Pass@32 average +1.7%), far short of SvS's gains. This suggests the benefit is not merely from increased data volume. However, the Ext baseline has a subtle limitation: it matches the total number of problem-solution pairs but not the diversity of problems. The synthetic problems in SvS are structurally varied variants, while Ext simply samples more solutions to the same fixed problems. A stronger baseline would be: augment the training set with an equivalent number of externally synthesized diverse problems (e.g., from MetaMath or another LLM) and train standard RLVR on this expanded but static dataset. This would isolate whether the benefit comes from problem diversity per se or from the online, capability-aligned nature of SvS's synthesis. The MetaMath experiment in Figure 2 is a step in this direction but is conducted only as a preliminary investigation with the 32B model, not as a systematic comparison across all configurations.

Similarly, the Eup baseline (Table 3) adds rollouts on underperforming problems without variational synthesis, improving Pass@32 by 6.3 points but reducing Pass@1 by 0.8 points. This demonstrates that raw exploration on hard problems trades off against exploitation efficiency. SvS's simultaneous improvement of both metrics suggests its variational problems provide something that additional rollouts on fixed problems do not — presumably, structured diversity that transfers across surface forms. But the paper does not fully disentangle the effects of (a) seeing more solution attempts, (b) seeing novel problem formulations, and (c) the regularization from the synthesis task.

Are the Gains on Competition-Level Benchmarks Partly Explained by Training Set Similarity?

The paper acknowledges a critical confound in the DAPO-17k experiments: DAPO-17k consists of integer-answer problems, and AIME (a primary evaluation benchmark) also has integer answers. The D25k experiment (Table 2) shows that adding 8k open-ended problems to the training set substantially improves performance on open-ended evaluation benchmarks while largely preserving AIME gains. This suggests that SvS's augmentation process amplifies whatever biases exist in the training data — if the training set skews toward integer-answer problems, the synthetic variants will also skew toward integer-answer problems, and the model will overfit to that format. The label "overfitting to DAPO-17k's integer-only format during augmentation" (Section 4.2) is entirely plausible given the 11.4-point drop on MATH-500. This is both a strength (SvS can be steered by training set composition) and a weakness (SvS requires careful training set curation to avoid amplifying unwanted biases).

The paper does not investigate whether SvS would outperform RLVR if both were trained on the same well-balanced D25k dataset from the start. The D25k SvS result (average 62.8%) is the best in Table 2, but it's compared against RLVR on D17k (average 58.6%), not RLVR on D25k. A fair comparison would require training standard RLVR on D25k as well, which is not reported.

Generalizability Claims Are Supported but Limited

The paper demonstrates SvS across three model sizes (3B, 8B, 32B), two model families (Qwen, LLaMA), two training datasets (MATH-12k, DAPO-17k), and two domains (math reasoning, code generation). This breadth of testing is a genuine strength and supports the claim of generalizability. The consistent improvements across all configurations (Table 2) suggest that SvS's core mechanism is robust to model architecture, scale, and domain.

However, all experiments use the GRPO optimizer with the specific DAPO enhancements (Clip-Higher, Token-Level Loss, Dynamic Sampling). The paper claims SvS is "agnostic to RLVR optimization algorithms and can be flexibly incorporated into other methods, such as PPO, GSPO, and Reinforce++" (Section 3.1), but provides no experimental evidence for this claim. Testing SvS with PPO or another algorithm would validate the claimed agnosticism, but is not done. Additionally, the code generation experiment (Figure 8) uses only one model (Qwen2.5-7B-Instruct) and a relatively small training run (~100 SvS steps vs. ~600 RLVR steps), providing proof-of-concept but not a comprehensive evaluation. The benchmarks reported for code generation are limited to the four PRIME-RL validation sets; standard code benchmarks like HumanEval, MBPP, or LiveCodeBench are not evaluated for the code-trained models.

What the Experiments Do Not Test

Several experiments that would strengthen the paper's claims are absent:

Varying the synthesis accuracy thresholds ([acc_l, acc_h] and [a\hat{c}_1, a\hat{c}_h]): The thresholds 12.5%–50% (underperforming) and 12.5%–62.5% (synthesis reward) are held constant. A sensitivity analysis showing how performance varies with these thresholds would clarify whether the precise values matter or whether the principle of "intermediate difficulty" is all that's needed. Given the dramatic failure when the reward shaping is removed entirely (Appendix G), understanding the acceptable range is important for practical adoption.
Scaling the synthesis budget (G_v): The number of variational problems per correct solution is fixed at 8. Would 4 or 16 change the results meaningfully? The paper provides no evidence. Given the computational overhead analysis (Figure 9), understanding the efficiency frontier of synthesis budget vs. performance gain would be practically valuable.
Domain transfer beyond code: The paper claims SvS generalizes beyond math reasoning, supporting this with code generation results. Testing on other verifiable-reward domains (e.g., formal theorem proving with Lean or Isabelle, where correctness is machine-checkable) would test the limits of the solution-to-problem synthesis paradigm.
Longer training horizons: All experiments run for fixed step counts (300 for MATH-12k, 600 for DAPO-17k). Figure 1 suggests SvS continues improving at 600 steps where RLVR has plateaued. Whether SvS eventually plateaus at some higher step count, or continues improving indefinitely, is unknown. The claim of "sustained" improvement is true relative to the training horizon tested but does not establish asymptotic behavior.
Interaction with model scale: The Pass@32 gains on AIME are largest for the 32B model (Table 1). The 3B and 8B results (Table 2) show smaller absolute improvements, though the relative improvements are comparable. Whether SvS's benefits scale with model size (larger models benefit more from self-play augmentation) or diminish (smaller models benefit more because they have more room for improvement) is not systematically tested.
Ablation of the synthesis prompt design: The synthesis prompt (Figure 20) instructs the model to "identify the topic and context," "infer the type of reasoning," and "determine the educational purpose." Whether these specific instructions improve synthesis quality over a simpler prompt (e.g., "reconstruct the original problem from this solution") is not tested. Given that the prompt is the primary mechanism for the solution-to-problem mapping, its design likely matters, but no ablation is provided.

Statistical Rigor

The paper's statistical reporting is notably sparse. None of the main results (Tables 1, 2, 3; Figures 1, 5, 6, 7) include confidence intervals, standard errors, or p-values. For the AIME benchmarks with Pass@1 averaged over 32 runs, the standard error of the mean depends on the per-problem variance, which is not reported. For Pass@k with the unbiased estimator, the variance depends on k and the number of samples used for estimation, which is also not reported. The trajectories in Figures 1, 2, 5, 7, and others are smoothed with a 5-step window, which visually clarifies trends but also obscures the underlying variability — the faint dots showing original data points are helpful but not a substitute for proper uncertainty quantification.

Given the relatively small size of some evaluation benchmarks (AIME 24 has 15 problems, AIME 25 has 15 problems), per-problem variance could be substantial. A single difficult problem on which SvS happens to succeed could meaningfully shift the overall accuracy. The instance-level analysis in Figure 12 partially addresses this by showing per-problem performance, confirming that SvS's gains are distributed across many problems rather than concentrated on a few outliers, but no formal statistical test is applied.

Practical Considerations Not Addressed

The paper's computation analysis (Section 5.5, Figure 9) shows that SvS training on DAPO-17k with the 32B model incurs roughly 2–3× the per-iteration time of standard RLVR in early training, declining to near parity as synthesis volume drops. The paper frames this as acceptable overhead, but two practical concerns are unaddressed:

GPU memory overhead: SvS requires generating, storing, and processing synthetic problems and their solutions within a single training iteration. For the 32B model with G = 8, G_v = 8, and 256 problems per batch, the peak memory could be substantially higher than standard RLVR, which only processes original problems. The paper mentions using 32 H100 GPUs but provides no memory profiling or discussion of whether SvS requires more GPUs than standard RLVR for the same batch size.
Wall-clock time to peak performance: While SvS achieves higher peak performance, the per-step overhead means that wall-clock time comparisons depend on whether the overhead is offset by faster convergence. The code generation experiment (Figure 8) shows SvS achieving RLVR's 600-step performance in ~100 steps, a dramatic speedup. But for math reasoning (Figure 1), SvS and RLVR are trained for the same number of steps (600), with SvS taking more wall-clock time per step. The asymptotic Pass@1 / Pass@32 advantage of SvS is clear, but the efficiency frontier (performance per GPU-hour) is not quantified.

Summary of Evidence Strength

The experiments convincingly demonstrate that SvS improves both Pass@1 and Pass@32 over standard RLVR across a range of model sizes and benchmarks (Tables 1, 2; Figures 1, 6). The evidence for the mechanism — that SvS works by maintaining policy entropy through continuous data renewal, and that this entropy maintenance causes the performance improvement — is correlational but compelling (Figure 5 shows stable entropy; the ablation in Table 3 shows that alternative augmentation strategies that don't maintain diversity underperform; Appendix G shows that removing the difficulty gate causes immediate collapse). The evidence for expanding reasoning boundaries is more qualified — the Pass@k scaling results (Figure 6) are suggestive but lack uncertainty quantification and are evaluated on the same benchmarks used for checkpoint selection. The strongest unique claim — that SvS can solve problems RLVR never solves (Figure 12, specific AIME problems) — is demonstrated qualitatively through instance-level analysis but not systematically quantified across a broader set of out-of-distribution problems. The generalizability to code generation (Figure 8) is promising but preliminary, with limited benchmark coverage and no comparison to standard RLVR trained for the same wall-clock time rather than the same number of steps.

Overall, the experimental evidence supports the paper's primary practical claim — SvS substantially outperforms standard RLVR — while leaving open questions about the precise mechanisms, the sensitivity to hyperparameters, the statistical reliability of the Pass@k estimates, and the efficiency frontier for practical deployment.

6. Limitations and Trade-offs

6.1 The Difficulty Estimation Cost Is Unaccounted for in the Training Budget

The assumption or constraint. SvS determines which problems to augment by computing the policy's group accuracy on each training problem, then selecting those whose accuracy falls within [12.5%, 50.0%] (Section 3.2, Algorithm 1 Line 11). This requires generating G = 8 full solution trajectories for every problem in the batch before the augmentation decision is made, and those accuracy computations are an integral part of every training iteration. The paper acknowledges the computational overhead of the synthesis itself (Section 5.5, Figure 9) but frames the accuracy filtering as a lightweight byproduct of the solution collection that standard RLVR already performs. Yet the filtering step means that SvS generates 8 solutions for every problem in the batch, retains only those with mixed accuracy for the buffer, and then — for the subset in the underperforming range — generates additional solutions (the synthetic problem solving phase) plus the synthesis and solving for variational problems. The paper's computation analysis (Figure 9) measures per-iteration time but does not break down what fraction of SvS's overhead comes from the accuracy-filtering cost itself versus the downstream synthesis and synthetic solving.

The consequence. The practical consequence is that SvS's per-iteration cost is not merely "2–3× standard RLVR in early training" as Figure 9 suggests — it is structurally higher because every iteration pays the cost of solving all 256 problems in the batch twice: once for the original solving phase, and once (for the underperforming subset) for the synthetic solving phase. Standard RLVR pays the original-solving cost once per problem per iteration. SvS pays it once, plus pays an additional solving cost for the synthetic problems (which can be substantial when the model is weak and many problems qualify as underperforming), plus the synthesis cost itself. This means the 4× efficiency framing that appears in prior sections (where SvS achieves in 64 generations what RLVR achieves in 256) does not account for the generation budget spent on filtering and synthesis within the training loop. A practitioner comparing SvS to standard RLVR on a total-FLOPs or GPU-hours basis would find the comparison less favorable than the headline accuracy numbers suggest, especially in early training when synthesis volume is highest. The paper reports SvS taking "roughly 2-3 times" the per-iteration time of RLVR (Figure 9), but this is measured in wall-clock time with 32 H100 GPUs — it does not isolate whether the bottleneck is generation, synthesis, or gradient computation, nor does it report total FLOPs to convergence.

What evidence exists in the paper. Figure 9 provides the only quantitative time comparison: iteration step time and standalone policy update time for SvS and RLVR on both MATH-12k and DAPO-17k. The two left panels show the number of variational problems and corresponding responses generated per iteration. For DAPO-17k, synthesis volume starts high and declines over training as the policy improves; for MATH-12k, it is low throughout. The paper notes that "the initial stages exhibit a large number of synthetic variational problems with responses because the policy's accuracy on most problems falls within the range [acc_l, acc_h]" and that "when training on the simpler MATH-12k dataset, the Qwen-32B model already achieves an initial accuracy of approximately 80% on the training set. Consequently, the number of synthetic variational problems generated is limited, and the SvS training time is comparable to that of standard RLVR" (Section 5.5). This analysis is honest about the overhead but only reports it for the fully realized SvS system — it does not compare SvS's total FLOPs to standard RLVR trained for longer, or to standard RLVR with a larger batch size, which would isolate whether the overhead is buying unique benefits or merely more gradient steps.

Mitigation status. The paper does not attempt to reduce the accuracy-filtering cost — e.g., by using a lightweight classifier to predict problem difficulty from the prompt text alone (as suggested in prior sections for test-time compute difficulty estimation), or by performing a cheaper preliminary sampling (e.g., G = 2 solutions to estimate whether a problem is in the underperforming range before committing to the full G = 8 synthesis pipeline). The paper flags this implicitly in Section 5.5 (calling it "computational overhead") but treats it as an acceptable cost rather than a limitation to be solved. No future work is suggested for reducing the filtering overhead. This is a significant gap for practitioners deploying SvS on large-scale training runs where GPU-hours are the binding constraint: the method assumes the cost of fully solving all problems in the batch before deciding which ones to augment, which is the correct experimental practice for establishing proof-of-concept but suboptimal for production efficiency.

6.2 Hardest Problems Remain Essentially Unsolved

The assumption or constraint. SvS augments training problems whose group accuracy falls within [12.5%, 50.0%] (Section 3.2, Appendix D.1). Problems with accuracy below 12.5% — where the policy gets at most 1 correct solution out of 8 — are explicitly excluded from augmentation: they are filtered out by the condition acc_l < Acc(x) in Algorithm 1 Line 11. The paper justifies this by arguing that these problems are either too hard for the current policy or unsolvable, and that augmenting them would be wasteful because "the policy hasn't demonstrated the ability to produce correct solutions that could serve as reliable context for synthesis." This is a rational efficiency choice, but it creates a hard boundary: problems below the 12.5% threshold receive no benefit from SvS's augmentation machinery.

The consequence. The downstream effect is visible in the experimental results: on the hardest problems, SvS provides minimal or no improvement over standard RLVR. Figure 12 (Appendix E.4) shows that on the hardest problems within AIME exams (e.g., AIME 24 problem 15, AIME 25 problem 15), SvS's accuracy gains over RLVR are smaller than on easier problems. On the most extreme out-of-distribution benchmarks, SvS actually underperforms RLVR: on OlymH (OlymMATH en-hard) with DAPO-17k training, SvS achieves Pass@32 of 16.7% versus RLVR's 18.3% (Δ = -1.6; Table 1). On OlymE with DAPO-17k, SvS Pass@32 is 43.4% versus 47.1% (Δ = -3.7). These are the hardest evaluation benchmarks in the paper's suite, and SvS's performance on them is either neutral or negative relative to the baseline.

The structural reason is not that SvS is harmful on hard problems, but that the 12.5% accuracy floor means problems the policy almost never solves correctly are excluded from the self-play loop. The synthesis pipeline requires at least one correct solution to use as context for variational problem generation (Algorithm 1 Line 12: "Select {(x, y_i)} such that I = {i | R_c(y_i, a) = 1}" — if this set is empty, no synthesis occurs). For genuinely difficult problems where the policy's pass rate is near zero, SvS reduces to standard RLVR without augmentation. For problems just above the threshold (e.g., 12.5% accuracy, exactly one correct solution out of eight), SvS generates synthetic variants from a single solution — a narrow synthesis base that may produce less diverse variants than problems with higher accuracy (where multiple correct solutions provide multiple seeds for synthesis). The paper does not analyze whether single-seed synthesis is sufficient for meaningful diversity or whether synthetic problem quality degrades when few correct solutions are available.

What evidence exists in the paper. The evidence is indirect but consistent. Table 1 shows SvS underperforming RLVR on OlymH across multiple training configurations. Figure 12 shows the accuracy gap between SvS and RLVR narrowing on the hardest AIME problems. The paper does not directly measure SvS's impact as a function of training problem difficulty — e.g., by binning the 256 training batch problems by their accuracy and showing SvS's augmentation volume and quality per bin. Such an analysis would reveal whether the method's benefits are concentrated on medium-difficulty problems (consistent with the underperforming range design) and absent for the hardest problems, which would characterize the capability bound. The paper's general statement about "no path forward for genuinely novel or out-of-distribution reasoning that exceeds the base model's training distribution" (a framing from prior analysis) applies here but is not explicitly quantified for SvS's specific mechanism.

Mitigation status. The paper does not address this limitation directly. The 12.5% floor is treated as a design parameter, not as a limitation to be overcome. The paper does not discuss strategies for extending SvS to harder problems — e.g., by allowing augmentation from partially correct solutions (where the model gets some steps right even if the final answer is wrong), by using a separate retrieval mechanism to find similar problems the policy can solve and using those solutions as synthesis seeds, or by periodically lowering the accuracy threshold as training progresses. The inability to help on problems where the policy rarely or never succeeds is a fundamental bound on SvS's "self-improvement" scope: the method amplifies existing competence but cannot create it from nothing. This is not a criticism of SvS per se (no method can improve on problems the model never solves), but it delineates the class of problems for which SvS provides value — those where the model already has non-trivial but imperfect performance — and warns practitioners that SvS will not help on problems entirely outside the model's current capability range.

6.3 Synthesis Accuracy Thresholds Are Held Constant Without Sensitivity Analysis or Tuning

The assumption or constraint. Two critical thresholds govern SvS's behavior: the underperforming problem selection range [acc_l, acc_h] = [12.5%, 50.0%] and the synthesis reward range [a\hat{c}_1, a\hat{c}_h] = [12.5%, 62.5%] (Appendix D.1). These values are held constant across all model sizes (3B, 8B, 32B), both model families (Qwen, LLaMA), both training sets (MATH-12k, DAPO-17k), and both domains (math, code). The paper provides no sensitivity analysis — no sweep of alternative threshold values, no measurement of how performance varies as the ranges are tightened or widened, no evidence that these particular thresholds are near-optimal for any configuration.

The consequence. The consequence is twofold. First, a practitioner cannot determine whether these thresholds are likely to transfer to a new model, dataset, or domain without replicating the experiments. If a different base model has a different accuracy profile on the training set (e.g., higher initial accuracy than the 80% reported for Qwen-32B on MATH-12k), the 12.5%–50% range might capture either too few or too many problems, changing the augmentation volume and potentially the effectiveness of SvS. The paper's cross-model consistency (Table 2) suggests the thresholds are reasonably robust across the tested configurations, but robustness across three tested settings does not guarantee optimality — it is possible that a wider or narrower range would yield substantially better results for any given configuration.

Second, the relationship between the two threshold ranges is underexplored. The underperforming range [12.5%, 50.0%] determines which problems get augmented; the synthesis reward range [12.5%, 62.5%] determines which synthetic problems are considered successfully generated. The asymmetry between these ranges — the synthesis reward upper bound (62.5%) is higher than the underperforming upper bound (50.0%) — is intentional (allowing synthetic problems to be somewhat easier than their sources) but its impact is never tested. Would a synthesis reward upper bound of 50.0% (matching the underperforming range exactly) improve or degrade performance? Would a wider gap (e.g., 75.0%) allow too-easy problems to enter the buffer? The failure of the naive reward in Appendix G (where all solvable problems get positive reward) demonstrates that the upper bound matters, but the paper does not characterize how much it matters or where the cliff is.

The most critical missing analysis concerns the synthesis reward lower bound. The 12.5% floor on synthesis reward ensures that synthetic problems are solvable at least some of the time. But it also means that problems where the policy achieves exactly one correct solution out of eight (12.5% accuracy) — the bare minimum for passing the filter — are treated identically to problems where the policy achieves five correct solutions out of eight (62.5% accuracy, the upper bound). Both receive R_V = 1.0. This binary quantization discards information about how appropriately difficult the problem is. A continuous reward (e.g., R_V = 1 - |Acc - 0.375|, centered at the midpoint of the range) might provide a more nuanced training signal. The paper does not consider this.

What evidence exists in the paper. Appendix G (Figure 18) provides the only threshold-related evidence: the contrast between the naive reward (no upper bound) and the shaped reward (range-gated). This demonstrates that some upper bound is necessary, but does not establish that 62.5% is the right value. The ablation SvS-Asp (Table 3) changes the problem selection range from [12.5%, 50.0%] to [37.5%, 75.0%] (augmenting simpler problems) and finds that it underperforms — establishing that the selection range matters and that augmenting easier problems is worse. But this tests only one alternative range, not a sweep. No ablation varies the synthesis reward range independently of the selection range.

Mitigation status. The paper implicitly treats the thresholds as fixed hyperparameters that are part of the SvS method definition. Section 5.3 and Appendix D.1 report the values without discussing their sensitivity. There is no statement in the paper acknowledging the lack of sensitivity analysis as a limitation, and no future work is suggested for adaptive or learned threshold selection. A practitioner adopting SvS would need to either (a) use the paper's exact thresholds and hope they transfer, or (b) conduct their own sensitivity analysis, which is expensive (each threshold configuration requires a full training run). The paper's strong empirical results across diverse settings suggest the thresholds are not brittle, but the absence of any characterization of their sensitivity is a notable gap in an otherwise thorough experimental evaluation.

6.4 SvS Is Evaluated on Benchmarks Used for Checkpoint Selection

The assumption or constraint. The paper uses a simple and transparent model selection protocol: "checkpoints selected based on the best average scores from AIME 24 and 25" (Section 5.3). This applies to the main results in Tables 1 and 2, the ablation studies in Table 3 and Figure 7, and the Pass@k scaling experiments in Figure 6 (which use the same AIME 24 and AIME 25 benchmarks as both selection and evaluation targets). The paper does not use a separate held-out validation set for model selection — the same benchmarks that determine which checkpoint is "best" are then used to measure the method's performance.

The consequence. This is a form of implicit overfitting, distinct from explicit data contamination (the model never sees the problems during training — AIME problems are not in the training sets). Because model selection is performed on the same benchmarks used for evaluation, the reported numbers represent the best observed performance during training rather than the expected performance on those benchmarks. If SvS training is noisier than standard RLVR training (e.g., because the synthesis process introduces additional variance in the training signal), the peak SvS checkpoint might represent a favorable noise fluctuation that would not generalize to a different evaluation set used for selection. Conversely, if RLVR training is more stable (monotonically improving then plateauing), its peak checkpoint is likely closer to its expected performance. The paper does not report training variance or the stability of checkpoint rankings under different selection criteria, so the magnitude of this selection bias cannot be estimated from the provided data.

The practical consequence for interpreting the results: the Pass@32 gains of +18.3 on AIME 24 and +22.8 on AIME 25 (Table 1) are the gaps at the best checkpoint as defined by these benchmarks. If selection were performed on a different validation set (e.g., the Beyond-AIME benchmark, or the AMC benchmark, or a random subset of AIME problems from a different year), the selected checkpoint might differ and the reported numbers would change. The paper reports significant gains on every individual benchmark (Table 1), suggesting the gains are not purely an artifact of overfitting to the AIME selection metric, but the precise magnitude of the gains cannot be separated from the selection procedure. Table 3 provides some partial validation: the alternative strategies (Ext, Eup, SvS-Asp) are also evaluated with the same AIME-average checkpoint selection, and they underperform full SvS. This suggests the selection criterion is not solely responsible for SvS's advantage — SvS genuinely outperforms the alternatives — but the absolute accuracy numbers should be interpreted as upper bounds on out-of-sample performance.

A secondary concern: AIME 24 and AIME 25 each contain only 15 problems. The "best average" is computed over 30 total problems. A single problem on which SvS happens to dramatically improve (e.g., from 0% to 80% accuracy) can meaningfully shift the average, and if that problem's difficulty or topic is unusual, checkpoint selection based on these 30 problems may favor models that happen to solve that particular problem type well. The instance-level analysis in Figure 12 shows that SvS's gains are distributed across many AIME problems, mitigating this concern but not eliminating it — the selection criterion still depends on only 30 data points.

What evidence exists in the paper. The paper is transparent about the selection criterion (stated in Section 5.3) but does not discuss its implications for the reported results. The intermediate evaluation plots (Figures 7, 19, and others) show training trajectories on multiple benchmarks, allowing readers to observe whether SvS's advantage over RLVR is consistent throughout training (suggesting robustness) or concentrated at specific checkpoints (suggesting noise sensitivity). In most cases, SvS outperforms RLVR across a range of checkpoints, not just at the peak, which strengthens confidence in the gains. But these trajectory plots are also smoothed (5-step window) and lack error bars, so the consistency cannot be rigorously assessed. The paper does not report results using alternative selection criteria (e.g., "best checkpoint by Beyond-AIME performance" or "best checkpoint by average across all non-AIME benchmarks"), which would quantify the sensitivity of the reported numbers to the selection procedure.

Mitigation status. The paper does not acknowledge this as a limitation or propose alternative validation protocols. A standard remedy in RL for reasoning — holding out a subset of training problems for validation, or using a separate benchmark for selection versus evaluation — is not employed because the evaluation benchmarks are fixed and limited (AIME has only two recent years with public problems; no comparable competition-level benchmark with similar difficulty is used). The paper's cross-validation for strategy selection (mentioned in prior sections of this analysis) is not fully described and does not appear to apply to the checkpoint selection for the final reported results. This limitation is partially inherent to the domain — there are few benchmarks at AIME-level difficulty — but its impact on the reliability of the headline numbers should be disclosed.

6.5 No Combination of Search or Inference-Time Compute Scaling with SvS Training

The assumption or constraint. SvS is purely a training-time data augmentation strategy — it modifies the RLVR training loop by injecting synthetic problems but does not alter how the trained model is used at inference. All evaluations use standard sampling (temperature 1.0, top-p 0.7, single responses for Pass@1, multiple independent samples for Pass@k) with no test-time search, verifier-guided selection, or inference-time compute scaling. The paper operates in a paradigm where the model is trained with SvS and then evaluated "as-is," without the inference-time strategies that have been shown to substantially boost reasoning performance in other work (e.g., majority voting over many samples, best-of-N with verifier reranking, tree search).

The consequence. The consequence is that SvS's reported performance represents a lower bound on what could be achieved by combining SvS-trained models with inference-time compute scaling, but also that the comparison between SvS and standard RLVR may be confounded by their differential interaction with inference-time strategies. A model trained with SvS has higher policy entropy and greater generation diversity (Figure 5). This diversity could make SvS-trained models more amenable to inference-time techniques like majority voting or best-of-N selection, because these techniques benefit from diverse candidate solutions that cover different reasoning approaches. If an SvS-trained model produces 32 diverse but sometimes-incorrect solutions, majority voting might extract the correct answer from the plurality; if a standard RLVR model produces 32 near-identical solutions (due to entropy collapse), majority voting adds little over a single sample. The Pass@32 results in Table 1 are consistent with this hypothesis (SvS's Pass@32 gains are proportionally larger than its Pass@1 gains), but the paper does not test whether explicitly combining SvS training with inference-time search or verifier reranking yields synergistic benefits beyond what either technique achieves alone.

The inverse concern: SvS's training cost (Section 6.1 above) means that for a fixed total compute budget (training + inference), it is unclear whether SvS training plus simple sampling outperforms standard RLVR training with additional inference-time compute. If SvS training costs 2–3× more per iteration and achieves +10.5 Pass@1 on AIME 24 over RLVR (Table 1), but standard RLVR with best-of-32 majority voting achieves a comparable or larger Pass@1 gain (not evaluated in the paper), the practical case for SvS depends on the relative cost of training compute versus inference compute — a tradeoff the paper does not analyze.

What evidence exists in the paper. The paper contains no experiments combining SvS with inference-time compute scaling strategies. The closest proxy is the Pass@k evaluation in Figure 6, which can be interpreted as the performance achievable with an oracle selection mechanism (picking the correct answer if it exists among k samples). This shows that SvS's Pass@k ceiling is substantially higher than RLVR's, suggesting that even with perfect inference-time selection, SvS-trained models outperform RLVR-trained models. But this is not the same as testing practical inference-time strategies (majority voting, verifier reranking, best-of-N weighted) that operate without oracle access.

Mitigation status. The paper acknowledges this implicitly through its focus on Pass@k as a metric of "reasoning boundaries" — the idea being that if the Pass@k ceiling is higher, practical inference-time strategies can (in principle) extract that performance. But this is a theoretical argument, not an empirical one. The paper does not evaluate whether standard inference-time techniques (majority voting among 32 or 64 samples) close the gap between Pass@1 and Pass@32 for SvS-trained models, nor does it compare the inference-time efficiency of SvS versus RLVR models (e.g., how many samples each needs to reach a target accuracy). The paper positions SvS as a training contribution and leaves the training-inference interface unexplored. This is a legitimate scope limitation for a methods paper, but it means a practitioner cannot determine the full-system performance of SvS (training + optimal inference) from the reported results alone.

6.6 Single Domain Paradigm (GRPO on Verifiable Rewards) Without Evidence on Alternative RL Algorithms

The assumption or constraint. All experiments in the paper use GRPO as the underlying RLVR optimizer, specifically the variant with Clip-Higher (ε = 0.28), Token-Level Loss, and Dynamic Sampling from DAPO (Yu et al., 2025a). The paper claims that "the SvS framework relies exclusively on the policy model itself, without any external guidance or distillation, achieving all improvements through end-to-end self-improvement. Moreover, the SvS augmentation is agnostic to RLVR optimization algorithms and can be flexibly incorporated into other methods, such as PPO, GSPO, and Reinforce++" (Section 3.1). This claim of algorithmic agnosticism is stated without experimental support.

The consequence. The claim that SvS is algorithm-agnostic is plausible — it operates at the data level, inserting synthetic problems into the training buffer before the policy update, and should in principle work with any RL algorithm that consumes (prompt, response, reward) tuples. But GRPO has specific properties that may interact with SvS in ways that other algorithms would not. GRPO uses group-relative advantages (Equation 5), which means the value of a synthetic problem depends not only on its absolute quality but on how its reward compares to other problems within the same batch. If SvS generates synthetic problems that tend to have systematically different reward distributions than original problems (e.g., slightly higher or lower average rewards), the group-relative normalization would treat them differently than an algorithm using a learned value baseline (PPO) or an absolute reward threshold. GRPO also discards groups with all-correct or all-incorrect solutions (advantages collapse to zero), which makes the mixed-correctness filtering in SvS (Lines 8 and 19 of Algorithm 1) especially important — synthetic problems that are too easy or too hard are filtered out. An algorithm that handles degenerate groups differently (e.g., by using a learned baseline that provides advantages even when all group rewards are equal) might need different filtering criteria.

The consequence for practitioners is uncertainty about transferability. If an organization has invested in a PPO-based RLVR infrastructure (with a critic model, value function training, GAE advantage estimation), the paper provides no evidence that SvS would work as well in that setting. The interaction between SvS's data augmentation and GRPO's group-relative mechanics could be a substantial part of SvS's effectiveness, or it could be incidental — the reader cannot tell from the reported experiments.

What evidence exists in the paper. None. The paper provides zero experiments with any algorithm other than GRPO. The claim of agnosticism is entirely theoretical. The paper does not discuss whether SvS's design choices — particularly the mixed-correctness filtering and the R_V reward shaping — are motivated by GRPO-specific considerations or are expected to generalize. For example, the R_V = I(12.5% ≤ Acc ≤ 62.5%) reward is a binary signal; in GRPO, this binary reward is group-normalized to compute advantages. In PPO, where advantages are computed using a learned value function and temporal difference errors, a binary reward with sparse positive signals might interact differently with the value function training, potentially requiring different reward scaling or a different threshold design. The paper does not address any of these algorithm-specific considerations.

Mitigation status. The paper does not acknowledge this as a limitation. The claim of algorithmic agnosticism is stated in the introduction (Section 3.1) as a feature of the method, but it remains an untested assertion. A minimal validation would involve running SvS with one alternative algorithm (e.g., PPO or Reinforce++) on a single model-dataset configuration and showing comparable improvements over the standard-RLVR baseline under that algorithm. This would establish that the gains are not GRPO-specific without requiring full-scale experiments across all configurations. The paper includes a comparison with Clip-CoV (Appendix F), which is an entropy regularization method applied on top of the same GRPO optimizer, and shows that SvS outperforms it — but this tests SvS against a different regularization strategy, not against a different optimization algorithm. The combination of SvS with Clip-CoV (Figure 17) similarly uses GRPO throughout. A practitioner considering SvS for a non-GRPO RLVR pipeline has no empirical guidance from this paper.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a data-centric reconceptualization of the entropy collapse problem in RLVR that shifts research attention from algorithmic fixes to data curriculum design. The dominant prior response to the well-documented finding that RLVR training causes policy entropy to decline and Pass@k to plateau (Cui et al., 2025b; Yue et al., 2025) was to treat it as an optimization pathology: add entropy regularization terms to the loss (Clip-CoV), adjust sampling temperatures (An et al., 2025; Chen et al., 2025), or modify advantage computation to incentivize diversity. These approaches accept the fixed-dataset RLVR paradigm and attempt to counteract its tendency toward entropy collapse through explicit constraints on the policy's output distribution.

SvS argues — and demonstrates through the preliminary experiment in Figure 2 — that entropy collapse is fundamentally a data stationarity problem, not an optimization problem. When the training set is fixed, the policy faces a stationary reward landscape where memorizing high-reward solution templates is the optimal strategy. Adding data diversity (rephrased problems in Figure 2) slows entropy decline; updating the diverse data mid-training actually reverses the decline, causing entropy to rise as the policy re-explores. No algorithmic changes were needed to produce this effect — the policy responded to environmental non-stationarity by resuming exploration, using the same GRPO optimizer that had previously driven it toward collapse. This is a clean demonstration that the policy retains the capacity for diverse generation throughout training; it merely converges to a low-entropy equilibrium because the fixed environment stops rewarding exploration.

This reframing has several consequences for how the field thinks about RLVR training:

It redirects research investment from algorithmic regularization to data pipeline design. Entropy regularization methods (Clip-CoV and its variants) address the symptom of declining output diversity but not the cause: memorization of previously successful solutions from a static problem set. A policy can maintain high entropy while still primarily reproducing memorized solutions intermixed with noise — entropy is a distributional property, not a guarantee of meaningful exploration. SvS addresses the root cause by continuously injecting novel problems that the policy has never seen, forcing genuine exploration of new reasoning strategies regardless of the policy's entropy level. The paper's comparison in Appendix F (Figure 16), where SvS consistently outperforms Clip-CoV across multiple benchmarks, empirically supports this distinction — the data-level intervention beats the algorithmic-level intervention. This suggests that future RLVR research should prioritize data curriculum design (how to maintain problem novelty throughout training) over loss-function engineering (how to force diversity from a stationary policy).

It reconciles contradictory claims about self-play and self-improvement. Prior work on self-play for LLM reasoning (Cheng et al., 2024; Liang et al., 2025) demonstrated promising results but did not explain when self-play helps versus when it might fail. The SvS framework provides specific boundary conditions: self-play augmentation works when (a) the synthetic problems are derived from correct solutions (preserving answer validity without external verification), (b) the problems are generated online (preventing memorization across iterations), and (c) the synthesis is constrained by the solver's difficulty profile (preventing the self-exploitation documented in Appendix G). The failure of the naive synthesis reward in Appendix G — where removing the difficulty gate caused complete training collapse — demonstrates that self-play is not inherently beneficial; it requires careful structural constraints to avoid degenerate feedback loops. This provides a diagnostic framework for evaluating future self-play proposals: check whether the generator is rewarded for solvability (which leads to trivial problems) or for appropriate difficulty (which leads to productive learning).

It elevates Pass@k to a first-class evaluation metric for reasoning advancement. While Yue et al. (2025) first observed that standard RLVR fails to improve Pass@k at scale, this paper operationalizes that observation as a primary evaluation framework and uses it to demonstrate a qualitative gap between SvS and standard RLVR that Pass@1 obscures. The finding that SvS's Pass@32 gains (+18.3 and +22.8 on AIME 24/25, Table 1) are proportionally larger than its Pass@1 gains (+10.5 on both benchmarks) indicates that SvS is not merely sharpening existing capabilities but expanding the set of problems the model can ever solve. This establishes a methodological standard: future RLVR work should report both Pass@1 and Pass@k at large k (at least 32, ideally 1024 as in Figure 6), and should interpret the relationship between them — if Pass@k gains are proportionally smaller than Pass@1 gains (as in standard RLVR), the method is primarily improving sampling efficiency; if Pass@k gains are proportionally larger (as in SvS), the method is expanding reasoning coverage. This distinction provides a principled way to separate "sharpening" from "learning" in RLVR evaluation.

It demonstrates that solution-to-problem synthesis is a viable self-supervised paradigm for verifiable-reward domains. The paper's most transferable conceptual contribution is the insight that conditioning data synthesis on outputs (solutions) rather than inputs (problem statements) solves the answer verification problem that constrains standard data augmentation. Because a correct solution contains the complete specification of the problem it answers, reconstructing a problem from a solution preserves the original ground-truth answer by construction. This paradigm generalizes beyond math reasoning: code generation (synthesize new coding problems from correct implementations, Figure 8), theorem proving (synthesize new theorems from correct proofs), or any domain where a verifiable output encodes the task specification. The paper shows this transfer concretely on code generation, where SvS achieves comparable performance to standard RLVR in ~5× fewer training steps (Figure 8). This suggests that the self-play synthesis paradigm is not math-specific but domain-general, opening a new class of self-improving training methods for any task with automatic verification.

It identifies difficulty-gated self-play as a necessary guardrail for self-improving systems. The failure mode documented in Appendix G — where the policy exploits naive synthesis rewards by embedding answer hints in problem text, causing the entire training loop to collapse — is a specific instance of a general problem: when a model both generates and evaluates its own training data, weak evaluation criteria create perverse incentives that destroy the training signal. The solution — constraining the generator by the solver's difficulty profile rather than just its success rate — establishes a design principle for self-play systems: reward the generator for producing data at the frontier of the solver's capability, not for producing data the solver can trivially succeed on. This principle has implications beyond RLVR, applying to any iterative self-improvement system (e.g., self-distillation, self-play for alignment, automatic curriculum learning) where a model generates training data that it then learns from.

Follow-Up Research This Work Enables

Adaptive and learned difficulty thresholds for synthesis gating. The paper uses fixed accuracy thresholds for underperforming problem selection (12.5%–50%) and synthesis reward (12.5%–62.5%) across all model sizes and training stages, with no sensitivity analysis. A natural extension is to make these thresholds adaptive: as training progresses and the policy improves, the thresholds should shift to maintain a constant fraction of problems in the "productive struggle" zone. A simple approach would be to set thresholds based on the current distribution of training accuracies — e.g., select problems in the 20th to 60th percentile of accuracy for augmentation — which automatically adjusts as the policy improves. A more ambitious version would train a lightweight predictor (perhaps a linear probe on the policy's intermediate representations) to estimate problem difficulty from the prompt text alone, enabling difficulty-gated synthesis without the cost of generating G = 8 solutions per problem first (addressing the limitation in Section 6.1). A strong follow-up would compare fixed vs. adaptive thresholds on DAPO-17k training with Qwen2.5-32B-Instruct, measuring both final performance and the fraction of training compute spent on synthesis filtering vs. actual learning.

Combining SvS with inference-time compute scaling (best-of-N, verifier reranking, tree search). The paper demonstrates that SvS-trained models have higher policy entropy and substantially higher Pass@k ceilings than standard RLVR models (Figure 6), but evaluates them only with independent sampling. A critical open question is whether SvS-trained models are more amenable to inference-time compute scaling strategies (majority voting, verifier-guided best-of-N, beam search over reasoning steps) than standard RLVR models. The hypothesis: because SvS maintains generation diversity, its 32 or 64 independent samples will cover a wider range of reasoning approaches, making aggregation techniques (majority voting) or selection techniques (verifier reranking) more effective — the verifier has more genuinely distinct candidates to choose from. This would be tested by training pairs of SvS and RLVR models to matched Pass@1, then comparing their Pass@k with practical inference-time strategies (majority voting among 64 samples, best-of-N weighted with a PRM verifier) rather than oracle selection. If SvS models show larger gains from inference-time compute than RLVR models at the same Pass@1, this would establish that training-time diversity and inference-time selection are complementary and should be co-designed.

Extending SvS to the hardest problems via partial-credit synthesis from incorrect solutions. The paper's 12.5% accuracy floor for augmentation means problems the policy rarely solves correctly receive no benefit from SvS (Section 6.2). A natural extension is to use partially correct solutions as synthesis context: even when the final answer is wrong, the solution may contain correct reasoning steps that reference the problem's constraints, variables, and structure. A partially correct solution to a near-impossible problem might still encode enough information to reconstruct a solvable variational problem — perhaps one that isolates a sub-problem the policy can handle. For example, if a complex geometry problem requires three sequential insights and the policy's incorrect solutions consistently get the first insight right, those partial solutions could be used to synthesize problems that test only the second and third insights (graduated difficulty). This would require a more sophisticated synthesis prompt that explicitly identifies which parts of the solution are correct, perhaps guided by a process reward model (PRM) that scores intermediate steps. A strong follow-up would test this on the DAPO-17k training set, identifying problems where the policy's accuracy is 0%–12.5% (below the current SvS threshold) but where a PRM identifies at least one correct reasoning step in some solutions, and measuring whether partial-credit synthesis expands the effective range of SvS to harder problems.

Stress-testing SvS on out-of-distribution generalization with truly novel benchmarks. The paper evaluates SvS on AIME 24 and AIME 25, the same benchmarks used for checkpoint selection (Section 6.4). A definitive test of whether SvS expands reasoning boundaries — rather than improving within-distribution performance — would evaluate on a held-out competition benchmark from a different source that was not used for any selection decisions. Candidates include problems from the International Mathematical Olympiad (IMO) shortlist, the Putnam competition, or the Chinese GaoKao mathematics section (distinct from the Gaokao-2023 benchmark already in the paper's suite). The key comparison: does SvS's Pass@k advantage over RLVR persist on problems that require genuinely different mathematical knowledge or reasoning paradigms than those in the training set? If SvS's gains are primarily from practicing diverse surface forms of known problem types, they may not transfer to problems requiring novel mathematical concepts. If SvS's gains do transfer, it would suggest that the self-play loop teaches general reasoning meta-skills (problem decomposition, verification, backtracking) rather than domain-specific pattern matching. This experiment would use the DAPO-17k SvS model evaluated zero-shot (no fine-tuning) on the held-out competition benchmark, with Pass@k measured at k = 1, 32, 256, and compared against the standard RLVR baseline and the base model.

Combining SvS with structured exploration mechanisms (MCTS, tree search during training). SvS maintains exploration by continuously renewing the training problems, but the exploration within each problem is still governed by standard temperature-based sampling (temperature = 1.0). A natural hybrid is to combine SvS's data-level exploration (novel problems each iteration) with algorithmic-level exploration (structured search over reasoning paths within each problem). For instance, instead of independent sampling of G = 8 solutions per problem, the policy could use Monte Carlo Tree Search (MCTS) guided by a process reward model to systematically explore different reasoning branches — and the correct solutions discovered through search could then serve as synthesis seeds for variational problem generation. This would address the limitation that SvS only augments from randomly discovered correct solutions; MCTS could discover correct solutions for harder problems that random sampling misses, expanding the synthesis base. A practical experiment would augment the GRPO rollout phase with tree-search during training (not just at inference), using a lightweight PRM trained on the base model's outputs, and measure whether MCTS-augmented SvS outperforms standard SvS on the hardest AIME problems (where standard SvS shows smaller gains, Figure 12). The key question is whether the additional compute spent on training-time search pays off in better synthetic problems that accelerate learning, or whether the overhead outweighs the benefit.

Generalizing the solution-to-problem paradigm to non-math verifiable domains. The code generation experiment (Figure 8) provides initial evidence that SvS transfers across domains, but it is tested on only one model (Qwen2.5-7B-Instruct) with limited benchmark coverage. A systematic extension would test the solution-to-problem synthesis paradigm on two additional verifiable-reward domains: (1) formal theorem proving using Lean or Isabelle, where a correct proof is machine-checkable and contains the complete statement of the theorem it proves — synthesizing new theorem statements from proofs and using the proof checker as the verifier; (2) reward modeling for RLHF, where a correct human preference judgment can be used to synthesize new comparison pairs that should elicit similar judgments. For theorem proving, the key adaptation is the synthesis prompt: the model would be fine-tuned to reconstruct theorem statements from proof scripts, then the synthesized theorems would be verified by attempting to prove them with the policy (and checking whether the policy's proof succeeds). For reward modeling, synthesizing preference pairs from judgments tests whether the solution-to-problem paradigm works for subjective/relative evaluation tasks, not just absolute correctness tasks. A strong follow-up would benchmark both domains with the same experimental structure as the math experiments (3B/8B/32B models, SvS vs. RLVR baselines, Pass@k evaluation where applicable), to characterize the domain-generality of the synthesis paradigm.

Ab initiation testing of SvS on base models without instruction tuning. All experiments in the paper start from instruction-tuned models (Qwen2.5-32B-Instruct, LLaMA-3.1-8B-Instruct). These models already have strong reasoning capabilities and have been exposed to diverse problem formats during supervised fine-tuning. An important stress test is whether SvS works when starting from a base model (no instruction tuning, no chain-of-thought training), where the initial policy has very low accuracy on training problems and very unstructured outputs. The hypothesis: SvS's underperforming problem filter (12.5%–50% accuracy) assumes the existence of problems with at least occasional correct solutions. A base model might have near-zero accuracy on most MATH problems, producing almost no correct solutions to use as synthesis seeds, causing SvS to degenerate to standard RLVR with no augmentation. If this hypothesis is correct, SvS requires a "warm-up" phase of standard RLVR or supervised fine-tuning before the self-play loop becomes productive — an important boundary condition on the method's applicability. If the hypothesis is wrong and SvS works even from very low initial accuracy (perhaps because even rare correct solutions are sufficient seeds, or because the synthesis prompt generalizes from the base model's pretraining knowledge), that would strengthen the claim of robustness. This experiment would train Qwen2.5-32B (base, non-instruct) with SvS on MATH-12k and compare against the instruct-model results.

Practical Applications and Downstream Use Cases

Cost-efficient training of competition-level reasoning models. The most direct application is for organizations training LLMs to perform at competition-math levels (AIME, IMO, Olympiad). The paper's headline result — +18.3 and +22.8 Pass@32 on AIME 24/25 over standard RLVR (Table 1) — translates to models that can solve ~70% of AIME problems given 32 attempts, compared to ~50% for standard RLVR. For an organization building a math reasoning API or competition-math tutoring system, this represents a step-change in capability without requiring a larger base model, additional training data, or human annotation. The D25k result (Table 2, average 62.8% across all benchmarks) demonstrates that SvS works with mixed-format training data and can be steered by training set composition to balance competition-level and general math performance. The practical deployment recipe emerging from the paper: start with a strong instruction-tuned model (32B parameter class), train on a diverse math dataset including both integer-answer competition problems and open-ended problems (to prevent format overfitting), apply SvS with the default thresholds (12.5%–50% underperforming, 12.5%–62.5% synthesis reward), train for 600 steps, and select checkpoints based on validation performance. The overhead relative to standard RLVR is manageable: on MATH-12k (where initial accuracy is already high), SvS training time is comparable to RLVR; on harder datasets, overhead is 2–3× in early training but declines as the policy improves (Figure 9).

Data-efficient domain adaptation to specialized reasoning tasks. SvS's self-play paradigm is particularly valuable when adapting a general reasoning model to a specialized domain with limited labeled data. If an organization has a small set of domain-specific problems with verified answers (e.g., 1,000 pharmaceutical calculation problems, 500 engineering statics problems, 300 legal logic problems), standard RLVR on this small set would quickly collapse to memorization. SvS can amplify the effective training data: each domain problem becomes a seed for dozens of variational problems, creating a continuously renewing curriculum that prevents overfitting while preserving domain-specific content. The key advantage over external data augmentation (hiring annotators, using GPT-4 to generate variants) is that SvS requires no additional annotation budget and produces problems aligned with the policy's current capability level. A practical implementation would: (1) fine-tune the model on the domain problems with standard supervised learning to establish a baseline, (2) apply SvS with the domain problem set as D, using the same synthesis prompt but potentially adjusting the accuracy thresholds if the domain problems have different difficulty profiles than MATH, and (3) monitor training entropy (Figure 5) to detect if the synthesis volume is appropriate — if entropy is still declining, the thresholds may need adjustment or the dataset may be too narrow for effective synthesis.

Self-improving code generation models with reduced training time. The code generation results (Figure 8) demonstrate a practical benefit that is distinct from the math reasoning results: SvS achieves comparable performance to standard RLVR in ~5× fewer training steps. For organizations training code generation models, this translates directly to reduced GPU-hours and faster iteration cycles. A practical deployment could use SvS as a rapid fine-tuning stage after initial supervised training: train a base coding model (e.g., Qwen2.5-Coder-7B) on a set of 10k–20k coding problems with unit tests, apply SvS for 100–200 steps (rather than 600+ for standard RLVR), and deploy. The synthesis prompt adaptation for code (Figure 20, bottom) is straightforward — it asks the model to reconstruct a programming problem from a correct solution, specifying inputs, outputs, and constraints. The key implementation detail is the verifier: for code, correctness is determined by executing the generated code against unit tests, which is faster and more reliable than math answer extraction. This makes the code domain an ideal testbed for SvS because the synthesis → solving → verification loop has lower latency and fewer false positives/negatives than math answer matching. A production system could use the unit test pass rate as both the RL reward and the synthesis difficulty signal, simplifying the pipeline.

Mitigating capability regression on general benchmarks during specialized RL training. A well-documented problem with RLVR training is that models improve on the training domain (math reasoning) but regress on general capabilities (instruction following, factual knowledge, broad QA). The paper demonstrates this for standard RLVR in Table 4 (Appendix E.3): Qwen2.5-32B-Instruct trained with standard RLVR on DAPO-17k drops from 73.35% to 72.99% average across 9 general benchmarks. SvS not only avoids this regression but improves general performance to 73.77%, exceeding both the initial model and the RLVR baseline. The mechanism is the variational problem synthesis task: by training the model to reconstruct diverse problem statements from solutions, SvS acts as a regularizer that preserves the model's general language understanding and generation capabilities. For organizations deploying RLVR-trained models in production (where the model must handle both specialized reasoning queries and general conversation), SvS offers a practical solution to the capability-regression problem without requiring multi-task training, data mixing, or KL regularization to a frozen reference model. The synthesis task is learned jointly with the reasoning task from the same self-generated data, adding no annotation cost. A practical deployment would track both target-domain accuracy and general benchmark performance during training, using the general benchmark scores as an early warning signal for overfitting — if general performance starts declining despite SvS, the synthesis volume may be too low or the training data may be too narrow.

When to Prefer This Method

The paper does not articulate an explicit tradeoff matrix against named alternatives (e.g., "use Clip-CoV when X, use SvS when Y"). However, its experiments establish implicit preference conditions through comparison with standard RLVR, Clip-CoV (Appendix F), and alternative augmentation strategies (Table 3, "Ext" and "Eup"). The following decision logic can be extracted from the paper's evidence:

Prefer SvS over standard RLVR when: (a) the training set is limited in size or diversity (MATH-12k, DAPO-17k class — 10k–20k problems), such that standard RLVR would converge to memorization; (b) the evaluation metric includes Pass@k at large k, not just Pass@1, since SvS's relative advantage grows with k (Figure 6); (c) the model will be deployed with inference-time sampling (majority voting, best-of-N), since SvS's diversity makes these techniques more effective; and (d) the training budget can absorb 1.1–3× per-iteration overhead (Figure 9), with overhead being lower when the base model already has high accuracy on the training set.
Prefer standard RLVR over SvS when: (a) the training set is already large and diverse enough to prevent entropy collapse (though the paper provides no threshold for "large enough" — Figure 2 suggests 12k problems is insufficient for the 32B model); (b) training compute is tightly constrained and the overhead of synthesis plus synthetic solving cannot be absorbed; (c) only Pass@1 matters for the deployment use case (though even then, Table 1 shows SvS improves Pass@1 by 5.4 points on average, so the tradeoff is not obvious); or (d) the base model's accuracy on training problems is near zero, meaning few or no correct solutions exist to seed synthesis (SvS degenerates to standard RLVR in this regime, as discussed in Section 6.2, and the overhead is wasted).
Prefer SvS over entropy regularization (Clip-CoV) when: the goal is to expand reasoning coverage (Pass@k) rather than just maintain output diversity. The paper shows that Clip-CoV underperforms SvS across benchmarks (Figure 16), and that SvS can be combined with Clip-CoV for additive gains (Figure 17). This suggests SvS addresses the root cause (data stationarity) while entropy regularization addresses a symptom — and if only one can be used, SvS provides larger benefits at the cost of additional training complexity.
Prefer SvS with D25k-style mixed-format training data when: the deployment requires strong performance on both integer-answer benchmarks (AIME, AMC) and open-ended-answer benchmarks (MATH-500, Olympiad-Bench). Training SvS on integer-only data (DAPO-17k) causes regression on open-ended benchmarks (Table 2, D17k SvS: MATH-500 drops from 87.0 to 75.6). Adding 8k open-ended problems (D25k) recovers this performance while largely preserving AIME gains.

These conditions are derived from the paper's empirical results rather than stated as explicit design guidance, and should be treated as hypotheses to be validated in deployment contexts that differ from the paper's experimental setup (different models, datasets, domains).