Pre-train Space Reinforcement Learning: From P(y|x) to P(y)

🎯 Pitch

Optimizing the marginal token distribution P(y) prunes entire incorrect reasoning subspaces and spontaneously amplifies reflective behaviors like self-correction by over 14×—without ever seeing a task prompt. This 'Negative Sample Reinforcement' effect can bootstrap post-training RL, yet too much pre-train RL produces over-exploratory models that sabotage downstream optimization.

1. Executive Summary

This paper introduces Pre-train Space RL (PreRL), a paradigm that applies reward-driven online updates directly to the marginal distribution P(y) rather than the standard conditional distribution P(y|x), and dissects its behavior into Positive Sample Reinforcement (PSR) and Negative Sample Reinforcement (NSR) on Qwen3 models trained on the MATH dataset. The central discovery is that NSR-PreRL prunes incorrect reasoning paths while stimulating endogenous reasoning capabilities — increasing transition and reflection thoughts by 14.89× and 6.54× respectively — and achieves comparable accuracy with 3× fewer training steps than standard RL. Building on this insight, the authors propose Dual Space RL (DSRL), a Policy Reincarnation strategy that initializes models with NSR-PreRL to expand the reasoning horizon before transitioning to standard GRPO, establishing that pre-train space optimization yields consistent gains over GRPO across AIME24 (+4.69 points), AIME25 (+2.50 points), and out-of-distribution benchmarks like MMLU-Pro (+5.37 points) only when the NSR warmup phase is carefully bounded — excessive warmup causes over-exploratory generation that undermines subsequent post-train optimization.

2. Context and Motivation

The Core Problem: RLVR's Gains Are Bounded by What the Base Model Already Knows

The paper addresses a fundamental limitation in how we currently apply reinforcement learning to improve LLM reasoning. Reinforcement learning with verifiable rewards (RLVR) — the paradigm behind DeepSeek-R1 and similar reasoning models — operates entirely in what the paper calls the Post-train Space: it optimizes the conditional distribution P(y|x), meaning the model learns to produce better reasoning trajectories given specific input questions. While this has produced remarkable results (Guo et al., 2025a; Team et al., 2026), a growing body of evidence suggests that RLVR's effectiveness has a hard ceiling. As the paper states:

"Existing work has demonstrated that incentivized reasoning paths already exist within the base model's output distribution, suggesting that the reasoning capacity of RLVR-trained models may be fundamentally bounded by the capabilities of the base model" (Section 1)

This claim is supported by recent empirical findings cited in the paper: Yue et al. (2025) argue that RLVR primarily "over-sharpens" the base model's existing sample distribution rather than creating genuinely new reasoning capabilities; Zuo et al. (2026) and Peng et al. (2025) similarly question whether RL can push reasoning beyond what was already latent in the pretrained parameters. In other words, if a base model has essentially zero probability of generating a correct solution to a hard problem, no amount of RLVR on P(y|x) can create that capability — it can only amplify what already exists.

This limitation is both theoretically significant and practically important. For model developers, it means that post-training RL alone cannot overcome fundamental capability gaps inherited from pretraining. For the broader goal of building reasoning systems that can tackle increasingly difficult problems, it implies that we need mechanisms that operate at a deeper level — modifying the model's intrinsic knowledge and exploration capacity rather than just its conditional sampling behavior.

Why This Problem Matters: The Exploration-Exploitation Bottleneck

The gap between what RLVR can achieve and what is theoretically possible has concrete implications that the paper is motivated by:

1. The initialization bottleneck. If RLVR can only sharpen existing capabilities, then the quality of the pretrained base model becomes the single most important factor determining final reasoning performance. This creates a "rich get richer" dynamic where larger, better-pretrained models have an inherent and potentially insurmountable advantage, and where simply scaling pretraining compute remains the dominant (and expensive) path to improvement.

2. The exploration deficit. RLVR optimizes by reinforcing trajectories that lead to high rewards. On hard problems where the base model rarely or never generates correct solutions by chance, this creates a chicken-and-egg problem: the model cannot reinforce what it cannot generate, and it cannot learn to generate what it has never seen reinforced. The paper frames this as an exploration-exploitation tradeoff (Section 5.1): standard RLVR is good at exploitation (selecting among known good strategies) but poor at exploration (discovering entirely new strategies not present in the base distribution).

3. Distribution shift between pretraining and reasoning. A deeper underlying issue is that pretrained models acquire their knowledge from broad web corpora through next-token prediction — a fundamentally different objective from the structured, multi-step reasoning required for mathematical problem-solving. As the paper notes:

"pre-training relies on static corpora for passive learning, leading to a pronounced distribution shift between pre-trained knowledge and the task distribution encountered during post-training, which hinders targeted reasoning enhancement" (Section 1)

This distribution gap means that even when relevant knowledge exists somewhere in the model's parameters (e.g., it has seen mathematical concepts during pretraining), it may not be organized in a way that supports effective reasoning. RLVR can redirect the output distribution toward higher-reward trajectories, but it cannot fundamentally restructure how knowledge is encoded internally.

The Pre-train Space Paradigm and Its Current Limitations

An alternative approach — one that the paper builds upon but argues has been underdeveloped — is to optimize directly in what the paper calls the Pre-train Space: the marginal distribution P(y). Unlike P(y|x), which conditions every update on a specific input question, P(y) represents the model's unconditional distribution over reasoning trajectories — its intrinsic capacity to generate structured, logical reasoning independent of any particular prompt.

The paper formalizes this distinction through two definitions (Section 2):

Post-train Space Optimization updates π_θ(y|x), conditioning every gradient step on the specific input x. This is what standard RLVR (GRPO, PPO, etc.) does.

Pre-train Space Optimization updates π_θ(y) directly, optimizing the intrinsic distribution of reasoning trajectories without the conditioning on x. This is closer in spirit to what pretraining does — encoding general capabilities into parameters — but unlike traditional pretraining, it can be guided by reward signals.

The key intuition is that optimizing P(y) should yield deeper, more fundamental improvements than optimizing P(y|x). By internalizing reasoning capability at the level of the marginal distribution, the model should acquire a broader exploration capacity, be better initialized for subsequent conditional fine-tuning, and potentially escape the exploration deficit that plagues standard RLVR on hard problems.

However, existing approaches to pre-train space optimization have a critical limitation. Traditional pre-training and continual pre-training (Gupta et al., 2023; Wang et al., 2024b; Zhou et al., 2025) operate on static corpora through passive next-token prediction. While this can inject domain-specific knowledge into P(y), it suffers from exactly the distribution shift problem the paper identifies: the model learns from pre-collected data that may not reflect the actual distribution of reasoning trajectories encountered during RL training. Moreover, passive learning cannot adaptively respond to the model's evolving capabilities — it is a one-shot injection of knowledge with no feedback loop.

As the paper frames it:

"This observation motivates a natural idea: introducing RL into the pre-train space to synergize knowledge internalization with reward-driven optimization" (Section 1)

Where Existing Approaches Fall Short

The paper identifies several specific gaps in the current landscape:

1. RLVR is confined to P(y|x). Despite the recognized importance of pre-train space optimization, no prior work has systematically studied whether and how RL can operate on the marginal distribution P(y) for reasoning tasks. Studies on exploration-exploitation tradeoffs in RLVR (Li et al., 2025a; Wang et al., 2026b; Yu et al., 2026) remain entirely within the conditional distribution framework. As the paper states:

"Whether and how such reward-driven mechanisms can operate on the marginal distribution P(y) remains unexplored" (Section 5.1)

2. Pre-train space methods lack active, reward-guided learning. Continual pre-training can encode domain knowledge but does so passively — it cannot distinguish between informative trajectories and uninformative ones, cannot prune incorrect reasoning patterns, and cannot adapt its learning strategy based on the model's current strengths and weaknesses. The paper argues that this passivity fundamentally limits how effectively P(y) can be aligned with reasoning tasks.

3. The relationship between P(y) and P(y|x) is not understood for RL purposes. Even if one wanted to apply RL to the pre-train space, a fundamental question remains unanswered: does optimizing the marginal log-likelihood actually improve the conditional policy needed for task performance? The gradients of log P(y) and log P(y|x) could, in principle, point in different or even opposing directions. Without theoretical and empirical validation of gradient alignment, pre-train space RL remains an unjustified gamble.

4. The role of positive versus negative reinforcement in the pre-train space is unknown. Standard RLVR uses both positive samples (to reinforce correct behavior) and negative samples (to suppress incorrect behavior). But the dynamics of these two mechanisms may differ dramatically in the pre-train space, where the absence of input conditioning changes the structure of what the model learns from each type of signal. Prior work (Zhu et al., 2025) has studied negative reinforcement in the post-train space, finding it surprisingly effective for reasoning tasks, but its behavior in the pre-train space is entirely uncharacterized.

5. There is no framework for combining pre-train and post-train RL. Even if pre-train space RL works, it is unlikely to be a complete replacement for standard RLVR — they target different aspects of model capability (foundational knowledge internalization vs. conditional trajectory refinement). Yet no existing work provides a principled strategy for sequencing or combining these two optimization phases.

How This Paper Positions Itself

The paper positions itself at the intersection of two research trajectories: RLVR for reasoning, and pre-train space optimization for foundational capability building. Its contribution is not to propose yet another RL algorithm or yet another pre-training dataset, but rather to open up the pre-train space as a legitimate domain for reinforcement learning and to characterize — theoretically, empirically, and through careful ablation — what happens when you do so.

The framing is explicitly incremental and analytical rather than claiming a wholesale paradigm shift. The paper does not argue that PreRL replaces standard RLVR. Instead, it investigates three concrete questions:

Can RL work in the pre-train space? This requires establishing gradient alignment between P(y) and P(y|x), validating that updating the marginal distribution doesn't harm the conditional policy.
If so, how should it be done? This involves decomposing PreRL into positive and negative sample reinforcement and characterizing their distinct dynamics — a decomposition that reveals NSR-PreRL as unexpectedly and disproportionately effective.
How should it be integrated with standard RL? This leads to the DSRL framework, which uses Policy Reincarnation (Agarwal et al., 2022; Liang et al., 2025) to treat NSR-PreRL as a warmup phase that establishes a stronger foundation before transitioning to standard GRPO.

The paper explicitly distinguishes PreRL from the emerging Reinforcement Learning Pre-Training (RLPT) paradigm (Section 5.2, Appendix A.2). RLPT methods (Hatamizadeh et al., 2025; Dong et al., 2025; Li et al., 2025b) also apply RL beyond post-training, but they (a) still optimize the conditional distribution P(y|x), (b) operate on pre-training corpora rather than reasoning tasks, and (c) use pseudo-rewards based on next-token prediction accuracy rather than verifiable task success. The paper argues that these proxy rewards are "simplistic and lacking in rigorous logical verification" (Appendix A.2), making RLPT more akin to active pre-training than to genuine reasoning-capability optimization.

In contrast, PreRL operates directly on reasoning tasks with truly verifiable rewards, but crucially strips away the input conditioning during gradient updates. This design choice is what makes PreRL novel: it bridges the gap between two previously disconnected paradigms — the foundational, knowledge-internalizing approach of pre-training (which operates on P(y) but passively) and the targeted, reward-driven approach of RLVR (which operates actively but only on P(y|x)). By combining the optimization target of the former with the learning mechanism of the latter, PreRL aims to achieve what neither can do alone: actively and adaptively encoding reasoning capability into model parameters in a way that preserves broad exploration capacity.

The Policy Reincarnation strategy (DSRL) then positions pre-train space optimization not as an alternative to standard RL but as a complementary phase — a warmup that "prunes universal incorrect patterns" (Section 4.3) so that subsequent post-train RL can focus on problem-specific refinement rather than struggling with fundamental error patterns. This sequencing reflects the paper's central insight that the benefits of pre-train space RL are most pronounced when it is used to establish a strong initialization, after which conditional optimization takes over for fine-grained improvement.

3. Technical Approach

3.1 Reader Orientation

This paper develops a training framework rather than a single model — a two-phase optimization procedure called Dual Space RL (DSRL) that first applies reinforcement learning to the model's unconditional output distribution (the "pre-train space") to prune incorrect reasoning and stimulate exploration, then transitions to standard conditional RL (the "post-train space") for fine-grained policy improvement. The core problem this framework solves is the exploration deficit in standard RLVR: when a base model has near-zero probability of generating correct solutions to hard problems, standard RL on P(y|x) cannot discover new reasoning strategies because it can only reinforce trajectories that already exist in the output distribution. The "shape" of the solution is a Policy Reincarnation strategy — a deliberately bounded warmup phase that modifies the foundational distribution P(y) to eliminate systematic error patterns and broaden exploration capacity, creating a better initialization from which standard RL can then operate more effectively.

3.2 Big-Picture Architecture (Diagram in Words)

The DSRL framework comprises five major components connected in a sequential pipeline:

Base Language Model (Qwen3-4B or Qwen3-8B) — the pretrained model that serves as the starting point for all optimization. It generates reasoning trajectories online (self-rollouts) and its parameters θ are updated throughout the entire process.
NSR-PreRL Warmup Phase — the first training stage where reinforcement learning is applied to the marginal distribution P(y). Crucially, only negative samples (incorrect reasoning trajectories) are used for updates during this phase; positive samples are ignored. The input question x is masked during gradient computation, so the model learns to suppress incorrect reasoning patterns in a question-agnostic way. This phase runs for a fixed number of steps S (optimally between 10 and 25).
Standard GRPO Phase — the second training stage where training switches to standard Group Relative Policy Optimization operating on the conditional distribution P(y|x). All samples (both positive and negative) are now used for updates, and the input condition x is included. This phase inherits the model checkpoint from the NSR-PreRL warmup as its initialization.
Verifiable Reward Function — a sparse binary reward signal: R(y) = 1 if the model's final answer matches the ground-truth, R(y) = 0 otherwise. All intermediate tokens receive reward 0. This is the only learning signal throughout both phases.
Policy Reincarnation Controller — the mechanism that governs the transition between the two phases. At step S, it switches the optimization target from P(y) to P(y|x), enables the use of positive samples, and begins standard GRPO training. This is not a learned controller; it is a fixed schedule determined by a hyperparameter sweep (Figure 7 in the paper).

The information flow is: a batch of training questions is sampled → the current model generates G = 8 rollouts per question → the verifiable reward function scores each rollout (0 or 1) → advantages are computed via group-based normalization → in the NSR-PreRL phase, only rollouts with negative advantage contribute to the gradient, and the input condition is stripped away → in the GRPO phase, all rollouts contribute with input conditioning → the model parameters are updated via clipped policy gradient → this repeats until the end of training.

3.3 Roadmap for the Deep Dive

First, the formal definition of the two optimization spaces (Post-train Space and Pre-train Space) and their mathematical objectives, because all subsequent analysis depends on understanding what exactly changes between standard RL and PreRL.
Second, the gradient alignment analysis that validates PreRL as a viable surrogate — this is the theoretical foundation that justifies the entire approach, and without it PreRL would be an unjustified gamble.
Third, the decomposition of PreRL into Positive Sample Reinforcement (PSR) and Negative Sample Reinforcement (NSR) and their distinct empirical dynamics — this is where the paper's key insight emerges (NSR works dramatically better than PSR in the pre-train space).
Fourth, the NSR-PreRL mechanism in detail: what it does to the model's reasoning behavior, why it stimulates exploration, and why it must be time-bounded — this sets up the motivation for DSRL.
Fifth, the DSRL framework itself — the Policy Reincarnation strategy, the transition mechanism, and the unified objective function — which brings the pre-train and post-train spaces together.
Sixth, the training configuration and implementation details — the specific hyperparameters, the GRPO algorithm used as the base optimizer, and the hardware/software setup.

3.4 Detailed, Sentence-Based Technical Breakdown

This is an empirical analysis paper with a theoretical motivation component whose core idea is that applying reinforcement learning directly to the marginal distribution P(y) — specifically, using only negative samples to prune incorrect reasoning patterns — creates a superior initialization for subsequent standard RL by eliminating systematic errors and stimulating endogenous exploration behaviors that standard RL cannot easily induce.

Formal Definition of Optimization Spaces

The paper begins by formally distinguishing two spaces in which language model policies can be optimized. A language model policy π_θ is equivalent to the output distribution P of the LLM parameterized by θ. The fundamental distinction between the spaces lies in what the update conditions on.

Definition 1 — Post-train Space Optimization: This is the standard paradigm for RLVR. The policy is optimized as π_θ(y|x), meaning every gradient update is strictly conditioned on a specific input query x drawn from the input distribution X. The learning objective computes gradients with respect to log π_θ(y|x), which captures how the model's probability of generating a particular reasoning trajectory y changes given a particular question x. This is what methods like GRPO, PPO, and RLOO all do.

Definition 2 — Pre-train Space Optimization: This is the alternative paradigm the paper introduces. The policy is optimized as π_θ(y), meaning the gradient update is not conditioned on the input query. The model optimizes the intrinsic, marginal distribution of reasoning trajectories directly. This is closer to what traditional pre-training does — encoding knowledge into parameters in a general, task-agnostic way — but PreRL does it using online reward signals rather than static next-token prediction targets.

The paper provides a detailed comparison with traditional pre-training and continual pre-training in Appendix A.1 (Table 4). All three paradigms optimize P(y), but they differ in training data (general web text vs. task-specific corpora vs. self-generated reasoning trajectories), data format (raw documents vs. QA pairs vs. response-only trajectories), learning signal (next-token prediction vs. next-token prediction vs. verifiable rewards), and learning paradigm (offline passive vs. offline passive vs. online active). PreRL is the only approach that combines the P(y) optimization target with active, reward-guided online learning.

The paper also explicitly distinguishes PreRL from the Reinforcement Learning Pre-Training (RLPT) paradigm in Appendix A.2 (Table 5). RLPT methods (Hatamizadeh et al., 2025; Dong et al., 2025; Li et al., 2025b) still optimize P(y|x), operate on pre-training corpora rather than reasoning tasks, and use pseudo-rewards based on next-token prediction accuracy. PreRL's distinguishing features are: it optimizes P(y) not P(y|x), it operates on genuine reasoning tasks with verifiable binary rewards, and it uses self-generated response-only trajectories in an online active learning loop.

Why these definitions matter: The distinction between P(y|x) and P(y) is not merely notational. When the model updates log π_θ(y|x), it learns "given this specific question, what reasoning patterns lead to success?" This creates question-specific associations that may not generalize. When the model updates log π_θ(y), it learns "what reasoning patterns are generally correct or incorrect?" The paper argues this question-agnostic learning should encode deeper, more transferable reasoning capabilities into the model's parameters.

The paper also provides a qualitative analysis of how P(y|x) and P(y) relate at the token level (Figures 10, 11, 12). In "aligned cases" (Figure 10), the token probability distributions with and without input conditioning show similar rankings — the tokens the model considers likely given the question are also tokens it considers likely in general. In "misaligned cases" (Figure 11), the top-ranked token under P(y|x) receives near-zero probability under P(y), indicating significant divergence. More systematically (Figure 12), log-probabilities align closely for high-probability, deterministic tokens (e.g., "the" following "So") but diverge notably for early-sequence or highly uncertain tokens where the input context dramatically reshapes the distribution. This analysis motivates the need for theoretical validation: if P(y) and P(y|x) diverge substantially in practice, can optimizing one reliably improve the other?

The Standard RL Objective in the Post-train Space

Before introducing PreRL, the paper establishes the standard RLVR formulation that serves as both a baseline and the second phase of DSRL. Language model generation is formulated as a token-level Markov Decision Process (MDP). At each step t, the state s_t = [x; y_{<t}] consists of the input question and all tokens generated so far. The policy π_θ(·|s_t) samples the next token y_t from the vocabulary, transitioning to s_{t+1} = [s_t; y_t]. Generation terminates upon producing an end-of-sequence token or reaching the maximum budget (8192 tokens).

The standard RL objective maximizes expected return conditioned on inputs:

$J_{RL}(\pi_\theta) = \mathbb{E}_{x \sim X}\left[\mathbb{E}_{y \sim \pi_\theta(\cdot|x)}[R(y)] - \beta D_{KL}[\pi_\theta(\cdot|x) || \pi_{ref}(\cdot|x)]\right]$

where R(y) = Σ_{t=1}^{|y|} r(y_t) is the cumulative return summed over all tokens in the trajectory, |y| is the total length of the generated response, π_ref is the reference policy (the frozen initial model), and β controls the strength of the KL penalty that prevents the policy from drifting too far from the reference.

What it computes: The objective has two terms. The first term E[R(y)] is the expected total reward — it encourages the policy to generate trajectories that achieve high rewards. The second term β·D_KL[π_θ||π_ref] penalizes divergence from the reference policy — it prevents the model from collapsing to a narrow set of high-reward trajectories and preserves generalization. The expectation over x ∼ X means this is averaged over the training distribution of questions.

Why this form: The KL penalty is standard in RLHF and RLVR to prevent reward hacking and maintain output diversity. However, the paper notes that it follows Hu et al. (2025a) in setting β = 0 for simplicity — removing the KL penalty entirely. This is a significant design choice because it means the policy is free to diverge arbitrarily from the reference model, placing full weight on reward maximization. The paper justifies this by noting that the verifiable reward signal (binary correctness) is sufficiently constrained that reward hacking is less of a concern than in RLHF with learned reward models, and that prior work has shown KL penalties can suppress exploration.

The policy gradient theorem gives the gradient of this objective:

$\nabla_\theta J_{RL}(\pi_\theta) = \mathbb{E}_{x \sim X}\left[\mathbb{E}_{y \sim \pi_\theta(\cdot|x)}\left[\sum_{t=1}^{|y|} \nabla_\theta \log \pi_\theta(y_t|x, y_{<t}) \cdot R(y)\right]\right]$

where ∇θ log π_θ(y_t|x, y{<t}) is the gradient of the log-probability of token y_t given the input and previous tokens, and R(y) is the scalar return for the complete trajectory.

What it computes: For each trajectory y, this computes the sum over all tokens of (gradient of token log-probability) × (trajectory return). Intuitively, it increases the probability of each token in high-reward trajectories and decreases the probability of each token in low-reward trajectories, with the magnitude of the update proportional to the reward. The expectation over x ∼ X and y ∼ π_θ averages this over the training distribution.

Why this form: This is the standard REINFORCE estimator adapted to the token level. The key property is that the same scalar reward R(y) is used for every token in the trajectory — this is credit assignment by temporal uniformity: all tokens share equally in the success or failure of the complete trajectory. This is appropriate for sparse-reward settings (only the final answer matters) where there is no per-step reward signal to distinguish good intermediate steps from bad ones.

The paper adopts sparse binary rewards: r_t = 0 for all t < |y| (intermediate tokens) and r_{|y|} ∈ {0, 1} (final answer token only, 1 if correct and 0 otherwise). This means R(y) = 1 for correct trajectories and R(y) = 0 for incorrect ones — it is purely a correctness signal with no partial credit.

GRPO: The Base RL Algorithm

The paper employs Group Relative Policy Optimization (GRPO) as the optimization algorithm in both the PreRL and standard RL phases. GRPO addresses a key practical challenge: standard policy gradient methods require a value network (critic) to estimate advantages, which doubles the model size and introduces additional training instability. GRPO eliminates the critic by using group-based relative advantage estimation.

For each input query x, GRPO samples a group of G responses {y_1, ..., y_G} from the old policy π_{θ_old} and computes their corresponding returns R = {R_1, ..., R_G}. The advantage for each token in response y_i is then estimated by normalizing the returns within the group:

$\hat{A}_{i,t} = \frac{R_i - \text{mean}(R)}{\text{std}(R)}$

where mean(R) and std(R) are the empirical mean and standard deviation of returns across the G responses in the group.

What it computes: For each group of G responses to the same question, this computes a normalized advantage for each response. Responses with above-average reward get positive advantages; responses with below-average reward get negative advantages. The normalization by standard deviation ensures that the magnitude of advantages is scale-invariant — it doesn't matter whether rewards range from 0 to 1 or 0 to 100; the relative ordering within the group determines the advantage.

Why this form: This group-relative normalization provides several benefits. It automatically adapts to question difficulty: on easy questions where most responses are correct, the mean is high and only exceptionally good responses get positive advantages; on hard questions where most responses are incorrect, the mean is low and even mediocre responses might get positive advantages. It eliminates the need for a separately trained value function, halving the memory and compute requirements. And it reduces variance compared to using raw returns, because it subtracts the group mean (which acts as a baseline).

The GRPO optimization objective is:

$J_{GRPO}(\pi_\theta) = \mathbb{E}_{x \sim X, \{y_i\}_{i=1}^G \sim \pi_{\theta_{old}}}\left[\frac{1}{\sum_{i=1}^G |y_i|} \sum_{i=1}^G \sum_{t=1}^{|y_i|} \min\left(\rho_{i,t} \hat{A}_{i,t}, \text{clip}(\rho_{i,t}, 1 - \epsilon, 1 + \epsilon) \hat{A}_{i,t}\right)\right]$

where ρ_{i,t} = π_θ(y_{i,t}|x, y_{i,<t}) / π_{θ_old}(y_{i,t}|x, y_{i,<t}) is the importance sampling ratio (probability ratio between the current and old policies), and ε is the clipping parameter that constrains policy updates.

What it computes: This is a PPO-style clipped objective adapted for group-based advantages. The outer expectation averages over questions x and groups of G responses sampled from the old policy. Inside, the sum runs over all responses and all tokens, computing a per-token update. The min() operation implements clipping: it takes the minimum of the unclipped update (ρ_{i,t} · Â_{i,t}) and the clipped update (clip(ρ_{i,t}, 1-ε, 1+ε) · Â_{i,t}). This prevents any single token from receiving an excessively large update by capping the importance sampling ratio.

Why this form: The clipped objective is the key innovation from PPO that stabilizes on-policy RL. Without clipping, a single large policy ratio could cause a catastrophically large update that destroys previously learned behavior. The clipping (typically ε = 0.2) ensures that the effective update magnitude is bounded even if the probability ratio becomes extreme. The min() between clipped and unclipped means the objective is a pessimistic bound — it only takes the larger update when it would actually hurt the objective (when the advantage and the ratio have opposite signs). The paper uses the optimized token-level policy gradient loss from Yu et al. (2025), which applies this clipping at the token level rather than the response level, giving finer-grained control.

The training hyperparameters are specified in Appendix B.1: optimizer is AdamW, learning rate 1e-6, prompt batch size 128, G = 8 rollouts per prompt, mini-batch size 128, training epochs 6 (on-policy iterations), sampling temperature 1.0, maximum response length 8192 tokens, and maximum prompt length 1024 tokens. Training uses the verl framework (Sheng et al., 2025).

PreRL: Applying RL to the Marginal Distribution P(y)

The key innovation of PreRL is that it modifies the gradient computation to remove the input condition x during the policy update. The policy gradient for PreRL is:

$\nabla_\theta J_{PreRL}(\pi_\theta) = \mathbb{E}_{x \sim X}\left[\mathbb{E}_{y \sim \pi_\theta(\cdot|x)}\left[\sum_{t=1}^{|y|} \nabla_\theta \log \pi_\theta(y_t|\cancel{x}, y_{<t}) \cdot R(y)\right]\right]$

where the notation \cancel{x} signifies that PreRL removes the input dependency during the gradient update. The trajectories y are still generated conditioned on x (the sampling process is unchanged — questions are still provided as input), but the gradient computation treats them as if they were generated unconditionally.

What it computes: This is structurally identical to the standard policy gradient, with one critical difference: the gradient is taken with respect to log π_θ(y_t|y_{<t}) rather than log π_θ(y_t|x, y_{<t}). For each token y_t, the gradient asks: "how should we change the model parameters to increase the unconditional probability of this token given the preceding tokens?" The reward R(y) still weights the update, so tokens in correct trajectories get their unconditional probability increased and tokens in incorrect trajectories get it decreased.

Why this form: The insight is that π_θ(y_t|y_{<t}) represents the model's intrinsic knowledge about how reasoning should proceed, independent of any particular question. By optimizing this marginal distribution, PreRL aims to encode general reasoning patterns into the model's parameters rather than question-specific associations. The question x is still used during sampling (to generate trajectories that are relevant to the task distribution), but the update itself is question-agnostic. This means the model learns that certain reasoning patterns — regardless of what question triggered them — are generally correct or generally incorrect.

The paper notes that a critical question must be answered for PreRL to be valid: does optimizing the marginal log-likelihood log π_θ(y_t|y_{<t}) actually improve the conditional log-likelihood log π_θ(y_t|x, y_{<t}) needed for task performance? There is no mathematical guarantee of this — the two objectives could, in principle, conflict.

Theoretical Justification: Gradient Alignment Between P(y) and P(y|x)

The paper provides a theoretical argument grounded in a fundamental premise: Shared Parameter Influence. The model parameters θ simultaneously govern both the marginal distribution π_θ(y) and the conditional distribution π_θ(y|x) — they are computed by the same neural network with the same weights; the only difference is whether the input x is prepended to the context. This shared parameterization creates a structural coupling between the two distributions.

Let θ' = θ + η·∇_θ log π_θ(y)·R(y) denote the parameters after one PreRL step that updates the marginal log-probability in the direction of the reward-weighted gradient (η is the learning rate). Using a first-order Taylor expansion, the effect of this update on the conditional log-probability is:

$\log \pi_{\theta'}(y|x) \approx \log \pi_\theta(y|x) + \eta \cdot R(y) \cdot \underbrace{[\nabla_\theta \log \pi_\theta(y)]^\top \nabla_\theta \log \pi_\theta(y|x)}_{\text{cross-gradient term}} + O(\eta^2)$

where log π_θ(y|x) is the conditional log-probability under the old parameters, η is the learning rate, R(y) is the scalar reward, and the cross-gradient term is the inner (dot) product between the gradient of the marginal log-probability and the gradient of the conditional log-probability.

What it computes: This expansion shows how a single PreRL update (which directly modifies only the marginal log-probability) indirectly affects the conditional log-probability through the shared parameters. The change in conditional log-probability decomposes into: the original value + η·R(y)·(gradient inner product) + higher-order terms. If the gradient inner product is positive, then the conditional log-probability moves in the same direction as the marginal update (scaled by the reward). If it is negative, the conditional and marginal objectives conflict.

Why this form: The Taylor expansion is the standard tool for understanding how local parameter updates affect related quantities. The key insight is that the cross-gradient term captures the structural coupling between the two distributions. This coupling is not something the paper designs or enforces — it emerges naturally from the shared parameterization. The paper's theoretical contribution is to identify this term as the critical quantity determining whether PreRL can work and to empirically verify that it is consistently positive.

The paper states the core premise explicitly: "for reasoning trajectories y which semantically aligned with x, the gradients of the marginal and conditional objectives exhibit a non-negative inner product: ⟨∇_θ log π_θ(y), ∇_θ log π_θ(y|x)⟩ ≥ 0." This ensures the cross-gradient term is non-negative, meaning that updating the marginal log π_θ(y) concurrently influences the conditional log π_θ(y|x) in the same direction.

Empirical validation of gradient alignment (Figure 2): The paper conducts a detailed empirical analysis on Qwen3-4B using 400 rollouts from the AMC23 dataset. Three complementary measurements confirm the alignment:

Observation 1 — Gradient dot product is consistently non-negative (Figure 2a). The inner product ⟨∇_θ log π_θ(y), ∇_θ log π_θ(y|x)⟩ is computed for each of the 400 rollouts. The distribution is entirely non-negative: maximum 46.18, minimum 0.94, mean 9.23. 100% of samples satisfy the non-negativity condition. This is strong evidence that, in practice, the marginal and conditional gradients point in aligned directions — optimizing one almost never conflicts with the other.

Observation 2 — Gradient cosine similarity is strongly positive (Figure 2b). While the dot product can be large simply because the gradient magnitudes are large, cosine similarity normalizes by magnitude to measure pure directional alignment. The cosine similarity distribution has maximum 0.71, minimum 0.06, and mean 0.44. These are moderately strong positive correlations — the gradients are not identical (cosine similarity of 1.0) but they consistently point in similar directions. The minimum of 0.06 (near-orthogonal) is notable, but even this is non-negative — no sample shows opposing gradient directions.

Observation 3 — The log-probability distributions are closely aligned (Figure 2c). The per-token difference |log P(y|x) - log P(y)| is small for most tokens: maximum 0.60, minimum 0.026, mean 0.16. This means that for a typical token, the conditional and marginal log-probabilities differ by only about 0.16 nats — the two distributions are quantitatively similar. The paper interprets this as validation that "log P(y) serves as a faithful surrogate for log P(y|x)."

Decomposing PreRL: Positive vs. Negative Sample Reinforcement

To understand PreRL's dynamics, the paper decomposes the optimization into two mechanisms based on the sign of the advantage:

Positive Sample Reinforcement (PSR): Samples with Â_i > 0 (above-group-average reward) act as positive signals. The gradient increases their probability: it moves the policy toward generating more trajectories like these.

Negative Sample Reinforcement (NSR): Samples with Â_i < 0 (below-group-average reward) act as negative signals. The gradient decreases their probability: it moves the policy away from generating trajectories like these.

This decomposition follows Zhu et al. (2025), who studied the surprising effectiveness of negative reinforcement in the post-train space. The paper extends this analysis to the pre-train space, where the dynamics are significantly different.

The paper's training dynamics analysis (Figure 3a) reveals a dramatic and asymmetric pattern:

PSR-PreRL suffers from on-policy learning collapse. When PreRL is applied using only positive samples (PSR-PreRL), the model's performance initially increases — it successfully raises the conditional probability π_θ(y|x), corroborating the gradient alignment argument. However, it then degrades substantially. The paper explains this failure: PSR-PreRL "fails to effectively learn from self-generated on-policy trajectories, ultimately degrading performance." The mechanism appears to be that maximizing π_θ(y) for self-generated correct trajectories causes the model to accumulate probability mass on its own outputs, potentially amplifying subtle errors or biases that the model cannot distinguish from genuine improvements.

The paper contrasts this with QFFT (Liu et al., 2025a), which "successfully optimizes the same objective max_θ π_θ(y) using out-of-distribution long-CoT trajectories from a teacher model." The key difference is that QFFT uses high-quality, externally-generated expert demonstrations rather than the model's own self-generated trajectories. This suggests a fundamental limitation: maximizing P(y) in the pre-train space strictly requires high-quality, out-of-distribution data, and self-generated on-policy trajectories do not meet this bar. The model cannot bootstrap itself upward using only positive reinforcement in the pre-train space — it needs external guidance.

NSR-PreRL demonstrates surprising effectiveness. When PreRL is applied using only negative samples (NSR-PreRL), the behavior is qualitatively different and far more successful. The training curve in Figure 3a shows that NSR-PreRL successfully decreases π_θ(y|x) (as expected, since it is suppressing incorrect trajectories) while simultaneously improving task performance. The paper reports that NSR-PreRL achieves 86% accuracy with roughly 3× fewer training steps than standard RL (approximately 20 steps vs. 60+ steps for GRPO to reach comparable performance).

The paper identifies two mechanisms driving NSR-PreRL's effectiveness:

Mechanism 1 — Pruning incorrect reasoning paths in the pre-train space. By suppressing incorrect trajectories in the marginal distribution P(y), NSR-PreRL removes probability mass from systematic error patterns. Because the gradient alignment is positive, this suppression in the pre-train space also reduces the probability of these incorrect patterns in the conditional distribution P(y|x). The effect is a "cleaning up" of the model's output distribution — it becomes less likely to generate common error patterns, which means subsequent standard RL has fewer mistakes to correct.

Mechanism 2 — Stimulating endogenous reasoning capabilities. NSR-PreRL "strongly elicits reasoning capabilities, significantly increasing response length." The paper quantifies this using the taxonomy from Chen et al. (2025a), which categorizes reasoning steps into transition thoughts (switching between approaches), reflection thoughts (checking and verifying work), and execution thoughts (direct computation). As shown in Figure 3b, after just 20 NSR-PreRL steps, the model generates 14.89× more transition thoughts and 6.54× more reflection thoughts compared to the vanilla model (zero RL steps). This substantially outperforms GRPO at 25 steps, which shows much more modest increases in these reasoning patterns.

The mechanism behind this stimulation is not fully explained but is characterized as the model "activating internal knowledge for deeper reasoning." The paper suggests that by suppressing incorrect trajectories, NSR-PreRL forces the model to explore alternative reasoning paths — since the "easy but wrong" paths are being pruned, the model must generate more elaborate, structured reasoning to achieve success. This is a form of implicit curriculum learning: by removing the model's ability to succeed via shallow or error-prone reasoning, NSR-PreRL incentivizes the emergence of deeper reasoning behaviors that were already latent in the pretrained parameters but not previously expressed.

The double-edged sword of NSR-PreRL. The paper is careful to note that the exploratory stimulation from NSR-PreRL is both its greatest strength and its fundamental limitation. On one hand, it "yields better performance and preserves exploration ability by redistributing probability mass away from incorrect trajectories in the pre-train space, effectively pruning wrong reasoning paths" (Section 2.3). On the other hand, it "eventually leads to excessively long outputs that hinder continuous training" (Section 2.3), citing Chen et al. (2024) and Tan et al. (2025a) on the overthinking problem where models generate ever-longer chains of reasoning without improving accuracy.

This tension — that NSR-PreRL stimulates beneficial exploration but eventually degenerates into unbounded output length — is what motivates the Policy Reincarnation strategy. The insight is that NSR-PreRL is most valuable as a bounded warmup phase rather than a sustained training paradigm. The key design choice is determining the optimal number of NSR-PreRL steps S before transitioning to standard RL.

The Dual Space RL (DSRL) Framework: Policy Reincarnation

DSRL combines NSR-PreRL with standard GRPO through a Policy Reincarnation strategy (Agarwal et al., 2022; Liang et al., 2025; Tan et al., 2025b). Policy Reincarnation is a meta-strategy where training is divided into phases, and the model checkpoint from one phase serves as the initialization for the next, potentially with different objectives, algorithms, or hyperparameters. The paper adapts this to the pre-train/post-train space transition.

The unified DSRL objective is:

$\nabla_\theta J_{DSRL}(\pi_\theta) = \mathbb{E}_{x \sim X, y \sim \pi_\theta(\cdot|x)}\left[\sum_{t=1}^{|y|} \nabla_\theta \log \pi_\theta(y_t|x \cdot \mathbb{I}[s > S], y_{<t}) \cdot R(y) \cdot \mathbb{I}[s > S \lor R(y) < 0]\right]$

where s denotes the current training step, S is the transition threshold (the number of NSR-PreRL warmup steps), and x·𝕀[s>S] controls whether the input condition is included.

What it computes: This objective has two indicator functions that encode the transition logic:

The term x·𝕀[s>S] controls input conditioning. When s ≤ S (PreRL phase), the indicator is 0, so the input condition is removed — the gradient is taken with respect to log π_θ(y_t|y_{<t}). When s > S (standard RL phase), the indicator is 1, so the input condition is included — the gradient is taken with respect to log π_θ(y_t|x, y_{<t}).

The term 𝕀[s > S ∨ R(y) < 0] controls which samples contribute to the update. When s ≤ S, the indicator is 1 only when R(y) < 0 — meaning only negative samples (incorrect trajectories) are used for updates. When s > S, the indicator is always 1 — meaning all samples (both positive and negative) are used.

Why this form: The objective encodes the entire DSRL training schedule in a single mathematical expression. The key design choices are:

Phase 1 (s ≤ S): NSR-PreRL — only negative samples, no input conditioning. This phase prunes incorrect reasoning patterns in the pre-train space while stimulating exploration. The exclusion of positive samples is crucial because PSR-PreRL causes collapse (as shown in Section 2.3). The exclusion of input conditioning ensures the updates are question-agnostic, encoding general reasoning improvements.
Phase 2 (s > S): Standard GRPO — all samples, with input conditioning. This phase inherits the NSR-PreRL-enhanced model and applies standard RLVR on P(y|x) to fine-tune the conditional policy. The full GRPO objective (including clipping and group-based advantages) is applied.

The transition between phases is a discrete switch at step S. The paper explores the optimal value of S through an ablation study (Figure 7), finding an inverted-U relationship: 10–25 steps is optimal. Fewer steps provide "insufficient stimulation" — the model doesn't benefit enough from pre-train space pruning. More than 25 steps causes "over-exploratory generation that hinders subsequent fine-grained optimization" — the excessive length and exploratory behavior from extended NSR-PreRL makes it harder for standard RL to converge.

In practice, the paper implements GRPO as the base algorithm for both phases, with the modifications encoded by the indicator functions. The training hyperparameters (learning rate 1e-6, batch size 128, 8 rollouts per prompt, etc.) remain constant across the transition.

Training Setup and Hyperparameters

The paper uses Qwen3-4B and Qwen3-8B (Yang et al., 2025) as base models. These are chosen because they "exhibit strong capability in advanced reasoning" and use non-thinking mode (no explicit chain-of-thought prompting by default), following Ma et al. (2025). The non-thinking mode is important because it means any reasoning behaviors that emerge (reflections, transitions, etc.) are genuinely elicited by the training process rather than being prompted by the system template.

Training data is the MATH dataset (Lewkowycz et al., 2022) containing 7,500 problems. The system prompt template (Appendix B.2, Figure 8) tells the model: "Please reason step by step, and put your final answer within \boxed{}." This explicit request for step-by-step reasoning combined with the non-thinking default mode creates a controlled setting where the model is asked to reason but isn't forced into a particular reasoning structure a priori.

Key training hyperparameters (Appendix B.1, Table 6):

Optimizer: AdamW
Policy learning rate: 1e-6
Training batch size (prompts per step): 128
Samples per prompt (G): 8
Mini-batch size: 128
Training epochs (on-policy iterations): 6
Max prompt length: 1024 tokens
Max response length: 8192 tokens
Rollout temperature: 1.0

The sampling temperature of 1.0 means the model samples from its full unmodified distribution during rollout generation — there is no temperature scaling to encourage either exploitation (low temperature) or exploration (high temperature). This is the "natural" sampling temperature for on-policy RL, where diversity of rollouts is important for estimating advantages.

The paper uses the verl framework (Sheng et al., 2025) for training and vLLM (Kwon et al., 2023) for evaluation, with evaluation temperature 1.0 and top-p 1.0.

NSR-PreRL Warmup vs. NSR-RL Warmup: Ablation Justifying the Pre-train Space

A critical ablation (Section 4.4, Table 3) validates that the pre-train space — not just negative sample reinforcement — is necessary for DSRL's gains. The paper compares three conditions:

GRPO (baseline): Standard RL only, no warmup — achieves average score 55.79.
NSR-RL Warmup + GRPO: 20 steps of negative-sample-only RL in the post-train space (with input conditioning), then standard GRPO — achieves 54.38, actually worse than the baseline.
NSR-PreRL Warmup + GRPO (DSRL): 20 steps of negative-sample-only RL in the pre-train space (without input conditioning), then standard GRPO — achieves 57.54, better than the baseline.

The finding is striking: NSR in the post-train space is actively harmful compared to standard GRPO, while NSR in the pre-train space is beneficial. By step 20, NSR-PreRL outperforms NSR-RL by an average of 6.6 points across AMC23, AIME24, and AIME25 (Figure 13). The paper does not provide a detailed mechanistic explanation for this asymmetry, but the implication is clear: the benefit of negative reinforcement comes specifically from suppressing incorrect patterns in the question-agnostic marginal distribution P(y), not from suppressing them in the question-conditional P(y|x). When negative reinforcement is applied in the post-train space, the model likely learns to avoid specific incorrect patterns for specific questions, which may not generalize and may even interfere with the model's ability to explore alternative approaches. When applied in the pre-train space, the suppression is of general reasoning patterns that are incorrect, leading to transferable improvements.

How the Ablations Validate Design Choices

The paper's experimental design includes several key ablations that validate specific architectural decisions:

Warmup step count (Figure 7): The inverted-U relationship validates that NSR-PreRL must be bounded. The optimal range (10–25 steps) balances sufficient pre-train space pruning against the risk of over-exploration. The paper uses a sweep across values from 5 to 30 steps, with GRPO's baseline score shown as a reference line (55.79). The peak at 10–25 steps (57.54) demonstrates that the Policy Reincarnation transition point is a real hyperparameter that must be tuned — it is not the case that "more PreRL is better."

Pre-train space vs. post-train space warmup (Table 3): This is the critical ablation distinguishing the paper's contribution. NSR applied in the post-train space is worse than no warmup at all. This validates that the pre-train space optimization target P(y) — not just the negative-reinforcement mechanism — is the essential ingredient.

PSR vs. NSR decomposition (Figure 3a): The full PreRL curve (which includes both PSR and NSR) initially matches standard RL before collapsing. The PSR-only curve degrades, while the NSR-only curve succeeds. This decomposition validates that positive reinforcement in the pre-train space is harmful and that NSR alone is the effective mechanism. This finding directly motivates the indicator function 𝕀[s > S ∨ R(y) < 0] in the DSRL objective, which excludes positive samples during the warmup phase.

Gradient alignment measurements (Figure 2): The consistently non-negative gradient dot products validate the theoretical premise that optimizing P(y) doesn't conflict with optimizing P(y|x). Without this validation, the entire PreRL approach would rest on an unverified assumption. The empirical confirmation is what transforms PreRL from a speculative idea into a justified method.

4. Key Insights and Innovations

Innovation 1: The Pre-train Space as a Legitimate Domain for Reinforcement Learning

The paper's most fundamental conceptual move is carving the optimization landscape of language models into two distinct spaces — the Post-train Space (where updates condition on input x, optimizing P(y|x)) and the Pre-train Space (where updates strip away x, optimizing P(y)) — and demonstrating that the latter is not merely a theoretical curiosity but a viable and complementary domain for RL. This is a reframing, not an incremental algorithmic advance.

Prior to this work, the dominant assumption in RLVR was that the input condition x is an essential component of the optimization — the model must learn "given this specific question, what is the right reasoning trajectory?" The idea that you could remove the question from the gradient update and still improve task performance would have seemed paradoxical. How could learning question-agnostic reasoning patterns help solve specific problems? The field's entire RL infrastructure — GRPO, PPO, RLOO, and their variants — operated exclusively on P(y|x) with no consideration of whether the marginal distribution could serve as a surrogate optimization target.

The paper's reframing is grounded in a diagnostic move: rather than assuming that P(y) and P(y|x) are related, the paper tests whether their gradients align empirically. The gradient alignment analysis in Figure 2 is not an incidental measurement — it is the condition of possibility for the entire framework. By showing that ⟨∇_θ log π_θ(y), ∇_θ log π_θ(y|x)⟩ is consistently non-negative (minimum 0.94, mean 9.23 across 400 AMC23 rollouts), the paper establishes that updating the marginal distribution is not a gamble — it is a structurally sound surrogate. The cosine similarity distribution (mean 0.44) further reveals that the gradients are not identical (which would make PreRL redundant with standard RL) but are sufficiently aligned to guarantee that improvements in P(y) transfer to P(y|x).

This distinguishes PreRL from the superficially related Reinforcement Learning Pre-Training (RLPT) paradigm (Hatamizadeh et al., 2025; Dong et al., 2025). RLPT methods apply RL during pre-training but still optimize P(y|x) using pseudo-rewards from next-token prediction. They never question whether the conditional distribution is the right target. PreRL's innovation is recognizing that the optimization target itself — conditional vs. marginal — is a design dimension separate from the learning algorithm, and that switching to P(y) opens up qualitatively different learning dynamics that standard RL cannot access.

The significance of this reframing extends beyond any single method. It implies that future work on RL for reasoning should consider two degrees of freedom — where the optimization happens (pre-train vs. post-train space) and how it happens (which algorithm, which reward structure) — rather than treating "RL" as a monolithic post-training step. The paper's own finding that NSR behaves oppositely in the two spaces (harmful in the post-train space, beneficial in the pre-train space — Table 3, where NSR-RL warmup scores 54.38 vs. GRPO's 55.79 while NSR-PreRL warmup scores 57.54) validates that space selection is not cosmetic but mechanistically consequential.

This is a fundamental conceptual contribution rather than an incremental method improvement. The paper does not claim to have solved pre-train space RL — it acknowledges PSR-PreRL's collapse and NSR-PreRL's eventual over-exploration — but it establishes the space as legitimate terrain for investigation, which is a more durable contribution than any specific algorithm.

Innovation 2: Identifying Negative Sample Reinforcement as the Dominant and Asymmetric Driver of Pre-train Space Learning

The paper's second major insight is exposing a fundamental asymmetry between positive and negative reinforcement in the pre-train space — PSR-PreRL causes on-policy learning collapse while NSR-PreRL is surprisingly effective — and demonstrating that this asymmetry is the mechanism that makes PreRL valuable. This is not merely an observation that "negative samples matter" (which Zhu et al., 2025 already showed in the post-train space); it is the discovery that the pre-train space inverts the relative importance of positive and negative reinforcement compared to standard RL.

In standard post-train space RL, both positive and negative samples contribute to policy improvement. Positive samples reinforce correct behavior; negative samples suppress incorrect behavior. The decomposition into PSR and NSR is symmetric — both are necessary components of a working system, and neither alone would be expected to suffice. Zhu et al. (2025) showed that negative reinforcement is "surprisingly effective" in the post-train space, but they did not claim that it works without positive reinforcement, and their analysis remained within the conditional distribution framework.

What the paper reveals in Figure 3a is that in the pre-train space, this symmetry breaks down entirely. PSR-PreRL — maximizing π_θ(y) for self-generated correct trajectories — fails catastrophically. The paper's diagnosis is precise: "maximizing P(y) in the pre-train space strictly requires high-quality, out-of-distribution expert demonstrations," citing the contrast with QFFT (Liu et al., 2025a) which succeeds on the same objective using teacher-generated trajectories. This is a negative result with diagnostic value: it reveals that the model cannot bootstrap itself upward through self-generated positive examples in the pre-train space because the probability mass it accumulates on its own outputs amplifies subtle errors and distributional biases that the binary reward signal cannot distinguish.

NSR-PreRL, in contrast, works precisely because it doesn't require the model to know what is correct — it only needs to identify what is incorrect. The reward signal R(y) = 0 is unambiguous for wrong answers, while R(y) = 1 for correct answers doesn't distinguish between genuinely good reasoning and lucky guesses that happen to produce the right final answer. In the pre-train space, where the optimization is question-agnostic, this asymmetry is magnified: suppressing a generally-incorrect reasoning pattern (e.g., a common algebraic mistake) transfers across questions, while reinforcing a specific correct trajectory may not generalize because the same reasoning steps could be wrong for a different problem.

The paper quantifies this asymmetry through two mechanisms (Figure 3b): NSR-PreRL increases transition thoughts by 14.89× and reflection thoughts by 6.54× compared to the vanilla model after just 20 steps, substantially outperforming GRPO at 25 steps. The interpretation — that pruning incorrect paths forces the model to explore more sophisticated reasoning strategies — is a novel causal claim about how negative reinforcement shapes exploration. Standard RL theory treats exploration as a function of stochasticity in the sampling process (temperature, entropy bonuses); this paper shows that selective suppression creates exploration pressure by removing the "easy wrong answers" that the model would otherwise default to.

This is a fundamental discovery about learning dynamics rather than an algorithmic contribution. The paper does not invent NSR — it inherits the decomposition from Zhu et al. (2025) — but it discovers that the pre-train space makes NSR sufficient (no PSR needed) and dramatically more powerful (eliciting endogenous reasoning behaviors that standard RL struggles to induce). This finding implies that future work on pre-train space RL should focus on improving what gets suppressed and how long suppression continues, rather than trying to incorporate positive reinforcement — a complete inversion of the standard RL design philosophy.

Innovation 3: Policy Reincarnation as a Phase-Transition Strategy Between Optimization Spaces

The paper's third innovation is elevating Policy Reincarnation from a training efficiency trick to a principled phase-transition strategy that bridges two incommensurable optimization regimes. Prior work on Policy Reincarnation (Agarwal et al., 2022; Liang et al., 2025) used it primarily to transfer knowledge between different RL algorithms or hyperparameter settings within the same optimization space — e.g., switching from one policy gradient variant to another once learning plateaus. DSRL repurposes this mechanism to transition between qualitatively different optimization targets (P(y) to P(y|x)), making the reincarnation not a convenience but a necessity: the two phases optimize fundamentally different objectives and cannot be meaningfully combined into a single loss function.

The paper's key insight is that NSR-PreRL and standard GRPO are not just different algorithms — they serve different functions in the capability-building pipeline. NSR-PreRL establishes the foundation by eliminating systematic error patterns in the question-agnostic marginal distribution; standard GRPO builds the superstructure by fine-tuning question-conditional behavior. You cannot do both simultaneously because they optimize different quantities (log P(y) vs. log P(y|x)) and have different sample requirements (only negative vs. all samples). The Policy Reincarnation strategy acknowledges this incommensurability and formalizes it as a sequential schedule.

The unified DSRL objective (Equation 6) encodes this phase transition through indicator functions that gate both the optimization target and the sample selection. This is a conceptual synthesis — the paper shows that the pre-train and post-train spaces are not alternatives to choose between but complementary phases to sequence. The evidence for why this synthesis is necessary rather than merely convenient comes from the warmup step ablation (Figure 7), which reveals an inverted-U relationship between NSR-PreRL duration and final performance. Too few steps (under 10) provide "insufficient stimulation" — the foundation isn't adequately prepared. Too many steps (over 25) cause "over-exploratory generation that hinders subsequent fine-grained optimization" — the foundation becomes so different from what standard RL expects that the transition fails. The optimal window (10–25 steps) is where the pre-train space has been sufficiently pruned but not so transformed that post-train RL cannot effectively build upon it.

This finding is significant because it characterizes the tradeoff between exploration and exploitability across optimization spaces. Extended NSR-PreRL generates richer reasoning behaviors (more transitions, more reflections, longer outputs), but these very behaviors make the policy harder for standard GRPO to optimize — the exploration that helps in the pre-train space becomes a liability in the post-train space. This is a more nuanced relationship than the standard exploration-exploitation tradeoff within a single space; it is a cross-space compatibility constraint that has no analog in prior RL-for-reasoning work.

The paper also provides diagnostic evidence that the phase transition genuinely improves the foundation rather than just adding complementary capabilities. In Figure 6, DSRL shows a sharp surge in "Fully Solved" questions (where all rollouts are correct) and a corresponding decline in "Fully Unsolved" questions (all rollouts incorrect) specifically during the NSR-PreRL phase, before standard RL begins. This means NSR-PreRL is systematically eliminating failure modes — the model transitions from "sometimes getting it right by chance" to "reliably getting it right" on a subset of problems. Standard RL then extends this reliability to additional problems. This two-stage dynamic — NSR-PreRL makes the model consistent, standard RL makes it capable on harder problems — is a novel decomposition of how reasoning capability is acquired.

This is an architectural contribution to training methodology. The paper doesn't claim to have discovered Policy Reincarnation (it cites Agarwal et al., 2022 explicitly), but it shows that reincarnation between optimization spaces — not just between algorithms — is a powerful design pattern for reasoning, and it provides the first characterization of how to schedule the transition.

Innovation 4: Pre-train Space Optimization as a Mechanism for Generalization Beyond the Training Distribution

The paper's fourth insight is that optimizing P(y) rather than P(y|x) produces superior out-of-distribution generalization, and that this generalization arises from the question-agnostic nature of the updates rather than from any explicit transfer-learning objective. This is not merely a performance claim (DSRL beats GRPO on OOD benchmarks) but a diagnostic finding about why pre-train space optimization generalizes: by suppressing incorrect reasoning patterns without conditioning on specific questions, NSR-PreRL eliminates errors that are universal across problem types, not just errors that are common in the training distribution.

The evidence for this claim comes from Table 2, where DSRL's gains over GRPO on out-of-distribution benchmarks are substantially larger proportionally than the in-distribution gains from Table 1. On Qwen3-4B, DSRL improves over GRPO on in-distribution benchmarks by roughly 1–3 points on average (57.54 vs. 55.79 across the six math benchmarks), but on OOD benchmarks the gains are more dramatic: +3.79 on GPQA-Diamond (graduate-level science QA), +5.37 on MMLU-Pro (broad knowledge assessment), and +2.04 on BBH (complex reasoning tasks). On Qwen3-8B, the pattern holds: +2.52 on GPQA-Diamond, +4.32 on MMLU-Pro, +2.44 on HumanEval (code generation).

These OOD gains are not guaranteed by the framework — nothing in the PreRL objective explicitly encourages generalization. The paper's implied explanation is that NSR-PreRL suppresses reasoning patterns that are generally incorrect — common logical fallacies, algebraic mistakes, incomplete verification — and that these patterns are shared across domains even when the surface content differs. A model that has been trained not to skip verification steps in math problems will also be less likely to skip verification in science problems or code generation. In contrast, standard RL on P(y|x) may learn question-specific associations (e.g., "when the question mentions percentages, always convert to decimals") that don't transfer because they're tied to surface features of the MATH training distribution.

This finding is significant because it suggests that the pre-train space is a natural domain for acquiring transferable reasoning skills, while the post-train space is better suited for domain-specific refinement. This is consistent with the broader intuition from pre-training (where next-token prediction on diverse corpora produces general-purpose representations) but applies it to the much more targeted context of RL-driven reasoning improvement. The paper doesn't develop this into a full theory of transfer, but the empirical pattern is clear and has practical implications: if you want your reasoning model to generalize, optimize the foundation in the pre-train space before fine-tuning in the post-train space.

This is a significant empirical finding with theoretical implications rather than a methodological innovation. It connects the pre-train space optimization literature (which has long argued that P(y) optimization enables broader transfer) with the RLVR literature (which has focused on maximizing in-distribution performance) and provides concrete evidence that the generalization benefits of P(y) optimization survive — and are perhaps even amplified by — the transition to reward-driven online learning.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary training dataset is MATH (Lewkowycz et al., 2022), containing 7,500 competition-level mathematical problems. For evaluation, the paper uses MATH500 (Lightman et al., 2023), AMC23 (MAA, 2023), AIME24 (MAA, 2024), AIME25 (MAA, 2025), Minerva (Lewkowycz et al., 2022), and OlympiadBench (He et al., 2024). Out-of-distribution evaluation additionally uses GPQA-Diamond (Rein et al., 2024), MMLU-Pro (Wang et al., 2024a), BBH (Suzgun et al., 2023), and HumanEval (Chen et al., 2021). The choice of MATH for training is deliberate: mathematical reasoning requires multi-step inference from knowledge the base model already possesses, making it an appropriate testbed for whether pre-train space optimization can internalize reasoning capability.
Base model(s). All experiments use Qwen3-4B and Qwen3-8B (Yang et al., 2025) in non-thinking mode. The paper states these models "exhibit strong capability in advanced reasoning" and the non-thinking mode ensures that any reasoning behaviors observed (reflections, transitions, etc.) are genuinely elicited by the training process rather than being artifacts of system-level chain-of-thought prompting. Both scales are tested to assess whether the benefits of pre-train space optimization are consistent across model sizes.
Metrics. The primary metric is Avg@32: Pass@1 averaged over 32 independent rollouts per problem, computed to reduce output variance inherent in reasoning tasks. Pass@K (for K ∈ [1, 256]) is estimated from n = 300 samples per problem using the standard unbiased estimator: Pass@K := E_x∼D[1 − (n−c_K) / (n_K)], where c is the number of correct completions out of n generated responses. For the reasoning behavior analysis (Section 4.3), four behavioral categories are tracked: Backtracking, Verification, Subgoal Setting, and Enumeration, identified using GPT-4o as an automated annotator.
Baselines. The paper compares against seven baselines: GRPO (Shao et al., 2024), PPO (Schulman et al., 2017), Reinforce++ (Hu, 2025), RLOO (Ahmadian et al., 2024), Dr.GRPO (Liu et al., 2025b), DAPO with only the clip-higher mechanism (Yu et al., 2025), and the Vanilla (untrained) base model. All RL-based baselines use the same training dataset (MATH), the same base models, and the same 6-epoch on-policy training regime, ensuring a fair comparison. The inclusion of both standard algorithms (PPO, GRPO) and their recent optimized variants (Dr.GRPO, DAPO) tests whether DSRL's gains are attributable to simply using a better base algorithm rather than the pre-train space warmup.
Generation budget / compute accounting. The training compute budget is implicitly equalized across methods: all RL approaches use the same number of on-policy training epochs (6), the same prompts per step (128), and the same rollouts per prompt (G = 8). The paper does not separately account for the cost of the NSR-PreRL warmup phase versus the standard GRPO phase — they operate under the same per-step sampling budget. This means DSRL's reported improvements are achieved within the same total training compute as standard GRPO, modulo the negligible overhead of masking the input condition during gradient computation in the warmup phase. At evaluation time, the temperature is 1.0 and top-p is 1.0 for all methods.
Cross-validation / statistical protocol. To control for evaluation variance, Avg@32 reports the mean Pass@1 over 32 independent rollouts. Pass@K curves use n = 300 samples per problem, providing stable estimates even at high K values. The paper does not report confidence intervals or statistical significance tests — results are reported as point estimates. The ablation study on warmup steps (Figure 7) implicitly provides a measure of robustness by showing how performance varies across different hyperparameter settings.

Main Quantitative Results

DSRL vs. Baselines on Mathematical Reasoning Benchmarks (Table 1)

The headline result is that DSRL achieves the highest average score across all six math benchmarks for both model scales: 57.54 for Qwen3-4B and 58.47 for Qwen3-8B, outperforming all seven baselines.

For Qwen3-4B, DSRL achieves an average of 57.54, compared to GRPO's 55.79 (a gain of 1.75 points), Dr.GRPO's 57.04 (+0.50), and DAPO's 55.82 (+1.72). The largest gains over GRPO appear on the most difficult benchmarks: AIME24 (51.15 vs. 46.46, +4.69) and AIME25 (43.44 vs. 40.94, +2.50). On AMC23, DSRL reaches 89.22 vs. GRPO's 87.81 (+1.41). MATH500 shows more modest improvement: 89.68 vs. 89.17 (+0.51). Minerva and OlympiadBench gains are smaller: 30.48 vs. 30.06 (+0.42) and 41.26 vs. 40.29 (+0.97) respectively.

For Qwen3-8B, DSRL's average is 58.47, compared to GRPO's 57.00 (+1.47), Dr.GRPO's 56.16 (+2.31), and DAPO's 56.56 (+1.91). The pattern of largest gains on hard benchmarks persists: AIME24 at 56.15 vs. 54.06 (+2.09), AIME25 at 42.19 vs. 39.37 (+2.82). However, PPO achieves 51.98 on AIME24 for Qwen3-8B, compared to DSRL's 56.15 — a substantial gap suggesting DSRL's pre-train space warmup provides benefits that strong post-train-only baselines cannot match. On AMC23, DSRL reaches 90.00 vs. GRPO's 88.05 (+1.95).

A notable pattern across both scales: DSRL's advantage over GRPO is larger on AIME24 and AIME25 (the hardest benchmarks) than on MATH500 and AMC23 (relatively easier). This aligns with the paper's claim that pre-train space optimization particularly helps with problems requiring deeper reasoning, where the base model's initial probability of generating correct solutions is low and exploration is critical.

The comparison with GRPO variants (Dr.GRPO and DAPO) is informative: DSRL outperforms both on average, suggesting that its gains are not simply recapitulating improvements that optimized GRPO variants can achieve through better clipping strategies or regularization. However, the margins are modest (0.50–2.31 points on average), particularly against Dr.GRPO on Qwen3-4B (57.54 vs. 57.04, a 0.50-point gap). This indicates that some of DSRL's benefit may overlap with improvements that can be achieved through better post-train RL algorithms, though DSRL's consistent edge — especially on hard benchmarks — suggests complementary mechanisms.

Pass@K Performance Scaling (Figure 4)

The Pass@K curves in Figure 4 show that DSRL's advantage over GRPO is not limited to Pass@1 metrics but extends across the entire range of sampling budgets (K from 1 to 256). For Qwen3-8B, DSRL achieves higher Pass@K for all K on AIME24, AIME25, AMC23, MATH500, OlympiadBench, Minerva, and the average across all benchmarks. The gap is maintained or widens at higher K values on challenging benchmarks — for AIME25 at K=256, DSRL reaches approximately 77% while GRPO reaches approximately 74%. For Qwen3-4B, DSRL outperforms GRPO across most K budgets, though the gap is narrower on some benchmarks (e.g., MATH500 where both methods saturate near 95% at K=256).

The robust Pass@K improvement is interpreted by the paper as evidence that "DSRL's pre-train space optimization fosters a more exploration-friendly policy landscape" (Section 4.2). Since Pass@K measures the probability that at least one of K independent samples is correct, higher Pass@K at a given K implies that the model's correct solutions are more diverse — they are not simply the same correct answer repeated K times, but represent genuinely distinct reasoning paths that cover different subsets of problems. This is consistent with the claim that NSR-PreRL broadens exploration by eliminating systematic errors, forcing the model to discover multiple valid approaches to the same problem rather than relying on a single successful template.

Out-of-Distribution Generalization (Table 2)

DSRL's advantage extends to four OOD benchmarks covering science QA (GPQA-Diamond), broad knowledge (MMLU-Pro), complex reasoning (BBH), and code generation (HumanEval). For Qwen3-4B, DSRL outperforms GRPO on GPQA-Diamond by +3.79 (43.18 vs. 39.39), MMLU-Pro by +5.37 (66.49 vs. 61.12), BBH by +2.04 (82.41 vs. 80.37), and matches HumanEval exactly (81.10 for both). For Qwen3-8B, gains are: GPQA-Diamond +2.52 (53.48 vs. 50.96), MMLU-Pro +4.32 (63.79 vs. 59.47), HumanEval +2.44 (90.24 vs. 87.80). BBH shows a negligible difference (82.31 vs. 82.35, −0.04).

The proportionality of gains is notable: DSRL's improvements over GRPO on OOD benchmarks are proportionally larger than on in-distribution math benchmarks. For Qwen3-4B, the average OOD gain is approximately +2.80 points vs. +1.75 on math benchmarks; for Qwen3-8B, +2.31 vs. +1.47. The largest absolute gain is on MMLU-Pro (+5.37 for Qwen3-4B), a broad knowledge benchmark that covers 57 subjects across STEM, social sciences, humanities, and more — precisely the kind of domain where question-agnostic reasoning improvements (from suppressing general error patterns) should transfer. Conversely, the near-identical performance on BBH and the tie on HumanEval for Qwen3-4B suggest that the benefits are not universal across all reasoning domains and may depend on the degree to which the task shares error patterns with mathematical reasoning.

The paper interprets these results as evidence that "pre-train space optimization not only enhances in-domain reasoning but also cultivates a highly generalizable policy with superior OOD transferability" (Section 4.2). However, the mechanism is not explicitly tested — the paper does not ablate whether the generalization comes specifically from the pre-train space optimization target (P(y)) or from the increased reasoning behaviors (more reflections, more transitions) that NSR-PreRL elicits, both of which would be present in the DSRL model.

Training Efficiency and Dynamics (Figure 1c, Figure 3a)

Figure 1c provides a comparative training curve between DSRL and GRPO, showing three dimensions: average accuracy, sample efficiency, and response length. DSRL consistently leads in accuracy, achieving a final average of 61.6 vs. GRPO's 57.7. The sample efficiency advantage is quantified: DSRL reaches 45.0% accuracy with approximately 2.5× fewer training steps than GRPO, and 58.0% accuracy with approximately 1.6× fewer steps. Response length grows more rapidly and to a higher plateau under DSRL, consistent with the claim that NSR-PreRL stimulates deeper reasoning.

Figure 3a (discussed in Section 2.3 but central to the experimental narrative) shows that NSR-PreRL alone — without any post-train space RL — achieves 86% accuracy on AMC23 with roughly 3× fewer training steps than standard RL. This result is the empirical foundation for the claim that NSR-PreRL is an "exceptionally effective driver" for reasoning. However, the paper does not report the equivalent NSR-PreRL-only accuracy on the full benchmark suite (Table 1 reports only full DSRL), which would clarify how much of DSRL's final performance is attributable to the warmup alone versus the warmup + GRPO combination.

Evolution of Reasoning Behaviors (Figure 5)

To test whether DSRL genuinely improves reasoning quality (not just final-answer accuracy), the paper tracks four cognitive behaviors — Subgoal Setting, Enumeration, Verification, and Backtracking — during training using GPT-4o as an automated annotator. Across all four behaviors, DSRL drives "rapid, sustained growth and achieves significantly higher frequency ceilings across all patterns" compared to GRPO (Section 4.3). For Backtracking specifically, the DSRL curve rises sharply to approximately 18 average count per response by step 250, while GRPO reaches only about 8 — a more than 2× difference. For Verification, DSRL peaks at approximately 6 count vs. GRPO's 3. The paper attributes this to DSRL "effectively removing conditional constraints, unlocking intrinsic capacities for both rigorous self-correction and complex, structured reasoning exploration."

The paper does not provide normalized counts (e.g., per-token or per-response-length), which is relevant because DSRL generates longer responses (Figure 1c). If DSRL's responses are simply longer, then raw counts of reasoning behaviors would naturally increase even if the density of reasoning behaviors per token remained constant. The paper's interpretation that DSRL induces higher-quality reasoning would be strengthened by showing that reasoning behavior density (not just absolute frequency) increases.

Foundation Diagnostics: Fully Solved and Fully Unsolved Questions (Figure 6)

To assess whether NSR-PreRL systematically eliminates errors rather than just improving average-case performance, the paper tracks the count of "Fully Solved" questions (all 8 rollouts correct) and "Fully Unsolved" questions (all 8 rollouts incorrect) during training. During the NSR-PreRL phase (the initial portion of DSRL training), there is a "sharp surge in Fully Solved counts and a decline in Fully Unsolved ones" (Section 4.3). This surge precedes the standard RL phase, confirming that NSR-PreRL alone — without any post-train space optimization — is responsible for the initial error elimination.

After the transition to standard GRPO, DSRL continues to increase Fully Solved counts and decrease Fully Unsolved counts at a faster rate than GRPO alone, ultimately achieving more Fully Solved questions and a lower error rate. The paper interprets this as evidence that "NSR-PreRL systematically eliminates failure modes by internalizing fundamental error logic" and that the subsequent GRPO phase "can then focus on refining problem-specific nuances" rather than struggling with basic errors.

This figure provides the strongest direct evidence for the paper's central mechanistic claim: that pre-train space optimization establishes a better foundation by eliminating universal error patterns, and that post-train space optimization then extends this foundation to harder problems. The fact that the Fully Solved surge is temporally localized in the NSR-PreRL phase (before standard RL begins) establishes a causal ordering consistent with the paper's narrative.

Ablation Studies and Robustness Checks

Warmup step count (Figure 7): The number of NSR-PreRL steps before transitioning to standard GRPO exhibits an inverted-U relationship with final performance. At S = 5 steps, average score is approximately 55.69; at S = 10, it rises to 57.74; at S = 15, 57.11; at S = 20, 57.54; at S = 25, 57.27; at S = 30, it drops to 56.32. GRPO's baseline is 55.79. The optimal window is S ∈ [10, 25], with the peak at S = 10 or S = 20 (the paper reports both values; Figure 7 shows 57.74 at S = 10 as the highest point, though Table 1's DSRL results use S = 20 based on the warmup strategy ablation in Table 3). Fewer than 10 steps provide "insufficient stimulation" — the model does not benefit enough from pre-train space pruning. More than 25 steps cause performance to degrade below the peak (and eventually below GRPO's baseline at extreme values), attributed to "over-exploratory generation that hinders subsequent fine-grained optimization." This is a critical robustness check: the benefit of NSR-PreRL is conditional on it being bounded, confirming that it serves as a warmup rather than a replacement for standard RL.
Pre-train space vs. post-train space warmup (Table 3): When NSR is applied in the post-train space (with input conditioning) as a warmup before standard GRPO, the final average score is 54.38 — worse than GRPO's 55.79. In contrast, NSR applied in the pre-train space (without input conditioning, i.e., DSRL) achieves 57.54. This ablation isolates the optimization space as the causal factor: negative reinforcement alone is not beneficial; it is negative reinforcement specifically in the pre-train space that produces gains. Figure 13 further shows that by step 20, NSR-PreRL outperforms NSR-RL by an average of 6.6 points across AMC23, AIME24, and AIME25. This is a strong negative result for the alternative hypothesis that NSR in any form is helpful, and it validates the paper's central design choice of stripping away the input condition during the warmup phase.
PSR-PreRL vs. NSR-PreRL decomposition (Figure 3a): The decomposition of PreRL into positive and negative sample components reveals that PSR-PreRL (using only positive samples in the pre-train space) degrades performance, while NSR-PreRL succeeds. The full PreRL curve (which implicitly includes both PSR and NSR) initially rises but then collapses — the paper attributes this to PSR's harmful effects eventually dominating. This ablation validates the indicator function in Equation 6 that excludes positive samples during the DSRL warmup phase: positive reinforcement in the pre-train space is not just unnecessary but actively harmful. The paper's explanation — that PSR-PreRL requires out-of-distribution expert demonstrations and fails on self-generated on-policy data — is consistent with the QFFT comparison (Liu et al., 2025a) but is not independently tested beyond the Figure 3a training curves.
Reasoning thought type analysis (Figure 3b): After 20 NSR-PreRL steps, the model generates 14.89× more transition thoughts and 6.54× more reflection thoughts compared to the vanilla model (zero RL steps). Standard GRPO at 25 steps shows much more modest increases: transition thoughts and reflection thoughts are noticeably lower than NSR-PreRL despite having more training steps. This ablation isolates NSR-PreRL as the specific mechanism driving the emergence of deeper reasoning behaviors — the effect is not simply a consequence of RL training in general, nor is it an artifact of longer training (since GRPO at 25 steps underperforms NSR-PreRL at 20 steps on both accuracy and reasoning thought diversity).
Model scale robustness (Tables 1 and 2): DSRL's benefits are replicated across both Qwen3-4B and Qwen3-8B, with consistent improvements over GRPO on all six math benchmarks and most OOD benchmarks. The average gain over GRPO is slightly larger for Qwen3-4B (1.75 points) than for Qwen3-8B (1.47 points) on math benchmarks, but the pattern is reversed or mixed on OOD benchmarks. The paper does not explore whether the benefits scale further to larger models (e.g., Qwen3-14B or Qwen3-32B), leaving open the question of whether pre-train space warmup becomes more or less important as base model capability increases.
Training data sensitivity: The paper uses MATH (7,500 problems) as the sole training dataset. There is no ablation on training data quantity — we cannot determine whether NSR-PreRL's benefits depend on having a large training set, or whether similar gains would appear with smaller datasets. Given the paper's claim that NSR-PreRL "prunes incorrect reasoning paths," training data diversity likely matters: with too few problems, the model might not encounter enough distinct error patterns for the pruning to generalize.

Critical Assessment

The experiments provide moderately strong evidence for the central claim that NSR-PreRL warmup improves final performance, with important caveats about effect size, mechanism isolation, and generalization. Below, I assess each major claim against the evidence presented.

Claim: PreRL is a viable surrogate for standard RL because gradients of P(y) and P(y|x) align.

The gradient alignment analysis (Figure 2) is comprehensive within its scope: 400 rollouts from a single model (Qwen3-4B) on a single dataset (AMC23) show consistently non-negative inner products (min 0.94, mean 9.23). However, the paper does not demonstrate that this alignment holds:

Across different model scales (8B not tested)
At different stages of training (only the base model's distribution is analyzed; alignment could degrade as the policy shifts)
For the specific trajectories that NSR-PreRL actually updates on (incorrect, low-reward trajectories, which might have different alignment properties than the uniformly sampled rollouts used in Figure 2)

The alignment evidence is necessary but not sufficient — it establishes that PreRL updates won't actively harm the conditional policy, but does not establish that they help enough to justify the added complexity. The paper implicitly addresses this through the NSR-RL vs. NSR-PreRL ablation (Table 3), which shows that NSR-PreRL outperforms NSR-RL by 6.6 points at step 20 — but this is a performance comparison, not a direct measurement of whether gradient alignment persists or strengthens during training.

Claim: NSR-PreRL is an exceptionally effective driver that rapidly prunes incorrect reasoning and stimulates endogenous reasoning.

The evidence for this claim is the strongest in the paper. Figure 3a shows NSR-PreRL achieving 86% accuracy with 3× fewer steps than standard RL. Figure 3b shows 14.89× increase in transition thoughts and 6.54× increase in reflection thoughts after 20 NSR-PreRL steps, substantially outperforming GRPO at 25 steps. Figure 6 shows a sharp surge in Fully Solved questions during the NSR-PreRL phase specifically.

However, the claim that NSR-PreRL "prunes incorrect reasoning paths" is inferred from performance improvements rather than directly measured. The paper does not show that specific incorrect reasoning patterns (e.g., particular algebraic errors, particular logical fallacies) are suppressed — only that overall accuracy and reasoning behavior diversity increase. A more direct measurement would track the probability of known error patterns before and after NSR-PreRL. The claim of "pruning" is a mechanistic interpretation consistent with the data but not uniquely supported by it — alternative explanations (e.g., NSR-PreRL increases response length, which independently improves accuracy through more computation) are not ruled out, particularly given the correlation between response length and accuracy in Figure 1c.

The 14.89× and 6.54× figures for thought type increases require careful interpretation. These are relative increases from a very low baseline (the vanilla model generates very few transition and reflection thoughts in non-thinking mode). An absolute comparison of thought counts between NSR-PreRL and GRPO (which Figure 3b shows visually) reveals that NSR-PreRL's absolute counts are higher than GRPO's, but the raw count difference is more modest than the multiplicative factors suggest. Additionally, the thought type classification (Appendix B.3, Table 7) uses simple keyword matching — the validity of this classification for capturing genuine cognitive behaviors is not validated beyond citation to Chen et al. (2025a).

Claim: DSRL consistently outperforms strong baselines through Policy Reincarnation.

Table 1 supports this claim across six benchmarks and two model scales, with gains of 0.50–4.69 points over GRPO and 0.50–2.31 points over the best baseline (Dr.GRPO or DAPO) depending on the benchmark. The gains are consistent in direction but modest in magnitude — the average improvement over GRPO is 1.75 points for Qwen3-4B and 1.47 for Qwen3-8B. Whether these gains are practically meaningful depends on the deployment context: a 1–2 point average gain on MATH benchmarks represents a modest but non-trivial improvement, while a 4.69-point gain on AIME24 is more substantial.

A significant gap in the experimental design is the absence of a longer-GRPO baseline. The paper compares DSRL (S steps of NSR-PreRL + remaining steps of GRPO) against GRPO with the same total number of steps. But if NSR-PreRL is simply more sample-efficient, then standard GRPO trained for more steps might catch up or surpass DSRL. The training curves in Figure 1c show DSRL maintaining a lead at the final step, but the curves do not clearly plateau — GRPO's accuracy is still rising at the end of training. Extending training for both methods would clarify whether DSRL's advantage is a convergence acceleration (reaching the same asymptote faster) or a genuine improvement in the asymptote itself.

The paper also does not compare against an NSR-PreRL-only model on the full benchmark suite. Table 1 reports only DSRL (NSR-PreRL + GRPO), not the NSR-PreRL checkpoint before transition. This makes it impossible to determine how much of DSRL's final performance is attributable to the warmup alone. If NSR-PreRL alone achieves 86% on AMC23 with 20 steps (Figure 3a), and DSRL achieves 89.22% after full training (Table 1), then the post-train GRPO phase contributes only ~3 additional points — a modest gain that might be achievable through other means (e.g., longer NSR-PreRL training, or simply more parallel samples at inference).

Claim: Pre-train space optimization steers the policy toward a "refined correct reasoning subspace."

The Pass@K curves (Figure 4) provide indirect support: higher Pass@K at a given K suggests more diverse correct solutions, consistent with a broader exploration of the correct reasoning subspace. The reasoning behavior analysis (Figure 5) shows DSRL models exhibit more backtracking, verification, subgoal setting, and enumeration — behaviors characteristic of structured reasoning. The OOD results (Table 2) show transfer beyond math, consistent with general reasoning improvement rather than domain-specific optimization.

However, the concept of a "refined correct reasoning subspace" is a geometric metaphor that the paper does not operationalize or measure directly. There is no analysis of the diversity or coverage of correct solutions (e.g., clustering of reasoning trajectories, semantic diversity of approaches). The Fully Solved / Fully Unsolved analysis (Figure 6) comes closest to showing structural improvement — the model transitions from "sometimes correct by chance" to "reliably correct" on certain problems — but this is a property of the output distribution rather than of the parameter-space representation.

Genuine weaknesses and missing experiments:

1. Single training dataset. All experiments use MATH (7,500 problems). The paper does not demonstrate that NSR-PreRL's benefits generalize to training on other reasoning domains (code, science, logic) or to larger/cleaner training sets. If the mechanism relies on pruning systematic error patterns, dataset composition — what kinds of errors the model makes and how diverse they are — should matter substantially.

2. No comparison against continual pre-training. The paper argues that PreRL is superior to passive continual pre-training on P(y) because it uses active, reward-guided learning. But there is no direct comparison. A fair baseline would be: take the MATH training data, format it as sequences without explicit question conditioning, and do a brief continual pre-training phase (next-token prediction) before standard GRPO. If this simpler approach achieves similar or better results, the complexity of NSR-PreRL (requiring online rollouts, reward computation, and advantage-based sample selection) would not be justified.

3. No extended NSR-PreRL with length penalty. The paper identifies that NSR-PreRL eventually produces excessively long outputs that hinder subsequent training. An obvious mitigation — adding a length penalty to the reward during the warmup phase — is not explored. This would test whether NSR-PreRL's benefits can be decoupled from its over-exploration failure mode, potentially extending the optimal warmup window beyond 25 steps.

4. Limited scale exploration. Results are shown only at 4B and 8B scales. The paper's central claim — that pre-train space optimization establishes a "stronger foundation" — would predict that the benefits should be larger for smaller models (which have weaker foundations) and perhaps diminish for larger models (which already encode more systematic reasoning patterns from pretraining). Testing this prediction at 0.5B, 1.5B, and 14B+ would strengthen the mechanistic narrative.

5. No confidence intervals or significance testing. All results are point estimates. With Avg@32 (32 rollouts per problem) and 500-question test sets, the standard error of the mean accuracy difference between DSRL and GRPO is likely on the order of 0.5–1.5 percentage points depending on the benchmark. For comparisons where the gap is under 1 point (e.g., 57.54 vs. 57.04 against Dr.GRPO on Qwen3-4B), the difference may not be statistically significant. The paper's claim that DSRL "consistently outperforms" should be qualified by the uncertainty in these estimates, particularly for the smaller-margin comparisons.

6. No exploration of why NSR-RL warmup is harmful (Table 3). The finding that NSR applied in the post-train space degrades performance (54.38 vs. 55.79 for GRPO) is diagnostically important but underexplored. The paper does not provide training curves or behavioral analysis for the NSR-RL condition, making it difficult to understand the mechanism of harm. Does NSR-RL also stimulate increased reasoning behaviors? Does it cause different patterns of collapse? Understanding this negative result would clarify the boundary conditions for pre-train space RL.

7. The OOD results may conflate reasoning improvement with training distribution effects. The OOD benchmarks in Table 2 cover diverse domains (science QA, knowledge, code). DSRL's improvements are attributed to better reasoning, but an alternative explanation is that NSR-PreRL somehow mitigates overfitting to the MATH training distribution — i.e., standard GRPO overspecializes to math-specific patterns while DSRL preserves general capabilities. The paper does not test whether DSRL maintains or degrades performance on the original MATH training distribution relative to GRPO, which would distinguish between "DSRL prevents math overfitting" and "DSRL genuinely improves general reasoning."

8. Reliance on automated thought classification (GPT-4o) for reasoning behavior analysis. The validity of the behavior counts in Figures 3b and 5 depends on GPT-4o's ability to accurately classify reasoning steps. The paper does not report inter-annotator agreement, human validation of the classifications, or examples of disagreements. Given that the behaviors are defined somewhat subjectively (what counts as "backtracking" vs. simply trying a different approach?), the classification may introduce systematic biases.

6. Limitations and Trade-offs

The Difficulty (Practicality) of Deploying Pre-Train Space Optimization

The paper introduces a new training paradigm — Pre-train Space RL (PreRL) — but does not address its practical deployment in existing RLVR pipelines. PreRL's gradient computation requires a non-trivial modification to the standard policy gradient: the input condition x must be removed from log π_θ(y_t|x, y_{<t}) to compute ∇_θ log π_θ(y_t|y_{<t}). This is not a standard operation in frameworks like verl or TRL, which are designed around conditional generation and compute gradients through the full sequence [x; y] jointly. Implementing PreRL requires either (1) a separate forward pass with an empty or dummy input prefix, (2) custom gradient masking that zeroes out contributions from input tokens to the log-probability computation, or (3) architectural modifications to the attention mask during the backward pass. The paper provides no implementation details, no code, and no discussion of computational overhead for this modified gradient computation (Section 2.2, Equation 4).

The consequence is that a practitioner attempting to replicate DSRL faces an unspecified engineering barrier. It is unclear whether PreRL's gradient can be computed efficiently within standard distributed RL training loops — for example, whether it requires a separate model forward pass (doubling the per-step computation), whether it interacts poorly with sequence parallelism or tensor parallelism, or whether it introduces numerical instability from computing log-probabilities without the full context. The paper reports that NSR-PreRL achieves "3× fewer training steps" than standard RL (Section 2.3), but if each PreRL step is significantly more expensive than a standard GRPO step, the wall-clock advantage narrows or vanishes.

The paper provides no measurement of this overhead. The training setup in Appendix B.1 describes using the verl framework with standard hyperparameters (batch size 128, 8 rollouts per prompt), but does not mention any modifications to the gradient computation pipeline. There is no ablation on training throughput (steps per second) for PreRL vs. standard GRPO. The claim of "3× fewer training steps" (Section 2.3) and "2.5× and 1.6× fewer steps" for DSRL's efficiency (Section 4.3) is based on step count, not wall-clock time or total FLOPs. This is a fundamental gap: the paper optimizes for sample efficiency (performance per training step) while claiming practical efficiency, but these are not equivalent if the per-step cost differs.

The paper does not attempt to mitigate this. No implementation strategy is proposed. The authors do not acknowledge this as a limitation, and no future work is suggested on making PreRL's gradient computation practical.

Hard Problems Remain Unsolved by Pre-Train Space Optimization

The paper's central claim is that pre-train space optimization broadens exploration and establishes a "stronger foundation" for reasoning. However, the evidence shows that NSR-PreRL's benefits are concentrated on easier problems, and the hardest problems see minimal or no improvement from the pre-train space warmup. This is a capability boundary that the paper does not adequately characterize or explain.

Table 1 shows that DSRL's largest absolute gains over GRPO occur on AIME24 and AIME25 — the hardest benchmarks — where accuracy is low for all methods. On Qwen3-4B, DSRL scores 51.15 on AIME24 and 43.44 on AIME25, compared to GRPO's 46.46 and 40.94. However, even DSRL's best scores (51–56% on AIME24, 42–44% on AIME25) mean that nearly half or more of hard problems remain incorrect. The absolute accuracy ceiling on these benchmarks is far below what would be considered "solved," and the gap between DSRL and GRPO, while meaningful (2–5 points), does not represent a qualitative breakthrough on hard-problem reasoning.

The paper's own analysis of reasoning behaviors (Figure 5) shows that DSRL increases backtracking, verification, and subgoal setting, but these behaviors do not translate into solving the most difficult problems. The Fully Solved / Fully Unsolved analysis (Figure 6) shows DSRL achieving a higher count of Fully Solved questions than GRPO, but the absolute counts are not reported — we cannot determine what fraction of the hardest problems are Fully Solved by DSRL vs. GRPO. The paper's decomposition of test-set performance by difficulty (Figure 3b and related discussion) is limited to the NSR-PreRL vs. GRPO comparison on thought types, not on accuracy by difficulty bin.

The consequence is a capability ceiling: pre-train space optimization can suppress systematic errors and stimulate reasoning behaviors, but it cannot create fundamentally new reasoning strategies that the base model lacks. This is consistent with the paper's own framing that "the reasoning capacity of RLVR-trained models may be fundamentally bounded by the capabilities of the base model" (Section 1), but NSR-PreRL does not escape this bound — it merely pushes performance closer to it with fewer steps. On problems where the base model has near-zero probability of generating a correct solution (the hardest AIME problems, for instance), NSR-PreRL pruning incorrect paths does not help because there are no correct paths in the distribution to discover.

The paper does not directly measure this limitation. There is no difficulty-stratified analysis of DSRL's accuracy (analogous to Figure 3 right in the reference example, which breaks search performance into five difficulty quintiles). The paper does not report pass@2048 for the base model on its evaluation benchmarks, which would establish the theoretical ceiling that PreRL is approaching. Without this analysis, we cannot determine whether DSRL's gains represent convergence toward the base model's latent capability ceiling, or genuinely expanded capability beyond what was already present in P(y).

The paper does not address this limitation. It presents DSRL's improvements as evidence that pre-train space optimization "establishes a more robust reasoning foundation," but does not discuss the boundary conditions — what kinds of problems remain unsolvable, and whether this ceiling is inherent to the approach or could be overcome with more pre-train space optimization. The suggestion that "subsequent RL can then focus on refining problem-specific nuances" (Section 4.3) implies that post-train RL addresses the remaining gap, but the evidence (Table 1) shows DSRL still scores below 60% on AIME24 and below 45% on AIME25 — the gap is not closed.

The Optimal NSR-PreRL Warmup Duration Is a Brittle Hyperparameter

The paper identifies that NSR-PreRL's benefits are conditional on the warmup phase being carefully bounded: too few steps provide "insufficient stimulation," while too many steps cause "over-exploratory generation that hinders subsequent fine-grained optimization" (Section 4.4). The ablation in Figure 7 quantifies this: performance exhibits an inverted-U relationship with warmup steps, peaking at S = 10–25 steps (scores of 57.11–57.74) and declining to 56.32 at S = 30. At extreme values (not plotted but implied), performance would fall below GRPO's baseline of 55.79.

This brittleness has two practical consequences. First, the optimal warmup duration S must be tuned for each model scale, training dataset, and potentially even each training run — the paper does not demonstrate that S = 10–25 generalizes beyond Qwen3-4B on MATH. The Qwen3-8B results (Table 1) use the same Policy Reincarnation strategy but the paper does not report whether the warmup duration was retuned for the larger model. If S must be re-tuned, the computational cost of the hyperparameter sweep (training multiple DSRL runs to different S values and evaluating on held-out data) erodes the claimed efficiency gains.

Second, the mechanism underlying the inverted-U is not well-characterized, making it difficult to predict S without running the full sweep. The paper attributes the decline at large S to "over-exploratory generation" — response lengths become excessive and hinder standard RL convergence — but does not provide a diagnostic metric (e.g., average response length, thought-type ratio, policy entropy) that could serve as an early-stopping criterion. A practitioner cannot monitor training and determine "NSR-PreRL has done enough" without running the full DSRL pipeline to evaluate downstream GRPO performance — a circular requirement.

The paper acknowledges this limitation indirectly by performing the sweep (Figure 7), but does not discuss the generalization of the optimal S across settings. The warmup strategy ablation (Table 3) uses S = 20 for both model scales, suggesting the hyperparameter may be stable, but this is tested at only two points on a single training dataset. There is no exploration of whether S depends on training data size, problem difficulty distribution, or base model capability — all of which are plausible confounding variables given the mechanism (NSR-PreRL prunes error patterns, which depend on what errors the base model makes).

The paper does not propose a mitigation. No adaptive transition criterion is discussed — for example, switching from PreRL to standard RL when the rate of Fully Solved questions plateaus, or when average response length exceeds a threshold. The Policy Reincarnation controller is a fixed step-count schedule, not a data-dependent rule.

The Framework Depends on an Unquantified Assumption About Gradient Alignment Persistence

The paper's theoretical justification for PreRL rests on the cross-gradient term in Equation 5: [∇_θ log π_θ(y)]^T ∇_θ log π_θ(y|x). The empirical validation of this alignment (Figure 2) is conducted on the base model (Qwen3-4B before any RL training) using 400 rollouts from AMC23. The gradients are measured from the initial parameter state θ_0.

However, PreRL updates the parameters — θ changes throughout the warmup phase as incorrect trajectories are suppressed. The gradient alignment at θ_0 does not guarantee alignment at θ_S (after S PreRL steps). As the marginal distribution P(y) is modified to prune incorrect patterns, the relationship between ∇_θ log π_θ(y) and ∇_θ log π_θ(y|x) may change. In the worst case, alignment could degrade or even reverse for the specific trajectories that NSR-PreRL is suppressing, meaning the cross-gradient term becomes negative and PreRL updates start actively harming the conditional policy — which would manifest as the performance collapse seen in the full PreRL curve in Figure 3a (where the full PreRL including PSR degrades after initial improvement).

The consequence is that the theoretical guarantee of PreRL's validity is time-limited: it holds at θ_0 but may not hold at θ_S. The paper's claim that "optimizing the marginal log-likelihood log π_θ(y_t|y_{<t}) concurrently influences the conditional log-likelihood log π_θ(y_t|x, y_{<t}) in the same direction" (Section 2.2) is an assertion about the local gradient structure at initialization, not a proven invariant of the training dynamics. If alignment degrades as NSR-PreRL proceeds, then the bounded warmup (S ≤ 25) is not just a practical constraint to avoid over-exploration — it may be a theoretical necessity to stop before the cross-gradient coupling breaks down.

The paper provides no measurement of gradient alignment at intermediate or final PreRL checkpoints. Figure 2 shows only the base model distribution. The ablation on warmup steps (Figure 7) provides indirect evidence: performance declines beyond S = 25, which could be caused by gradient alignment degradation, over-exploration, or both — but these are not distinguished. The paper does not track the cross-gradient inner product through training, nor does it analyze whether the trajectories that NSR-PreRL updates on (incorrect, low-reward samples) have systematically different alignment properties than the "aligned cases" shown in Figure 10.

The paper does not address this limitation. The theoretical analysis in Section 2.2 is presented as a general justification for PreRL, not as a result that may be conditional on the distance from initialization. The empirical validation (Figure 2) is treated as sufficient without discussion of its time-dependence.

The Gains Are Modest Against Strong Baselines and the Mechanism Is Not Isolated

DSRL's average improvement over GRPO is 1.75 points for Qwen3-4B and 1.47 points for Qwen3-8B across six math benchmarks (Table 1). Against the strongest baseline — Dr.GRPO for Qwen3-4B (57.04) and PPO for Qwen3-8B (57.34) — the margins shrink to 0.50 and 1.13 points respectively. On individual benchmarks, DSRL sometimes underperforms a baseline: DAPO scores 88.20 on AMC23 for Qwen3-4B vs. DSRL's 89.22 (a 1.02-point gain), but PPO scores 87.89 vs. DSRL's 89.22 (a 1.33-point gain) — the variation between baselines is comparable to DSRL's advantage over them.

This matters because the paper claims that DSRL is a framework-level contribution — that pre-train space optimization provides a qualitatively different and better training strategy. If the gains are within 1–2 points of strong but simpler baselines, the practical case for the added complexity (modified gradient computation, hyperparameter tuning for warmup duration, two-phase training) is weaker. A practitioner might reasonably ask: is implementing PreRL's gradient masking and tuning S worth 1.5 average points on my benchmark suite, or would improving my base GRPO configuration yield similar gains?

More importantly, the mechanism is not isolated. DSRL combines two changes relative to GRPO: (1) a warmup phase and (2) negative-sample-only updates with input conditioning removed. The paper does not ablate whether a simpler warmup — for example, 20 steps of standard GRPO on a subset of the data, or 20 steps of standard GRPO with a higher learning rate — would provide similar benefits through improved initialization. The NSR-RL warmup ablation (Table 3, 54.38 vs. 55.79) rules out negative-sample-only warmup in the post-train space, but does not rule out that any warmup followed by standard GRPO outperforms GRPO from scratch. Curriculum learning and learning rate warmup are standard techniques that improve RL training stability; DSRL may be benefiting from the warmup structure itself rather than specifically from the pre-train space optimization.

The paper addresses this partially: Figure 6 shows that the Fully Solved surge occurs during the NSR-PreRL phase specifically, and Figure 3b shows reasoning thought increases that GRPO at comparable steps does not match. These are correlational, not causal — they show that NSR-PreRL induces certain behaviors, but do not prove that these behaviors are necessary for the final performance gains. A missing experiment is a control warmup (e.g., standard GRPO for 20 steps, then restart with fresh optimizer state for the main GRPO phase) that would isolate the effect of the warmup structure from the effect of the pre-train space optimization.

The paper does not discuss the effect size relative to baseline variance. All results are point estimates without confidence intervals, making it impossible to assess whether the 0.50-point gap over Dr.GRPO is statistically reliable. The evaluation protocol (Avg@32, n=300 for Pass@K) reduces variance but the paper does not report standard errors, and the test sets (e.g., AIME24 with 30 problems) are small enough that a few-question swing can produce changes of several percentage points.

No Comparison Against Continual Pre-Training or Other P(y) Optimization Methods

The paper's central distinction is between PreRL (active, reward-guided optimization of P(y)) and traditional pre-training / continual pre-training (passive, next-token-prediction-based optimization of P(y)). The paper argues in Section 5.2 and Appendix A.1 that PreRL is superior because it replaces "passive data consumption with active, reward-guided learning" and "transforms knowledge internalization from a static, data-driven process into a dynamic, feedback-driven one." This is a strong claim: that active, online, reward-driven optimization of P(y) is better than passive, offline optimization of P(y) for preparing a model for subsequent RL.

However, the paper never tests this claim. There is no comparison against a continual pre-training baseline: taking the MATH training data, formatting it as sequences (potentially without explicit question conditioning, to match PreRL's question-agnostic objective), and running a brief next-token prediction phase before standard GRPO. This baseline would use the same training questions, the same base model, and the same total compute budget, but would replace the online RL-based PreRL warmup with a simpler offline pre-training warmup. If continual pre-training achieves similar or better results, the entire motivation for PreRL — that active RL in the pre-train space is necessary — would be undermined.

The paper also does not compare against mid-training approaches (Wang et al., 2025b; Zhang et al., 2025; Zhou et al., 2025), which also optimize P(y) through continued pre-training on domain-specific corpora before RLVR. The paper cites these works as motivation (Section 5.2) but does not empirically position DSRL against them. Without this comparison, the paper cannot claim that the active, RL-driven nature of PreRL is the essential ingredient — it may be that any P(y) optimization before standard RL helps, and the specific mechanism (NSR-PreRL vs. continual pre-training) is incidental.

The consequence is a missing baseline that weakens the paper's central thesis. The paper claims that "PreRL fundamentally departs from both paradigms by replacing passive data consumption with active, reward-guided learning" (Appendix A.1) and that this departure yields benefits. But if passive P(y) optimization (continual pre-training) also improves subsequent RLVR performance — a plausible hypothesis given the literature on mid-training — then the paper's claimed contribution is not that pre-train space optimization helps, but that it helps more than the simpler alternative. This comparison is absent.

The paper does not acknowledge this as a limitation. The comparison with pre-training paradigms is purely conceptual (Table 4, Table 5), and the experimental section evaluates only against post-train space RL baselines (GRPO, PPO, etc.). No ablation or baseline addresses the question: does any optimization of P(y) before GRPO help, or is NSR-PreRL specifically required?

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a reframing of the RL-for-reasoning design space rather than a paradigm shift or a direct replacement for existing methods. The core conceptual move — distinguishing Pre-train Space optimization (targeting P(y)) from Post-train Space optimization (targeting P(y|x)) and demonstrating that the former is a viable domain for RL — expands the set of design dimensions researchers must consider from one (which RL algorithm, which reward structure) to two (which algorithm AND which optimization target). This is significant not because it immediately displaces GRPO or PPO, but because it reveals that the optimization target itself is a consequential design choice that changes the qualitative dynamics of learning.

The empirical evidence for why this reframing matters is specific and falsifiable. The paper shows that Negative Sample Reinforcement behaves oppositely in the two spaces: NSR applied in the Post-train Space as a warmup degrades performance below standard GRPO (54.38 vs. 55.79, Table 3), while NSR applied in the Pre-train Space as a warmup improves it (57.54). This asymmetry — where a mechanism that is harmful in one optimization space becomes beneficial in the other — is not derivable from standard RL theory and would have been missed entirely if researchers had continued to treat "RL for reasoning" as synonymous with "optimizing P(y|x)." The paper thus provides a new diagnostic axis: before asking whether positive or negative reinforcement helps, one must first ask in which space the reinforcement is being applied.

The paper also reconciles a tension in the RLVR literature that the authors identify but do not fully articulate. Prior work has shown that RLVR improves reasoning (DeepSeek-R1, Guo et al., 2025a) but that its gains are "fundamentally bounded by the capabilities of the base model" (Yue et al., 2025; Section 1). This creates a puzzle: if RLVR only sharpens existing capabilities, how do we improve capabilities that the base model lacks? The paper's framework provides a partial resolution by distinguishing two loci of improvement: error elimination (removing systematic mistakes from the model's distribution) and capability expansion (enabling the model to solve problems it could not solve before). NSR-PreRL primarily achieves the former — it prunes incorrect reasoning paths, as evidenced by the sharp surge in Fully Solved questions during the warmup phase (Figure 6). Standard GRPO, building on this pruned foundation, achieves the latter — extending success to harder problems. This decomposition suggests that prior negative findings about RLVR's limits may have conflated both types of improvement under a single optimization regime that is poorly suited for error elimination.

The research directions this paper makes more attractive are clear: work on pre-train space optimization, understanding gradient alignment dynamics during RL, phase-transition training strategies, and the asymmetric role of negative samples. The directions it makes less attractive (or at least reframes) are: pure algorithmic improvements to post-train space RL that ignore the optimization target, and continual pre-training work that does not incorporate active, reward-guided learning. The paper does not argue that post-train space RL is obsolete — DSRL explicitly retains it as the second phase — but it suggests that foundational error elimination should be treated as a distinct phase with distinct optimization characteristics, not as something that standard RLVR alone can accomplish efficiently.

The magnitude of this reframing should be kept in perspective. The empirical gains are modest: DSRL improves over GRPO by 1.75 points on average for Qwen3-4B and 1.47 for Qwen3-8B (Table 1), with the strongest baseline (Dr.GRPO) within 0.50 points on Qwen3-4B. These are not breakthrough improvements that would cause practitioners to immediately abandon standard RL pipelines. The paper's primary contribution is conceptual and diagnostic — it opens up a new axis of investigation — rather than providing a method that dramatically outperforms existing approaches. The long-term impact will depend on whether follow-up work can amplify these modest gains through better pre-train space algorithms, more principled transition strategies, or application to domains where the exploration deficit is more severe than in mathematical reasoning.

Follow-Up Research This Work Enables

Measuring gradient alignment as a dynamic quantity throughout PreRL training. The paper validates gradient alignment between ∇_θ log P(y) and ∇_θ log P(y|x) only at the base model initialization (Figure 2, 400 rollouts from Qwen3-4B at θ_0). This is a static snapshot that does not address whether alignment persists as PreRL updates parameters and modifies the marginal distribution. A critical follow-up experiment would track the cross-gradient inner product every k steps during NSR-PreRL training, separately for positive and negative samples, and correlate alignment degradation with the performance collapse observed in full PreRL (Figure 3a). If alignment degrades specifically for the trajectories being suppressed (incorrect samples with negative advantage), this would provide a mechanistic explanation for why NSR-PreRL must be bounded to S ≤ 25 steps — the warmup is not just avoiding over-exploration but stopping before the cross-gradient coupling breaks. The experiment requires: (1) checkpointing the model every 5 steps during a 40-step NSR-PreRL run, (2) computing the gradient inner product distribution at each checkpoint using a fixed set of rollouts from the base model distribution, and (3) plotting the mean and minimum inner product against training step and final DSRL performance. A finding that alignment remains stable while performance declines would rule out gradient decoupling as the failure mode and point toward over-exploration alone; a finding that alignment collapses at S > 25 would establish a theoretically-grounded early-stopping criterion.

Comparing NSR-PreRL against continual pre-training on the same data budget. The paper's central claim is that active, reward-guided PreRL is superior to passive continual pre-training for optimizing P(y) before standard RL (Section 5.2, Appendix A.1), but this comparison is never made empirically. A direct head-to-head experiment would use the MATH training set (7,500 problems) under matched compute budgets: (1) Continual pre-training baseline: format each training problem and its ground-truth solution as a single sequence (without explicit question-answer separation, to approximate the marginal distribution objective), run next-token prediction for the equivalent number of token updates as S = 20 steps of NSR-PreRL (counting total tokens processed, not steps), then train standard GRPO for the remaining budget. (2) NSR-PreRL baseline: the standard DSRL pipeline as described in the paper. (3) No-warmup baseline: standard GRPO for the full budget. If continual pre-training matches or exceeds NSR-PreRL, the paper's thesis — that active, reward-guided learning is essential — would be undermined, and the practical recommendation would simplify to "do a brief continual pre-training phase before GRPO," which is far easier to implement than PreRL's gradient masking. If NSR-PreRL outperforms continual pre-training, the comparison quantifies the value of active learning in the pre-train space. A strong version of this experiment would also test a hybrid: continual pre-training on the training set followed by NSR-PreRL warmup (to see whether passive and active P(y) optimization are additive or redundant).

Testing whether NSR-PreRL's benefits transfer to domains without clean verifiable rewards. The paper's entire framework depends on verifiable binary rewards (correct/incorrect final answers) to identify negative samples for suppression. Mathematical reasoning provides clean verification through ground-truth answers, but many important reasoning domains — legal analysis, medical diagnosis, strategic planning, creative problem-solving — lack unambiguous correctness signals. A stress-test experiment would apply DSRL to code generation (HumanEval, MBPP), where unit tests provide verifiable rewards but the reasoning structure differs from math (code often requires algorithmic decomposition rather than symbolic manipulation). The key measurement is whether DSRL's behavioral improvements (increased backtracking, verification, subgoal setting) emerge in code generation and whether they correlate with Pass@K improvements. A negative result — DSRL shows no gain over GRPO on code despite showing gains on math — would suggest that NSR-PreRL's mechanism is domain-specific, perhaps because mathematical error patterns (algebraic mistakes, incomplete case analysis) are more systematically suppressible than coding error patterns (off-by-one errors, API misuse) which may be more idiosyncratic. A positive result would strengthen the paper's OOD generalization narrative (Table 2) by showing that the mechanism transfers across reasoning modalities, not just across problem distributions within the same modality.

Exploring whether NSR-PreRL's exploration benefits can be decoupled from its length-explosion failure mode. The paper identifies that extended NSR-PreRL (S > 25) causes "over-exploratory generation" with excessively long outputs that hinder subsequent GRPO (Section 2.3, Figure 7). This conflates two effects: the desirable exploration of diverse reasoning strategies, and the undesirable length explosion. A targeted experiment would add a length penalty to the reward during the NSR-PreRL phase — for example, R'(y) = R(y) − α·|y| where α is a small coefficient tuned to keep average response length within a target range. Three conditions would be compared: (1) standard DSRL with S = 20 and no length penalty, (2) DSRL with S = 40 and length penalty tuned to match the response length of the S = 20 condition, (3) DSRL with S = 40 and no length penalty (expected to fail). If condition (2) matches or exceeds condition (1), it would demonstrate that longer NSR-PreRL provides additional benefits (more thorough pruning) that are currently masked by length explosion, and a simple length penalty would extend the practical warmup window. If condition (2) performs similarly to condition (3) (still degrades), it would suggest that the failure mode is not length per se but some other pathology of extended NSR-PreRL — perhaps the model begins suppressing trajectories that are actually correct but fall outside the current policy's high-probability region, effectively over-pruning the exploration space.

Scaling NSR-PreRL to larger models to test the "stronger foundation" hypothesis. The paper's mechanistic narrative predicts that NSR-PreRL's benefits should be largest for smaller, weaker base models that make more systematic errors and have more room for foundational improvement. Testing this prediction requires running DSRL at multiple scales: Qwen3-1.5B, Qwen3-4B (existing), Qwen3-8B (existing), Qwen3-14B, and Qwen3-32B, all on the same MATH training data and evaluation benchmarks. The key measurement is the relative gain of DSRL over GRPO as a function of model scale. If the gain shrinks monotonically with scale (e.g., +3 points at 1.5B, +1.75 at 4B, +1.5 at 8B, +0.5 at 14B, +0.2 at 32B), this confirms the prediction and implies that pre-train space optimization is primarily valuable for smaller models where the exploration deficit is most severe. If the gain is flat or U-shaped across scales, the mechanism is more complex — perhaps larger models have more latent reasoning patterns to unlock despite also having fewer systematic errors. A related measurement is whether the optimal warmup duration S changes with scale: if larger models make fewer systematic errors, they might need fewer NSR-PreRL steps to prune those errors, yielding a decreasing S(scale) function. This scaling experiment would transform the paper's conceptual framework into a predictive theory of when pre-train space optimization matters.

Diagnosing why NSR-RL warmup is harmful using behavioral and gradient alignment analysis. The paper's most diagnostically important negative result — that NSR applied in the post-train space degrades performance (Table 3, 54.38 vs. 55.79 for GRPO) — is reported as a single accuracy number with no mechanistic analysis. A detailed follow-up would replicate the NSR-RL warmup condition and track: (1) the same reasoning behavior metrics as Figure 3b and Figure 5 (transition thoughts, reflection thoughts, backtracking, verification, subgoal setting, enumeration), (2) the Fully Solved / Fully Unsolved trajectories from Figure 6, (3) the gradient alignment between ∇_θ log P(y) and ∇_θ log P(y|x) for the specific negative samples being suppressed in the post-train space (with input conditioning), and (4) the token-level probability changes for correct vs. incorrect trajectories separately. The hypothesis to test is that NSR-RL suppresses question-specific incorrect patterns that are actually correct for other questions — i.e., the model learns "for this specific AIME problem, approach X is wrong" but approach X might be correct for a different problem, and suppressing it in the conditional distribution hurts generalization. If this hypothesis is correct, we would expect to see: (a) Fully Solved counts do not increase during NSR-RL warmup (unlike the surge in NSR-PreRL), (b) behavioral metrics show less exploration (since the model is learning to avoid specific patterns for specific questions, not exploring new ones), and (c) gradient alignment for negative samples is lower than for positive samples (the conditional gradient for incorrect trajectories points in a different direction than the marginal gradient). This experiment would transform an unexplained negative result into a precise characterization of why the pre-train space is essential.

Practical Applications and Downstream Use Cases

Cost-efficient reasoning model training for organizations with limited compute budgets. The paper's finding that DSRL reaches 45.0% accuracy with ~2.5× fewer training steps and 58.0% accuracy with ~1.6× fewer steps than GRPO (Figure 1c) translates directly to reduced GPU-hours for training competitive reasoning models. For a team training a Qwen3-8B-scale model on 7,500 MATH problems, the standard GRPO pipeline with 6 epochs requires generating and training on ~360,000 rollouts (7,500 prompts × 8 rollouts × 6 epochs). DSRL with S = 20 warmup steps reduces the effective training to reach comparable or better performance by approximately 1.6–2.5× in the early-to-mid training regime, meaning the team could either (a) achieve the same accuracy with 40–60% of the compute budget, or (b) achieve higher accuracy with the same budget. However, this application is currently constrained by the unresolved implementation overhead: if PreRL's gradient masking doubles per-step wall-clock time, the step-count advantage may not translate to wall-clock savings. The practical value for this use case therefore depends on engineering PreRL's gradient computation to be as efficient as standard GRPO's — a specific engineering target that the paper identifies but does not solve.

Improving reasoning diversity for ensemble and best-of-N sampling at inference time. DSRL's Pass@K improvement over GRPO across K ∈ [1, 256] (Figure 4) means that for a fixed inference budget of K samples, DSRL-trained models are more likely to produce at least one correct answer. This is directly applicable to deployment scenarios where inference compute is cheap relative to the cost of errors — for example, automated math tutoring systems that can afford to generate 64 candidate solutions per problem and select the majority answer. On AIME25 with Qwen3-8B, DSRL's Pass@64 is approximately 70% versus GRPO's 65% (estimated from Figure 4), meaning the tutoring system would correctly solve ~5 more problems per 100 student queries. The mechanism — DSRL broadens exploration by pruning systematic errors, forcing the model to discover multiple distinct correct approaches — is specifically valuable in this setting because best-of-N selection benefits from diversity among correct solutions, not just Pass@1 accuracy. A practitioner deploying DSRL for this use case would need to tune K based on the latency requirements and cost of the specific application, but the Pass@K curves in Figure 4 provide the necessary data for making that tradeoff.

Domain-adaptive fine-tuning where training data and deployment data differ in distribution. DSRL's OOD generalization gains — +5.37 on MMLU-Pro for Qwen3-4B, +4.32 for Qwen3-8B (Table 2) — are specifically relevant when the available training data for RLVR is narrower than the intended deployment distribution. For example, a team building a general science QA system might only have verifiable training data for mathematical reasoning (where answers are unambiguous) but needs the model to generalize to physics, chemistry, and biology problems (where reasoning patterns transfer even if surface content differs). Training with DSRL on MATH data before deploying on GPQA-Diamond (+3.79 improvement for Qwen3-4B) provides a measurable boost without requiring domain-specific verifiable rewards for the target domain. This use case is practical today because it requires no changes to the deployment pipeline — only changes to the training pipeline — and the training data (MATH) is publicly available. The primary risk is that the OOD gains, while consistent, are modest in absolute terms (+2–5 points) and may not justify the implementation complexity for all teams. A practitioner considering this use case would want to validate on their specific target domain before committing to the DSRL training pipeline.

When to Prefer This Method

The paper does not articulate an explicit decision rule comparing DSRL against named alternatives (continual pre-training, mid-training, longer standard RL, or alternative exploration methods). It positions NSR-PreRL as a warmup that precedes standard GRPO — in all experiments, DSRL includes standard GRPO as a second phase — rather than as a standalone method that might be preferred over something else in specific circumstances. The ablation comparing DSRL against NSR-RL warmup and against varying warmup durations (Figure 7, Table 3) characterizes how to do the warmup, not whether to do it versus an alternative approach. Given this framing, a decision matrix would be speculative and is not provided here.