Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Reward

ArXiv: 2510.03222

🎯 Pitch

Standard RL training for reasoning models accidentally kills off rare but crucial exploratory words like "wait" and "however," causing a creativity collapse—while generic fixes that boost randomness just amplify useless noise. This paper shows you can fix the collapse and hit a new state-of-the-art by selectively protecting these "reasoning sparks" with a clever probability filter.

---

1. Executive Summary

This paper investigates the exploration collapse that limits reinforcement learning with verifiable rewards (RLVR) for large language models, identifying that the systematic elimination of valuable low-probability exploratory tokens — termed reasoning sparks (e.g., "wait," "however," "perhaps") — causes performance plateaus, while indiscriminate entropy boosting amplifies irrelevant noise (e.g., "cost," "fine") and accelerates collapse. The authors introduce Low-probability Regularization (Lp-Reg), which constructs a less-noisy proxy distribution by filtering out presumed noise tokens below a probability threshold and renormalizing over the remaining candidates, then uses a forward KL divergence penalty to shield these reasoning sparks from over-penalization during GRPO training. On five math benchmarks using Qwen3-14B-Base and Qwen2.5-32B-Base, Lp-Reg achieves state-of-the-art performance with a 60.17% average accuracy, improving 2.66% over prior methods, while sustaining continuous on-policy scaling across 3,000 training steps and 81,204 GPU-hours where entropy-control baselines collapse, establishing that preserving exploration quality through selective low-probability token protection is more effective than indiscriminate entropy maximization only when meaningful exploratory tokens are distinguished from semantically irrelevant noise.

2. Context and Motivation

The Core Problem: RLVR Training Plateaus Due to Exploration Collapse, Not Just Entropy Decay

The fundamental problem this paper addresses is deceptively specific: during reinforcement learning with verifiable rewards (RLVR) for mathematical reasoning, language models systematically lose their ability to explore alternative solution paths, causing performance to plateau and then collapse. This matters because RLVR has become the dominant paradigm for training large reasoning models — systems like OpenAI's o1 and DeepSeek-R1 that generate extended chain-of-thought reasoning to solve complex problems. If the training process inherently extinguishes exploration, it imposes a hard ceiling on how far these models can scale.

The paper identifies a precise mechanistic failure behind this exploration collapse. The issue is not that the model stops being "random" in general — it's that specific, semantically meaningful low-probability tokens get systematically eliminated from the model's output distribution during RLVR training. These tokens, which the authors call reasoning sparks, include words like "wait," "however," "perhaps," and "alternatively." They function as logical pivots — expressions of uncertainty or reconsideration that naturally initiate new reasoning branches. When a model is about to produce a wrong answer (probability 0.97 for the incorrect path in Figure 1a), sampling "wait" at probability 0.03 represents a crucial exploratory action that can redirect the reasoning toward a correct solution.

The paper's key diagnostic finding (Figure 1c) is that standard GRPO training forces these tokens' sampling distributions to collapse toward higher probabilities. In other words, the model learns to produce "wait" only when it's already highly confident — turning what should be an exploratory pivot into a deterministic pattern. As the aggregated violin plots in Figure 1c show, after training the model can no longer sample these tokens at low probabilities: the distribution has shifted entirely toward the high-probability regime, meaning the model has lost its capacity for the kind of low-confidence exploration that generates diverse reasoning pathways.

This is a more precise and causal diagnosis than the commonly observed phenomenon of "entropy collapse." Prior work recognized that policy entropy decays rapidly during RL training (Yu et al., 2025; Cui et al., 2025; Wang et al., 2025), but treated entropy as a correlational symptom — the quantity of randomness decreases, so we should boost it. The paper's central insight is that the quality of exploration matters more than the quantity of randomness, and that indiscriminately boosting entropy actually makes things worse because it amplifies semantically irrelevant noise alongside genuine exploration.

Why This Problem Matters: The Scaling Bottleneck

This exploration collapse is not merely an inconvenience — it is the primary bottleneck preventing continuous scaling of RLVR training. The paper documents that standard GRPO training plateaus and collapses, as shown in Figure 1b: accuracy peaks around 60% then declines, while training entropy drops from approximately 0.5 to near zero. This pattern is consistent across model sizes and training configurations, and it means that simply training longer does not produce better models.

The practical stakes are substantial. Training runs for reasoning models consume enormous computational resources — the paper reports 81,204 GPU-hours for the 3,000-step continuous scaling experiments on Qwen2.5-32B-Base. If training collapses partway through, all that compute is wasted. More importantly, if exploration collapse is the mechanism limiting performance, then simply throwing more compute at RLVR training without addressing the exploration bottleneck cannot yield further progress. The field needs a principled intervention that preserves meaningful exploration throughout training, not just at the beginning.

Beyond the immediate training stability concerns, this problem has theoretical significance for our understanding of how LLMs learn to reason. The paper's analysis of reasoning sparks suggests that the pre-trained model already possesses a rich capacity for exploratory reasoning — the token distributions that encode "wait, maybe I should reconsider" are present in the base model. RLVR training, by penalizing incorrect outputs, inadvertently extinguishes this capacity because those exploratory tokens appear more frequently in unsuccessful reasoning trajectories. This reveals a fundamental tension in RLVR: the reward signal that incentivizes correct reasoning simultaneously punishes the very exploratory behaviors that enable the model to discover correct reasoning in the first place. Understanding and resolving this tension is crucial for the entire paradigm of reinforcement learning for language models.

Prior Approaches and Where They Fall Short

The paper positions its contribution against a landscape of methods that all attempt to address entropy collapse but do so through indirect, correlational mechanisms:

Entropy regularization (GRPO + Entropy Loss) represents the most direct prior approach: add a policy entropy bonus to the GRPO objective function, following Maximum Entropy RL principles, to explicitly encourage the model to maintain randomness in its token predictions. The paper's results (Figure 1b) show this does not work — in fact, GRPO + Entropy Loss collapses even faster than standard GRPO, with accuracy dropping precipitously and entropy initially spiking then crashing. The paper's diagnostic analysis reveals why: the entropy bonus indiscriminately amplifies the low-probability sampling of irrelevant tokens like "cost" and "fine" (Figure 1d, Figure 18), which are semantically out of context for mathematical reasoning. This creates noise that destabilizes training rather than enhancing meaningful exploration. As the authors state:

"an indiscriminate focus on maximizing randomness risks amplifying noise and destabilizing training, suggesting a deeper issue beyond simply the quantity of randomness"

Clip-Higher (Yu et al., 2025), a core component of DAPO, modifies the PPO clipping mechanism by using an asymmetric clipping range that raises the upper clip bound. The intuition is that standard clipping constrains the policy from increasing probabilities of high-advantage tokens; by relaxing the upper bound while maintaining the lower bound, the method allows the policy to more aggressively increase probabilities of tokens that lead to good outcomes. This indirectly encourages higher entropy because tokens that receive positive advantages can have their probabilities raised more freely. However, Clip-Higher operates entirely through the advantage-clipping mechanism — it doesn't distinguish between types of exploration or directly protect low-probability tokens from being penalized when they receive negative advantages. The paper's results (Figure 4, 6) show Clip-Higher performs competitively on Qwen2.5-32B but degrades on Qwen3-14B, which the authors attribute to its reliance on off-policy importance sampling and its fragility across model scales.

Selective High-Entropy Training (80/20) (Wang et al., 2025) takes a more targeted approach: rather than regularizing all tokens, it restricts policy gradient updates to only the top 20% of tokens with the highest generation entropy. The rationale is that high-entropy tokens represent "forking" points where the model is uncertain and exploration is most valuable. While this represents progress toward selectivity — recognizing that not all tokens matter equally for exploration — it still targets high entropy as a proxy for exploration value. The paper's analysis in Section 6.1 and Figure 11 provides a direct refutation of this proxy: the top 1% of highest-entropy tokens are dominated by common functional terms ("sqrt," "times") and formatting characters ("\n"), not the semantically meaningful exploratory markers that actually drive diverse reasoning. Targeting high-entropy tokens therefore risks regularizing precisely the wrong tokens — amplifying noise while missing the low-probability reasoning sparks that the paper identifies as crucial.

KL-Cov (Cui et al., 2025) prevents entropy collapse by applying a selective KL-divergence penalty to tokens with the highest covariance between their log probabilities and advantages. The idea is that high-covariance tokens are those where the advantage signal varies most with the token's probability, making them potential sources of instability. By regularizing these specific tokens, the method aims to smooth the optimization landscape and prevent the policy from collapsing onto deterministic patterns. This approach shares with Lp-Reg the insight that selective intervention is better than global entropy control, but it identifies intervention targets based on statistical properties of the training dynamics (covariance with advantages) rather than semantic properties of the tokens themselves (meaningfulness for exploration).

GSPO (Zheng et al., 2025) modifies the clipping mechanism to operate at the sequence level rather than the token level, promoting higher training entropy through a different structural reformulation of the policy gradient objective. Like the other methods, GSPO addresses entropy indirectly through the optimization procedure rather than directly examining and preserving exploratory tokens.

The critical gap across all these prior approaches is that they monitor policy entropy, which is correlational rather than causal to exploration. Entropy measures how spread out the probability distribution is — it's a summary statistic that conflates meaningful exploratory uncertainty with meaningless noise. A distribution that assigns equal probability to "wait" and "the" has the same entropy as one that assigns equal probability to "the" and "cost," but only the former supports mathematical reasoning exploration. By operating on entropy directly, prior methods cannot disambiguate these cases.

The paper explicitly frames this distinction:

"While these methods primarily operate by monitoring policy entropy, which is correlational rather than causal to exploration, our analysis delves directly into the next-token prediction distribution. This allows for a more semantically grounded and causally-informed investigation of the probabilities of individual candidates and their role in exploration dynamics."

How the Paper Positions Itself

The paper positions Lp-Reg as a fundamentally different approach to the exploration problem — one that operates on the semantics of exploration rather than its statistics. The key conceptual moves are:

1. From entropy to probability: a causal perspective. Instead of asking "how much randomness does the model have?" (entropy), the paper asks "which specific tokens enable exploration, and what is happening to their probabilities during training?" This reframing enables a causal diagnosis: the problem is not that entropy is low, but that the probabilities of specific exploratory tokens are being driven to zero by the RL penalty on incorrect outputs. The solution is not to increase entropy globally, but to selectively prevent the elimination of these specific tokens.

2. From quantity to quality of exploration. The paper demonstrates through multiple analyses (Figures 1d, 11, 18) that boosting exploration quantity indiscriminately amplifies irrelevant noise alongside meaningful exploration — and that the noise component is destructive. Lp-Reg's filtering mechanism explicitly separates the two, preserving reasoning sparks while suppressing noise. This quality-over-quantity framing is a direct challenge to the Maximum Entropy RL paradigm that underlies entropy regularization approaches.

3. Leveraging the model's intrinsic confidence. The paper identifies a statistical regularity that makes selective protection possible: within the low-probability range, meaningful exploratory tokens consistently exhibit higher average next-token probabilities than semantically irrelevant tokens (Figure 14). This probabilistic gap — which the authors attribute to the LLM's intrinsic confidence calibration (citing Nguyen et al., 2025; Fu et al., 2025) — provides a principled basis for filtering. The model itself distinguishes between plausible alternatives and noise, even in the low-probability tail. Lp-Reg exploits this: by setting a probability threshold τ, the method can discard the extreme low-probability tail (dominated by noise) while retaining the slightly-higher-probability meaningful tokens. This is a fundamentally different strategy from entropy control — rather than fighting the model's confidence by artificially boosting randomness, Lp-Reg works with the model's confidence to identify which low-probability tokens are worth protecting.

4. A self-contained, policy-intrinsic regularization. Unlike Clip-Higher and other methods that rely on off-policy importance sampling corrections, Lp-Reg's regularization is entirely derived from the current policy's own probability distribution. The proxy distribution is constructed by filtering and renormalizing πθ itself, and the forward KL divergence provides a soft constraint rather than a strict imitation target. This makes the method equally effective in on-policy and off-policy settings — a practical advantage that the paper's results confirm (Figure 4, Table 1), where Lp-Reg is one of the only methods that performs strongly across both training regimes.

The paper explicitly contrasts its approach with the broader literature on entropy minimization for LLMs (Gao et al., 2025; Agarwal et al., 2025), which encourages models to commit to high-confidence solution paths. While entropy minimization improves inference performance by sharpening confidence distributions, it would exacerbate the training-time problem that Lp-Reg addresses by further suppressing low-probability exploration. Lp-Reg occupies a middle ground: it allows the model to sharpen its distribution in general (entropy can decrease as the model learns) while selectively preventing the complete elimination of specific tokens that enable recovery from mistakes.

3. Technical Approach

3.1 Reader Orientation

This paper builds a regularization plugin for reinforcement learning training — a modification to the GRPO algorithm that prevents the model from permanently forgetting how to explore alternative reasoning paths. The core problem it solves is that standard RLVR training systematically eliminates low-probability tokens like "wait" and "however" that serve as exploratory pivots, and the solution takes the form of a carefully-designed KL divergence penalty that shields these specific tokens from extinction without amplifying irrelevant noise, using only the model's own probability distribution to decide which tokens to protect.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components that interact during each training step:

1. Base Language Model (πθ) — the policy being trained, initialized from a pretrained checkpoint (Qwen3-14B-Base or Qwen2.5-32B-Base). It generates multiple candidate solutions for each math problem, and its token-level probabilities form the foundation for all downstream computation.

2. GRPO Policy Gradient Engine — the standard training loop that samples groups of outputs, computes reward-based advantages using Equation 2, and updates the policy using a clipped surrogate objective. Lp-Reg modifies this engine's clipping behavior and adds a regularization term to its objective.

3. Proxy Distribution Constructor — a filtering-and-renormalization pipeline that takes the current policy's next-token probability distribution and produces a cleaned-up version (πproxy) where tokens below a confidence threshold τ are zeroed out and remaining tokens' probabilities are rescaled to sum to one. This proxy distribution serves as the regularization target.

4. Lp-Reg Penalty Module — a selective forward KL divergence penalty that fires only under specific conditions: the token being updated has low probability (below batch percentile δBρ), survives the proxy filter (πproxy > 0), and receives a negative advantage signal (Ai,t < 0). When all three conditions hold, the penalty pushes πθ away from zero for that token, preventing its elimination.

Information flows through these components at each training step: the policy generates outputs → the GRPO engine computes advantages → for each token in each output, the Lp-Reg module computes the current policy probability, constructs the proxy distribution, and checks the three activation conditions → if activated, the forward KL penalty is added to the GRPO loss → the combined loss updates the policy. The proxy distribution is always constructed from the data-generating policy (πθold in off-policy mode, the current πθ in on-policy mode), ensuring the filtering reflects the model's actual confidence at the time of generation.

3.3 Roadmap for the Deep Dive

• First, the GRPO foundation and Lp-Reg's modifications to it — understanding the base algorithm is essential because Lp-Reg alters both its clipping behavior and its objective function.
• Second, the proxy distribution construction (filtering threshold τ and renormalization) — this is the core mechanism that distinguishes reasoning sparks from noise, and the design choices here determine everything downstream.
• Third, the full Lp-Reg objective function and its three activation conditions — each condition encodes a specific insight about when and why low-probability tokens need protection.
• Fourth, the forward KL divergence choice and why reverse KL fails — this is a non-obvious design decision with significant empirical consequences.
• Fifth, the training configuration and hyperparameters — understanding the scale, computational cost, and sensitivity of the method.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a methodological paper whose core idea is that selectively protecting low-probability tokens through a filtered proxy distribution regularization enables sustained exploration in RLVR training, and that this selectivity — distinguishing reasoning sparks from noise — is what makes the regularization effective where entropy-based approaches fail.

---

GRPO Foundation and Lp-Reg's Structural Modifications

Lp-Reg is built on top of Group Relative Policy Optimization (GRPO), so understanding the base algorithm is necessary before examining what Lp-Reg changes.

Standard GRPO objective. GRPO is a value-free policy gradient method that estimates advantages using multiple samples drawn from the same prompt rather than learning a separate value function. For each prompt, the model generates G outputs (G = 8 in all experiments), computes a reward for each using a rule-based verifier, and defines the advantage of each token in output i as:

$A_{i,t} = \frac{R(o_i) - \text{mean}(G)}{\text{std}(G)}$

where G = {R(o1), ..., R(oG)} is the set of rewards for the group of outputs, R(oi) is the scalar reward for the i-th output (1.0 for correct final answer, 0.0 otherwise), mean(G) and std(G) are the empirical mean and standard deviation of rewards within the group, and Ai,t is the advantage assigned uniformly to every token in output oi. This group-relative normalization means that tokens in outputs that score above the group average get positive advantages (they should become more likely), while tokens in below-average outputs get negative advantages (they should become less likely).

The standard GRPO objective that updates the policy is:

$J_{\text{GRPO}}(\theta) = \mathbb{E}_{(q,a) \sim D, \{o_i\}_{i=1}^G \sim \pi_{\theta_{\text{old}}}(\cdot|q)} \left[ \frac{1}{G} \sum_{i=1}^G \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \min\left(r_{i,t} A_{i,t}, \text{clip}(r_{i,t}, 1-\epsilon, 1+\epsilon) A_{i,t}\right) - \beta D_{\text{KL}}(\pi_\theta \| \pi_{\text{ref}})\right]$

where:

• q is the math problem prompt, a is the reference answer
• oi is the i-th generated output, |oi| is its length in tokens
• ri,t = πθ(oi,t | q, oi,<t) / πθold(oi,t | q, oi,<t) is the importance sampling ratio for the t-th token in output i
• ε is the clipping ratio (controls how far the new policy can deviate from the behavior policy)
• β controls the strength of KL regularization from the reference policy πref (typically the initial pretrained model)
• The min(·, ·) and clip(·, ·, ·) operations implement PPO-style trust region: when Ai,t > 0, the ratio is clipped at 1+ε to prevent the policy from increasing token probabilities too aggressively; when Ai,t < 0, the ratio is clipped at 1-ε to prevent the policy from decreasing token probabilities too aggressively

What this computes operationally: For each token in each generated output, the GRPO objective multiplies the importance sampling ratio by the advantage. If a token appeared in a good output (Ai,t > 0), the gradient pushes πθ(oi,t) higher, making that token more likely. If a token appeared in a bad output (Ai,t < 0), the gradient pushes πθ(oi,t) lower. The clipping prevents any single update from changing token probabilities too drastically.

The problem with standard GRPO for exploration: The standard clipping range [1-ε, 1+ε] is symmetric. This means that when a token receives a negative advantage (Ai,t < 0), the policy is allowed to reduce its probability down to (1-ε)·πθold. Over many training steps, tokens that consistently appear in incorrect outputs will have their probabilities driven toward zero — this is precisely what happens to reasoning sparks. The symmetric clipping provides no special protection for exploratory tokens that happen to appear in unsuccessful trajectories.

Lp-Reg's first modification: asymmetric clipping. Lp-Reg removes the lower bound of the clipping range while maintaining a large upper bound:

$\text{clip}(r_{i,t}(\theta), 0, U) \cdot A_{i,t}$

where U = 10 is the upper bound on the importance sampling ratio. The critical change is the lower bound: standard GRPO clips at max(1-ε, ri,t), which means that even for tokens with very low ri,t (the new policy assigns them much lower probability than the old policy), the effective ratio used in the gradient is 1-ε, softening the penalty. Lp-Reg uses max(0, ri,t) instead — zero is the floor. This means the policy gradient can drive token probabilities down more aggressively when the advantage signal is negative, without the protection of the standard 1-ε floor.

This might seem counterproductive for protecting low-probability tokens — why remove the floor that limits how much a token can be penalized? The answer is that Lp-Reg handles protection through a separate mechanism (the Lp-Reg penalty term), and removing the lower bound simplifies the interaction between the policy gradient and the regularization. The standard clipping's lower bound creates an artificial constraint that interferes with the regularization's targeted protection. By handling downward pressure through the policy gradient (which can now push harder) and upward protection through the Lp-Reg penalty (which selectively prevents elimination), the two forces can balance more cleanly.

Why U = 10: The upper bound of 10 on ri,t means that if the new policy assigns a token more than 10x the probability it had under the old policy, the effective ratio used in the gradient update is capped at 10. This prevents numerical instability from extreme policy changes while still allowing substantial increases for tokens that receive strong positive advantages.

---

Proxy Distribution Construction: Filtering and Renormalization

The proxy distribution πproxy is the central innovation of Lp-Reg. It is a modified version of the current policy's probability distribution over the vocabulary, constructed to preserve meaningful tokens while discarding presumed noise. The construction happens in two steps applied to every token prediction in every generated sequence.

Step 1: Filtering noise tokens by probability threshold. For a given context (q, oi,<t), the current policy πθ produces a probability distribution over the vocabulary V (the set of all possible next tokens). Lp-Reg discards any token whose probability falls below a threshold τ, treating these ultra-low-probability tokens as presumed noise:

$\text{Filtered tokens} = \{o \in V : \pi_\theta(o|q, o_{i,<t}) \leq \tau\}$

The paper explores two strategies for setting τ:

• Fixed threshold: τ is a constant hyperparameter (e.g., τ = 0.02). Every token whose probability is ≤ 2% is discarded.
• Min-p threshold (default): τ is defined adaptively relative to the maximum probability in the distribution: τ = κ · maxo'∈V πθ(o'|q, oi,<t), where κ ∈ (0, 1) is a hyperparameter (default κ = 0.02). This makes the filtering adaptive to the distribution's sharpness — if the distribution is very peaked (one token has probability 0.95), then τ is 0.019, filtering tokens below ~2% of the peak probability. If the distribution is flatter (maximum probability 0.3), then τ is 0.006, filtering fewer tokens. The key insight is that the model's confidence varies across contexts, and a fixed absolute threshold might be too aggressive in diffuse distributions (filtering tokens that are actually competitive) or too lenient in peaked distributions (retaining tokens that are clearly noise).

The paper's primary experiments use the min-p strategy for its adaptiveness, but ablation studies (Figure 8) confirm that fixed thresholds also work — the core filtering principle is what matters, and adaptiveness provides a modest additional benefit.

Why this threshold exists conceptually: The filtering is crucial because low-probability tokens are not uniformly valuable. Some low-probability tokens are meaningful exploratory pivots ("wait," "however") that should be protected. Others are semantically irrelevant intrusions ("cost," "fine," "ere") whose amplification would destabilize training. The authors' key empirical finding (Figure 14) provides the justification for threshold-based filtering: within the low-probability range (specifically, the 0 to 0.1 range, limited to the top-64 most probable tokens due to storage constraints), meaningful exploratory tokens consistently exhibit higher average next-token probabilities than irrelevant tokens across all training stages. This probabilistic gap — which the authors attribute to the LLM's intrinsic confidence calibration — means that a probability threshold can serve as a rough but principled filtering mechanism, discarding the extreme tail of the distribution (dominated by noise) while retaining the slightly-higher-probability meaningful tokens.

Step 2: Probability renormalization. After filtering, the proxy distribution πproxy is constructed by assigning zero probability to discarded tokens and renormalizing the probability mass among the surviving tokens:

$\pi_{\text{proxy}}(o| \cdot) = \begin{cases} \frac{\pi_\theta(o|\cdot)}{\sum_{o' \text{ s.t. } \pi_\theta(o'|\cdot) > \tau} \pi_\theta(o'|\cdot)} & \text{if } \pi_\theta(o|\cdot) > \tau \\ 0 & \text{otherwise} \end{cases}$

Or in plain language:

• For each surviving token (πθ(o|·) > τ): its proxy probability is its original probability divided by the sum of probabilities of all surviving tokens, so that the proxy probabilities sum to 1.
• For each discarded token (πθ(o|·) ≤ τ): its proxy probability is zero.

What this computes operationally: Every token that survives the filtering gets a probability boost in the proxy distribution because the probability mass from discarded tokens is redistributed proportionally. A token with original probability 0.05 that survives filtering might get a proxy probability of 0.06 (if 20% of the probability mass was discarded and redistributed), while a token with original probability 0.001 that gets discarded gets proxy probability 0. This makes the proxy distribution a "sharpened" version of the original, where the model's confidence is amplified by removing the long tail.

Why this form matters for the downstream regularization: The proxy distribution is used as the reference in a KL divergence penalty. If a token has zero probability in πproxy, the forward KL divergence DKL(πproxy ∥ πθ) does not penalize the policy for also assigning it low probability — there's no protection. If a token has non-zero probability in πproxy, the forward KL imposes a significant penalty when πθ assigns it probability approaching zero. So the filtering step directly controls which tokens get protected: only tokens that survive the filter (πproxy > 0) are shielded from elimination by the regularization. The renormalization amplifies the relative probability of surviving tokens in the proxy, making the protection signal stronger for tokens that were on the borderline of the filter.

Why min-p over a fixed threshold: The min-p formulation makes the filtering adaptive to the model's confidence in each specific context. The authors note that "the dynamic nature of min-p provides a more robust estimate of the model's confidence across different contexts, better preserving genuine reasoning sparks." In contexts where the model is uncertain (flat distribution), the min-p threshold is lower, preserving more tokens — this is exactly when exploration is most valuable. In contexts where the model is confident (peaked distribution), the threshold is higher, filtering more aggressively — this is when noise filtering is most important because the model already knows what to do and random exploration would just introduce errors. Figure 2 provides a visual example of this renormalization process, showing how a distribution with many tokens below the threshold gets cleaned up into a distribution over only the high-confidence candidates.

---

The Full Lp-Reg Objective Function

The Lp-Reg objective combines the modified GRPO policy gradient with the selective Lp-Reg penalty:

$J_{\text{Lp-Reg}}(\theta) = \mathbb{E}_{B \sim D, (q,a) \sim B, \{o_i\}_{i=1}^G \sim \pi_{\theta_{\text{old}}}(\cdot|q)} \left[ \frac{1}{\sum_{i=1}^G |o_i|} \sum_{i=1}^G \sum_{t=1}^{|o_i|} \left( \text{clip}(r_{i,t}(\theta), 0, U) \cdot A_{i,t} - \beta \cdot \mathbb{I}[\text{conditions}] \cdot D_{\text{KL}}(\pi_{\text{proxy}}(\cdot|q, o_{i,<t}) \| \pi_\theta(\cdot|q, o_{i,<t})) \right) \right]$

where:

• B is a training batch sampled from the dataset D (Dapo-Math-17K)
• The first term is the modified GRPO policy gradient with asymmetric clipping (lower bound 0, upper bound U = 10)
• β = 1.0 is the Lp-Reg regularization coefficient (controls the strength of the penalty relative to the policy gradient)
• The indicator function I[conditions] gates the penalty — it is 1 only when all three conditions are met, and 0 otherwise, making the penalty selective rather than global

The three activation conditions are:

1. Low probability condition: πθ(oi,t | q, oi,<t) < δBρ. This checks whether the token being updated was sampled with probability below the batch-level low-percentile threshold. The threshold δBρ is computed dynamically for each training batch B as the ρ-th percentile of the sampling probabilities of all tokens in that batch. With ρ = 0.5% for Qwen2.5-32B and ρ = 1% for Qwen3-14B, δBρ represents the probability value below which the lowest 0.5% or 1% of tokens fall. A token satisfies this condition if it was a genuinely unlikely choice — the model sampled it despite (or because of) its low probability. This condition restricts the penalty to tokens that are actually low-probability, avoiding unnecessary regularization of high-probability token choices.

2. Not-noise condition: πproxy(oi,t | q, oi,<t) > 0. This checks whether the token survived the proxy filtering — that is, whether its probability in the current policy's distribution was above the noise threshold τ. If the token was filtered out (its probability was too low even for the proxy distribution), this condition is false, and the penalty does not activate. This is the critical selectivity mechanism: only tokens that the model considers at least minimally plausible (above τ) get protected; tokens in the extreme tail (probabilities below τ) are left unprotected and can be driven to zero by the policy gradient.

3. Negative advantage condition: Ai,t < 0. This checks whether the token appeared in an output that received a below-average reward. If the token was in a good output (positive advantage), the policy gradient is already pushing its probability up, and regularization is not needed — in fact, regularizing in that case would slow down learning of correct patterns. If the token was in a bad output (negative advantage), the policy gradient is pushing its probability down, and the Lp-Reg penalty counterbalances this to prevent complete elimination.

What this computes operationally — the gating logic: For each token in each generated output, the system performs a three-part check: (a) was this token sampled at low probability? (b) is this token probably meaningful, not noise? (c) is the policy gradient trying to reduce this token's probability? Only if the answer is "yes" to all three does the Lp-Reg penalty fire, pushing back against the gradient's downward pressure. If any answer is "no," the penalty is zero and the standard GRPO update proceeds unchanged.

The expected effect is: when a reasoning spark like "wait" appears in an incorrect output (so Ai,t < 0), it was likely sampled at low probability (the model was leaning toward a different continuation), and it survives the proxy filter (its probability is above τ), so all three conditions are met and the penalty protects it from being extinguished. When a noise token like "cost" appears in an incorrect output, it was likely sampled at even lower probability, fails the proxy filter (its probability is below τ), so the penalty does not protect it — the GRPO gradient can drive its probability to zero as desired.

Why these three conditions together: Each condition addresses a specific failure mode. Removing the low-probability condition would apply the penalty to high-probability tokens in bad outputs, which would prevent the model from learning to avoid obvious mistakes — if the model confidently produces a wrong answer, the policy gradient should reduce that confidence. Removing the not-noise condition would protect extreme-tail tokens, amplifying the noise problem that the paper shows destabilizes training (Figure 8, "Lp-Reg w/o τ" causes catastrophic collapse). Removing the negative advantage condition would apply the penalty to exploratory tokens in good outputs, which would unnecessarily slow down the reinforcement of correct patterns. The three conditions together ensure the regularization is targeted precisely at the problem — the elimination of meaningful exploratory tokens from unsuccessful trajectories.

Ablation of the conditions (Section 5.3):

• Regularizing both positive and negative samples (Figure 9, "Regularize Pos. & Neg.") shows that applying the penalty to all low-probability tokens (not just those in negative-advantage outputs) still helps compared to no regularization, validating that protecting low-probability tokens is broadly beneficial. However, it learns slower than negative-only regularization because "extending the penalty to positive samples provides little additional exploratory advantage" and "introduces minor noise into the gradient updates."
• Removing the noise filter (Figure 8, "Lp-Reg w/o τ") causes catastrophic collapse — entropy explodes and accuracy crashes. This confirms that without filtering, the penalty protects extreme-tail noise tokens that "consist of irrelevant noise that destabilizes training if regularized."
• Targeting high-entropy tokens instead of low-probability tokens (Figure 7, "Highest H Regularization") fails to sustain exploration and collapses after an initial entropy spike. This directly validates the paper's central claim: high-entropy tokens are dominated by "common function words or formatting characters, not the meaningful, low-probability exploratory tokens."

---

Forward KL Divergence: Why This Form and Not Reverse KL

The Lp-Reg penalty uses the forward KL divergence DKL(πproxy ∥ πθ), where the proxy distribution is the reference (first argument) and the current policy is the target (second argument). This choice is non-obvious and empirically consequential.

Definition: For a given context, the forward KL divergence between πproxy and πθ over the vocabulary V is:

$D_{\text{KL}}(\pi_{\text{proxy}} \| \pi_\theta) = \sum_{o \in V} \pi_{\text{proxy}}(o|\cdot) \log \frac{\pi_{\text{proxy}}(o|\cdot)}{\pi_\theta(o|\cdot)}$

This is the expected log-ratio of probabilities, where the expectation is taken with respect to πproxy. It penalizes the policy πθ when it assigns low probability to tokens that πproxy considers plausible.

What this computes operationally: For each token o in the vocabulary, the term πproxy(o) · log(πproxy(o)/πθ(o)) contributes to the penalty. When πθ(o) is close to πproxy(o), the ratio is close to 1, the log is near 0, and the contribution is small. When πθ(o) is much smaller than πproxy(o) — meaning the policy is assigning very low probability to a token that the proxy distribution considers reasonable — the ratio is large, the log is positive, and the contribution is large. When πθ(o) is zero (the policy has eliminated the token), the log diverges to infinity, imposing an infinite penalty — this is why the forward KL provides strong protection against token elimination. When πproxy(o) is zero, the term contributes nothing regardless of πθ(o), so tokens filtered out as noise receive no protection.

Why forward KL over reverse KL DKL(πθ ∥ πproxy): The reverse KL divergence would be DKL(πθ ∥ πproxy) = Σo πθ(o) · log(πθ(o)/πproxy(o)). In this form, the expectation is taken with respect to πθ, and the penalty pushes πθ to match πproxy everywhere. This is a much stronger constraint — it forces the policy to imitate the proxy distribution, which is itself a heuristic derived from the current policy, not a ground-truth target. The reverse KL "penalizes any deviation of πθ from πproxy, effectively forcing the policy to strictly imitate this non-ideal, heuristic target" and "constrains the protection of potentially valuable exploratory tokens" because it prevents the policy from exploring beyond what πproxy considers plausible.

The forward KL is asymmetric in a specific beneficial way: it heavily penalizes the policy for ignoring tokens that the proxy considers plausible (mode-dropping), but it does not penalize the policy for exploring tokens that the proxy assigns zero probability. This is exactly what Lp-Reg needs: prevent the elimination of reasoning sparks (mode-dropping in the low-probability tail) without forcing the policy to mimic the proxy's exact probability ratios for those tokens. The policy can still adjust the relative probabilities among protected tokens based on the GRPO advantage signal; the forward KL only prevents it from driving them to zero.

The empirical comparison in Figure 10 confirms this: forward KL significantly outperforms reverse KL, with reverse KL showing substantially worse accuracy (around 0.45 at 1,000 steps vs. forward KL's ~0.60 on Qwen3-14B).

Why not just add a standard entropy bonus: An entropy bonus -H(πθ) = Σo πθ(o) log πθ(o) would push all token probabilities toward uniformity, without any selectivity. This would protect noise tokens equally with reasoning sparks, which the paper shows causes collapse (GRPO + Entropy Loss in Figure 1b). The forward KL to a filtered proxy achieves selectivity because the proxy distribution itself encodes the filtering: noise tokens have zero probability in πproxy, so they contribute nothing to the KL penalty, regardless of what πθ does with them. Reasoning sparks have non-zero probability in πproxy, so the KL penalty protects them.

---

Dynamic Batch-Level Threshold for Low-Probability Identification

The threshold δBρ that determines whether a token is "low probability enough" to trigger the Lp-Reg penalty is computed dynamically from the current training batch:

$\delta^B_\rho = \text{Percentile}(\{\pi_{\theta_{\text{old}}}(o_{i,t} | q, o_{i,<t}) : \text{all tokens in batch } B\}, \rho)$

where ρ = 0.5% for Qwen2.5-32B-Base and ρ = 1% for Qwen3-14B-Base.

What this computes operationally: For all tokens in all outputs in the current training batch, collect their sampling probabilities (the probability the behavior policy assigned to that token at generation time). Sort these probabilities and find the value at the ρ-th percentile — the value below which ρ% of all token samples fall. This dynamic threshold adapts to the batch: if the batch contains many low-probability tokens (the policy is exploring widely), δBρ will be higher, triggering the penalty more broadly. If the batch contains mostly high-probability tokens (the policy is behaving deterministically), δBρ will be lower, triggering the penalty only for the most extreme low-probability cases.

Why dynamic rather than fixed: A fixed probability threshold (e.g., penalize all tokens with probability < 0.01) would fail as training progresses and the policy's probability distribution shifts. In early training, when the model is relatively uncertain, many meaningful tokens might have probabilities around 0.01-0.05, and a fixed threshold of 0.01 would miss them. In late training, when the model is very confident, even genuinely meaningful alternative tokens might have probabilities well below 0.01, and a fixed threshold would protect all of them (including borderline noise). The percentile-based threshold adapts to the distribution's scale, always targeting the lowest ρ% of token samples regardless of the absolute probability values.

The training dynamics in Figure 16 show that δBρ decreases over training steps — as the model learns and becomes more confident, the absolute threshold for what counts as "low probability" shifts downward. Simultaneously, the regularization ratio (the fraction of below-threshold tokens that survive the proxy filter and actually get regularized) also decreases, suggesting that "as training progresses, the extreme low-probability range becomes increasingly dominated by irrelevant tokens, constituting what we term irrelevant noise" while "semantically meaningful exploratory tokens are lifted into higher-probability regions, thus requiring less regularization."

Hyperparameter sensitivity (Appendix A, Figure 15): The paper tests ρ values of 0.005, 0.010, and 0.015, finding that "training trajectories for average test accuracy are comparable" and "final performance across all three settings is highly comparable." Similarly, κ values of 0.01, 0.02, and 0.03 for the min-p ratio show "a similar trend of stability." This insensitivity is desirable — it means practitioners don't need extensive hyperparameter tuning to get Lp-Reg working.

---

Training Configuration and Scale

Models and datasets: Experiments use Qwen3-14B-Base (Yang et al., 2025) and Qwen2.5-32B-Base as the pretrained starting points. Training data is Dapo-Math-17K (Yu et al., 2025), consisting of 17,000 mathematical reasoning problems with verifiable answers. Maximum response length is 8,192 tokens. The dataset choice is deliberate: math problems have clean, binary reward signals (correct or incorrect final answer) that enable RLVR without reward model training.

GRPO configuration: Group size G = 8 outputs per prompt. Global batch size of 256 prompts. For off-policy training, mini-batch size of 32 (8 gradient updates per rollout, since 256/32 = 8, and each "step" in the paper refers to one rollout — so one step = 8 gradient updates). This distinction between "steps" (rollouts) and "gradient updates" matters for interpreting the training curves: the x-axis in all figures shows rollout steps, not gradient updates.

Optimization: Constant learning rate of 1 × 10−6 without a warmup schedule. The paper doesn't specify the optimizer, but based on the framework (verl), it likely uses AdamW with default betas. The policy gradient clipping uses an upper bound of U = 10 on the importance sampling ratio.

Lp-Reg hyperparameters:

• Probability percentile threshold ρ = 0.5% (32B) or 1% (14B)
• KL regularization coefficient β = 1.0
• Min-p ratio κ = 0.02 (so the noise threshold τ is 2% of the maximum token probability in each context)
• The proxy distribution is constructed from the data-generating policy: πθold in off-policy mode, the current πθ in on-policy mode

Baseline configurations: All baselines use hyperparameters from their original public implementations. Specifically, GRPO + Entropy Loss uses an entropy coefficient of 0.002, implemented within the verl framework. This coefficient controls the relative weight of the entropy bonus versus the policy gradient objective.

Training duration and compute:

• Main comparisons: approximately 1,000 training steps for Qwen3-14B-Base (~8,000 GPU-hours on 32 NVIDIA H20 GPUs) and 800 steps for Qwen2.5-32B (~16,000 GPU-hours on 64 NVIDIA H20 GPUs)
• Continuous scaling experiment: 3,000 steps for Qwen2.5-32B with Lp-Reg on-policy, totaling 81,204 GPU-hours
• For methods that experienced training collapse (defined as >10% accuracy drop), training was stopped early to conserve resources

Evaluation protocol: Five math benchmarks are used: AIME24, AIME25 (both with sampled decoding at temperature 0.6, 16 responses per problem, majority voting), MATH-500, OlympiadBench, and Minerva Math (all with greedy decoding). This dual evaluation strategy reflects the benchmarks' sizes: AIME has small test sets (15 problems each), so sampling and voting provides more stable estimates; the other benchmarks are large enough for greedy decoding to give reliable scores. The paper reports the best accuracy achieved during the entire training run for each method, not the final accuracy.

Cross-method fairness: The paper explicitly trains "models until their performance saturated" to "mitigate unfair comparisons arising from different convergence speeds during the early stages of training." This means methods that converge faster don't get an artificial advantage from being evaluated at a fixed step count before slower methods reach their peak. The exception is methods that collapsed, where early stopping was applied.

---

On-Policy vs. Off-Policy Training

Lp-Reg is evaluated in both on-policy and off-policy training regimes, which have fundamentally different data generation and update characteristics.

Off-policy training uses the standard GRPO approach: generate outputs with the behavior policy πθold, compute advantages, perform multiple gradient updates on the same batch of data using importance sampling corrections (the ri,t ratios in Equation 4). The proxy distribution is constructed from πθold (the data-generating policy), ensuring the filtering reflects the distribution that actually produced the tokens. This is the regime used by most baseline methods (Clip-Higher, KL-Cov, 80/20, GSPO).

On-policy training generates fresh outputs after each gradient update and uses those for the next update, with no importance sampling corrections needed (ri,t = 1 for all tokens because the behavior policy and current policy are the same). The proxy distribution is constructed from the current πθ. This regime is more stable because there's no distribution mismatch between the data-generating policy and the policy being updated, but it's also more computationally expensive because each update requires new rollouts.

The paper finds that "on-policy training exhibits better training stability and testing performance compared to off-policy training" (Figure 4). This advantage is attributed to "the inherent stability of on-policy updates, which avoid distribution shifts caused by mismatched data-sampling and training policies." Moreover, "off-policy methods, such as Clip-Higher, often rely on importance sampling clipping, leading to instability" — a problem that Lp-Reg's self-contained regularization avoids because it doesn't depend on importance sampling corrections for its protection mechanism.

Lp-Reg is one of the only methods evaluated in both regimes that performs strongly across both. The 80/20 method also supports on-policy training, but most other baselines are off-policy only. The on-policy results are the primary results (Table 1), with on-policy Lp-Reg achieving 60.17% on Qwen3-14B and 52.54% on Qwen2.5-32B, both the highest in their respective model scales.

---

Summary of Design Choices and Their Justifications

• Min-p thresholding over fixed threshold: adapts to the model's confidence per context, better preserving reasoning sparks in uncertain situations and filtering noise more aggressively when the model is confident (Figure 8 shows fixed threshold still works but dynamic is better). The κ = 0.02 value means the threshold is 2% of the maximum token probability, which is empirically stable across κ ∈ {0.01, 0.02, 0.03} (Figure 15b).

• Forward KL over reverse KL: provides soft protection against mode-dropping without forcing strict imitation of a heuristic proxy (Figure 10 confirms significantly better performance). This is the key technical insight that makes the proxy distribution work as a regularization target rather than a distillation target.

• Three-condition gating over unconditional penalty: the negative-advantage condition focuses protection where it's needed (tokens that would otherwise be eliminated); the not-noise condition prevents amplifying noise that destabilizes training; the low-probability condition avoids regularizing high-confidence tokens. Each condition's necessity is ablated (Figures 7, 8, 9).

• Asymmetric clipping (0, U=10) over symmetric (1-ε, 1+ε): removes the artificial floor that interferes with the Lp-Reg penalty's targeted protection, allowing the two forces (GRPO pushing down, Lp-Reg pushing up) to balance without the clipping adding a confounding constraint.

• Batch-level percentile threshold δBρ over fixed probability threshold: adapts to shifting probability scales during training, ensuring the penalty always targets the lowest ρ% of sampled tokens regardless of how the absolute probabilities evolve. This is more important than it might appear because policy probabilities change dramatically during RLVR training — what counts as "low probability" at step 100 (when entropy is still moderate) is very different from what counts at step 500 (when entropy has collapsed).

• Proxy distribution from the data-generating policy: using πθold (off-policy) or πθ (on-policy) ensures the filtering reflects the distribution that actually generated the tokens being updated. Using a different policy (e.g., πref) for the proxy would create a mismatch between what was sampled and what gets protected, potentially protecting the wrong tokens.

4. Key Insights and Innovations

Innovation 1: Reframing Exploration as Token-Level Survival Rather Than Entropy Maintenance

The dominant conceptual framework for addressing exploration collapse in RLVR has been entropy management: the model's randomness decreases, so we should boost it — either through entropy bonuses (Maximum Entropy RL), asymmetric clipping that allows probability increases (Clip-Higher), or selective gradient application to high-entropy "forking" tokens (80/20). All of these treat entropy as the thing that matters, and low entropy as the problem to fix.

This paper fundamentally reframes the issue. The problem is not that the model becomes insufficiently random in the aggregate — it's that specific functionally important tokens are being systematically eliminated from the model's output distribution. The "reasoning sparks" concept (tokens like "wait," "however," "perhaps") introduces a semantic category of tokens whose value lies not in what they mean directly, but in what they enable: pivoting between reasoning paths, initiating reconsideration, expressing uncertainty that leads to alternative solutions. These tokens are functionally analogous to exploration actions in a discrete Markov decision process, but with a crucial difference — they are not explicitly designated as such by any reward signal. The model discovers their utility through pre-training, and RLVR inadvertently extinguishes them.

What makes this reframing intellectually significant is that it inverts the causal arrow from how the field has thought about exploration and entropy. Prior work implicitly assumed that low entropy causes poor exploration — if we could just keep entropy high, the model would continue exploring diverse reasoning paths. The paper's diagnostic evidence (Figures 1c, 1d, 17, 18) shows that this gets causality backwards: entropy is low because specific exploratory tokens have been eliminated, and indiscriminately boosting entropy fails because it amplifies semantically irrelevant noise rather than restoring the specific tokens that matter (Figure 1d). The GRPO + Entropy Loss baseline collapses even faster than standard GRPO (Figure 1b), which is a strong negative result that the entropy-centric framework cannot explain but the token-survival framework predicts: adding noise to an already-noisy distribution doesn't create meaningful exploration, it just destabilizes the optimization.

This is not merely a terminological shift. It implies a completely different class of solutions: instead of manipulating summary statistics of the output distribution (entropy), intervene directly on the probabilities of specific semantically-identified tokens. The challenge then becomes how to identify which tokens are worth protecting without human annotation — a challenge that the Lp-Reg method addresses through its filtering mechanism, but which exists conceptually independent of any particular solution.

The connection to prior work on intrinsic confidence (Saurav et al., 2022; Nguyen et al., 2025; Fu et al., 2025) is important here. Those works established that LLMs' next-token probabilities encode meaningful uncertainty signals — tokens with higher probabilities tend to be more contextually appropriate. Lp-Reg operationalizes this insight for exploration: if the model's intrinsic confidence can distinguish plausible alternatives from noise even in the low-probability tail (Figure 14), then we can use probability thresholds to selectively protect the plausible alternatives without protecting the noise. This connects the exploration problem in RLVR to the broader literature on model calibration and confidence, which had not previously been applied to reinforcement learning for reasoning.

This is a fundamental reframing, not an incremental refinement. It changes what you measure (token-level probabilities rather than distribution-level entropy), what you intervene on (specific token survival rather than global randomness), and what failure mode you diagnose (token elimination rather than entropy collapse). The distinction matters because the two frameworks make different predictions: if entropy is the problem, any method that increases entropy should help, and methods that decrease entropy should hurt. But the paper shows this is false — entropy minimization (Gao et al., 2025) improves inference, and entropy boosting (GRPO + Entropy Loss) hurts training. The token-survival framework correctly predicts that the type of randomness matters more than its quantity, and that selective protection of semantically meaningful low-probability tokens is sufficient for sustained exploration.

Innovation 2: Exploiting the Model's Own Confidence Distribution as a Noise Filter

Prior approaches to selective exploration in RLVR (Selective High-Entropy Training, KL-Cov) all identify intervention targets based on training dynamics: which tokens have high entropy, which tokens have high covariance between probability and advantage. These are behavioral indicators — they tell you where the training process is unstable or where the model is uncertain, but they don't tell you whether intervening at those points will produce useful exploration or harmful noise.

Lp-Reg introduces a fundamentally different criterion: use the model's own confidence — as encoded in its next-token probability distribution — to distinguish valuable exploratory tokens from noise. The key empirical finding that makes this possible is Figure 14: across all training stages, meaningful exploratory tokens ("wait," "perhaps") consistently exhibit higher average next-token probabilities than semantically irrelevant tokens ("cost," "fine") within the same low-probability range (0 to 0.1). This probabilistic gap is not large in absolute terms — both classes of tokens have very low probabilities — but it is consistent and systematic, reflecting the model's intrinsic sense that certain tokens are more contextually plausible than others even when both are unlikely.

The conceptual move here is subtle but significant. Rather than fighting the model's confidence (as entropy boosting does, by pushing probabilities toward uniformity) or ignoring it (as most exploration methods do, by treating all low-probability tokens as equivalent), Lp-Reg works with the model's confidence as a signal for filtering. The threshold τ is set relative to the model's own maximum probability, so what counts as "noise" depends on how confident the model is in its top prediction. This creates an adaptive filter that is more aggressive when the model is confident (filtering out more of the tail, since the model has a clear idea of what comes next and low-probability alternatives are likely noise) and more permissive when the model is uncertain (preserving more alternatives, since the model itself is unsure and exploration is more valuable).

This is a conceptual bridge between two previously disconnected literatures. The exploration-in-RL literature (Maximum Entropy RL, curiosity-driven exploration, count-based exploration) has focused on generating diversity through explicit randomness or novelty bonuses, treating the policy as something to be pushed away from its natural tendencies. The confidence-and-calibration literature has focused on whether models know what they know, treating confidence as an evaluation metric. Lp-Reg shows that confidence can serve a third function: as an operational signal for selective exploration, telling the training algorithm which departures from the model's preferred outputs are worth encouraging and which are just noise. This is a novel synthesis with implications beyond Lp-Reg — it suggests that any method for exploration in language model training could potentially benefit from using the model's confidence to filter exploration targets, rather than relying solely on behavioral statistics like entropy or advantage covariance.

The ablation in Figure 8 (Lp-Reg w/o τ vs. w/ τ) provides the crucial evidence: removing the confidence-based filter causes catastrophic collapse, with entropy exploding and accuracy crashing. This is not a minor implementation detail — it's the mechanism that makes the entire approach work. Without filtering, the regularization protects noise tokens equally with reasoning sparks, and the noise destabilizes training exactly as the entropy bonus does. With filtering, the regularization is targeted and stable. The fact that a fixed threshold (τ = 0.02) also works, just slightly worse than the adaptive min-p threshold, confirms that the core principle — using probability as a proxy for meaningfulness — is what matters, not the specific adaptation mechanism.

This is a fundamental conceptual contribution that reframes how to think about exploration quality, though the specific implementation (min-p thresholding) is an adaptation of an existing inference-time sampling technique (Nguyen et al., 2025). The contribution lies in the idea of using the model's internal confidence hierarchy as a filter for exploration targets, which had not been proposed or validated in the RLVR context before.

Innovation 3: The Diagnostic Concept of "Reasoning Sparks" and Its Empirical Validation

The paper does not merely propose a method — it introduces a diagnostic category (reasoning sparks) and provides systematic empirical evidence for its existence, functional role, and vulnerability to RLVR training. This is a contribution to our understanding of how language models explore, independent of the proposed solution.

Prior work had observed that entropy collapses during RLVR training and that exploration suffers, but had not characterized what specifically is lost. The paper's analysis in Section 6 and Appendix B provides this characterization through multiple complementary analyses:

• Aggregated probability density evolution (Figures 1c, 1d, 17, 18): showing that the sampling distributions of tokens like "wait" and "however" shift from broad (including low-probability samples) to narrow (only high-probability samples) under GRPO, while entropy bonuses fail to restore the low-probability tail and instead amplify noise tokens like "cost" and "fine."

• Word cloud analysis (Figure 11): visually demonstrating that high-entropy tokens are dominated by function words and formatting characters, while low-probability tokens include semantically meaningful exploratory markers. This provides a direct, intuitive refutation of entropy-based targeting.

• Probability-entropy scatter plots (Figure 12, 19): showing that under GRPO, exploratory tokens cluster in low-entropy, high-probability regions (they become deterministic patterns rather than exploratory pivots); under GRPO + Entropy Loss, they scatter into extremely high-entropy regions (uncontrolled noise); under Lp-Reg, they occupy a broad, structured range of entropy values (genuine exploration).

• Frequency tracking (Figure 13): demonstrating that Lp-Reg maintains higher usage of exploratory tokens throughout training, confirming that the preserved probability distributions translate into actual behavioral differences.

What makes this a genuine innovation rather than just a description is that it operationalizes a previously vague notion ("the model stops exploring") into a specific, measurable, token-level phenomenon with clear functional implications. The "reasoning spark" concept is falsifiable: if preserving these tokens' low-probability sampling doesn't improve exploration, the hypothesis is wrong. The paper's ablation (Figure 7, comparing low-probability regularization to high-entropy regularization) provides exactly this test, and the result validates the concept — targeting the identified tokens works, targeting high-entropy tokens doesn't.

This diagnostic contribution also explains the paper's seemingly paradoxical finding that GRPO + Entropy Loss collapses faster than standard GRPO (Figure 1b). Under the reasoning sparks framework, this makes sense: the entropy bonus is protecting the wrong tokens. It's amplifying semantically irrelevant noise while doing nothing to restore the specific low-probability reasoning sparks that GRPO eliminated. The extra randomness from the entropy bonus makes the policy gradient updates noisier (since more tokens with random advantages get updated) without improving the model's ability to reconsider its reasoning — a net negative effect.

The broader significance of this diagnostic concept extends beyond the paper's specific method. If reasoning sparks are a general phenomenon — tokens whose low-probability sampling serves an exploratory function — then their elimination may be a bottleneck across many RLVR applications, not just mathematical reasoning. The identification of these tokens as a category enables future work to measure their presence and preservation in other domains, model families, and training algorithms, providing a concrete diagnostic tool for assessing exploration quality that is more informative than aggregate entropy.

This is a conceptual innovation with strong empirical grounding. It's not a method contribution — Lp-Reg is the method — but it's the diagnostic framework that makes the method's design principled rather than ad-hoc, and it provides testable predictions that the paper's experiments confirm.

Innovation 4: Forward KL as a Mode-Dropping Prevention Mechanism Rather Than a Distribution-Matching Objective

KL divergence regularization is common in RL for language models — it's used in the standard GRPO objective (Equation 3, the DKL(πθ ∥ πref) term) to prevent the policy from drifting too far from a reference model, typically the initial pretrained checkpoint. In that role, the KL penalty serves as a global proximity constraint: keep the entire output distribution close to the reference, preventing catastrophic forgetting and reward hacking.

Lp-Reg uses KL divergence in a fundamentally different way that has received little attention in the RLVR literature. The forward KL DKL(πproxy ∥ πθ) is deployed as a selective mode-dropping prevention mechanism: it heavily penalizes the policy only when it assigns near-zero probability to tokens that the proxy distribution considers plausible, but imposes minimal penalty when the policy assigns higher-than-proxy probability to those tokens or when the proxy assigns zero probability to begin with. This asymmetry — which the paper explicitly contrasts with reverse KL in Figure 10 — is what makes the regularization work as exploration protection rather than distribution matching.

The conceptual distinction matters because the two uses of KL have opposite relationships to the reference distribution. Global proximity KL (reverse KL, DKL(πθ ∥ πref)) says "stay close to the reference everywhere" — it penalizes the policy for being more confident than the reference about tokens the reference considers unlikely. This is useful for preventing catastrophic forgetting but counterproductive for exploration: it would penalize the policy for increasing a reasoning spark's probability above the reference model's low baseline. Mode-dropping prevention KL (forward KL, DKL(πproxy ∥ πθ)) says "don't completely ignore tokens the proxy considers plausible, but feel free to adjust their exact probabilities based on reward" — it penalizes only the elimination of tokens, not their up-weighting.

This is a conceptual reframing of how KL divergence can be used in RLVR, from a broad constraint that limits change to a targeted mechanism that prevents specific types of harmful change (mode collapse) while allowing others. The paper's ablation (Figure 10) shows that this distinction is not academic — reverse KL performs substantially worse, likely because it forces the policy to imitate a heuristic proxy distribution rather than using the proxy as a soft guide for which tokens must survive.

The connection to the mode-dropping problem in generative modeling is worth noting, though the paper doesn't make it explicitly. In GAN training, mode collapse refers to the generator producing only a subset of the data distribution's modes. The forward KL divergence's property of penalizing mode-dropping (because zero probability under the generator for a non-zero probability mode under the data distribution yields infinite KL) is well-known in that literature. The paper effectively imports this insight into RLVR: the "modes" are the reasoning sparks, and the "data distribution" is the proxy distribution. This cross-pollination from generative modeling to reinforcement learning for language models is conceptually fertile, though the paper doesn't explore it deeply.

This is an incremental technical refinement in the specific context of Lp-Reg (since KL divergence is already widely used), but it represents a conceptual reframing of what role KL regularization can play in exploration — from a global stability mechanism to a selective anti-extinction mechanism. The empirical validation that forward KL substantially outperforms reverse KL for this purpose (Figure 10) provides the evidence that the distinction matters in practice.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All experiments use the Dapo-Math-17K dataset (Yu et al., 2025) for RLVR training, consisting of 17,000 mathematical reasoning problems with verifiable answers. The dataset was chosen because math problems provide clean, binary reward signals (correct or incorrect final answer) that enable RLVR without requiring a separate reward model. For evaluation, five benchmarks are used: AIME24 (15 problems), AIME25 (15 problems), MATH-500 (500 problems; Hendrycks et al., 2021), OlympiadBench (He et al., 2024a), and Minerva Math (Lewkowycz et al., 2022).

• Base model(s). Two model scales are tested: Qwen3-14B-Base (Yang et al., 2025) and Qwen2.5-32B-Base. The paper uses these models because Qwen3-14B represents a state-of-the-art base model with strong pre-trained reasoning capacity (making it a good substrate for observing reasoning spark emergence), while Qwen2.5-32B allows testing at larger scale to validate that findings generalize beyond a single model size. Both are pretrained base models, not instruction-tuned, so their reasoning behaviors emerge purely from RLVR training rather than being pre-shaped by supervised fine-tuning.

• Metrics. The primary metric is average test accuracy (%) — the fraction of problems for which the model's final answer matches the ground truth. For AIME24 and AIME25 (small test sets), accuracy is computed using sampled decoding at temperature 0.6 with 16 independent responses per problem and majority voting to obtain a robust estimate. For MATH-500, OlympiadBench, and Minerva Math (larger test sets), greedy decoding is used. The paper reports the best accuracy achieved during the entire training run for each method rather than final accuracy, which accounts for performance saturation and collapse patterns differently across methods. Training metrics include policy entropy (computed over the token-level probability distributions during generation) and response length (the average number of tokens in generated outputs).

• Baselines. The paper compares against six methods:

  • GRPO (Shao et al., 2024a): the standard Group Relative Policy Optimization algorithm with symmetric PPO clipping, serving as the foundational baseline. Evaluated in both on-policy and off-policy modes.
  • GRPO + Entropy Loss: GRPO augmented with a Maximum Entropy RL-style policy entropy bonus (coefficient = 0.002), representing classical entropy regularization.
  • GSPO (Zheng et al., 2025): Group Sequence Policy Optimization, which modifies clipping to operate at the sequence level to promote higher training entropy. Off-policy only.
  • Clip-Higher (Yu et al., 2025): a core component of DAPO that uses asymmetric PPO clipping with a raised upper bound to encourage higher entropy by allowing probability increases for high-advantage tokens. Off-policy only.
  • KL-Cov (Cui et al., 2025): applies a selective KL-divergence penalty to tokens with the highest covariance between their log probabilities and advantages, aiming to prevent entropy collapse. Off-policy only.
  • 80/20 (Selective High-Entropy Training; Wang et al., 2025): restricts policy gradient updates to only the top 20% of tokens with the highest generation entropy, representing selective entropy-based exploration. Evaluated in both on-policy and off-policy modes.

• Generation budget / compute accounting. The primary unit of compute is training steps, where one "step" corresponds to a single rollout (generating G = 8 outputs for each prompt in a global batch of 256 prompts). For off-policy methods, each rollout is followed by 8 gradient updates (mini-batch size 32 on the 256-prompt batch), but the paper's x-axes consistently count rollout steps, not gradient updates. This ensures fair comparison between on-policy and off-policy methods despite their different update frequencies. Total compute is reported in GPU-hours: approximately 8,000 GPU-hours on 32 NVIDIA H20 GPUs for 1,000 Qwen3-14B steps, 16,000 GPU-hours on 64 H20 GPUs for 800 Qwen2.5-32B steps, and 81,204 GPU-hours for the 3,000-step continuous scaling experiment.

• Cross-validation / statistical protocol. The paper does not use cross-validation for model selection or hyperparameter tuning. Instead, it explicitly trains all methods until performance saturation (approximately 1,000 steps for Qwen3-14B, 800 for Qwen2.5-32B) to avoid unfair comparisons from different convergence speeds. For methods that experience training collapse (defined as a >10% accuracy drop), training was stopped early to conserve resources, and the best accuracy achieved before collapse is reported. This protocol means methods that are stable but converge slowly are not penalized relative to methods that peak early and then collapse. Hyperparameters for all baseline methods are taken directly from their original public implementations, and Lp-Reg's hyperparameters (ρ, κ, β) are set based on the model scale without per-benchmark tuning. Sensitivity analysis in Appendix A confirms that Lp-Reg is robust to these choices across reasonable ranges.

Main Quantitative Results

Continuous Scaling Stability (Figure 3)

The paper's headline stability result is that Lp-Reg (on-policy) enables continuous reinforcement learning scaling for 3,000 training steps on Qwen2.5-32B-Base, totaling 81,204 GPU-hours, without performance collapse. Figure 3 tracks three metrics across these 3,000 steps:

• Average test accuracy climbs from approximately 0.20 at initialization to approximately 0.58 by step 500, then continues a slow upward trend reaching approximately 0.60 by step 3,000, with no visible degradation. This contrasts sharply with all baseline entropy-control methods, which collapse well before 1,000 steps (as shown in the main comparison figures).
• Training entropy decreases from approximately 0.65 to approximately 0.30 by step 500, then gradually recovers and stabilizes around 0.35–0.40 for the remaining 2,500 steps. This multi-phase trajectory — initial decrease (learning core patterns), subsequent recovery (exploration resumes as accuracy improves), and final stabilization — is identified by the paper as a signature of healthy exploration-exploitation balance unique to Lp-Reg.
• Response length grows from approximately 1,000 tokens at initialization to roughly 8,000 tokens by step 1,000, then remains stable between 7,000–8,000 tokens for the remaining 2,000 steps. This sustained long-response generation indicates the model maintains extended chain-of-thought reasoning rather than collapsing to short, superficial answers.

No other method in the paper is evaluated beyond approximately 1,000 steps, so Lp-Reg's stability at 3,000 steps represents a qualitative threshold — the paper demonstrates that Lp-Reg can operate in a regime where all compared methods have already failed. However, the paper does not report whether any baseline methods were attempted at 3,000 steps and collapsed (beyond the early stopping criterion), so the direct head-to-head comparison exists only at ≤1,000 steps.

Main Benchmark Results (Table 1, Figures 4, 5, 6)

Qwen3-14B-Base results (1,000 training steps). The on-policy Lp-Reg achieves the highest average accuracy of 60.17% across all five benchmarks, representing an improvement of +2.66% over the next best method (80/20 on-policy at 57.51%) and +2.74% over the best off-policy method (KL-Cov at 57.29%). The per-benchmark breakdown reveals where Lp-Reg's advantage concentrates:

• AIME24: Lp-Reg (on.) achieves 50.83%, compared to 47.29% for 80/20 (on.), 49.17% for KL-Cov (off.), and 46.04% for GRPO (on.). This +3.54% gap over the next best on-policy method is substantial on a 15-problem benchmark.
• AIME25: Lp-Reg (on.) achieves 37.92%, compared to 34.58% for GSPO (off.) and 34.38% for GRPO (on.). The 3.34% absolute improvement is again concentrated on the hardest benchmark.
• MATH-500: Lp-Reg (on.) achieves 94.40%, compared to 93.00% for GRPO (on.) and 95.00% for Clip-Higher (off.). This is the one benchmark where Lp-Reg is not the top performer — Clip-Higher edges it by 0.60% — but it still exceeds most baselines.
• Minerva Math: Lp-Reg (on.) achieves 49.26%, compared to 50.74% for GSPO (off.) and 50.37% for 80/20 (on.). Lp-Reg is slightly below the top methods on this benchmark.
• OlympiadBench: Lp-Reg (on.) achieves 68.44%, compared to 65.78% for 80/20 (on.) and 65.19% for GRPO (on.). This +2.66% gap is the largest single-benchmark improvement among on-policy methods.

The pattern across benchmarks is notable: Lp-Reg's advantage is largest on the hardest benchmarks (AIME24, AIME25, OlympiadBench) and smallest or negligible on the benchmarks where performance is already high (MATH-500 at 94.40%) or where the task may depend less on exploratory reasoning patterns. This aligns with the paper's conceptual framework — reasoning sparks matter most when the model needs to reconsider and pivot, which is more common on difficult problems.

On-policy vs. off-policy comparison. Lp-Reg (on.) at 60.17% outperforms Lp-Reg (off.) at 57.08% by +3.09%, and this on-policy advantage is consistent with the broader pattern in Table 1. Across all methods that are evaluated in both regimes (GRPO, 80/20, Lp-Reg), on-policy training consistently outperforms off-policy: GRPO (on.) 57.43% vs. GRPO (off.) 51.13% (+6.30%); 80/20 (on.) 57.51% vs. 80/20 (off.) 55.88% (+1.63%). The paper attributes this to "the inherent stability of on-policy updates, which avoid distribution shifts caused by mismatched data-sampling and training policies." However, the magnitude of the on-policy advantage varies substantially across methods — Lp-Reg gains +3.09% while GRPO gains +6.30%, suggesting that GRPO's off-policy importance sampling corrections introduce instability that Lp-Reg's self-contained regularization partially mitigates even in the off-policy setting.

Qwen2.5-32B-Base results (800 training steps). The pattern differs from the Qwen3-14B results in important ways:

• Lp-Reg (on.) achieves 52.54% average accuracy, compared to 51.49% for 80/20 (off.), 51.48% for Clip-Higher (off.), and 50.73% for 80/20 (on.). The gap between Lp-Reg and the next best method is only +1.05%, substantially smaller than the +2.66% on Qwen3-14B.
• On AIME24: Lp-Reg (on.) achieves 38.12%, tied with 80/20 (off.) at 38.12% and slightly behind Clip-Higher (off.) at 38.33%.
• On AIME25: Lp-Reg (on.) achieves 27.08%, behind Clip-Higher (off.) at 29.79% and 80/20 (off.) at 28.75%.
• On MATH-500: Lp-Reg (off.) achieves 90.20%, the highest across all methods (the next best is Lp-Reg (on.) at 90.00% and 80/20 (on.) at 89.40%).
• On OlympiadBench: Lp-Reg (on.) achieves 61.19%, compared to 60.30% for GRPO (on.) and 58.37% for 80/20 (off.).

The paper explicitly notes this difference between model scales:

"Lp-Reg's advantage is more pronounced on the newer Qwen3-14B base model compared to the older Qwen2.5-32B in the first 1,000 training steps. We hypothesize that as base models improve, their capacity for nuanced, low-probability reasoning increases, creating a richer substrate for the emergence of valuable reasoning sparks, which Lp-Reg can then effectively protect and leverage."

This hypothesis is plausible but untested — the paper does not provide a mechanistic analysis of why Qwen3-14B would have more reasoning sparks than Qwen2.5-32B, nor does it compare the probability distributions of reasoning sparks between the two base models before training. The observation that Lp-Reg's advantage is larger on the newer model is interesting but remains a post-hoc interpretation rather than a validated finding.

Training dynamics (Figures 4, 5, 6). The training curves reveal how different methods evolve over time:

• GRPO (on.) on Qwen3-14B (Figure 4, right panel): accuracy rises from approximately 0.47 to a peak of approximately 0.58 around step 200, then declines gradually to approximately 0.56 by step 600, after which it stabilizes with slight degradation. This matches the classic "plateau then decline" pattern the paper identifies as exploration collapse.
• GRPO + Entropy Loss (on.): accuracy climbs to approximately 0.53 around step 100, then crashes spectacularly to approximately 0.48 by step 200 — below its starting point. This is the fastest collapse among all methods, visually confirming the paper's claim that indiscriminate entropy boosting is worse than doing nothing.
• Clip-Higher (off.): accuracy climbs steadily to approximately 0.57 by step 400, plateaus, then shows high variance with peaks around 0.58 and drops to 0.55, indicating instability. On Qwen2.5-32B (Figure 6), Clip-Higher climbs to approximately 0.53 by step 200 and maintains roughly that level, but its entropy (Figure 5, left panel) shows a continuous artificial increase — the paper identifies this as an undesirable signature of forced randomness rather than genuine exploration.
• 80/20 (on.): on Qwen3-14B, accuracy climbs more gradually than Lp-Reg, reaching approximately 0.57 by step 400 and roughly matching GRPO (on.) through much of training. Its entropy trajectory (Figure 5, middle panel) remains low and stable, similar to GRPO. On Qwen2.5-32B (Figure 6), 80/20 (on.) shows a nearly identical accuracy curve to Lp-Reg (on.), which explains the narrow final gap (52.54% vs. 50.73%).
• Lp-Reg (on.): on Qwen3-14B, accuracy climbs rapidly to approximately 0.57 by step 200, continues improving to approximately 0.60 by step 400, and maintains roughly 0.59–0.60 for the remaining 600 steps with remarkably low variance compared to other methods. On Qwen2.5-32B (Figure 6), Lp-Reg (on.)'s accuracy curve is nearly indistinguishable from 80/20 (on.) for the first 400 steps, then separates slightly in the 400–800 range, finishing at 52.54% vs. 50.73%.

Entropy trajectory comparison (Figure 5). The middle panel of Figure 5 comparing Clip-Higher, GRPO (on.), and Lp-Reg (on.) is particularly informative:

• Clip-Higher (off.): entropy starts around 0.2, then increases continuously to approximately 0.5 by step 800 — a steady, monotonic upward trend that the paper characterizes as "artificial" because it's driven by the asymmetric clipping mechanism rather than by genuine exploration needs.
• GRPO (on.): entropy starts around 0.5, drops rapidly to approximately 0.2 by step 200, then continues declining to near 0.1 by step 800. This is the classic entropy collapse pattern.
• Lp-Reg (on.): entropy starts around 0.5, drops to approximately 0.3 by step 200 (as the model learns core patterns), then gradually increases to approximately 0.35–0.40 by step 800. This non-monotonic U-shaped trajectory — decrease, then increase, then stabilization — is unique to Lp-Reg and the paper presents it as evidence that Lp-Reg achieves "a distinct entropy signature indicative of a healthy exploration-exploitation balance."

Response length comparison (Figure 5). Lp-Reg's outputs are consistently longer than GRPO's: starting from approximately 2,000 tokens and growing to 6,000–7,000 by step 400, compared to GRPO's growth to approximately 4,000–5,000 before declining. This is relevant because longer responses in mathematical reasoning typically indicate more detailed chain-of-thought, suggesting Lp-Reg is not merely preserving exploration but also sustaining the model's willingness to generate extended reasoning. Clip-Higher produces the longest responses (peaking at approximately 8,000 tokens), which is consistent with its entropy-boosting mechanism but does not translate to better accuracy, again demonstrating that quantity of output (like quantity of entropy) does not equate to quality of reasoning.

Analysis of Exploration Dynamics (Figures 11, 12, 13, 14)

Beyond benchmark performance, the paper provides diagnostic analyses that characterize how Lp-Reg preserves exploration:

Probability-entropy distribution of exploratory tokens (Figure 12). This scatter plot shows the joint distribution of sampled probability (x-axis) and per-token entropy (y-axis) for five exploratory tokens ("but," "wait," "perhaps," "alternatively," "however") at training step 256 on Qwen3-14B:

• GRPO: n = 42,023 samples; mean probability = 0.91 ± 0.20; mean entropy = 0.16 ± 0.27. These tokens are overwhelmingly sampled at high probability and low entropy — they have become deterministic patterns.
• GRPO + Entropy Loss: n = 34,470 samples; mean probability = 0.46 ± 0.32; mean entropy = 1.60 ± 1.47. The distribution scatters widely, with many samples at extremely high entropy (some exceeding 10) — uncontrolled noise.
• Lp-Reg: n = 74,079 samples; mean probability = 0.72 ± 0.31; mean entropy = 0.62 ± 0.59. The distribution is broad but structured, spanning a healthy range of probability and entropy values — genuine exploration.

The much larger sample count for Lp-Reg (74,079 vs. 42,023 for GRPO) indicates these tokens are actually being used more frequently, not just preserved as low-probability options. The individual token-level breakdowns in Appendix B.2 (Figure 19a–e) confirm this pattern is consistent across all five tokens, not driven by a single outlier.

Frequency tracking during training (Figure 13). Across training steps 256, 512, and 768, Lp-Reg maintains consistently higher frequency of exploratory tokens than GRPO. For example, at step 256, "but" appears at ~0.0035 frequency under Lp-Reg vs. ~0.0020 under GRPO; by step 768, Lp-Reg's frequency has dropped to ~0.0025 while GRPO's has collapsed to ~0.0010. This confirms that the preserved probability distributions translate into actual differences in token usage throughout training.

Probabilistic distinction between reasoning sparks and noise (Figure 14). This is the foundational empirical finding that justifies Lp-Reg's filtering design. Analyzing the top-64 most probable tokens within the low-probability range (0 to 0.1) across training steps 50–400 under standard GRPO training, the paper tracks the mean probability of two token classes:

• Exploratory tokens ("wait," "perhaps," etc.): mean probability starts at ~0.007 at step 50 and decreases gradually to ~0.004 by step 400.
• Irrelevant tokens ("cost," "fine," etc.): mean probability starts at ~0.004 at step 50 and decreases to ~0.003 by step 400.

The key finding is that exploratory tokens' mean probability is consistently higher than irrelevant tokens' across all training stages — the gap narrows over time but never closes. The paper interprets this as evidence that the model's intrinsic confidence systematically distinguishes meaningful alternatives from noise even within the low-probability tail, providing a principled basis for threshold-based filtering. An important caveat: this analysis is limited to the top-64 most probable tokens (due to storage limitations), so it captures the upper portion of the low-probability range but not the full extreme tail. Tokens in positions 65+ in the sorted probability distribution are not analyzed, and these might include additional reasoning sparks or noise that could alter the picture.

Word cloud comparison (Figure 11). A qualitative complement to the quantitative analyses: the top 1% lowest-probability tokens include semantically meaningful markers like "But," "Wait," "Perhaps," "Alternatively," "_Wait" (underscore prefix variant), and "Let" — tokens that signal reconsideration or branching. The top 1% highest-entropy tokens are dominated by function words ("sqrt," "times," "frac," "and") and formatting characters ("\n," "_"), with no clear exploratory function. This visualization makes intuitively clear why entropy-based targeting fails: even a perfect entropy-based method would primarily regularize tokens that are common and functional, missing the specific low-probability tokens that enable exploration.

Ablation Studies and Robustness Checks

Low-probability tokens vs. high-entropy tokens (Figure 7): The paper's central claim that low-probability tokens (not high-entropy tokens) are the correct target for exploration preservation is tested by applying the identical Lp-Reg regularization mechanism (forward KL, proxy distribution, same hyperparameters) to the top 1% highest-entropy tokens instead of the bottom 1% lowest-probability tokens. This "Highest H Regularization" variant fails: accuracy initially rises similarly to standard Lp-Reg but then collapses around step 300, falling from ~0.55 to ~0.48 by step 500 — worse than the no-regularization GRPO baseline. Entropy spikes to approximately 1.0 at step 100, then crashes to near zero by step 400. Response length drops from 7,000+ tokens to ~2,000. This ablation directly validates the paper's claim that high-entropy regularization "pollutes the learning signal without protecting the structured, low-probability reasoning paths necessary for progress." It also demonstrates that the benefits of Lp-Reg are specific to targeting low-probability tokens, not just any form of selective regularization — the mechanism matters, not just the selectivity.

Noise filtering necessity (Figure 8): Three configurations are compared on Qwen3-14B:

• Lp-Reg w/o τ: removing the noise filter entirely (all low-probability tokens get protected regardless of proxy distribution status). This causes catastrophic failure: entropy explodes from ~0.5 to ~1.4 by step 100, accuracy crashes from ~0.55 to ~0.25 by step 300, and response length plummets. This is the most dramatic ablation result in the paper, visually confirming that filtering is not just beneficial but necessary — without it, Lp-Reg is actively harmful.
• Lp-Reg w/ fixed τ: using a constant threshold τ = 0.02 instead of the dynamic min-p threshold. Accuracy reaches approximately 0.57 by step 600, compared to ~0.60 for dynamic τ and ~0.55 for GRPO. Entropy stabilizes around 0.3–0.4, comparable to dynamic τ. The paper concludes that "while the core filtering principle is effective, the dynamic nature of min-p provides a more robust estimate of the model's confidence across different contexts, better preserving genuine reasoning sparks."
• Lp-Reg w/ dynamic τ: the default configuration described in Section 4, achieving the best performance.

The ordering (dynamic > fixed > none) confirms that both the existence of filtering and its adaptiveness matter, but the gap between fixed and dynamic is modest (~0.03 accuracy), suggesting the filtering principle is the primary driver and adaptiveness provides incremental benefit.

Negative sample regularization (Figure 9): Three conditions are compared:

• Regularize None: standard GRPO, achieving approximately 0.55 accuracy by step 1,000.
• Regularize Neg.: Lp-Reg penalty applied only to tokens with Ai,t < 0 (the default Lp-Reg configuration). Accuracy reaches approximately 0.60, with faster learning in the 200–600 step range compared to Regularize Pos. & Neg.
• Regularize Pos. & Neg.: Lp-Reg penalty applied to all low-probability tokens regardless of advantage sign. Accuracy also reaches approximately 0.60 by step 1,000, but the trajectory is slightly slower — at step 400, Regularize Neg. is at ~0.59 while Regularize Pos. & Neg. is at ~0.57. Entropy trajectories are similar.

The paper interprets this as evidence that "concentrating the regularization solely on negative samples leads to a faster learning rate" because "once [reasoning sparks in negative samples] are protected, extending the penalty to positive samples provides little additional exploratory advantage" and "introduces minor noise into the gradient updates." Both configurations substantially outperform no regularization, confirming that protecting low-probability tokens per se is beneficial, but negative-only targeting is more efficient.

Forward KL vs. Reverse KL (Figure 10): The forward KL DKL(πproxy ∥ πθ) achieves approximately 0.60 accuracy and stable training, while the reverse KL DKL(πθ ∥ πproxy) causes training to collapse around step 300, with accuracy dropping from ~0.55 to ~0.35 by step 500 and entropy plunging to near zero. This is a striking difference — swapping the order of arguments in the KL divergence entirely changes the method from beneficial to harmful. The paper attributes this to the reverse KL "forcing the policy to strictly imitate this non-ideal, heuristic target" which "constrains the protection of potentially valuable exploratory tokens" because it penalizes deviations from the proxy in both directions (both lowering and raising probabilities relative to the proxy). The forward KL only penalizes lowering probabilities below the proxy, allowing the policy to increase probabilities of protected tokens based on reward signal.

Hyperparameter sensitivity (Appendix A, Figure 15):

• Percentile threshold ρ (Figure 15a): values of 0.005, 0.010, and 0.015 produce "comparable" training trajectories and final performance, all reaching approximately 0.58–0.60 accuracy by step 600 with similar entropy and response length dynamics. This robustness is important because ρ determines how many tokens get protected — the insensitivity suggests the method works across a 3× range of this parameter.
• Min-p ratio κ (Figure 15b): values of 0.01, 0.02, and 0.03 show "a similar trend of stability" with "consistently high and tightly clustered" final performance, all reaching approximately 0.58–0.60 accuracy. This 3× range covers noise thresholds from 1% to 3% of the maximum token probability.

The paper notes that Lp-Reg's "effectiveness is not contingent on extensive, fine-grained hyperparameter tuning, highlighting its practical applicability." This is a meaningful practical claim, though the tested ranges are relatively narrow (ρ ∈ {0.005, 0.010, 0.015}, κ ∈ {0.01, 0.02, 0.03}) — performance might degrade outside these ranges, but the insensitivity within these ranges is convincingly demonstrated.

Training dynamics of regularized tokens (Appendix B.3, Figure 16): The probability threshold δBρ (the batch-level percentile threshold from the Lp-Reg activation condition) decreases from approximately 0.075 at step 128 to approximately 0.025 by step 768, reflecting that the model becomes more confident overall and the "low-probability" boundary shifts downward. Simultaneously, the regularization ratio — the fraction of below-threshold tokens that survive the proxy filter (πproxy > 0) and actually get regularized — decreases from approximately 26% at step 128 to approximately 8% by step 768. The paper interprets this decline as evidence that "as training progresses, the extreme low-probability range becomes increasingly dominated by irrelevant tokens, constituting what we term irrelevant noise" while "semantically meaningful exploratory tokens are lifted into higher-probability regions, thus requiring less regularization." This is a self-limiting property: Lp-Reg's intervention naturally decreases as training progresses and the policy learns to maintain exploration on its own.

Critical Assessment

Does Lp-Reg genuinely sustain exploration, or does it merely prevent entropy from dropping?

The paper's central claim is that Lp-Reg sustains meaningful exploration, not just randomness. The evidence supporting this is multifaceted but relies heavily on qualitative and correlational analyses. The probability-entropy scatter plots (Figure 12) show that exploratory tokens occupy a broader, more structured distribution under Lp-Reg than under baselines, and the frequency tracking (Figure 13) confirms these tokens are actually used more often. The word cloud (Figure 11) demonstrates that low-probability tokens include semantically meaningful exploratory markers while high-entropy tokens do not. Together, these analyses paint a coherent picture: Lp-Reg preserves tokens that look like they should enable exploration, and these tokens are indeed sampled more frequently and in more diverse contexts.

However, the paper does not provide a direct causal demonstration that preserved reasoning sparks cause better exploration which causes better accuracy. The evidence is: (a) Lp-Reg preserves reasoning sparks (Figures 12, 13, 17), (b) Lp-Reg achieves better accuracy (Table 1), (c) removing protection causes collapse (Figure 8, Lp-Reg w/o τ). This is a classic A→B and A→C, therefore B→C inference pattern, which is suggestive but not definitive. Alternative explanations are possible: perhaps Lp-Reg's regularization has a beneficial optimization effect (smoothing the loss landscape, preventing sharp gradient updates) that is independent of whether it preserves exploratory tokens specifically. The ablation comparing low-probability vs. high-entropy regularization (Figure 7) partially addresses this — if the benefit were purely from regularization smoothness, high-entropy regularization should also help — but the catastrophic failure of high-entropy regularization could be due to the specific tokens it targets being actively harmful rather than because low-probability tokens are specifically beneficial. A stronger test would be: manually remove reasoning sparks from the model's outputs (e.g., by masking them during generation) and show that accuracy degrades even with Lp-Reg, or conversely, manually inject reasoning sparks into a baseline model's outputs and show improved exploration.

The paper's analysis in Figure 14 (probabilistic distinction between exploratory and irrelevant tokens) provides the mechanistic basis for why filtering works, but this analysis is limited in scope: only the top-64 tokens are examined, only within probability range [0, 0.1], and only for a specific set of five pre-selected exploratory tokens and five pre-selected irrelevant tokens. The generalizability of this probabilistic gap to the full vocabulary and full probability range is assumed but not demonstrated. If the gap narrows or reverses for other tokens or probability ranges, the filtering mechanism might be less principled than the paper suggests.

Does the 2.66% improvement represent a meaningful advance or an incremental gain?

The +2.66% improvement over the next best method (80/20 on-policy at 57.51% vs. Lp-Reg at 60.17%) on Qwen3-14B is statistically meaningful but modest in absolute terms. The paper does not report confidence intervals, standard deviations, or statistical significance tests for any benchmark results, making it impossible to assess whether this difference could arise from random variation. Given that AIME24 and AIME25 have only 15 problems each, a difference of 3–4 percentage points on these benchmarks corresponds to less than one problem — a single additional correct answer could flip the ranking. The larger benchmarks (MATH-500 with 500 problems, OlympiadBench) provide more stable estimates, but the paper's practice of reporting only the single best accuracy during training (not the final accuracy or an average over the last N checkpoints) introduces potential selection bias: methods with higher variance might have higher "best" scores by chance.

The paper's claim of state-of-the-art performance is qualified by the specific model scale (14B and 32B) and the specific baselines compared. The paper does not compare against methods that combine multiple exploration strategies or that use substantially more compute — the 3,000-step continuous scaling experiment (Figure 3) shows Lp-Reg continuing to improve slowly, but no baseline is run for 3,000 steps to see if they would also improve given enough compute (they likely would not, given the collapse patterns, but this is unverified).

Is the continuous scaling result (3,000 steps) genuinely due to sustained exploration, or could it be achieved by other methods given the same compute?

The 3,000-step, 81,204 GPU-hour training run (Figure 3) is presented as evidence that Lp-Reg enables continuous scaling where baselines collapse. However:

• No baseline is run for 3,000 steps. The paper's early stopping criterion (>10% accuracy drop) means baselines that collapsed at 400–600 steps were terminated. It's possible — though unlikely given the collapse trajectories — that some baselines would recover if allowed to continue training through the collapse.
• The accuracy improvement from step 500 (~0.58) to step 3,000 (~0.60) is only ~0.02 over 2,500 steps — a very slow rate of improvement. Whether this counts as "continuous scaling" or "performance saturation with minor fluctuation" is a matter of interpretation. The paper does not provide a trend line or statistical test for whether the accuracy is genuinely increasing over steps 500–3,000 versus fluctuating around a plateau.
• The computational cost of 81,204 GPU-hours for a ~0.02 improvement is enormous. The paper does not discuss cost-effectiveness — whether the same compute invested in other approaches (larger models, more data, different algorithms) would yield larger gains. This is a significant omission for a paper that makes claims about scalability.

Does the on-policy advantage reflect a genuine property of Lp-Reg or a confounding factor?

Lp-Reg (on.) outperforms Lp-Reg (off.) by +3.09% on Qwen3-14B, but GRPO (on.) outperforms GRPO (off.) by +6.30% — the on-policy advantage is actually larger for the baseline method. This suggests that on-policy training itself provides a substantial benefit independent of Lp-Reg, and Lp-Reg's advantage over baselines in the on-policy setting (+2.66% over 80/20 on.) is partly because it handles on-policy training well while some baselines (Clip-Higher, GSPO, KL-Cov) don't support on-policy training at all. The paper's claim that Lp-Reg's "self-contained, policy-intrinsic regularization ensures its effectiveness in both on-policy and off-policy settings" is supported by Lp-Reg being the best off-policy method on Qwen3-14B (57.08% vs. 57.29% for KL-Cov, a 0.21% difference, essentially tied), but the off-policy gap between methods is small and the ranking differs between model scales (on Qwen2.5-32B, Clip-Higher at 51.48% edges Lp-Reg off-policy at 50.60%).

What experiments would strengthen the paper's claims?

Several experiments are notably absent:

1. Direct measurement of exploration diversity: The paper shows that reasoning sparks are preserved but never directly measures whether this preservation leads to more diverse reasoning paths. Metrics like the number of distinct solution strategies generated, the variance in final answers before majority voting, or the diversity of intermediate reasoning steps would provide direct evidence that exploration quality improved, not just that specific tokens survived.

2. Comparison against a properly tuned entropy regularization baseline: The GRPO + Entropy Loss baseline uses a single entropy coefficient (0.002). If this coefficient is poorly chosen, the baseline's collapse might reflect poor hyperparameter selection rather than an inherent flaw of entropy regularization. A sweep over entropy coefficients to find the best possible entropy-regularized baseline would strengthen the claim that entropy boosting is fundamentally inferior to Lp-Reg.

3. Stress-test with different reward sparsity: The Dapo-Math-17K dataset provides binary rewards (correct/incorrect) for each problem. If rewards were sparser (e.g., only provided every N problems) or denser (e.g., step-level rewards from a PRM), would Lp-Reg's advantage change? This would test whether Lp-Reg's benefit is specific to the binary reward structure or generalizes to other RLVR settings.

4. Transfer to non-mathematical reasoning domains: All experiments are on mathematical reasoning. The reasoning sparks concept should apply to any domain where reconsideration and branching are valuable — code generation, logical reasoning, multi-step planning. Without evidence from other domains, the generality of the approach remains speculative.

5. Larger-scale comparison: The paper shows results up to 32B parameters. Whether Lp-Reg's benefits persist, diminish, or amplify at larger scales (70B, 405B) is unknown. The hypothesis that "as base models improve, their capacity for nuanced, low-probability reasoning increases" would predict larger benefits at larger scales, but this is untested.

6. Ablation of the proxy distribution construction method: The proxy distribution uses min-p filtering and renormalization. Would a different filtering method (e.g., top-K, nucleus sampling) work equally well? The paper tests fixed vs. dynamic threshold but does not explore alternative filtering paradigms. This matters because min-p filtering is just one of many ways to separate "plausible" from "implausible" tokens — the paper's claim that the model's confidence provides a principled basis for filtering would be stronger if multiple confidence-based filtering methods all worked.

Summary of evidential support

The paper's central claims are supported but with important caveats:

• Lp-Reg sustains exploration by protecting reasoning sparks: Supported by correlated evidence (Figures 12, 13, 17) and ablation (Figure 7), but lacks direct causal demonstration that preserved tokens cause improved exploration diversity.
• Lp-Reg outperforms entropy-control baselines: Supported by Table 1 (+2.66% on Qwen3-14B), but the margin is modest, statistical significance is unreported, and the comparison omits hyperparameter-optimized entropy baselines.
• Lp-Reg enables continuous scaling where baselines collapse: Supported for Lp-Reg (Figure 3), but no baseline is run at comparable scale, and the accuracy gain from 500–3,000 steps is marginal (~0.02).
• The probabilistic gap between reasoning sparks and noise justifies filtering: Supported within the analyzed scope (Figure 14), but the analysis is limited to top-64 tokens and pre-selected token sets, and generalizability to the full distribution is assumed.
• On-policy Lp-Reg is the best overall method: Supported for Qwen3-14B and Qwen2.5-32B (Table 1), but the on-policy advantage is confounded with baseline training stability and the ranking differs across model scales and benchmarks.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Not Included in Any Efficiency Calculation

The assumption or constraint. The entire Lp-Reg mechanism depends on constructing a proxy distribution πproxy that filters tokens based on their probability relative to a threshold τ. For min-p filtering, this threshold is τ = κ · maxo'∈V πθ(o'|·), which requires computing πθ for the entire vocabulary at every token position to identify the maximum probability and determine which tokens fall below the threshold. The paper does not account for this computational cost in any efficiency metric, nor does it compare the wall-clock overhead of Lp-Reg against baseline methods. The proxy distribution construction, KL divergence computation, and three-condition gating logic all add per-token computation beyond standard GRPO. Section 5.1.2 reports total GPU-hours for training runs (8,000 GPU-hours for Qwen3-14B, 16,000 for Qwen2.5-32B, 81,204 for the 3,000-step scaling experiment) but never disaggregates how much of this compute is baseline GRPO versus Lp-Reg overhead, and never reports tokens-per-second or step-time comparisons.

The consequence. A practitioner evaluating whether to adopt Lp-Reg cannot determine its throughput cost. If the proxy distribution construction and KL penalty computation add, say, 20% overhead per training step, then the reported gains must be discounted — a method that achieves +2.66% accuracy at +20% compute cost is less impressive than one that achieves it at cost parity. More subtly, the dynamic batch-level threshold δBρ requires collecting sampling probabilities for all tokens in the current batch and computing a percentile — an operation that scales with batch size and sequence length. For the 3,000-step continuous scaling experiment (81,204 GPU-hours), even a modest per-step overhead compounds to thousands of additional GPU-hours that are attributed to Lp-Reg's "success" rather than to its overhead.

What evidence exists in the paper. The paper provides no wall-clock time measurements, no throughput comparisons, and no FLOPs or memory analyses. The training dynamics figures (Figures 3, 4, 5, 6, 7, 8, 9, 10) all use "Training Steps" on the x-axis, which treats each step as equivalent across methods regardless of per-step computational cost. The fact that Lp-Reg is compared against methods that add their own overhead (entropy bonus computation, selective token filtering, covariance-based KL penalties) partially mitigates this concern — Clip-Higher's asymmetric clipping, KL-Cov's covariance computation, and 80/20's entropy sorting all add compute — but the paper makes no attempt to quantify relative overheads, so relative comparisons are incomplete.

Mitigation status. The paper does not address this at all. There is no discussion of implementation efficiency, no profiling of the proxy distribution construction, and no suggestion that the overhead could be reduced through approximation, caching, or compute-sharing with the GRPO forward pass. Given that the proxy distribution is constructed from the data-generating policy's output distribution (πθold in off-policy mode, πθ in on-policy mode), and those probabilities are already computed during generation, the filtering and renormalization steps might be cheap relative to the transformer forward pass itself — but the paper provides no evidence either way. This is a notable omission for a method paper that makes practical deployment claims.

---

The Reasoning Sparks Framework Is Empirically Defined Only for a Hand-Selected Set of Five Tokens

The assumption or constraint. The paper's diagnostic framework hinges on a specific set of tokens identified as "reasoning sparks": "but," "wait," "perhaps," "alternatively," and "however." These five tokens are used throughout the analysis to demonstrate that Lp-Reg preserves meaningful exploration (Figures 1c, 12, 13, 17, 19) while entropy baselines fail. However, the paper never defines a procedure for identifying which tokens count as reasoning sparks beyond these five examples, never provides a complete list of all tokens the method aims to protect, and never demonstrates that the probabilistic distinction between "exploratory" and "irrelevant" tokens (Figure 14) generalizes beyond the two hand-selected groups of five tokens each. The proxy distribution's filtering mechanism (Section 4.1) operates over the entire vocabulary based purely on probability thresholds — not on any semantic identification of reasoning sparks — but the paper's entire narrative about why Lp-Reg works depends on the claim that it preserves a semantically meaningful category of tokens. The connection between "tokens with probability above τ" and "tokens that serve an exploratory function" is asserted but never systematically validated.

The consequence. If the five selected reasoning sparks are not representative of the broader set of tokens that actually drive exploration in mathematical reasoning, then the paper's central diagnostic claim — that Lp-Reg works because it preserves reasoning sparks — may be an artifact of token selection. The proxy distribution filtering might preserve many tokens above τ that are not meaningful explorers, and might filter out some tokens below τ that are. The paper's analysis would still show that the five selected tokens are preserved, but this wouldn't establish that exploration is preserved — it would only establish that the selected tokens happen to have probabilities above τ and therefore survive filtering. A practitioner wanting to extend Lp-Reg to a new domain (e.g., code generation, where the relevant exploratory tokens might be "refactor," "rewrite," "simplify") has no principled way to identify which tokens constitute reasoning sparks in that domain, and no way to verify that τ-based filtering will protect them.

What evidence exists in the paper. The word cloud (Figure 11) provides qualitative evidence that the lowest 1% probability tokens include a broader set of meaningful markers beyond the five selected ("Let," "But," "Wait," "Perhaps," "Alternatively," "_Wait"), but this is a single-snapshot visualization from one training step, not a systematic analysis. The frequency tracking (Figure 13) covers only the five pre-selected tokens. The probability-entropy scatter plots (Figures 12, 19) cover only the five pre-selected tokens. The aggregated violin plots (Figures 1c, 1d) cover the five pre-selected tokens plus five pre-selected irrelevant tokens. The probabilistic distinction plot (Figure 14) uses "a group of meaningful exploratory tokens (e.g., 'wait', 'perhaps') and a group of irrelevant tokens (e.g., 'cost', 'fine')" — the "e.g." is the paper's only acknowledgment that these are examples, but no complete list is provided, no selection criteria are specified, and no analysis of false positives or false negatives in the filtering is attempted.

Mitigation status. The paper acknowledges none of this. The reasoning sparks concept is presented as an established category when it is in fact a small, hand-selected illustrative set. The method itself (Lp-Reg) does not depend on semantic token identification — it filters purely by probability — so this limitation is primarily about the paper's explanation of why the method works, not about whether the method works. However, for practitioners who need to understand the method's domain of applicability and whether it will transfer to their setting, the gap between "filtering by probability preserves tokens above τ" and "this preserves tokens that serve an exploratory function" matters. If the correlation between "probability above τ" and "exploratory function" is specific to these five tokens in this mathematical reasoning dataset, the method might accidentally protect useless tokens in other domains while the paper's diagnostic framework provides false reassurance that exploration is being preserved.

---

The Method Is Validated Only on a Single Task Family (Mathematical Reasoning) with a Single Dataset (Dapo-Math-17K)

The assumption or constraint. All RLVR training experiments use the Dapo-Math-17K dataset (Yu et al., 2025), consisting of 17,000 mathematical reasoning problems with verifiable answers. All five evaluation benchmarks (AIME24, AIME25, MATH-500, OlympiadBench, Minerva Math) are mathematical reasoning benchmarks. The paper makes no attempt to evaluate Lp-Reg on non-mathematical reasoning domains — code generation, logical reasoning, scientific question answering, multi-step planning, or any other task where RLVR with binary reward signals can be applied. The reasoning sparks concept is introduced as a general property of language model exploration ("these tokens... naturally initiate diverse reasoning pathways"), but its empirical validation is confined entirely to mathematics.

The consequence. A practitioner considering Lp-Reg for code generation (where RLVR with unit test verification is increasingly common) or logical reasoning has no evidence that the method transfers. The specific tokens identified as reasoning sparks in the math context ("wait," "however," "perhaps," "alternatively") may not be the relevant exploratory tokens in code generation (where "refactor," "optimize," "alternatively," "instead" might play analogous roles). More fundamentally, the probabilistic gap between exploratory and irrelevant tokens that Lp-Reg relies on (Figure 14) may not exist in other domains — the model's intrinsic confidence calibration might differ for code tokens versus math tokens, changing the effectiveness of probability-based filtering. The paper's hypothesis about Qwen3-14B versus Qwen2.5-32B — that "as base models improve, their capacity for nuanced, low-probability reasoning increases" — implicitly assumes that the reasoning sparks phenomenon generalizes across model scales and capabilities, but this is not tested across task families.

What evidence exists in the paper. None outside of mathematics. The paper cites prior work on intrinsic confidence in LLMs in general terms (Saurav et al., 2022; Nguyen et al., 2025; Fu et al., 2025), suggesting the probabilistic distinction between token classes might be domain-general, but provides no empirical evidence that this is true for RLVR training specifically. The related work section notes that RLVR has been applied to other domains ("code generation, logical reasoning"), and the introduction frames the problem as relevant to "complex reasoning" generally, but the experimental scope does not match this framing.

Mitigation status. The paper does not acknowledge this as a limitation. There is no discussion of domain transfer, no suggestion that math-specific properties (e.g., the formulaic structure of chain-of-thought reasoning in mathematics) might make reasoning sparks more or less important than in other domains, and no future work direction pointing toward multi-domain evaluation. Given that many RLVR applications involve code generation, this is a significant omission — a reader evaluating Lp-Reg for a code-focused training pipeline would have to extrapolate from math results with no principled basis for doing so. The paper's code release (linked in the abstract) might enable reproduction in other domains, but the paper itself provides no guidance on whether to expect the method to work.

---

The 2.66% Improvement Is Not Subjected to Statistical Significance Testing and Uses Best-Checkpoint Selection

The assumption or constraint. The paper reports the best accuracy achieved during the entire training run for each method, not the final accuracy or an average over the last N checkpoints. Section 5.1.2 states: "we trained models until their performance saturated... comparisons are based on approximately 1,000 training steps for the Qwen3-14B-Base model and 800 for the Qwen2.5-32B model." For methods that collapsed (>10% accuracy drop), training was stopped early, and the best pre-collapse accuracy is reported. No standard deviations, confidence intervals, error bars, or statistical significance tests are reported for any of the five benchmark scores. The AIME24 and AIME25 benchmarks have only 15 problems each, making individual scores highly sensitive to a single problem's outcome.

The consequence. The headline claim of "an improvement of 2.66% over prior methods" on Qwen3-14B must be interpreted with caution. If the true performance of Lp-Reg (on.) and 80/20 (on.) were identical at 59.0% with a standard deviation of ±1.5% across different random seeds, a 2.66% gap could arise from sampling noise alone. The per-benchmark breakdown reveals that Lp-Reg's advantage is concentrated on specific benchmarks: +3.54% on AIME24 (15 problems — this corresponds to approximately 0.5 more correct problems out of 15), +3.34% on AIME25 (15 problems — approximately 0.5 more correct problems), -0.60% on MATH-500, -1.48% on Minerva, +2.66% on OlympiadBench. The AIME benchmarks' small size makes the per-benchmark differences on the hardest problems particularly noisy: a single additional correct AIME24 answer changes accuracy by 6.67 percentage points. The best-checkpoint selection protocol introduces additional variance: methods with more volatile training trajectories (like Clip-Higher, which shows significant accuracy fluctuations in Figure 5) might have higher "best" scores due to lucky peaks, while more stable methods (like Lp-Reg) have "best" scores closer to their typical performance.

What evidence exists in the paper. The paper provides no error estimates of any kind. The training curves (Figures 4, 5, 6, 7, 8, 9, 10) show single trajectories without shaded regions or error bands, implying (or allowing the reader to infer) that only a single training run was performed per method per model scale. Table 1 reports scalar point estimates without ± notation. The paper's robustness analysis focuses on hyperparameter sensitivity (Appendix A, Figure 15) rather than run-to-run variability, which addresses a different question (does performance degrade if you choose the wrong hyperparameter?) than statistical reliability (if you run the same configuration multiple times, how much does performance vary?).

Mitigation status. The paper does not acknowledge this as a limitation. The standard in the RLVR training literature for reporting single best-checkpoint scores is unfortunately common (Yu et al., 2025; Cui et al., 2025; Wang et al., 2025 all report similar point estimates), but this does not excuse the omission of basic statistical reporting for a method paper that claims state-of-the-art with a 2.66% margin. A practitioner trying to decide whether to adopt Lp-Reg over 80/20 would reasonably want to know whether the 2.66% gap is larger than run-to-run variance. The paper's code release enables reproduction, but running 81,204 GPU-hour experiments for statistical validation is not feasible for most practitioners, putting the burden of establishing statistical reliability on the authors — a burden that is not met.

No Comparison Against an Optimized Entropy Regularization Baseline

The assumption or constraint. The GRPO + Entropy Loss baseline uses a single entropy coefficient of 0.002, implemented "within the verl framework" (Section 5.1.2). The paper does not report any hyperparameter sweep over entropy coefficients, does not justify the choice of 0.002, and does not show whether a different coefficient would avoid the catastrophic collapse that defines this baseline's failure. Given that the paper's central claim is that Lp-Reg's selective low-probability protection is superior to indiscriminate entropy boosting, the strength of this claim depends on comparing against the best possible version of entropy boosting — not an arbitrarily chosen one.

The consequence. If the entropy coefficient of 0.002 is poorly tuned (too high, causing the catastrophic collapse in Figure 1b; too low, providing insufficient exploration incentive), then the paper's comparison understates the potential of entropy regularization and overstates Lp-Reg's relative advantage. A properly tuned entropy baseline might still underperform Lp-Reg, but the margin might be smaller, and the claim of "superiority over indiscriminate entropy-control methods" would need to be qualified as "superiority over a specific entropy coefficient choice." The paper's own Lp-Reg method is shown to be robust to hyperparameter choices (Appendix A, Figure 15: ρ ∈ {0.005, 0.010, 0.015} and κ ∈ {0.01, 0.02, 0.03} all work), but the baseline it critiqued most heavily receives no equivalent robustness analysis. This asymmetry undermines the paper's central conceptual argument — that the principle of selectivity matters more than the principle of entropy boosting — because the entropy boosting principle is not given a fair empirical test.

What evidence exists in the paper. The entropy coefficient of 0.002 is stated in Section 5.1.2 with no justification. The GRPO + Entropy Loss training curves (Figure 1b, Figure 4) show the fastest collapse of any method, with accuracy dropping from ~0.53 to ~0.48 by step 200 on Qwen3-14B (below the starting accuracy). The entropy trajectory for this baseline is not shown separately but can be inferred from Figure 1b (top-right): entropy spikes sharply then crashes. The paper's analysis of entropy loss failure in Figures 1d and 18 (amplifying irrelevant token sampling) suggests why the chosen coefficient fails, but does not demonstrate that a smaller coefficient (which might add less noise while still providing some exploration incentive) would also fail.

Mitigation status. The paper does not address this at all. There is no ablation over entropy coefficients, no discussion of coefficient selection, and no acknowledgment that the baseline comparison is sensitive to this choice. The paper's argument for why entropy boosting fails (it amplifies noise without protecting reasoning sparks) is theoretically motivated and qualitatively supported by the token-level analyses (Figures 1d, 18), which provide convergent evidence beyond the single-coefficient result. If entropy boosting amplifies noise at any non-zero coefficient (because even a small entropy bonus protects all low-probability tokens equally, including noise), then the coefficient choice matters less — the principle would fail regardless of tuning. But the paper doesn't make this argument explicitly, and without a sweep showing that all reasonable coefficients fail, the baseline remains vulnerable to the criticism of poor tuning.

Training Stability at 3,000 Steps Is Demonstrated Only for Lp-Reg, Not for Baselines

The assumption or constraint. The continuous scaling experiment (Figure 3) shows Lp-Reg (on-policy) training stably for 3,000 steps on Qwen2.5-32B-Base, totaling 81,204 GPU-hours, with accuracy improving from ~0.20 to ~0.60. The paper presents this as evidence that Lp-Reg "enables continuous on-policy scaling for around 3,000 steps, a regime where many entropy-control methods have collapsed." However, no baseline method is trained for 3,000 steps. The paper's early stopping criterion (>10% accuracy drop) means baselines that collapsed at 400–800 steps in the main experiments were terminated, and no attempt was made to see whether they would recover if training continued, or whether they would collapse later at 2,000 or 3,000 steps. The claim that baselines "collapse" implicitly assumes the collapse is permanent and would persist at longer horizons, but this is not tested.

The consequence. The paper cannot distinguish between two possibilities: (a) baselines truly collapse irrecoverably and would show degraded or zero performance at 3,000 steps, or (b) baselines experience a temporary instability phase that they might recover from given enough training steps. The GRPO training curve in Figure 1b shows accuracy peaking around step 400 and then declining — if this decline is transient and GRPO would recover and eventually reach 0.60 accuracy at step 2,500, then Lp-Reg's claimed advantage in "continuous scaling" is largely about when the performance is achieved (faster, with less instability) rather than whether it is achieved at all. The paper's own characterization of Lp-Reg's entropy trajectory as "a dynamic, multi-phase entropy trajectory: entropy initially decreases as the model learns core reasoning patterns, then gradually increases to foster exploration as performance improves, and finally stabilizes" (Section 5.2) could describe a process that baseline methods also undergo, just at different timescales or with different noise characteristics.

What evidence exists in the paper. The baseline training curves (Figures 4, 5, 6) show trajectories for approximately 1,000 steps, and most show clear signs of collapse or plateau by that point. GRPO (on.) on Qwen3-14B (Figure 4, right) peaks around step 200 and declines gradually to step 600, after which it appears to stabilize (not continue declining). GRPO + Entropy Loss crashes by step 200 and does not recover within the remaining 800 steps. Clip-Higher (off.) shows high variance but maintains approximately 0.57 accuracy through step 1,000. None of these curves show evidence of imminent recovery — the collapse patterns appear persistent over the observed window. However, "persistent over 1,000 steps" is not the same as "would persist over 3,000 steps," and the paper's 3,000-step claim uses only Lp-Reg data.

Mitigation status. The paper does not acknowledge this gap. The early stopping criterion is described pragmatically ("for models that experienced a training collapse... we implemented early stopping to conserve computational resources"), which is a reasonable practical constraint — running collapsed baselines for 3,000 steps would waste substantial compute. However, the paper then makes claims about what baselines would do at 3,000 steps ("a regime where many entropy-control methods have collapsed") without the corresponding evidence. A more circumspect framing would note that Lp-Reg demonstrates stability at 3,000 steps while baselines show collapse within 1,000 steps, without asserting that the collapse is permanent. The computational cost of running even a single baseline to 3,000 steps (tens of thousands of GPU-hours) makes this a genuinely difficult limitation to address, but the paper should at minimum acknowledge that the 3,000-step comparison is one-sided and frame its claims accordingly.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a diagnostic reframing of the exploration collapse problem in RLVR that shifts attention from aggregate distribution statistics to token-level survival dynamics. The field's prior approach—monitoring policy entropy as a proxy for exploration quality and designing interventions to maintain it—is shown to be not merely suboptimal but actively counterproductive in the case of entropy regularization, which the paper demonstrates accelerates collapse (Figure 1b). The insight that which tokens survive matters more than how spread out the distribution is represents a genuine conceptual reorientation, not just a new method within the existing entropy-management paradigm.

The magnitude of this shift is diagnostic rather than paradigmatic. The paper does not overturn the RLVR training framework itself—GRPO remains the foundation, and the core mechanics of group-relative advantage estimation and policy gradient updates are unchanged. What changes is how we understand failure: exploration collapse is not "the model becomes too deterministic" but rather "the model systematically eliminates specific tokens that enable it to reconsider its reasoning." This is a more precise and causally grounded diagnosis, and it implies a different class of solutions: interventions on token survival rather than interventions on distribution shape.

The paper also provides a unifying resolution to a tension in the existing literature. Entropy minimization approaches (Gao et al., 2025; Agarwal et al., 2025) show that sharpening model confidence improves inference performance, while entropy regularization approaches (GRPO + Entropy Loss, Clip-Higher) attempt to maintain or increase entropy during training. These seem contradictory—should we make the model more or less random? The paper's token-survival framework resolves this: entropy minimization at inference is beneficial because it reduces noise (irrelevant low-probability tokens) without affecting the model's ability to deploy exploratory pivots when needed, since those pivots have been preserved through training. Entropy boosting at training time is harmful because it indiscriminately amplifies both noise and exploration—and the noise component, the paper shows, is destructive. The distinction is not "high entropy good vs. low entropy good" but rather "preserve specific exploratory tokens while allowing entropy to naturally decrease elsewhere."

For the research landscape, this work makes several directions more attractive and others less so:

More attractive:

• Token-level diagnostics for RL training. The paper demonstrates that tracking the probability distributions of specific token categories (Figures 1c, 1d, 17, 18) reveals training dynamics invisible in aggregate entropy curves. Future RLVR research should routinely report not just policy entropy but also the survival rates of tokens identified as functionally important for the target domain.
• Selective regularization informed by model confidence. Lp-Reg's core mechanism—using the model's own probability distribution to filter intervention targets—suggests a broader principle: wherever the model exhibits systematic confidence differences between meaningful and meaningless alternatives (as Figure 14 shows for low-probability tokens), confidence-informed selectivity can outperform uniformity. This could extend beyond exploration to other RLVR challenges like reward hacking prevention or catastrophic forgetting mitigation.
• Understanding pre-trained model capabilities for exploration. The paper's finding that reasoning sparks exist in pre-trained models and are extinguished by RLVR suggests that pre-trained LMs have richer exploratory capacities than RLVR training reveals. Research into what other latent capabilities are lost during fine-tuning, and how to preserve them, becomes more pressing.

Less attractive:

• Designing new entropy-control mechanisms. The paper provides strong evidence that manipulating aggregate entropy is an indirect and error-prone approach to exploration. Clip-Higher's continuous entropy inflation (Figure 5), GRPO + Entropy Loss's catastrophic collapse (Figure 1b), and the high-entropy regularization ablation's failure (Figure 7) collectively suggest that the entropy-as-exploration framework has fundamental limitations that incremental improvements to entropy-control mechanisms cannot overcome.
• Treating all low-probability tokens as equally valuable for exploration. The paper's noise-filtering ablation (Figure 8, Lp-Reg w/o τ) shows that protecting all low-probability tokens causes catastrophic failure—the extreme tail is dominated by semantically irrelevant noise. This means methods that boost exploration by uniformly encouraging low-probability sampling (e.g., temperature scaling without filtering) are likely to introduce more noise than signal.

Follow-Up Research This Work Enables

Directly measuring whether preserved reasoning sparks cause improved exploration diversity. The paper shows that Lp-Reg preserves reasoning sparks (Figures 12, 13) and that Lp-Reg achieves better accuracy (Table 1), but the causal chain—preserved tokens → more diverse reasoning → better accuracy—is inferred rather than demonstrated. A strong follow-up would intervene causally: train two models with Lp-Reg, but in one, manually suppress reasoning sparks during inference (e.g., by masking "wait," "however," "perhaps" from the sampling distribution) and measure whether accuracy degrades to GRPO-baseline levels. Conversely, take a baseline GRPO-trained model (where reasoning sparks have been eliminated) and manually inject them during inference by boosting their sampling probabilities to pre-collapse levels, measuring whether accuracy recovers. If accuracy moves in the predicted directions, the causal claim is validated. If not—if suppressing sparks doesn't hurt Lp-Reg or injecting them doesn't help GRPO—then the paper's diagnostic framework would need revision: the benefit might come from some other property of Lp-Reg's regularization (e.g., optimization landscape smoothing) rather than from preserving exploratory tokens per se. The experiment requires no additional training, only controlled inference-time intervention on models already trained in this paper's setup, making it unusually cheap to execute.

Characterizing reasoning sparks in non-mathematical domains to test generality. The paper identifies five reasoning sparks in mathematical reasoning ("but," "wait," "perhaps," "alternatively," "however") but provides no procedure for identifying them in other domains. A systematic study would apply Lp-Reg to code generation RLVR (using, e.g., the APPS or CodeContests datasets with unit-test-based binary rewards) and ask: (a) Do exploratory tokens analogous to reasoning sparks exist in code? Candidate tokens might include "refactor," "rewrite," "simplify," "instead," "alternatively." (b) Does the probabilistic gap between meaningful and irrelevant tokens (Figure 14) replicate in code, and at what probability threshold? (c) Does Lp-Reg with math-derived hyperparameters (ρ = 1%, κ = 0.02) transfer directly, or do code-specific tokens require different filtering thresholds? If Lp-Reg transfers without modification, the reasoning sparks concept gains domain-generality. If it fails or requires domain-specific tuning, the concept would be shown to be math-specific, and the paper's general claims would need qualification. This experiment is feasible because the verl framework already supports code RLVR, and the computational cost would be comparable to the paper's ~8,000 GPU-hour runs.

Stress-testing Lp-Reg under reward sparsity and noise to map its failure modes. The paper's experiments use dense, clean binary rewards (correct/incorrect final answer for every problem). Many RLVR applications have sparser or noisier rewards: only some problems have verifiable answers, rewards may be delayed across multiple reasoning steps, or verification may be stochastic. A systematic stress-test would vary reward sparsity (providing rewards for only 25%, 50%, or 75% of problems, with the rest receiving zero reward regardless of correctness) and reward noise (flipping a fraction of binary rewards) during Lp-Reg training on MATH. The key question: does Lp-Reg's advantage over GRPO grow, shrink, or reverse as rewards degrade? If Lp-Reg is robust because its protection mechanism depends on the proxy distribution (derived from the policy, not the reward), then the advantage might grow under reward sparsity (because GRPO would collapse even faster with less signal). If Lp-Reg is fragile because the proxy distribution's filtering relies on the policy having already learned meaningful distinctions (which sparse rewards delay), then the advantage might shrink or reverse. This experiment maps the boundary conditions of Lp-Reg's applicability and would be important for practitioners considering it for reward-constrained settings.

Testing whether the probabilistic gap between reasoning sparks and noise (Figure 14) is a general property of pre-trained LMs or an artifact of the specific token selection and analysis window. The paper's Figure 14, which provides the empirical foundation for Lp-Reg's filtering mechanism, has several analytic constraints: only the top-64 most probable tokens are examined, only within the probability range [0, 0.1], and only for two hand-selected groups of five tokens each. A comprehensive replication would: (a) extend the analysis to the full vocabulary (or at least the top-1,000 tokens) to see whether the gap persists across all tokens or is specific to the selected set; (b) test whether the gap exists in pre-trained models before any RLVR training—if it does, it's a property of pre-trained LMs that Lp-Reg exploits; if it doesn't, the gap might be a product of early RLVR training dynamics that Lp-Reg then preserves; (c) test across multiple model families (Qwen, Llama, DeepSeek) and sizes (7B, 14B, 32B, 70B) to see whether the gap scales with model capability, as the paper's Qwen3 vs. Qwen2.5 comparison hints but doesn't establish. This analysis requires no training, only inference on base models to collect next-token probability distributions on math prompts, making it computationally cheap and high-impact for establishing the generality of the paper's core mechanistic claim.

Combining Lp-Reg with other exploration mechanisms to test whether the benefits are additive or redundant. The paper compares Lp-Reg against individual baselines but never combines it with them. A natural combination experiment: train with Lp-Reg + Clip-Higher (asymmetric clipping from DAPO) or Lp-Reg + 80/20 (selective updates from Wang et al.) on Qwen3-14B. The question is whether Lp-Reg's token-protection mechanism and these methods' entropy/update mechanisms address different aspects of the exploration problem (in which case combining them should yield additive gains) or whether Lp-Reg already subsumes their benefits (in which case adding them provides no improvement or even interferes). If Lp-Reg + Clip-Higher achieves, say, 63% average accuracy (vs. Lp-Reg's 60.17% and Clip-Higher's 56.16% individually), that would demonstrate complementarity and point toward a combined exploration strategy. If performance is unchanged or degraded, that would suggest Lp-Reg's selective protection is sufficient and additional entropy mechanisms are either redundant or harmful. This experiment requires no new methodological development, only combining existing implementations in the verl framework, and would directly inform practitioners about whether to adopt Lp-Reg as a standalone solution or as one component of a broader exploration toolkit.

Validating whether Lp-Reg's sustained training stability translates to larger models (70B+). The paper's results span 14B and 32B models, with the hypothesis that "as base models improve, their capacity for nuanced, low-probability reasoning increases, creating a richer substrate for the emergence of valuable reasoning sparks." If true, Lp-Reg's advantage should grow at larger scales, making it increasingly important as the field moves toward 70B, 405B, and beyond. A scaling study training Lp-Reg vs. GRPO on Qwen2.5-72B (or the largest feasible model) for ~1,000 steps would test this hypothesis. The prediction: the accuracy gap between Lp-Reg and GRPO should be larger at 72B than at 32B, and larger at 32B than at 14B. If instead the gap shrinks or reverses at larger scales, the paper's claims about scaling behavior would need revision. This experiment is computationally expensive (~40,000+ GPU-hours for 72B) but is the natural next step for establishing whether Lp-Reg is a practical solution for the scale at which state-of-the-art reasoning models are actually trained.

Practical Applications and Downstream Use Cases

Training open-source reasoning models with reduced collapse risk. The most direct application of Lp-Reg is in RLVR training pipelines for open-source reasoning models. The paper demonstrates that standard GRPO training on Qwen3-14B collapses (Figure 4: accuracy peaks then declines), and that Lp-Reg prevents this collapse while improving final accuracy by 2.66% (60.17% vs. 57.51% for the next best method). For teams training reasoning models on mathematical or scientific problem-solving—where RLVR with verifiable rewards is the dominant paradigm—adopting Lp-Reg means training runs can safely continue for more steps without collapse, potentially extracting more performance from a given base model and dataset. The practical benefit is not just the +2.66% accuracy gain but the elimination of the need to carefully monitor for collapse and implement early stopping, which simplifies training orchestration and reduces the risk of wasting compute on runs that will degrade. The method's hyperparameter robustness (Figure 15: ρ ∈ {0.005, 0.010, 0.015} and κ ∈ {0.01, 0.02, 0.03} all work) means adoption requires minimal tuning—practitioners can use the paper's defaults (ρ = 1% for 14B models, ρ = 0.5% for 32B, κ = 0.02, β = 1.0) with reasonable confidence they will work. The primary practical cost is the additional per-step computation for proxy distribution construction and KL penalty evaluation, which the paper does not quantify but which (given that proxy construction uses probabilities already computed during generation) is likely a small fraction of the transformer forward pass.

Improving data efficiency in self-improvement and distillation pipelines. When using RLVR-trained models to generate training data for subsequent fine-tuning (as in rejection sampling, STaR, or ReST-style self-improvement loops), the quality and diversity of generated solutions directly determine the quality of the distilled model. Lp-Reg's sustained exploration—evidenced by broader probability-entropy distributions for reasoning sparks (Figure 12) and higher frequency of these tokens throughout training (Figure 13)—means Lp-Reg-trained models produce a wider variety of reasoning paths, including paths that involve reconsideration and branching. For a self-improvement pipeline, this diversity translates to higher-quality training data: instead of generating 1,000 solutions that follow the same collapsed reasoning pattern, an Lp-Reg-trained generator produces solutions with diverse structures, including self-correction and alternative approaches, which provide richer supervision for the next training iteration. The paper's finding that Lp-Reg maintains higher exploratory token frequency at step 768 than GRPO does at step 256 (Figure 13: "but" at ~0.0025 under Lp-Reg vs. ~0.0010 under GRPO) suggests this diversity advantage persists throughout training, not just in early stages. A practitioner running a multi-round self-improvement loop could use Lp-Reg in each round's RLVR phase to maintain solution diversity across rounds, potentially avoiding the diversity collapse that limits the effectiveness of iterative self-improvement.

Enabling longer training horizons for resource-constrained teams. The paper's 3,000-step continuous scaling experiment (Figure 3) demonstrates that Lp-Reg enables training stability well beyond the point where standard GRPO collapses. For teams with limited GPU budgets who cannot afford to train multiple short runs with different hyperparameters or early stopping points, Lp-Reg's stability means they can run a single long training job with confidence that performance will not catastrophically degrade partway through. The practical implication: instead of training 5 separate 800-step runs with different seeds or hyperparameters (totaling 5 × 16,000 = 80,000 GPU-hours on 32B, comparable to the paper's 81,204 GPU-hour single run) and selecting the best, a team could run one 3,000-step Lp-Reg job and achieve better or comparable final performance with less orchestration overhead. The paper shows Lp-Reg at 3,000 steps reaching ~0.60 accuracy on Qwen2.5-32B (Figure 3), compared to the best 800-step result of 52.54% for Lp-Reg (on.) and 51.49% for 80/20 (off.) in Table 1—suggesting the longer run provides substantial additional gains. For teams where engineering time for training orchestration is scarcer than GPU time, this reliability is a significant practical advantage.