F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare

ArXiv: 2602.06717

🎯 Pitch

Intermediate group sizes in RLVR actively destroy solution diversity: the probability of missing rare-correct trajectories peaks exactly where most practical training operates, causing models to forget how to solve hard problems even as easy-prompt accuracy improves. F-GRPO fixes this with a focal weighting coefficient that selectively downscales gradient updates on already-solved prompts, recovering pass@256 performance at small group sizes without extra computation.

---

1. Executive Summary

This paper analyzes how finite group size in group-relative RLVR drives a tail-miss failure mode—a non-monotonic phenomenon where active training updates at intermediate rollout counts (the most common practical regime) systematically miss rare-correct trajectories while concentrating probability onto common solutions, reconciling conflicting prior guidance that small groups preserve diversity through inactivity and large groups through coverage. The authors propose Focal weighting for Group-Relative Policy Optimization (F-GRPO), a difficulty-aware advantage scaling coefficient inspired by Focal loss that down-weights updates on high-success prompts by multiplying the group-relative advantage by $(1 - \hat{\mu}_{\text{pos}})^{\gamma}$, directly targeting the reward-positive regime where unsampled-correct mass shrinks even as total correct mass grows. On Qwen2.5-7B trained on DeepScaleR across in-domain math benchmarks (AIME, MATH500, Olympiad) and out-of-domain tasks (IFEval, GPQA), F-GRPO at $N = 8$ improves pass@256 by +6.2 points over GRPO (64.1 → 70.3), matches or exceeds GRPO at $N = 32$ using 4× fewer rollouts, and generalizes as a drop-in modification to DAPO (+3.2 pass@256) and CISPO (+3.6 pass@256), establishing that preserving solution diversity under practical compute constraints requires suppressing gradient contributions from prompts where the concentration mechanism is most active, not increasing group size.

2. Context and Motivation

The Core Problem: RLVR Sharpens Distributions It Should Be Diversifying

The fundamental question this paper tackles is subtle but consequential: when we train language models with group-relative RLVR at practical group sizes, does the optimization dynamics systematically suppress rare but correct solutions? The answer, the paper argues, is yes—and the mechanism is not obvious.

RLVR has become the dominant paradigm for post-training LLMs on reasoning tasks (Section 1). The recipe is straightforward: sample multiple rollouts per prompt, assign binary rewards based on verifiable correctness, compute advantages relative to the group mean, and update the policy. This has driven impressive gains on mathematical reasoning (Li et al., 2024), code generation (Jimenez et al., 2023), and general problem-solving benchmarks (Chollet et al., 2025), and has been adopted at scale by major labs (Guo et al., 2025; Yang et al., 2025; Team et al., 2025; Shao et al., 2024).

But a growing body of evidence suggests something troubling. RLVR does not primarily teach the model new reasoning strategies; it sharpens the output distribution toward solutions the base model already knows how to produce (Yue et al., 2025; Ni et al., 2025; Wu et al., 2025a; Dang et al., 2025). The empirical signature of this "sharpening" is a specific pattern in pass@k metrics: RLVR-trained models improve their pass@1 (single-attempt accuracy goes up), but degrade at large sampling budgets (pass@256, pass@1024), indicating that the model has concentrated its probability mass onto fewer solutions, losing diversity. This is not merely an academic concern—it directly limits the test-time scaling capabilities of RLVR-trained models, since the benefit of drawing many samples and selecting the best one is predicated on the model producing a diverse set of candidate solutions.

The paper frames this as a tension at the heart of RLVR (Section 1):

"RLVR does not primarily introduce new knowledge, but instead sharpens the output distribution toward solutions already accessible to the base model"

Yet simultaneously, other work suggests that "prolonged or carefully scaled RL can expand the effective reasoning boundary" (Liu et al., 2025b; Yuan et al., 2025). This leaves the role of RLVR genuinely ambiguous: is it fundamentally constrained to redistributing existing capability, or can it, under the right conditions, create new capability? The paper does not resolve this deeper question, but it identifies a specific, addressable mechanism through which one flavor of sharpening—the loss of rare-correct modes—occurs.

Why This Problem Matters

The practical stakes are high for several interconnected reasons:

Test-time compute scaling is predicated on diversity. The standard approach to improving reasoning performance at inference time is to sample many candidate solutions and select the best one (e.g., best-of-N, majority voting, or verifier-guided search). If RLVR training concentrates the policy onto a narrow set of solutions, pass@k for large k degrades, directly undermining the value of test-time compute. The paper's motivation crystallizes in Figure 1(b,c): GRPO at $N = 8$ improves pass@1 over $N = 2$, but pass@256 drops sharply—the model is getting better at its first try but worse at producing diverse correct solutions when given many attempts.

Group size is a fixed computational bottleneck. Most RLVR systems use group-relative methods like GRPO (Shao et al., 2024), DAPO (Yu et al., 2025), or CISPO (Chen et al., 2025a), where advantages are computed from $N$ rollouts per prompt. The group size $N$ directly controls which trajectories receive learning signal (advantages) and which do not (when all rewards in the group are identical, the advantage is zero and no update occurs). Yet $N$ cannot be scaled freely—each rollout costs GPU memory and wall-clock time, and $N$ multiplies the number of forward passes per training step. In practice, $N$ is constrained to small values (2–32) that leave the training dynamics in an unresolved regime.

Conflicting prior guidance paralyzes practitioners. The literature provides no clear answer to "what $N$ should I use?" Wu et al. (2025b) show that $N = 2$ is theoretically sound and computationally efficient, connecting GRPO to DPO. Hu et al. (2025) advocate for large $N$ to broaden exploration and stabilize learning, showing that scaling $N$ ensures the change in total correct mass is non-negative. He et al. (2025) observe that $N = 32$ improves pass@1 but degrades pass@k for large k—a sharpening regime. These recommendations are contradictory because they evaluate different metrics and operate at different points in the group-size spectrum. A practitioner reading the literature has no principled basis for choosing $N$.

The problem is about what is not sampled, not just what is. This is the paper's key conceptual reframing. Prior work on group-relative RLVR focused on the properties of sampled trajectories—how advantages are computed, how clipping affects updates, how rewards are assigned. The paper shifts attention to what happens to unsampled trajectories during an update. When a group of $N$ rollouts is drawn, some correct solutions are inevitably not sampled. In reward-positive batches (where the group mean reward $S_R > 0$), the one-step logit update derived in Section 2.3 (Equation 7) shows that unsampled actions receive $\Delta z_i = -\frac{\eta}{N} S_R \, p_i$: their logits are pushed downward simply because they were not sampled, regardless of whether they are correct. The probability mass of these unsampled-correct actions can shrink even while total correct mass is increasing (Proposition 3.2). This is the "sharpening" mechanism: RLVR concentrates mass onto whichever correct solutions happen to appear in the sampled group.

Where Existing Approaches Fall Short

The paper identifies limitations in several lines of prior work, each of which captures part of the picture but misses the full dynamics:

1. Prior theoretical analyses focus on total correct mass, not within-correct redistribution. Hu et al. (2025) derive the one-step change in total correct mass $Q_{\text{pos}}$ (Equation 8) and show that scaling $N$ ensures $\Delta Q_{\text{pos}} \geq 0$—the model becomes more accurate on average. But this analysis aggregates all correct actions together. It does not distinguish between a model that spreads its correct mass across 10,000 distinct solution strategies and one that concentrates all its mass onto a single strategy, even though the pass@k behavior of these two models would be radically different. The paper's Proposition 3.2 addresses this gap directly: it derives $\Delta Q_{\text{u, pos}}$, the change in mass of unsampled-correct actions, showing it can be negative even when $\Delta Q_{\text{pos}} > 0$ (Section 3.2):

"RLVR can increase total correct mass while concentrating it onto sampled-correct actions at the expense of unsampled-correct ones."

2. Prior empirical work documents sharpening but does not explain the group-size mechanism. Yue et al. (2025), Dang et al. (2025), and Wu et al. (2025a) all observe the pass@1-improves-but-pass@k-degrades pattern and attribute it to distribution sharpening, but they do not provide a mechanistic account of why the sharpening occurs or how it depends on $N$. He et al. (2025) propose an "unlikeliness reward" to up-weight rare correct trajectories, correctly identifying the symptom (rare-correct modes are being suppressed) but not the sampling-dynamics cause (the non-monotonic tail-miss probability). The paper's Lemma 3.1 provides this missing causal mechanism: the probability that an active update misses rare-correct regions, $\Pr(\mathcal{B}_\tau)$, is maximized at intermediate $N$—exactly the regime where sharpening is empirically observed.

3. Difficulty-aware methods address correlated but distinct problems. Zhou et al. (2025) rebalance loss contributions across difficulty groups to equalize loss scale. He et al. (2025) modify trajectory-level rewards to up-weight rare-correct trajectories. Gai et al. (2025) apply differential smoothing to correct versus incorrect trajectories. These approaches modify what signal individual trajectories receive. The paper's Focal weighting is orthogonal: it modifies which prompts contribute to the gradient at all, scaling the entire group-relative advantage by $(1 - \hat{\mu}_{\text{pos}})^{\gamma}$ (Section 4). This is a coarser intervention (prompt-level rather than trajectory-level), but it directly targets the $S_R > 0$ regime where the unsampled-correct concentration mechanism is most active, rather than trying to identify which trajectories within a group are rare.

4. Entropy and KL regularization are blunt instruments. Common diversity-preserving techniques like entropy bonuses or KL penalties to a reference model apply uniform pressure regardless of prompt difficulty or the concentration dynamics of the current batch. The paper's comparison with these methods (Appendix G, Table 4) shows that GRPO-KL achieves slightly higher pass@256 on math (72.0 vs. F-GRPO's 70.3) but requires maintaining a reference model in memory and underperforms on OOD pass@256 (60.0 vs. 63.3). Entropy bonuses show mixed results. These regularizers treat all prompts identically, diluting their effect: they apply diversity pressure where it is not needed (on hard prompts where the model already explores broadly) and may apply insufficient pressure where it is needed (on high-success prompts where concentration is aggressive).

How This Paper Positions Itself

The paper positions itself at the intersection of two lines of work—theoretical analyses of group-relative RLVR sampling dynamics and empirical observations of distribution sharpening—while filling a gap neither line addresses: a mechanistic understanding of how group size controls the tradeoff between accuracy improvement and diversity loss, and a practical intervention that mitigates the problem without increasing computational cost.

The theoretical contribution (Section 3) derives a closed-form expression for the probability that an active RLVR update misses rare-correct trajectories (Lemma 3.1, Equation 11):

$\Pr(\mathcal{B}_\tau) = (1 - \tau)^N - (\mu_{\text{pos}} - \tau)^N - (1 - \mu_{\text{pos}})^N$

This expression unifies the seemingly contradictory prior findings into a single framework. Three regimes emerge naturally from the non-monotonic shape (Figures 1(a) and 2):

• Small $N$: The update is rarely active ($\Pr(\mathcal{A}_N)$ is low), so the policy changes slowly. Diversity is preserved because the model does not move much from the base distribution. This corresponds to the Wu et al. (2025b) finding that $N = 2$ works well.

• Intermediate $N$: Updates are frequently active, but coverage of rare-correct modes is poor. $\Pr(\mathcal{B}_\tau)$ peaks. This is the sharpening regime observed by He et al. (2025) at $N = 32$ and by Yue et al. (2025) at practical group sizes. The model improves pass@1 (it concentrates onto common correct solutions) but loses diversity (rare-correct modes shrink).

• Large $N$: Coverage improves as $(1 - \tau)^N \to 0$, and unsampled mass becomes negligible. Both pass@1 and pass@k can improve. This is the Hu et al. (2025) regime.

The paper's key insight is that practical training almost always operates in the intermediate regime where $\Pr(\mathcal{B}_\tau)$ peaks, because computational constraints prevent scaling $N$ to the point where rare-correct coverage is reliable. The question then becomes: can we stay in this computationally feasible regime while mitigating the concentration failure mode?

The proposed solution—Focal weighting (Section 4)—is deliberately minimal. It does not require changing the group size, modifying the reward structure, adding auxiliary networks, or introducing new hyperparameters beyond the Focal exponent $\gamma$. It is a single scalar multiplier $g(x) = (1 - \hat{\mu}_{\text{pos}}(x))^{\gamma}$ applied uniformly to all rollouts from a prompt, derived directly from the theoretical analysis: since the concentration mechanism is driven by $S_R > 0$ (reward-positive batches), and $S_R$ is non-decreasing in the empirical success rate $\hat{\mu}_{\text{pos}}$ (Corollary B.2), suppressing gradient contributions from high-$\hat{\mu}_{\text{pos}}$ prompts directly targets the regime where unsampled-correct mass is most vulnerable.

The paper explicitly frames F-GRPO as orthogonal to existing modifications of the clipping mechanism (DAPO) or importance weighting (CISPO), stating (Section 4.2):

"The concentration phenomenon we address arises from the sampling dynamics of group-relative advantage estimation, not from these algorithmic choices. The Focal weight $g(x)$ is thus orthogonal and can be applied independently."

This positioning is important because it implies the concentration problem is fundamental to group-relative RLVR, not an artifact of GRPO's specific objective. Any method that computes advantages by comparing trajectories within a sampled group will exhibit the tail-miss dynamics, and Focal weighting should provide benefits across all of them—a prediction verified in Table 1.

3. Technical Approach

3.1 Reader Orientation

This paper develops a theoretical diagnosis and a lightweight fix for a specific failure mode in group-relative reinforcement learning with verifiable rewards (RLVR). The fix is a single scalar multiplier—called Focal weighting—that is multiplied onto the existing advantage estimates in algorithms like GRPO, DAPO, or CISPO. The problem it solves is that at practical (intermediate) group sizes, training updates systematically miss rare-but-correct solution strategies while still being "active" (i.e., containing mixed correct/incorrect rewards), causing the policy to concentrate probability onto common solutions and lose the diversity needed for test-time compute scaling. The solution's shape is simple: identify which prompts are most likely to be in the dangerous region (high empirical success rate), and reduce their gradient contribution using a difficulty-aware coefficient inspired by Focal loss.

3.2 Big-Picture Architecture (Diagram in Words)

The system is not a new training pipeline but a modification to the advantage computation inside any existing group-relative RLVR loop. The components are:

1. Base policy ($\pi_\theta$): The LLM being fine-tuned. It generates $N$ i.i.d. rollouts per prompt.
2. Verifiable reward function: An automatic checker that assigns a binary reward (correct/incorrect) to each complete rollout.
3. Group-relative advantage estimator (e.g., GRPO): Computes a normalized advantage for each rollout by comparing its reward to the mean and standard deviation of rewards within the sampled group of size $N$.
4. Empirical success rate estimator ($\hat{\mu}_{\text{pos}}$): Counts what fraction of the $N$ rollouts for a given prompt are correct. This is computed per prompt, per batch.
5. Focal weight ($g(x)$): A scalar in $[0, 1]$ computed as $(1 - \hat{\mu}_{\text{pos}})^\gamma$. It multiplies the group-relative advantage for all rollouts from the same prompt.
6. Policy gradient objective (GRPO/DAPO/CISPO): The standard clipped surrogate loss, now using the Focal-weighted advantage instead of the raw one.

The information flow is: a batch of prompts → $N$ rollouts per prompt sampled from $\pi_\theta$ → binary rewards assigned by the verifier → $\hat{\mu}_{\text{pos}}$ computed per prompt (fraction correct) → raw advantages computed per rollout (reward minus group mean, divided by group std) → Focal weight $g(x) = (1 - \hat{\mu}_{\text{pos}})^\gamma$ multiplies all advantages from that prompt → clipped surrogate objective is computed using weighted advantages → policy is updated via gradient ascent.

3.3 Roadmap for the Deep Dive

• First, the formal group-relative RLVR setup (GRPO, DAPO, CISPO objectives and advantage computation), since F-GRPO is a modification inside these existing objectives and the reader must understand what is being modified.
• Second, the categorical policy framework from Hu et al. (2025), because Proposition 3.2 (how unsampled-correct mass changes) is derived in this simplified setting, and the one-step logit update (Equation 7) is the mathematical engine behind the concentration mechanism.
• Third, the tail-miss probability analysis (Lemma 3.1), since this is the core theoretical contribution that identifies when rare-correct modes are most vulnerable—the non-monotonic dependence on $N$ that unifies prior contradictory findings.
• Fourth, the unsampled-correct mass analysis (Proposition 3.2), since this explains the mechanism by which concentration happens—the direct drift and normalization coupling terms that can make $\Delta Q_{\text{u,pos}}$ negative even when total correct mass increases.
• Fifth, the Focal weight construction, connecting the theoretical $S_R > 0$ regime to the observable $\hat{\mu}_{\text{pos}}$ proxy via the monotonicity result (Corollary B.2), and explaining the integration with GRPO/DAPO/CISPO.
• Sixth, the categorical simulation setup (Appendix J), since it provides controlled empirical validation of the three-regime framework before the full LLM experiments.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a theoretical analysis paper with a lightweight algorithmic intervention whose core idea is that finite group size in group-relative RLVR creates a non-monotonic tail-miss probability that peaks at intermediate $N$, causing distribution sharpening, and that this can be mitigated by down-weighting gradient contributions from high-success prompts using a Focal-inspired scaling factor.

---

Group-Relative RLVR: The Base Objectives Being Modified

GRPO (Group Relative Policy Optimization) is the primary baseline that F-GRPO modifies. The core idea of GRPO is to eliminate the learned value function (critic) that is standard in PPO, and instead compute advantages by comparing each rollout's reward to the statistics of the other rollouts sampled from the same prompt.

For a given prompt $x$, the policy $\pi_\theta$ generates $N$ complete rollouts (trajectories) $\{o_i\}_{i=1}^N$. Each rollout receives a binary reward based on verifiable correctness:

$R_i = R_w + (R_c - R_w) \cdot \mathbb{I}[o_i \text{ is correct}]$

where $R_c > R_w$, typically $R_c = 1$ and $R_w \in \{0, -1\}$. This means correct rollouts get a positive reward, incorrect rollouts get a lower (possibly zero or negative) reward.

The group-relative advantage for rollout $i$ is:

$\hat{A}_i^{\text{GRPO}} = \frac{R_i - \bar{R}}{\sigma_R + \epsilon}$

where $\bar{R} = \frac{1}{N} \sum_{j=1}^N R_j$ is the mean reward across the $N$ rollouts in the group, $\sigma_R = \text{std}(\{R_j\}_{j=1}^N)$ is the standard deviation of rewards in the group, and $\epsilon$ is a small constant for numerical stability.

What it computes: the normalized deviation of rollout $i$'s reward from the group average. If rollout $i$ is correct and most others are incorrect, $\hat{A}_i^{\text{GRPO}}$ is positive (encouraging that behavior). If rollout $i$ is correct but all rollouts are correct, $\sigma_R = 0$, so the denominator is zero and the advantage is undefined—in practice, GRPO sets the advantage to zero in this case, yielding no learning signal. The same happens when all rollouts are incorrect.

Why this form: the group-relative normalization removes the need for a separately trained value function, which simplifies the training pipeline and eliminates a source of bias. The standard deviation normalization ensures that the scale of advantages is comparable across prompts with different levels of difficulty (a prompt where the model gets 50% correct will have larger raw reward differences than one where it gets 90% correct). The clipping in the denominator via $\epsilon$ handles the edge case where all rewards are identical.

GRPO optimizes a clipped surrogate objective. Let $o_i = (y_{i,1}, \dots, y_{i,T_i})$ be the token sequence for rollout $i$, and define the importance ratio at each token:

$r_{i,t}(\theta) = \frac{\pi_\theta(y_{i,t} \mid x, y_{i,<t})}{\pi_{\theta_{\text{old}}}(y_{i,t} \mid x, y_{i,<t})}$

This ratio measures how much more (or less) likely the updated policy is to produce token $y_{i,t}$ compared to the policy that generated the rollout. The GRPO objective is:

$\mathcal{L}_{\text{GRPO}}(\theta) = \mathbb{E}_{x} \left[ \frac{1}{N} \sum_{i=1}^N \frac{1}{T_i} \sum_{t=1}^{T_i} \min\left(r_{i,t} \hat{A}_i, \text{clip}(r_{i,t}, 1-\varepsilon, 1+\varepsilon) \hat{A}_i\right) - \beta \mathbb{D}_{\text{KL}}(\pi_\theta \| \pi_{\text{ref}}) \right]$

where $\varepsilon$ is the clipping parameter, $\hat{A}_i = \hat{A}_i^{\text{GRPO}}$, and $\mathbb{D}_{\text{KL}}$ is a KL divergence penalty to a reference policy (typically the base model before RLVR training). The paper sets $\beta = 0$ following DAPO (Yu et al., 2025), removing the KL penalty entirely.

What the clipped objective does: the $\min(\cdot, \cdot)$ term ensures that if the advantage is positive, the policy is not updated to increase the probability of a token by more than a factor of $(1+\varepsilon)$ relative to the old policy; if the advantage is negative, the policy is not updated to decrease probability by more than a factor of $(1-\varepsilon)$. This prevents destructively large updates from a single batch.

DAPO modifies GRPO by using asymmetric clipping bounds: $\text{clip}(r_{i,t}, 1 - \varepsilon_{\text{low}}, 1 + \varepsilon_{\text{high}})$ where $\varepsilon_{\text{high}} > \varepsilon_{\text{low}}$. The paper uses $\varepsilon_{\text{low}} = 0.2$ and $\varepsilon_{\text{high}} = 0.28$. The higher upper bound relaxes the constraint on increasing probability for low-probability actions, making it easier for the policy to up-weight rare correct solutions—a design choice that partially addresses the sharpening problem but through a different mechanism than Focal weighting.

CISPO takes a different approach: instead of clipping the product $r_{i,t} \hat{A}_i$, it clips the importance weights directly and uses a REINFORCE-style objective:

$\hat{r}_{i,t} = \text{clip}(r_{i,t}, 1 - \varepsilon_{\text{low}}^{\text{IS}}, 1 + \varepsilon_{\text{high}}^{\text{IS}})$

$\mathcal{L}_{\text{CISPO}}(\theta) = \mathbb{E}_{i,t} \left[ \text{sg}(\hat{r}_{i,t}) \hat{A}_i^{\text{GRPO}} \log \pi_\theta(y_{i,t} \mid x, y_{i,<t}) \right]$

where $\text{sg}(\cdot)$ denotes stop-gradient: the clipped importance weight is treated as a constant coefficient, not differentiated through. The paper uses much wider clipping bounds for CISPO: $\varepsilon_{\text{low}}^{\text{IS}} = 1.0$ and $\varepsilon_{\text{high}}^{\text{IS}} = 5.0$, following Khatri et al. (2025).

The critical property shared by all three methods: the learning signal vanishes when all rewards in a group are identical ($\sigma_R = 0$), because the advantage becomes zero for all rollouts. This occurs when either all $N$ rollouts are correct or all $N$ are incorrect. This property is the starting point for the paper's theoretical analysis: it creates a fundamental link between the group size $N$, the prompt's difficulty $\mu_{\text{pos}}$, and whether or not an update actually occurs.

---

The Categorical Policy Framework

To analyze how RLVR updates redistribute probability mass—particularly between sampled and unsampled correct actions—the paper adopts the categorical policy framework introduced by Hu et al. (2025). This is a simplified model that strips away token-level complexity to focus on the core sampling dynamics.

Consider a policy $p = \text{softmax}(z)$ over a finite action space $\mathcal{A}$, partitioned into correct actions $\mathcal{P}$ and incorrect actions $\mathcal{N} = \mathcal{A} \setminus \mathcal{P}$. The "policy" here is a categorical distribution, not an LLM; the "actions" are abstract correct/incorrect choices, not tokens. Define:

$Q_{\text{pos}} := \sum_{i \in \mathcal{P}} p_i, \quad Q_{\text{neg}} := 1 - Q_{\text{pos}}$

as the total correct and incorrect probability masses respectively. These are the scalar summaries: what fraction of the policy's total probability is allocated to correct versus incorrect actions.

Draw $N$ i.i.d. samples from $p$. Let $A \subseteq \mathcal{P}$ be the set of sampled correct actions, $B \subseteq \mathcal{N}$ the set of sampled incorrect actions, and $U = \mathcal{A} \setminus (A \cup B)$ the unsampled actions (those that did not appear in any of the $N$ draws). Define the sampled probability masses and concentration measures:

$P_{\text{pos}} := \sum_{i \in A} p_i, \quad P_{\text{neg}} := \sum_{i \in B} p_i, \quad A_2 := \sum_{i \in A} p_i^2, \quad B_2 := \sum_{i \in B} p_i^2$

$A_2$ and $B_2$ are second moments (sums of squared probabilities) of the sampled correct and incorrect actions. These measure how concentrated the sampled actions are: if one action has probability 0.9 and another 0.1, the sum of squares is $0.9^2 + 0.1^2 = 0.82$; if they are equally likely at 0.5 each, the sum of squares is $0.5^2 + 0.5^2 = 0.5$. Higher second moments mean the sampled set is dominated by fewer actions.

For unsampled actions, define analogous quantities: $U_{\text{pos},2} := \sum_{i \in U \cap \mathcal{P}} p_i^2$ and $U_{\text{neg},2} := \sum_{i \in U \cap \mathcal{N}} p_i^2$. The batch baseline $S_R$ is the average reward, weighted by the probability mass of distinct sampled actions:

$S_R := R_c P_{\text{pos}} + R_w P_{\text{neg}}$

This is not the same as $\bar{R} = \frac{1}{N} \sum_j R_j$ used in GRPO's advantage computation. $\bar{R}$ treats each sampled rollout equally regardless of whether multiple rollouts land on the same action; $S_R$ weights by the policy's probability mass of each distinct action, which better captures how strongly the baseline pushes unsampled actions.

The one-step logit update (Equation 7). Under a TRPO-style linear surrogate objective and its unbiased Monte Carlo estimate, differentiating the sample surrogate with respect to the logits $z_j$ yields:

$\Delta z_i = \frac{\eta}{N} p_i (R_i - S_R)$

where $\eta$ is the learning rate, $p_i$ is the current probability of action $i$, $R_i$ is its reward ($R_c$ for correct, $R_w$ for incorrect, 0 for unsampled), and $S_R$ is the batch baseline.

What it computes: for each action, the logit update is proportional to the action's current probability times the deviation of its reward from the batch baseline. Actions with reward above the baseline get their logits increased; actions with reward below the baseline get their logits decreased. The factor $p_i$ means that actions with already-high probability get larger absolute updates (because the derivative of the softmax is proportional to $p_i$).

Critical case for unsampled actions: for $i \in U$ (unsampled), $R_i = 0$ (no reward was observed since the action was never drawn). The update becomes $\Delta z_i = -\frac{\eta}{N} S_R \, p_i$. When $S_R > 0$ (reward-positive batch), unsampled actions have their logits pushed downward, with the magnitude proportional to their current probability and the batch baseline. This is the mathematical root of the concentration mechanism: correct actions that happen not to be sampled in a reward-positive batch lose probability, even though they are correct.

The one-step change in total correct mass (Equation 8). Summing the probability changes from Equation 7 over all correct actions yields, after algebraic manipulation:

$\Delta Q_{\text{pos}} = \frac{\eta}{N} \left[ (R_c - S_R) Q_{\text{neg}} A_2 + (S_R - R_w) Q_{\text{pos}} B_2 + S_R (Q_{\text{pos}} U_{\text{neg},2} - Q_{\text{neg}} U_{\text{pos},2}) \right]$

What it computes: the total change in probability mass assigned to all correct actions after one gradient step, as a function of the sampled and unsampled masses and second moments.

The three terms decomposed:

• First term: $(R_c - S_R) Q_{\text{neg}} A_2$. This is always non-negative (since $R_c \geq S_R$ when $S_R$ is computed from the batch, and all other factors are non-negative). It represents mass transferred from the incorrect pool to the correct pool: sampled-correct actions gain probability, drawing from the incorrect mass $Q_{\text{neg}}$, proportional to how concentrated those correct actions are ($A_2$).
• Second term: $(S_R - R_w) Q_{\text{pos}} B_2$. Also always non-negative (since $S_R \geq R_w$ for the observed batch). It represents mass that stays in the correct pool but was at risk of being lost: sampled-incorrect actions lose probability ($R_w - S_R < 0$), and some of that freed mass flows to the correct pool, proportional to the current correct mass $Q_{\text{pos}}$ and the concentration of sampled-incorrect actions ($B_2$).
• Third term: $S_R (Q_{\text{pos}} U_{\text{neg},2} - Q_{\text{neg}} U_{\text{pos},2})$. This is the unsampled coupling term and can be positive or negative. When $S_R > 0$, this term is negative if $Q_{\text{neg}} U_{\text{pos},2} > Q_{\text{pos}} U_{\text{neg},2}$—that is, if the unsampled-correct second moment weighted by incorrect mass exceeds the unsampled-incorrect second moment weighted by correct mass. This term captures the redistribution within the unsampled set.

As $N$ grows, the unsampled second moments $U_{\text{pos},2}$ and $U_{\text{neg},2}$ decay toward zero (more actions get sampled as $N$ increases), so this third term vanishes and $\Delta Q_{\text{pos}}$ becomes unambiguously non-negative. This is the Hu et al. (2025) result: scaling $N$ ensures the total correct mass monotonically increases.

---

The Tail-Miss Probability (Lemma 3.1)

This is the paper's core theoretical contribution: a closed-form expression for the probability that an RLVR update is active (contains both correct and incorrect rollouts) yet misses rare-correct regions entirely.

Consider a single prompt $x$ with success probability $\mu_{\text{pos}}(x)$ (the probability that a single rollout from $\pi_\theta$ is correct) and rare-correct mass $\tau(x)$ (the probability of producing a specific rare-correct solution). By definition, $0 < \tau(x) < \mu_{\text{pos}}(x)$, meaning the rare-correct region is a subset of all correct trajectories, with possibly much smaller probability.

Let $X$ be the number of correct rollouts among $N$ i.i.d. samples. Define the active event:

$\mathcal{A}_N := \{0 < X < N\}$

This is the event that the group contains at least one correct and at least one incorrect rollout, so $\sigma_R \neq 0$ and the advantages are non-zero (the update is "active"). The probability of this event is:

$\Pr(\mathcal{A}_N) = 1 - \mu_{\text{pos}}^N - (1 - \mu_{\text{pos}})^N$

where $\mu_{\text{pos}}^N$ is the probability that all $N$ rollouts are correct, and $(1 - \mu_{\text{pos}})^N$ is the probability that all are incorrect. For $N = 1$, $\Pr(\mathcal{A}_1) = 0$ (a single rollout is always homogeneous). As $N \to \infty$, $\Pr(\mathcal{A}_N) \to 1$ (with enough samples, you almost certainly get both correct and incorrect rollouts, unless $\mu_{\text{pos}} = 0$ or $\mu_{\text{pos}} = 1$).

Let $Y_i = \mathbb{I}[\text{rollout } i \in \mathcal{C}_{\text{rare}}(x)]$ be the indicator that rollout $i$ comes from the rare-correct region. The tail-miss event is:

$\mathcal{B}_\tau := \mathcal{A}_N \cap \left\{\sum_{i=1}^N Y_i = 0\right\}$

The update is active (mixed rewards) but none of the $N$ rollouts came from the rare-correct region.

Lemma 3.1 derives the probability of this event in closed form:

$\Pr(\mathcal{B}_\tau) = (1 - \tau)^N - (\mu_{\text{pos}} - \tau)^N - (1 - \mu_{\text{pos}})^N$

where $\tau$ is the rare-correct mass and $\mu_{\text{pos}}$ is the total correct mass.

Derivation logic (Appendix A): each rollout falls into one of three disjoint regions: the rare-correct region ($\mathcal{C}_{\text{rare}}$) with probability $\tau$, the remaining correct region ($\mathcal{C} \setminus \mathcal{C}_{\text{rare}}$) with probability $\mu_{\text{pos}} - \tau$, or the incorrect region ($\Omega \setminus \mathcal{C}$) with probability $1 - \mu_{\text{pos}}$. The probability that no rollout hits the rare-correct region is $(1 - \tau)^N$. Conditioned on no rare-correct rollouts, the group is inactive (so $\mathcal{B}_\tau$ does not occur) if either all rollouts are correct-but-not-rare (probability $(\mu_{\text{pos}} - \tau)^N$) or all are incorrect (probability $(1 - \mu_{\text{pos}})^N$). Subtracting these from $(1 - \tau)^N$ gives the probability of the tail-miss event.

What it computes: for a given prompt difficulty ($\mu_{\text{pos}}$), rare-correct fraction ($\rho = \tau / \mu_{\text{pos}}$), and group size ($N$), the probability that a training step on this prompt will produce an active update that nonetheless provides zero learning signal for the rare-correct region.

Why this form is important: it makes the non-monotonic dependence on $N$ explicit. Figure 2 plots $\Pr(\mathcal{B}_\tau)$ as a function of $N$ for various combinations of $\mu_{\text{pos}}$ and $\rho$. The shape is consistently an inverted-U:

• At small $N$ (e.g., $N = 2$): $\Pr(\mathcal{B}_\tau)$ is low because $\Pr(\mathcal{A}_N)$ is low—most groups are homogeneous (all correct or all incorrect), so the update is inactive and the policy barely moves. Rare-correct modes are preserved passively through inaction.

• At intermediate $N$ (the exact peak depends on $\mu_{\text{pos}}$ and $\rho$, but is typically in the range $N = 4$ to $N = 64$ for plausible parameter values): $\Pr(\mathcal{B}_\tau)$ reaches its maximum. Updates are frequently active (mixed rewards), but the probability of sampling any specific rare-correct mode is still low because $\tau$ is small. The policy is actively learning to up-weight common correct solutions while the baseline $S_R > 0$ pushes unsampled-correct logits downward (Equation 7). This is the sharpening regime.

• At large $N$: $(1 - \tau)^N \to 0$ as coverage of rare-correct modes improves, so $\Pr(\mathcal{B}_\tau)$ decays toward zero. Both common and rare correct solutions get sampled, the update reinforces all of them, and diversity is preserved through coverage.

The reconciliation of prior work: this framework directly explains why Wu et al. (2025b) found $N = 2$ works well (small-$N$ regime, diversity preserved through inactivity), why He et al. (2025) observed sharpening at $N = 32$ (intermediate-$N$ regime, where $\Pr(\mathcal{B}_\tau)$ peaks), and why Hu et al. (2025) advocate for very large $N$ (large-$N$ regime, diversity preserved through coverage). The contradictions in the literature are not contradictions at all—they are observations of the same non-monotonic curve at different points.

Practical implication: computational constraints typically limit $N$ to the range 2–32 (in this paper, $N = 8$ is the primary experimental setting). This is precisely the intermediate regime where $\Pr(\mathcal{B}_\tau)$ peaks for plausible values of $\mu_{\text{pos}}$ and $\tau$, meaning standard RLVR training operates in the zone of maximum sharpening risk. The paper's solution, F-GRPO, is motivated by the observation that increasing $N$ to escape this regime is too expensive, so we need a way to mitigate the concentration within the computationally feasible regime.

---

Unsampled-Correct Mass Dynamics (Proposition 3.2)

While Lemma 3.1 identifies when rare-correct modes are vulnerable, it does not explain the mechanism by which their probability mass decreases. Proposition 3.2 fills this gap by analyzing how the one-step update affects the subset of correct actions that were not sampled.

Define the unsampled-correct mass:

$Q_{\text{u,pos}} := \sum_{i \in U \cap \mathcal{P}} p_i = Q_{\text{pos}} - P_{\text{pos}}$

This is the total probability mass assigned to correct actions that did not appear in any of the $N$ samples. When $Q_{\text{u,pos}}$ decreases, the model is concentrating its correct probability onto the sampled-correct actions at the expense of unsampled ones—the mathematical signature of sharpening.

Proposition 3.2: Under the one-step surrogate update (Equation 7),

$\Delta Q_{\text{u,pos}} = \frac{\eta}{N} \left[ - S_R \, U_{\text{pos},2} - Q_{\text{u,pos}} \left( (R_c - S_R) A_2 + (R_w - S_R) B_2 - S_R U_2 \right) \right]$

where $U_{\text{pos},2}$ is the sum of squared probabilities of unsampled-correct actions, $U_2$ is the sum of squared probabilities of all unsampled actions, and the other quantities are as defined in Section 2.3.

What it computes: the one-step change in the probability mass of correct actions that were not sampled in the current batch. A negative value means unsampled-correct mass is shrinking.

The two terms decomposed:

Direct drift term: $-S_R U_{\text{pos},2}$. This comes from directly applying Equation 7 to unsampled-correct actions. Since $R_i = 0$ for $i \in U$, the update is $\Delta z_i = -\frac{\eta}{N} S_R p_i$. Through the softmax Jacobian, this translates to a probability change proportional to $-S_R U_{\text{pos},2}$. When $S_R > 0$ (reward-positive batch), this term is negative: the batch baseline directly pushes unsampled-correct logits downward, and the magnitude scales with the concentration of unsampled-correct mass ( $U_{\text{pos},2}$ ). If the unsampled-correct region contains one high-probability action, $U_{\text{pos},2}$ is large and the direct drift is strong; if it contains many low-probability actions, $U_{\text{pos},2}$ is small and the drift is weak.

Normalization coupling term: $-Q_{\text{u,pos}} \left( (R_c - S_R) A_2 + (R_w - S_R) B_2 - S_R U_2 \right)$. This captures the indirect effect through softmax normalization. The factor in parentheses is the total sum $\sum_j p_j \Delta z_j$ (derived in Equation 24), which represents the net change in the system due to all logit updates. This net change is then multiplied by $-Q_{\text{u,pos}}$, the current unsampled-correct mass. The three components of the coupling are:

• $(R_c - S_R) A_2 \geq 0$: sampled-correct actions gain probability, which through normalization draws mass away from all other actions—including unsampled-correct ones. This contributes negatively to $\Delta Q_{\text{u,pos}}$.
• $(R_w - S_R) B_2 \leq 0$: sampled-incorrect actions lose probability, which through normalization donates mass to all other actions—including unsampled-correct ones. This contributes positively to $\Delta Q_{\text{u,pos}}$.
• $-S_R U_2$: when $S_R > 0$, unsampled actions (both correct and incorrect) lose probability, which through normalization donates mass to sampled actions. The interaction with $Q_{\text{u,pos}}$ depends on the sign of $S_R$.

When does $\Delta Q_{\text{u,pos}} < 0$ while $\Delta Q_{\text{pos}} > 0$? Consider a reward-positive batch ($S_R > 0$) on a prompt with high success probability. In this regime:

• The direct drift $-S_R U_{\text{pos},2}$ is negative, actively pushing unsampled-correct mass down.
• The normalization coupling is dominated by the positive $(R_c - S_R) A_2$ term (since many sampled actions are correct in a high-$\mu_{\text{pos}}$ batch), which further drains unsampled-correct mass through renormalization.
• Meanwhile, Equation 8 gives $\Delta Q_{\text{pos}} > 0$ because the first two terms (mass transfer from incorrect to correct pool) outweigh the unsampled coupling.

The consequence: RLVR can increase total correct mass while concentrating it onto the sampled-correct subset, shrinking the probability of correct actions that happen not to be sampled. This is precisely the sharpening phenomenon: the model becomes more accurate on average (higher $Q_{\text{pos}}$), but its correct mass is increasingly dominated by a few high-probability solution strategies that were lucky enough to appear in training batches, while alternative correct strategies that are equally valid but rarer under the current policy get squeezed out.

Connection to group size: as $N$ increases, $U_{\text{pos},2}$ and $U_2$ decay because the probability that any specific action is unsampled decreases exponentially with $N$. At very large $N$, the direct drift term approaches zero and the unsampled coupling becomes negligible, so $\Delta Q_{\text{u,pos}} \approx 0$—unsampled-correct mass is no longer shrinking. But at intermediate $N$, where $U_{\text{pos},2}$ is still substantial and the batch is reward-positive, $\Delta Q_{\text{u,pos}}$ can be significantly negative.

---

F-GRPO: Focal Weighting Construction

The theoretical analysis identifies $S_R > 0$ as the condition driving concentration ($\Delta Q_{\text{u,pos}} < 0$). At the trajectory level, we cannot directly observe $S_R$ because it depends on the distinct sampled-correct probability mass, which requires knowing the policy's internal probabilities. But we can observe the empirical success rate:

$\hat{\mu}_{\text{pos}}(x) := \frac{\bar{R}(x) - R_w}{R_c - R_w} = \frac{X}{N} \in [0, 1]$

where $X$ is the number of correct rollouts in the group of $N$, and $\bar{R}(x) = \frac{1}{N} \sum_{i=1}^N R_i$ is the group mean reward. This is an unbiased estimator: $\mathbb{E}[\hat{\mu}_{\text{pos}}(x)] = \mu_{\text{pos}}(x)$.

The justification for using $\hat{\mu}_{\text{pos}}$ as a proxy. Corollary B.2 (proved in Appendix B) establishes that $\mathbb{E}[S_R \mid X = k]$ is non-decreasing in $k$, the number of correct rollouts. The proof works by conditioning on $X = k$: the $k$ correct rollouts are i.i.d. from the conditional distribution over correct trajectories, and the $N - k$ incorrect rollouts are i.i.d. from the conditional distribution over incorrect trajectories. The probability that any specific correct action is sampled at least once is $1 - (1 - q_{\text{pos}}(o))^k$, which is non-decreasing in $k$. Therefore, $\mathbb{E}[P_{\text{pos}} \mid X = k]$, the expected distinct sampled-correct mass, is non-decreasing in $k$, and $\mathbb{E}[P_{\text{neg}} \mid X = k]$ is non-increasing in $k$. Since $S_R = R_c P_{\text{pos}} + R_w P_{\text{neg}}$ with $R_c > 0$ and $R_w \leq 0$, $\mathbb{E}[S_R \mid X = k]$ is non-decreasing in $k$.

This means that prompts with higher empirical success rate $\hat{\mu}_{\text{pos}} = X/N$ are more likely to be in the $S_R > 0$ regime where the concentration mechanism is active. Therefore, down-weighting updates on prompts with high $\hat{\mu}_{\text{pos}}$ directly targets the dangerous region identified by the theory.

The Focal weight. The paper adopts a functional form inspired by Focal loss (Lin et al., 2017):

$g(x) := (1 - \hat{\mu}_{\text{pos}}(x))^{\gamma}, \quad \gamma \geq 0$

When $\hat{\mu}_{\text{pos}}$ is high (the prompt is easy for the current policy), $g(x)$ is close to 0, strongly suppressing the gradient. When $\hat{\mu}_{\text{pos}}$ is low (the prompt is hard), $g(x)$ is close to 1, leaving the gradient at full strength. The exponent $\gamma$ controls the steepness of the down-weighting: $\gamma = 0$ recovers standard GRPO (no down-weighting); higher $\gamma$ more aggressively suppresses high-success prompts.

Integration with group-relative advantages. The Focal weight is applied by scaling the advantage:

$\hat{A}_i^{\text{F-GRPO}} := g(x) \cdot \hat{A}_i^{\text{GRPO}}$

This single scalar $g(x) \in [0, 1]$ is computed once per prompt and applied uniformly to all $N$ rollouts from that prompt. It is then plugged into the standard GRPO/DAPO/CISPO objective in place of the raw $\hat{A}_i^{\text{GRPO}}$. The Focal-weighted variants are denoted F-GRPO, F-DAPO, and F-CISPO.

What this modification does operationally: for a prompt where 7 out of 8 rollouts are correct ($\hat{\mu}_{\text{pos}} = 0.875$), with $\gamma = 0.5$, the Focal weight is $(1 - 0.875)^{0.5} = \sqrt{0.125} \approx 0.354$. The advantages for all 8 rollouts from this prompt are multiplied by 0.354, reducing the effective learning rate for this prompt by roughly 65%. For a prompt where only 1 out of 8 rollouts is correct ($\hat{\mu}_{\text{pos}} = 0.125$), the weight is $(1 - 0.125)^{0.5} = \sqrt{0.875} \approx 0.935$, leaving the gradient nearly unchanged.

Why this form and not something else:

• Smoothness and monotonicity: $(1 - \hat{\mu}_{\text{pos}})^{\gamma}$ is a smooth, monotonically decreasing function of $\hat{\mu}_{\text{pos}}$, so the down-weighting is continuous and intuitive: harder prompts get more gradient contribution.
• No thresholding: alternatives like hard thresholding (zero out updates when $\hat{\mu}_{\text{pos}} > 0.8$) would introduce discontinuities and require tuning the threshold. The Focal form naturally interpolates.
• Single hyperparameter $\gamma$: the only new hyperparameter is $\gamma$, swept over $\{0.5, 1.0\}$ in the paper's experiments. The method is shown to be robust to this choice (Table 6 shows best results at both $\gamma = 0.5$ and $\gamma = 1.0$ across different configurations).
• Connection to Focal loss: in object detection, Focal loss down-weights the loss contribution of well-classified examples so the model focuses on hard cases. The analogy here is that high-$\hat{\mu}_{\text{pos}}$ prompts are "well-classified" (the model already succeeds on them), and their updates are most likely to drive concentration; down-weighting them redirects gradient budget toward prompts where the model struggles.
• Orthogonality to clipping modifications: Focal weighting modifies the advantage magnitude, not the clipping mechanism or importance weighting. It can be stacked with DAPO's asymmetric clipping or CISPO's direct importance-weight clipping without interference. This is verified empirically in Table 1.

The key insight that makes this work: Focal weighting does not attempt to identify which trajectories within a prompt are rare-correct and which are common-correct. That would require a separate rarity model or trajectory-level reward modification, as in He et al. (2025). Instead, it operates at the prompt level: it identifies prompts that are likely in the $S_R > 0$ regime (high $\hat{\mu}_{\text{pos}}$) and reduces the entire gradient contribution from those prompts. This is a coarser intervention but directly targets the condition that drives the concentration mechanism, without needing to distinguish rare from common trajectories. The effect is that high-success prompts contribute less to the update, reducing the pressure that would otherwise concentrate mass onto whatever correct solutions happened to be sampled, while hard prompts (where the model still needs to improve, and where the $S_R > 0$ condition is less likely or the batch baseline is weaker) receive full gradient contribution.

---

Categorical Simulation Setup (Appendix J)

To validate the theoretical framework in a controlled setting before scaling to LLMs, the paper runs categorical policy simulations following Hu et al. (2025) with one modification to the learning rate.

Setup details: the policy is a softmax distribution over $|\mathcal{A}| = 128,000$ actions. Of these, $10,000$ actions are designated as correct ($\mathcal{A}_+$, reward $+1$), and the remaining $118,000$ are incorrect (reward $-1$). Logits are initialized as:

• One "anchor" correct action: $z_{\text{anchor}} = 5.0$
• All other 9,999 correct actions: $z = 3.0$
• All 118,000 incorrect actions: $z = 0.0$

Under softmax with temperature $\tau = 1$, this yields initial total correct mass $Q_{\text{pos}} \approx 0.63$, anchor probability $p_{\text{anchor}} \approx 4.7 \times 10^{-4}$, and probability $\tau_{\text{leaf}} \approx 6.3 \times 10^{-5}$ for each non-anchor correct action.

Tail-miss probability in this setting (Figure 5): for a typical non-anchor correct action with $\tau = 6.3 \times 10^{-5}$ and $\mu_{\text{pos}} = 0.64$, Lemma 3.1 gives a tail-miss probability that rises steeply for small $N$, plateaus near 1 for intermediate values, and only declines toward zero for $N \gtrsim 2^{15} = 32,768$. At $N = 131,072 = 2^{17}$, $\Pr(\mathcal{B}_\tau) < 10^{-3}$, predicting that only at this very large group size should non-anchor correct actions be reliably preserved.

Training procedure: at each step, sample $N$ actions i.i.d. from the current policy, compute group-relative advantages $\tilde{r}_j = R_j - \frac{1}{N} \sum_k R_k$, and update logits via gradient ascent on $\mathcal{L} = \frac{1}{N} \sum_j \tilde{r}_j p_j$. When Focal weighting is applied, the objective is scaled by $g = (1 - \hat{\mu}_{\text{pos}})^\gamma$. The learning rate is $\eta = 10^{-2}$ (ten times higher than the $\eta = 10^{-3}$ used by Hu et al., 2025; the paper notes that at the lower rate, policy entropy after 1,000 steps remained above 4 even for $N = 65,536$, whereas LLM generation entropy during RLVR training is typically below 1—the higher rate produces dynamics that better reflect the concentration regimes observed in practice). The sweep covers $N \in \{2, 4, \dots, 131,072\}$ and $\gamma \in \{0, 1\}$, running $T = 1,000$ steps per configuration, averaged over 4 random seeds.

Metrics tracked:

• Total correct mass: $Q_{\text{pos}}(t) = \sum_{a \in \mathcal{A}_+} \pi_t(a)$, the sum of probabilities over all correct actions at step $t$.
• Retained positive mass (Equation 26):

$\mathcal{M}_{\text{ret}}(t) = 1 - \frac{\sum_{a \in \mathcal{A}_+} \max(0, \pi_0(a) - \pi_t(a))}{\sum_{a \in \mathcal{A}_+} \pi_0(a)}$

This measures what fraction of initial correct-action probability has not decreased below its starting value. $\mathcal{M}_{\text{ret}} = 1$ means no correct action has lost mass; $\mathcal{M}_{\text{ret}} \approx 0$ means most correct actions have lost mass, with probability concentrated onto a smaller subset.

What this simulation provides: a direct test of whether $\Delta Q_{\text{pos}} > 0$ (total correct mass increases) can coexist with diversity loss ($\mathcal{M}_{\text{ret}}$ decreases), and whether the group-size-dependence predicted by Lemma 3.1 is borne out in the actual training dynamics. The results are discussed in Section 5.1 and Figure 4: yes, both predictions hold, and Focal weighting ($\gamma = 1$) improves $\mathcal{M}_{\text{ret}}$ retention particularly in the intermediate-$N$ concentration zone.

4. Key Insights and Innovations

Innovation 1: The Non-Monotonic Tail-Miss Probability as a Unifying Diagnostic Framework

The paper's most fundamental intellectual contribution is not Focal weighting itself, but the diagnostic framework that the tail-miss probability $\Pr(\mathcal{B}_\tau)$ provides for understanding the group-size dependence of RLVR. Prior to this work, the literature on optimal group size was genuinely contradictory: Wu et al. (2025b) argued $N = 2$ suffices, Hu et al. (2025) advocated scaling $N$ as large as possible, and He et al. (2025) showed $N = 32$ sharpens the distribution. A practitioner reading this literature had no way to reconcile these claims—they appeared to be describing fundamentally different phenomena.

The paper's key conceptual move is to recognize that these findings are not contradictory but are observations of different points on the same non-monotonic curve. Lemma 3.1 derives the closed-form expression $\Pr(\mathcal{B}_\tau) = (1 - \tau)^N - (\mu_{\text{pos}} - \tau)^N - (1 - \mu_{\text{pos}})^N$, which is an inverted-U function of $N$ for all parameter values (Figure 2). This shape directly explains the three regimes:

• Small $N$: $\Pr(\mathcal{B}_\tau)$ is low because $\Pr(\mathcal{A}_N)$ is low—updates are rarely active, so the policy changes slowly and preserves the base model's diversity. This is the Wu et al. (2025b) regime, where $N = 2$ "works" because it hardly trains at all.

• Intermediate $N$: $\Pr(\mathcal{B}_\tau)$ peaks—updates are frequently active but systematically miss rare-correct modes. This is the He et al. (2025) sharpening regime, where pass@1 improves but pass@k degrades.

• Large $N$: $(1 - \tau)^N \to 0$ as coverage improves, so $\Pr(\mathcal{B}_\tau)$ decays. This is the Hu et al. (2025) regime, where scaling $N$ ensures $\Delta Q_{\text{pos}} \geq 0$ and diversity is preserved through coverage.

What makes this a genuine intellectual advance rather than a minor taxonomy is that it reframes the group-size question from "what $N$ is best?" to "what regime are you operating in, and what failure mode does that create?" The answer is not a single optimal $N$ but a diagnosis of the risk profile at whatever $N$ is computationally feasible. The paper does not claim that intermediate $N$ is always bad—it claims that intermediate $N$ is where sharpening risk is maximized, which is a precise, falsifiable statement about the mechanism.

The practical bite of this reframing is substantial. It explains why the $N = 32$ that He et al. (2025) found to sharpen their distribution might be in the "large $N$" regime for a different model or dataset (the paper notes this explicitly in Section 3.1: "in our setup $N = 32$ falls in the large-$N$ regime, whereas for He et al. (2025) it was intermediate. This shift in regime boundaries, determined by $\mu_{\text{pos}}$, $\tau$, and their evolution during training, also explains the smaller degradation on OOD IFEval"). The diagnostic generalizes: the regime boundaries depend on the model's current success probability and the rarity of the modes being considered, so the same $N$ can be safe or dangerous depending on context. This is a fundamentally more useful conceptual tool than a fixed recommendation like "always use $N = 32$."

The validation of this framework in the categorical simulation (Figure 4, Section 5.1) is particularly elegant: rather than testing a single hypothesis, the simulation sweeps $N$ across six orders of magnitude and tracks both $Q_{\text{pos}}$ (total correct mass) and $\mathcal{M}_{\text{ret}}$ (retained positive mass), showing that $\Delta Q_{\text{pos}} > 0$ can coexist with $\mathcal{M}_{\text{ret}}$ collapse—the exact signature predicted by the theory. The three-regime labeling (I: slow growth, diversity preserved; II: concentration zone; III: both metrics high) makes visible a pattern that would be invisible to any analysis tracking only accuracy or only diversity.

A subtle but important contribution here is the introduction of $\mathcal{M}_{\text{ret}}$ itself as a metric. Prior work on RLVR dynamics (Hu et al., 2025) tracked total correct mass $Q_{\text{pos}}$, which measures the aggregate but not the distribution within the correct set. The retained positive mass metric (Equation 26) directly quantifies how much of the initial correct-action probability remains at or above its starting value—a direct measure of diversity loss that is orthogonal to aggregate accuracy. This metric makes the redistribution mechanism empirically observable in a way that was previously unavailable, and it is general enough to apply to any categorical policy analysis beyond the specific setting of this paper.

Innovation 2: The Unsampled-Correct Mass Analysis as a Mechanism for Sharpening

While Lemma 3.1 identifies when rare-correct modes are vulnerable, the paper's second conceptual contribution is the characterization of the mechanism by which their mass actually decreases—Proposition 3.2's decomposition of $\Delta Q_{\text{u,pos}}$ into a direct drift term ($-S_R U_{\text{pos},2}$) and a normalization coupling term. This is a genuine theoretical advance because it answers a question that prior work did not even formulate: under what conditions can total correct mass increase while unsampled-correct mass decreases?

The significance of this question is easy to miss if one only reads the Hu et al. (2025) analysis. That work derived $\Delta Q_{\text{pos}}$ (Equation 8) and showed that scaling $N$ ensures it is non-negative—a result that seems to imply RLVR is well-behaved. But $\Delta Q_{\text{pos}} \geq 0$ says nothing about which correct actions are gaining mass. A policy that assigns 100% probability to a single correct action has $\Delta Q_{\text{pos}} > 0$ (it went from, say, 0.8 to 1.0) but has lost all diversity and will have terrible pass@k for large $k$. The Hu et al. analysis, by aggregating all correct actions, masks this redistribution.

The paper's decomposition of $\Delta Q_{\text{u,pos}}$ makes the redistribution explicit and names its drivers: the direct drift term $-S_R U_{\text{pos},2}$ arises because unsampled actions receive zero reward but are still affected by the baseline subtraction in the advantage computation (Equation 7: $\Delta z_i = -\frac{\eta}{N} S_R p_i$ for $i \in U$), and the normalization coupling term captures how probability gains by sampled-correct actions draw mass away from unsampled-correct ones through the softmax normalization. Both terms are negative when $S_R > 0$, meaning that in reward-positive batches—precisely the batches most common on easy-to-medium prompts—the mechanism actively concentrates mass onto whichever correct solutions happened to be sampled.

What makes this a fundamental rather than incremental advance is that it identifies a structural property of group-relative RLVR, not an implementation detail. The concentration mechanism arises from the interaction of three design choices that are pervasive in RLVR: (1) sampling a finite group of rollouts per prompt, (2) computing a batch baseline from that group, and (3) using that baseline to normalize advantages. Any algorithm with these three properties—which includes GRPO, DAPO, CISPO, and likely future variants—will exhibit the unsampled-correct mass shrinkage characterized by Proposition 3.2. The paper makes this explicit (Section 4.2): "The concentration phenomenon we address arises from the sampling dynamics of group-relative advantage estimation, not from these algorithmic choices [clipping mechanisms, importance weighting]."

This has an important negative implication that the paper does not trumpet but that follows directly: no amount of tuning the clipping parameters, importance-weighting scheme, or KL penalty can eliminate this concentration mechanism, because it is not caused by those choices. Entropy bonuses and KL penalties can slow the rate of concentration by applying uniform pressure against distributional change, but they do not target the specific condition ($S_R > 0$) that drives unsampled-correct mass shrinkage. This explains why the entropy and KL baselines in Appendix G (Table 4) provide only partial mitigation: they are fighting the symptom (narrowing distribution) rather than the cause (the baseline-driven downward drift on unsampled actions in reward-positive batches).

The counterintuitive nature of this finding is itself notable. The intuitive expectation is that RLVR should diversify the policy because it reinforces correct trajectories and penalizes incorrect ones, and there are typically many more distinct correct solution strategies than any single batch captures. The paper shows that the opposite can happen: the baseline subtraction actively penalizes correct strategies that happen not to be sampled, because from the perspective of the advantage computation, an unsampled-correct solution is indistinguishable from noise—it receives zero reward and is pushed downward by $S_R > 0$. The model learns not just to prefer correct over incorrect, but to prefer sampled-correct over unsampled-correct. This is a subtle failure mode that would be easy to miss without the categorical analysis, and Proposition 3.2 provides the mathematical language to diagnose it.

Innovation 3: Prompt-Level Focal Down-Weighting as a Solution Orthogonal to Trajectory-Level Interventions

The paper's third intellectual contribution is the recognition that addressing the concentration mechanism does not require identifying which individual trajectories are "rare"—it can be done at the coarser granularity of entire prompts, using a single observable statistic ($\hat{\mu}_{\text{pos}}$) that is already computed as part of the training loop. This is a conceptual move that distinguishes F-GRPO from all prior work on distribution sharpening.

The dominant approach in the literature—exemplified by He et al. (2025) and Gai et al. (2025)—is to modify trajectory-level rewards or losses to differentially treat rare versus common correct solutions. He et al. (2025) propose an "unlikeliness reward" that up-weights rare-correct trajectories based on their probability under the current policy. This requires estimating per-trajectory rarity, which introduces additional computation and potential estimation error. Gai et al. (2025) apply differential smoothing to correct versus incorrect trajectories, again requiring trajectory-level discrimination.

The paper's key insight is that Corollary B.2—the proof that $\mathbb{E}[S_R \mid X = k]$ is non-decreasing in $k$—means the concentration mechanism is strongest on prompts where the empirical success rate is high, regardless of which specific trajectories within that prompt are rare or common. This obviates the need for trajectory-level rarity estimation. Instead of asking "which of these correct rollouts is rare?" and up-weighting it, F-GRPO asks "is this prompt likely to be in the $S_R > 0$ regime?" and down-weights the entire prompt if the answer is yes.

This is a fundamentally different intervention point from prior work, with practical advantages that go beyond the performance gains in Table 1:

• Computational simplicity: the Focal weight $g(x) = (1 - \hat{\mu}_{\text{pos}})^\gamma$ is computed from $\hat{\mu}_{\text{pos}} = X/N$, which is already tracked for advantage normalization. No additional forward passes, no auxiliary networks, no trajectory-level probability estimation. The paper is explicit that "no additional networks are required" (Section 4.2).

• Single hyperparameter: $\gamma$ is the only new knob, and the paper shows robustness across $\gamma \in \{0.5, 1.0\}$ (Table 6). By contrast, trajectory-level reward modifications typically introduce per-trajectory hyperparameters, and entropy/KL regularizers require trading off regularization strength against task performance.

• Orthogonality to method variants: because Focal weighting modifies the advantage magnitude rather than the clipping mechanism or importance-weighting scheme, it stacks with DAPO's asymmetric clipping and CISPO's direct importance-weight clipping without interference. The consistent gains across all three methods in Table 1 (F-GRPO: +6.2 pass@256; F-DAPO: +3.2; F-CISPO: +3.6) empirically validate this claim of orthogonality.

• No distributional assumptions about rarity: trajectory-level methods like He et al. (2025) implicitly assume that low-probability trajectories under the current policy are rare-correct modes worth preserving. This conflates several things: a trajectory might be low-probability because it is genuinely rare and valuable, or because it is verbose, or because it contains an unusual but irrelevant phrasing. F-GRPO sidesteps this issue entirely by operating at the prompt level.

The connection to Focal loss (Lin et al., 2017) is more than a superficial naming choice. The original Focal loss addressed a specific class imbalance problem in object detection: the vast majority of candidate regions are easy negatives that dominate the loss, swamping the gradient from rare positive examples. By down-weighting well-classified examples, Focal loss focused training on hard cases. The analogy here is precise: in RLVR, high-$\hat{\mu}_{\text{pos}}$ prompts are "well-classified" (the model already succeeds on them), and their updates are most likely to drive concentration; down-weighting them redirects gradient budget toward harder prompts where the model still needs to improve and where the concentration mechanism is weaker. The paper's contribution is recognizing that this domain-agnostic reweighting strategy, originally developed for supervised object detection, transfers naturally to the RLVR setting and addresses the specific failure mode identified by the tail-miss and unsampled-correct analyses.

A subtle but important property of the Focal weight that the paper does not emphasize: it is self-adjusting during training. As the policy improves and $\mu_{\text{pos}}$ increases for some prompts, $\hat{\mu}_{\text{pos}}$ will tend to increase as well, and the Focal weight will automatically reduce those prompts' gradient contribution. This means F-GRPO implicitly implements a form of curriculum learning without any explicit difficulty scheduling: early in training, when the model struggles on most prompts, $\hat{\mu}_{\text{pos}}$ is low for most prompts and Focal weighting has little effect; later in training, as the model masters easy prompts, those prompts are automatically down-weighted and gradient budget shifts to remaining hard prompts. This property is not analyzed in the paper but follows directly from the weight's dependence on $\hat{\mu}_{\text{pos}}$.

Innovation 4: The Three-Regime Taxonomy as a Lens for Characterizing the Pass@1 vs. Pass@k Tradeoff

The paper's final conceptual contribution is the framing of RLVR's effect on model behavior not as a single metric (accuracy improves or degrades) but as a regime-dependent tradeoff between two metrics: pass@1 (single-attempt accuracy) and pass@k (diversity-dependent multi-attempt accuracy). The three-regime taxonomy from Figure 4 makes visible a relationship that prior work treated as a simple tension: "RLVR improves pass@1 but hurts pass@k." The paper shows that this is not a universal tension but a specific property of the intermediate-$N$ regime.

The Tables 2 and 3 results make this concrete. At $N = 2$, pass@256 is highest (75.0 on in-domain math) but pass@1 is lowest (36.2). At $N = 8$, pass@1 improves (37.3) but pass@256 drops sharply (64.1). At $N = 32$, both metrics improve (39.2 / 70.1). F-GRPO at $N = 8$ achieves 38.6 / 70.3, nearly matching the $N = 32$ pass@256 while retaining most of the pass@1 improvement.

What makes this intellectually distinctive is not the empirical observation itself—Yue et al. (2025) and Dang et al. (2025) already documented that RLVR degrades pass@k—but the causal attribution of the tradeoff to a specific, quantifiable sampling phenomenon ($\Pr(\mathcal{B}_\tau)$) rather than a vague "sharpening" tendency. The non-monotonic shape of $\Pr(\mathcal{B}_\tau)$ predicts that pass@1 and pass@256 should move in opposite directions at intermediate $N$ (active updates that miss rare-correct modes will improve the probability of common correct solutions at the expense of rare ones) but in the same direction at small $N$ (slow improvement on both) and large $N$ (improvement on both with diversity preserved). The empirical data in Table 3 qualitatively matches this prediction, lending credence to the causal claim.

The $\Delta$NLL$_{\text{rare}}$ metric in Table 2 is a particularly clever empirical validation of the redistribution mechanism. By tracking the negative log-likelihood of a fixed set of trajectories that were correct but low-probability under the base model, the metric provides a direct window into whether the policy is "forgetting" rare-correct solutions. The ordering $\Delta$NLL$_{\text{rare}}(N=2) < \Delta$NLL$_{\text{rare}}(N=32) < \Delta$NLL$_{\text{rare}}(N=8)$ mirrors the pass@256 ordering—the model at $N = 8$ deviates most from the base distribution on these rare-correct trajectories, consistent with the concentration mechanism being strongest at intermediate $N$. F-GRPO at $N = 8$ achieves an intermediate $\Delta$NLL$_{\text{rare}}$ value (0.46), reflecting reduced concentration relative to the baseline. This is the kind of diagnostic evidence that supports a mechanistic claim rather than merely an empirical correlation.

The broader significance of this reframing is that it provides a vocabulary for discussing RLVR quality beyond scalar accuracy. The three-regime taxonomy gives practitioners a way to diagnose what is happening to their model: if pass@1 is improving but pass@256 is degrading, they are likely in the intermediate-$N$ concentration zone and should consider either increasing $N$ (if computationally feasible) or applying Focal weighting (if not). If neither metric is moving, they are in the small-$N$ regime and should increase $N$ to get any learning. If both are improving, they are in the large-$N$ regime and can be confident that accuracy gains are not coming at the expense of diversity. This diagnostic utility extends beyond the specific methods evaluated in the paper.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All LLM experiments use DeepScaleR (Luo et al., 2025), a challenging dataset of competition-level mathematics problems. The paper filters samples longer than 1024 tokens and removes duplicates with conflicting answers, retaining 39,202 training samples. The system prompt "Please reason step by step, and put your final answer within \boxed{}." is prepended to all inputs. For evaluation, in-domain benchmarks include MATH500 (Hendrycks et al., 2021), AIME24/25 (Art of Problem Solving, 2024a), AMC23 (Art of Problem Solving, 2024b), Minerva Math (Lewkowycz et al., 2022), and Olympiad Bench (He et al., 2024). Out-of-domain benchmarks include GPQA Diamond (Rein et al., 2023) (graduate-level science QA), IFEval (Zhou et al., 2023) (instruction following), and SynLogic (Liu et al., 2025a) (synthetic logical reasoning).

• Base model(s). Three model families at different scales are evaluated: Qwen2.5-7B (Yang et al., 2024a), Qwen2.5-1.5B-Math (Yang et al., 2024b), and Llama-3.2-3B-Instruct (Grattafiori et al., 2024). The primary results are on Qwen2.5-7B, with the other two serving as robustness checks across model families and scales. The choice covers both general-purpose instruction-tuned models (Qwen2.5-7B, Llama-3.2-3B-Instruct) and a math-specialized model (Qwen2.5-1.5B-Math).

• Metrics. The paper reports two primary metrics: pass@1 and pass@256. Pass@k is the probability that at least one of k samples is correct, computed using the unbiased estimator $1 - \binom{n-c}{k} / \binom{n}{k}$ where $n = 256$ is the total number of samples and $c$ is the number of correct samples. Pass@1 measures single-attempt accuracy (sharpened distribution should improve this); pass@256 measures solution diversity (concentration should degrade this). For the categorical simulation experiments (Section 5.1), the paper tracks total correct mass $Q_{\text{pos}}$ and retained positive mass $\mathcal{M}_{\text{ret}}$ (Equation 26). For the group-size comparison (Section 5.3), the paper additionally reports $\Delta$NLL$_{\text{rare}}$, the increase in negative log-likelihood on trajectories that were correct but low-probability under the base model—a direct proxy for whether the policy is deviating from rare-correct solutions (higher values indicate more deviation; Appendix F.2).

• Baselines. The paper compares against three group-relative methods without Focal weighting: GRPO (Shao et al., 2024), DAPO (Yu et al., 2025), and CISPO (Chen et al., 2025a). These are evaluated at $N = 8$ (the primary practical setting) and, for the group-size analysis, at $N \in \{2, 8, 32\}$. Additional baselines include GRPO with entropy bonus (GRPO-$\mathcal{H}$, coefficient 0.001) and GRPO with KL penalty to a reference model (GRPO-KL, coefficient 0.001), both tuned for best average math pass@1 (Appendix G). The categorical simulation baselines follow the setup of Hu et al. (2025) with the learning rate adjustment noted in Appendix J.

• Generation budget / compute accounting. The group size $N$ is the primary unit of compute, since each training step requires $N$ rollouts per prompt and $N \times$ (batch size) forward passes. The paper uses $N = 8$ as the default practical setting, comparing against $N = 2$ and $N = 32$ for the group-size analysis. F-GRPO at $N = 8$ uses exactly the same number of rollouts as GRPO at $N = 8$—the Focal weight is a scalar multiplication on already-computed advantages, adding negligible computational overhead. The paper explicitly claims $4\times$ fewer rollouts when comparing F-GRPO at $N = 8$ against GRPO at $N = 32$ (Section 5.3). Training uses 16 NVIDIA H100 GPUs with FSDP2, the verl framework (Sheng et al., 2024), and sglang (Zheng et al., 2023) for rollout generation. Key training hyperparameters: global batch size 256, mini-batch size 64, learning rate $1 \times 10^{-6}$, 10 training epochs, 1 PPO epoch, sampling temperature 1.0, maximum response lengths of 3072 tokens for Qwen2.5-1.5B-Math and 8192 tokens for other models (Appendix H.2, Table 5).

• Cross-validation / statistical protocol. Checkpoint selection is based on the best average math pass@1 over training epochs. The paper saves checkpoints at the end of each epoch and selects the best baseline checkpoint by average math pass@1, then compares to the best F-GRPO checkpoint obtained with equal or less compute (Appendix H.4). Statistical significance is assessed using a paired m-out-of-n subsampling test (Politis et al., 1999) with $n = 1024$ solutions per problem, $m = 256$ subsample size, and 50,000 subsampling iterations. Differences are considered statistically significant at $p < 0.05$, which equivalent to the 95% confidence interval not containing zero (Appendix I). The Focal exponent $\gamma$ is swept over $\{0.5, 1.0, 2.0\}$ and selected by average in-domain math pass@1 at the best checkpoint (Appendix H.3, Table 6).

Main Quantitative Results

Categorical Simulation: Validation of the Three-Regime Framework (Section 5.1, Figure 4)

The categorical simulation validates the core theoretical predictions in a controlled setting before scaling to LLMs. Figure 4 presents results across group sizes from $N = 2$ to $N = 131,072$.

Total correct mass increases for all group sizes (Figure 4a): $Q_{\text{pos}}$ rises throughout training for every $N$, consistent with Hu et al. (2025). However, this metric alone obscures the diversity dynamics.

Retained positive mass exhibits non-monotonic behavior (Figure 4b): $\mathcal{M}_{\text{ret}}$ behaves dramatically differently across group sizes. At $N = 2$, $\mathcal{M}_{\text{ret}}$ remains near 1—diversity is preserved because most batches are homogeneous and updates are inactive. At intermediate $N$ (e.g., $N = 32$ to $N = 512$), $\mathcal{M}_{\text{ret}}$ collapses toward zero—the concentration zone where total correct mass grows rapidly but correct probability is increasingly dominated by a subset of actions. At $N = 131,072$, $\mathcal{M}_{\text{ret}} \approx 1$ throughout training—diversity is preserved through coverage.

The three regimes are clearly visible (Figure 4c): The final metrics across all $N$ form three distinct regions labeled in the paper: Regime I (small $N$, e.g., $N = 2$–8) where $Q_{\text{pos}}$ grows slowly but $\mathcal{M}_{\text{ret}}$ is high; Regime II (intermediate $N$, shaded region, e.g., $N = 16$–4096) where $Q_{\text{pos}}$ grows rapidly but $\mathcal{M}_{\text{ret}}$ collapses; Regime III (large $N$, e.g., $N = 8192$–131,072) where both metrics are high. This directly validates Lemma 3.1's prediction that intermediate group sizes maximize the probability of active updates that miss rare-correct actions.

Focal weighting improves $\mathcal{M}_{\text{ret}}$ in the concentration zone (dashed lines, $\gamma = 1$): The dashed curves in Figure 4b and the dashed markers in Figure 4c show that Focal weighting partially mitigates the diversity collapse in Regime II. The effect is most visible at intermediate $N$ (e.g., $N = 64$ to $N = 512$), where $\mathcal{M}_{\text{ret}}$ is substantially higher with $\gamma = 1$ than with $\gamma = 0$. At the extremes ($N = 2$ or $N = 131,072$), Focal weighting has minimal effect because $\hat{\mu}_{\text{pos}}$ is rarely high enough to trigger significant down-weighting.

Tail-miss probability predicts the simulation dynamics (Figure 5): For a non-anchor correct action with $\tau = 6.3 \times 10^{-5}$ and $\mu_{\text{pos}} = 0.64$, Lemma 3.1 gives $\Pr(\mathcal{B}_\tau)$ that rises steeply for small $N$, plateaus near 1 for intermediate $N$, and only declines toward zero for $N \gtrsim 2^{15} = 32,768$. At $N = 131,072$, $\Pr(\mathcal{B}_\tau) < 10^{-3}$. This directly predicts that only the very largest group size should reliably preserve probability mass on non-anchor correct actions—and indeed, $N = 131,072$ is the only configuration that maintains $\mathcal{M}_{\text{ret}} \approx 1$ throughout training in Figure 4b.

The paper notes that these categorical simulation results should not be interpreted as quantitative predictions for LLM training—the specific boundaries of the concentration zone depend on the initial distribution. The key insight is the qualitative pattern: intermediate group sizes can exhibit worse diversity than either extreme, and Focal weighting mitigates this.

Group Size Regimes and Focal Weighting: LLM Experiments (Section 5.3, Table 2, Table 3)

Table 2 reports the comparison of GRPO at $N \in \{2, 8, 32\}$ versus F-GRPO at $N = 8$ on Qwen2.5-7B, with the $\Delta$NLL$_{\text{rare}}$ metric as a direct probe of redistribution away from rare-correct solutions.

GRPO exhibits non-monotonic behavior in pass@256 consistent with the three regimes:

• At $N = 2$: highest pass@256 (75.0 in-domain math, 67.3 OOD) but lowest pass@1 (36.2). This is the small-$N$ regime: diversity is preserved through inactivity, but limited learning signal means pass@1 improves slowly.
• At $N = 8$: pass@1 improves to 37.3 but pass@256 drops sharply to 64.1 (in-domain) and 55.9 (OOD). This is the intermediate-$N$ sharpening regime: active updates systematically miss rare-correct modes.
• At $N = 32$: both metrics improve (pass@1 39.2, pass@256 70.1 in-domain; 17.7 / 61.7 OOD). This is the large-$N$ regime where coverage improves.

Per-benchmark results in Table 3 reveal the same pattern on individual benchmarks: AIME25 pass@256 drops from 56.0 at $N = 2$ to 40.8 at $N = 8$, then recovers to 49.5 at $N = 32$; IFEval pass@256 drops from 77.2 at $N = 2$ to 70.3 at $N = 8$, then partially recovers to 71.4 at $N = 32$.

F-GRPO at $N = 8$ matches or exceeds GRPO at $N = 32$ using $4\times$ fewer rollouts: On in-domain math, F-GRPO at $N = 8$ achieves pass@1 38.6 and pass@256 70.3, compared to GRPO at $N = 32$ which achieves 39.2 / 70.1. On OOD benchmarks, F-GRPO at $N = 8$ achieves 19.2 / 63.3, exceeding GRPO at $N = 32$ (17.7 / 61.7). The pass@1 on in-domain shows a small trade-off (38.6 vs. 39.2), but OOD pass@1 improves (19.2 vs. 17.7).

$\Delta$NLL$_{\text{rare}}$ tracks the pass@256 ordering: The ordering is $\Delta$NLL$_{\text{rare}}(N=2) = 0.19 < \Delta$NLL$_{\text{rare}}(N=32) = 0.52 < \Delta$NLL$_{\text{rare}}(N=8) = 0.68$. This means the model at $N = 8$ deviates most strongly from the base distribution on rare-correct trajectories—the signature of the concentration mechanism. F-GRPO at $N = 8$ achieves $\Delta$NLL$_{\text{rare}} = 0.46$, an intermediate value reflecting reduced concentration relative to the baseline at the same group size.

Figure 1(b,c) visualization: The empirical consequences are shown visually: GRPO at $N = 8$ improves pass@1 over $N = 2$ but degrades pass@256 on both AIME 2025 (in-domain) and IFEval (OOD). F-GRPO at $N = 8$ recovers pass@256 while maintaining pass@1, and achieves higher absolute pass@256 than GRPO at $N = 32$ on both benchmarks (52.6 vs. 49.5 on AIME 2025; 75.7 vs. 71.4 on IFEval), using $4\times$ fewer rollouts.

Focal Weighting Across Methods: Main Results (Section 5.4, Table 1)

Table 1 reports the primary evaluation of Focal weighting applied to GRPO, DAPO, and CISPO at $N = 8$ across three model families. All results are presented as pass@1 / pass@256.

Qwen2.5-7B (the main model): Focal weighting produces consistent and substantial pass@256 improvements across all three base methods, with pass@1 remaining stable or improving:

• F-GRPO vs. GRPO: pass@256 improves from 64.1 to 70.3 (+6.2) on in-domain math, and from 55.9 to 63.3 (+7.4) on OOD. Pass@1 improves from 37.3 to 38.6 (in-domain) and 17.1 to 19.2 (OOD). Individual benchmarks: AIME25 pass@256 improves from 40.8 to 52.6 (+11.8); AMC23 pass@256 from 87.3 to 96.3 (+9.0); IFEval pass@256 from 70.3 to 75.7 (+5.4); GPQA pass@256 from 46.2 to 57.3 (+11.1).
• F-DAPO vs. DAPO: pass@256 improves from 69.3 to 72.5 (+3.2) on math, and from 58.4 to 63.6 (+5.2) on OOD. Pass@1 improves from 39.4 to 40.5 (math) and 15.7 to 17.9 (OOD).
• F-CISPO vs. CISPO: pass@256 improves from 73.2 to 76.8 (+3.6) on math, and from 59.0 to 65.9 (+6.9) on OOD. Pass@1 is stable on math (39.5 vs. 39.5) and improves on OOD (14.9 to 18.1). CISPO already achieves the highest baseline pass@256 among the three methods (73.2), and F-CISPO pushes this further to 76.8—the highest value in the table for Qwen2.5-7B.

Statistically significant improvements (underlined in Table 1) are widespread: on Qwen2.5-7B, F-GRPO shows significant improvements on AIME24 pass@256, AIME25 pass@256, AMC23 pass@256, Olympiad pass@256, IFEval pass@1 and pass@256, and GPQA pass@1 and pass@256. F-DAPO and F-CISPO show similarly broad significance patterns (see bold/underline annotations in Table 1).

Qwen2.5-1.5B-Math (smaller, math-specialized): Focal weighting shows consistent pass@256 improvements, though pass@1 shows minor trade-offs in some configurations:

• F-GRPO vs. GRPO: pass@256 improves from 74.4 to 74.5 (+0.1) on math, and from 43.1 to 46.5 (+3.4) on OOD. Pass@1 is essentially flat (36.7 vs. 36.3).
• F-DAPO vs. DAPO: pass@256 improves from 74.3 to 76.0 (+1.7) on math, and from 45.4 to 46.3 (+0.9) on OOD. Pass@1 is stable (37.7 vs. 37.8).
• F-CISPO vs. CISPO: pass@256 improves from 72.9 to 76.1 (+3.2) on math, and from 41.0 to 47.7 (+6.7) on OOD. Pass@1 shows a small decrease (38.9 to 37.4). Notably, F-CISPO achieves the single highest per-benchmark pass@256 in the table: 99.1 on AMC23.

Llama-3.2-3B-Instruct (different family, smaller scale): All methods show pass@256 gains with stable pass@1:

• F-GRPO vs. GRPO: pass@256 improves from 59.9 to 63.4 (+3.5) on math, and from 56.5 to 57.6 (+1.1) on OOD. Pass@1 is unchanged (23.0 vs. 23.0).
• F-DAPO vs. DAPO: pass@256 improves from 54.2 to 62.3 (+8.1) on math, and from 51.3 to 55.4 (+4.1) on OOD. Pass@1 improves slightly (24.3 to 24.8). This is the largest single pass@256 gain in the table (+8.1), and it partially recovers DAPO's baseline underperformance on this model (note that baseline DAPO at 54.2 pass@256 is substantially lower than baseline GRPO at 59.9).
• F-CISPO vs. CISPO: pass@256 improves from 58.0 to 59.7 (+1.7) on math, and from 52.5 to 53.0 (+0.5) on OOD. Pass@1 improves slightly (24.1 to 24.5).

Cross-cutting patterns across all 9 method-model combinations (18 comparisons total): Focal weighting improves both math and OOD pass@256 in all 18 cases. Pass@1 improves or is stable in 15/18 cases. OOD pass@1 improves in 7/9 cases (average +1.1). The average pass@256 improvement across all configurations is +3.5 on math and +3.8 on OOD.

Figure 1(b,c) contextualizes the main results: The comparison in the figure uses specific benchmarks (AIME 2025 for in-domain, IFEval for OOD) rather than averages. GRPO at $N = 8$ improves pass@1 over $N = 2$ on both benchmarks (from 6.7 to ~6.7 on AIME 2025? — the exact numbers for $N = 2$ are in Table 3, not Figure 1; the figure shows the $N = 8$ vs. $N = 32$ comparison). The core visual message is that F-GRPO at $N = 8$ (green bars) achieves higher pass@256 than GRPO at $N = 32$ (blue bars) on both benchmarks, with comparable pass@1, while using $4\times$ fewer rollouts.

Comparison with Entropy and KL Regularization (Section 5.5, Table 4)

Table 4 in Appendix G compares F-GRPO against two common diversity-preserving regularizers on Qwen2.5-7B at $N = 8$:

• F-GRPO: math pass@1 38.6, pass@256 70.3; OOD pass@1 19.2, pass@256 63.3.
• GRPO-$\mathcal{H}$ (entropy bonus, coefficient 0.001): math pass@1 37.8, pass@256 69.5; OOD pass@1 18.7, pass@256 59.9.
• GRPO-KL (KL penalty, coefficient 0.001): math pass@1 37.2, pass@256 72.0; OOD pass@1 19.4, pass@256 60.0.

F-GRPO achieves the best math pass@1 (38.6) and the best OOD pass@256 (63.3). GRPO-KL achieves the best math pass@256 (72.0 vs. F-GRPO's 70.3), but requires maintaining a reference model in memory, increasing computational overhead. GRPO-$\mathcal{H}$ underperforms F-GRPO on all four aggregate metrics.

Per-benchmark, F-GRPO achieves the best or second-best performance on most OOD benchmarks: IFEval pass@256 75.7 (best), SynLogic pass@256 57.0 (second-best to GRPO-$\mathcal{H}$'s 59.9), GPQA pass@256 57.3 (tied for best). On in-domain math, GRPO-KL leads on AIME24 pass@256 (53.4 vs. 46.2) and AIME25 pass@256 (53.7 vs. 52.6), while F-GRPO leads on AMC23 pass@256 (96.3 vs. 95.9) and MATH500 pass@1 (76.2 vs. 76.7—a near tie).

Ablation Studies and Robustness Checks

• Focal exponent $\gamma$ sensitivity (Appendix H.3, Table 6): The paper sweeps $\gamma \in \{0.5, 1.0, 2.0\}$ for each Focal-weighted method and selects by best average math pass@1. Table 6 shows that the best results are attained at both $\gamma = 0.5$ (F-GRPO and F-DAPO for most models) and $\gamma = 1.0$ (F-CISPO for Qwen2.5 models, F-DAPO for Llama). No configuration selects $\gamma = 2.0$, suggesting diminishing returns or oversuppression at higher exponents. The paper states the method is "robust to the choice of $\gamma$" (Appendix H.3).

• Group size sweep with F-GRPO (Table 2, Table 3): Rather than only comparing F-GRPO at $N = 8$ against GRPO at the same $N$, the paper tests GRPO at $N \in \{2, 8, 32\}$ and compares against F-GRPO at $N = 8$. This provides evidence that Focal weighting at a computationally feasible $N$ can recover the diversity benefits of a larger $N$. The $\Delta$NLL$_{\text{rare}}$ ordering across group sizes (0.19 for $N = 2$, 0.68 for $N = 8$, 0.52 for $N = 32$, 0.46 for F-GRPO at $N = 8$) directly supports the mechanistic claim: concentration pressure peaks at intermediate $N$, and Focal weighting reduces it.

• Method-agnostic integration (Table 1, all rows): Focal weighting is tested on three different base methods with different clipping mechanisms: GRPO (symmetric clipping, $\varepsilon_{\text{low}} = \varepsilon_{\text{high}} = 0.2$), DAPO (asymmetric clipping, $\varepsilon_{\text{high}} = 0.28$), and CISPO (direct importance-weight clipping with wide bounds, $\varepsilon_{\text{low}}^{\text{IS}} = 1.0$, $\varepsilon_{\text{high}}^{\text{IS}} = 5.0$). The consistent pass@256 gains across all three (average +4.3 on math, +5.8 on OOD for Qwen2.5-7B) empirically validate the claim that the concentration phenomenon is orthogonal to these algorithmic choices (Section 4.2).

• Model family and scale robustness (Table 1, all panels): The method is evaluated on three model families (Qwen2.5, Llama-3.2) across three scales (1.5B, 3B, 7B), including general-purpose instruct models and a math-specialized model. The consistent pass@256 improvements in 18/18 cases (all method-model combinations across math and OOD) suggest the approach generalizes beyond a single model family.

• $\Delta$NLL$_{\text{rare}}$ as a mechanistic probe (Table 2, Appendix F.2): The paper constructs a fixed set of 1,263 trajectories that were correct but low-probability under the base model (top 1% by base-model NLL among correct trajectories from 800 rollouts per prompt on 256 training prompts). The NLL of this fixed subset under each trained model serves as a direct measure of whether the policy is "forgetting" rare-correct solutions. The ordering $\Delta$NLL$_{\text{rare}}(N=2) < \Delta$NLL$_{\text{rare}}(\text{F-GRPO } N=8) < \Delta$NLL$_{\text{rare}}(N=32) < \Delta$NLL$_{\text{rare}}(N=8)$ quantitatively supports the claim that the concentration mechanism is strongest at intermediate $N$ and that Focal weighting partially mitigates it. This is a non-trivial validation: if the pass@256 improvements were coming from some other mechanism (e.g., better optimization, different regularization), the $\Delta$NLL$_{\text{rare}}$ ordering would not necessarily align with the tail-miss predictions.

• Categorical simulation as a controlled testbed (Figure 4, Appendix J): The categorical simulation strips away the complexities of token-level LLM training (variable-length outputs, credit assignment across tokens, KL penalties, clipping) to isolate the core sampling dynamics. The observation that the three regimes appear in this minimal setting provides strong evidence that the tail-miss mechanism is fundamental to group-relative RLVR and not an artifact of LLM-specific implementation details.

• Learning rate sensitivity in categorical simulation (Appendix J): The paper uses $\eta = 10^{-2}$ rather than the $\eta = 10^{-3}$ used by Hu et al. (2025), noting that at the lower rate, policy entropy after 1,000 steps remained above 4 even for $N = 65,536$, whereas LLM generation entropy during RLVR training is typically below 1. This is a thoughtful design choice: the paper explicitly connects the simulation learning rate to the empirical regime it aims to model, rather than blindly replicating prior work.

• Entropy and KL regularization comparison (Table 4): The explicit comparison with entropy bonus and KL penalty provides evidence that Focal weighting is not merely replicating the effect of these standard regularizers. If it were, the pass@256 values would be similar. Instead, F-GRPO achieves better OOD pass@256 (63.3 vs. 59.9/60.0) and better math pass@1 (38.6 vs. 37.8/37.2) while being simpler (no reference model, no additional loss terms). The GRPO-KL result of higher math pass@256 (72.0) but lower OOD pass@256 (60.0) suggests that KL regularization may be more effective at preserving in-domain diversity but less effective for OOD transfer.

Critical Assessment

Claim 1 from the executive summary: F-GRPO at $N = 8$ improves pass@256 by +6.2 points over GRPO (64.1 → 70.3).

This claim is directly supported by Table 1 (Qwen2.5-7B row, GRPO vs. F-GRPO, average math pass@256 column). The result is consistent across the individual math benchmarks in Table 1: AIME24 (+8.5), AIME25 (+11.8), AMC23 (+9.0), MATH500 (+2.3), Minerva (+0.1), Olympiad (+5.8). However, the Minerva improvement is negligible (+0.1), and MATH500 is modest (+2.3). The claim holds in aggregate but the per-benchmark breakdown shows uneven gains—most of the improvement is concentrated in AIME and AMC, which are competition-level problems where solution diversity matters more than on MATH500 (which may be easier for this model). The paper does not discuss this heterogeneity.

Claim 2 from the executive summary: F-GRPO at $N = 8$ matches or exceeds GRPO at $N = 32$ using 4× fewer rollouts.

Supported by Table 2: math pass@256 70.3 vs. 70.1 (essentially tied), OOD pass@256 63.3 vs. 61.7 (F-GRPO exceeds). Pass@1 shows a small trade-off on in-domain (38.6 vs. 39.2) but an improvement on OOD (19.2 vs. 17.7). The "4× fewer rollouts" claim is accurate ($N = 8$ vs. $N = 32$). However, the comparison is specifically between F-GRPO at $N = 8$ and GRPO at $N = 32$—it does not test whether GRPO at $N = 32$ with Focal weighting (hypothetical F-GRPO at $N = 32$) would perform even better. The paper does not run F-GRPO at $N = 32$, so the upper bound on what Focal weighting plus large groups can achieve is unknown. If F-GRPO at $N = 32$ yields pass@256 of 75+, the claim that $N = 8$ with Focal weighting replaces the need for larger groups would be weaker.

Claim 3 from the executive summary: F-GRPO generalizes as a drop-in modification to DAPO (+3.2 pass@256) and CISPO (+3.6 pass@256).

Supported by Table 1. The gains are smaller than for GRPO (+6.2), but still positive in all cases. On Qwen2.5-7B, F-DAPO improves pass@256 from 69.3 to 72.5 (+3.2) and F-CISPO from 73.2 to 76.8 (+3.6). On Llama-3.2-3B-Instruct, F-DAPO shows the largest gain (+8.1) but this partly reflects DAPO's baseline underperformance on this model (54.2 pass@256 vs. GRPO's 59.9)—Focal weighting is partly recovering from a worse starting point. The "drop-in" claim is accurate: the modification is a single scalar multiplication on the already-computed advantage, with no changes to the underlying training loop, optimizer, or clipping scheme.

Claim 4 (implicit in the theoretical motivation): The pass@256 improvements are caused by mitigating the unsampled-correct mass concentration mechanism, not by some other effect of Focal weighting.

This is the hardest claim to verify empirically from the presented results. The $\Delta$NLL$_{\text{rare}}$ metric (Table 2) supports the mechanistic story: F-GRPO at $N = 8$ (0.46) shows less deviation from the base distribution on rare-correct trajectories than GRPO at $N = 8$ (0.68), consistent with reduced concentration pressure. However, Focal weighting could also improve pass@256 through a different mechanism: by reallocating gradient budget from easy prompts to hard prompts, it might simply improve the model's overall optimization, leading to better pass@256 as a side effect of better training, not specifically because concentration is reduced. The paper does not include an ablation that separates "reduced concentration on easy prompts" from "more training on hard prompts." A useful diagnostic would be to train with Focal weighting on only a subset of prompts (e.g., only the easiest 50%) and compare the diversity metrics, or to compare Focal weighting against a baseline that simply up-samples hard prompts without down-weighting easy ones. Without such an ablation, the causal chain (Focal weight → reduced $S_R$ on high-$\hat{\mu}_{\text{pos}}$ prompts → less unsampled-correct mass shrinkage → higher pass@256) is plausible but not directly tested.

Genuine weaknesses in the experimental design:

• Single training dataset (DeepScaleR). All experiments use DeepScaleR for training. The paper does not test whether Focal weighting transfers to other RLVR training datasets with different difficulty distributions, subject domains (code, science), or reward structures. If DeepScaleR has an unusually high concentration of easy problems, the benefits of Focal weighting might be dataset-specific.

• No combination of Focal weighting with large $N$. As noted above, the paper never tests F-GRPO at $N = 32$ or $N = 64$. If Focal weighting at $N = 32$ provides additive benefits over either intervention alone, the practical recommendation changes from "use Focal weighting at $N = 8$ instead of scaling $N$" to "use both." The paper cannot distinguish these scenarios with the current experiments.

• Limited $\gamma$ sweep. The paper sweeps $\gamma \in \{0.5, 1.0, 2.0\}$ and reports that no method selects $\gamma = 2.0$ as best. But the sweep does not test $\gamma < 0.5$ (e.g., 0.1, 0.25), $\gamma$ between 1.0 and 2.0 (e.g., 1.5), or $\gamma$ values that might be optimal for specific method-model combinations. The "robustness" claim is based on only three tested values.

• No trajectory-level diversity metrics beyond pass@256. Pass@256 is a downstream metric that reflects diversity but does not directly measure it. The paper does not report metrics like the number of distinct correct solutions, entropy of the output distribution, or coverage of solution strategies—metrics that would directly test Proposition 3.2's prediction about unsampled-correct mass shrinkage. The $\Delta$NLL$_{\text{rare}}$ metric is a step in this direction but requires constructing a fixed rare-correct trajectory set, which is itself a heuristic.

• The categorical simulation uses a different learning rate justification that could be tested. The paper increases the learning rate from $\eta = 10^{-3}$ (Hu et al., 2025) to $\eta = 10^{-2}$ to better match LLM entropy regimes. This is a reasonable adjustment, but the paper does not report results at $\eta = 10^{-3}$ to confirm that the three-regime pattern is robust to this choice, nor does it sweep learning rates to show where the qualitative patterns hold.

• Statistical significance is reported but the procedure may be anti-conservative. The m-out-of-n subsampling test reuses the same 1024 solutions per problem with 256-sized subsamples, which may underestimate variance if the solutions are correlated (e.g., many are minor variants of the same approach). The paper reports that "a difference is considered statistically significant if the two-sided $p$-value is less than 0.05" (Appendix I), but does not apply multiple comparison corrections across the many method-benchmark pairs in Table 1. The widespread significance annotations should be interpreted cautiously.

• The OOD improvement story is strong but not explained. F-GRPO consistently improves OOD pass@1 and pass@256 (Table 1). The paper's theoretical framework—Focal weighting reduces concentration on in-domain training prompts—does not directly predict OOD generalization benefits. It is plausible that preserving solution diversity on in-domain math problems transfers to better reasoning strategies on OOD tasks, but the paper does not provide evidence for this mechanism. The OOD results could also be explained by better optimization (less overfitting to common in-domain solutions) rather than diversity preservation per se.

• No comparison with trajectory-level rarity methods. The paper positions Focal weighting as an alternative to trajectory-level interventions like He et al. (2025)'s unlikeliness reward or Gai et al. (2025)'s differential smoothing. However, it does not experimentally compare against these methods. Such a comparison would test whether prompt-level down-weighting is as effective as trajectory-level reweighting, or whether combining both yields further gains. The current results show Focal weighting is useful; they do not show it is better than or complementary to trajectory-level approaches.

Experiments that would have strengthened the paper:

1. F-GRPO at $N = 32$ to test whether Focal weighting and larger groups are additive or redundant.
2. Ablation of Focal weighting on only the easiest prompts to isolate whether the benefit comes from reduced concentration specifically or from redistributed gradient budget generally.
3. Direct diversity metrics (number of distinct correct solutions, solution-level entropy, coverage of MATH solution categories) to validate Proposition 3.2's mechanism directly.
4. Comparison with trajectory-level reweighting methods (He et al., 2025; Gai et al., 2025) to position Focal weighting in the broader landscape of diversity-preserving RLVR interventions.
5. Training on a different dataset (e.g., NuminaMath, MetaMathQA) to test whether the benefits are specific to DeepScaleR's difficulty distribution.
6. Analysis of which prompts are most affected: Does Focal weighting primarily help on prompts where baseline GRPO most aggressively concentrates? Does it ever hurt on prompts where concentration was not a problem? The paper reports only aggregate metrics.

Summary of experimental support: The central empirical claims—that Focal weighting improves pass@256 while preserving pass@1 across multiple methods and model families, and that it can match larger-group performance at a fraction of the cost—are well-supported by Tables 1–3 and Figure 4. The mechanistic claim—that these improvements arise specifically from mitigating the unsampled-correct mass concentration characterized in Proposition 3.2—is supported by the $\Delta$NLL$_{\text{rare}}$ results and the categorical simulation, but alternative explanations (improved optimization through gradient reallocation) are not ruled out. The paper provides strong evidence that Focal weighting works; it provides plausible but incomplete evidence for why it works.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unaccounted For in the Efficiency Claims

The assumption or constraint. The Focal weight $g(x) = (1 - \hat{\mu}_{\text{pos}}(x))^\gamma$ depends on the empirical success rate $\hat{\mu}_{\text{pos}} = X/N$, which is computed from the same $N$ rollouts already used for advantage estimation. The paper rightly claims this adds "no additional networks" and "negligible computational overhead" (Section 4.2) — the computation is essentially free, since $\hat{\mu}_{\text{pos}}$ is already tracked. However, Corollary B.2's monotonicity result — that $\mathbb{E}[S_R \mid X = k]$ is non-decreasing in $k$ — establishes that $\hat{\mu}_{\text{pos}}$ is a valid proxy for the concentration-driving regime, but it does not establish that $\hat{\mu}_{\text{pos}}$ is a reliable estimator at the group sizes where F-GRPO is deployed. At $N = 8$, $\hat{\mu}_{\text{pos}}$ can only take 9 discrete values ($0, 1/8, 2/8, \dots, 8/8$), producing coarse and noisy estimates of the true success probability $\mu_{\text{pos}}$. For a prompt with $\mu_{\text{pos}} = 0.5$, the standard error of $\hat{\mu}_{\text{pos}}$ with $N = 8$ is $\sqrt{0.5 \times 0.5 / 8} \approx 0.177$, meaning the estimated success rate can easily be off by $\pm 0.35$ — a large enough error to shift the Focal weight $g(x)$ by a factor of $\sim 2$ at $\gamma = 1.0$ (compare $(1 - 0.3)^1 = 0.7$ to $(1 - 0.7)^1 = 0.3$).

The consequence. At small $N$, the Focal weight is a noisy function of which specific rollouts happened to be correct, not a stable estimate of prompt difficulty. This introduces gradient variance: the same prompt can receive substantially different Focal weights on different training steps depending on sampling luck, potentially destabilizing training or causing the weight to mis-identify which prompts are truly "high-success" versus merely lucky in a particular batch. The paper does not analyze whether this noise degrades the effectiveness of Focal weighting, or whether the benefits documented at $N = 8$ would be larger or smaller at even smaller group sizes where the noise is worse.

What evidence exists in the paper. None directly. The paper sweeps $N \in \{2, 8, 32\}$ for GRPO (Table 2) but does not report F-GRPO results at $N < 8$. The categorical simulation (Figure 4) uses the true population $\hat{\mu}_{\text{pos}}$ computed from the full categorical distribution (since the simulation knows the exact softmax probabilities at each step), which removes the estimation noise entirely — it is a best-case scenario that the LLM experiments cannot replicate. The paper does not report the variance of $g(x)$ across training steps for the same prompt, nor does it analyze how Focal weighting interacts with the discrete, noisy $\hat{\mu}_{\text{pos}}$ estimates at practical group sizes.

Mitigation status. Not addressed. The paper acknowledges the estimation question only indirectly through the Appendix B proof of monotonicity, which establishes that $\mathbb{E}[S_R \mid X = k]$ is well-behaved in expectation but says nothing about variance at finite $N$. No variance reduction techniques (e.g., exponential moving average of $\hat{\mu}_{\text{pos}}$ across training steps, smoothing across prompts, or Bayesian shrinkage toward a population mean) are proposed or tested. A practitioner deploying F-GRPO at $N = 8$ would need to independently assess whether the noisy $\hat{\mu}_{\text{pos}}$ estimates are undermining the method.

---

The Causal Mechanism Is Not Isolated: Gradient Reallocation Is Confounded with Concentration Mitigation

The assumption or constraint. The paper's theoretical framework (Lemma 3.1, Proposition 3.2) identifies the unsampled-correct mass concentration mechanism as the problem, and Focal weighting as the solution that directly targets it by down-weighting high-$\hat{\mu}_{\text{pos}}$ prompts where $S_R > 0$. However, Focal weighting simultaneously does something else: it reallocates gradient budget from easy prompts to hard prompts. When $g(x)$ suppresses the gradient contribution of prompts with $\hat{\mu}_{\text{pos}} \approx 1$, the effective learning rate on hard prompts (where $\hat{\mu}_{\text{pos}}$ is low and $g(x) \approx 1$) increases relative to easy prompts. This means the pass@256 improvements could arise from either mechanism: (a) reduced concentration pressure on easy prompts (the claimed mechanism), or (b) simply more optimization steps devoted to hard prompts where the model has more room to improve, regardless of concentration dynamics.

The consequence. If the benefit comes primarily from mechanism (b) — better overall optimization through curriculum-like gradient reallocation — then the paper's theoretical narrative (tail-miss probability → unsampled-correct mass shrinkage → Focal weighting as antidote) is a plausible but unverified causal story, and the paper's contribution is empirical (Focal weighting works) rather than mechanistic (Focal weighting works because it mitigates concentration specifically). A practitioner who accepts the mechanistic story might incorrectly conclude that train-time diversity interventions are necessary to preserve pass@k, when a simpler curriculum-learning approach (e.g., up-sampling hard prompts, down-sampling easy ones) might achieve similar pass@256 gains. Conversely, if mechanism (a) is the true driver, trajectory-level rarity reweighting (He et al., 2025) might outperform Focal weighting because it directly targets the rare-correct trajectories rather than bluntly down-weighting entire prompts.

What evidence exists in the paper. The $\Delta$NLL$_{\text{rare}}$ metric (Table 2) provides partial evidence for mechanism (a): F-GRPO at $N = 8$ (0.46) shows less deviation from the base model on rare-correct trajectories than GRPO at $N = 8$ (0.68), consistent with reduced concentration. However, this does not rule out mechanism (b) as a contributing factor, since better overall optimization on hard prompts could also reduce the NLL deviation on the fixed rare-correct set if those rare-correct trajectories tend to appear in harder prompts. The paper does not include an ablation that separates the two mechanisms — for instance, a baseline that reweights prompts by difficulty without the Focal form (e.g., uniform up-weighting of low-$\hat{\mu}_{\text{pos}}$ prompts, or down-weighting high-$\hat{\mu}_{\text{pos}}$ prompts using a step function rather than the smooth $(1 - \hat{\mu}_{\text{pos}})^\gamma$ curve), which would test whether any difficulty-aware reweighting produces similar benefits or whether the specific shape matters.

Mitigation status. Not addressed. The paper does not acknowledge this confound. Section 4.3 states that Focal weighting "reduces gradient contribution from prompts where the concentration mechanism of Section 3.2 is most active," implying a direct causal link, but does not discuss alternative explanations for why reducing gradient contribution from easy prompts might improve pass@256. The comparison with entropy and KL regularization (Table 4) partially addresses whether other diversity-preserving methods achieve similar results (they don't, at least for OOD pass@256), but does not address whether the benefit is specifically from concentration mitigation or from gradient reallocation.

---

All Results Are on a Single Training Dataset (DeepScaleR) with Math-Only Benchmarks

The assumption or constraint. Every model in the paper is trained on DeepScaleR (Luo et al., 2025), a dataset of competition-level mathematics problems. All in-domain evaluation benchmarks are mathematical reasoning tasks (AIME, AMC, MATH500, Minerva, Olympiad). The out-of-domain benchmarks (GPQA, IFEval, SynLogic) test generalization but the training is exclusively on math. The paper does not train or evaluate on code generation, scientific reasoning, multi-step planning, or any non-math domain where RLVR is also widely applied.

The consequence. The paper's central empirical claim — that Focal weighting at $N = 8$ recovers the diversity of larger group sizes — may be specific to the difficulty distribution, reward structure, and solution-type diversity of the DeepScaleR dataset. Competition math problems have several properties that could interact with Focal weighting in ways that do not generalize: (1) they typically admit multiple distinct solution strategies (algebraic, geometric, casework-based), so concentration onto a single strategy is a genuine diversity loss; (2) they have clean, automatically verifiable binary rewards (the answer is either exactly correct or it isn't), making $\hat{\mu}_{\text{pos}}$ well-defined and the $S_R > 0$ regime clearly delineated; (3) the difficulty distribution within DeepScaleR is not documented in the paper, so it is unknown whether the dataset skews toward problems where the base model already has non-trivial success probability (the regime where Focal weighting is most impactful) or toward impossible problems (where $\hat{\mu}_{\text{pos}} \approx 0$ for all prompts, making Focal weighting irrelevant). For domains like code generation, where rewards may be partial (passing some test cases but not all), or where "correctness" is graded on a continuous spectrum, the binary reward assumption underlying both the tail-miss analysis and the $\hat{\mu}_{\text{pos}}$ computation may not hold.

What evidence exists in the paper. The OOD benchmarks (IFEval, GPQA, SynLogic) provide some evidence of generalization — F-GRPO improves OOD pass@256 by +7.4 (GRPO), +5.2 (DAPO), and +6.9 (CISPO) on Qwen2.5-7B (Table 1) — suggesting the diversity preservation benefits transfer at least to instruction-following and science QA. However, OOD evaluation tests whether a model trained on math transfers its reasoning capabilities to other tasks; it does not test whether Focal weighting would work if the training were on those other tasks directly. A practitioner training RLVR on a code dataset cannot assume that because F-GRPO improves OOD transfer from math to IFEval, it will also improve code-generation diversity.

Mitigation status. Not addressed. The paper does not discuss the generalizability of the findings to non-math RLVR training, does not test on code or science training datasets, and does not analyze whether the DeepScaleR difficulty distribution is representative of RLVR training datasets more broadly. The authors do not flag this as a limitation.

---

No Test of Focal Weighting at Larger Group Sizes: The Additivity Question Is Unanswered

The assumption or constraint. The paper frames F-GRPO at $N = 8$ as an alternative to scaling $N$ to 32: "F-GRPO at $N = 8$ matches or exceeds GRPO at $N = 32$, achieving higher pass@256... while using $4\times$ fewer rollouts" (Section 5.3, Figure 1). The natural next question — whether Focal weighting at $N = 32$ or $N = 64$ yields additive benefits beyond either intervention alone — is never tested. The paper runs GRPO at $N \in \{2, 8, 32\}$ without Focal weighting, and runs F-GRPO/F-DAPO/F-CISPO at $N = 8$, but never runs the Focal-weighted variants at any group size other than 8.

The consequence. The practical recommendation is ambiguous. If Focal weighting at $N = 32$ produces pass@256 of, say, 75 on in-domain math (compared to 70.3 for F-GRPO at $N = 8$ and 70.1 for GRPO at $N = 32$), then the correct advice is "use both Focal weighting and the largest group size you can afford." The paper cannot rule this out. Conversely, if Focal weighting at $N = 32$ produces pass@256 no better than at $N = 8$ (saturation), then the $4\times$ compute savings claim is robust. The existing results are consistent with either scenario. This matters because group size is not purely a computational constraint — it is a design choice that practitioners must make. If a team has the budget for $N = 32$, should they use it with or without Focal weighting? The paper provides no guidance.

Additionally, the theoretical framework predicts that at sufficiently large $N$, $\Pr(\mathcal{B}_\tau) \to 0$ and $U_{\text{pos},2} \to 0$ (the unsampled-correct mass approaches zero), so Focal weighting should become unnecessary — the coverage mechanism alone preserves diversity. The paper does not test where this crossover occurs for realistic LLM training. If $N = 32$ is already in the large-$N$ regime for this model-dataset combination (and the Table 2 results suggest it might be, since pass@256 recovers from the $N = 8$ dip), then Focal weighting at $N = 32$ might add nothing. But the paper does not measure this, leaving the boundary between the intermediate and large-$N$ regimes empirically uncharacterized for LLMs.

What evidence exists in the paper. Table 2 includes GRPO at $N = 2, 8, 32$ and F-GRPO at $N = 8$ only. The categorical simulation (Figure 4) sweeps Focal weighting ($\gamma = 1$) across all group sizes from $N = 2$ to $N = 131,072$, showing that Focal weighting has minimal effect at the smallest ($N = 2$) and largest ($N = 131,072$) group sizes — its benefits are concentrated in the intermediate concentration zone. If this pattern transfers to LLMs, F-GRPO at $N = 32$ might indeed show diminishing returns. But the categorical simulation is a simplified model with 128,000 abstract actions, not an LLM; the quantitative mapping from its group-size regimes to LLM group-size regimes is unknown.

Mitigation status. Not addressed. The paper does not flag the absence of Focal weighting at larger group sizes as a limitation, and does not discuss whether the categorical simulation results predict saturation of Focal weighting benefits as $N$ increases in LLM training.

---

No Direct Diversity Metrics: Pass@256 Is a Downstream Proxy, Not a Measurement of the Mechanism

The assumption or constraint. The paper's theoretical contribution is a mechanistic account of how unsampled-correct probability mass shrinks even as total correct mass grows (Proposition 3.2). The empirical validation of this mechanism relies on two metrics: pass@256 (a downstream task metric that reflects diversity indirectly) and $\Delta$NLL$_{\text{rare}}$ (a log-likelihood-based proxy constructed from a fixed set of rare-correct trajectories; Table 2, Appendix F.2). Neither metric directly measures the key quantities from the theory: the unsampled-correct mass $Q_{\text{u,pos}}$, the concentration of probability onto a subset of correct solutions, or the number of distinct correct solution strategies the model can produce.

The consequence. The paper demonstrates that Focal weighting improves pass@256, but it does not demonstrate that pass@256 improves because unsampled-correct mass is better preserved. Pass@256 could increase for reasons unrelated to diversity: better calibration of the model's probabilities (so that when it is correct, it is more confident, making the best-of-256 selection more reliable), improved optimization that increases the absolute number of correct solutions without changing their relative concentration, or reduced overfitting to spurious features of common solutions. The $\Delta$NLL$_{\text{rare}}$ metric (ordering: $N = 2$ at 0.19 < F-GRPO $N = 8$ at 0.46 < $N = 32$ at 0.52 < $N = 8$ at 0.68) is consistent with the concentration story but does not prove it — a model that is simply better optimized overall might assign lower NLL to all trajectories (including the rare-correct ones) without specifically preserving rare-correct modes at the expense of common ones.

Specifically, Proposition 3.2 predicts that unsampled-correct mass shrinks in reward-positive batches. A direct test would measure, for a fixed set of correct solutions, how their aggregate probability changes during RLVR training under different configurations (GRPO vs. F-GRPO, different $N$). The paper does not perform this measurement. The $\Delta$NLL$_{\text{rare}}$ metric measures NLL (not probability mass directly) on a fixed set of 1,263 trajectories (not all correct solutions), selected as the top 1% by base-model NLL (a heuristic for "rareness" that may not align with the $\tau(x)$ in the theory, which is defined by the model's own probability, not the base model's).

What evidence exists in the paper. The categorical simulation (Figure 4b) tracks $\mathcal{M}_{\text{ret}}$, which is a direct measurement of how much initial correct-action probability mass is retained — exactly the quantity Proposition 3.2 is about. In that controlled setting, the mechanism is validated: $\mathcal{M}_{\text{ret}}$ collapses at intermediate $N$, and Focal weighting ($\gamma = 1$) partially mitigates the collapse. However, the paper does not port this analysis to the LLM setting. It does not report $\mathcal{M}_{\text{ret}}$-equivalent metrics for the LLM experiments (e.g., measuring how many distinct correct MATH500 solutions the model produces post-training, or tracking the model's probability mass over a fixed set of correct solutions). The gap between the simulation validation and the LLM validation is bridged only by the $\Delta$NLL$_{\text{rare}}$ metric, which is a weaker and less direct probe.

Mitigation status. Partially addressed through the $\Delta$NLL$_{\text{rare}}$ metric (Table 2) and the categorical simulation (Figure 4). The paper does not claim to have directly measured unsampled-correct mass in LLM training, and the $\Delta$NLL$_{\text{rare}}$ construction is documented transparently (Appendix F.2). However, the paper does not acknowledge the gap between the theory's predictions (about probability mass redistribution) and the empirical measurements (pass@256, NLL on a heuristic rare set) as a limitation. A reader who takes the mechanistic story at face value might overestimate the strength of the empirical evidence for it specifically, as opposed to the empirical evidence that Focal weighting improves pass@256 through some combination of mechanisms.

---

The OOD Improvement Mechanism Is Unexplained: A Gap Between Theory and Transfer Results

The assumption or constraint. The paper's theoretical framework — tail-miss probability (Lemma 3.1) and unsampled-correct mass shrinkage (Proposition 3.2) — is entirely about the training distribution: it explains why, on the prompts the model is being trained on, rare-correct modes get suppressed at intermediate group sizes, and why Focal weighting mitigates this by down-weighting high-$\hat{\mu}_{\text{pos}}$ training prompts. The OOD benchmarks (IFEval, GPQA, SynLogic) are not in the training distribution — they test whether properties acquired during math RLVR training transfer to instruction-following, science QA, and logical reasoning respectively. The theory provides no direct account of why preserving diversity on in-domain math training should improve OOD pass@256 or pass@1.

The consequence. The paper's OOD results — F-GRPO improves OOD pass@256 by +7.4 (GRPO), +5.2 (DAPO), +6.9 (CISPO) on Qwen2.5-7B, and OOD pass@1 improves in 7/9 method-model combinations (Table 1) — are among the strongest empirical results in the paper, but they are the least explained. A practitioner who reads the theory and then sees the OOD gains might reasonably conclude that Focal weighting is simply a better general-purpose training method, but the paper provides no framework for predicting which OOD tasks will benefit. Does preserving diversity on math solutions transfer to IFEval because instruction-following benefits from diverse reasoning strategies? Or because Focal weighting prevents the model from overfitting to math-specific surface patterns that would otherwise interfere with general instruction-following ability? Or because the gradient reallocation effect (mechanism (b) from the second limitation above) improves overall optimization, which coincidentally helps OOD performance? These have different implications for whether Focal weighting would help on, say, code generation OOD tasks.

There is also a potential negative result hiding in the data: on Qwen2.5-1.5B-Math, F-GRPO's OOD pass@1 improves slightly (8.3 vs. 7.9 for GRPO), but F-CISPO's math pass@1 decreases (37.4 vs. 38.9) while its OOD pass@256 improves substantially (47.7 vs. 41.0). This suggests a tradeoff where Focal weighting may sacrifice some in-domain single-attempt accuracy for better diversity and OOD transfer. The paper does not analyze or explain this pattern, which matters for practitioners who must decide whether to prioritize pass@1 or pass@256 for their deployment.

What evidence exists in the paper. Table 1 reports OOD results for all method-model combinations. The group-size comparison in Table 2 includes OOD metrics and shows that the non-monotonic pass@256 pattern (dip at $N = 8$, recovery at $N = 32$) holds for OOD as well as in-domain, and that F-GRPO at $N = 8$ achieves the best OOD pass@256 (63.3) and pass@1 (19.2). The paper notes that "OOD pass@1 also improves in 7/9 cases (average +1.1), suggesting that preserving solution diversity benefits generalization without sacrificing single-attempt accuracy" (Section 5.4). But this is an interpretation, not an explanation — the paper does not provide a mechanism for why preserving diversity on math training helps on non-math tasks.

Mitigation status. Not addressed. The paper treats OOD improvement as a welcome empirical finding that "suggests" a connection between diversity preservation and generalization, but does not develop this theoretically or test alternative explanations. The OOD results are presented as evidence that Focal weighting is broadly useful, not as something the theory predicts. Acknowledging this gap — that the strongest transfer results are the least theoretically grounded — would strengthen the paper's transparency about the scope of its explanatory framework.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not propose a new training algorithm, a new model architecture, or a new benchmark. Its contribution is a diagnostic lens — a way of seeing the group-size-dependent dynamics of RLVR that was previously unavailable — and a surgical intervention that follows directly from that lens. The effect on the field is less about a single technique and more about changing what researchers monitor and optimize for when training reasoning models with RLVR.

The most immediate conceptual shift is the reframing of group size $N$ from a hyperparameter to a regime selector. Before this paper, practitioners chose $N$ based on computational budget or prior convention (GRPO's original paper used $N = 64$, but that was for a specific model/dataset combination). The question was "what $N$ gives the best pass@1?" — a scalar optimization. This paper demonstrates that $N$ sets a risk profile for distribution sharpening: it controls the probability that an active update misses rare-correct modes entirely ($\Pr(\mathcal{B}_\tau)$), and that probability is non-monotonic. A practitioner who only monitors pass@1 during training may observe improvement at intermediate $N$ and assume their training is working well, while pass@256 silently degrades. The paper's key reframing is that RLVR evaluation must be multi-metric: pass@1 and pass@k for large k together diagnose whether the model is in the concentration zone, the inactive zone, or the coverage zone (Figure 4c).

This is not a paradigm shift — the underlying algorithms (GRPO, DAPO, CISPO) are unchanged, and RLVR's basic structure remains intact. But it is a genuinely new diagnostic framework that explains a pattern multiple prior works observed in isolation but could not unify. The Wu et al. (2025b) finding that $N = 2$ suffices, the Hu et al. (2025) finding that scaling $N$ stabilizes learning, and the He et al. (2025) finding that $N = 32$ sharpens the distribution — these appeared contradictory because they were taken as recommendations about the "correct" group size. The paper shows they are observations of the same non-monotonic curve at different points, and the correct question is not "what $N$ is best?" but "given my computational budget and my base model's current $\mu_{\text{pos}}$ distribution, which regime am I in, and what is the corresponding failure mode?" This transforms group-size selection from guesswork into a diagnostic process.

The paper also reconciles a deeper tension in the RLVR literature: whether RLVR primarily sharpens existing capabilities or can expand the reasoning boundary (Yue et al., 2025; Dang et al., 2025 vs. Liu et al., 2025b; Yuan et al., 2025). The tail-miss analysis suggests an answer that is more nuanced than either pole: RLVR can sharpen or diversify depending on the operating regime. At intermediate $N$, the unsampled-correct mass shrinkage characterized by Proposition 3.2 actively concentrates probability onto common solutions — that's sharpening. At large $N$, the coverage mechanism preserves diversity, and with sufficient coverage, RLVR can reinforce rare-correct modes that happen to be sampled — that's genuine capability expansion. The paper does not fully resolve this tension (it doesn't train at the $N > 100$ needed for coverage in realistic settings, and it doesn't demonstrate new-capability acquisition), but it provides the mathematical vocabulary for discussing when sharpening versus expansion occurs, rather than treating it as a binary property of RLVR.

Methodologically, the paper shifts attention from what is sampled to what is not sampled. Prior work on group-relative RLVR focused on the properties of sampled trajectories: how to compute advantages (GRPO vs. DAPO vs. CISPO), how to clip updates, how to assign rewards. This paper's key conceptual move is to notice that the batch baseline $S_R$ creates a downward drift on all unsampled actions, regardless of their correctness (Equation 7: $\Delta z_i = -\frac{\eta}{N} S_R p_i$ for $i \in U$). This is a structural property of group-relative advantage estimation that no amount of tuning the clipping parameters or reward function can eliminate. It means that the concentration mechanism is fundamental to the method class, not an accident of a specific implementation. This insight should refocus research attention: rather than developing ever-more-sophisticated clipping schemes (which modify how sampled trajectories' updates are constrained), the priority should be understanding and mitigating the baseline-driven drift on unsampled actions.

The practical consequence for the field is likely a re-evaluation of default group sizes in RLVR pipelines. The paper's result that GRPO at $N = 8$ degrades pass@256 compared to $N = 2$ (64.1 vs. 75.0 on in-domain math; Table 2) should give pause to anyone deploying RLVR at common intermediate group sizes (8–32) without monitoring diversity metrics. The demonstration that F-GRPO can recover most of that degradation (+6.2 pass@256) at zero additional computational cost makes the intervention easy to adopt, and the framework's prediction that intermediate $N$ maximizes sharpening risk applies regardless of whether Focal weighting is used. I expect that future RLVR papers will routinely report pass@k for multiple k as a diagnostic, much as the GAN literature eventually adopted FID alongside visual inspection, and that group-size selection will be informed by explicit estimation of $\mu_{\text{pos}}$ distributions rather than convention.

Finally, the paper makes the "unsampled-correct" concept empirically observable through the $\Delta$NLL$_{\text{rare}}$ metric (Table 2). This is a small but important methodological contribution: prior work could only observe sharpening through its symptoms (pass@k degradation), but $\Delta$NLL$_{\text{rare}}$ probes the mechanism directly by tracking whether the policy is deviating from rare-correct solutions that existed in the base model. The fact that the ordering $\Delta$NLL$_{\text{rare}}(N=2) < \Delta$NLL$_{\text{rare}}(N=32) < \Delta$NLL$_{\text{rare}}(N=8)$ mirrors the pass@256 ordering (a dip at $N = 8$ with partial recovery at $N = 32$) provides evidence that the two are causally linked — exactly the kind of diagnostic that turns a theoretical mechanism into an empirically grounded explanation. Future work that adopts this metric (or variants of it) will be able to distinguish between diversity loss caused by the unsampled-correct drift mechanism and diversity loss from other sources (e.g., reward hacking, optimization instability).

Follow-Up Research This Work Enables

Connecting Focal weighting benefits to the unsampled-correct drift mechanism through a "gradient reallocation" ablation. The paper demonstrates that Focal weighting improves pass@256, but cannot distinguish whether the improvement comes from reduced concentration pressure on easy prompts (the claimed mechanism) or from simply spending more gradient budget on hard prompts (an alternative explanation). A clean ablation would compare three conditions at $N = 8$: (a) baseline GRPO, (b) F-GRPO (down-weights high-$\hat{\mu}_{\text{pos}}$ prompts), and (c) a "hard prompt up-weighting" baseline that increases the learning rate on low-$\hat{\mu}_{\text{pos}}$ prompts without down-weighting high-$\hat{\mu}_{\text{pos}}$ ones (e.g., multiply advantages by $1 + \alpha(1 - \hat{\mu}_{\text{pos}})$ rather than $(1 - \hat{\mu}_{\text{pos}})^\gamma$). If condition (c) matches F-GRPO's pass@256 while showing higher $\Delta$NLL$_{\text{rare}}$ (more deviation from base-model rare solutions), the concentration-mitigation story is validated. If condition (c) matches both pass@256 and $\Delta$NLL$_{\text{rare}}$, the benefit is from gradient reallocation, and the unsampled-correct mechanism — while theoretically real — is not the primary driver of the empirical gains. Such an experiment would also test whether the Focal shape specifically matters or whether any difficulty-aware reweighting suffices, which directly informs whether practitioners should use the $(1 - \hat{\mu}_{\text{pos}})^\gamma$ form or could adopt a simpler scheme.

Direct measurement of unsampled-correct mass shrinkage in LLM training. The categorical simulation (Figure 4b, Appendix J) directly validates Proposition 3.2 by tracking $\mathcal{M}_{\text{ret}}$, the fraction of initial correct-action probability that has not decreased. The paper does not port this analysis to the LLM setting. A natural follow-up would track, during RLVR training, a fixed set of distinct correct solutions (e.g., clustered by solution strategy using embedding similarity or by final answer equivalence after deduplication) and measure how their aggregate probability mass evolves under GRPO vs. F-GRPO at different $N$. This requires computing the policy's log-probability for each solution at each checkpoint, which is computationally expensive but feasible for a modest held-out set. The key prediction is that under GRPO at $N = 8$, rare-correct solutions (low base-model probability) should lose mass relative to common-correct solutions, and F-GRPO should partially arrest this loss. If this pattern holds, it provides a direct confirmation that the mechanism identified in Proposition 3.2 operates in LLM-scale training, not just in categorical simulations. If it does not hold, the pass@256 improvements from Focal weighting must come from some other mechanism (e.g., improved credit assignment across tokens, reduced overfitting), which would redirect theoretical attention.

Does Focal weighting at $N = 32$ or $N = 64$ yield additive benefits, or does the coverage mechanism dominate? The paper's core practical claim is that F-GRPO at $N = 8$ can match GRPO at $N = 32$ using 4× fewer rollouts. But it never tests F-GRPO at $N = 32$. The categorical simulation (Figure 4c) shows that Focal weighting's benefits are concentrated in the intermediate-$N$ concentration zone and diminish at very large $N$ — but the mapping from categorical group sizes to LLM group sizes is unknown. A natural experiment would train F-GRPO at $N = 32$, $N = 64$, and $N = 128$ and measure pass@1, pass@256, and $\Delta$NLL$_{\text{rare}}$. If F-GRPO at $N = 32$ achieves, say, pass@256 of 75+ on in-domain math (compared to 70.3 for F-GRPO at $N = 8$ and 70.1 for GRPO at $N = 32$), then the practical recommendation becomes "use Focal weighting with the largest $N$ you can afford" rather than "use Focal weighting at $N = 8$ instead of scaling $N$." If F-GRPO at $N = 32$ shows no improvement over $N = 8$, the paper's claim of a 4× compute savings is robust. In either case, knowing the answer would directly inform how RLVR practitioners should allocate their rollout budget. The experiment would also characterize empirically where the boundary between the intermediate-$N$ and large-$N$ regimes falls for realistic LLM training — a question the paper's theory raises but does not answer quantitatively.

Does Focal weighting transfer to RLVR training on non-math domains with non-binary or partial rewards? All experiments in the paper use competition mathematics problems with binary correctness rewards (the answer is either exactly right or exactly wrong). The theoretical analysis — Lemma 3.1, Proposition 3.2, the $S_R > 0$ regime identification — assumes binary rewards. In code generation, rewards are often partial (e.g., fraction of test cases passed) or continuous (e.g., functional correctness scores). In multi-step reasoning with process rewards, correctness is evaluated at the step level rather than only at the outcome level. A natural test would be to apply F-GRPO to RLVR training on a code generation dataset (e.g., using the SWE-bench or APPS training splits with test-case-based rewards) and measure whether pass@k improves. The key question: does $\hat{\mu}_{\text{pos}}$ remain a valid proxy for the $S_R > 0$ regime when rewards are non-binary? With partial rewards, a prompt where all rollouts get 0.8/1.0 might have $\hat{\mu}_{\text{pos}} \approx 0.8$ (if thresholded) but $S_R$ substantially below $R_c$, weakening the drift on unsampled actions. If Focal weighting provides no benefit in this setting, it would delineate the scope of the method to binary-reward RLVR. If it provides benefits, the theoretical framework would need to be extended to account for non-binary rewards — a natural next step for the categorical analysis.

Focal weighting combined with trajectory-level rarity interventions: complementary or redundant? The paper positions Focal weighting as orthogonal to trajectory-level rarity reweighting methods like He et al. (2025)'s unlikeliness reward and Gai et al. (2025)'s differential smoothing (Section 6). These methods modify per-trajectory rewards or losses to up-weight rare-correct trajectories; Focal weighting modifies per-prompt gradient contribution. They operate at different granularities and target different aspects of the problem (rarity identification vs. baseline-driven drift). A natural experiment: train with F-GRPO + unlikeliness reward and compare to either method alone on pass@1, pass@256, and diversity metrics. If the combination outperforms both individually, it suggests the two mechanisms (unsampled-correct drift from baseline subtraction, and insufficient learning signal for rare trajectories) are distinct and additive — and the optimal RLVR pipeline would incorporate both prompt-level and trajectory-level diversity interventions. If the combination shows no improvement over the better of the two, it suggests they address the same underlying phenomenon through different mechanisms and one suffices, which would simplify the intervention space for practitioners.

Characterizing the $\hat{\mu}_{\text{pos}}$ distribution's evolution during training to predict when the concentration zone is entered. The paper's three-regime framework is static: for a given $N$, $\mu_{\text{pos}}$, and $\tau$, it predicts $\Pr(\mathcal{B}_\tau)$. But $\mu_{\text{pos}}$ changes during training — as the policy improves, prompts that were hard become medium, and prompts that were medium become easy. This means a model can drift into the concentration zone during training without changing $N$: early in training, most prompts have low $\mu_{\text{pos}}$ (the model struggles), so $S_R$ is rarely positive enough for strong concentration. Later, as the model masters some prompts, $\hat{\mu}_{\text{pos}}$ rises and the concentration pressure intensifies on those prompts. F-GRPO partially addresses this through the self-adjusting property of the Focal weight (as $\hat{\mu}_{\text{pos}}$ increases, $g(x)$ automatically decreases), but the dynamic is not analyzed. A valuable diagnostic study would track the distribution of $\hat{\mu}_{\text{pos}}$ across prompts at each training epoch, along with pass@1 and pass@256, to identify the point where concentration begins to dominate. This would enable dynamic schedule adjustment: when the fraction of prompts with $\hat{\mu}_{\text{pos}} > 0.75$ crosses a threshold, increase $\gamma$, increase $N$, or switch to a diversity-preserving objective. Such a study would transform Focal weighting from a fixed hyperparameter choice into an adaptive training intervention, and would provide the monitoring tools that practitioners need to diagnose concentration in their own training runs.

Practical Applications and Downstream Use Cases

Cost-efficient RLVR training pipelines where pass@k matters at deployment. Many production LLM systems use best-of-N sampling or majority voting at inference time (e.g., generating 64 candidate answers to a math problem and selecting the most common one, or running verifier-guided search over 256 candidates). For these systems, training-time pass@256 is at least as important as pass@1 because the deployment strategy directly depends on the model's ability to produce diverse correct solutions. The paper's result that baseline GRPO at $N = 8$ degrades pass@256 from 75.0 (at $N = 2$) to 64.1 on in-domain math (Tables 2–3) means that a team deploying standard GRPO at a typical practical group size is silently reducing their model's test-time compute scaling capability. F-GRPO recovers to 70.3 pass@256 at the same $N = 8$, representing a +6.2 point improvement with zero additional inference cost. For a deployment that draws 256 samples per query and uses majority voting, a 6-point pass@256 improvement translates directly to a 6-percentage-point improvement in end-to-end accuracy. The compute savings claim is also practically meaningful: F-GRPO at $N = 8$ matches GRPO at $N = 32$ on pass@256 (70.3 vs. 70.1 on in-domain math, 63.3 vs. 61.7 on OOD; Table 2), meaning a team can achieve the same downstream accuracy with 4× fewer training rollouts per step. For an organization training on 39,202 prompts with batch size 256, using 16 H100 GPUs, reducing $N$ from 32 to 8 cuts the total training FLOPs by approximately 4× — from roughly 32 rollouts × 39,202 prompts × 10 epochs × (generation cost per rollout) to 8 rollouts × the same. If generation dominates the training time (as is typical for long-context reasoning), this is a substantial wall-clock time reduction.

On-device or edge deployment of small models that need test-time scaling. The paper evaluates on Qwen2.5-1.5B-Math and Llama-3.2-3B-Instruct, showing consistent Focal weighting benefits at these smaller scales (Table 1). For edge deployment scenarios — running a 1.5B model on a phone or a 3B model on a laptop — test-time compute scaling (sampling many answers and selecting the best) is often the only way to approach the accuracy of larger models, since the hardware cannot run a 70B model at interactive latency. The paper's finding that baseline GRPO at $N = 8$ can degrade pass@256 relative to $N = 2$ (the inactive regime) means that standard RLVR training of these small models may actually be reducing their deployment-time effectiveness. F-GRPO provides a drop-in fix: on Qwen2.5-1.5B-Math, F-GRPO achieves pass@256 of 74.5 (vs. GRPO's 74.4 — a small in-domain gain) but OOD pass@256 of 46.5 (vs. 43.1, +3.4), and F-CISPO achieves math pass@256 of 76.1 (vs. 72.9, +3.2) and OOD pass@256 of 47.7 (vs. 41.0, +6.7). For a 1.5B model deployed on-device for math tutoring or science Q&A, a 3–7 point pass@256 improvement on out-of-domain tasks (GPQA, IFEval) is a meaningful real-world accuracy improvement — and it comes at zero additional training or inference cost. This use case is particularly compelling because on-device models cannot increase $N$ at training time (the group size is constrained by memory), so they are forced into the intermediate-$N$ concentration zone. Focal weighting is one of the few diversity-preserving interventions that does not require more memory or more rollouts.

Self-improvement and synthetic data generation pipelines that depend on solution diversity. A growing paradigm in LLM post-training is to use RLVR to improve a model, then use the improved model to generate synthetic training data for further fine-tuning (the STaR/ReST approach; Zelikman et al., 2022; Singh et al., 2024). The quality and diversity of this synthetic data depend directly on the pass@k of the generator model — if the trained model produces only a narrow set of correct solutions, the synthetic dataset will lack the diversity needed to teach the next generation of models alternative reasoning strategies. The paper's finding that standard RLVR at intermediate $N$ concentrates probability onto common solutions (manifested as degraded pass@256) implies that RLVR-trained models used as data generators may produce lower-diversity training sets than base models. F-GRPO directly addresses this: by mitigating concentration, it preserves the solution diversity needed for high-quality synthetic data generation. The OOD transfer results (F-GRPO improves OOD pass@256 by +7.4 over GRPO on Qwen2.5-7B; Table 1) suggest that the diversity benefits are not limited to the training domain, which is important for self-improvement pipelines that aim to generalize. A concrete deployment scenario: a team trains a 7B model on competition math with F-GRPO at $N = 8$, then uses it to generate 256 solutions per training prompt, filters for correctness, and fine-tunes the next model iteration on the resulting diverse correct solutions. The F-GRPO-trained generator would provide both higher coverage of correct solutions (better pass@256) and more diverse strategies within the correct set, directly improving the quality of the next generation's training data, without requiring the 4× increase in training compute that $N = 32$ would demand.

</response>