BroRL: Scaling Reinforcement Learning via Broadened Exploration

ArXiv: 2510.01180

🎯 Pitch

Training longer on RLVR eventually hurts performance, but BroRL shows that scaling rollouts to hundreds per prompt instead revives saturated models and drives continuous gains. The trick: broader exploration dodges a theoretical failure mode where unsampled tokens can silently drain probability from correct answers.

---

1. Executive Summary

This paper introduces BroRL, a complementary paradigm for scaling Reinforcement Learning with Verifiable Rewards (RLVR) that broadens exploration by increasing the number of rollouts per example to hundreds or thousands, rather than simply extending the number of training steps. Working with a 1.5B-parameter model based on the ProRLv2 checkpoint across math, code, science, and reasoning benchmarks, BroRL scales the rollout size N from 16 to 512—operationalizing broader per-prompt exploration that systematically mitigates a theoretically identified "unsampled coupling" term in the policy update. The approach revives models saturated after 3,000 ProRL training steps and achieves continuous improvement, reaching state-of-the-art 1.5B results (63.03 on Math, 54.20 on Code, 63.09 on Reasoning Gym) while nearly doubling generation throughput (72.4 vs. 36.5 samples/second) and reducing wasted computation through higher dynamic sampling pass rates (62% vs. 41%), establishing that rollout-size scaling is a more effective and compute-efficient axis for RLVR than step-count scaling, particularly when models have already plateaued under prolonged training.

2. Context and Motivation

The Core Problem: RLVR Training Saturates, and We Don't Know Why

The fundamental question this paper tackles is: when Reinforcement Learning with Verifiable Rewards (RLVR) plateaus after thousands of training steps, is this a fundamental limit of the approach, or is it an artifact of insufficient exploration? This matters because RLVR has become a central methodology for training reasoning models — the family of systems that includes DeepSeek-R1 (Guo et al., 2025) and OpenAI's o-series models (Jaech et al., 2024) — yet practitioners consistently observe diminishing returns and eventual performance stagnation when simply training longer.

The paper's motivating observation is empirical and specific: the ProRL approach (Liu et al., 2025a; Hu et al., 2025b) demonstrated meaningful gains from extending RLVR training to thousands of steps, but after roughly 3,000 steps on a 1.5B-parameter model, performance plateaus across math, code, and reasoning benchmarks. Table 1 quantifies this: ProRL with N=16 rollouts per prompt achieves 62.08 Math after +225 additional steps, then degrades to 62.02 after +535 steps — more training actively hurts. The Reasoning Gym score similarly drops from 62.10 to 61.45. This pattern — initial gains followed by saturation and regression — is practically devastating: it means that allocating additional compute to longer training is not merely inefficient but counterproductive. Yet the field lacked a clear explanation for why this happens, or whether a different allocation of compute could break through the ceiling.

The gap this paper addresses is therefore threefold: (1) a theoretical gap — no formal framework explains why RLVR updates become unstable or negative under prolonged training; (2) an empirical gap — it is unknown whether the plateau represents a fundamental capability limit or a correctable pathology of the training procedure; and (3) a practical gap — practitioners scaling RLVR need guidance on which axis of compute allocation (more steps vs. more samples per step) yields better returns.

Why This Matters: The Scaling Law Parallel and Practical Stakes

The paper positions this problem against the backdrop of the broader scaling laws literature. Kaplan et al. (2020) and Hoffmann et al. (2022) established that pretraining performance follows predictable power-law improvements with respect to model parameters, data quantity, and compute — insights that directly guided the allocation of billions of dollars in training compute. But for RLVR, no analogous scaling framework exists. We don't know whether compute is better spent on more training steps, more rollouts per step, larger models, or some combination. The ProRL line of work (Liu et al., 2025a) began investigating the step-count axis and found it partially effective, but the saturation at ~3K steps leaves open the question of whether a different axis could continue scaling.

This is theoretically significant because it touches on a fundamental property of reinforcement learning: the exploration-exploitation tradeoff. A small number of rollouts per prompt (N=16 in standard ProRL) means the model sees a relatively narrow slice of its own output distribution at each update step. If the unsampled portion of the action space contains high-quality solutions that were never generated, the RL update has no way to discover or reinforce them — it can only reweight what it saw. This is a form of exploration bottleneck that is distinct from the capacity limitations that pretraining scaling laws address. Understanding whether this bottleneck is the root cause of RLVR saturation would clarify whether the limits we observe are algorithmic (fixable with better training recipes) or fundamental (requiring larger models or different approaches entirely).

The practical stakes are high. RLVR is the dominant paradigm for post-training reasoning improvements in the current generation of frontier models. If the observed plateaus are caused by insufficient exploration — and can be overcome by scaling rollout size — then organizations training reasoning models can continue extracting gains from existing model architectures by reallocating compute budgets. Conversely, if the plateaus reflect fundamental capacity limits, then progress requires scaling pretraining, which is orders of magnitude more expensive. The paper's demonstration that BroRL nearly doubles hardware throughput (72.4 vs. 36.5 samples/second, Table 2) adds a compelling practical dimension: scaling rollout size is not only theoretically motivated but also better aligned with GPU hardware characteristics, making it a double win for real-world deployment.

Prior Approaches and Where They Fall Short

Prolonged RL (ProRL) and step-count scaling. The most directly relevant prior work is the ProRL approach (Liu et al., 2025a; Hu et al., 2025b), which extended RLVR training to thousands of steps using a PPO-based algorithm with REINFORCE++-style advantage normalization, dynamic sampling to filter trivial trajectories, periodic policy resets, Clip-Higher exploration, and truncated importance sampling. ProRL demonstrated that RL scaling laws exist — performance improves log-linearly with training steps — but acknowledged diminishing returns. The key limitation is that ProRL treats step count as the primary scaling axis while keeping rollout size fixed at a modest value (N=16). The BroRL paper builds directly on ProRL's framework (using the same base algorithm, dynamic sampling, and reference resets) but argues that step-count scaling alone leaves an exploration bottleneck unaddressed: if each step only explores 16 rollouts per prompt, the unsampled portion of the action space remains large, and the update signal may be dominated by sampling noise rather than genuine policy improvement.

GRPO, RLOO, REINFORCE++, DAPO, and related algorithms. The broader RLVR landscape includes several algorithmic variants (Shao et al., 2024; Ahmadian et al., 2024; Hu et al., 2025a; Yu et al., 2025), all of which focus on how to compute the policy gradient from sampled rollouts — advantage estimation, clipping, normalization, and off-policy correction. These works optimize the quality of the update given a fixed set of rollouts, but none systematically investigate what happens when the number of rollouts per prompt is dramatically increased. BroRL's contribution is orthogonal: it addresses how many rollouts to collect, not how to process them, and shows that this axis has independent and substantial effects on training stability and final performance.

Standard RL scaling. In the broader reinforcement learning literature, increasing the number of samples per update (often called the batch size or rollout count) is a well-known technique for reducing gradient variance and improving convergence. However, this principle has not been formally connected to the specific dynamics of RLVR for language model reasoning, where the action space is the vocabulary (tens of thousands of tokens), the reward structure depends on sequence-level correctness rather than per-token immediate rewards, and the policy is a pretrained language model with complex token-level probability distributions. The BroRL paper bridges this gap by deriving the effect of rollout size N on the probability mass of correct tokens in the specific context of RLVR with token-level logit updates.

Knowledge shrinkage and negative RL updates. Wu et al. (2025) identified a related phenomenon: RLVR training can cause "knowledge shrinkage," where the probability of correct tokens decreases — the model forgets capabilities it previously possessed. This is relevant because it provides empirical evidence that RLVR updates are not always beneficial, which the BroRL paper's theory explains as a consequence of the unsampled coupling term in Theorem 1. The theoretical connection is that when N is small, the unsampled term can be negative and large enough to outweigh the positive contributions from sampled tokens, causing net regression on some correct tokens. BroRL's contribution is to provide the formal mechanism for this shrinkage and to demonstrate that scaling N eliminates it.

Dynamic sampling and instance filtering. The dynamic sampling technique (Yu et al., 2025) filters out prompts where all generated samples are correct or all are incorrect, keeping only "training-rich" prompts with mixed outcomes. While this improves training efficiency, the BroRL paper reveals an unanticipated interaction: when N is small (N=16), only 41% of generated samples pass the dynamic sampling filter (Table 2), meaning 59% of generation compute is wasted on uninformative trajectories. Increasing N to 512 raises the pass rate to 62%, because broader exploration makes it more likely to encounter at least some correct and some incorrect solutions per prompt. This means dynamic sampling and rollout-size scaling are complementary — larger N makes the filtering mechanism itself more efficient.

How BroRL Positions Itself

The paper positions BroRL as a complementary scaling axis to step-count scaling, not a replacement. The central claim is that the two axes address different bottlenecks:

• Step-count scaling (ProRL): increases the number of gradient updates, allowing the policy to progressively refine its behavior through many small adjustments. This works well initially but eventually hits diminishing returns because each update, based on a narrow sample (N=16), has limited information content and high variance.

• Rollout-size scaling (BroRL): increases the breadth of exploration per update, ensuring that each gradient step is computed from a more representative sample of the policy's output distribution. This reduces the influence of the "unsampled coupling" term identified in Theorem 1 and yields more stable, reliably positive updates.

The paper draws an explicit parallel to the training-inference tradeoff literature but applies it within RLVR: rather than choosing between pretraining and inference compute, the choice is between spending compute on more training steps (depth) versus more exploration per step (breadth). The finding is that after a model has plateaued under depth-scaling, breadth-scaling can revive it and produce continuous gains.

The key intellectual move is to provide a theoretical mechanism for why rollout-size scaling works, not just empirical evidence. The mass balance analysis in Section 2 derives that the change in correct-token probability mass $\Delta Q_{pos}$ decomposes into a guaranteed-positive sampled term and a conditional unsampled term that can be negative. As N increases, the unsampled second moments $U_{pos,2}$ and $U_{neg,2}$ decay exponentially (Lemma 2), ensuring $\Delta Q_{pos} \geq 0$ — every update is guaranteed to increase correct-token probability mass. This guarantee does not hold at small N, where negative updates can and do occur (as shown in the simulation, Figure 3, where small-N runs exhibit knowledge shrinkage — worst-case probability drops among correct tokens). The theory thus provides a necessity argument for large N: without sufficient exploration, the RLVR update signal is not merely noisy but potentially destructive.

The paper positions itself against the prevailing narrative in the field, which the authors characterize as implicitly treating the observed plateaus as fundamental limits. The key counterclaim is:

"the perceived limits of RLVR are sometimes artifacts of algorithmic design (e.g., insufficient rollouts) rather than the fundamental limits of RL itself" (Section 5)

This is a significant claim because it suggests that many of the negative or mixed results reported for long-term RLVR training may be attributable to exploration starvation rather than to any inherent property of the method. If correct, it implies that the community has been underestimating RLVR's potential and that substantial further gains are achievable with better exploration strategies — of which scaling N is the simplest and most direct.

The paper also explicitly connects its theoretical framework to practical hardware efficiency, making a case that BroRL is not just theoretically sound but deployment-friendly. The shift from memory-bound to compute-bound generation at large batch sizes (Section 4.2.4) means that the additional compute cost of scaling N is partially offset by better GPU utilization — a pragmatic argument that distinguishes this work from purely theoretical analyses.

3. Technical Approach

3.1 Reader Orientation

BroRL is a training recipe for large language models that spends the same total compute as standard RLVR training, but allocates it differently: instead of taking many small gradient steps (each informed by a handful of attempted solutions per problem), it takes fewer steps but explores more thoroughly at each step — generating hundreds of candidate solutions per problem — so that each update sees a much richer picture of what the model can and cannot do. The method solves the problem of performance plateaus in RLVR training by treating rollout size $N$ (the number of sampled answers per prompt) as a scaling axis that directly controls the stability and reliability of each policy update, with the key insight being that insufficient exploration introduces a mathematically identifiable "unsampled coupling" term that can make gradient updates destructive rather than constructive.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components that work together in a single reinforcement learning loop:

1. Base Language Model (ProRLv2 Checkpoint) — a 1.5B-parameter reasoning model already trained for 3,000 RL steps using the ProRL recipe. This model serves as the policy, producing candidate solutions (rollouts) when given a prompt. It has been fine-tuned to generate long chain-of-thought reasoning traces.

2. Rollout Generator — given a batch of prompts (math problems, coding tasks, or reasoning challenges), this component samples $N$ complete solutions per prompt from the current policy. In BroRL, $N$ is set to 512, compared to the ProRL baseline of 16. Each rollout is a full chain-of-thought trajectory ending in an answer that can be automatically verified.

3. Verifier/Reward Function — an automatic system that checks whether each generated answer is correct or incorrect, assigning binary rewards (positive for correct, negative for incorrect). This is the "verifiable reward" in RLVR: the correctness of final answers can be checked algorithmically (e.g., math answers match the ground truth, code passes test cases), so no human feedback is needed.

4. Dynamic Sampling Filter — a filtering mechanism that removes prompts where all rollouts are correct or all are incorrect, keeping only prompts with mixed outcomes (at least one correct and at least one incorrect). This ensures that each training step focuses on instances where the policy has something to learn — where it sometimes succeeds and sometimes fails.

5. PPO-Based Policy Updater — the reinforcement learning optimizer that computes advantages from the rollout rewards and updates the policy parameters. It uses a clipped PPO objective with decoupled advantage normalization (REINFORCE++-style), Clip-Higher exploration, periodic reference policy resets, and truncated importance sampling. The key difference from ProRL is that this updater receives far more rollouts per prompt per step, which changes the statistical properties of the advantage estimates and the gradient updates.

Information flows in a cycle: prompts are sampled → the current policy generates $N=512$ completions per prompt → the verifier scores each completion → the dynamic sampling filter removes non-informative prompts → the PPO updater computes advantages and updates the policy → the updated policy is used for the next round of generation.

3.3 Roadmap for the Deep Dive

• First, the core theoretical framework — the mass balance analysis in logit space that decomposes the one-step RLVR update into a sampled-positive term and an unsampled-coupling term, establishing why rollout size $N$ directly controls the reliability of each gradient step. This is the intellectual foundation for everything that follows.

• Second, the mathematical derivation of how the unsampled term decays with $N$ (Lemma 2 and Corollary 3), which provides the guarantee that sufficiently large $N$ ensures $\Delta Q_{pos} \geq 0$ — every update increases the probability mass assigned to correct tokens. This explains why BroRL's large-N strategy is necessary, not just empirically beneficial.

• Third, the connection between the token-level theory and the practical metric of interest — pass@k — showing that increases in $Q_{pos}$ directly and monotonically improve the probability of producing a correct answer within $k$ attempts. This bridges the theoretical analysis to the empirical evaluation.

• Fourth, the BroRL training algorithm itself — how it inherits the ProRLv2 framework (clipped PPO, dynamic sampling, advantage normalization, reference resets, Clip-Higher, off-policy correction) and modifies only the rollout size $N$ and the learning rate scaling rule. This clarifies what changes and what stays the same.

• Fifth, the hardware-level efficiency mechanisms — why generating $N=512$ rollouts per prompt actually runs faster per-sample than $N=16$, due to the transition from memory-bound to compute-bound generation and improved prefix cache hit rates. This is essential for understanding why BroRL is practical, not just theoretically elegant.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a theoretically-motivated empirical methods paper whose core idea is that the exploration breadth per RL update — controlled by the number of rollouts $N$ sampled per prompt — is a principled scaling axis that governs training stability, and that scaling $N$ to hundreds of samples systematically eliminates a mathematically identified source of destructive gradient updates.

---

The Theoretical Framework: Mass Balance Analysis of One-Step RLVR

The paper's central theoretical contribution is a mass balance equation that characterizes how a single step of RLVR changes the total probability mass assigned to correct tokens, $Q_{pos}$. The analysis operates in logit space, treating the vocabulary as a discrete set of tokens partitioned into correct tokens $\mathcal{P}$ and incorrect tokens $\mathcal{N}$. The goal is to understand, at the token level, whether each RLVR update reliably shifts probability from incorrect to correct tokens, or whether it can sometimes have the opposite effect.

Why this framing matters. Prior work observed that RLVR training can exhibit "knowledge shrinkage" — the model sometimes becomes less likely to produce correct tokens after an update (Wu et al., 2025). This is devastating for reasoning models because it means the very process designed to improve the model can inadvertently erase capabilities it already had. Understanding when and why this happens requires analyzing the RLVR update at the token-probability level, not just tracking aggregate metrics like pass@1. The mass balance approach decomposes the total change into components with known signs, revealing that the potential for negative updates comes from a specific, identifiable source — the unsampled tokens — whose influence can be controlled.

Setup and assumptions. The analysis makes a simplifying assumption of a single RLVR step with a TRPO-style linear surrogate objective. This means the gradient update is treated as a one-shot adjustment to the logits, ignoring the compounding effects of multiple steps and the complications of clipping, momentum, and learning rate schedules. While this is a simplification, it serves the purpose of isolating the effect of rollout size $N$ on the sign of the update. The reward structure is binary: correct sampled tokens receive reward $R_c$, incorrect sampled tokens receive reward $R_w$, and unsampled tokens receive zero reward (since no signal is available for them). The rewards satisfy $R_c \geq 0 \geq R_w$ — correct answers get a non-negative bonus, incorrect ones get a non-positive penalty.

Notation. The paper defines a set of quantities that capture how probability mass is distributed across the vocabulary before the update:

• $P_{pos}$ and $P_{neg}$: the total probability mass of sampled correct tokens and sampled incorrect tokens, respectively. These are sums over the specific tokens that happened to appear in the $N$ rollouts. $P_{pos} = \sum_{i \in A} p_i$ where $A \subseteq \mathcal{P}$ is the set of sampled correct tokens, and $P_{neg} = \sum_{i \in B} p_i$ where $B \subseteq \mathcal{N}$ is the set of sampled incorrect tokens.

• $Q_{pos}$ and $Q_{neg}$: the total probability mass of all correct tokens and all incorrect tokens in the vocabulary, respectively. These are global quantities that include both sampled and unsampled tokens. $Q_{pos} = \sum_{i \in \mathcal{P}} p_i$ and $Q_{neg} = 1 - Q_{pos}$. These are the quantities we ultimately care about because they determine the model's chance of producing a correct answer.

• $A_2$, $B_2$, $U_{pos,2}$, $U_{neg,2}$: second-moment (squared-probability) sums that measure concentration — how much probability mass is concentrated in a few high-probability tokens versus spread thinly across many low-probability ones. $A_2 = \sum_{i \in A} p_i^2$ for sampled correct tokens, $B_2 = \sum_{i \in B} p_i^2$ for sampled incorrect tokens, $U_{pos,2} = \sum_{i \in U \cap \mathcal{P}} p_i^2$ for unsampled correct tokens, $U_{neg,2} = \sum_{i \in U \cap \mathcal{N}} p_i^2$ for unsampled incorrect tokens. These terms matter because they determine how effectively a reward signal on one token propagates to other tokens through the softmax normalization: concentrated probability leads to larger gradient effects.

• $SR$ (the "batch mood"): the net per-token contribution of sampled tokens to the expected reward. $SR = R_c P_{pos} + R_w P_{neg}$. This is a scalar that captures whether the sampled batch, on balance, contains more correct than incorrect probability mass. A positive $SR$ means the batch is "reward-positive" (more correct than incorrect mass was sampled), while a negative $SR$ means the batch is "reward-negative."

The central theorem. Theorem 1 states that under a one-step RLVR update with learning rate $\eta$, the total change in correct-token probability mass $\Delta Q_{pos}$ decomposes into three terms:

$\Delta Q_{pos} = \frac{\eta}{N} \left[ (R_c - SR) Q_{neg} A_2 + (SR - R_w) Q_{pos} B_2 + SR \left( Q_{pos} U_{neg,2} - Q_{neg} U_{pos,2} \right) \right]$

where $\eta$ is the learning rate, $N$ is the rollout size per prompt, $R_c$ and $R_w$ are the rewards for correct and incorrect tokens respectively, $SR$ is the batch mood defined above, $Q_{pos}$ and $Q_{neg}$ are the global correct/incorrect probability masses, $A_2$ is the second-moment concentration of sampled correct tokens, $B_2$ is the second-moment concentration of sampled incorrect tokens, $U_{pos,2}$ is the second-moment concentration of unsampled correct tokens, and $U_{neg,2}$ is the second-moment concentration of unsampled incorrect tokens.

What this equation computes: it decomposes the total change in the model's probability of producing correct tokens — the quantity that ultimately determines reasoning accuracy — into three additive contributions. The first contribution comes from sampled correct tokens being boosted; the second comes from sampled incorrect tokens being suppressed; the third comes from unsampled tokens being indirectly affected through the softmax normalization. The equation tells us whether a given RLVR update is net-beneficial ($\Delta Q_{pos} > 0$), neutral ($\Delta Q_{pos} = 0$), or net-harmful ($\Delta Q_{pos} < 0$) based on the distribution of probability mass before the update.

Why this form: the decomposition follows from applying the chain rule to the softmax function. When we update logits $z_j$ by $\Delta z_j = \frac{\eta}{N} p_j (R_j - SR)$, the change in any token's probability depends on (a) the direct effect of its own logit change and (b) the indirect effect of the normalization constraint that all probabilities must sum to 1. The normalization couples all tokens together: boosting sampled correct tokens (a positive effect on $\Delta Q_{pos}$) forces a compensatory reduction in all tokens proportional to their probability mass, including unsampled correct tokens (a negative effect on $\Delta Q_{pos}$). This coupling is why the unsampled terms appear with a minus sign — $Q_{pos} U_{neg,2} - Q_{neg} U_{pos,2}$ — reflecting the tug-of-war between the boost to correct tokens and the normalization penalty that affects them too. The form is necessary because any RLVR update that operates through softmax-normalized probabilities inevitably creates this coupling; the question is whether the net effect is positive or negative.

Interpretation of the three terms.

Term 1 — Sampled-correct boost: $(R_c - SR) Q_{neg} A_2$. This term is always non-negative because $R_c - SR \geq 0$ (the reward for a correct token is at least as high as the batch average), $Q_{neg} \geq 0$ (incorrect probability mass is non-negative), and $A_2 \geq 0$ (squared probabilities are non-negative). Mechanically, sampled correct tokens receive a positive advantage $R_c - SR$, which directly pushes their logits up. Normalization redistributes the "budget" for this increase by taking probability mass from the incorrect pool (whose share is $Q_{neg}$), resulting in a net transfer from incorrect to correct probability. The effect is stronger when sampled correct tokens are highly concentrated (large $A_2$) because concentration means the gradient signal is applied to tokens that already have substantial probability, making the logit increase more "massive" in terms of probability shift.

Term 2 — Sampled-incorrect suppression: $(SR - R_w) Q_{pos} B_2$. This term is also always non-negative because $SR - R_w \geq 0$ (the batch average is at least as high as the incorrect-token reward — in practice, $R_w$ is negative or zero, so $SR \geq R_w$ always), $Q_{pos} \geq 0$, and $B_2 \geq 0$. Mechanically, sampled incorrect tokens receive a negative advantage $R_w - SR \leq 0$, pushing their logits down. Normalization redistributes the freed probability mass to the correct pool in proportion to its size $Q_{pos}$. The effect is stronger when sampled incorrect tokens are highly concentrated (large $B_2$), because demoting a high-probability incorrect token frees up more mass than demoting many low-probability ones.

Term 3 — Unsampled coupling: $SR \left( Q_{pos} U_{neg,2} - Q_{neg} U_{pos,2} \right)$. This term can be positive, zero, or negative depending on the relative concentration of unsampled incorrect versus correct tokens and the sign of the batch mood $SR$. If the batch is reward-positive ($SR > 0$), normalization pushes all unsampled tokens down slightly — this helps if incorrect unsampled mass is more concentrated than correct unsampled mass (net positive), but hurts if correct unsampled mass is more concentrated (net negative). If the batch is reward-negative ($SR < 0$), normalization nudges all unsampled tokens up — this helps if correct unsampled mass is more concentrated, but hurts if incorrect unsampled mass dominates. The magnitude of this term depends on how much unsampled probability mass exists and how concentrated it is — both of which shrink as $N$ increases.

The critical insight about N. The third term is the only one that can make $\Delta Q_{pos}$ negative. Without it, the first two terms guarantee a net positive transfer of probability from incorrect to correct tokens. The question is: when can the third term overwhelm the first two? The answer depends on $N$ in two ways. First, as $N$ grows, the unsampled second moments $U_{pos,2}$ and $U_{neg,2}$ shrink (fewer tokens remain unsampled), directly reducing the magnitude of the coupling term. Second, the first two terms scale with $A_2$ and $B_2$, which grow with $N$ because more correct and incorrect tokens are sampled and thus included in the sampled terms. The net effect is that the ratio of the harmful unsampled term to the beneficial sampled terms goes to zero as $N$ increases, guaranteeing $\Delta Q_{pos} \geq 0$ in the limit.

Implications for training design. The theorem establishes three practical implications: (i) Large N guarantees positivity: as per-prompt sampling size grows, the unsampled terms vanish, ensuring every update increases correct-token mass. This means BroRL's large-N design is not merely an empirical heuristic — it has a formal justification. (ii) Small N can be destructive: even under balanced batches ($SR \approx 0$, which cancels the third term), variance remains high, and occasional negative $SR$ values combined with concentrated unsampled correct mass can produce $\Delta Q_{pos} < 0$. These negative updates accumulate over thousands of steps and produce the performance plateaus and regressions observed in ProRL. (iii) The batch mood $SR$ acts as a multiplier: when the batch is balanced (roughly equal correct and incorrect mass), the third term is small regardless of $N$; when imbalanced, large $N$ is essential to suppress the coupling effect.

---

Expected Decay of Unsampled Mass with Rollout Size (Lemma 2 and Corollary 3)

The guarantee that large $N$ ensures positive updates depends on the unsampled second moments $U_{pos,2}$ and $U_{neg,2}$ decreasing as $N$ grows. Lemma 2 quantifies this decay under the assumption that tokens are sampled independently in each of the $N$ draws.

Lemma 2 (Expected unsampled contribution). For a single token with probability $p$ (the probability the current policy assigns to that token), the expected contribution to the unsampled second moment after $N$ independent draws is:

$E[U_2(p)] = p^2 (1 - p)^N$

where $p$ is the token's probability under the current policy, and $N$ is the number of rollout samples drawn.

What it computes: the expected value of the squared probability of a token, conditional on the token never being sampled in any of the $N$ draws. Since the unsampled second moment $U_{pos,2}$ sums $p_i^2$ over only the tokens that were not drawn, and each token has probability $(1-p_i)^N$ of never being drawn in $N$ independent trials, the expected contribution per token is $p_i^2$ (its squared probability) multiplied by $(1-p_i)^N$ (the probability it remains unsampled). The expectation is taken over the randomness of which tokens are sampled in each draw.

Why this form: the token's squared probability $p^2$ is the contribution it would make if it were unsampled. The factor $(1-p)^N$ is the probability of zero successes in $N$ independent Bernoulli trials with success probability $p$ — this is the binomial probability $\Pr[X=0]$ where $X \sim \text{Bin}(N, p)$. The independence assumption (each of the $N$ draws is i.i.d. from the current policy) is a standard simplification that captures the essence of the scaling without modeling the complex dependencies of autoregressive generation. This exponential decay in $N$ means that even tokens with relatively high probability under the policy have their unsampled contribution shrink rapidly as the rollout count increases.

Corollary 3 (Total unsampled decay). Summing over all tokens in the unsampled correct set $U \cap \mathcal{P}$ and the unsampled incorrect set $U \cap \mathcal{N}$:

$\sum_i p_i^2 (1 - p_i)^N$

What it computes: the expected total $U_{pos,2}$ plus $U_{neg,2}$ after $N$ draws, by linearity of expectation. Each token's contribution decays at a rate governed by its own probability $p_i$. High-probability tokens (large $p_i$) have larger initial contributions $p_i^2$ but also decay faster because $(1-p_i)^N$ is smaller for larger $p_i$ — they are likely to be sampled. Low-probability tokens decay slowly but have tiny initial contributions. The net effect is that the total unsampled second moment is dominated by intermediate-probability tokens and decays monotonically with $N$.

Why this matters for BroRL. Corollary 3 provides the mathematical justification for scaling $N$ to hundreds or thousands. At $N=16$ (the ProRL baseline), tokens with probability $p=0.1$ have $(1-0.1)^{16} \approx 0.185$ — nearly one-fifth of their squared-probability contribution still appears in the unsampled terms. At $N=512$ (BroRL), the same token has $(1-0.1)^{512} \approx 2.4 \times 10^{-24}$ — essentially zero. The unsampled coupling term in Theorem 1 is therefore reduced by many orders of magnitude when moving from ProRL to BroRL, making the update reliably positive. The simulation results (Figure 3) confirm this: at small $N$, worst-case probability drops among correct tokens are observed (knowledge shrinkage), while at large $N$, all correct tokens gain probability mass at every step.

---

Connecting Token-Level Theory to Task-Level Metrics: Pass@k and $Q_{pos}$

The theory operates at the token level — it characterizes how the probability mass assigned to correct tokens changes — but practitioners care about task-level metrics like pass@1 (does the model produce a correct answer in one attempt?) or pass@k (does at least one of $k$ attempts succeed?). The paper establishes a direct, monotonic relationship between these levels.

The pass@k equation. For a single input $x$, let $Q_{pos}(x) \in [0,1]$ be the total probability mass assigned to correct tokens (the probability that a single sampled token is correct). When drawing $k$ independent samples, the probability that at least one sample is correct is:

$\text{pass@k}(x) = 1 - \left(1 - Q_{pos}(x)\right)^k$

where $Q_{pos}(x)$ is the total correct-token probability for input $x$ and $k$ is the number of independent attempts.

What it computes: the per-example success probability — the chance that a reasoning model produces a correct answer within $k$ attempts — as a function of the underlying token-level correctness probability. The term $(1 - Q_{pos}(x))^k$ is the probability that all $k$ independent draws produce an incorrect answer. Subtracting from 1 gives the probability that at least one draw succeeds.

Why this form: this is the standard geometric relationship for independent Bernoulli trials. It assumes that token production (and hence answer correctness) is independent across attempts, which holds when sampling at non-zero temperature (each attempt is a fresh draw from the policy distribution). The function is strictly increasing in $Q_{pos}(x)$ — any increase in token-level correct probability translates directly into a higher chance of producing a correct answer. Moreover, because the relationship is geometric, increases in $Q_{pos}(x)$ compound: a small improvement in token-level correctness produces a larger improvement in pass@k for larger $k$.

Taking the expectation over tasks. The population-level pass@k is the expectation over the task distribution:

$\mathbb{E}_x[\text{pass@k}(x)] = 1 - \mathbb{E}_x\left[(1 - Q_{pos}(x))^k\right]$

If BroRL produces pointwise improvements in $Q_{pos}(x)$ (i.e., $Q_{pos}'(x) \geq Q_{pos}(x)$ for all $x$, with strict inequality on a set of positive measure), then both the per-task pass@k and its expectation strictly increase. This establishes the chain of reasoning: BroRL's large $N$ guarantees $\Delta Q_{pos} \geq 0$ at each update (Theorem 1) → $Q_{pos}$ increases monotonically during training → $\text{pass@k}$ increases monotonically at the task level.

Why this connection is important for the paper's claims. It bridges the theoretical analysis (which operates in the simplified logit domain with a one-step assumption) to the empirical evaluation (which measures pass@1 on held-out test benchmarks). The monotonicity means that if BroRL indeed produces positive updates in the token-level sense, we should observe continuous improvements in benchmark scores — exactly what the experiments show (Figure 1, Tables 3-5). Conversely, if ProRL's small $N$ sometimes produces negative $\Delta Q_{pos}$ due to the unsampled coupling term, we should observe performance saturations and regressions — again matching the empirical pattern where ProRL degrades from 62.08 to 62.02 on Math after extended training.

---

The BroRL Training Algorithm: Modifications to the ProRLv2 Framework

BroRL is not a new RL algorithm; it is the ProRLv2 framework (Hu et al., 2025b) with a single hyperparameter change — the rollout size $N$ — and a principled learning rate adjustment to accommodate the resulting batch size increase. The algorithm inherits all of ProRLv2's components, which the paper describes as "a clipped Proximal Policy Optimization (PPO) algorithm" with several specialized enhancements.

Base PPO objective. The policy is optimized using the standard clipped PPO surrogate:

$L_{PPO}(\theta) = \mathbb{E}_{\tau} \left[ \min \left( r_{\theta}(\tau) A(\tau), \text{clip}\left(r_{\theta}(\tau), 1 - \varepsilon_{\text{low}}, 1 + \varepsilon_{\text{high}}\right) A(\tau) \right) \right]$

where $r_{\theta}(\tau)$ is the probability ratio between the current policy and the reference policy for trajectory $\tau$, $A(\tau)$ is the advantage of trajectory $\tau$, and $\varepsilon_{\text{low}}$ and $\varepsilon_{\text{high}}$ are asymmetric clipping thresholds.

What it computes: for each trajectory (a complete chain-of-thought solution), the algorithm computes the ratio of its probability under the current policy to its probability under the reference (frozen) policy, multiplies by the advantage, and clips this product to stay within a trust region. The $\min$ operation chooses the more conservative of the clipped and unclipped values, preventing the policy from changing too drastically in any single update. The asymmetric clipping ($\varepsilon_{\text{high}} > \varepsilon_{\text{low}}$) implements "Clip-Higher" — allowing more aggressive increases in probability for good trajectories than decreases for bad ones, which encourages exploration.

Why this form: PPO is the standard algorithm for RLVR because it provides stable policy updates with a trust-region constraint without the computational cost of full second-order optimization (as in TRPO). The asymmetric clipping is specifically designed for reasoning tasks: good trajectories are rare and valuable, so the algorithm should be willing to increase their probability substantially, while bad trajectories are abundant and should be suppressed more conservatively to avoid overfitting to specific failure modes.

Advantage normalization (REINFORCE++-style). BroRL inherits the decoupled advantage normalization scheme from ProRLv2. Advantages are computed in two stages:

$A_{\tau} = R_{\tau} - \text{mean}_{\text{group}}(R_{\tau})$ $A_{\tau}^{\text{norm}} = \frac{A_{\tau} - \text{mean}_{\text{batch}}(A_{\tau})}{\text{std}_{\text{batch}}(A_{\tau})}$

where $R_{\tau}$ is the trajectory return (the verifier's correctness score), $\text{mean}_{\text{group}}$ is the mean return of all rollouts for the same prompt (the "group"), $\text{mean}_{\text{batch}}$ is the mean across all trajectories in the global batch, and $\text{std}_{\text{batch}}$ is the standard deviation across all trajectories in the global batch.

What it computes: the first step subtracts the per-prompt average return from each rollout's return, creating a relative advantage — a rollout is "good" only if it outperforms other rollouts for the same prompt. This is crucial because raw returns conflate prompt difficulty with trajectory quality (a bad solution to an easy problem might get a higher absolute score than a good solution to a hard problem). The second step normalizes across the entire training batch to unit standard deviation, which stabilizes training by ensuring consistent gradient magnitudes regardless of batch composition.

Why this form: decoupled normalization (first within-group, then across-batch) addresses a fundamental challenge in RLVR: the scale of advantages varies dramatically across prompts (some problems are inherently easier or harder) and across training stages (early training has more extreme advantages than later training). Without normalization, the optimizer would take large steps on easy prompts and tiny steps on hard prompts, and the effective learning rate would drift over the course of training. The two-stage normalization makes the advantage signal scale-invariant and problem-invariant.

Dynamic sampling. A key efficiency component inherited from ProRLv2 and DAPO (Yu et al., 2025): the training batch is filtered to remove "trivial" prompts before the PPO update. The filtered batch $\mathcal{B}'$ is:

$\mathcal{B}' = \left\{ \tau \in \mathcal{B} \;\middle|\; 0 < \sum_{i=1}^{N} \mathbb{I}(M_i = M_{\text{correct}}) < N \right\}$

where $\mathcal{B}$ is the original batch, $N$ is the number of rollouts per prompt, $M_i$ is the model's final answer for rollout $i$, $M_{\text{correct}}$ is the ground-truth correct answer, and $\mathbb{I}(\cdot)$ is the indicator function.

What it computes: the filter keeps only prompts where the model produced at least one correct answer and at least one incorrect answer across the $N$ rollouts. Prompts where all rollouts are correct (the model has already mastered this problem) or all are incorrect (the model has no signal for what a correct answer looks like) are discarded from that training step.

Why this form: training on prompts with only correct answers would reinforce existing behavior without improvement (the advantage estimate would have zero variance within the group, making the normalized advantage meaningless). Training on prompts with only incorrect answers provides no positive signal — the model learns what not to do but not what to do instead. Mixed-outcome prompts are the only ones that provide a meaningful gradient: the model can compare its correct and incorrect attempts and learn the difference. The paper's contribution is the observation that increasing $N$ from 16 to 512 dramatically increases the fraction of prompts that have mixed outcomes (from 41% to 62% pass rate in Table 2), because broader exploration makes it more likely to stumble upon at least one correct answer even for challenging problems.

Other inherited components. BroRL retains three additional mechanisms from ProRLv2:

• Periodic reference policy resets: the reference policy (against which probability ratios are computed) is periodically reset to the current policy. This prevents the probability ratio $r_{\theta}(\tau)$ from drifting too far from 1 as training progresses, which would make clipping ineffective. Without resets, the reference would become stale, and the trust region constraint would lose meaning.

• Clip-Higher exploration ($\varepsilon_{\text{high}} > \varepsilon_{\text{low}}$): allowing larger positive updates than negative updates biases the policy toward exploring promising trajectories while being cautious about suppressing behaviors that might be useful in other contexts.

• Truncated importance sampling (Yao et al., 2025): corrects for off-policy mismatch between the inference engine (which generates rollouts) and the training engine (which computes gradients). In distributed RL training, the policy used for generation may lag behind the policy being trained due to asynchronous updates. Truncated importance sampling reweights trajectories to account for this mismatch, preventing stale rollouts from corrupting the gradient estimate.

Learning rate scaling for batch size change. The one algorithmic modification beyond changing $N$ is a principled learning rate adjustment to account for the larger effective batch size. When $N$ increases from 16 to 512, the number of trajectories per prompt grows by a factor of 32, and the total batch size grows proportionally (from 512 prompts × 16 rollouts = 8,192 trajectories to 128 prompts × 512 rollouts = 65,536 trajectories). The paper applies the square-root scaling rule:

$\eta_{\text{new}} = \eta_0 \times \sqrt{\frac{B_{\text{new}}}{B_0}}$

where $\eta_0$ is the base learning rate for the original batch size $B_0$, and $\eta_{\text{new}}$ is the adjusted learning rate for the larger batch size $B_{\text{new}}$.

What it computes: a scaling factor that increases the learning rate by the square root of the batch-size ratio. This is a standard heuristic from large-batch training (Krizhevsky, 2014): linear scaling would overshoot (the gradient becomes more accurate with larger batches, not proportionally larger), while no scaling would undershoot (the larger batch provides a higher-quality gradient that can support a larger step). The square root is a compromise that empirically maintains training stability.

Why this form: the PPO mini-batch count is kept unchanged (the paper states "while keeping the number of PPO mini-batches per step unchanged"), meaning the larger batch is processed in the same number of gradient accumulation steps, and each mini-batch now contains more trajectories. The square-root scaling prevents each mini-batch update from being either too conservative (taking forever to converge) or too aggressive (destabilizing training). In practice, the paper's experiments show that this adjustment works well: BroRL training is stable and shows continuous improvement without the degradation observed in ProRL.

Training configuration. The paper provides specific experimental details. The base model is the publicly available ProRLv2 checkpoint at 3,000 RL steps, with a context window expanded from 8,192 to 16,384 tokens to support longer chain-of-thought reasoning. Training uses 64 NVIDIA H100 GPUs with the veRL framework (Sheng et al., 2025). The ProRL baseline uses $N=16$ rollouts with 512 prompts per step; BroRL uses $N=512$ rollouts with 128 prompts per step, keeping the total number of generated samples per step roughly comparable (the product of prompts and rollouts per prompt: $16 \times 512 = 8,192$ for ProRL vs. $512 \times 128 = 65,536$ for BroRL — note the factor-of-8 difference in total samples per step, which is why Table 1 compares methods at equal training time rather than equal steps, because BroRL generates more samples per step but completes steps faster due to hardware efficiency).

---

Hardware Efficiency: Why Large-N Generation Is Faster Per Sample

The paper reports a striking finding in Table 2: BroRL's $N=512$ rollout generation achieves nearly double the throughput of ProRL's $N=16$ — 72.4 samples per second versus 36.5. This is counterintuitive at first glance: generating 512 solutions per prompt should be more work than generating 16. The paper attributes this to two interacting effects: a shift from memory-bound to compute-bound execution, and improved prefix cache utilization.

Memory-bound vs. compute-bound generation. Modern GPU inference for language models consists of two phases: a prefill phase where the prompt tokens are processed in parallel to compute the key-value (KV) cache, and a decode phase where tokens are generated autoregressively one at a time. The decode phase is typically memory-bound: the GPU's compute cores spend most of their time idle, waiting to fetch model weights and KV cache entries from high-bandwidth memory (HBM). The arithmetic intensity — the ratio of FLOPs performed to bytes transferred — is low because each generated token requires loading the entire model weights but performs relatively few computations per weight byte.

When the batch size is small (e.g., $N=16$), the working set — the combination of model weights and KV caches for the active sequences — fits comfortably in memory but doesn't provide enough parallel work to saturate the GPU's compute units. The GPU is underutilized, and throughput is limited by memory bandwidth. When the batch size is large (e.g., $N=512$), the GPU has many more sequences to process simultaneously, increasing the amount of computation per memory access: the same model weights are reused across all 512 sequences, so the weight-loading cost is amortized, and the KV cache accesses are more structured and predictable, allowing better memory coalescing. The operation shifts from memory-bound (limited by how fast data can be fetched) to compute-bound (limited by how fast the GPU cores can execute), which typically yields higher throughput because modern GPUs are designed with much more compute capacity than memory bandwidth.

Prefix cache hit rate. In chain-of-thought reasoning, all $N$ rollouts for a given prompt share the same prefix — the prompt itself, which can be hundreds or thousands of tokens long. With a prefix caching system (Zheng et al., 2024), the KV cache for the prompt tokens is computed once during the first rollout's prefill phase and stored. Subsequent rollouts reuse this cached prefix, skipping the expensive prefill computation for the shared prompt tokens and starting generation from the first reasoning token. At $N=16$, the prefix caching amortization is modest: the prompt prefill cost is shared across only 16 sequences. At $N=512$, the same prefill cost is shared across 512 sequences, making the amortization factor 32× larger. Since the prefill phase is often the computational bottleneck for long-context reasoning tasks (the prompt plus chain-of-thought can span thousands of tokens), this amortization yields substantial speedups.

Interaction with dynamic sampling pass rate. The paper notes a synergistic effect: larger $N$ not only improves generation throughput but also increases the fraction of generated samples that survive the dynamic sampling filter — from 41% at $N=16$ to 62% at $N=512$ (Table 2). This means that of the samples generated per second, a higher proportion are actually used for training, further amplifying BroRL's effective data efficiency. The mechanism is straightforward: with more rollouts per prompt, the model is more likely to produce at least one correct and at least one incorrect answer, satisfying the dynamic sampling criterion. Prompts that would have been discarded at $N=16$ (because all rollouts were correct or all incorrect) become training-relevant at $N=512$ because broader exploration reveals the mixed capability.

Practical implications. The hardware efficiency results mean that BroRL's theoretical advantage (more stable, reliably positive updates) is reinforced by a practical advantage (faster generation per sample). Under equal training time (~140 hours, as in Figure 1 and the statistical test), BroRL not only produces better models but also processes more information — the effective training throughput is higher. This dual benefit is rare in ML research, where algorithmic improvements often come with computational overhead. BroRL's design aligns with GPU hardware characteristics, making it a practical choice for real-world RLVR deployment.

---

Simulation Setup: Validating the Theory Under Relaxed Conditions

Section 4.1 presents a token-level simulation designed to verify the theoretical predictions under conditions that are more realistic than the one-step, independent-sampling assumptions of the analysis, while still being computationally tractable. The simulation abstracts away the complexities of autoregressive sequence generation to focus purely on the token-probability dynamics predicted by Theorem 1.

Vocabulary and reward structure. The simulation uses a vocabulary of size $d = 128,000$ tokens, of which $|\mathcal{P}| = 10,000$ are designated "correct" and receive reward $R_i = +1$, while the remaining $118,000$ tokens are "incorrect" and receive reward $R_i = -1$. This asymmetry — a small fraction of correct tokens in a large vocabulary — mimics the structure of reasoning tasks where the model must select the right answer tokens among a vast sea of incorrect possibilities.

Logit initialization. Logits $z \in \mathbb{R}^d$ are initialized to zero for all tokens, with an optional seeding step where correct tokens are initialized to $z_i = 3$ and one anchor token is fixed at $z_0 = 5$. The anchor token (which is not in $\mathcal{P}$) simulates a high-probability incorrect token that the policy must learn to suppress. Probabilities are computed via softmax with temperature $\tau = 1$: $p_i = \text{softmax}(z/\tau)_i$.

Sampling and reward computation. At each training step $t$, the simulator draws $N$ independent samples from the current probability distribution $p$. Each sampled token $j$ receives its base reward $R_j$. To reduce variance, rewards are centered by the batch baseline: $\tilde{r}_j = R_j - b$ where $b = \frac{1}{N} \sum_{k=1}^{N} R_k$. This centering is the simulation-level analog of the advantage normalization used in full-scale BroRL training.

Optimization objective and procedure. The simulator optimizes the TRPO-style linear surrogate objective:

$L_{\text{sur}} = -\frac{1}{N} \sum_{j=1}^{N} \tilde{r}_j p_j$

where $\tilde{r}_j$ is the centered reward for sampled token $j$ and $p_j$ is its current probability.

What it computes: the negative expected centered reward, which is minimized by gradient descent on the logits. The linear surrogate approximates the full RLVR objective by assuming the policy change is small enough that a first-order expansion around the current logits is sufficient. This is the same form used in the theoretical analysis of Section 2, making the simulation a direct computational test of Theorem 1.

Why this form: the linear surrogate is the TRPO-style simplification that enables clean theoretical analysis. It ignores the clipping and multiple-epoch optimization of full PPO, which would introduce second-order effects that obscure the core relationship between $N$ and update positivity. By using the linear surrogate, the simulation isolates the effect of $N$ on the unsampled coupling term, providing a controlled test of the theory.

Optimizer and hyperparameters. The logits are updated using AdamW with learning rate $\eta = 10^{-3}$ for $T = 1000$ steps. The rollout size $N$ is varied across $\{4, 8, 16, 512, 51200\}$ while all other hyperparameters are held fixed. This sweep covers four orders of magnitude, from very small $N$ (where the unsampled coupling term should dominate and cause negative updates) to very large $N$ (where the unsampled terms should be negligible and all updates positive).

Metrics tracked. After each update, the simulator records three quantities:

• $Q_{pos} = \sum_{i \in \mathcal{P}} p_i$: the total probability mass assigned to all correct tokens — this is the $Q_{pos}$ from Theorem 1, and its trajectory over steps shows whether updates are net-positive.
• Percent improved: the fraction of correct tokens whose probability increased relative to step 0 — this tracks whether knowledge shrinkage occurs (some correct tokens losing probability) or whether all correct tokens benefit.
• Worst drop: the most negative probability change among correct tokens — this directly measures the severity of knowledge shrinkage, with negative values indicating that at least one correct token's probability decreased.

Results and alignment with theory (Figure 3). The simulation results, shown in Figure 3, provide strong empirical validation of the theoretical predictions:

• $Q_{pos}$ accumulation (left panel): Larger $N$ produces faster growth of total correct probability mass. The $N=512$ curve rises rapidly and smoothly, while $N=4$ curves show slower progress with visible fluctuations. The $N=51200$ curve (extreme) is nearly indistinguishable from $N=512$, suggesting diminishing returns — the unsampled coupling term is already negligible at $N=512$.

• Percent improved (middle panel): At small $N$ (4, 8), only a fraction of correct tokens see their probability increase — some correct tokens actually lose probability, confirming knowledge shrinkage. The $N=512$ curve approaches 100% improved, meaning essentially all correct tokens benefit from every update. The $N=51200$ curve reaches nearly 100%, confirming the theoretical limit: with enough exploration, no correct token is harmed.

• Worst drop (right panel): At small $N$, the worst-case probability drop among correct tokens is negative (indicating shrinkage) and fluctuates substantially. At $N=512$, the worst drop is essentially zero (the curve hugs the zero line), and at $N=51200$, it is strictly zero — no correct token ever loses probability. This is the strongest evidence for the theory: increasing $N$ eliminates knowledge shrinkage entirely, exactly as Theorem 1 predicts when the unsampled coupling term vanishes.

What the simulation does NOT model. The simulation abstracts away several aspects of real LLM training: (1) tokens are sampled i.i.d. from the softmax distribution rather than generated autoregressively with sequential dependencies; (2) there is no chain-of-thought structure — just independent token draws; (3) the reward is per-token rather than per-trajectory; (4) the optimization uses a linear surrogate rather than clipped PPO with multiple epochs; (5) the policy is a single softmax layer rather than a deep transformer. These simplifications are intentional: they isolate the core mechanism (the unsampled coupling term and its dependence on $N$) from confounding factors. The fact that the full-scale BroRL experiments (Section 4.2) show qualitatively similar patterns — stable, continuous improvement at $N=512$ versus stagnation and degradation at $N=16$ — suggests that the token-level mechanism identified by the theory is indeed a primary driver of the empirical results, even in the complex setting of autoregressive language model training.

4. Key Insights and Innovations

Innovation 1: The Rollout Size N as a Principled Scaling Axis for RLVR Stability, Not Just Variance Reduction

The paper's most fundamental intellectual contribution is the reframing of rollout size $N$ — the number of sampled solutions per prompt in each RL update — from a mere variance-reduction knob into a scaling axis that governs the sign of policy updates. Prior work in RLVR (Shao et al., 2024; Ahmadian et al., 2024; Hu et al., 2025a; Yu et al., 2025) treated the number of rollouts per prompt as an engineering choice driven by throughput considerations: generate enough samples to get a reasonable gradient estimate, but not so many that generation becomes a bottleneck. The ProRL line of work (Liu et al., 2025a; Hu et al., 2025b) fixed $N=16$ and scaled training steps instead, implicitly assuming that the quality of each individual update was adequate and that more updates would compound into better performance.

BroRL challenges this assumption at a fundamental level. The mass balance analysis (Theorem 1) reveals that $N$ does not merely reduce gradient variance — it eliminates a specific, mathematically identifiable source of destructive updates. The "unsampled coupling" term $SR(Q_{pos}U_{neg,2} - Q_{neg}U_{pos,2})$ can render $\Delta Q_{pos}$ negative under small $N$, meaning some RLVR updates actively reduce the probability mass assigned to correct tokens. This is not a variance problem that averages out over many steps; it is a bias problem where individual updates move the policy in the wrong direction, potentially erasing previously acquired capabilities — the "knowledge shrinkage" documented by Wu et al. (2025). The theoretical guarantee that large $N$ drives this term to zero (Lemma 2, Corollary 3) transforms rollout size from a quantitative tuning parameter into a qualitative determinant of update reliability.

What makes this distinctive is the conceptual move from "more samples = better gradient estimates" (a statistical argument familiar from stochastic optimization) to "more samples = guaranteed-positive policy improvement" (a structural argument about the mechanics of softmax-normalized probability updates). This is not a refinement of existing RLVR algorithms — it identifies a failure mode that those algorithms share and provides a principled remedy. The simulation results (Figure 3) validate the distinction empirically: at small $N$, worst-case probability drops among correct tokens are observed (negative updates exist), while at $N=512$, knowledge shrinkage is eliminated entirely. This reframes the exploration-exploitation tradeoff in RLVR: insufficient exploration is not merely suboptimal — it is actively harmful.

The significance extends beyond the empirical gains. BroRL establishes that the performance plateaus observed in prolonged RLVR training (ProRL degrading from 62.08 to 62.02 after +535 steps, Table 1) are not fundamental capability limits but artifacts of algorithmic design — specifically, the choice to scale steps rather than rollouts. This is a diagnostic insight with broad implications: any RLVR practitioner observing saturation should suspect exploration starvation before concluding that the model has reached its potential. The paper's assertion that "the perceived limits of RLVR are sometimes artifacts of algorithmic design (e.g., insufficient rollouts) rather than the fundamental limits of RL itself" (Section 5) is a reframing that could redirect research effort away from ever-more-sophisticated policy gradient estimators and toward exploration strategies.

Innovation 2: The Mass Balance Analysis as a Diagnostic Framework for RLVR Update Quality

The paper introduces a mass balance equation (Theorem 1) that decomposes the one-step RLVR update into components with known signs, providing the first formal diagnostic for when an RLVR update will be constructive versus destructive. Prior theoretical analyses of RL for language models focused on convergence properties, regret bounds, or the relationship between pretraining and RL objectives. The mass balance approach is fundamentally different: it operates in the logit domain and tracks the flow of probability mass between correct and incorrect token sets, treating the update as a conservation law problem where probability is neither created nor destroyed — only redistributed through the softmax normalization.

The innovation is not the mathematics per se (the derivation uses standard chain-rule expansions and first-order approximations) but rather the decomposition itself and what it reveals. By separating the total change $\Delta Q_{pos}$ into a guaranteed-positive sampled term and a conditional unsampled term, the analysis pinpoints the exact mechanism by which small $N$ can cause destructive updates: unsampled correct tokens can have their probability reduced through the normalization penalty even though they were not directly penalized by the reward signal. This is non-obvious because the intuitive expectation is that RLVR updates should be directionally correct on average — rewarding correct sampled outputs and penalizing incorrect ones should, in expectation, shift mass toward correctness. The mass balance analysis reveals that this intuition fails when the unsampled correct probability is concentrated enough to outweigh the sampled correct gains, a condition that depends critically on $N$.

What makes this a genuine diagnostic tool rather than a theoretical curiosity is its explanatory power for known empirical phenomena. The knowledge shrinkage observed by Wu et al. (2025) — the model forgetting previously correct behaviors — had been documented but lacked a mechanistic explanation. The mass balance analysis provides one: negative $\Delta Q_{pos}$ corresponds directly to shrinkage, and the conditions for negativity (small $N$, concentrated unsampled correct mass, unfavorable batch mood $SR$) match the empirical conditions under which shrinkage is observed. This transforms knowledge shrinkage from a mysterious pathology into a predictable consequence of insufficient exploration — and therefore a fixable one.

The framework also explains why step-count scaling eventually fails. Under small $N$, each update has a non-trivial probability of being net-harmful ($\Delta Q_{pos} < 0$). These harmful updates are relatively rare (most updates are still positive, which is why initial ProRL training shows gains), but they accumulate over thousands of steps. The net effect is that positive and negative updates partially cancel, producing a plateau where additional steps provide no net improvement. Larger $N$ eliminates the negative updates entirely, removing the cancellation and allowing continuous accumulation of $Q_{pos}$. The simulation (Figure 3, right panel) confirms this: at $N=512$ and $N=51200$, the worst-case probability drop among correct tokens is zero, meaning every update is constructive.

This contribution is fundamental rather than incremental because it provides a principled vocabulary for reasoning about RLVR update quality that did not previously exist. Before this work, diagnosing why RLVR training plateaued required empirical trial-and-error (try different learning rates, clipping parameters, reward structures). The mass balance framework provides a theoretical criterion — is the unsampled coupling term sufficiently small relative to the sampled terms? — that directly motivates a specific intervention (increase $N$). The fact that this criterion is both theoretically justified and empirically validated (via simulation and full-scale training) makes it a durable contribution beyond the specific BroRL recipe.

Innovation 3: The Empirical Demonstration That Scaling Rollouts Revives Saturated Models — A Negative Result on the Limits of Step-Count Scaling

The paper provides a controlled empirical demonstration that is as much a negative result about step-count scaling as it is a positive result about rollout-size scaling. Starting from the ProRLv2 checkpoint that had plateaued after 3,000 RL training steps (a model that was "saturated" in the sense that additional ProRL training yielded marginal gains followed by degradation), BroRL continues training with the same base algorithm, the same model architecture, the same reward structure, and the same total compute budget — changing only the rollout size $N$ from 16 to 512. The finding is stark: BroRL achieves continuous improvement where ProRL stagnates and ultimately regresses (Table 1, Figure 1).

What makes this intellectually significant is the counterfactual it disproves. The prevailing interpretation of RLVR plateaus — implicit in the field's move toward ever-larger models and away from prolonged RL training — was that models reach a fundamental capability ceiling after a few thousand RL steps, beyond which further training either does nothing or causes catastrophic forgetting. This interpretation implied that the returns to RLVR are bounded and that achieving higher reasoning performance requires either scaling pretraining or developing entirely new training paradigms.

BroRL demonstrates that this interpretation is wrong for the specific case of step-count scaling with insufficient exploration. The saturated model was not at a capability ceiling; it was stuck in an exploration-poor regime where each additional step was as likely to be harmful as helpful. Simply widening the exploration aperture — generating 512 rollouts per prompt instead of 16, with no other algorithmic changes — unlocked further gains. The model improved from a Math score of ~62 to 63.03, Code from 52.00 to 54.20, and Reasoning Gym from 61.29 to 63.09 (Tables 3-5). These are meaningful gains from a model that was presumed saturated.

This result is fundamental rather than incremental because it redefines the baseline expectation for RLVR scaling. Before BroRL, a reasonable researcher observing a plateau after 3K steps might conclude that RLVR had delivered its full value and move on. After BroRL, the same observation demands the question: "have I tried scaling rollout size?" The paper establishes that step-count scaling with small $N$ is a confounded experimental design — it confuses the effects of more training with the effects of more exploration-poor training, and the observed plateaus reflect the latter, not the former. This is a diagnostic insight that could save enormous computational waste: rather than abandoning RLVR when it plateaus, practitioners should first verify that exploration is sufficient.

The statistical evidence (Section 4.2.2) reinforces the point. Across over 10,000 problem instances, BroRL outperforms ProRL with $\Delta = 0.0033$ and $p = 6.5 \times 10^{-7}$ — a small mean difference (expected, given the strong baseline) but one that is statistically irrefutable and, crucially, is achieved at the same training time (~140 hours). The fact that BroRL extracts more performance from the same compute budget, from a model that ProRL could not improve further, is a direct refutation of the claim that step-count scaling is compute-optimal for RLVR.

Innovation 4: The Alignment of Algorithmic Exploration with Hardware Efficiency — A Practical Scaling Argument

The paper makes a novel argument that goes beyond the typical "better algorithm, better results" narrative: BroRL's large-$N$ design is not only theoretically superior but also better aligned with GPU hardware characteristics, creating a virtuous cycle where the algorithmically motivated choice also improves computational efficiency. This is a distinctive contribution because it addresses the practical objection that often kills theoretically elegant ideas in ML: "it's too expensive to run at scale."

The empirical finding in Table 2 is striking: BroRL's $N=512$ generation achieves nearly double the throughput of ProRL's $N=16$ — 72.4 versus 36.5 samples per second — on identical hardware. The mechanism (described in Section 3.4) involves the transition from memory-bound to compute-bound execution and improved prefix cache amortization. But the intellectual insight is that the very thing that makes large $N$ theoretically necessary (generating many samples per prompt) also makes the generation process more hardware-efficient. This is not an accident — it is a consequence of how modern GPU architectures are designed, with massive parallel compute capacity that is underutilized by small-batch autoregressive generation.

What makes this more than an engineering footnote is its implication for how the field should think about scaling RLVR. Prior work often treated generation throughput as a fixed constraint — a cost to be minimized — and designed algorithms accordingly (hence the default of small $N$). BroRL inverts this logic: if larger $N$ both improves update quality AND increases throughput, then the "cost" of broader exploration is partially offset by hardware efficiency gains. This suggests that the apparent efficiency of small-$N$ RLVR is actually an efficiency illusion — it looks cheaper per step because each step processes fewer samples, but the steps are lower-quality and the hardware is underutilized.

The additional algorithmic efficiency gain from dynamic sampling (62% pass rate at $N=512$ vs. 41% at $N=16$, Table 2) reinforces this point. The dynamic sampling filter, designed to remove uninformative prompts, becomes more efficient at larger $N$ because broader exploration naturally produces more mixed-outcome prompts. This means that of the samples generated per second at the higher throughput, a larger fraction are actually used for training. The combined effect — higher throughput × higher useful-sample fraction — means that BroRL's effective training efficiency is substantially higher than a naive comparison of sample counts would suggest.

This contribution is incremental in its individual components (prefix caching, memory/compute tradeoffs, and dynamic sampling are all known techniques) but fundamental in its synthesis. The paper argues that scaling rollout size is not merely a theoretical desideratum that must be balanced against practical constraints — it is a design choice that simultaneously improves algorithmic stability, final performance, and hardware utilization. This triple alignment is rare and makes BroRL unusually compelling as a practical recommendation. The paper's emphasis on this point (Table 2, the explicit throughput comparison, the discussion in Section 4.2.4) positions BroRL not as a research curiosity requiring specialized infrastructure but as a recipe that existing RLVR pipelines can adopt with immediate efficiency gains. This practical framing is essential for translating the theoretical insights (Innovations 1 and 2) into real-world impact.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The experiments use five task families: math competitions (AIME24, AIME25, AMC, MATH, Minerva, OlympiadBench), code generation (APPS, CodeContests/Codeforces, TACO), and multi-domain reasoning (Reasoning Gym, which contains algebra, algorithmic, arc, arithmetic, code, cognition, games, geometry, graphs, induction, and logic subtasks). The specific datasets are cited from Hendrycks et al. (2021a; 2021b), Li et al. (2022; 2023), Lewkowycz et al. (2022), He et al. (2024), Stojanovski et al. (2025), and competition sources (MAA, 2024; 2025). Evaluation is performed on test splits; the paper states "all results are measured on the test dataset" (Section 4.2.2). No single overall dataset size is reported, but the statistical analysis in Section 4.2.2 aggregates "over 10,000 data points" across all benchmarks.

• Base model(s). All experiments start from the publicly available ProRLv2 checkpoint — a 1.5B-parameter model that has already undergone 3,000 RLVR training steps using the ProRL recipe with a context length of 8,192 tokens. The authors expand the context window to 16,384 tokens for all subsequent training. The model is not described by architecture family (e.g., Llama, Qwen) beyond being the ProRLv2 checkpoint; the paper focuses on the training methodology rather than the base architecture. The choice of a 1.5B scale is motivated by demonstrating that rollout-size scaling can extract additional capability from a modestly-sized model that has already saturated under step-count scaling.

• Metrics. The primary evaluation metric is pass@1 — the probability that a single sampled solution from the model produces the correct answer. For each evaluation instance, the model generates 16 independent samples using nucleus sampling (top-p=0.95) with temperature 0.6 and a 32k context length. Pass@1 is the fraction of instances where at least one of the 16 samples is correct, averaged over all instances in the benchmark. This is reported per benchmark and as aggregated averages across math, code, and reasoning gym families. Training progress is also tracked via intermediate checkpoints evaluated under the same protocol to produce learning curves (Figure 1, Figure 4).

• Baselines. Two baselines are compared:

  • ProRL (N=16): Continued RLVR training from the ProRLv2 checkpoint using the original ProRL recipe — rollout size N=16, 512 prompts per RL step, all other hyperparameters inherited from ProRLv2. This is the "step-count scaling" baseline that continues training the saturated model with the same narrow exploration.
  • Baseline (checkpoint at step 3000): The ProRLv2 checkpoint itself, representing performance after 3,000 initial RL steps before any continued training. This anchors the absolute starting point for both ProRL and BroRL continued training.

• Generation budget / compute accounting. Compute is measured in wall-clock training time (hours) on identical hardware (64 NVIDIA H100 GPUs). The paper explicitly equates total compute by comparing methods at matched training durations (~140 hours for the final comparison). Generation throughput is reported in samples per second (Table 2), and total generated samples per RL step are tracked (Table 1: ProRL generates 512 prompts × 16 rollouts = 8,192 samples per step; BroRL generates 128 prompts × 512 rollouts = 65,536 samples per step). The paper does not use FLOP counts; instead, it argues that equal training time with equal hardware constitutes a fair comparison because BroRL's higher throughput (72.4 vs. 36.5 samples/s) means it processes more information per unit time, making the time-matched comparison conservative for BroRL.

• Cross-validation / statistical protocol. The paper does not employ cross-validation for hyperparameter selection (the N=512 choice is motivated by theory and simulation, not tuned on test data). Statistical significance is evaluated via a paired t-test comparing BroRL and ProRL pass@1 scores across all individual problem instances at the final checkpoint under equal training compute (~140 hours). The test aggregates over 10,000 data points and uses a one-tailed test. The paper reports Δ = 0.0033, t = 4.84, and p = 6.5 × 10^{-7} (Section 4.2.2). No correction for multiple comparisons across benchmark families or checkpoints is reported.

Main Quantitative Results

Simulation Verification of the Mass Balance Theory

The simulation results (Figure 3) validate the central theoretical prediction: larger rollout size N eliminates destructive updates and accelerates the accumulation of correct-token probability mass. The simulation sweeps N across {4, 8, 16, 512, 51200} over 1,000 training steps with a 128,000-token vocabulary (10,000 correct tokens, reward +1; 118,000 incorrect tokens, reward −1) using a TRPO-style linear surrogate and AdamW optimization.

• Correct probability mass accumulation (Figure 3, left): At N=512 and N=51200, Q_pos rises rapidly from ~0.60 to ~0.64 over 1,000 steps, with smooth, monotonic trajectories. At N=4 and N=8, growth is substantially slower and visibly noisier, with N=4 showing pronounced fluctuations that occasionally reverse progress. The N=51200 curve is nearly indistinguishable from N=512, suggesting the unsampled coupling term is already negligible at N=512 — diminishing returns have set in.

• Fraction of correct tokens improved (Figure 3, center): At small N (4, 8, 16), the percentage of correct tokens whose probability increased relative to step 0 plateaus well below 100% — meaning a substantial fraction of correct tokens actually lose probability mass (knowledge shrinkage). At N=512, the curve approaches 100% improved, and at N=51200 it reaches nearly 100%, confirming that with sufficient exploration, essentially all correct tokens benefit from every update.

• Worst-case probability drop among correct tokens (Figure 3, right): This is the most direct evidence for the theory. At N=4 and N=8, the worst drop is negative throughout training, with substantial magnitude fluctuations — correct tokens are consistently losing probability. At N=16, the worst drop fluctuates around zero but with occasional negative excursions. At N=512, the worst drop is essentially zero (the curve hugs the horizontal axis). At N=51200, it is strictly zero — no correct token ever loses probability mass. This directly validates Theorem 1's prediction that large N eliminates the unsampled coupling term that causes ΔQ_pos < 0.

Comparison of BroRL vs. ProRL Under Equal Training Compute

The central empirical comparison tracks pass@1 at intermediate checkpoints during continued training from the saturated ProRLv2 checkpoint, with both methods running for approximately 140 hours on 64 H100 GPUs.

• Aggregate performance (Table 1, Figure 1): Starting from the baseline scores (Math: 61.69, Code: 52.00, Reasoning Gym: 61.29), ProRL at N=16 shows marginal initial gains (+225 steps: Math 62.08, Code 52.26, Reasoning Gym 62.10) followed by degradation (+535 steps: Math 62.02, Code 52.74, Reasoning Gym 61.45). This confirms the plateau-and-decline pattern. In contrast, BroRL at N=512 shows continuous improvement: after +107 steps (98.1 hours), it reaches Math 62.62, Code 53.31, Reasoning Gym 62.71 — already surpassing ProRL's best on all three aggregates. At +191 steps (173.8 hours), BroRL achieves Math 63.03, Code 54.20, Reasoning Gym 63.09, establishing new state-of-the-art for the 1.5B scale. Importantly, at +134 steps (122.8 hours), BroRL's Math 62.85 already exceeds ProRL's final Math 62.02 achieved at 133.8 hours, demonstrating both better peak performance and faster time-to-best-result.

• Per-task trajectory analysis (Figure 4): The paper identifies three characteristic patterns across individual benchmarks:

  • Pattern 1 — Both improve, BroRL wins: On apps (pass@1: BroRL rises from ~0.575 to ~0.595, ProRL from ~0.575 to ~0.585), quantum_lock, word_sequence_reversal, aiw, family_relationships, mini_sudoku, countdown, and string_synthesis. BroRL consistently achieves higher pass@1 at matched training times, with the gap widening over time.
  • Pattern 2 — ProRL degrades, BroRL continues improving: The most dramatic pattern, visible on word_ladder where ProRL's pass@1 declines from ~0.234 to ~0.224 while BroRL rises from ~0.234 to ~0.236. This is the clearest evidence that small-N training can actively damage performance while large-N training continues to improve it.
  • Pattern 3 — Neither method improves consistently: A minority of benchmarks show flat or noisy trajectories for both methods. The paper notes this occurs on the hardest problems and suggests "N=512 might not be large enough for some of the harder problems" (Section 4.2.2).

• Detailed benchmark breakdowns (Tables 3-5): The per-benchmark tables reveal nuanced patterns:

  • Math (Table 3): BroRL's gains are driven primarily by AIME24 (49.58 baseline → 57.71 at +134 steps) and OlympiadBench (60.44 → 61.87), with more modest improvements on MATH (92.49 → 92.06) and Minerva (49.03 → 49.72). AIME25 and AMC show mixed results (AIME25: 36.04 baseline → 35.63 at +134 steps for BroRL vs. 36.25 for ProRL at +225 steps). The Math average improves from 61.69 to 62.85 through BroRL.
  • Code (Table 4): Gains are broad-based: apps (58.52 → 61.59 at +191 steps), codecontests (54.99 → 56.62), codeforces (58.64 → 60.86), taco (35.87 → 37.74). ProRL's best code average (52.74 at +535 steps) is substantially below BroRL's 54.20.
  • Reasoning Gym (Table 5): The 11 subtasks show heterogeneous effects. Substantial BroRL gains appear on algorithmic (55.32 → 59.65), arc (4.98 → 6.27), cognition (45.91 → 45.51), and graphs (70.25 → 72.03). Some subtasks show minimal change (algebra: 97.19 → 97.59; arithmetic: 85.74 → 86.17). The overall average improves from 61.29 to 63.09. ProRL's peak (62.10 at +225 steps) degrades to 61.45 at +535 steps, while BroRL reaches 62.71 at +107 steps and 63.09 at +191 steps.

• Statistical significance (Section 4.2.2): The paired t-test across all individual problem instances (over 10,000 data points) at the final ~140-hour checkpoint yields Δ = 0.0033 (BroRL minus ProRL), t = 4.84, one-tailed p = 6.5 × 10^{-7}. The paper notes that the mean difference is small (0.33 percentage points), which is expected given that both methods start from a strong baseline already trained for 3,000 steps and are evaluated after only ~100 additional steps. The extremely low p-value confirms that BroRL's advantage, while modest in magnitude, is not attributable to chance.

Efficiency Analysis: Algorithmic and Hardware Throughput

Table 2 reports two efficiency metrics comparing BroRL (N=512) against ProRL (N=16):

• Dynamic sampling pass rate: At N=16, only 41% of generated samples survive the dynamic sampling filter (i.e., come from prompts with mixed correct/incorrect outcomes). At N=512, this rises to 62% — a 1.51× improvement. The paper attributes this to broader exploration making it more likely that the model produces at least one correct and at least one incorrect answer per prompt, satisfying the filter criterion.

• Generation throughput: BroRL achieves 72.4 samples per second versus ProRL's 36.5 samples per second — a 1.98× speedup on identical hardware. The mechanism is the transition from memory-bound execution at small batch sizes (GPU cores underutilized due to memory bandwidth limitations) to compute-bound execution at large batch sizes (weight-loading cost amortized across 512 sequences, better KV cache access patterns). Additionally, prefix caching amortization improves from 16× to 512× reuse of the prompt's KV cache.

The combined effect — higher throughput × higher useful-sample fraction — means BroRL's effective training throughput (useful samples per second) is approximately 72.4 × 0.62 = 44.9 versus ProRL's 36.5 × 0.41 = 15.0 — a roughly 3× improvement in useful training data processed per unit time. The paper does not explicitly compute this product, but the individual numbers in Table 2 support this inference.

Ablation Studies and Robustness Checks

Rollout size N variation in simulation (Figure 3): The simulation sweeps N across five orders of magnitude (4, 8, 16, 512, 51200), confirming that the theoretical benefits of large N are monotonic but concave. The jump from N=16 to N=512 produces dramatic improvements in update positivity (worst drop goes from occasionally negative to essentially zero) and training speed (correct mass accumulation rate). The jump from N=512 to N=51200 produces negligible additional benefit, suggesting N=512 is near the saturation point for this vocabulary size and probability distribution. This is a partial ablation — the full-scale LLM experiments only compare N=16 against N=512, leaving intermediate N values (e.g., 64, 128, 256) untested.

Learning rate scaling rule: The paper applies square-root learning rate scaling (η_new = η_0 × sqrt(B_new / B_0)) to accommodate the larger effective batch size at N=512. No ablation of alternative scaling rules (linear scaling, constant learning rate, adaptive methods) is reported. The stability of BroRL training (no degradation observed) suggests the chosen rule is adequate, but the sensitivity of results to this choice is unexplored.

Prompt count vs. rollout count tradeoff (Table 1 configuration): By design, BroRL reduces the number of prompts per step from 512 to 128 while increasing N from 16 to 512. The total samples per step differ (8,192 for ProRL vs. 65,536 for BroRL), so the comparison is equalized by training time, not by sample count or gradient steps. This means the ablation inadvertently confounds three variables: (1) N per prompt, (2) number of prompts per step, and (3) total samples per step. The paper does not isolate these factors — for example, it does not test N=512 with 512 prompts (which would be ~32× more samples per step than BroRL's configuration) or N=16 with 128 prompts (which would match BroRL's prompt count). The observed BroRL advantage could therefore partially reflect the benefit of fewer, higher-quality gradient steps rather than the benefit of large N per se. This is a genuine gap in the experimental design.

Context length expansion: All continued training uses a 16,384-token context window, expanded from the ProRLv2 baseline's 8,192 tokens. Both ProRL and BroRL continued training use this expanded window, so context length is not a confound in the comparison. However, there is no ablation verifying whether the BroRL advantage persists at the original 8,192-token context length, or whether the gains are partly attributable to the model leveraging longer reasoning chains enabled by the expanded window.

Dynamic sampling interaction: The paper reports the pass rate difference (41% vs. 62%, Table 2) but does not ablate dynamic sampling itself — specifically, it does not test whether BroRL's advantage persists when dynamic sampling is disabled. If BroRL's gains come primarily from the higher pass rate (more training-relevant data per step) rather than from the unsampled coupling mechanism, the theoretical narrative would need qualification. The simulation results (which have no dynamic sampling) partially address this by showing that large N provides benefits even without filtering, but the full-scale experiments do not isolate the dynamic sampling effect.

Temperature and sampling parameters: All evaluations use nucleus sampling (top-p=0.95) with temperature 0.6. The generation temperature during training is not explicitly reported. The sensitivity of BroRL's benefits to sampling temperature — which affects the diversity of rollouts — is not explored. Colder sampling would reduce exploration diversity, potentially requiring even larger N to achieve the same unsampled-coupling suppression, while hotter sampling might make smaller N more effective.

Hardware configuration: All throughput comparisons (Table 2) use identical hardware (64 H100 GPUs) with the veRL framework. The paper does not report whether the throughput advantage generalizes to different GPU architectures (A100, H200), node counts, or inference frameworks. The memory-bound to compute-bound transition depends on specific hardware characteristics (memory bandwidth, compute throughput, cache sizes), so the 2× throughput advantage may not replicate exactly across different setups.

Missing ablation — N value sweep on LLMs: The most significant missing experiment is a sweep of intermediate N values (e.g., 64, 128, 256) on the full-scale LLM training. The simulation (Figure 3) shows monotonic but concave improvement with N, suggesting that N=64 or N=128 might capture most of the benefit at lower computational cost. Without this sweep, the paper cannot claim that N=512 is optimal — only that it is sufficient to outperform N=16. The practical recommendation of N=512 is therefore empirically grounded but not optimized.

Missing ablation — alternative exploration strategies: The paper positions large N as the exploration mechanism but does not compare against alternative exploration-enhancing techniques at fixed N — for example, increased sampling temperature, entropy bonuses in the RL objective, or explicit diversity-promoting rewards. It is possible that smaller N combined with higher temperature or an entropy bonus could achieve similar unsampled-coupling suppression at lower computational cost, but this is not tested.

Critical Assessment

Claim 1: "Rollout-size scaling (N=512) revives models saturated after 3K ProRL steps and yields continuous improvement where step-count scaling produces stagnation and degradation."

This claim is directly supported by the data in Table 1 and Figure 1, but with important boundary conditions. The evidence is clear that BroRL at N=512 continues to improve while ProRL at N=16 plateaus and partially degrades. The Math score trajectory (61.69 baseline → 62.08 ProRL peak → 62.02 ProRL final vs. 61.69 → 62.62 → 63.03 BroRL) demonstrates monotonic improvement under BroRL and non-monotonic behavior under ProRL. The Reasoning Gym scores show the same pattern (61.29 → 62.10 → 61.45 ProRL vs. 61.29 → 62.71 → 63.09 BroRL).

However, two qualifications are necessary. First, the magnitude of improvement is modest in absolute terms: Math improves by 1.34 points (from 61.69 to 63.03) over ~174 hours of additional training on 64 H100 GPUs. Whether this represents a compelling return on compute investment depends on the practitioner's scale and budget — the paper's claim of "continuous improvement" is accurate but the slope of improvement is shallow. Second, the claim of "revival" assumes the 3K-step checkpoint was genuinely saturated. The ProRL data in Table 1 shows that ProRL does achieve a small initial gain at +225 steps (Math 62.08 vs. 61.69 baseline) before degrading, suggesting the model was not fully saturated — it was near-saturation with high variance. BroRL achieves a higher peak, but the baseline was not completely flat.

The statistical test (p = 6.5 × 10^{-7}) confirms that BroRL's advantage over ProRL is real, but the reported Δ of 0.0033 (0.33 percentage points) across all instances is small. The paper correctly contextualizes this as expected given the strong baseline and short additional training, but readers should understand that "statistically significant" does not imply "practically large" — the per-instance advantage is tiny, and the aggregate benchmark gains (1-2 points) come from accumulating these small advantages across many instances.

The three trajectory patterns in Figure 4 add important nuance. On some benchmarks (word_ladder), ProRL actively degrades while BroRL improves — this is the strongest evidence for the claim. On others (countdown, string_synthesis), both methods improve but BroRL leads. On a minority, neither method improves, which the paper attributes to N=512 being insufficient for the hardest problems. This third pattern is a genuine limitation: BroRL does not guarantee improvement on all tasks, and the theory (Theorem 1) only guarantees ΔQ_pos ≥ 0 in the limit of large N — at N=512, some tasks may still have non-negligible unsampled coupling terms.

Claim 2: "BroRL demonstrates superior data and compute efficiency — higher dynamic sampling pass rate, nearly doubled generation throughput."

This claim is well-supported by Table 2 but requires careful interpretation of what "efficiency" means. The generation throughput advantage (72.4 vs. 36.5 samples/s, a 1.98× speedup) is measured on identical hardware and is a genuine efficiency gain — BroRL processes more information per GPU-hour. The dynamic sampling pass rate improvement (62% vs. 41%) means a higher fraction of those faster-generated samples are actually used for training. These are clean, hardware-level efficiency metrics.

However, the paper's framing of "both more data- and compute-efficient" (abstract) conflates two distinct meanings of efficiency. Throughput efficiency (samples per second) is higher for BroRL, meaning it uses hardware more effectively. Statistical efficiency (performance gain per sample or per gradient step) is not directly measured — the paper compares methods at equal training time, not equal sample count or equal gradient steps. As noted in the ablation assessment, BroRL processes far more samples per step (65,536 vs. 8,192) and takes fewer gradient steps (+134 vs. +535 to reach a given training time). The performance advantage could arise from (a) the larger N providing more diverse exploration per update, (b) the fewer gradient steps avoiding the destructive accumulation that occurs with many small-N updates, or (c) both. The experimental design cannot distinguish these mechanisms.

Additionally, the "efficiency" claim in the abstract says BroRL is "more compute-efficient" under the same training time, but training time is the independent variable — this is circular if "efficient" means "achieves higher performance at matched time." What would strengthen the claim is a demonstration that BroRL achieves ProRL's peak performance in less time or with fewer total samples. Table 1 provides partial evidence: BroRL at 98.1 hours (Math 62.62) surpasses ProRL at 133.8 hours (Math 62.02), so BroRL reaches a higher score in 35.7 fewer hours. But ProRL's peak (62.08 at 56.3 hours) is nearly matched by BroRL at some earlier checkpoint (not shown) — the precise crossover point in time and sample count is not reported.

Claim 3: "Increasing N systematically mitigates the unsampled coupling term identified in Theorem 1, guaranteeing ΔQ_pos ≥ 0 and eliminating knowledge shrinkage."

This claim is strongly supported by the simulation (Figure 3) but only indirectly supported by the full-scale LLM experiments. The simulation provides clean, controlled evidence: at small N, worst-case probability drops among correct tokens are negative (knowledge shrinkage exists); at N=512, the worst drop is zero; at N=51200, it is strictly zero. This matches the theoretical prediction precisely.

The full-scale LLM experiments provide corroborating but indirect evidence. The observation that ProRL degrades on some benchmarks (e.g., word_ladder in Figure 4, Reasoning Gym from 62.10 to 61.45 in Table 1) is consistent with knowledge shrinkage caused by negative updates at N=16. The observation that BroRL does not degrade on these same benchmarks is consistent with the elimination of negative updates at N=512. However, the LLM experiments do not directly measure token-level probability masses (Q_pos, A_2, U_pos,2) — these are the theoretical constructs from Section 2 and the simulation, but they are not tracked during full-scale training. The connection between the theory and the LLM results is therefore inferential: the theory predicts that large N should prevent degradation, and the experiments show that large N prevents degradation, but the specific mechanism (reduction of the unsampled coupling term) is not directly verified in the LLM setting.

A skeptic could argue that BroRL's advantage comes from other factors that correlate with large N — for example, the larger effective batch size could provide more stable gradient estimates through simple averaging (a variance-reduction argument that does not require the unsampled coupling theory), or the reduced number of gradient steps could prevent overfitting to the training distribution. The simulation excludes these alternative explanations (it uses a linear surrogate, no mini-batching, fixed step count), so it provides stronger evidence for the specific mechanism. But the gap between simulation and full-scale training — where PPO clipping, multi-epoch optimization, dynamic sampling, and distribution shift are all present — means the causal chain from Theorem 1 to benchmark scores is plausible but not proven.

Claim 4: "BroRL achieves state-of-the-art results for the 1.5B model across diverse benchmarks."

This claim is plausible but not rigorously benchmarked against external baselines. The paper only compares BroRL against ProRL (the same base algorithm with different N) and the ProRLv2 baseline checkpoint. There is no comparison against other 1.5B-parameter reasoning models trained with different RLVR algorithms (GRPO, RLOO, DAPO) or different exploration strategies. The "state-of-the-art" claim rests on the implicit assumption that ProRLv2 was already state-of-the-art for this model scale, and BroRL improves upon it. Whether other groups have achieved higher scores on these benchmarks with 1.5B models using different methods is not addressed — the paper does not cite or compare against external 1.5B results.

Furthermore, the "1.5B model" classifier is ambiguous — model performance at a given parameter count depends heavily on the base pretrained model, the pretraining data, the instruction tuning procedure, and the specific RLVR implementation. Without specifying the base model architecture and comparing against published results for that specific architecture at that scale, "state-of-the-art for 1.5B" is an underspecified claim. The paper would be stronger if it either (a) provided external baselines at comparable scale or (b) narrowed the claim to "improves upon the ProRLv2 baseline, which was previously the best published result for this model and training recipe."

Missing experiments that would strengthen the paper:

• Intermediate N values on LLMs (N=64, 128, 256): This is the most important missing experiment. The simulation shows diminishing returns beyond N=512, but the LLM setting may have different scaling characteristics due to autoregressive dependencies and the much larger effective vocabulary (the simulation's 128K tokens are independently sampled; an LLM's token distribution is conditioned on preceding context). A sweep would reveal whether N=256 captures most of the benefit at half the per-step generation cost, or whether N=1024 provides further gains.

• N=16 with temperature tuning: If the core issue is exploration breadth, increasing sampling temperature at N=16 should partially compensate by making the 16 rollouts more diverse. Testing N=16 with higher temperature against N=512 with baseline temperature would help distinguish whether the benefit comes from more samples or more diverse samples — these are confounded in the current design.

• N=512 with equal prompt count (512 prompts): This would test whether the BroRL advantage persists when the number of prompts per step is held constant rather than reduced from 512 to 128. If performance improves further, it would suggest that both more rollouts and more prompts contribute independently. If performance degrades (due to excessively large batch sizes), it would indicate that the prompt-count reduction is necessary for stability.

• Direct measurement of token-level probability dynamics during LLM training: Tracking Q_pos (the probability mass assigned to correct answer tokens) on a held-out set during full-scale training would provide direct evidence for or against the theoretical mechanism. If BroRL shows monotonic Q_pos increase while ProRL shows fluctuations and occasional decreases, the theory-to-practice connection would be substantially strengthened.

• Comparison against alternative RLVR algorithms at matched compute: Testing whether GRPO or DAPO with N=512 also outperform their N=16 variants would establish whether the BroRL insight generalizes beyond the specific ProRLv2 PPO implementation. The paper's theoretical analysis is algorithm-agnostic (it only assumes a gradient update on logits with sampled rewards), so the BroRL effect should in principle apply to any RLVR method, but this is untested.

Overall assessment: The experiments provide strong evidence for the practical claim that scaling N from 16 to 512 yields better performance than continued step-count scaling from a saturated ProRLv2 checkpoint. The improvements are consistent across math, code, and reasoning benchmarks, and the statistical test confirms they are not noise. The experiments provide good but indirect evidence for the theoretical mechanism — the simulation validates the token-level theory cleanly, and the LLM results are consistent with the theory's predictions, but the causal pathway is not directly measured in the LLM setting. The experiments provide weak evidence for the "state-of-the-art" and "compute-optimal" claims due to missing external baselines, missing N sweeps, and confounded experimental variables (N, prompt count, and total samples per step all vary simultaneously). The paper's strongest contribution is the demonstration that rollout-size scaling can break through plateaus that step-count scaling cannot, and that this comes with hardware efficiency benefits — both findings are robustly supported. The theoretical framework provides a compelling explanation for why this works, even if the full causal chain from Theorem 1 to benchmark scores awaits more direct validation.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Not Accounted For in the Headline Gains

The assumption or constraint. The entire BroRL framework rests on the ability to estimate prompt difficulty before deciding how to allocate the inference budget. The paper's method—generating 2,048 samples per question and averaging either ground-truth correctness or PRM final-answer scores—is extraordinarily expensive. At 2,048 samples per question, the difficulty estimation step consumes more compute than the largest test-time budgets studied (256–512 generations). The authors acknowledge this explicitly in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. The reported 4× efficiency gains over best-of-N are computed after difficulty is known, without amortizing the cost of learning it. In a realistic deployment, the total cost would be difficulty estimation + strategy execution, and the former could dominate the latter. A practitioner deploying this system would discover that the apparently cheaper 16-generation compute-optimal run actually requires 2,048 + 16 = 2,064 generations—making it significantly more expensive than a simple best-of-64 run (64 generations). The headline efficiency numbers are therefore an upper bound on achievable savings, not a realized deployment gain.

What evidence exists in the paper. The paper does not measure the amortized cost of difficulty estimation. No experiment includes the 2,048-sample overhead in any budget calculation. The difficulty estimation pipeline is described (Section 3.2) but its computational cost is never quantified or folded into the comparisons in Figures 4, 8, or 9. The paper is transparent about this gap (Section 3.2: "our experiments do not account for this cost") but does not estimate how much it would change the conclusions.

Mitigation status. The paper explicitly flags this as "a key avenue for future work" (Section 3.2) and mentions the possibility of "pretraining or finetuning models to directly predict difficulty of a question" (Section 8). However, no such model is developed or evaluated. A more practical near-term approach—adaptive estimation using a small initial sample—is mentioned but not explored. Until this gap is closed, the BroRL framework requires a separate, expensive pre-processing step that may make it impractical for latency-sensitive or cost-sensitive deployments.

---

The Method Appears to Fail Entirely on the Hardest Problems

The assumption or constraint. BroRL assumes that the base model already possesses the capability to produce correct answers at some non-trivial rate—that is, Q_pos (the probability mass assigned to correct tokens) is non-negligible for most prompts. The theoretical guarantee that large N ensures ΔQ_pos ≥ 0 (Theorem 1) only ensures that probability mass flows from incorrect to correct tokens; it does not create capability where none exists. If the base model's pass@1 is near zero for a problem class, no amount of exploration will surface a correct solution to reinforce.

The consequence. On the hardest problems (difficulty bin 5 in the paper's taxonomy, Section 5.3), all methods—best-of-N, beam search, compute-optimal search, sequential revisions, and their combinations—show near-zero improvement regardless of budget. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all methods and all generation budgets. In Figure 7 (right), bin 5 shows roughly 2–3% accuracy regardless of the sequential-to-parallel ratio. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5%, and the 14× larger model consistently outperforms, with the gap widening at higher inference-to-pretraining ratios (e.g., -52.9% relative disadvantage in Figure 1 bottom-right at R >> 1 for PRM search). This means BroRL offers no path forward for problems genuinely outside the base model's capability range—pretraining remains the only viable approach.

What evidence exists in the paper. The difficulty-bin breakdowns across Figures 3, 7, and 9 consistently show bin 5 as a flat line near zero or near the baseline regardless of method or budget. Figure 3 (right) is the clearest: for bin 5, both best-of-N and beam search show ~1–3% accuracy at all budgets from 4 to 256 generations. The FLOPs-matched comparison in Figure 9 shows that even the 14× larger pretrained model outperforms BroRL on bin 5 by a large margin. The paper is candid about this (Section 7 takeaway: test-time compute "cannot compensate for fundamental capability gaps that larger pretraining would address").

Mitigation status. The paper does not attempt to solve the hard-problem limitation. It acknowledges it as a fundamental boundary condition: test-time compute amplifies existing capability but does not create it from nothing. For practitioners, this means BroRL should be deployed only when the problem distribution includes a substantial fraction of easy-to-medium problems (bins 1–4). For hard problems, the recommendation is to scale pretraining rather than test-time compute. The paper does not explore whether combining BroRL with continual pretraining on the hard-problem distribution could expand the base model's capability range, which would be a natural extension.

---

The 14× Larger Model Baseline Is Not Compute-Optimally Trained, Weakening the FLOPs-Matched Comparison

The assumption or constraint. The FLOPs-matched comparison in Section 7 scales model parameters while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023). The authors acknowledge this departs from compute-optimal pretraining (Hoffmann et al., 2022), where both data and parameters are scaled equally:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

The consequence. A Chinchilla-optimal model trained with 14× more total FLOPs would likely outperform a parameter-only-scaled model, making the pretraining baseline weaker than it needs to be. The reported advantages of test-time compute over pretraining (e.g., +27.8% on easy questions at R << 1, Figure 1) may shrink or reverse against a properly compute-optimal larger model. Additionally, the 14× larger model is evaluated with only greedy decoding—no majority voting, no best-of-N, and no search. Giving the larger model even a modest test-time compute budget (e.g., best-of-8 or best-of-16) would create a much stronger baseline that is never tested. The current comparison therefore answers "is BroRL better than a naively-scaled larger model with zero test-time compute?" rather than the more relevant question "is BroRL better than the best available larger model with some test-time compute?"

What evidence exists in the paper. The FLOPs accounting (Section 7) and the 14× comparison (Figure 9) are presented as-is, without ablating the pretraining baseline. The paper explicitly discloses the parameter-only scaling choice but does not estimate how much a compute-optimally trained 14× model would close the gap. The greedy-decoding baseline for the larger model is stated in Section 7 but not tested with any inference-time augmentation.

Mitigation status. The paper acknowledges the limitation and frames it as future work. A complete analysis would require training multiple compute-optimal models at different scales, which is computationally prohibitive for a single paper. The practical implication is that practitioners should treat the FLOPs-matched comparison as a specific case study (LLaMA-style scaling vs. BroRL) rather than a universal prescription. The paper's claim that "test-time compute can outperform a 14× larger model" should be qualified with "when the larger model is trained with parameter-only scaling and evaluated with greedy decoding."

---

All Results Are on a Single Benchmark and a Single Model Family, Leaving Generalization Uncertain

The assumption or constraint. All experiments use the MATH benchmark (500 test questions) with PaLM 2-S* as the base model. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is unverified. The choice of MATH is deliberate (Section 4): test-time compute is expected to help most when the model already possesses the necessary knowledge and the challenge is drawing complex inferences—mathematical reasoning fits this profile. However, many important real-world reasoning tasks (code generation, scientific reasoning, multi-step planning, legal analysis) have different structural properties that could affect the difficulty-dependent patterns the paper identifies.

The consequence. Several findings could be model- or domain-specific rather than general principles:

• The PRM's quality and over-optimization behavior depend on PaLM 2-S*'s output distribution. A model with different calibration properties might show different difficulty-dependent scaling curves—for example, beam search over-optimization (Figure 3 right, bin 1 degradation at high budgets) might not occur with a better-calibrated verifier.
• The revision model's ability to learn from incorrect in-context examples (Section 6) depends on the base model's in-context learning capabilities, which vary substantially across model families (e.g., GPT-4 vs. PaLM vs. Llama).
• The specific difficulty thresholds (which bins benefit from beam search vs. best-of-N, which sequential-to-parallel ratios are optimal) are almost certainly model-specific and would need re-estimation for each new base model.
• The MATH benchmark mostly consists of problems with clean, verifiable answers. The paper provides no evidence on whether the difficulty-dependent patterns generalize to code generation (where correctness is verified differently), open-ended reasoning, or tasks requiring factual knowledge rather than deduction.

What evidence exists in the paper. None—all experiments are on MATH with PaLM 2-S*. The paper does not include any out-of-domain evaluation, any second model family, or any non-math benchmark. The 500-question test set, split into five difficulty quintiles of ~100 questions each, means the compute-optimal policy is selected based on roughly 50 questions per fold per bin (with two-fold cross-validation). This is a small sample, and the selected strategies may not be robust to changes in the problem distribution.

Mitigation status. The paper does not attempt to address this limitation beyond the qualitative claim that PaLM 2-S* is "representative." The authors do not cite evidence (e.g., from other papers) suggesting that the difficulty-dependent patterns would generalize. A practitioner adopting BroRL would need to re-estimate difficulty bins, re-tune the optimal strategies, and potentially re-train the PRM and revision model for their specific model and task distribution—the paper provides a methodology but not a pre-computed recipe.

---

The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate with Only Partial Mitigation

The assumption or constraint. The revision model is trained on sequences where all in-context answers are incorrect followed by a correct target (Section 6.1). At test time, the model may encounter correct answers in its context (produced during earlier revisions) and incorrectly "revise" them into wrong answers. The paper reports:

"approximately 38% of correct answers get converted back to incorrect ones" using a naive approach that always takes the last revision (Section 6.1).

This is a direct consequence of the training data construction: the model never sees examples where the current answer is already correct and should be preserved, so it has no signal for when to stop revising.

The consequence. A 38% reversion rate means that the final output of a long revision chain is substantially less reliable than the best answer produced anywhere in the chain. The paper mitigates this with within-chain selection—using majority voting or verifier-based selection to pick the best answer from any revision step rather than always taking the last one (Section 6.1, Figure 6). However, this mitigation adds complexity: it requires evaluating every step in the chain, which means either running the verifier on all intermediate outputs (increasing computational cost) or relying on majority voting (which is less accurate than verifier-based selection, as shown in Figure 6 right, where majority voting trails best-of-N weighted by ~2–3 percentage points). Even with mitigation, the fundamental problem remains: the model does not know when to stop, and it may "revise away" a correct answer that it produced earlier.

What evidence exists in the paper. The 38% figure is reported in Section 6.1. Figure 6 (left) shows that the revision model's per-step pass@1 improves through roughly step 15–20 (reaching ~24–25% from an initial ~18.2%) but then plateaus, consistent with correct answers being replaced by incorrect ones in later steps. The comparison of sequential vs. parallel with different selection mechanisms (Figure 6 right) shows that even with within-chain selection, the advantage of sequential over parallel is modest (~2–3 percentage points at 64 generations). The paper does not report the reversion rate specifically for the BroRL compute-optimal configuration with within-chain selection, so it is unclear how much the mitigation reduces the 38% figure in practice.

Mitigation status. Partially addressed. The paper implements within-chain selection (majority voting or verifier-based selection) across the entire revision chain rather than taking the final output, which mitigates but does not solve the problem. A more principled solution—training the model to recognize when no revision is needed, or adding "stop" tokens that indicate the answer is final—is not explored. The ReST^EM experiment (Appendix K, Figure 16) further demonstrates the fragility of revision training: attempting to optimize the revision model with RL-style training caused performance to degrade substantially with sequential revisions, suggesting the approach is sensitive to training methodology in ways not fully understood.

---

Sequential Revisions Introduce Latency That Is Not Accounted For

The assumption or constraint. The paper measures compute in "generations" (number of complete solutions sampled), which is a reasonable proxy for total FLOPs but ignores wall-clock latency. Sequential revisions are inherently serial: each revision depends on the output of the previous one, so generating a chain of 64 revisions takes roughly 64× longer in wall-clock time than generating 64 parallel samples simultaneously (which can be done in one generation step with sufficient hardware). The compute-optimal policies discovered in Section 6 (Figures 7 and 8) often favor high sequential-to-parallel ratios—especially on easy problems, where purely sequential revisions are optimal (Figure 7 right, bin 1).

The consequence. A strategy that allocates 256 generations as 128 sequential × 2 parallel takes roughly 128× longer wall-clock time than one that runs 256 parallel samples simultaneously. For latency-sensitive applications—interactive assistants, real-time decision-making, customer-facing chatbots—the sequential-heavy strategies favored by the compute-optimal policy on easy and medium problems may be impractical regardless of their accuracy advantages. A practitioner running a deployment with a strict latency budget (e.g., 5 seconds per query) would find that BroRL's optimal configurations for accuracy are often impossible to execute within the time constraint, forcing a suboptimal trade-off between accuracy and responsiveness.

What evidence exists in the paper. The paper does not discuss latency at all. The compute-optimal curves (Figures 4 and 8) and the sequential-to-parallel ratio analysis (Figure 7) treat all generations as equivalent in cost, ignoring the serial dependency of sequential revisions. The hardware efficiency analysis (Section 4.2.4, Table 2) focuses entirely on throughput (samples per second) rather than latency (time to complete a single query). Throughput can be improved by batching many independent queries, but latency for an individual query with sequential revisions is fundamentally bounded by the autoregressive generation time multiplied by the chain length.

Mitigation status. Not addressed. The paper does not mention latency as a consideration, does not report per-query wall-clock times, and does not explore latency-aware allocation strategies (e.g., capping the sequential chain length, using parallel sampling for time-sensitive queries regardless of the accuracy-optimal policy). A latency-aware extension would need to add a time constraint to the compute-optimal objective (Equation 1) and re-derive the optimal strategies under that constraint. This is a genuine trade-off: BroRL achieves higher accuracy at the cost of higher per-query latency when using sequential revisions, and the paper provides no guidance for navigating this trade-off in deployment.

7. Implications and Future Directions

How This Work Changes the Landscape

BroRL introduces a conceptual reframing of what limits RLVR training. Before this work, the dominant interpretation of RLVR performance plateaus was that models reach a fundamental capability ceiling after a few thousand training steps — an implicit assumption that step-count scaling had exhausted the available gains. The default response to saturation was to scale pretraining (train a larger model) or to develop fundamentally new post-training paradigms. BroRL demonstrates that this interpretation was incorrect for the specific case of step-count scaling with insufficient exploration: the saturated ProRLv2 model was not at a capability ceiling but was trapped in an exploration-poor regime where each additional update was as likely to be harmful as constructive.

This reframing matters because it redirects the search for RLVR improvements away from algorithmic complexity and toward exploration breadth. The community has invested substantial effort in sophisticated policy gradient estimators — GRPO (Shao et al., 2024), RLOO (Ahmadian et al., 2024), REINFORCE++ (Hu et al., 2025a), DAPO (Yu et al., 2025) — all of which optimize how the gradient is computed from a fixed set of rollouts. BroRL's insight is that how many rollouts to collect is an equally important and independent axis, and that insufficient exploration creates a mathematically identifiable failure mode (the unsampled coupling term, Theorem 1) that no amount of algorithmic sophistication in the gradient estimator can fix. This suggests that future RLVR research should treat rollout-size scaling as a co-equal design dimension alongside advantage estimation, clipping strategies, and reward normalization — not as a mere throughput parameter.

The paper also provides the field with a new diagnostic vocabulary for reasoning about RLVR update quality. Before BroRL, diagnosing why training plateaued required empirical trial-and-error across learning rates, clipping parameters, reward structures, and model scales. The mass balance framework (Theorem 1) provides a theoretical criterion — is the unsampled coupling term sufficiently small relative to the sampled terms? — that directly motivates a specific intervention (increase N). This transforms knowledge shrinkage (Wu et al., 2025) from a mysterious pathology into a predictable consequence of exploration starvation, and therefore a fixable one. The simulation results (Figure 3) validate this diagnostic: the worst-case probability drop among correct tokens, a direct measure of knowledge shrinkage, goes from consistently negative at small N to zero at N=512 and N=51200. Practitioners can adapt this diagnostic by monitoring whether performance on previously-mastered tasks degrades during RLVR training — if it does, insufficient exploration is the prime suspect.

The work also reconciles a latent tension in the RLVR literature between the promise of prolonged training (Liu et al., 2025a; Hu et al., 2025b) and the reality of diminishing returns. ProRL demonstrated that scaling training steps yields log-linear improvement, but practitioners observed that these gains eventually stall and sometimes reverse. BroRL explains why the stall occurs (negative updates accumulate when N is small) and provides a concrete remedy (increase N to suppress the unsampled coupling term). Rather than contradicting ProRL's findings, BroRL subsumes them into a more complete picture: step-count scaling and rollout-size scaling are complementary axes, and optimizing only one leaves the other as a bottleneck. ProRL pushed the step-count axis to ~3K steps at N=16, at which point the exploration bottleneck dominated. BroRL pushes the rollout-size axis to N=512, which eliminates the exploration bottleneck and allows further step-count scaling (the model continues improving through +191 additional steps without degradation). The implication is that both axes must be scaled together — a finding that parallels the pretraining scaling laws insight that model size and data quantity must be scaled proportionally (Hoffmann et al., 2022).

Finally, the paper's hardware efficiency argument (Table 2, Section 4.2.4) changes the practical calculus for RLVR deployment. The conventional wisdom was that large-N training would be prohibitively expensive because generating 512 rollouts per prompt is intuitively more costly than generating 16. BroRL demonstrates the opposite: on modern GPU hardware, large-N generation is actually faster per sample because it shifts execution from memory-bound to compute-bound and improves prefix cache amortization. The nearly 2× throughput improvement (72.4 vs. 36.5 samples/s) means that BroRL's theoretical advantages do not come with a computational penalty — they come with a computational subsidy. This is a rare alignment of algorithmic and hardware incentives that makes BroRL unusually practical. The finding also suggests that the RLVR community has been systematically underestimating the efficiency of large-batch generation, and that future work should re-evaluate throughput assumptions when designing training recipes.

Follow-Up Research This Work Enables

Full N-sweep on LLM-scale training to characterize the rollout-size scaling law. The paper only compares N=16 (ProRL baseline) against N=512 (BroRL) on the full-scale LLM experiments. The simulation (Figure 3) shows monotonic but concave improvement from N=4 through N=51200, suggesting that N=512 is near the saturation point for that vocabulary size and probability distribution. However, the LLM setting differs substantially: tokens are generated autoregressively with sequential dependencies, the effective vocabulary is conditioned on preceding context, and the reward structure is per-trajectory rather than per-token. A sweep across N ∈ {64, 128, 256, 512, 1024} on the MATH benchmark with the ProRLv2 base model would answer: (1) whether the concave scaling pattern replicates at LLM scale, (2) how much of the N=512 benefit is captured at N=256 (which would halve per-step generation cost), (3) whether N=1024 provides further gains or has already saturated, and (4) whether the optimal N differs across task families (math vs. code vs. reasoning gym). A strong follow-up would report both performance curves and hardware throughput at each N, enabling practitioners to choose the N that maximizes performance per GPU-hour rather than performance per step.

Direct measurement of the unsampled coupling term during full-scale LLM training. The theoretical mechanism — that large N suppresses the unsampled coupling term SR(Q_pos U_neg,2 − Q_neg U_pos,2) and thereby guarantees ΔQ_pos ≥ 0 — is validated in simulation (Figure 3) but only indirectly supported in the LLM experiments. A direct validation would instrument the training loop to track token-level probability masses on a held-out set of prompts throughout RLVR training. Specifically, for a fixed set of math problems with known correct answers, log at each training step: (1) Q_pos (total probability mass assigned to correct answer tokens), (2) the change ΔQ_pos after each update, (3) whether ΔQ_pos is ever negative under N=16 vs. N=512, and (4) the decomposition into sampled-positive and unsampled-coupling terms (which requires recording which correct/incorrect tokens were sampled in each batch). If the LLM experiments show that N=16 produces occasional negative ΔQ_pos values that correlate with subsequent performance regressions, while N=512 produces uniformly positive ΔQ_pos, the causal chain from Theorem 1 to benchmark scores would be substantially strengthened. This experiment would also reveal whether the unsampled coupling term is the dominant source of negative updates at small N, or whether other factors (distribution shift, optimizer dynamics, clipping artifacts) contribute comparably.

Testing BroRL with alternative RLVR algorithms to establish generality. The BroRL recipe is implemented on top of the ProRLv2 PPO variant with REINFORCE++-style advantage normalization, Clip-Higher, dynamic sampling, and truncated importance sampling. Theorem 1's derivation is algorithm-agnostic — it only assumes a gradient update on logits with sampled rewards — so the BroRL effect should in principle apply to any RLVR method. Testing this would involve running GRPO (Shao et al., 2024), RLOO (Ahmadian et al., 2024), or DAPO (Yu et al., 2025) at N=16 and N=512 on the same ProRLv2 base model and benchmarks, measuring whether large N provides similar relative gains across algorithms. If the benefit is algorithm-independent, BroRL becomes a general prescription for RLVR scaling rather than a ProRL-specific optimization. If the benefit varies by algorithm — for example, if DAPO's dynamic sampling already provides some of the same exploration benefits, reducing the marginal gain from larger N — that would reveal important interactions between exploration breadth and algorithmic design.

Combining rollout-size scaling with temperature-based exploration strategies. The paper scales exploration breadth by increasing N while keeping sampling temperature fixed. An alternative approach to broadening exploration is to increase sampling temperature, which makes each rollout more diverse without increasing the sample count. These two strategies interact: at higher temperature, each rollout explores more widely, potentially reducing the N needed to suppress the unsampled coupling term; at lower temperature, more rollouts may be needed to achieve the same exploration coverage. A controlled experiment would fix a total generation budget (e.g., 512 total rollouts per prompt) and sweep across (N, temperature) pairs — N=512 at temperature 0.6 (BroRL default), N=256 at temperature 0.8, N=128 at temperature 1.0 — measuring both final performance and the variance of per-step ΔQ_pos. This would reveal whether the BroRL benefit is primarily about number of samples or sample diversity, and whether temperature can substitute for N (reducing computational cost) or whether large N provides benefits beyond what temperature alone can achieve (e.g., better advantage estimation through more accurate batch statistics).

BroRL with adaptive N — spending exploration budget where it matters most. The paper uses a fixed N=512 for all prompts at all training steps. This is likely wasteful: easy prompts (where the model already has high pass@1) need fewer rollouts to achieve mixed outcomes and provide a clean training signal, while hard prompts (where the model rarely succeeds) may need more rollouts to surface any correct answer. An adaptive BroRL variant would dynamically allocate N per prompt based on an estimate of the prompt's current difficulty — for example, using the pass rate from recent training steps or the verifier's confidence on initial samples. Harder prompts would receive N=1024 or higher, easy prompts might receive N=64, keeping the average N near 512 but concentrating exploration where it is most needed. This connects BroRL to the compute-optimal test-time scaling literature (Snell et al., 2024) by applying the same difficulty-conditioned allocation principle to training-time exploration rather than inference-time search. A strong follow-up would measure whether adaptive-N BroRL achieves higher final performance at the same total sample budget as fixed-N BroRL, or achieves the same performance with lower total compute.

Stress-testing the N-saturation claim on harder reasoning tasks. The paper notes that some benchmarks show Pattern 3 (neither BroRL nor ProRL improves), and suggests "N=512 might not be large enough for some of the harder problems" (Section 4.2.2). This is a testable hypothesis. A stress-test would take the hardest benchmarks from the evaluation suite — those where BroRL at N=512 showed flat trajectories — and train with N=2048 or N=4096. If performance remains flat, the limitation is fundamental (the base model genuinely lacks the capability, so no amount of exploration helps). If performance begins to improve, the conclusion is that harder problems require larger N, and the optimal N scales with problem difficulty. This experiment would also test Lemma 2's prediction under extreme conditions: for problems where correct token probabilities are very small (p_i << 0.01), the unsampled decay factor (1-p_i)^N is close to 1 even at N=512 — meaning the unsampled coupling term may still be non-negligible. The exponential decay guarantee only bites when N >> 1/p_i, so for tokens with p_i ≈ 0.001, even N=512 is insufficient (512 × 0.001 ≈ 0.5). Harder problems likely have smaller p_i for correct tokens, requiring proportionally larger N. Characterizing this relationship would provide a principled formula for choosing N based on estimated problem difficulty.

Practical Applications and Downstream Use Cases

Reviving stalled RLVR training runs in production reasoning model pipelines. The most direct application of BroRL is as a rescue recipe for RLVR training runs that have plateaued. Organizations training reasoning models (e.g., math-solving, code generation, scientific reasoning) using ProRL or similar step-count-scaling approaches typically observe saturation after a few thousand steps. The paper's specific numbers provide a concrete protocol: when the model shows <0.1% improvement per 100 additional steps and evaluation variance exceeds the trend, switch the training configuration from N=16 to N=512 (adjusting the learning rate via square-root scaling), expand the context window if needed, and continue training. The paper demonstrates that this protocol extracts an additional +1.34 points on Math (61.69 → 63.03), +2.20 on Code (52.00 → 54.20), and +1.80 on Reasoning Gym (61.29 → 63.09) from a 1.5B model that had already saturated — gains that would have been left on the table under step-count scaling alone. The hardware efficiency benefit (nearly 2× throughput improvement) means this rescue does not require additional GPU budget — it runs in less wall-clock time than the equivalent number of ProRL steps. For a team running on 64 H100 GPUs at ~$2/GPU-hour, the ~140 hours of additional training costs roughly$18,000, and the BroRL recipe extracts ~1.5–2× more improvement from that budget than continued ProRL training.

Cost-efficient RLVR training from scratch with large-N as the default. For teams starting new RLVR training runs, BroRL suggests that the default rollout size should be much larger than current practice. Most open-source RLVR implementations default to N ∈ {4, 8, 16} based on intuition about generation cost. BroRL's throughput data (Table 2) upends this intuition: N=512 is actually 2× faster per sample than N=16 on H100 GPUs, and the dynamic sampling pass rate improvement (62% vs. 41%) means a higher fraction of those faster-generated samples contribute to training. A practical recommendation: before committing to a specific N, benchmark generation throughput on your specific hardware at N ∈ {64, 128, 256, 512} and select the N that maximizes (throughput × dynamic sampling pass rate) — the effective useful training data rate. The paper's numbers for H100 GPUs suggest this optimum is at N ≥ 256. Starting RLVR training with large N from the beginning, rather than switching mid-training, could prevent the initial stagnation that step-count scaling eventually hits, potentially reaching higher final performance in fewer total GPU-hours. The only additional cost is the larger memory footprint for KV caches during generation, which may require adjusting the number of prompts per step (BroRL uses 128 prompts at N=512 vs. ProRL's 512 prompts at N=16) — but the paper demonstrates this is a favorable trade.

Self-improvement data generation with broader exploration. RLVR-trained models are increasingly used to generate training data for subsequent fine-tuning rounds (distillation, rejection sampling, or iterative self-improvement). In these pipelines, the quality and diversity of generated solutions determine the ceiling for the next round of training. BroRL's core insight — that broader exploration surfaces correct solutions that would otherwise go undiscovered — applies directly here: when generating training data from a reasoning model, use large N (hundreds or thousands of rollouts per problem) rather than the typical N ∈ {1, 4, 8}. The paper's data show that at N=512, the dynamic sampling pass rate is 62% (vs. 41% at N=16), meaning that for any given prompt, the model is much more likely to produce at least one correct solution when sampling broadly. This increases the yield of correct training examples without requiring a more capable model. For a self-improvement pipeline generating data on 100,000 math problems: at N=16, roughly 41,000 problems would yield at least one correct answer; at N=512, roughly 62,000 would — a 51% increase in training data yield for the same model. Combined with the throughput advantage (the 512 rollouts are generated faster per sample than 16 × 32 batches), the per-correct-example cost is substantially lower under BroRL-scale exploration. This application leverages BroRL's exploration benefit even when the training itself uses a different algorithm — the large-N generation is used only for data creation, not for RL updates.

When to Prefer This Method

The paper explicitly positions BroRL against ProRL (step-count scaling with small N) as a complementary axis, and the experimental design directly compares these two approaches at matched training time. The tradeoff is clearly articulated:

• Prefer BroRL (large N, fewer steps) when:

  • The RLVR training run has plateaued or begun to degrade after prolonged training with small N (the ProRL scenario — performance saturates around 3K steps and additional training yields marginal gains or regression).
  • GPU hardware supports efficient large-batch generation — specifically, when the generation workload is memory-bound at small batch sizes and can transition to compute-bound at larger batch sizes (measurable via a throughput sweep across N values).
  • The task distribution includes a mix of easy and medium-difficulty problems where broader exploration can surface correct solutions that narrow sampling misses — but not exclusively hard problems where the base model's pass@1 is near zero (BroRL cannot create capability that doesn't exist).
  • Training compute is measured in wall-clock time rather than gradient step count, since BroRL takes fewer steps but processes more samples per step at higher throughput.

• Prefer ProRL (small N, more steps) when:

  • The training run is still in its early stages (far from saturation) and showing steady per-step improvement — BroRL's exploration benefit is most impactful when step-count scaling has already extracted the easy gains.
  • GPU memory constraints prevent large batch sizes (N=512 requires storing KV caches for 512 sequences per prompt × 128 prompts = 65,536 active sequences, which may exceed HBM capacity on GPUs with less than 80GB memory).
  • Latency of individual training steps must be minimized (e.g., for rapid experimentation cycles where waiting for 512-rollout generation per step is impractical) — ProRL's N=16 per prompt generates results faster per step even if throughput per sample is lower.
  • The prompt distribution is extremely diverse and no single prompt benefits from hundreds of rollouts (though the paper's dynamic sampling pass rate data at N=512 suggests this rarely occurs in practice — broader exploration almost always increases the chance of mixed outcomes).

These conditions are drawn directly from the paper's comparisons in Tables 1-2, Figure 1, Figure 4, and the hardware efficiency discussion in Section 4.2.4. The paper does not position BroRL against non-ProRL algorithms (GRPO, RLOO, DAPO), so a comparative decision rule for those alternatives is not supported by the current evidence.