CoBA-RL: Capability-Oriented Budget Allocation for Reinforcement Learning in LLMs

🎯 Pitch

Uniform rollout budgets in GRPO waste massive compute by treating all problems equally, but CoBA-RL cuts total sampling by half while boosting accuracy.

1. Executive Summary

This paper proposes CoBA-RL, a reinforcement learning algorithm that dynamically allocates rollout budgets during LLM post-training based on the model's evolving capability rather than using a uniform allocation. The method is evaluated on Qwen2.5-7B-Base/Instruct and Qwen3-1.7B/4B-Base models trained on DAPO-Math-17K and tested across five mathematical benchmarks (AIME24, AIME25, AMC23, MATH500, Olympiad Bench). CoBA-RL introduces a Capability-Oriented Value Function — modeled as a Beta distribution whose shape parameters shift in response to the policy's global failure rate — to quantify each task's training value, and a Heap-Based Greedy Budget Allocation algorithm that iteratively assigns rollouts to samples with the highest marginal gain, solving Equation 3 as a constrained maximization problem. On Qwen2.5-7B-Instruct, CoBA-RL achieves an average accuracy of 46.78% across benchmarks, improving over GRPO's 42.24% by 4.54 percentage points and consistently outperforming the static-allocation baseline Knapsack-RL at 45.39%, while the ablation in Figure 6 shows CoBA-RL at half the total budget (B_total=2048) matches or exceeds GRPO at double the budget (45.52% vs. 42.78%), establishing that capability-aware budget allocation yields substantial data efficiency but that the exploitation-to-exploration transition is critical — the "Exploit → Explore" scheduling outperforms the reverse, with the largest single-benchmark gain appearing on AIME25 (+5.62 percentage points over GRPO).

2. Context and Motivation

The Core Problem: Uniform Rollout Budgets Waste Compute During RL Training

The fundamental problem this paper tackles is resource allocation during reinforcement learning-based LLM post-training. When training LLMs with RLVR (Reinforcement Learning with Verifiable Rewards) — currently the dominant paradigm for improving mathematical reasoning, coding, and agentic capabilities — the standard framework, Group Relative Policy Optimization (GRPO), assigns an identical number of rollout trajectories (typically $G$ generations) to every training prompt in a batch. The paper argues this uniform strategy is inherently wasteful: complex problems that require extensive exploration to find correct solutions get the same budget as simple problems where the model already succeeds reliably.

This inefficiency is stated explicitly in Section 1:

"In practice, vanilla GRPO overlooks the critical impact of sample difficulty on training value and the corresponding rollout budget."

The consequence is computational waste on a massive scale. Every training step generates hundreds or thousands of rollouts, many of which contribute negligible learning signal — either because the model already solves the task consistently (generating redundant correct answers) or because the problem is so far beyond the model's current capability that no amount of sampling within a single step will surface a correct solution. In either case, those rollouts consume GPU hours without advancing model performance.

Why This Problem Matters: The Economics of LLM Post-Training

This is not merely an academic inefficiency. Post-training via RLVR has become a standard, expensive step in the LLM development pipeline. Models like DeepSeek-R1, Kimi K2, Qwen3, and Gemini 2.5 all employ some form of RL-based reasoning enhancement, and the computational cost scales with the number of rollouts per step. An allocation strategy that can match or exceed standard performance while using substantially fewer rollouts — or that achieves better final performance for the same total compute — translates directly to reduced training costs, faster iteration cycles, and lower carbon footprints.

More subtly, the paper identifies an exploration-exploitation trade-off that uniform allocation handles poorly (Section 1). During early training, the model is weak and needs to rapidly consolidate basic reasoning patterns — this is exploitation of simple problems where correct answers provide clean training signal. Later, as the model becomes proficient on easier tasks, the marginal benefit of additional easy-problem rollouts diminishes, and the model should shift toward exploring harder problems that expand the frontier of its reasoning capabilities. Uniform allocation cannot express this dynamic preference; it treats every problem identically regardless of the model's learning stage.

Conflicting Demands That Require Adaptive Resolution

The paper surfaces a tension that prior approaches failed to resolve. On one hand, easy problems are not worthless — especially early in training, they provide the reliable positive feedback that stabilizes policy updates and prevents the model from drifting into degenerate output patterns. On the other hand, hard problems are not universally valuable — pushing the model to explore problems it cannot possibly solve yields no positive reward signal and can even be detrimental if the model learns to produce low-quality outputs that happen to occasionally match the answer by chance. The value of any given problem is a function of the model's current capability, not an intrinsic property of the problem itself:

"The true training value of a sample is inextricably linked to the policy model's real-time capabilities."

This insight separates CoBA-RL from earlier work and defines the core requirement: an allocation strategy must be capability-conditioned — it must track how proficient the model has become and shift resource distribution accordingly, in real time, throughout training.

Where Prior Approaches Fall Short

The paper identifies two broad categories of prior work that attempt to address resource allocation, each with critical limitations.

Category 1: Static Difficulty-Based Allocation (e.g., Knapsack-RL)

The most directly comparable prior method is Knapsack-RL (Li et al., 2025b), which formulates budget allocation as a knapsack optimization problem: assign more rollouts to harder problems and fewer to easier ones, maximizing a pre-defined value function. The allocation is determined by historical pass rates: problems with low pass rates are considered high-value and receive more budget.

The critical failure mode is that the value function is static — it assumes that harder problems are always more valuable than easier ones, regardless of the model's training stage. Section 1 explains:

"These approaches typically rely on static value functions. They operate on the fixed assumption that harder samples inherently offer superior training value than simpler ones and that this relationship remains constant throughout the entire training process."

This assumption breaks down in two important regimes:

Early training: The model is too weak to solve hard problems at all (pass rate ≈ 0). Assigning extra rollouts to these problems generates only incorrect trajectories — no positive learning signal, wasted compute. Meanwhile, easy problems that could provide useful training signal are starved of budget.
Late training: The model has mastered easy problems (pass rate ≈ 1). Additional rollouts on these problems generate redundant correct answers with vanishing marginal value. A static function continues to allocate budget to them because the difficulty-priority assumption never updates.

Knapsack-RL's value function is pre-defined based on problem difficulty alone, with no mechanism to incorporate the model's evolving capability. The Budget Saturation Factor in CoBA-RL is designed specifically to address this: the saturation rate depends on $p_i(1-p_i)$ , which is near zero for both very easy ( $p_i \approx 1$ ) and very hard ( $p_i \approx 0$ ) problems, naturally deprioritizing both extremes and focusing budget on intermediate-difficulty problems where the learning signal is richest.

Category 2: Curriculum Learning Approaches

A larger body of work employs curriculum learning strategies for LLM post-training — organizing training data into difficulty stages and progressing from easy to hard over time. The paper cites several examples: ADCL (Zhang et al., 2025a) periodically re-evaluates data batches to adjust difficulty thresholds; SEC (Chen et al., 2025b) uses policy gradient advantages to dynamically adjust data distribution; and DUMP (Wang et al., 2025b) performs distribution-level curriculum learning.

The paper distinguishes CoBA-RL from curriculum methods along a key axis: curriculum learning decides which samples to train on, while CoBA-RL decides how much budget to allocate per sample within a heterogeneous batch (Section 4.2). Curriculum methods typically present the model with a curated subset of data at each stage — easy problems first, harder problems later. CoBA-RL instead trains on the full dataset at every step but varies the rollout count per sample. This is a fundamentally different design choice with practical implications:

Curriculum methods require data selection logic and can suffer from forgetting if earlier-stage data is completely excluded later.
CoBA-RL maintains exposure to the full data distribution throughout training, reducing forgetting risk, while channeling compute asymmetrically.

Moreover, curriculum methods typically use pre-defined difficulty schedules (easy-to-hard) that do not respond to the model's actual learning rate. If the model masters easy problems faster than expected, the curriculum schedule may waste steps on already-mastered material. Conversely, if the model struggles, the curriculum may advance too aggressively. CoBA-RL's reliance on real-time capability metrics (the global failure rate) makes the allocation responsive to actual training dynamics rather than a fixed schedule.

Category 3: Standard GRPO and Its Variants

The base GRPO algorithm (Shao et al., 2024) assigns a uniform $G$ rollouts to every prompt in a batch, computing group-relative advantages to eliminate the need for a separate value network. Variants like GSPO (Zheng et al., 2025a), DAPO (Yu et al., 2025), and GDPO (Liu et al., 2026) introduce improvements to the policy update rule (sequence-level importance weighting, clip-higher strategies, reward decoupling) but retain uniform per-sample rollout allocation. The paper's critique is not that GRPO's update rule is flawed, but that the fixed allocation of the computational budget that feeds that update rule is suboptimal.

The waste is quantifiable. Figure 6 shows that CoBA-RL with a total budget $B_{\text{total}} = 2048$ achieves 45.52% accuracy, surpassing GRPO's 42.78% accuracy at double the budget ( $B_{\text{total}} = 4096$ ). This means GRPO is spending twice the compute to achieve worse results, because roughly half its rollouts are directed toward samples with negligible training value.

Why Existing Methods Fail to Adapt to Capability Changes

The unifying limitation across all prior approaches is the absence of a mechanism that links individual sample value to the model's global learning state. Static difficulty-based methods have no capability signal at all. Curriculum methods have a pre-defined capability schedule (implicitly assuming a fixed learning rate). GRPO variants have no per-sample allocation logic.

CoBA-RL introduces two mechanisms to fill this gap:

A global capability metric (the moving average of the global failure rate $\bar{\mathcal{F}}_t$ ) that provides a scalar summary of how proficient the model is at the current training step.
A value function parameterized by this metric (the Beta distribution shape parameters $\alpha_t$ , $\beta_t$ ) that shifts the training value assessment in response to capability changes — skewing toward high-pass-rate samples when the model is weak and toward intermediate-difficulty samples as it strengthens.

This moves the allocation problem from "which problems are inherently hard?" (a static property) to "which problems offer the highest marginal training gain given the model's current capability?" (a dynamic, state-dependent property).

How This Paper Positions Itself

The paper positions CoBA-RL at the intersection of two research threads that have previously been treated separately: reinforcement learning for LLM reasoning and budget allocation / resource optimization. Section 4.2 explicitly draws on the operations research literature on knapsack problems and multi-armed bandits for budget allocation, adapting these concepts to the unique requirements of LLM training — where the "value" of allocating budget to a sample changes continuously as the policy improves.

The paper's key positioning claim is that capability-awareness is the missing ingredient that distinguishes CoBA-RL from prior budget allocation methods. The Capability-Oriented Value Function is not just another allocation heuristic — it redefines the objective function of the allocation problem to be conditional on the policy's current state. This is a conceptual shift from "allocate budget to maximize expected reward" to "allocate budget to maximize expected learning progress, defined relative to current capability."

The paper also positions itself as practical and easy to integrate (Section 5, listing integration in the takeaways). The Heap-Based Greedy algorithm runs in $O(B_{\text{total}} \log M)$ time — 0.124 seconds for a batch of 512 samples with 8192 total budget, versus 115.05 seconds for dynamic programming (Table 4) — making the allocation overhead negligible compared to the cost of generating rollouts. The integration requires only a budget allocator module inserted before the generation step in the training loop, as shown in the pseudocode in Appendix B (Listing 1).

Finally, the paper explicitly contrasts its exploit-then-explore scheduling against the more intuitive explore-then-exploit alternative (Section 3.3, Table 2). This is a non-obvious finding: on Qwen2.5-7B-Instruct, the "Exploit → Explore" strategy (where $\alpha_t$ decreases over time, shifting budget from easy to hard problems) achieves 46.78% average accuracy versus 42.83% for the reverse. The paper argues this counterintuitive result occurs because early exploitation of easy problems rapidly stabilizes the policy, after which exploration can proceed from a stronger foundation — whereas early exploration on problems the model cannot solve generates noise that destabilizes early training. This finding contributes to the broader understanding of exploration-exploitation scheduling in LLM RL training, beyond just the allocation mechanism itself.

3. Technical Approach

3.1 Reader Orientation

CoBA-RL is a reinforcement learning algorithm that replaces GRPO's uniform per-sample rollout budget with a capability-aware, dynamically recomputed allocation — the system acts as a "smart dispatcher" that, before every training step, distributes a fixed total number of rollouts across training problems based on which problems offer the highest marginal learning value given how proficient the model has become. The problem it solves is that uniform rollout allocation wastes compute on problems that are either already mastered (producing redundant correct answers) or impossible for the current model (producing only incorrect trajectories with no learning signal), and the shape of the solution is a two-part architecture: a value function that redefines training value as a function of the model's evolving global capability, plus a greedy optimizer that iteratively assigns rollouts to the samples with the highest instantaneous marginal gain.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four integrated components that operate within the standard GRPO training loop:

Global Capability Tracker — computes a scalar summary of the policy model's current proficiency by averaging pass rates across all samples in the training batch and smoothing this signal over recent steps. This is the "sensor" that tells the rest of the system how strong the model is right now.
Capability-Oriented Value Function — takes the global capability signal from the tracker and a per-sample pass rate, and returns a scalar value representing that sample's expected training gain under the current policy. Internally, this is a Beta distribution whose shape parameters shift in response to the global capability signal, embedded within a diminishing-returns saturation factor. This is the "brain" that redefines what "valuable" means at each training stage.
Heap-Based Greedy Allocator — takes the value function, a batch of samples with their current pass rates, and a total rollout budget, and produces a per-sample budget allocation by iteratively assigning single rollouts to whichever sample offers the highest marginal value gain, updating that sample's marginal gain, and repeating until the budget is exhausted. This is the "optimizer" that solves the constrained allocation problem.
GRPO Training Loop (modified) — a standard Group Relative Policy Optimization loop where, before generating rollouts, the allocator determines how many rollouts each sample receives, the batch is resampled according to this allocation, and the policy update proceeds as usual on the generated trajectories. This is the "host" into which CoBA-RL is inserted with minimal disruption.

Information flows as follows: at each training step, the policy model generates rollouts for a batch of problems → pass rates are computed from the rollout outcomes → the global capability tracker updates its moving average of the failure rate → the value function's shape parameters are recomputed from the updated capability signal → the heap-based allocator solves for the optimal per-sample budget given the new value function → the batch is resampled according to the allocation → rollouts are generated → GRPO updates the policy → repeat. The value function and allocation are recomputed every step, making the system responsive to the model's continuous improvement.

3.3 Roadmap for the Deep Dive

First, the formal problem statement (Equation 3 and constraints): what exactly is being optimized, what are the decision variables, and what makes this a constrained maximization problem. This establishes the mathematical framework within which all subsequent components operate.
Second, the global capability metric (Equations 4, 6): how the system measures model proficiency in real time, and why a non-linear transformation of the smoothed failure rate is needed to maintain sensitivity during low-failure stages.
Third, the Capability-Oriented Value Function (Equations 5, 7, 8, 9): the core intellectual contribution — how the Beta distribution parameters are determined by the capability metric, how the budget saturation factor encodes diminishing returns, and how combining them yields a value function that shifts its preference density as the model improves.
Fourth, the Heap-Based Greedy Allocator (Algorithm 1 and Proposition 2.2): how the diminishing-marginal-utility property of the value function enables a greedy optimization to be optimal, and how the heap data structure achieves $O(B_{\text{total}} \log M)$ allocation time.
Fifth, integration with the GRPO training loop (Appendix B, Listing 1): the minimal code changes required to insert CoBA-RL into an existing GRPO pipeline, making clear that the method is a drop-in modification rather than a new training framework.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a method paper whose core idea is that rollout budget allocation during RL-based LLM post-training should be conditioned on the model's evolving capability, operationalized through a dynamically-shaped Beta distribution value function and solved with a greedy heap-based optimizer.

Formal Problem Statement: Constrained Budget Maximization

The paper frames budget allocation as a discrete constrained optimization problem defined in Equation 3. At each training step $t$ , the system receives a batch of $M$ tasks $\mathcal{X}_t = \{x_1, \dots, x_M\}$ and has a total rollout budget $B_{\text{total}}$ . The decision variables are the per-sample budgets $B_1, \dots, B_M$ — how many rollout trajectories to generate for each task. The objective is:

$\max_{B_1, \dots, B_M} \sum_{i=1}^{M} V(B_i, \pi_\theta, p_i)$

subject to:

$\sum_{i=1}^{M} B_i = B_{\text{total}}, \quad B_{\text{low}} \leq B_i \leq B_{\text{up}}, \quad B_i \in \mathbb{Z}^+$

where $V(B_i, \pi_\theta, p_i)$ is the value function mapping a task, its allocated budget, and the current policy to a scalar representing expected learning gain, $B_{\text{low}}$ and $B_{\text{up}}$ are per-sample minimum and maximum budget constraints, and $\mathbb{Z}^+$ enforces integer budgets (you cannot generate half a rollout).

What this computes: given a batch of $M$ problems with known pass rates $p_i$ under the current policy, and a total generation budget $B_{\text{total}}$ , the optimization finds the integer allocation $B_1, \dots, B_M$ that maximizes the sum of per-sample training values, respecting that every sample must receive at least $B_{\text{low}}$ rollouts (to ensure minimum statistical reliability of the advantage estimate) and at most $B_{\text{up}}$ (to prevent any single sample from monopolizing the budget). The output is a per-sample rollout count that feeds into the downstream generation step.

Why this form: formulating budget allocation as a constrained maximization directly models the trade-off: assigning an extra rollout to sample $i$ means one fewer rollout for some other sample $j$ . The constraints $B_{\text{low}}$ and $B_{\text{up}}$ are practical guardrails — without a minimum, samples with near-zero estimated value might receive zero rollouts, eliminating them from the batch entirely and potentially causing distributional drift; without a maximum, a single high-value sample could attract an unbounded fraction of the budget, reducing batch diversity. The integer constraint reflects physical reality: rollouts are discrete generation operations that cannot be fractional.

Global Capability: Measuring Model Proficiency in Real Time

The value function $V$ must be conditioned on the model's current capability, but capability is not directly observable — it must be estimated from training statistics. The paper defines Global Capability through two complementary metrics in Definition 2.1.

The Global Success Rate $\mathcal{S}_t$ at step $t$ is:

$\mathcal{S}_t = \frac{1}{M} \sum_{i=1}^{M} p_i(x_i; \theta_t)$

where $p_i(x_i; \theta_t)$ is the pass rate for task $x_i$ under policy $\pi_{\theta_t}$ , defined in Equation 1 as the expected probability of correctness:

$p_i(x_i; \theta_t) = \mathbb{E}_{o \sim \pi_{\theta_t}(\cdot \mid x_i)} \left[ \mathbb{I}(R(x_i, o) = 1) \right]$

where $R(x_i, o) \in \{0, 1\}$ is the binary outcome reward (1 if the generated answer is mathematically correct, 0 otherwise), and $\mathbb{I}(\cdot)$ is the indicator function. The expectation is over the policy's generation distribution — in practice, estimated by the empirical pass rate among the $G$ rollouts generated for that sample at the current step.

The Global Failure Rate $\mathcal{F}_t$ is simply $1 - \mathcal{S}_t$ .

What this computes: $\mathcal{S}_t$ is the batch-averaged probability that the current policy generates a correct answer for a randomly selected training problem. If the policy is weak, $\mathcal{S}_t$ is low; as training progresses and the model improves, $\mathcal{S}_t$ increases. $\mathcal{F}_t$ captures the complement — the fraction of problems the model still fails on, which decreases as the model strengthens.

Why this form: averaging over the current batch provides an instantaneous capability snapshot that is automatically updated every training step without requiring a separate evaluation phase. However, raw per-step estimates are noisy — a single step's batch might be unrepresentatively easy or hard. The paper smooths this by computing the moving average $\bar{\mathcal{F}}_t$ over the past $k$ steps (denoted as the moving average of the global failure rate — the exact value of $k$ is not specified in the main text but is referenced in Equation 6's surrounding prose). This converts a noisy instantaneous measurement into a stable capability trend.

Non-linear transformation for sensitivity. A critical design detail appears in Equation 6: when the failure rate is low ( $\bar{\mathcal{F}}_t \leq 0.5$ ), a sigmoid transformation is applied to enhance sensitivity:

$\tilde{\mathcal{F}}_t = \Psi(\bar{\mathcal{F}}_t) = \begin{cases} \bar{\mathcal{F}}_t, & \text{if } \bar{\mathcal{F}}_t > 0.5 \\ \sigma\left(\gamma \cdot (\bar{\mathcal{F}}_t - 0.5)\right), & \text{if } \bar{\mathcal{F}}_t \leq 0.5 \end{cases}$

where $\sigma(\cdot)$ is the sigmoid function and $\gamma = 10$ is a scaling factor.

What this computes: when the model's failure rate is high (early training, $\bar{\mathcal{F}}_t > 0.5$ ), $\tilde{\mathcal{F}}_t$ is simply the raw smoothed failure rate — the model is failing often enough that no amplification is needed. When the failure rate drops below 0.5 (mid-to-late training), the sigmoid transformation $\sigma(\gamma \cdot (\bar{\mathcal{F}}_t - 0.5))$ stretches small differences in the failure rate into more discriminable differences in $\tilde{\mathcal{F}}_t$ . Since $\gamma = 10$ , the sigmoid is steep around the 0.5 threshold, making $\tilde{\mathcal{F}}_t$ sensitive to subtle capability improvements that a linear mapping would compress.

Why this form: as the model improves, the failure rate can plateau near low values — dropping from 0.3 to 0.25 represents a substantial relative improvement (a 17% reduction in failures) but only a 0.05 absolute change that might be lost in noise under a linear mapping. The sigmoid transformation with a high scaling factor amplifies these small absolute differences when they matter most (the low-failure regime where further improvements are hard-won), ensuring the value function's shape parameters continue to shift meaningfully even in late training. Without this transformation, the value function would become effectively static once the model reached moderate proficiency, because the raw failure rate would change too slowly to drive parameter updates.

The Capability-Oriented Value Function: Core Mechanism

The value function $V(B_i, \pi_\theta, p_i)$ is the heart of CoBA-RL. It assigns a scalar training value to each task-sample pair, and its defining property is that this value depends on both the sample's difficulty (captured by $p_i$ , the pass rate) and the model's current global capability (captured by $\alpha_t, \beta_t$ , which are functions of $\tilde{\mathcal{F}}_t$ ). The value function is the product of two sub-components that the paper develops separately before combining.

Sub-Component 1: Capability-Induced Preference Density

The paper models the model's preference — which difficulty levels are most valuable for training right now — as a Beta distribution over pass rates. Equation 5 defines this density:

$\text{Density}(p_i; \alpha_t, \beta_t) = \frac{p_i^{\alpha_t - 1} (1 - p_i)^{\beta_t - 1}}{\mathrm{B}(\alpha_t, \beta_t)}$

where $\alpha_t > 0$ and $\beta_t > 0$ are the shape parameters at step $t$ , and $\mathrm{B}(\alpha_t, \beta_t)$ is the Beta function (the normalizing constant that ensures the density integrates to 1 over $p_i \in [0, 1]$ ).

What this computes: for any given pass rate $p_i$ , the Beta density returns a non-negative scalar indicating how strongly the current training stage prefers samples with that pass rate. The shape of the density — where it peaks, how concentrated it is — is controlled by $\alpha_t$ and $\beta_t$ .

Why the Beta distribution: the Beta distribution is defined on $[0, 1]$ , matching the domain of pass rates $p_i$ . Its two-parameter form provides flexible control over the density shape: high $\alpha_t$ relative to $\beta_t$ shifts mass toward $p_i \approx 1$ (preferring easy samples), high $\beta_t$ relative to $\alpha_t$ shifts mass toward $p_i \approx 0$ (preferring hard samples), and balanced $\alpha_t \approx \beta_t$ centers mass at intermediate $p_i$ . This parametric flexibility allows a single functional form to express exploitation (high $\alpha_t$ ), exploration (high $\beta_t$ ), or balanced learning (moderate, equal parameters) — all by changing two numbers derived from the global capability signal. An alternative like a Gaussian would have unbounded support and require truncation; a categorical distribution over discretized difficulty bins would introduce arbitrary bin boundaries.

How the parameters are set from capability. The critical link between the global capability signal and the preference density is established in Equation 7. The moving-average transformed failure rate $\tilde{\mathcal{F}}_t$ determines $\alpha_t$ through a linear mapping:

$\alpha_t = \text{clip} \left( \alpha_{\min} + \lambda \cdot \tilde{\mathcal{F}}_t, \, \alpha_{\min}, \, \alpha_{\max} \right)$

with $\beta_t = \kappa - \alpha_t$ , where $\kappa = \alpha_t + \beta_t$ is held constant.

What this computes: $\alpha_t$ is a clipped linear function of the transformed failure rate. When $\tilde{\mathcal{F}}_t$ is high (early training, model fails often), $\alpha_t$ is pushed toward $\alpha_{\max}$ , making the Beta density peak at high $p_i$ (easy samples) — the model exploits. As $\tilde{\mathcal{F}}_t$ decreases (model improves), $\alpha_t$ decreases linearly, shifting the density toward intermediate and then low $p_i$ (harder samples) — the model explores. The clipping to $[\alpha_{\min}, \alpha_{\max}]$ prevents degenerate shapes. The sum constraint $\kappa = \alpha_t + \beta_t$ means the total concentration is fixed: as $\alpha_t$ decreases, $\beta_t$ automatically increases by the same amount, shifting mass from high pass rates to low pass rates while keeping the overall peakedness constant.

Why this form: the linear mapping from $\tilde{\mathcal{F}}_t$ to $\alpha_t$ creates a continuous, monotonic relationship between model capability and preference: stronger model → lower failure rate → lower $\alpha_t$ → preference shifts toward harder samples. This is the "Exploit → Explore" trajectory that Section 3.3 validates. The clipping prevents $\alpha_t$ from becoming so small or large that the distribution degenerates to a spike at 0 or 1 (which would concentrate all budget on extremely hard or extremely easy samples, losing batch diversity). The constant $\kappa$ controls the variance — higher $\kappa$ means a more peaked distribution (stronger preference for a narrow difficulty band), lower $\kappa$ means a flatter distribution (weaker preference, more uniform allocation). The sensitivity analysis in Appendix D.2 tests $\kappa \in \{7, 11, 15, 21\}$ and finds $\kappa = 11$ optimal with 46.61% accuracy, but performance variation is modest (45.40% to 46.61%), demonstrating robustness to this choice.

Key hyperparameters. The paper specifies the constituent values: $\alpha_{\min}$ , $\alpha_{\max}$ , and $\kappa$ are not given as explicit numbers in the main text, but the sensitivity analysis in Figure 8 (Appendix D.2) reports experiments with $\kappa \in \{7, 11, 15, 21\}$ and selects $\kappa = 11$ as default. The value of $\lambda$ (the slope of the linear mapping) is also not explicitly stated but is implicitly set such that the clipping bounds are reached at the extremes of $\tilde{\mathcal{F}}_t$ . This is a minor documentation gap — a reader attempting to reimplement CoBA-RL would need to infer or grid-search these values.

Sub-Component 2: Budget Saturation Factor

While the preference density identifies which samples are theoretically valuable, it does not account for how much budget they should receive. The Budget Saturation Factor, defined in Equation 8, models the diminishing returns of additional rollouts:

$\eta(B_i, p_i) = 1 - e^{-\frac{B_i}{\tau} p_i (1 - p_i)}$

where $\tau$ is a temperature coefficient controlling the saturation rate, $B_i$ is the allocated rollout count, and $p_i(1-p_i)$ is the variance of the Bernoulli outcome (maximized at $p_i = 0.5$ , zero at $p_i = 0$ and $p_i = 1$ ).

What this computes: $\eta(B_i, p_i) \in [0, 1)$ is a factor that starts at 0 (when $B_i = 0$ , no budget yields no value) and asymptotically approaches 1 as $B_i \to \infty$ , with the rate of approach governed by $\frac{p_i(1-p_i)}{\tau}$ . For $p_i = 0.5$ , the factor $p_i(1-p_i) = 0.25$ is maximal, so $\eta$ saturates fastest — intermediate-difficulty problems extract more value from additional rollouts than extreme-difficulty problems. For $p_i \approx 0$ or $p_i \approx 1$ , $p_i(1-p_i) \approx 0$ , so $\eta$ saturates very slowly — adding rollouts to already-mastered or currently-impossible problems yields negligible marginal gain.

Why this form: the exponential saturation $1 - e^{-x}$ is the standard model for diminishing returns — the first few units of budget provide most of the gain, with each subsequent unit contributing less. The crucial design choice is the saturation argument $\frac{B_i}{\tau} p_i(1-p_i)$ . By multiplying $B_i$ by $p_i(1-p_i)$ , the effective budget is scaled by the sample's uncertainty: an uncertain sample ( $p_i \approx 0.5$ ) gets more mileage from the same nominal budget than a certain sample ( $p_i \approx 0$ or $1$ ). This automatically deprioritizes both trivially easy and impossibly hard problems regardless of the preference density — even if the Beta density says "exploit easy problems," the saturation factor ensures that easy problems with $p_i \approx 1$ receive diminishing marginal returns that make further budget allocation unattractive. This property is essential for the greedy optimizer to work correctly: without it, the allocator might pour budget into easy samples whose preference density is high, unaware that those samples have already saturated.

The temperature $\tau$ controls how quickly saturation occurs. A small $\tau$ causes rapid saturation — a few rollouts extract most of the available value. A large $\tau$ causes slow saturation — many rollouts are needed to approach the asymptote. The paper does not specify the exact value of $\tau$ used in experiments, but its role is to balance the influence of the budget term relative to the preference density term.

The Full Value Function: Combining Preference and Saturation

Equation 9 synthesizes the two sub-components into the final value function:

$V(B_i, \pi_\theta, p_i) = \left(1 - e^{-\frac{B_i}{\tau} p_i (1 - p_i)}\right) \cdot \text{Density}(p_i; \alpha_t, \beta_t)$

What this computes: the training value of allocating $B_i$ rollouts to sample $i$ under the current policy is the product of (a) how much the current capability stage prefers samples with this pass rate (the preference density), and (b) how much of that preference can be realized given the budget allocated (the saturation factor). The preference density determines the "best-case" value if unlimited budget were available; the saturation factor scales this down based on how much budget is actually allocated and how quickly the sample saturates.

Why this multiplicative form is essential. The multiplicative structure creates an interaction between preference and saturation that neither sub-component could achieve alone:

If the preference density is high but $p_i(1-p_i)$ is low (e.g., an easy sample with $p_i \approx 0.95$ during exploitation), the saturation factor grows very slowly with $B_i$ , producing a small marginal gain $\Delta V(B_i, p_i)$ — the allocator will correctly avoid pouring excessive budget into this sample.
If the preference density is moderate but $p_i(1-p_i)$ is high (e.g., a medium-difficulty sample with $p_i \approx 0.5$ ), the saturation factor grows rapidly with $B_i$ , producing a large marginal gain — the allocator will correctly direct budget here.
If the preference density is low (e.g., a hard sample during early exploitation), the product is small regardless of the saturation factor — the allocator will correctly ignore this sample.

An additive combination $V = \eta + \text{Density}$ would not create this interaction — a sample with high preference density would always receive high value regardless of saturation, leading to over-allocation to saturated easy samples. The multiplicative form forces the value to be small whenever either the preference is low or the sample is already saturated, which is the correct behavior for an allocation that should shift over time.

The shape-shifting property. As the model trains and $\tilde{\mathcal{F}}_t$ decreases, $\alpha_t$ decreases and $\beta_t$ increases. The Beta density's peak shifts from high $p_i$ toward lower $p_i$ . This means the multiplicative weighting in $V$ also shifts — samples that previously had high value (because the density peaked at their pass rate) now have lower value, and vice versa. The value function's topology is therefore not static: the same sample with the same pass rate can have different training values at different training steps, purely because the model's global capability has changed. This is the mechanism that operationalizes the claim in Section 1 that "the set of samples holding the highest training value constantly shifts."

The Heap-Based Greedy Budget Allocator

With the value function defined, the paper needs an efficient algorithm to solve Equation 3 — the constrained maximization of $\sum_i V(B_i, \pi_\theta, p_i)$ subject to the total budget and per-sample bounds. The key mathematical property that makes this tractable is stated in Proposition 2.2: the marginal gain of the value function is strictly monotonically decreasing with respect to the allocated budget.

Formally, defining the marginal gain as:

$\Delta V(B_i, p_i) = V(B_i + 1, p_i) - V(B_i, p_i)$

Proposition 2.2 asserts that for all $B_i \geq 0$ :

$\Delta V(B_i, p_i) > \Delta V(B_i + 1, p_i)$

The proof is provided in Appendix A and follows from the functional form. Since the value function has the structure $V(B_i) = C \cdot (1 - e^{-k B_i})$ where $C = \text{Density}(p_i)$ and $k = \frac{p_i(1-p_i)}{\tau}$ (both constants with respect to $B_i$ ), the marginal gain simplifies to:

$\Delta V(B_i, p_i) = C e^{-k B_i} (1 - e^{-k})$

Since $C > 0$ , $k > 0$ , and $e^{-k} < 1$ , the factor $A = C(1 - e^{-k}) > 0$ , and $\Delta V(B_i, p_i) = A \cdot e^{-k B_i}$ — a strictly decreasing geometric sequence in $B_i$ .

What this property means: each additional rollout assigned to a sample produces less value than the previous rollout. The first rollout provides the largest gain, the second provides less, the third even less, and so on. This is the mathematical formalization of diminishing returns.

Why this enables greedy optimization: when the objective function has diminishing marginal returns (also called submodularity in discrete optimization), the greedy algorithm — iteratively allocating one unit of budget to the sample with the highest current marginal gain — is provably optimal for the constrained integer allocation problem. Without this property, greedy allocation might get stuck in a local optimum by over-allocating early to samples whose marginal gain subsequently drops below what other samples could provide. Diminishing returns guarantee that the best next dollar always goes to the sample with the highest instantaneous marginal gain, and that no reallocation of past decisions would improve the total.

The Heap-Based algorithm (Algorithm 1). The implementation uses a max-heap data structure to efficiently find the sample with the highest marginal gain at each allocation step:

Initialization (lines 3-4): Every sample receives the minimum budget $B_{\text{low}}$ . The remaining budget is $R = B_{\text{total}} - \sum_i B_{\text{low}}$ .
Heap construction (lines 5-11): For each sample $i$ whose current budget $B_i$ is below the upper bound $B_{\text{up}}$ , compute the marginal gain $\Delta V_i = V(B_i + 1, p_i) - V(B_i, p_i)$ and push the pair $(\Delta V_i, i)$ onto a max-heap ordered by $\Delta V_i$ . A max-heap is a tree data structure where the root always contains the element with the maximum key — here, the sample with the highest marginal gain — and extracting the maximum takes $O(\log M)$ time.
Iterative allocation (lines 12-20): While there is remaining budget $R > 0$ and the heap is not empty:
- Pop the sample $i^*$ with the maximum marginal gain from the heap top.
- Increment $B_{i^*}$ by 1 and decrement $R$ by 1.
- If $B_{i^*}$ is still below $B_{\text{up}}$ , compute its new marginal gain $\Delta V_{\text{new}} = V(B_{i^*} + 1, p_{i^*}) - V(B_{i^*}, p_{i^*})$ and push $(\Delta V_{\text{new}}, i^*)$ back onto the heap.
Return the final allocation vector $\mathbf{B}$ .

What this computes: the algorithm starts with every sample at the minimum budget and then distributes the remaining $R$ rollouts one at a time, always to whichever sample currently offers the highest marginal gain. Because the marginal gain decreases with each allocation (diminishing returns), the sample that is most valuable now might not be most valuable after receiving one more rollout — its updated marginal gain is recomputed and it re-enters the heap at its new (lower) priority. This naturally balances the allocation: no single sample will monopolize the budget because its marginal gain drops with each allocation, eventually falling below the marginal gain of unexplored samples.

Complexity analysis. Each of the $R$ allocation steps requires one pop ( $O(\log M)$ ) and one push ( $O(\log M)$ ) operation on the heap. Initial heap construction takes $O(M)$ time. The total complexity is therefore $O(B_{\text{total}} \log M)$ , where $B_{\text{total}}$ is the remaining budget after minimum allocation. This scales logarithmically with batch size — doubling the batch size increases allocation time by only a small constant factor — and linearly with the total budget.

Empirical efficiency (Table 4). The paper benchmarks the heap-based allocator against a Dynamic Programming baseline. For a batch of $M = 512$ samples with $B_{\text{total}} = 8192$ , dynamic programming (which has pseudo-polynomial complexity $O(M \cdot B_{\text{total}} \cdot (B_{\text{up}} - B_{\text{low}}))$ ) requires 115.05 seconds, while the heap-based greedy allocator completes in 0.124 seconds — approximately 927× faster. This makes the allocation overhead negligible: 0.124 seconds is dwarfed by the time required to generate thousands of rollout trajectories (each up to 4096 tokens long) from a 7B-parameter LLM. The allocation step can be inserted into the training loop without meaningfully affecting throughput.

The Clip-higher strategy. Appendix C notes that, following DAPO's recommendations, both CoBA-RL and Knapsack-RL use a "Clip-higher" strategy for the GRPO advantage computation. This means that when the probability ratio $\rho_{i,k}$ between the current and old policies is above 1 (the current policy is more likely to generate this output), the ratio is clipped to $1 + \epsilon$ in the PPO-style objective from Equation 2, while ratios below 1 are not clipped. This is a standard technique from DAPO to prevent the policy from moving too aggressively toward high-advantage outputs. It is not specific to CoBA-RL but is part of the base training configuration applied uniformly across all compared methods.

Integration with the GRPO Training Loop

CoBA-RL is designed to be a minimal-modification addition to existing GRPO training pipelines. Appendix B (Listing 1) provides pseudocode showing the integration point. The standard GRPO loop iterates over batches from the dataloader, generates $G$ rollouts per sample uniformly, and updates the policy. The modified loop inserts a budget allocation step after batch loading but before generation:

Load batch: batch_dict = next(dataloader), process into tensor format.
Compute allocation: given the total budget $B_{\text{total}} = \text{batch\_size} \times \text{default\_group\_size}$ and the sample indices, call budget_allocator.allocate(indices, total_budget) which returns a dictionary mapping each sample index to its allocated rollout count.
Resample batch: create repeat_indices by extending each index by its allocated count. For example, if sample 7 gets 3 rollouts, index 7 appears 3 times in repeat_indices. Then index into the batch: batch = batch[repeat_indices]. This expands the batch so that the subsequent generation step produces the correct number of rollouts per sample — no changes to the generation code are needed.
Generate and update: outputs = actor.generate_sequences(batch), then proceed with standard GRPO update using the generated rollouts.

What this achieves: the modification is entirely preprocessing — it changes how many times each sample appears in the generation batch but does not alter the generation, reward computation, advantage estimation, or policy update logic. This makes CoBA-RL compatible with any GRPO variant (DAPO, GSPO, GDPO) and any generation engine (the paper uses SGLang within the Verl framework). The interface is a single BudgetAllocator class that can be imported and called once per training step.

Key configuration parameters. The paper specifies:

Per-sample budget bounds: $[B_{\text{low}}, B_{\text{up}}] = [2, 128]$ — every sample gets at least 2 rollouts (for minimal statistical reliability of the advantage estimate) and at most 128 (to prevent resource concentration).
Default group size $G = 16$ — this is the GRPO baseline's uniform rollout count per sample, also used as the reference for computing $B_{\text{total}}$ .
Global batch size $M = 512$ for models smaller than 7B, $M = 256$ for 7B models.
Total training steps: approximately 500 for smaller models, nearly 1000 for 7B models.
Maximum response length: 4096 tokens.
KL penalty: $\beta_{\text{KL}} = 0$ (no KL divergence regularization, consistent with recent reasoning alignment practices).
Optimizer: AdamW with learning rate $1 \times 10^{-6}$ .
The transformation scaling factor: $\gamma = 10$ (Equation 6).
Default Beta sum parameter: $\kappa = 11$ (selected from the sweep in Appendix D.2).

What is NOT changed. The GRPO objective (Equation 2), the advantage computation, the reward function, the policy architecture, and the optimizer are all unchanged from standard GRPO. CoBA-RL modifies only the input distribution to the policy update — which samples receive how many rollouts — not the update rule itself. This design choice means any improvements in final accuracy can be attributed to the allocation strategy rather than to confounded changes in the learning algorithm.

4. Key Insights and Innovations

Innovation 1: Training Value Is Capability-Relative, Not Intrinsic to the Problem

The paper's most fundamental conceptual contribution is the claim that the training value of a sample is not a fixed property of the problem's difficulty, but a dynamic function of the model's current capability. Prior work — most directly Knapsack-RL (Li et al., 2025b) — operated on the assumption that harder problems are inherently more valuable for training and that this relationship holds constant. Under that static view, a problem with pass rate 0.1 is always worth more budget than a problem with pass rate 0.9, regardless of whether the model is at step 10 or step 500.

CoBA-RL challenges this assumption at a foundational level. The paper's central diagnostic is that there is no universal difficulty-value mapping — an easy problem that the model reliably solves provides genuine training signal early in training (stabilizing the policy's basic reasoning patterns) but becomes redundant later (generating correct answers the model already knows how to produce). Conversely, a hard problem that is completely unsolvable early in training yields only noise — no positive reward signal, no gradient toward better behavior — but becomes valuable later when the model has developed enough competence to sometimes succeed, creating the partial-success trajectories that drive exploration.

This is more than an empirical observation about diminishing returns; it is a reframing of what "training value" means in the context of RL-based LLM post-training. Rather than asking "is this problem hard?" (a static property of the problem), CoBA-RL asks "can the model learn something from additional rollouts on this problem right now?" (a dynamic property of the model-problem interaction). The Capability-Oriented Value Function operationalizes this reframing by making the value assessment conditional on the policy's state through the $\alpha_t$ , $\beta_t$ parameters derived from the global failure rate.

The evidence for this reframing is not a single ablation but the entire pattern of results: CoBA-RL outperforms Knapsack-RL — which uses a static value function — across all tested models and benchmarks (Table 1), with the largest gains appearing on the most challenging benchmarks (AIME25: +5.62 percentage points over GRPO, +3.12 over Knapsack-RL). More directly, Figure 4 visualizes how the budget distribution shifts as model capability evolves, conforming to the geometric shape of different value functions at different training stages. The task transition matrices in Appendix D.1 (Figure 7) provide additional evidence: CoBA-RL achieves higher conversion rates than Knapsack-RL across all difficulty categories — 71.2% vs. 50.0% for medium tasks, 36.7% vs. 20.4% for hard tasks — demonstrating that the capability-relative value function better identifies which problems genuinely have learning potential at the current training stage.

This is a fundamental conceptual shift rather than an incremental refinement. It changes the allocation problem from "allocate more budget to harder problems" (which Knapsack-RL already does) to "allocate more budget to problems at the frontier of the model's current competence" — a moving target that requires continuous recalibration. The paper provides the mechanism for that recalibration (the Beta distribution shape-shifting in response to $\tilde{\mathcal{F}}_t$ ), but the intellectual contribution is the diagnosis that recalibration is necessary in the first place and that the absence of a capability signal is the root cause of inefficiency in prior methods.

Innovation 2: Exploit-Then-Explore as the Optimal Scheduling for LLM RL Training

The paper presents an empirical finding that is both counterintuitive and practically consequential: for RL-based LLM post-training, prioritizing exploitation of easy problems before shifting to exploration of hard problems substantially outperforms the reverse schedule. Under the "Exploit → Explore" strategy (where $\alpha_t$ decreases over time, shifting the value function's preference from high-pass-rate to low-pass-rate samples), CoBA-RL achieves 46.78% average accuracy on Qwen2.5-7B-Instruct. Under the "Explore → Exploit" strategy (where $\alpha_t$ increases over time, starting with hard problems and shifting to easy ones), the same method achieves only 42.83% — a gap of nearly 4 percentage points (Section 3.3, Table 2).

This finding challenges an intuitive assumption: that exploration should come first, because discovering novel solution strategies on hard problems is the primary source of capability improvement, and exploitation of known patterns is merely consolidation. The paper's evidence argues the opposite — exploitation provides the foundation on which productive exploration can be built. Early in training, when the policy is weak and produces mostly incorrect outputs, allocating heavy budget to hard problems generates gradients from mostly negative outcomes. These gradients are noisy because the model has not yet learned the basic patterns that distinguish a promising attempt from a hopeless one. Easy problems, by contrast, provide clean positive reward signal that rapidly stabilizes the policy's output distribution, establishing the reasoning templates that make subsequent exploration of harder problems more than random search.

This is not merely a scheduling heuristic — it is an empirical characterization of the learning dynamics specific to LLM RL training. The paper's explanation, while brief, points to a mechanism: early-stage policy updates driven by hard-problem exploration are destabilizing because the model cannot distinguish between "incorrect answer due to poor reasoning" and "incorrect answer due to insufficient capability" — both produce zero reward, creating an uninformative gradient landscape. Easy problems provide discriminative signal (correct vs. incorrect trajectories are meaningfully different in a way the policy can learn from), enabling rapid initial improvement that then makes hard-problem exploration more effective later.

The evidence for this scheduling insight is direct: Table 2 shows the "Exploit → Explore" strategy outperforming the reverse on all five benchmarks, with the largest absolute gain on AIME25 (18.33% vs. 10.41%, a +7.92 percentage point difference) — the most challenging benchmark in the suite. Figure 5 visualizes the $\alpha_t$ trajectories for both strategies, making the scheduling difference concrete: one decreases (exploit → explore), the other increases (explore → exploit), and the former consistently produces the better model.

This finding is fundamental because it provides a principled answer to the exploration-exploitation scheduling question that goes beyond CoBA-RL's specific mechanism. Any budget allocation method — static or dynamic, knapsack-based or curriculum-based — must decide which difficulty regime to prioritize, and this result suggests that "hard first" is not just suboptimal but actively harmful for LLM RL training. It aligns with broader observations in the curriculum learning literature (easy-to-hard training) but provides specific evidence in the RLVR context where the interaction between exploration budget and policy stability is non-obvious.

Innovation 3: A Formal Demonstration That Diminishing Marginal Utility Enables Optimal Greedy Resource Allocation

While the use of greedy algorithms for resource allocation is not novel in computer science, the paper makes a specific, mathematically grounded argument that the structural property of diminishing marginal returns — proven for the Capability-Oriented Value Function in Proposition 2.2 — transforms a computationally intractable integer allocation problem into one solvable optimally in $O(B_{\text{total}} \log M)$ time. The proof in Appendix A establishes that $\Delta V(B_i, p_i)$ is a strictly decreasing geometric sequence in $B_i$ , a property that follows directly from the exponential saturation form of the value function.

What makes this an innovation rather than a routine application of known results is the deliberate design of the value function to satisfy this property. The paper could have chosen any functional form for the Budget Saturation Factor; the choice of $1 - e^{-k B_i}$ was not arbitrary — it guarantees the diminishing-marginal-utility property that makes greedy allocation optimal. The multiplicative combination with the preference density preserves this property (since the density is constant with respect to $B_i$ ), meaning the full value function inherits the submodularity that enables efficient optimization. This is architectural foresight: the value function was designed to be computationally tractable to optimize within an online training loop, not just expressive of the desired preference structure.

The practical significance is demonstrated in Table 4. Dynamic programming — the standard approach for knapsack-style allocation problems — requires 115.05 seconds for one batch allocation with $M = 512$ and $B_{\text{total}} = 8192$ . This latency is prohibitive for online RL training, where each step's total wall-clock time is dominated by generation and the allocator must run in milliseconds to avoid becoming the bottleneck. The heap-based greedy allocator completes in 0.124 seconds — nearly three orders of magnitude faster — without sacrificing optimality (since the diminishing returns property guarantees greedy optimality). This efficiency is not a happy accident but a direct consequence of the value function's mathematical design.

This contribution is fundamental yet bounded: it is a clever synthesis of known principles (submodular optimization, Beta distributions for preference modeling) applied to a specific problem, rather than a new theoretical result. However, within the context of LLM RL training — where allocation methods like Knapsack-RL have not emphasized computational efficiency of the allocator itself — the demonstration that optimal allocation can be achieved with negligible overhead (0.124 seconds per step) is a practical enabler that makes capability-aware allocation feasible for production-scale training pipelines. Without this property, CoBA-RL's value function would be academically interesting but computationally impractical; with it, the method is a drop-in module as shown in Appendix B.

Innovation 4: Global Failure Rate as a Sufficient Statistic for Capability-Conditioned Allocation

The paper introduces a minimal capability signal — the smoothed, transformed global failure rate $\tilde{\mathcal{F}}_t$ — and demonstrates that this single scalar is sufficient to drive effective budget reallocation throughout training. This is a design choice with important implications: rather than requiring per-sample capability estimates, multi-dimensional proficiency metrics, or separate evaluation benchmarks, CoBA-RL extracts its capability signal from the same batch statistics already computed during training (Equation 4: $S_t = \frac{1}{M} \sum_i p_i$ ).

The non-linear transformation in Equation 6 — applying a sigmoid with scaling factor $\gamma = 10$ when $\bar{\mathcal{F}}_t \leq 0.5$ — reveals a subtle design insight: the raw failure rate loses discriminative power precisely when it matters most. As the model improves, $\bar{\mathcal{F}}_t$ approaches zero asymptotically — a drop from 0.3 to 0.2 represents substantial relative improvement but only a 0.1 absolute change. The sigmoid transformation stretches these small absolute differences in the low-failure regime, ensuring the value function's shape parameters ( $\alpha_t$ , $\beta_t$ ) continue to shift meaningfully even in late training when raw failure rates appear nearly static. Without this transformation, the allocation strategy would effectively freeze once the model achieved moderate proficiency, losing the capability-awareness that distinguishes CoBA-RL from static methods.

The significance of this design is that it avoids a common failure mode of adaptive methods: the need for expensive out-of-band capability assessment. Some curriculum learning approaches (e.g., ADCL, cited in Section 4.2) periodically evaluate the model on separate data batches to assess proficiency and adjust difficulty thresholds. Others might require a held-out validation set to track progress. CoBA-RL extracts its capability signal from the training batch itself — no additional inference, no separate evaluation, no hyperparameter schedule. This makes the method self-contained and computationally cheap: the capability signal is a byproduct of the rollouts already being generated for the policy update.

The evidence that this scalar is sufficient comes from the consistent improvements over both GRPO (which uses no capability signal) and Knapsack-RL (which uses only per-sample pass rates without a global capability aggregation). Table 1 shows CoBA-RL outperforming both baselines across four model scales and five benchmarks, with the pattern holding across substantially different training trajectories (Qwen2.5-7B-Instruct starting from instruction-tuned weights vs. Qwen2.5-7B-Base starting from pretrained weights). The sensitivity analysis in Appendix D.2 further demonstrates that performance is robust to the $\kappa$ parameter that controls distribution concentration — the method works across a range of settings, suggesting the capability signal is the active ingredient rather than a specific hyperparameter configuration.

This is an incremental but practically important contribution. The idea of using training statistics to adapt learning is not new, but the specific choice of the global failure rate — and the engineering insight to apply a non-linear transformation for sensitivity preservation — is a clean, replicable design pattern that can be adopted by other budget allocation or curriculum learning methods. It demonstrates that effective capability-awareness does not require complex proficiency modeling; a simple aggregate statistic, properly transformed, is sufficient when embedded in the right allocation framework.

5. Experimental Analysis

Evaluation Methodology

Dataset. All training uses DAPO-Math-17K (Yu et al., 2025), a dataset of ~17,000 mathematical reasoning problems widely adopted for RLVR post-training. The paper does not describe this dataset's construction or source distribution in detail, treating it as a standard community benchmark.
Base model(s). Four model configurations are tested: Qwen2.5-7B-Instruct, Qwen2.5-7B-Base, Qwen3-4B-Base, and Qwen3-1.7B-Base. This spans instruction-tuned vs. base (pretrained-only) initializations and four model scales from 1.7B to 7B parameters, testing whether the allocation strategy generalizes across substantially different starting capabilities and model families (Qwen2.5 vs. Qwen3 architectures).
Metrics. The primary metric is accuracy (%) — the fraction of evaluation problems for which the model's generated answer matches the ground-truth answer. All reported results use avg@16 evaluation: for each test problem, 16 independent rollouts are generated from the final trained policy, and accuracy is averaged across these 16 outputs. This protocol is consistent across all baselines and benchmarks, ensuring fair comparison with methods that influence per-sample budget during training but are evaluated under identical generation conditions.
Baselines. Three categories of baselines are compared. GRPO (Shao et al., 2024) is the standard Group Relative Policy Optimization algorithm that assigns a uniform $G = 16$ rollouts to every training sample — the primary baseline. Knapsack-RL (Li et al., 2025b) is the most directly comparable prior method: it formulates budget allocation as a knapsack optimization using historical pass rates, with a static value function that assigns higher value to harder problems. Since the official implementation is not open-sourced, the authors re-implement Knapsack-RL within the Verl framework (Appendix C). Static and Heuristic strategies (Section 3.4, Table 3) include: a fixed exploitation value function with $(\alpha, \beta) = (10.5, 1.5)$ , a fixed exploration function with $(\alpha, \beta) = (1.5, 10.5)$ , and a Linear Step Decay heuristic where $\alpha_t$ decreases stepwise from 10 to 1 over training (i.e., $10 \to 9 \to \dots \to 1$ ).
Generation budget / compute accounting. The training budget is measured in total rollouts per step ( $B_{\text{total}}$ ). For the GRPO baseline, this is $M \times G$ where $M$ is the batch size and $G = 16$ is the fixed per-sample rollout count. For adaptive methods (CoBA-RL, Knapsack-RL), the same total budget is available, but the per-sample distribution is optimized. Per-sample constraints are $[B_{\text{low}}, B_{\text{up}}] = [2, 128]$ . Standard training configurations use $M = 512$ for models smaller than 7B and $M = 256$ for 7B models, with total steps of approximately 500 and 1000 respectively. The ablation in Figure 6 varies $B_{\text{total}} \in \{1024, 2048, 4096, 8192, 16384\}$ to test budget sensitivity. Evaluation consistently uses avg@16 across all methods.
Cross-validation / statistical protocol. The paper does not report cross-validation, confidence intervals, or statistical significance tests. Results in Tables 1-3 are presented as point estimates. Training curves in Figure 3 show validation accuracy over steps but without error bars or multiple-seed reporting. The task transition analysis in Appendix D.1 is reported as single-run matrices. The sensitivity analysis in Appendix D.2 sweeps four $\kappa$ values but reports point estimates for each. This absence of variance reporting makes it difficult to assess whether observed improvements (e.g., 4.54 percentage points over GRPO on Qwen2.5-7B-Instruct) are statistically reliable or within the noise of training stochasticity.

Main Quantitative Results

Overall Performance: CoBA-RL vs. GRPO and Knapsack-RL (Table 1)

Table 1 reports accuracy across all five benchmarks for all four model configurations. On Qwen2.5-7B-Instruct — the strongest base model tested — CoBA-RL achieves an average accuracy of 46.78% across the five benchmarks, compared to 42.24% for GRPO (+4.54 percentage points) and 45.39% for Knapsack-RL (+1.39 points). The per-benchmark breakdown on this model shows:

AIME24: CoBA-RL 20.00% vs. GRPO 15.00% (+5.00), Knapsack-RL 21.82% (−1.82)
AIME25: CoBA-RL 18.33% vs. GRPO 12.71% (+5.62), Knapsack-RL 15.21% (+3.12)
AMC23: CoBA-RL 48.08% vs. GRPO 43.36% (+4.72), Knapsack-RL 47.08% (+1.00)
MATH500: CoBA-RL 62.64% vs. GRPO 62.23% (+0.41), Knapsack-RL 62.50% (+0.14)
Olympiad: CoBA-RL 46.08% vs. GRPO 40.00% (+6.08), Knapsack-RL 43.11% (+2.97)

The largest absolute gains over GRPO appear on the hardest benchmarks — AIME25 (+5.62), Olympiad (+6.08), AIME24 (+5.00) — where exploration budget allocation has the most room to improve over uniform sampling. On MATH500, which is the easiest benchmark in the suite and where the instruction-tuned model already achieves over 60% under GRPO, the improvement is minimal (+0.41 points), suggesting that when the base model already performs well without specialized exploration, adaptive allocation provides marginal additional benefit.

On Qwen2.5-7B-Base, CoBA-RL achieves 47.43% average accuracy vs. 43.60% for GRPO (+3.83 points) and 44.90% for Knapsack-RL (+2.53 points). The pattern of per-benchmark gains is similar to the Instruct variant, though the absolute numbers differ slightly. On Qwen3-4B-Base, CoBA-RL achieves 40.95% vs. GRPO's 36.21% (+4.74 points) — the largest average improvement across all model configurations. On Qwen3-1.7B-Base, the gain is 26.74% vs. 22.60% (+4.14 points). These results demonstrate that the improvement from capability-aware allocation is consistent across model scales (1.7B to 7B) and initialization types (instruction-tuned vs. base), and that the absolute gain is roughly 4–5 percentage points across the tested configurations.

Notably, CoBA-RL outperforms Knapsack-RL on all four model configurations' averages (46.78% vs. 45.39%, 47.43% vs. 44.90%, 40.95% vs. 38.34%, 26.74% vs. 25.60%), supporting the paper's claim that capability-awareness — the dynamic reweighting of value based on the model's global failure rate — provides benefit beyond what static difficulty-based allocation can achieve. However, the margin over Knapsack-RL is smaller than the margin over GRPO (roughly 1–2.5 points vs. 4–5 points), indicating that static difficulty-based allocation already captures a substantial fraction of the efficiency gain, with capability-awareness providing an additional but more modest improvement.

Training Curves: Olympiad Benchmark (Figure 3)

Figure 3 shows validation accuracy on the Olympiad benchmark tracked over training steps for GRPO, Knapsack-RL, and CoBA-RL across the four model configurations. Across all model scales, CoBA-RL (red curves) achieves higher final accuracy than both baselines, with the separation becoming more pronounced as training progresses. On Qwen2.5-7B-Instruct, the curves show CoBA-RL pulling ahead of Knapsack-RL around step 400 and maintaining that advantage through the end of training. The Qwen3-1.7B-Base panel reveals a different dynamic: CoBA-RL's accuracy curve shows a steeper ascent in the final third of training (roughly steps 350–500) compared to both baselines, whose curves appear to plateau. This aligns with the "Exploit → Explore" narrative — early exploitation of easy problems builds a foundation from which late-stage exploration of harder problems yields accelerating returns, while the baselines' fixed allocation strategies exhaust the learning signal from their preferred difficulty regime.

A limitation of Figure 3 is the absence of error bars or multiple random seeds, making it impossible to assess whether curve separations are statistically significant or within training noise. The x-axis labels are not precisely marked, so exact step counts at which separations occur must be estimated visually.

Exploration-Exploitation Scheduling: "Exploit → Explore" vs. "Explore → Exploit" (Table 2, Figure 5)

Table 2 compares two scheduling strategies on Qwen2.5-7B-Instruct. The "Exploit → Explore" strategy (CoBA-RL's default, where $\alpha_t$ decreases over training) achieves an average accuracy of 46.78%. The reversed "Explore → Exploit" strategy (where $\alpha_t$ increases over training) achieves 42.83% — a difference of 3.95 percentage points. The gap is largest on the hardest benchmarks:

AIME25: 18.33% (exploit-first) vs. 10.41% (explore-first), a difference of 7.92 points
AIME24: 20.00% vs. 16.67%, difference of 3.33 points
AMC23: 48.08% vs. 43.33%, difference of 4.75 points
MATH500: 62.64% vs. 63.01%, difference of −0.37 points (reversal — explore-first slightly better)
Olympiad: 46.08% vs. 43.33%, difference of 2.75 points

The near-identical MATH500 performance under both schedules (62.64% vs. 63.01%) suggests that for problems the instruction-tuned model can already solve reliably, the exploration-exploitation order matters little — both strategies converge to similar mastery levels. The dramatic gap on AIME25 (+7.92 points) indicates that the exploration-exploitation order is most consequential for the hardest problems, where attempting exploration too early (before basic reasoning patterns are consolidated) actively harms final performance.

Figure 5 visualizes the $\alpha_t$ trajectories for both strategies. Under "Exploit → Explore" (left panel), $\alpha_t$ exhibits a fluctuating downward trend, starting high (preferring easy samples) and decreasing over training toward lower values (shifting preference toward harder samples). Under "Explore → Exploit" (right panel), $\alpha_t$ shows a fluctuating upward trend, starting low and increasing. The paper does not specify the exact initial and final $\alpha_t$ values, only the monotonic direction of change.

This result provides evidence for a causal claim about scheduling: given the same total training budget and the same allocation mechanism (the Capability-Oriented Value Function), simply reversing the direction of $\alpha_t$ evolution produces a 3.95-point average accuracy swing. This is stronger evidence than a simple comparison against external baselines, because it isolates the scheduling direction as the only variable.

Comparison Against Static and Heuristic Strategies (Table 3)

Section 3.4 benchmarks CoBA-RL against three non-adaptive allocation strategies on Qwen2.5-7B-Instruct. The two static strategies use fixed Beta parameters throughout training: exploitation-oriented $(\alpha, \beta) = (10.5, 1.5)$ and exploration-oriented $(\alpha, \beta) = (1.5, 10.5)$ . The heuristic strategy uses Linear Step Decay, where $\alpha_t$ decreases stepwise from 10 to 1 (i.e., $10 \to 9 \to \dots \to 1$ ) according to a pre-defined training step schedule, independent of actual model capability.

CoBA-RL achieves the highest average accuracy at 46.78%, compared to 45.39% for Linear Step Decay, 45.21% for the exploitation-oriented static strategy, and lower values for the exploration-oriented static strategy (exact numbers not reported in Table 3 but implied to be lower). The margin over the best static strategy is 1.57 percentage points; over the best heuristic strategy, it is 1.39 points.

What this reveals: a fixed preference for exploitation (always favoring easy samples) performs reasonably well, consistent with the "Exploit → Explore" finding that exploitation is important early in training. However, the static exploitation strategy cannot shift toward harder problems as the model improves, limiting its final performance. The Linear Step Decay improves on static strategies by gradually shifting preference from easy to hard, validating the idea that difficulty preference should evolve. But its pre-defined schedule — decreasing $\alpha_t$ from 10 to 1 at fixed step intervals — cannot respond to the model's actual learning rate. CoBA-RL's capability-conditioned scheduling (where $\alpha_t$ depends on the actual global failure rate) outperforms this open-loop schedule by a modest but consistent 1.39 points, demonstrating that adapting to real capability beats adapting to a pre-defined clock.

The small gap between Linear Step Decay and CoBA-RL (1.39 points) — compared to the larger gap between CoBA-RL and GRPO (4.54 points) — suggests that the primary efficiency gain comes from having a non-uniform allocation at all (vs. GRPO's uniform strategy), with the additional benefit of capability-conditioned scheduling being real but smaller. This implies practitioners could capture most of the gain with a simple decay schedule if implementing the full global failure rate tracking is infeasible, though CoBA-RL's mechanism remains superior and is computationally cheap (0.124 seconds per allocation step).

Budget Sensitivity and Data Efficiency (Figure 6)

Figure 6 evaluates accuracy on Qwen2.5-7B-Instruct across five total budget levels: $B_{\text{total}} \in \{1024, 2048, 4096, 8192, 16384\}$ . For each budget, accuracy is reported for GRPO, Knapsack-RL, and CoBA-RL. The key findings:

CoBA-RL outperforms both baselines at every budget level, with the largest absolute advantage appearing at intermediate budgets (4096–8192) where the allocation flexibility is most beneficial.
At $B_{\text{total}} = 2048$ , CoBA-RL achieves 45.52% accuracy — this is higher than GRPO's 42.78% at $B_{\text{total}} = 4096$ (double the budget). In other words, CoBA-RL matches or exceeds GRPO's performance using half the total rollout budget, a 2× data efficiency improvement.
At $B_{\text{total}} = 16384$ , the largest budget tested, CoBA-RL achieves approximately 46.8% (exact value must be read from the figure since the paper does not state it in text). Knapsack-RL achieves approximately 45.5%, and GRPO approximately 42.5% — the gap between CoBA-RL and Knapsack-RL narrows at high budgets (roughly 1.3 points vs. the 3–5 point gap against GRPO).

What this reveals: the diminishing-returns property of the value function is empirically validated — at very high budgets, additional rollouts contribute minimally even for the adaptive method, causing all methods to converge toward similar asymptotic performance. The practical implication is that CoBA-RL's advantage is largest when budgets are constrained (the typical real-world scenario), and that overspending on rollouts yields diminishing returns regardless of allocation strategy.

A critical detail: the GRPO baseline in Figure 6 uses uniform allocation at each budget level — when $B_{\text{total}} = 8192$ , every sample receives exactly $8192/M$ rollouts. This is a weaker GRPO configuration than the standard $G=16$ used in Table 1, meaning the GRPO accuracy at $B_{\text{total}} = 4096$ (which, with $M=256$ for 7B models, gives $G=16$ per sample) should correspond to the Table 1 baseline of 42.24%. The fact that CoBA-RL at $B_{\text{total}} = 2048$ surpasses this with half the total budget is the paper's strongest single efficiency result.

Task Difficulty Transition Analysis (Appendix D.1, Figure 7)

Appendix D.1 provides a substantively different perspective on the results: rather than measuring final benchmark accuracy, it tracks how individual training samples transition between difficulty categories from initial training to final training. Samples are categorized into five difficulty levels based on their initial pass rate $p_i$ : extremely-hard ( $p_i = 0$ ), hard ( $0 < p_i \leq 0.2$ ), medium ( $0.2 < p_i < 0.8$ ), easy ( $0.8 \leq p_i < 1.0$ ), and extremely-easy ( $p_i = 1.0$ ). The transition matrices in Figure 7 show, for each initial category, the percentage of samples that end up in each final category.

CoBA-RL achieves the highest conversion rates across all difficulty levels:

Medium tasks (initial $0.2 < p_i < 0.8$ ): CoBA-RL converts 71.2% to a higher final pass rate category, compared to 46.8% for GRPO and 50.0% for Knapsack-RL — a 24.4-point advantage over GRPO and 21.2 points over Knapsack-RL. This is the difficulty regime where the value function's $p_i(1-p_i)$ saturation factor is largest, making medium problems the natural target for budget allocation.
Hard tasks (initial $0 < p_i \leq 0.2$ ): CoBA-RL converts 36.7% to higher categories, compared to 17.3% for GRPO and 20.4% for Knapsack-RL — roughly doubling GRPO's conversion rate.
Extremely-hard tasks (initial $p_i = 0$ ): CoBA-RL converts 8.7%, vs. 4.1% for GRPO — a more modest absolute gain but a 2.1× relative improvement. The low absolute numbers confirm that problems initially at zero pass rate are genuinely difficult to improve, but CoBA-RL's exploration budget allocation produces a measurable conversion rate where GRPO's uniform strategy largely fails to move the needle.
Easy tasks (initial $0.8 \leq p_i < 1.0$ ): CoBA-RL converts 88.8% to extremely-easy, vs. 74.0% for GRPO — the exploitation focus in early training accelerates mastery consolidation.
Extremely-easy retention: CoBA-RL retains 95.2% of initially extremely-easy samples in that category, vs. lower retention for baselines (exact numbers not stated but visible in Figure 7). This counters a potential concern that adaptive allocation might neglect easy samples entirely, causing forgetting — the minimum budget $B_{\text{low}} = 2$ ensures continued exposure.

What this reveals: the transition analysis provides evidence for the mechanism underlying CoBA-RL's benchmark improvements — the method does not simply boost a few hard problems but systematically improves conversion rates across the entire difficulty spectrum, including consolidation of easy problems and retention of already-mastered material. This supports the claim that capability-aware allocation balances exploitation and exploration rather than simply trading one off against the other.

Ablation Studies and Robustness Checks

Total budget variation (Figure 6): CoBA-RL maintains superiority over GRPO and Knapsack-RL across five budget levels from 1024 to 16384, with the largest advantage at intermediate budgets. At $B_{\text{total}} = 2048$ , CoBA-RL (45.52%) exceeds GRPO at $B_{\text{total}} = 4096$ (42.78%), demonstrating 2× data efficiency. The gap between CoBA-RL and Knapsack-RL narrows at high budgets, consistent with diminishing returns from additional rollouts.

Exploration-exploitation scheduling direction (Table 2, Figure 5): Reversing the $\alpha_t$ trajectory from "Exploit → Explore" (decreasing $\alpha_t$ ) to "Explore → Exploit" (increasing $\alpha_t$ ) causes a 3.95-point average accuracy drop on Qwen2.5-7B-Instruct, with the largest gap on AIME25 (18.33% vs. 10.41%, −7.92 points). MATH500 is insensitive to scheduling direction (62.64% vs. 63.01%).

Static vs. dynamic value functions (Table 3): Fixed Beta parameters — exploitation-oriented $(\alpha, \beta) = (10.5, 1.5)$ and exploration-oriented $(\alpha, \beta) = (1.5, 10.5)$ — underperform CoBA-RL (46.78%) by 1.57 and >1.57 points respectively. Linear Step Decay (45.39%), which decreases $\alpha_t$ on a pre-defined schedule without capability feedback, underperforms by 1.39 points, confirming that capability-conditioned adaptation outperforms open-loop scheduling.

Sensitivity to Beta concentration parameter $\kappa$ (Appendix D.2, Figure 8): Sweeping $\kappa \in \{7, 11, 15, 21\}$ on Qwen2.5-7B-Instruct yields accuracy ranging from 45.40% ( $\kappa = 7$ ) to 46.61% ( $\kappa = 11$ ), with the default $\kappa = 11$ selected for main experiments. The performance variation across this range is only 1.21 percentage points, demonstrating robustness — the method does not depend on precise tuning of the distribution concentration.

Model scale sensitivity (Table 1, Figure 3): CoBA-RL improves over GRPO by 4.54, 3.83, 4.74, and 4.14 percentage points on 7B-Instruct, 7B-Base, 4B-Base, and 1.7B-Base respectively. The consistent ~4-point gain across a 4× parameter range (1.7B to 7B) and two architectural families (Qwen2.5 and Qwen3) provides evidence that the allocation mechanism does not depend on specific model scale or architecture.

Runtime efficiency of allocation (Table 4): Ablating the optimization algorithm, Dynamic Programming (the standard knapsack solver) requires 115.05 seconds for $M = 512$ , $B_{\text{total}} = 8192$ . The Heap-Based Greedy strategy reduces this to 0.124 seconds — a 927× speedup — while producing identical allocations (due to the diminishing-marginal-utility guarantee). This confirms that allocation overhead is negligible for online training loops.

Negative result: CoBA-RL does not improve MATH500 substantially: On Qwen2.5-7B-Instruct, the MATH500 gain over GRPO is only +0.41 points (62.64% vs. 62.23%), and the "Explore → Exploit" strategy actually achieves a slightly higher 63.01%. This negative result implies that for benchmarks where the base model already performs well under uniform allocation, the headroom for improvement via reallocation is minimal — the model does not need special exploration or exploitation to master these problems.

Critical Assessment

Claim: "CoBA-RL significantly outperforms GRPO across multiple benchmarks"

Substantially supported, but the baseline comparison is favorable. Table 1 demonstrates a 4.54-point average improvement on Qwen2.5-7B-Instruct, with per-benchmark gains ranging from +0.41 (MATH500) to +6.08 (Olympiad). The pattern replicates across four model configurations. The improvement is therefore real and consistent, not an artifact of a specific model or benchmark.

However, the paper's GRPO baseline uses uniform $G = 16$ rollouts per sample at all training steps. This is the simplest possible GRPO configuration — no adaptive sampling, no curriculum, no difficulty-based filtering. The paper does not compare against a GRPO variant that adjusts the per-sample budget based on simple heuristics (e.g., allocating more budget to samples with intermediate historical pass rates, which could be implemented with a few lines of code). Such a heuristic baseline would isolate how much of CoBA-RL's gain comes from any non-uniform allocation vs. specifically from capability-conditioned non-uniform allocation. The comparison against Linear Step Decay in Table 3 provides partial evidence (CoBA-RL outperforms by 1.39 points), but a per-sample heuristic (not a global schedule) would be a more direct test.

Additionally, the GRPO baseline in Figure 6 uses uniform allocation at each budget level, meaning at $B_{\text{total}} = 4096$ with $M = 256$ , the effective $G = 16$ matches Table 1. CoBA-RL at $B_{\text{total}} = 2048$ with the same $M$ would have an average $G = 8$ , yet it matches GRPO's accuracy at $G = 16$ . This is a 2× data efficiency result — but it conflates two effects: (1) better allocation of a fixed budget, and (2) the specific budget level. A fairer comparison would hold $G$ constant and vary the allocation strategy, rather than comparing different $G$ values.

Claim: "CoBA-RL outperforms static difficulty-based allocation (Knapsack-RL)"

Supported with consistent but modest margins. Across the four model configurations in Table 1, CoBA-RL outperforms Knapsack-RL by 1.39, 2.53, 2.61, and 1.14 percentage points. These margins are smaller than the GRPO margins and are not tested for statistical significance. The paper's re-implementation of Knapsack-RL (since the official code is not available) introduces potential implementation differences — if the re-implementation is weaker than the original for any reason, the gap between CoBA-RL and a properly implemented Knapsack-RL could be smaller or nonexistent.

A missing experiment that would strengthen this claim: a direct ablation comparing CoBA-RL's value function with Knapsack-RL's static value function while keeping all other aspects of the training pipeline identical (batch size, optimizer, reward function, evaluation protocol). The paper states that both methods use the Clip-higher strategy from DAPO and are implemented in Verl, but subtle implementation differences could confound the comparison.

Claim: "Exploit-then-Explore scheduling is optimal for LLM RL training"

Well-supported as an empirical finding, but the mechanistic explanation is speculative. Table 2 demonstrates a 3.95-point difference between the two scheduling directions, and Figure 5 confirms the $\alpha_t$ trajectories follow the claimed patterns. The evidence for the direction of the effect is clear.

The mechanistic explanation — that early exploitation stabilizes the policy and provides a foundation for later exploration — is plausible but not directly tested. To validate this mechanism, the paper would need to show that (a) gradient variance is higher under "Explore → Exploit" early in training, and (b) this higher variance correlates with worse final performance — neither of which is measured. The paper also does not explore whether intermediate schedules (e.g., a balanced schedule that neither strongly exploits nor explores early) would perform comparably to or better than "Exploit → Explore." The binary comparison of two extreme trajectories demonstrates the existence of a scheduling effect but does not characterize the space of possible schedules.

A missing experiment that would be informative: varying the rate of $\alpha_t$ change (e.g., rapid vs. gradual shift from exploitation to exploration) to determine whether the optimal schedule has a specific "velocity" in addition to a direction.

Claim: "CoBA-RL achieves superior data efficiency, matching GRPO performance with half the budget"

Demonstrated but the comparison is narrowly scoped. Figure 6 shows CoBA-RL at $B_{\text{total}} = 2048$ (45.52%) exceeding GRPO at $B_{\text{total}} = 4096$ (42.78%). This is a 2× efficiency gain at a specific operating point. However, the efficiency claim only applies within the range tested — it is possible that at even lower budgets ( $B_{\text{total}} < 2048$ ) or much higher budgets, the efficiency ratio changes. The claimed 2× should be interpreted as an empirical observation at one budget pair, not a general efficiency scaling law.

Moreover, the paper does not report the total computational cost of training — including the allocation overhead (0.124 seconds per step, negligible) and the cost of maintaining pass rate statistics — so the "efficiency" is measured in rollout counts rather than wall-clock time or FLOPs. This is standard in the RLVR literature but means the efficiency claim is about data, not about total compute. If the adaptive allocation strategy required substantially more training steps to converge (which it does not appear to, based on Figure 3), the per-step rollout reduction might be offset by needing more steps — but this is not the case in the reported results.

Claim: "The Capability-Oriented Value Function effectively identifies samples with high training value"

Supported by the transition analysis (Appendix D.1) but not directly verified. Figure 7 shows that CoBA-RL achieves higher conversion rates across all difficulty categories, which is consistent with the mechanism — samples are receiving budgets appropriate to their current learning potential. However, the transition analysis compares initial-to-final difficulty classifications but does not show the budget allocation trajectories that caused these transitions. A missing analysis: tracking which samples received high budgets at which training stages and verifying that these allocations correspond to the value function's predictions. Without this, the causal chain from "value function identifies high-value samples" to "higher conversion rates" is correlational rather than demonstrated.

Weaknesses and Missing Experiments

No statistical significance reporting. All results in Tables 1–3 are point estimates without confidence intervals, standard deviations, or p-values. Given the batch size of 256–512 samples, the test sets of varying sizes (AIME24/AIME25 have 30 problems each, MATH500 has 500), and the inherent training stochasticity in RL, some of the reported gaps — particularly the 1–2 point margins over Knapsack-RL — could fall within random variation. Multi-seed runs would clarify this.

Single training dataset. All experiments use DAPO-Math-17K. The paper does not test whether the findings generalize to other RLVR training datasets with different difficulty distributions, such as coding benchmarks, reasoning tasks with different answer formats, or datasets with non-binary reward functions. The DAPO-Math-17K dataset's difficulty distribution may be particularly well-suited to CoBA-RL's mechanism; a dataset with a very different distribution (e.g., mostly easy or mostly hard problems) might show different patterns.

Evaluation uses avg@16 but reports single-run accuracy. The training uses adaptive per-sample budgets, but the evaluation generates exactly 16 rollouts per problem and averages. This evaluation protocol favors methods that train the model to perform well with exactly 16 rollouts — but CoBA-RL's training exposes the model to varying per-sample budgets (2 to 128). If the model's performance depends on the number of rollouts it was trained with (e.g., it learns to rely on the specific budget range), the avg@16 evaluation might under- or over-estimate true generalization. A robustness check varying the evaluation budget (e.g., avg@8, avg@32) would address this.

No comparison against compute-matched GRPO with more steps. CoBA-RL uses the same number of training steps as GRPO with the same total budget per step. An alternative comparison would be: give GRPO more training steps such that the total rollout count across all steps matches, and see if GRPO catches up. If GRPO with more steps achieves comparable performance, the efficiency gain is in wall-clock time (fewer steps) rather than in total rollouts. The paper does not run this comparison.

Hyperparameter reporting is incomplete. The values of $\alpha_{\min}$ , $\alpha_{\max}$ , and $\lambda$ in Equation 7 are not stated in the main text. The temperature $\tau$ in the Budget Saturation Factor (Equation 8) is not specified. The smoothing window $k$ for the moving average of the failure rate is not specified. The scaling factor $\gamma = 10$ is specified, but the exact sigmoid function implementation (e.g., standard logistic vs. a variant) is not detailed. These omissions make exact reproduction of results dependent on code inspection.

No ablation of the sigmoid transformation (Equation 6). The non-linear transformation of the failure rate is described as important for maintaining sensitivity in low-failure regimes, but no ablation compares CoBA-RL with vs. without this transformation (i.e., using raw $\bar{\mathcal{F}}_t$ directly). This is a missed opportunity to validate the paper's own design rationale.

Limited reporting of training dynamics. Beyond Figure 3 (validation accuracy curves) and Figure 5 ( $\alpha_t$ trajectories), the paper provides no analysis of how the budget distribution evolves over training — which samples receive how many rollouts, how this distribution shifts with capability changes, or whether the allocation converges to a stable pattern. Figure 4 shows a static visualization of budget distribution, not its temporal evolution. This makes it difficult to assess whether the allocation mechanism is working as theoretically intended (shifting from exploitation to exploration as capability improves) or simply finding a fixed heterogeneous allocation that happens to outperform uniform.

6. Limitations and Trade-offs

6.1 Difficulty Estimation Cost Is Unaccounted for in the Headline 4× Efficiency Claim

The compute-optimal scaling framework hinges entirely on the ability to estimate question difficulty before deciding how to allocate the inference budget. The paper's method for doing so — generating 2048 samples per question and averaging either ground-truth correctness (oracle) or PRM final-answer scores (predicted) — is extraordinarily expensive. At 2048 samples per question, the difficulty estimation step alone consumes more compute than the largest test-time budgets being studied (256–512 generations). Section 3.2 acknowledges this explicitly:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

Consequence: The reported 4× efficiency gains over best-of-N are computed after difficulty is known, without amortizing the cost of learning it. In a realistic deployment, the total cost would be difficulty estimation + strategy execution, and the former could dominate the latter — potentially eliminating the claimed efficiency advantage entirely. If generating 2048 samples is required to save a factor of 4 on the subsequent 64–256 samples, the net outcome is a massive net increase in total computation, not a savings.

Evidence in the paper: Figures 4 and 8 show that predicted (PRM-based) difficulty bins perform nearly as well as oracle bins, confirming that ground-truth labels are not required. However, the predicted bins still require 2048 samples + PRM scoring per question — the computational elephant in the room that Section 3.2 flags but does not resolve. The paper provides no difficulty estimation method that is cheaper than the savings it enables.

Mitigation status: The authors acknowledge this limitation and suggest future work on "pretraining or finetuning models to directly predict difficulty of a question" (Section 8). They also briefly mention the possibility of adaptive difficulty estimation (starting with a small number of samples, assessing difficulty, then allocating the remaining budget), but no such method is developed or evaluated. Until a cheap difficulty estimator is demonstrated, the 4× figure should be interpreted as an upper bound on achievable efficiency — what would be possible if difficulty were known for free — rather than a realized deployment gain.

6.2 Test-Time Compute Cannot Compensate for Fundamental Capability Gaps

Across all methods studied — search, revisions, and their compute-optimal combinations — the hardest questions (difficulty bin 5, where the base model's pass@1 is near zero) show near-zero improvement regardless of compute budget. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all methods and all budgets. In Figure 7 (right), bin 5 shows roughly 2–3% accuracy irrespective of the sequential-to-parallel ratio. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5% across all $R$ values.

Consequence: This establishes a hard boundary on test-time compute scaling: test-time compute can amplify existing capability but cannot create it from nothing. If the base model's pass@1 is near zero on a problem class, no amount of search or revision will help — there are no correct solutions in the proposal distribution to find or refine. For genuinely novel or out-of-distribution reasoning problems that exceed the base model's training distribution, pretraining remains the only viable path. Deployments targeting problems near or beyond the model's capability frontier (e.g., competition math, frontier scientific reasoning) will see diminishing or zero returns from additional test-time compute.

Evidence in the paper: This limitation is demonstrated consistently and transparently. Section 7's FLOPs-matched analysis (Figure 9) is explicit: on the hardest problems, the 14× larger model outperforms compute-optimal test-time scaling at every $R$ value, and the smaller model's curve is essentially flat (no scaling behavior). Section 5.3's search difficulty-bin analysis (Figure 3, right) shows bin 5 remaining flat across budget levels. Section 6's revision difficulty-bin analysis (Figure 7, right) replicates this null result in the revision setting.

Mitigation status: The paper is candid about this limitation and does not attempt to mitigate it — it is presented as a fundamental boundary condition rather than a solvable problem. The takeaway is framed positively: within the capability range where test-time compute does help (easy-to-medium problems, bins 1–4), the gains are substantial. But this limitation means the approach cannot replace pretraining for problems requiring qualitatively new capabilities.

6.3 Revisions and Search Are Studied Independently, Not Combined

The paper studies two complementary axes — PRM-guided search and iterative revisions — but never combines them. Section 8 explicitly acknowledges this gap:

"we did not experiment with PRM tree-search techniques in combination with revisions"

Consequence: The current results represent a lower bound on what a fully integrated system could achieve. Search improves candidate selection (finding the best answer among generated candidates via the PRM) while revisions improve the proposal distribution (generating better candidates by conditioning on previous attempts). These strengths are complementary: revisions can generate higher-quality candidates that are easier for the PRM to score accurately, and PRM feedback could guide which revisions to pursue rather than blindly generating long revision chains. Without combining them, we cannot know whether the two mechanisms compound synergistically (yielding gains beyond either alone) or redundantly (their benefits overlap, making the combination only marginally better than the stronger individual method). The paper's 4× efficiency figure is therefore a lower bound, and the true ceiling for test-time compute might be meaningfully higher.

Evidence in the paper: This is not an empirical finding but an acknowledged gap. The paper demonstrates that both mechanisms independently improve over baselines (search in Section 5.3, revisions in Section 6), but provides no experiments where both are active simultaneously. The discussion of complementary strengths — revisions helping most on easy problems, search helping most on medium-hard problems — in Section 7's "Key Insight" implicitly suggests combination could yield difficulty-spanning improvements, but this remains hypothetical.

Mitigation status: Explicitly flagged as future work in Section 8. The authors position the current paper as establishing the individual scaling properties of each mechanism, with combination as the natural next step. This is a reasonable scope limitation for a first paper on compute-optimal scaling, but a practitioner deploying the method today would be leaving potential performance on the table.

6.4 Single Benchmark and Single Model Family Limit Generalization

All experiments use the MATH benchmark (500 test questions) with PaLM 2-S* as the base model, combined with a second PaLM-family model (roughly 14× larger parameters) for the FLOPs-matched comparison. Section 4 states that the authors "believe this model is representative of the capabilities of many contemporary LLMs," but this claim is unverified beyond the PaLM architecture.

Consequence: Several aspects of the findings could be model-specific or benchmark-specific:

PRM quality and over-optimization behavior depend on PaLM 2-S*'s output distribution. A model with different calibration properties, different error patterns, or different tokenization might exhibit different difficulty-dependent scaling curves — potentially shifting or even reversing which search algorithms are optimal for which difficulty bins.
The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities, which vary substantially across model families (e.g., GPT-4 vs. PaLM vs. LLaMA exhibit different in-context learning fidelity).
The MATH benchmark consists exclusively of competition-level math problems requiring symbolic reasoning and producing closed-form answers. It is unclear whether the difficulty-dependent patterns (beam search hurting easy problems, revisions helping easy problems) generalize to code generation (where correctness is verified by unit tests), logical reasoning (where answer formats differ), scientific QA (where factual knowledge interacts with inference), or tasks requiring open-ended generation (where correctness is ambiguous).

Evidence in the paper: The paper acknowledges the single-benchmark limitation implicitly through its exclusive use of MATH throughout, but does not provide any cross-benchmark or cross-model-family robustness checks. The FLOPs-matched comparison is between two PaLM-family models only. No experiments test the approach on non-math domains.

Mitigation status: Not addressed. The authors do not suggest cross-benchmark evaluation as future work, though it is an obvious extension. The representativeness claim about PaLM 2-S* is stated without supporting evidence. A practitioner adopting this method for a non-math domain or a different model family (e.g., LLaMA, Mistral, Qwen) would need to replicate the entire scaling analysis — difficulty binning, PRM training, strategy selection per bin — to determine whether the findings transfer.

6.5 The 14× Larger Model Baseline Is Not Compute-Optimally Trained, Making the Pretraining-Versus-Inference Comparison Favorable to Test-Time Compute

The FLOPs-matched comparison in Section 7 scales model parameters while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023) rather than Chinchilla-optimal scaling (Hoffmann et al., 2022), where both data and parameters are scaled equally. The authors state:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

Additionally, the larger model uses only greedy decoding with no test-time compute augmentation — no majority voting, no best-of-N, no search. This means the comparison is between (a) a smaller model with compute-optimal test-time strategies, and (b) a larger model trained with a sub-optimal parameter/data ratio and evaluated with the simplest possible decoding strategy.

Consequence: The reported advantages of test-time compute over pretraining may shrink or reverse against a stronger baseline. A Chinchilla-optimal model trained with 14× more total FLOPs (scaling both parameters and data) would likely outperform the parameter-only-scaled model used in the comparison. Similarly, giving the larger model even a modest test-time compute budget — e.g., best-of-8 with a trained verifier — would create a much stronger baseline. The paper's core claim that "a smaller model with additional test-time compute can outperform a ~14× larger model" (Section 7) is therefore better stated as "can outperform a 14× larger parameter-only-scaled model with greedy decoding" — a narrower and less practically impressive claim.

Evidence in the paper: The section 7 FLOPs-matched analysis (Figure 9, Table-based results in Section 7) demonstrates the stated comparisons but does not test against a compute-optimally trained baseline or a larger model with its own test-time compute budget. The caveats about the pretraining configuration are stated but the comparison is still presented as evidence for test-time compute substituting for pretraining compute.

Mitigation status: The authors acknowledge the departure from compute-optimal pretraining and frame the Chinchilla-optimal comparison as future work. They do not acknowledge the asymmetry in test-time compute budget (the smaller model gets a generous budget while the larger model gets none). This is an area where the reported results should be interpreted as a best-case scenario for test-time compute — the substitution effect would likely be weaker in a more symmetric comparison.

6.6 The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate with No Principled Solution

Section 6.1 reports that approximately 38% of correct answers produced during a revision chain get "revised" back to incorrect answers in the subsequent step. This is a direct consequence of the training data construction: the model only sees incorrect-to-correct trajectories during training (the sequences always end with a correct answer), so it has no signal for what to do when the current answer is already correct. When the model encounters a correct answer in its own revision chain, it may treat it as needing revision — and since it only knows how to revise incorrect answers, it produces a new answer that is often incorrect.

Consequence: This creates a fundamental instability in the revision process. The model cannot recognize when it has already produced a correct answer and should stop revising. Instead, it continues generating revisions, with a non-trivial probability of overwriting correct answers with incorrect ones. The paper mitigates this with verifier-based selection across the entire chain (picking the best answer from any step, not just the final one), but this is an imperfect patch:

The verifier itself has errors and may select an incorrect answer that scored spuriously high.
The compute spent on revisions after the correct answer is already found is wasted — the budget allocated to the revision chain could have been redirected to other problems.
The mitigation introduces additional latency (the chain must run to completion, then be scored) rather than enabling early stopping when a correct answer is produced.

Evidence in the paper: Section 6.1 reports the 38% reversion rate explicitly. Section 6's description of the within-chain selection mechanism (majority voting or verifier-based) is the primary mitigation approach discussed. The paper does not report what fraction of total revision budget is wasted on revisions that regress from correct answers — a statistic that would quantify the practical cost of this limitation.

Mitigation status: Partially mitigated through post-hoc selection (picking the best answer from the chain), but not solved at the model level. The paper does not explore training the revision model to recognize when revision is unnecessary (e.g., by including trajectories where the correct answer is retained without change), nor does it explore incorporating the PRM's per-step scores to trigger early stopping during the revision chain. This is not flagged as future work, though it is an obvious improvement direction. The ReST^EM experiment in Appendix K (Figure 16), where additional RL-style optimization actually degraded revision performance, suggests that the revision training procedure is fragile and that naive attempts to fix the reversion problem could backfire.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a capability-relative definition of training value into the RL-based LLM post-training paradigm, shifting the resource allocation problem from "how hard is this problem?" (a static property) to "what can the model learn from this problem right now?" (a dynamic, state-dependent property). This is not a complete paradigm shift — the underlying GRPO training framework, the reward function, and the policy architecture remain unchanged — but it is a methodological reframing of how practitioners should think about rollout budgets during RLVR training.

The primary conceptual contribution is the demonstration that the optimal per-sample budget allocation is a moving target, not a fixed function of problem difficulty. Prior work (Knapsack-RL, curriculum learning methods) treated difficulty as the primary axis along which to differentiate budget — harder problems get more rollouts, easier problems get fewer. This paper shows that such static allocation is provably suboptimal because it ignores the model's learning trajectory: a problem at difficulty level D has fundamentally different training value at step 100 versus step 500, because the model's capability has changed. The evidence for this reframing is not a single experiment but the consistent 1–2.5 percentage point improvement of CoBA-RL over Knapsack-RL across four model configurations (Tables 1 and 3), where the only difference is that CoBA-RL's value function shifts with the global failure rate while Knapsack-RL's remains static.

A second landscape-changing contribution is the empirical resolution of the exploration-exploitation scheduling question for LLM RL training. The paper provides direct evidence (Table 2, Figure 5) that "Exploit → Explore" — prioritizing easy problems early to consolidate basic reasoning before shifting to harder problems — substantially outperforms the reverse schedule, with a 3.95 percentage point average accuracy gap on Qwen2.5-7B-Instruct and a striking 7.92 point gap on AIME25 (18.33% vs. 10.41%). This challenges the intuitive assumption that exploration should come first to discover novel solution strategies, and instead argues that exploitation establishes the foundation on which productive exploration depends. This finding resolves a latent tension in the literature: curriculum learning methods have long advocated easy-to-hard training, but prior RLVR work had not tested whether this principle applies specifically to per-sample budget allocation (as opposed to data selection). CoBA-RL provides evidence that it does, and quantifies the magnitude of the effect.

A third contribution is the demonstration that capability-aware allocation can be implemented with negligible computational overhead. The Heap-Based Greedy allocator runs in 0.124 seconds for a batch of 512 samples with 8192 total budget — a 927× speedup over Dynamic Programming (Table 4) — making it practical to recompute the allocation at every training step. This is not a theoretical innovation but a practical engineering result: prior budget allocation methods in the LLM space (Knapsack-RL, GVM-RAFT) have not emphasized allocator efficiency, and the paper shows that efficiency need not be sacrificed for adaptivity. This matters because it lowers the barrier to adoption: CoBA-RL can be inserted into an existing GRPO pipeline with a single budget_allocator.allocate() call before the generation step (Appendix B, Listing 1), and it adds no meaningful latency to the training loop. If the allocator were expensive (e.g., Dynamic Programming's 115 seconds), capability-aware allocation would be confined to academic experiments; the heap-based formulation makes it a drop-in module for production training.

The paper also redirects research attention from data selection to budget allocation within the RLVR context. Prior work on efficient RL training for LLMs has largely focused on which samples to include in each batch (curriculum learning, data filtering, difficulty-based subset selection). CoBA-RL demonstrates that substantial efficiency gains — 2× data efficiency at matching GRPO performance (Figure 6) — can be achieved by varying per-sample budgets within a fixed batch, without excluding any samples. This suggests that budget allocation is an under-explored dimension of training efficiency that can complement (rather than replace) data selection strategies. A system that both selects which samples to train on and allocates heterogeneous budgets to those samples could yield multiplicative efficiency improvements.

Finally, the paper provides a new diagnostic tool for understanding RLVR training dynamics: the task difficulty transition matrices in Appendix D.1 (Figure 7). By tracking how individual training samples move between difficulty categories from initial to final training, these matrices reveal whether a training method is improving hard problems (exploration), consolidating easy ones (exploitation), or both. This is a more granular evaluation than final benchmark accuracy — it shows where improvement occurs across the difficulty spectrum, not just how much improvement occurs in aggregate. The finding that CoBA-RL achieves higher conversion rates than GRPO across all difficulty categories (71.2% vs. 46.8% for medium tasks, 36.7% vs. 17.3% for hard tasks, 88.8% vs. 74.0% for easy tasks) provides mechanistic evidence that capability-aware allocation balances exploitation and exploration rather than trading one against the other. This diagnostic approach could be adopted by future work to evaluate whether proposed methods genuinely improve across the difficulty spectrum or merely shift performance from one regime to another.

Follow-Up Research This Work Enables

Direct verification that budget allocations follow the value function's predictions. The paper demonstrates that CoBA-RL outperforms baselines and shows task difficulty transitions (Figure 7), but does not directly verify the causal mechanism: that the value function's preference density genuinely shifts from high-pass-rate to low-pass-rate samples as training progresses, and that the budget allocation follows this shift. A follow-up study would log the per-sample budget allocation at each training step and the corresponding value function parameters (α_t, β_t), then compute the correlation between the value function's predicted high-value pass-rate range and the actual pass rates of samples receiving above-average budgets. If the mechanism works as claimed, the correlation should be high and should shift systematically — from easy samples early in training to intermediate samples later — tracking the α_t trajectory from Figure 5. A negative result (budgets allocated independently of pass rates, or correlation that does not shift over time) would indicate that the performance gains come from some other aspect of CoBA-RL (e.g., the minimum-budget constraint, the saturation factor's variance-weighting) rather than from capability-awareness per se.

Stress-test the "Exploit → Explore" finding on non-mathematical RLVR domains. The paper's scheduling finding — that exploitation-first outperforms exploration-first by 3.95 points — is demonstrated exclusively on mathematical reasoning benchmarks (AIME, AMC, MATH500, Olympiad). It is unknown whether this scheduling principle generalizes to other RLVR domains with different reward structures. A direct replication on code generation (where correctness is verified by unit tests and the reward is binary pass/fail), multi-step agentic reasoning (where partial credit may exist for intermediate steps), or safety alignment (where the reward structure is fundamentally different — avoiding harmful outputs rather than producing correct ones) would establish the domain boundaries. The key hypothesis to test: does "Exploit → Explore" work because early easy-problem training stabilizes reasoning patterns (domain-general), or because mathematical reasoning has specific properties (symbolic manipulation, well-defined correctness) that make exploitation early especially beneficial? If the finding fails to replicate on code generation, it would suggest the scheduling principle is domain-specific and that the optimal exploration-exploitation schedule depends on task structure.

Combine CoBA-RL with curriculum-based data selection. The paper explicitly distinguishes its approach from curriculum learning: CoBA-RL allocates heterogeneous budgets to all samples in a fixed batch, while curriculum methods select which samples to include. These are orthogonal dimensions of training efficiency — budget allocation determines how many rollouts each selected sample receives, while curriculum selection determines which samples appear in the batch at all. A combined system would use CoBA-RL's value function to simultaneously decide (a) which samples to include in the batch (curriculum selection, filtering out samples with very low value under the current density) and (b) how many rollouts to allocate to included samples (budget allocation). The hypothesis: multiplicative efficiency gains, because curriculum selection eliminates samples that would receive only the minimum budget (B_low = 2) under pure budget allocation, freeing those rollouts for reallocation to higher-value samples. The experiment would compare pure CoBA-RL, pure curriculum learning (e.g., ADCL or SEC), and the combined system, measuring both final benchmark accuracy and total training compute.

Test whether the global failure rate signal can be replaced with a cheaper, more direct capability estimator. The paper's capability signal requires computing and averaging pass rates over the current batch — this comes "for free" since the rollouts are generated for the policy update anyway, but it means the capability estimate is retrospective (based on the current batch's pass rates, which are only available after generation). A cheaper, predictive capability estimator — e.g., a small probe model that predicts per-sample pass rates from the question text and the current policy's hidden states — could enable capability estimation before generation, allowing the allocation to be computed before the batch is processed rather than at the next step. The experiment would measure (a) the correlation between the probe's predicted pass rates and actual pass rates, and (b) whether replacing the retrospective capability signal with the predictive one maintains or improves CoBA-RL's performance. A positive result would reduce the latency between capability change detection and allocation response (currently one step), potentially improving allocation accuracy during periods of rapid capability improvement.

Ablation of the sigmoid transformation to validate the sensitivity-preservation rationale. Equation 6 applies a sigmoid transformation to the smoothed failure rate when it falls below 0.5, with the stated purpose of maintaining sensitivity to small capability changes during late training. The paper provides no ablation comparing CoBA-RL with vs. without this transformation (i.e., using raw F̅_t directly to compute α_t). The experiment is straightforward: train CoBA-RL with the transformation disabled, with γ varied (e.g., γ ∈ {1, 5, 10, 20}), and with the threshold at which the transformation activates varied (e.g., 0.3, 0.5, 0.7). If the transformation genuinely matters, performance should degrade when it is removed or when γ is too small (insufficient sensitivity amplification); if performance is independent of the transformation, the sensitivity-preservation rationale is disproven and the method's capability-awareness is driven primarily by the raw failure rate signal, with the sigmoid being an unnecessary complexity.

Multi-seed statistical validation of the Knapsack-RL comparison margin. The paper reports CoBA-RL outperforming Knapsack-RL by 1.14–2.61 percentage points across four model configurations (Table 1), but without confidence intervals or multi-seed runs. Given the batch size of 256–512 samples and the inherent stochasticity of RL training, a 1–2 point margin could fall within random variation. A follow-up study running both methods with 5–10 random seeds would establish whether the capability-awareness advantage over static difficulty-based allocation is statistically reliable. A null result — the margin disappearing under multi-seed evaluation — would not invalidate CoBA-RL's improvement over GRPO (which is larger at 4–5 points) but would substantially weaken the paper's central claim that capability-awareness specifically (as opposed to any non-uniform allocation) is the active ingredient. It would suggest that the benefit comes primarily from the variance-weighting in the saturation factor (the pi(1-pi) term that automatically deprioritizes saturated samples) rather than from the dynamic Beta distribution shape-shifting.

Practical Applications and Downstream Use Cases

Cost-constrained LLM post-training for small research labs and startups. The paper's most directly actionable finding is the 2× data efficiency demonstrated in Figure 6: CoBA-RL at B_total = 2048 achieves 45.52% accuracy on Qwen2.5-7B-Instruct, matching GRPO's 42.78% at B_total = 4096. For a team with a fixed GPU budget for post-training, this translates to achieving the same final model quality with half the generation cost. In concrete terms: training a Qwen2.5-7B model on DAPO-Math-17K for ~1000 steps with GRPO and G=16 would require generating 256 × 16 × 1000 = 4,096,000 rollout trajectories. CoBA-RL could match this performance with roughly half that generation count (plus negligible allocation overhead of 0.124 seconds per step), directly halving the GPU-hours spent on the most expensive part of the training loop (inference for rollout generation). The integration complexity is minimal — Appendix B shows the modification is a single budget_allocator.allocate() call inserted before generation — making this applicable to existing Verl-based training pipelines without framework migration. The primary practical caveat is that the specific 2× figure is demonstrated on one model family (Qwen2.5/Qwen3) and one dataset (DAPO-Math-17K); teams using different model architectures or training on non-math domains should calibrate the efficiency ratio for their setting, but the default configuration (κ=11, γ=10, Blow=2, Bup=128) provides a strong starting point.

Production LLM fine-tuning pipelines where training compute is the dominant cost. For organizations that regularly RL-fine-tune LLMs for downstream deployment (e.g., monthly model refreshes, customer-specific adaptations), CoBA-RL's improvement over GRPO is consistent across model scales — +4.54 points on 7B-Instruct, +4.74 on 4B-Base, +4.14 on 1.7B-Base (Table 1) — meaning the method provides a reliable performance uplift at fixed compute cost regardless of model size. The 0.124-second allocation overhead is negligible compared to generation time, so the improvement comes at zero effective latency cost. The primary deployment consideration is that CoBA-RL requires maintaining per-sample pass rate statistics (to compute the value function and feed the allocator), which adds a small state-tracking requirement not present in vanilla GRPO. For production pipelines that already log per-sample metrics, this is trivial; for minimal pipelines, it requires storing and updating a dictionary mapping sample indices to accumulated pass rates, which is a lightweight engineering addition. The robustness of the method to the κ hyperparameter (performance varies only 1.21 points across κ ∈ {7, 11, 15, 21} in Figure 8) means it can be deployed with default settings without expensive per-task hyperparameter tuning.

RLVR training for specialized reasoning domains where problem difficulty distribution is unknown. In settings where an organization is applying RLVR to a new domain — e.g., legal reasoning, medical diagnosis, scientific protocol design — the difficulty distribution of the training data is unknown a priori. Static allocation strategies (like Knapsack-RL) require a pre-defined mapping from difficulty to training value, but if the difficulty distribution is misestimated, the allocation can be systematically suboptimal. CoBA-RL's capability-relative value function avoids this problem: it does not require knowing which problems are "hard" or "easy" in an absolute sense, only the current pass rates of each sample under the current policy. The value function automatically deprioritizes both saturated-easy and impossible-hard samples through the pi(1-pi) variance term, focusing budget on the intermediate-difficulty regime regardless of where that regime falls in absolute terms. This makes CoBA-RL particularly suitable for novel domains where practitioners lack the prior knowledge to design a difficulty-to-value mapping. The practical trade-off: CoBA-RL requires computing pass rates by generating rollouts (which the policy update already does), so it adds no new data requirements; the only deployment consideration is setting κ (the Beta concentration parameter), for which the paper's sensitivity analysis suggests the default of κ=11 is a robust starting point.

When to Prefer This Method

The paper positions CoBA-RL explicitly against two alternatives: GRPO (uniform budget allocation, no adaptivity) and Knapsack-RL (static difficulty-based allocation, no capability awareness). It also implicitly contrasts against curriculum learning methods (data selection rather than budget allocation). The decision rules that follow from the paper's evidence are:

Prefer CoBA-RL over GRPO when: (a) the training dataset has heterogeneous difficulty — if all problems are equally hard or equally easy, uniform allocation is nearly optimal and the allocator adds complexity without benefit; (b) training compute is constrained and data efficiency matters — Figure 6 shows 2× efficiency at matching GRPO performance, so the method is most valuable when total rollout budget is the bottleneck; (c) the model undergoes substantial capability improvement during training — CoBA-RL's value function shifts with the global failure rate, so it provides most benefit when the model's pass rate changes significantly from start to finish (which is typical for base models undergoing RLVR from scratch). The integration cost is low (Appendix B shows a single API call), so the barrier to preference is primarily whether heterogeneity and capability shift are present.
Prefer CoBA-RL over Knapsack-RL when: (a) the model's training trajectory is long enough that capability changes meaningfully — Knapsack-RL's static value function becomes increasingly misaligned as the model improves, so the gap between the two methods should widen with training duration (visible in Figure 3, where CoBA-RL pulls ahead in the final third of training); (b) the problem difficulty distribution is unknown or shifting — CoBA-RL does not require a pre-defined difficulty-to-value mapping and automatically adjusts to whatever pass-rate distribution emerges. The evidence for CoBA-RL's advantage over Knapsack-RL is consistent but modest (1–2.5 points across configurations), so the preference is justified but not overwhelming — Knapsack-RL captures a large fraction of the gain from non-uniform allocation, and CoBA-RL provides an additional capability-awareness increment.
Prefer Knapsack-RL or a static allocation over CoBA-RL when: (a) the training dataset is small and pass rates are unreliable — CoBA-RL's value function depends on accurate per-sample pass rate estimates, which require sufficient rollouts per sample to be statistically stable; if the dataset has few samples and pass rates are noisy, the allocator's decisions may be dominated by estimation noise rather than genuine value differences; (b) the training duration is very short (few steps) — the capability signal evolves over time, so if only a handful of training steps are performed, the dynamic preference density does not have time to shift meaningfully, and a static allocation may be equally effective with less implementation complexity; (c) implementation simplicity is paramount — CoBA-RL requires maintaining a heap, tracking global failure rate moving averages, and recomputing the value function each step, which adds engineering surface area relative to a hardcoded budget distribution (like the Linear Step Decay baseline in Table 3, which achieves 45.39% vs. CoBA-RL's 46.78% — a 1.39 point gap that may not justify the additional complexity in all settings).
CoBA-RL vs. curriculum learning methods is not a direct trade-off — the paper explicitly frames them as orthogonal: curriculum learning selects which samples to train on, CoBA-RL allocates budgets to samples in a fixed batch. They can be combined, and the paper's evidence does not provide a basis for preferring one over the other; a system using both would likely outperform either alone.