$V_{0.5}$: Generalist Value Model as a Prior for Sparse RL Rollouts

ArXiv: 2603.10848

🎯 Pitch

Sparse rollouts with only 4 samples already beat dense 16-sample baselines—provided you statistically test whether a frozen value model is hallucinating before trusting its predictions. By adaptively blending the prior only when real-time hypothesis tests confirm reliability, V0.5 slashes variance without importing bias, delivering over 10% accuracy gains across six math benchmarks.

---

1. Executive Summary

This paper introduces V0.5, a framework that adaptively fuses a frozen generalist value model's prior predictions with sparse empirical rollouts to construct robust advantage baselines for RL-based LLM post-training. Evaluated across six mathematical reasoning benchmarks (AIME 2024, AIME 2025, Olympiad Bench, MATH500, Minerva Math, AMC 2023) using Qwen3-4B-Instruct-2507 as the base policy, V0.5 integrates two named mechanisms—Empirical Shrinkage Fusion (a convex combination of the empirical mean and the V0 prior, weighted by a real-time hypothesis test that detects prior hallucinations) and Sequential OSLA Allocation (a dynamic stopping rule that triggers additional rollouts only when the observed discrepancy statistically exceeds the noise floor)—to minimize baseline MSE while bounding induced bias to O(1/√k). V0.5 with only 4 initial rollouts outperforms standard GRPO and DAPO at 16 rollouts, achieving over 10% absolute accuracy improvement and faster convergence, establishing that a generalist value prior can safely substitute for extensive Monte Carlo sampling only when the prior's reliability is validated through real-time statistical testing rather than assumed uniformly.

2. Context and Motivation

The Core Problem: Baseline Estimation Under Sparse Rollouts Must Trade Bias Against Variance With No Principled Safety Guarantees

The paper addresses a specific statistical bottleneck in Reinforcement Learning with Verifiable Rewards (RLVR) for LLM post-training: how to construct a stable, low-variance advantage baseline when you can afford only a handful of online rollouts per prompt. This is not a toy concern — it is the operational reality for long-horizon reasoning tasks where each rollout generates thousands of tokens of chain-of-thought, making extensive Monte Carlo sampling prohibitively expensive. The baseline serves as the anchor for computing the advantage $A = r - \mu$; subtract an accurate $\mu$, and the policy gradient updates reinforce only those behaviors that exceed the model's current expected performance. Subtract a noisy or biased $\mu$, and the gradient signal becomes corrupted in ways that cascade through training.

The paper identifies this as a strict bias-variance tradeoff with no safe default option under sparse rollouts. On one side sits the empirical mean $\bar{v}_G = \frac{1}{G}\sum_{k=1}^G r_k$ — unbiased (its expectation equals the true expected return $\mu_{\text{true}}$), but with observation variance $\sigma^2_{\text{noise}} \propto 1/G$. With a small group size $G = 4$ or $G = 8$, this variance is large enough to produce wildly fluctuating baselines, destabilizing policy updates. On the other side sits a parameterized value model — low variance by construction (a single forward pass produces a smooth prediction), but requiring synchronous training alongside the evolving policy. This synchronous coupling is the deployment bottleneck: the value model must be updated every training step to track the non-stationary policy, consuming GPU memory, compute cycles, and engineering complexity that rival the policy model itself. Worse, if the value model fails to generalize to the policy's current output distribution (an out-of-distribution problem that worsens as the policy drifts), it introduces systematic bias that the policy gradients cannot distinguish from genuine improvement signals.

Neither approach provides a principled mechanism to detect when its estimates are unreliable. The empirical mean has no way to know whether its high variance in a particular batch is producing a misleading baseline; the value model has no way to know whether it has entered an OOD regime where its predictions are hallucinatory. In both cases, the policy consumes the corrupted advantage signal and degrades silently.

Why This Matters: Gradient Variance Amplification in Billion-Parameter Models

The practical importance of this problem is not merely about getting slightly better accuracy on math benchmarks. It follows from a structural fact about policy gradients in LLMs that the paper formalizes in Theorem 3.1: the trace of the policy gradient covariance matrix is bounded by

$\text{Tr}(\text{Var}(\hat{g}(\theta))) \leq \text{Var}_{\text{oracle}} + \Phi_{\text{score}} \cdot \text{MSE}(b) + L \cdot |\text{Bias}(b)|$

The critical term here is $\Phi_{\text{score}} = \mathbb{E}_\pi[\|\nabla_\theta \log \pi_\theta(y|x)\|^2]$ — the expected squared norm of the score function. For LLMs with billions of parameters, $\Phi_{\text{score}}$ is inherently massive. This means that any estimation error in the baseline, measured as $\text{MSE}(b)$, gets multiplied by a factor that dwarfs it by many orders of magnitude. A variance of $1/G$ in the empirical mean under $G = 4$ translates to gradient noise that can cause the loss to oscillate wildly between steps, produce exploding gradient norms, and drive the policy into degenerate regions of parameter space (evidenced in Figure 3 and Figure 4).

This amplification also explains a commonly observed pathology in GRPO training: rapid entropy collapse. Figure 4 shows GRPO's policy entropy decaying sharply during training under sparse rollouts. When the baseline fluctuates randomly due to high variance, the policy receives spurious reinforcement signals — sometimes rewarding behaviors that are actually average or below-average, other times penalizing good behaviors. These noisy gradients act as a random forcing term that pushes the policy distribution toward low-entropy, over-confident modes, destroying the exploration needed for complex reasoning tasks.

The theoretical significance is that this problem is structural, not accidental. It follows from the scale of modern LLMs and the fundamental limits of Monte Carlo estimation under small sample sizes. No amount of hyperparameter tuning or careful initialization can eliminate it — the only paths are to increase the group size (which is what we want to avoid for computational reasons) or to introduce additional statistical structure that suppresses the noise.

Prior Approaches and Their Specific Shortcomings

The paper situates itself relative to three families of baseline estimation methods, each of which makes a different tradeoff but none of which provides a verified safe fusion of external knowledge with sparse data.

1. Synchronous Value Models (PPO, VAPO). The standard Actor-Critic architecture trains a parameterized value function $V_\phi$ in lockstep with the policy. Schulman et al. (2017) formalized this for LLMs via PPO, and Yue et al. (2025) extended it for reasoning tasks in VAPO. The value model reduces gradient variance substantially when well-calibrated. However, the coupling dilemma is severe: $V_\phi$ must track $\pi_\theta$ as the policy evolves, requiring gradient updates of comparable cost to the policy itself. For the Qwen3-4B model used in this paper, training a separate value model of comparable capacity would roughly double the GPU memory and compute requirements. Moreover, the value model's predictions on novel prompts generated by an updated policy inherently lag behind or extrapolate incorrectly — an OOD generalization failure that introduces bias the system cannot detect because it has no independent ground truth to compare against.

2. Empirical Group Sampling (GRPO, ReMax, OPO, and variants). GRPO (Shao et al., 2024) eliminates the value model entirely, using the intra-group empirical mean as the baseline. This is unbiased by construction, satisfying $\mathbb{E}[\bar{v}_G] = \mu_{\text{true}}$, and requires no additional parameters. However, the paper identifies a direct statistical cost: under sparse rollouts ($G$ small), the observation variance $\propto 1/G$ is high enough to destabilize training. Subsequent work has attempted to patch specific failure modes of the empirical mean:

• Outlier sensitivity: MC-GRPO (Kim, 2026) replaces the mean with the median and normalizes by Median Absolute Deviation to resist extreme reward values that flip advantage signs. This helps with outlier robustness but does nothing to reduce the inherent $1/G$ variance of the central tendency estimate.

• Bias from group-level aggregation: HA-DW (Yang et al., 2026) identifies that group means systematically underestimate the advantage of difficult prompts (because difficult prompts have lower expected reward but the group mean treats all prompts equally within a batch). They introduce Kalman filter-based history-aware anchors to correct this. While this addresses a genuine bias, it adds complexity without touching the variance problem.

• Quantile-based filtering: QAE (Wu et al., 2026) uses K-quantile dual-state gating to filter noisy advantages. BNPO (Xiao et al., 2025) models rewards as a Beta distribution and normalizes via moment estimation. Both refine how the empirical distribution is summarized but remain fundamentally bounded by the information content of the sparse samples themselves.

• Credit assignment granularity: Turn-PPO (Li et al., 2026) and GiGPO (Feng et al., 2025) design baselines for multi-turn or hierarchical decision structures. Tree-OPO (Huang et al., 2025) formulates advantage computation as quadratic programming over Monte Carlo trees. These improve the baseline's structural alignment with the task but still operate within the sample-limited regime.

The unifying limitation across all these empirical methods is that they treat the sparse rollouts as the sole source of information about expected returns. When $G = 4$, they are trying to estimate $\mu_{\text{true}} \in [-1, 1]$ from four binary observations — a problem with fundamental information-theoretic limits that no statistical trick can circumvent.

3. Generalist Value Models (V0). Zhang et al. (2026) recently introduced V0, a frozen value model that leverages in-context learning to estimate the expected return for any policy on any prompt without gradient updates. Rather than embedding policy information in trained weights, V0 accepts a context set $C_\pi = \{(x_i, r_i)\}_{i=1}^N$ of historical query-performance pairs and produces a prediction $V = V_0(x, C_\pi)$ via a single forward pass through a specialized architecture (embedding backbone, residual query adapter, and TabPFN-based probabilistic inference head). Because V0 is fully pre-trained offline and frozen during RL, it breaks the coupling dilemma: no synchronous training, no gradient memory overhead, no computational cost beyond inference.

However, the paper identifies a critical vulnerability that prevents naive use of V0 as a direct baseline replacement. As a generalist model trained on diverse but finite historical data, V0 is susceptible to hallucinations on out-of-distribution prompts — problems whose difficulty, format, or reasoning structure differs from its training distribution. These hallucinations manifest as systematic prediction errors $\Delta^2 = (V - \mu_{\text{true}})^2 > 0$. If V0's prediction were used directly as the baseline without verification, these errors would introduce persistent, uncorrected bias into the advantage calculation. Unlike the zero-mean noise of the empirical mean, this bias would systematically push policy updates in wrong directions — the policy would be reinforced to match V0's flawed estimate rather than the true expected return.

This creates the central dilemma that V0.5 is designed to resolve: V0 provides zero-variance guidance at zero training cost, but with unknown and potentially severe bias. The empirical mean provides unbiased estimation at zero bias, but with high variance under sparsity. Neither can detect its own failures. The statistical tradeoff is strict — you cannot minimize both bias and variance simultaneously — so the only way forward is a mechanism that verifies the prior's reliability before using it and adaptively shifts between the two estimators based on real-time evidence.

How V0.5 Positions Itself: Safe Fusion Through Real-Time Hypothesis Testing

The paper's positioning is distinctive because it does not propose a new value model architecture, a new policy gradient estimator, or a new reward normalization scheme. Instead, it proposes a meta-statistical layer that wraps the prior and the empirical mean in a decision-theoretic framework. The key intellectual move is to treat the value model's prior not as an estimate to be trusted or discarded wholesale, but as a statistical hypothesis to be tested against live data.

The V0.5 framework operates through two coupled mechanisms that together implement this verification logic:

Empirical Shrinkage Fusion constructs a convex combination of the prior $V$ and the empirical mean $\bar{v}_k$: $\mu^* = w\bar{v}_k + (1-w)V$. The weight $w$ is not a fixed hyperparameter — it is computed per-prompt in real-time via an empirical estimate of the optimal shrinkage weight $w^* = \Delta^2 / (\Delta^2 + \sigma^2_{\text{noise}})$, which itself follows from minimizing the MSE of the combined estimator. Crucially, the empirical estimate of the prior bias $\hat{\Delta}^2_k = \max(0, (\bar{v}_k - V)^2 - 1/k)$ incorporates a positive-part truncation that the paper proves is functionally equivalent to a hypothesis test with null hypothesis $H_0: \Delta^2 = 0$ (the prior is correct) against $H_1: \Delta^2 > 0$ (the prior is hallucinating). When the observed discrepancy $(\bar{v}_k - V)^2$ falls within the theoretical noise bound $1/k$, the truncation activates ($\hat{\Delta}^2_k = 0$), meaning the prior is fully trusted ($w = 0$) and all variance is suppressed. When the discrepancy exceeds the noise bound, the truncation deactivates, the system estimates the true bias by subtracting the expected noise floor, and the weight shifts toward the empirical mean — isolating the prior's influence.

Sequential OSLA Allocation extends this logic into a dynamic budget allocation problem. Relying on a fixed small $k$ can produce false rejections of an accurate prior due to the sheer randomness of limited sampling. The OSLA mechanism frames the decision to allocate an additional rollout as a cost-benefit analysis: compute the expected reduction in empirical MSE from one more sample and compare it to the marginal compute cost $c$. The paper derives a closed-form stopping rule (Theorem 3.6):

$K^* = \inf\left\{k \geq k_{\min} : k \geq \frac{1}{\sqrt{c}} - \frac{1}{\hat{\Delta}^2_k}\right\}$

The first term $1/\sqrt{c}$ sets the maximum budget ($\approx 16$ for the paper's chosen $c = 0.0039$), while the second term $1/\hat{\Delta}^2_k$ acts as a dynamic discount: when the prior is accurate ($\hat{\Delta}^2_k$ small), $1/\hat{\Delta}^2_k$ is large, the stopping condition is met early, and the system halts with minimal compute. When the prior is hallucinating ($\hat{\Delta}^2_k$ large), $1/\hat{\Delta}^2_k$ is small, the budget expands, and the system collects more samples to override the prior with empirical data. This creates an automatic difficulty-aware compute scheduler: easy prompts (where V0 is reliable) consume fewer rollouts; hard or OOD prompts (where V0 hallucinates) receive more compute to correct the baseline.

The paper's positioning relative to the prior work is therefore not competitive but integrative. GRPO uses pure empirical sampling; PPO uses pure value model guidance; V0.5 uses empirically-verified value model guidance, where the verification is statistical rather than heuristic. The contribution is a framework that makes it safe to use a generalist prior — the safety comes from the hypothesis test that actively monitors for hallucinations and the dynamic allocator that corrects them when detected. This is fundamentally different from both "trust the prior" (which would inherit its bias) and "ignore the prior" (which would inherit the variance), and it explains why V0.5 can outperform GRPO at $G = 16$ using only $k_{\text{init}} = 4$ rollouts: it is not doing more with less data, but rather doing more by knowing when the data it has is sufficient and when it needs more.

The Specific Gap V0.5 Fills

To summarize the precise gap: prior to V0.5, there was no principled method for combining a frozen generalist value prior with sparse empirical rollouts in a way that (1) formally minimizes the baseline MSE, (2) provides bounded guarantees on induced bias, (3) detects and isolates prior hallucinations without human intervention, and (4) dynamically scales compute only when statistically necessary. The existence of V0 created the opportunity for such a framework — a zero-cost prior with known susceptibility to OOD errors — but the framework itself was missing. V0.5 fills this gap with a specific, mathematically grounded design whose components (shrinkage estimation, sequential hypothesis testing, OSLA stopping) are individually known from classical statistics (James-Stein estimation, Wald's sequential analysis) but had not been synthesized into a coherent inference-time compute allocator for RLVR. The paper's repeated emphasis on safety — that the system must bound the damage from prior hallucinations while exploiting the prior's variance reduction — reflects the core design constraint that distinguishes V0.5 from a naive ensemble or fixed-weight interpolation.

3. Technical Approach

3.1 Reader Orientation

V0.5 is a meta-statistical framework that wraps a frozen generalist value model (V0) and a policy model's sparse rollouts in a real-time hypothesis-testing and dynamic budget allocation loop to produce low-variance, bias-bounded advantage baselines for policy gradient updates. The problem it solves is the strict bias-variance tradeoff in sparse-rollout RLVR: the empirical mean is unbiased but high-variance, the prior is zero-variance but potentially biased, and neither can detect its own failures. The shape of the solution is a verification-then-fusion pipeline — test the prior against live data, weight it according to how much the data supports it, and allocate additional compute only when the test indicates the prior is unreliable.

3.2 Big-Picture Architecture (Diagram in Words)

The V0.5 system has four major components connected in a sequential decision loop:

1. Generalist Value Model (V0) — A frozen, pre-trained model that accepts a policy's historical performance context and a target prompt, and outputs a prior prediction $V$ of the expected return. It runs once per prompt, before any rollouts are generated.

2. Sparse Rollout Generator — The policy model $\pi_\theta$ itself, which generates $k$ candidate responses $\{o_1, \ldots, o_k\}$ for a given prompt $x$, receiving binary rewards $\{r_1, \ldots, r_k\} \in \{-1, 1\}$ from a verifier. The number $k$ starts small ($k_{\text{init}} = 4$) and may grow dynamically.

3. Empirical Shrinkage Fusion Engine — Computes the empirical mean $\bar{v}_k = \frac{1}{k}\sum r_i$, estimates the prior bias $\hat{\Delta}^2_k = \max(0, (\bar{v}_k - V)^2 - 1/k)$, and constructs the fused baseline $\hat{\mu}^* = \hat{w}_k \bar{v}_k + (1 - \hat{w}_k)V$ with weight $\hat{w}_k = \hat{\Delta}^2_k / (\hat{\Delta}^2_k + 1/k)$. This engine also standardises the advantage $A_i = (r_i - \hat{\mu}^*) / \hat{\sigma}^*$ where $\hat{\sigma}^* = \sqrt{1 - (\hat{\mu}^*)^2}$.

4. Sequential OSLA Allocator — Evaluates the stopping condition $k \geq 1/\sqrt{c} - 1/\hat{\Delta}^2_k$ after each round of rollouts. If satisfied, it halts and passes the fused baseline to the policy update. If not, it triggers an additional batch of 2 rollouts and loops back to the Fusion Engine, up to a maximum of ~16 per prompt.

Information flows: prompt $x$ enters $\rightarrow$ V0 produces prior $V$ $\rightarrow$ policy generates $k_{\text{init}} = 4$ rollouts $\rightarrow$ Fusion Engine computes $\hat{\mu}^*$ and tests for prior hallucination $\rightarrow$ OSLA Allocator decides stop or continue $\rightarrow$ (if continue) policy generates 2 more rollouts $\rightarrow$ loop back to Fusion $\rightarrow$ (if stop) final advantage $A_i$ passed to surrogate objective $\rightarrow$ policy updated.

3.3 Roadmap for the Deep Dive

• First, the theoretical motivation (Theorem 3.1) that establishes why baseline MSE matters for LLM policy gradients — this provides the optimisation objective that all subsequent mechanisms serve.
• Second, the Empirical Shrinkage Fusion mechanism — the MSE decomposition (Theorem 3.2), the optimal static weight (Theorem 3.3), the empirical approximations (Equations 6–9), and the bias safety bounds (Theorem 3.4). This is the core statistical engine.
• Third, the Sequential OSLA Allocation mechanism — the empirical risk function, the marginal return derivation (Theorem 3.5), the closed-form stopping rule (Theorem 3.6), and the base group size analysis (Appendix A.8). This is the dynamic compute scheduler.
• Fourth, the end-to-end workflow — how these components connect in the five-step implementation described in Section 4.1.3, including practical engineering choices like batch padding and minimum dispatch thresholds.
• Fifth, the V0 model architecture and training pipeline — what produces the prior that V0.5 consumes, since the prior's properties (zero cost, susceptibility to OOD hallucination) define the fusion problem.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a statistical decision-making paper whose core idea is that a generalist value model's prior should be treated as a hypothesis to be tested against live rollouts, not as a fixed estimate to be trusted or ignored. The mechanisms (shrinkage fusion, OSLA stopping) are derived from classical statistics — James-Stein estimation and Wald's sequential analysis — but synthesised into a specific architecture for RLVR baseline construction that provides formal guarantees on both bias and variance.

---

Theoretical Motivation: Why Baseline MSE Dominates Policy Gradient Variance (Theorem 3.1)

The paper begins not with a method but with a justification of the objective. Before designing any mechanism, it establishes precisely what happens when a baseline has estimation error and why that error is catastrophic in the LLM regime.

Consider a single-step policy gradient estimator:

$\hat{g}(\theta) = \nabla_\theta \log \pi_\theta(o|x)(r - \mu)$

where $\theta$ are the policy parameters, $o$ is a generated response for prompt $x$, $r \in \{-1, 1\}$ is the verifier reward, and $\mu$ is the baseline subtracted to compute the advantage. The true policy gradient is $g_{\text{true}} = \mathbb{E}_o[\nabla_\theta \log \pi_\theta(o|x)(r - \mu_{\text{true}})]$ with $\mu_{\text{true}} = \mathbb{E}_{o \sim \pi_\theta}[r|x]$ being the true expected return. Any baseline $\mu$ independent of the current action $o$ preserves unbiasedness ($\mathbb{E}[\hat{g}(\theta)] = g_{\text{true}}$), but different choices of $\mu$ produce different gradient variance.

The paper proves that the trace of the gradient covariance matrix satisfies:

$\text{Tr}(\text{Var}(\hat{g}(\theta))) \leq \text{Var}_{\text{oracle}} + \Phi_{\text{score}} \cdot \text{MSE}(\mu) + L \cdot |\text{Bias}(\mu)|$

where $\text{Var}_{\text{oracle}}$ is the irreducible variance when $\mu = \mu_{\text{true}}$ (perfect baseline), $\text{MSE}(\mu) = \mathbb{E}[(\mu - \mu_{\text{true}})^2]$ is the baseline's mean squared error, $\Phi_{\text{score}} = \mathbb{E}_\pi[\|\nabla_\theta \log \pi_\theta(y|x)\|^2]$ is the expected squared norm of the policy's score function (a measure of how sensitive the log-probability is to parameter changes), and $L$ is a constant from the cross-perturbation penalty term $2\mathbb{E}_{o,\mu}[\|\nabla_\theta \log \pi_\theta(o|x)\|^2(r - \mu_{\text{true}})(\mu_{\text{true}} - \mu)]$.

What it computes: The total variance of the policy gradient estimator, decomposed into three sources — the minimum achievable variance (oracle), the variance contributed by baseline estimation error scaled by the model's gradient sensitivity, and a penalty from systematic baseline bias interacting with the reward signal.

Why this form: The decomposition follows from expanding the squared advantage $(r - \mu)^2 = (r - \mu_{\text{true}})^2 + (\mu_{\text{true}} - \mu)^2 + 2(r - \mu_{\text{true}})(\mu_{\text{true}} - \mu)$ and substituting into the second moment $\mathbb{E}[\|\hat{g}(\theta)\|^2]$. The cross-term $2(r - \mu_{\text{true}})(\mu_{\text{true}} - \mu)$ evaluates to a constant times the baseline bias after separating the score function's independence from the baseline. The three terms are orthogonal in the sense that they capture irreducible sampling noise, baseline quality, and bias-correlation effects respectively.

Why this matters (the "insight" part): For LLMs with billions of parameters, $\Phi_{\text{score}}$ is inherently massive — it scales with the number of parameters. This means that even a modest $\text{MSE}(\mu)$ gets multiplied by an enormous factor, causing gradient variance to explode far beyond what one would expect from the raw MSE magnitude. Conversely, the bias term is scaled only by $L$ (not $\Phi_{\text{score}}$), making it cheaper in gradient-variance terms to tolerate a bounded bias than to tolerate unbounded variance. This asymmetry — bias hurts less than variance at LLM scale — is the mathematical justification for V0.5's entire design philosophy: intentionally accept a small, controlled bias (from the prior) in exchange for a large reduction in MSE (from the fusion). A method that insisted on strict unbiasedness (like GRPO) would pay an enormous $\Phi_{\text{score}} \cdot \text{MSE}$ penalty that no amount of careful implementation can avoid — it is a structural consequence of the model size.

---

Empirical Shrinkage Fusion: The Static MSE-Minimising Baseline (Theorems 3.2, 3.3, Equations 6–9, Theorem 3.4)

With the objective established — minimise baseline MSE while bounding bias — the paper constructs the core statistical estimator.

The shrinkage estimator form. Given $k$ sparse rollouts producing empirical mean $\bar{v}_k = \frac{1}{k}\sum_{i=1}^k r_i$ and a prior prediction $V$ from the frozen V0 model, the fused baseline is defined as a convex combination:

$\mu^* = w \bar{v}_k + (1 - w)V$

where $w \in [0, 1]$ controls the reliance on empirical data versus the prior. When $w = 0$, the baseline is purely the prior; when $w = 1$, it is purely the empirical mean; intermediate values blend both.

Theorem 3.2 — Orthogonal MSE decomposition. Substituting this form into the MSE definition yields:

$\text{MSE}(w) = \mathbb{E}[(\mu^* - \mu_{\text{true}})^2] = w^2 \sigma^2_{\text{noise}} + (1 - w)^2 \Delta^2$

where $\sigma^2_{\text{noise}} = \text{Var}(\bar{v}_k)$ is the variance of the empirical mean (from observation noise), and $\Delta^2 = (V - \mu_{\text{true}})^2$ is the squared systematic bias of the prior (its deviation from the true expected return).

What it computes: The expected squared error of the fused baseline as a weighted sum of two independent error sources — the empirical estimator's variance (scaled by $w^2$) and the prior's bias (scaled by $(1-w)^2$). The cross-term vanishes because $\bar{v}_k$ is unbiased ($\mathbb{E}[\bar{v}_k - \mu_{\text{true}}] = 0$), making the covariance between $(\bar{v}_k - \mu_{\text{true}})$ and the constant $(V - \mu_{\text{true}})$ exactly zero.

Why this form: The orthogonality is what enables clean optimisation — the two error sources don't interact, so the optimal weight can be found by balancing them directly. If the prior were not independent of the rollouts (e.g., if it were trained on the same data), the cross-term would couple them and prevent closed-form optimisation. The frozen, independently-trained nature of V0 is what makes this decomposition hold: $V$ is fixed before any rollouts are generated, so it is statistically independent of $\bar{v}_k$.

Theorem 3.3 — Optimal static weight. Differentiating $\text{MSE}(w)$ with respect to $w$ and setting to zero:

$\frac{d}{dw}\left(w^2 \sigma^2_{\text{noise}} + (1-w)^2 \Delta^2\right) = 2w\sigma^2_{\text{noise}} - 2(1-w)\Delta^2 = 0$

Solving yields the unique global minimum (second derivative $2\sigma^2_{\text{noise}} + 2\Delta^2 > 0$):

$w^* = \frac{\Delta^2}{\Delta^2 + \sigma^2_{\text{noise}}}$

What it computes: The weight that optimally balances prior bias against empirical variance. When the prior is accurate ($\Delta^2 \to 0$), $w^* \to 0$ — trust the prior completely. When the prior is severely biased ($\Delta^2 \gg \sigma^2_{\text{noise}}$), $w^* \to 1$ — rely almost entirely on the empirical mean. When both error sources are comparable, the weight interpolates smoothly.

Why this form: The ratio $\Delta^2 / (\Delta^2 + \sigma^2_{\text{noise}})$ has the property that it approaches 0 or 1 at the extremes but produces non-trivial interpolation when both error sources matter. An alternative like a hard threshold ($w = 0$ if $\Delta^2 < \tau$, else $w = 1$) would create a discontinuity that could cause the baseline to jump between two very different values on adjacent prompts, destabilising training. The smooth interpolation avoids this.

From theory to practice — empirical weight estimation. The optimal weight $w^*$ requires the true prior bias $\Delta^2$ and true observation variance $\sigma^2_{\text{noise}}$, neither of which is known at inference time. The paper approximates both from the $k$ real-time rollouts:

1. Variance estimate (Equation 6):

$\hat{\sigma}^2_{\text{noise}} = \frac{1}{k}$

where the variance of a single Bernoulli($p$) reward is $p(1-p) \leq 1/4$ (maximum at $p = 0.5$), but the paper uses the more conservative bound of 1 (the maximum possible variance for any distribution on $[-1, 1]$ under a uniform bound). The variance of the empirical mean of $k$ independent samples is then $\text{Var}(\bar{v}_k) = \text{Var}(r_i)/k \leq 1/k$.

What it computes: A guaranteed upper bound on the observation variance of $\bar{v}_k$ that holds regardless of the true success probability. It is conservative (the true variance may be smaller) but requires no estimation from data, making it robust to the small-sample regime where variance estimates themselves would be noisy.

Why this form: Using $1/k$ rather than an empirical variance estimate $\hat{p}(1-\hat{p})/k$ avoids the circular problem of estimating variance with the same $k$ samples used to estimate the mean. With $k = 4$, an empirical variance estimate would itself have enormous uncertainty — using the theoretical bound is more stable. The cost is conservatism (overestimating variance), which biases the fusion weight slightly toward the prior — a direction that is safe because the hypothesis test (Equation 7) will detect and correct if the prior is actually wrong.

2. Empirical bias estimate (Equation 7):

$\hat{\Delta}^2_k = \max\left(0, (\bar{v}_k - V)^2 - \frac{1}{k}\right)$

What it computes: An estimate of the true prior bias $\Delta^2 = (V - \mu_{\text{true}})^2$ from the observed discrepancy $(\bar{v}_k - V)^2$, after subtracting the expected contribution of observation noise ($1/k$). The $\max(0, \cdot)$ operator ensures the estimate is non-negative — if the observed discrepancy is smaller than the noise floor, the system attributes it entirely to random sampling variation and sets the estimated bias to zero.

Why this form — the hypothesis test equivalence: This is the most subtle and important design choice in the paper. Under the null hypothesis $H_0: V = \mu_{\text{true}}$ (the prior is correct, $\Delta^2 = 0$), the expected value of the squared discrepancy is $\mathbb{E}[(\bar{v}_k - V)^2 | H_0] = \mathbb{E}[(\bar{v}_k - \mu_{\text{true}})^2] = \sigma^2_{\text{noise}} \leq 1/k$. So if the observed $(\bar{v}_k - V)^2$ is at most $1/k$, it is entirely consistent with the prior being correct — there is no statistical evidence of hallucination. The max operator sets $\hat{\Delta}^2_k = 0$ in this case, which forces $\hat{w}_k = 0$ (Equation 8), meaning the prior is fully trusted.

When $(\bar{v}_k - V)^2 > 1/k$, the observed discrepancy exceeds what pure sampling noise can explain under $H_0$. The system rejects the null, concludes the prior has genuine bias, and estimates that bias by subtracting the noise floor: $(\bar{v}_k - V)^2 - 1/k$. This is functionally a one-sided hypothesis test with rejection region $(\bar{v}_k - V)^2 > 1/k$.

The alternative of using the raw squared discrepancy $(\bar{v}_k - V)^2$ without noise correction would overestimate $\Delta^2$ whenever $k$ is small, because part of the observed discrepancy is always due to sampling noise. This overestimation would push $\hat{w}_k$ toward 1 even when the prior is actually correct, needlessly discarding the prior's variance reduction. The noise subtraction is what makes the test fair — it gives the prior the benefit of the doubt up to the theoretical noise ceiling.

3. Adaptive fusion weight (Equation 8):

$\hat{w}_k = \frac{\hat{\Delta}^2_k}{\hat{\Delta}^2_k + \hat{\sigma}^2_{\text{noise}}} = \frac{\hat{\Delta}^2_k}{\hat{\Delta}^2_k + 1/k}$

What it computes: The empirical analogue of the optimal weight $w^*$, plugging in the real-time estimates of bias and variance. When $\hat{\Delta}^2_k = 0$ (prior appears correct), $\hat{w}_k = 0$ and the fused baseline becomes $\hat{\mu}^* = V$ (pure prior). When $\hat{\Delta}^2_k$ is large (prior appears wrong), $\hat{w}_k \to 1$ and the fused baseline approaches $\bar{v}_k$ (pure empirical mean).

Why this form: It inherits the smooth interpolation property of $w^*$ while operating entirely from observable quantities. No hyperparameter tuning is needed — the weight adapts automatically to each prompt's statistical evidence.

4. Final fused baseline (Equation 9):

$\hat{\mu}^* = \hat{w}_k \bar{v}_k + (1 - \hat{w}_k)V$

What it computes: The convex combination of empirical mean and prior, producing a scalar $\hat{\mu}^* \in [-1, 1]$ that serves as the baseline for computing advantages.

Why this form (the crucial property): The weight $\hat{w}_k$ is a function of the random variable $\bar{v}_k$, which makes $\hat{\mu}^*$ a nonlinear shrinkage estimator rather than a simple linear combination. This nonlinearity introduces a statistical bias that was absent in the static $\mu^*$ — because $\bar{v}_k$ appears both in the estimate and in the weight that controls how much the estimate is used, the estimator is no longer unbiased. The paper quantifies and bounds this induced bias in Theorem 3.4.

Theorem 3.4 — Bias safety bounds. The empirical estimator $\hat{\mu}^*$ has the following guarantees:

1. Finite-sample bound: $|\text{Bias}(\hat{\mu}^*)| = |\mathbb{E}[\hat{\mu}^*] - \mu_{\text{true}}| \leq 1/\sqrt{k}$ for any $k$.

2. Asymptotic decay: If $\Delta \neq 0$ (the prior truly has some bias), the induced bias decays as $O(1/k)$ as $k$ increases, which is faster than the $O(1/\sqrt{k})$ rate of the empirical mean's standard deviation.

What this means operationally: The first bound says that even in the worst case with $k = 4$, the bias from the fusion is at most $1/\sqrt{4} = 0.5$. This is a constant bound — it doesn't grow with the model size or the prompt difficulty. Combined with Theorem 3.1, this means the bias term in the gradient variance bound contributes at most $L \cdot 0.5$, which is independent of $\Phi_{\text{score}}$. In exchange, the MSE term gets reduced dramatically (because the prior's zero variance is leveraged when $\hat{\Delta}^2_k = 0$, and the empirical mean's variance is reduced even when $\hat{\Delta}^2_k > 0$ by the weight interpolation). The second bound says that if you do increase the rollout budget (e.g., via OSLA allocation), the bias disappears super-linearly fast — faster than the variance of a pure empirical estimator would, making the fusion estimator increasingly superior.

Proof sketch for the $1/\sqrt{k}$ bound: The paper rewrites $\hat{\mu}^* = \bar{v}_k - (1 - \hat{w}_k)(\bar{v}_k - V)$ and isolates the bias in the correction term $\mathbb{E}[(1 - \hat{w}_k)(\bar{v}_k - V)]$. Substituting the weight formula, this becomes $\mathbb{E}[Z]$ where $Z = \frac{1/k}{\hat{\Delta}^2_k + 1/k}(\bar{v}_k - V)$. A case analysis on the truncation threshold $(\bar{v}_k - V)^2 \leq 1/k$ versus $> 1/k$ shows that $|Z| \leq 1/\sqrt{k}$ pointwise, establishing the bound. The $O(1/k)$ asymptotic follows from Hoeffding's inequality for the tail probability of $\bar{v}_k$ deviating from $\mu_{\text{true}}$ plus the $1/\sqrt{k}$ bound for the rare-event contribution.

Standardisation for advantage computation. The final advantage for rollout $o_i$ is:

$A_i = \frac{r_i - \hat{\mu}^*}{\hat{\sigma}^*}$

where $\hat{\sigma}^* = \sqrt{1 - (\hat{\mu}^*)^2}$ is an intrinsic standard deviation derived from the fused baseline under the assumption of bounded rewards in $[-1, 1]$ with mean $\hat{\mu}^*$.

Why standardise: This follows GRPO's convention of dividing by the group's standard deviation, which normalises the advantage to unit scale and prevents the policy from overfitting to the magnitude of rewards. Using $\sqrt{1 - (\hat{\mu}^*)^2}$ rather than the empirical standard deviation of the $k$ rewards avoids another source of small-sample estimation noise — with $k = 4$, an empirical standard deviation estimate would be extremely unstable.

The surrogate objective. The policy is updated by maximising:

$J_{\text{V0.5}}(\theta) = \mathbb{E}_{x \sim \mathcal{D}_{\text{prompt}}, \{o_i\}_{i=1}^k \sim \pi_{\theta_{\text{old}}}}\left[\frac{1}{k}\sum_{i=1}^k \min\left(\rho_i(\theta) A_i, \text{clip}(\rho_i(\theta), 1-\epsilon, 1+\epsilon) A_i\right)\right]$

where $\rho_i(\theta) = \pi_\theta(o_i|x) / \pi_{\theta_{\text{old}}}(o_i|x)$ is the importance sampling ratio and $\epsilon$ is the clipping parameter (standard PPO-style clipped surrogate).

What it computes: The expected clipped advantage over the $k$ rollouts, where the clipping prevents the policy update from being too aggressive when the new policy diverges too far from the old one. The mean is taken over prompts and rollouts.

Why this form: This is the standard PPO/GRPO clipped objective, not a V0.5 innovation. The key difference from GRPO is that $A_i$ uses the fused baseline $\hat{\mu}^*$ rather than the group mean, and $k$ may be dynamically determined by OSLA rather than fixed.

---

Sequential OSLA Allocation: Dynamic Budget Scheduling (Theorems 3.5, 3.6, Appendix A.7, A.8)

The Shrinkage Fusion mechanism optimally uses a fixed budget $k$. However, with extreme sparsity ($k = 4$), the hypothesis test can produce false rejections — random sampling variation makes an accurate prior appear wrong, causing the system to shift weight toward a noisy empirical mean when the prior would actually have been better. The Sequential OSLA Allocation mechanism addresses this by allowing $k$ to grow adaptively until the statistical evidence is sufficient to make a reliable decision.

The empirical risk function (Equation 11):

$R(k) = \widehat{\text{MSE}}(k) + c \cdot k$

where

$\widehat{\text{MSE}}(k) = \frac{\hat{\Delta}^2_k}{k \hat{\Delta}^2_k + 1}$

and $c$ is the marginal compute cost per additional rollout (set to $c = 0.0039$ in experiments).

What it computes: The total cost of using $k$ rollouts, expressed as the sum of the empirical estimation error (the MSE of the fused baseline using the current bias estimate) and the linear compute cost ($c$ times the number of rollouts). This is a standard regularised risk formulation: more rollouts reduce the MSE but cost compute; the optimal $k$ balances these.

Why this form: $\widehat{\text{MSE}}(k)$ is derived by substituting the empirical estimates $\hat{\Delta}^2_k$ and $\hat{\sigma}^2_{\text{noise}} = 1/k$ into the MSE formula $\text{MSE} = w^2 \sigma^2_{\text{noise}} + (1-w)^2 \Delta^2$ with $w = \hat{w}_k = \hat{\Delta}^2_k / (\hat{\Delta}^2_k + 1/k)$. This substitution yields:

$\widehat{\text{MSE}}(k) = \left(\frac{\hat{\Delta}^2_k}{\hat{\Delta}^2_k + 1/k}\right)^2 \cdot \frac{1}{k} + \left(\frac{1/k}{\hat{\Delta}^2_k + 1/k}\right)^2 \cdot \hat{\Delta}^2_k = \frac{\hat{\Delta}^2_k}{k \hat{\Delta}^2_k + 1}$

The marginal return lower bound (Theorem 3.5):

The expected reduction in empirical MSE from one additional rollout is:

$g(k) = \widehat{\text{MSE}}(k) - \widehat{\text{MSE}}(k+1) = \frac{\hat{\Delta}^4_k}{(k \hat{\Delta}^2_k + 1)((k+1)\hat{\Delta}^2_k + 1)}$

Under the One-Step-Look-Ahead (OSLA) assumption that $\hat{\Delta}^2_{k+1} \approx \hat{\Delta}^2_k$ (the bias estimate doesn't change dramatically from one additional sample), the paper derives a lower bound:

$g(k) > \frac{\hat{\Delta}^4_k}{((k+1)\hat{\Delta}^2_k + 1)^2}$

What it computes: A guaranteed minimum improvement in baseline quality from collecting one more rollout. Because the denominator uses the larger factor $((k+1)\hat{\Delta}^2_k + 1)$ in place of $(k \hat{\Delta}^2_k + 1)$, the bound is conservative — the actual improvement is always at least this large.

Why this form: The OSLA assumption is what makes the analysis tractable. Without it, the future $\hat{\Delta}^2_{k+1}$ would depend on the unknown outcome of the $(k+1)$-th rollout, making the decision a full dynamic programming problem. The OSLA approximation ($\hat{\Delta}^2_{k+1} \approx \hat{\Delta}^2_k$) treats the bias estimate as locally stable, which is reasonable because a single binary observation changes $\bar{v}_k$ by at most $2/k$ (in the binary reward setting), which translates to a small change in $\hat{\Delta}^2_k$ for $k \geq 4$.

The optimal stopping rule (Theorem 3.6):

The system should continue allocating rollouts as long as the expected statistical return exceeds the marginal cost. Setting the lower bound envelope equal to $c$:

$\frac{\hat{\Delta}^4_k}{((k+1)\hat{\Delta}^2_k + 1)^2} = c$

Taking square roots and solving for $k+1$:

$k + 1 = \frac{1}{\sqrt{c}} - \frac{1}{\hat{\Delta}^2_k}$

This yields the stopping condition — halt when the current $k$ satisfies:

$K^* = \inf\left\{k \geq k_{\text{min}} : k \geq \frac{1}{\sqrt{c}} - \frac{1}{\hat{\Delta}^2_k}\right\}$

with $k_{\text{min}} = 4$ (derived in Appendix A.8 and explained below).

What it computes: The smallest number of rollouts at which the marginal benefit of another sample no longer justifies its cost. The first term $1/\sqrt{c}$ sets the absolute maximum budget — when $\hat{\Delta}^2_k$ is infinite (extreme hallucination), $1/\hat{\Delta}^2_k \to 0$ and the stopping condition requires $k \geq 1/\sqrt{c} \approx 16$ (for $c = 0.0039$). The second term $1/\hat{\Delta}^2_k$ acts as an adaptive discount: when the prior appears accurate, $\hat{\Delta}^2_k$ is small, $1/\hat{\Delta}^2_k$ is large, the right-hand side becomes small or negative, and the condition is met immediately (at $k = k_{\text{min}}$). When the prior appears biased, $\hat{\Delta}^2_k$ is large, $1/\hat{\Delta}^2_k$ is small, and the required $k$ grows toward the maximum.

Why this form — the feedback control interpretation: The stopping rule creates a closed-loop system where the prior's apparent quality directly controls the compute budget. An accurate prior ($\hat{\Delta}^2_k \approx 0$) makes $1/\hat{\Delta}^2_k$ very large, so the right-hand side is far below $k_{\text{min}}$ and the system stops immediately — saving compute. A hallucinating prior ($\hat{\Delta}^2_k \gg 0$) reduces the discount, pushing the required $k$ upward — spending compute to correct the baseline. A marginally inaccurate prior ($\hat{\Delta}^2_k$ small but nonzero) produces intermediate budgets — spending just enough compute to refine the baseline.

This is fundamentally different from a fixed budget allocation (always 16 rollouts) or a simple threshold ("use more if the discrepancy exceeds $\tau$"). The continuous dependence on $\hat{\Delta}^2_k$ means the system can distinguish between "prior is slightly off — need a few more samples" and "prior is completely wrong — need many samples" and allocate proportionally. The $\sqrt{c}$ scaling means the maximum budget is set by the engineer's cost tolerance, not by any property of the prompt — making the system tunable for different cost regimes.

Regret bound (Appendix A.7). The paper proves that compared to an oracle that knows the true $\Delta^2$ and directly computes the optimal $k^*_{\text{oracle}} = 1/\sqrt{c} - 1/\Delta^2$, the OSLA scheduler's excess cost is bounded by $O(c)$. For $c = 0.0039$, this means the expected additional compute spent due to estimation noise is equivalent to at most a constant number of extra rollouts (on the order of 1–2). The proof uses a Taylor expansion of the risk function around the oracle optimum, the Delta method to translate $\hat{\Delta}^2_k$ estimation error into stopping-time variance, and renewal theory to correct for the nonlinearity of the stopping boundary.

Practical significance: This bound guarantees that the dynamic allocation does not risk unbounded compute consumption due to decision errors in the small-sample regime. Even when the system makes suboptimal stopping decisions because $\hat{\Delta}^2_k$ is noisy, the total excess cost is bounded by a small constant — making the OSLA mechanism safe for deployment.

The base group size analysis (Appendix A.8): The paper establishes $k_{\text{min}} = 4$ through an analysis of the discrete structure of binary rewards. With rewards $r \in \{-1, 1\}$, the empirical mean $\bar{v}_k$ can only take $k+1$ discrete values (from $-1$ to $1$ in steps of $2/k$). The gap between adjacent possible values is $\text{Gap}(k) = 2/k$. The hypothesis test's tolerance radius (the maximum discrepancy attributed to noise) is $\text{Threshold}(k) = 1/\sqrt{k}$.

For the test to be statistically valid, the tolerance radius must be at least as large as the discrete gap — otherwise, a single additional positive rollout could jump the empirical mean by more than the entire noise allowance, causing the test to flip from accepting to rejecting the prior based on random sampling variation. The condition $\text{Threshold}(k) \geq \text{Gap}(k)$ gives $1/\sqrt{k} \geq 2/k \implies k \geq 4$.

At $k = 4$, the gap is $2/4 = 0.5$ and the threshold is $1/\sqrt{4} = 0.5$ — exactly equal. This means the test can absorb one complete discrete jump while maintaining its decision. For $k < 4$, the gap exceeds the threshold, meaning normal sampling variation routinely triggers false rejections. The paper illustrates this with a concrete example: with a prior $V = 0.8$, at $k = 4$, if 3 of 4 rollouts are correct ($\bar{v}_4 = 0.5$), the discrepancy $|0.5 - 0.8| = 0.3 \leq 0.5$ — the prior is accepted, correctly attributing the single wrong answer to noise. At $k = 3$, the same outcome pattern (2 correct, 1 wrong, $\bar{v}_3 \approx 0.333$) gives discrepancy $|0.333 - 0.8| \approx 0.467$ which would need to be compared against a threshold of $1/\sqrt{3} \approx 0.577$ — still passing, but with less margin. At $k = 2$, the gap is 1.0 while the threshold is only 0.707, making false rejections likely.

This analysis explains the experimental finding in Figure 5: V0.5 fails at $k = 1, 2$ because the hypothesis test structurally cannot distinguish prior hallucination from sampling noise, leading to excessive false rejections and the loss of the prior's stabilising influence.

---

End-to-End Implementation Workflow

Section 4.1.3 specifies the exact operational sequence that integrates the theoretical mechanisms into a practical training loop. The system maintains a global Support Buffer of capacity 512 storing recent prompt-performance pairs $(x_i, r_i)$ from the ongoing RL training. When a new batch of prompts arrives, the workflow is:

Step 1: Construct Context and Acquire Prior $V$. The system randomly samples 256 pairs from the Support Buffer to form the capability context $C_\pi$. It queries the frozen V0 model in batches (Query Batch Size) to obtain the predicted success probability $p \in [0, 1]$ for each target prompt. The prior $V$ is this probability linearly mapped to the reward range: $V = 2p - 1 \in [-1, 1]$. This happens once per prompt, before any rollouts.

Step 2: Cold Start Allocation. The policy model generates $k_{\text{init}} = 4$ candidate responses for each prompt in the batch, at temperature 1.0. The verifier scores each response as $r \in \{-1, 1\}$ (correct = 1, incorrect = -1). The empirical mean $\bar{v}_4$ is computed.

Step 3: Initial Bias Evaluation. The system computes $(\bar{v}_4 - V)^2$ and compares it to the noise bound $1/4 = 0.25$. If $(\bar{v}_4 - V)^2 \leq 0.25$, then $\hat{\Delta}^2_4 = 0$ — the prior is deemed reliable, and no additional compute is needed. If $(\bar{v}_4 - V)^2 > 0.25$, then $\hat{\Delta}^2_4 > 0$ — potential hallucination detected, triggering the OSLA allocator.

Step 4: Sequential OSLA Allocation. For prompts where $\hat{\Delta}^2_k > 0$, the system computes the target budget $k_{\text{target}} = 1/\sqrt{c} - 1/\hat{\Delta}^2_k$ where $c = 0.0039$ (so $1/\sqrt{c} \approx 16$). If the current $k$ is less than $k_{\text{target}}$, 2 additional rollouts are generated for that prompt, the empirical mean is updated to $\bar{v}_{k+2}$, and the bias estimate is recomputed. This loop continues until $k \geq k_{\text{target}}$ or the global stop condition triggers.

Two practical engineering constraints are applied to maintain distributed training efficiency:

• Global stop condition: If fewer than 25% of the batch's prompts still require additional rollouts, dynamic generation halts for the entire batch. This prevents a handful of slow prompts from blocking the GPU pipeline while most prompts are ready to proceed.

• Tensor parallelism padding: The allocated budget for a single dispatch is automatically padded to a multiple of 32 to prevent resource fragmentation during tensor-parallel generation. This means the actual budget may slightly exceed the theoretical $k_{\text{target}}$ to align with hardware-efficient batch sizes.

Step 5: Final Fusion and Advantage Calculation. Once dynamic allocation concludes, each prompt has $k$ rollouts where $4 \leq k \leq 16$ (the maximum imposed by $1/\sqrt{c}$). The system computes the final $\bar{v}_k$, $\hat{\sigma}^2_{\text{noise}} = 1/k$, $\hat{\Delta}^2_k = \max(0, (\bar{v}_k - V)^2 - 1/k)$, $\hat{w}_k = \hat{\Delta}^2_k / (\hat{\Delta}^2_k + 1/k)$, $\hat{\mu}^* = \hat{w}_k \bar{v}_k + (1 - \hat{w}_k)V$, and $\hat{\sigma}^* = \sqrt{1 - (\hat{\mu}^*)^2}$. The standardised advantage $A_i = (r_i - \hat{\mu}^*) / \hat{\sigma}^*$ is computed for each rollout and passed to the clipped surrogate objective (Equation 10) for the policy update.

Design choice — why $c = 0.0039$: This value sets $1/\sqrt{c} \approx 16$, which the paper describes as the maximum budget. This means the OSLA mechanism can never request more than 16 rollouts per prompt. The choice appears motivated by aligning the maximum V0.5 budget with GRPO's fixed $G = 16$ — making the worst-case compute cost of V0.5 equal to GRPO's fixed cost, while the average cost is substantially lower (since most prompts stop at $k = 4$). The paper does not explicitly justify the specific value $0.0039$ beyond this alignment, but the framework could be tuned for different cost regimes by adjusting $c$.

Design choice — why global stop at 25%: This threshold balances the statistical benefit of resolving individual prompt uncertainties against the throughput cost of straggler prompts. In a synchronous distributed training setting (32 GPUs across 4 nodes using sglang), all GPUs must wait for the slowest prompt in a batch before proceeding to the next step. If 75% of prompts have already satisfied their stopping condition but 25% are still requesting rollouts, the throughput penalty of waiting is deemed acceptable relative to the baseline quality improvement. Below 25%, the marginal benefit of resolving a small fraction of prompts is outweighed by the collective wait time.

---

The V0 Value Model: Architecture and Training

The prior $V$ that V0.5 consumes is produced by V0 (Zhang et al., 2026), and understanding its architecture is essential to understanding the nature of the prior — specifically, why it can hallucinate and what kinds of errors it is prone to.

Architecture. V0 consists of three components:

1. Semantic-Perception Backbone: A frozen LLM embedding model (Qwen3-Embedding-0.6B, $d_{\text{embed}} = 1024$) that maps each instruction (prompt $x$) into a semantic vector. This component is not fine-tuned during V0 training — it provides a fixed, general-purpose representation.

2. Residual Query Adapter: A set of 168 learnable query vectors that project the entangled semantic features from the backbone into a structured, compressed latent space. It uses a projection dimension of 6 and 3 Multi-Head Attention (MHA) layers with a residual mechanism. The adapter's purpose is to extract capability-relevant features from the raw semantic embeddings while discarding prompt-level details irrelevant to difficulty prediction.

3. Probabilistic In-Context Head: TabPFN-v2.5 (a transformer-based Bayesian inference model designed for small tabular datasets) performs single-pass Bayesian inference on the historical query-performance pairs $C_\pi = \{(x_i, r_i)\}_{i=1}^N$. Given the adapter's representation of the target query and the representations of the $N$ context queries (with their associated binary rewards), TabPFN outputs a predicted success probability $p \in [0, 1]$ for the target query.

Why this architecture: The three-component design explicitly separates semantic understanding (backbone), capability-relevant feature extraction (adapter), and in-context probabilistic inference (TabPFN head). The frozen backbone means V0 can be pre-trained once and deployed across many different policies without retraining. The query adapter is the only trainable component that processes individual prompts, keeping the training cost manageable (40 hours on 128 GPUs). TabPFN enables the model to perform Bayesian inference over the context set without requiring gradient-based optimisation at inference time — a single forward pass produces both a prediction and implicit uncertainty.

V0's In-Context Mechanism. Crucially, V0 does not learn a mapping from policy parameters to expected returns (as a traditional value model would). Instead, it learns to infer a policy's capability from a context of its past performance. The context $C_\pi$ is a set of $(x_i, r_i)$ pairs — prompts the policy has previously attempted and the binary outcomes. This means V0 estimates the expected return for a new prompt $x$ by reasoning: "Given that this policy succeeded on these kinds of problems and failed on those kinds of problems, what is its likely success probability on this new problem?" This is fundamentally an analogical reasoning task rather than a regression task, and it is what enables V0 to generalise to policies it has never seen during training (as long as it has seen sufficiently diverse policies and prompts during pre-training).

Enhanced Training Data. For the V0.5 experiments, the authors trained an enhanced V0 model on a substantially expanded dataset. They sampled GRPO training trajectories from LLMs across multiple architectural scales, capturing checkpoints throughout training (over 200 checkpoints per trajectory, approximately 20k rollouts per checkpoint). The model pool included:

• Original V0 models: Qwen3-4B-Instruct-2507, Qwen2.5-7B-Instruct, DeepSeek-R1-Distill-Qwen-1.5B
• Extended models: Full Qwen3 series (0.6B to 30B parameters), including Base, Instruct, and Thinking variants

This data synthesis produced approximately 424k training pairs (each pair consisting of a context set and a target query with its ground-truth outcome). The diversity is critical: by training on policies at different scales (0.6B to 30B), different training stages (early to late checkpoints), and different model variants (base vs. instruct vs. thinking), V0 learns to recognise capability patterns across a wide range — enabling it to estimate the expected return of a 4B policy at any point in its RL training.

Training Hyperparameters. V0 training used the same architecture and hyperparameters as the original paper: the backbone is frozen Qwen3-Embedding-0.6B, the adapter uses 168 static queries with projection dimension 6 and 3 MHA layers, the head is TabPFN-v2.5. During training, 256 query-performance pairs are randomly sampled to form each capability context. The model was pre-trained on 128 GPUs for approximately 40 hours.

Why V0 can hallucinate. The limitation that V0.5 is designed to address follows directly from V0's design. Because V0 relies on analogical reasoning from its training data, it can produce inaccurate predictions when the target prompt or the policy's current state falls outside the distribution of its training pairs. This can happen for several reasons:

• Novel prompt difficulty: The prompt may require reasoning patterns or mathematical domains not well-represented in the training data. V0's analogical reasoning may map it to a superficially similar but actually different class of problems, producing a biased estimate.

• Policy drift during RL: The policy's capability profile evolves during training in ways that may not correspond to any checkpoint V0 has seen. Intermediate policies may have erratic performance patterns (e.g., good at algebra but suddenly worse at geometry due to forgetting) that don't match any training trajectory.

• Sparse context: The 256 context pairs sampled from the Support Buffer may not be representative of the policy's true capability distribution, especially early in training when the buffer is small or when the buffer contains stale entries from much earlier checkpoints.

These hallucinations manifest as systematic errors $\Delta^2 > 0$ — V0 predicts a success probability $p$ that differs from the true $\mu_{\text{true}}$ by a non-trivial margin. It is these errors that the V0.5 hypothesis test is designed to detect and the OSLA allocator is designed to correct.

Why the prior is worth using despite hallucinations: The critical property that makes V0 valuable even with its imperfections is that its predictions are zero-variance — for a given prompt and context, V0 produces a single deterministic output. In the sparse-rollout regime where $\bar{v}_4$ has variance up to $1/4$, a prior with some unknown bias but zero variance provides a statistical anchor that, when verified by the hypothesis test, can dramatically reduce the baseline MSE. The fusion mechanism exploits this: when the test confirms the prior is reliable (discrepancy within the noise bound), the baseline inherits the prior's zero variance and the MSE drops to zero. When the test rejects the prior, the system has effectively detected its own potential error and compensates with additional data — a capability that neither pure empirical sampling nor pure value model guidance possesses.

4. Key Insights and Innovations

Innovation 1: Reframing the Generalist Prior as a Hypothesis to Be Verified, Not an Estimate to Be Trusted

The most distinctive conceptual move in this paper is not the fusion mechanism itself (shrinkage estimators are classical statistics dating to James and Stein, 1961) but the epistemological stance it takes toward the generalist value model's predictions. Prior work that incorporated external knowledge into RL baselines treated that knowledge as a direct estimate — something to be weighted, ensembled, or used as an initialisation — but ultimately trusted as correct by default. PPO trusts its value model's predictions implicitly, updating only through gradient descent without any explicit correctness verification. HA-DW (Yang et al., 2026) augments group means with history-aware anchors, but the anchors are incorporated as additive corrections, not as hypotheses subject to rejection.

V0.5 does something fundamentally different: it treats the prior $V$ as a rebuttable presumption. The system assumes the prior is correct (null hypothesis $H_0: V = \mu_{\text{true}}$) but actively gathers evidence against this assumption through live rollouts. The positive-part truncation in $\hat{\Delta}^2_k = \max(0, (\bar{v}_k - V)^2 - 1/k)$ is not merely a numerical convenience — it operationalises a genuine statistical test where the burden of proof falls on the data to overcome the prior. When the observed discrepancy falls within the noise floor, the system does not merely "weight the prior heavily" — it concludes there is no evidence the prior is wrong and assigns it full trust ($w = 0$). The prior is innocent until proven guilty.

This stance is what makes the framework "safe" in the paper's repeated framing. Safety here has a specific statistical meaning: the system bounds the damage from prior hallucinations not by limiting how much the prior can influence the baseline (a soft constraint), but by actively detecting hallucinations and quarantining them when found (a hard switch). The difference is analogous to a circuit breaker versus a current limiter — the former disconnects entirely when a fault is detected, while the latter continuously attenuates. V0.5's hypothesis test is a circuit breaker for prior bias.

The practical consequence is that V0.5 can afford to use a prior that is known to be sometimes wrong — a generalist model trained on finite, potentially mismatched data — because it has a mechanism to identify when it is wrong on a specific prompt and override it with empirical data. This lowers the bar for what constitutes an acceptable prior. Prior work that used value models required them to be accurate across the entire prompt distribution to avoid silently corrupting training; V0.5 requires only that the prior be accurate enough of the time that the hypothesis test's false rejection rate is manageable. This is a qualitatively different — and much more practical — requirement for deploying generalist models in RL pipelines.

Evidence: The difficulty-bin analysis implicit in the OSLA stopping rule (Theorem 3.6) shows this epistemological stance in action. On prompts where V0 is accurate ($\hat{\Delta}^2_k \approx 0$), the system stops early at $k = 4$ — it verifies the prior's correctness with minimal data and proceeds. On prompts where V0 hallucinates ($\hat{\Delta}^2_k \gg 0$), the budget expands toward the maximum of ~16 — the system recognises the prior's failure and gathers enough data to override it. The system does not need to know a priori which prompts are easy or hard for V0; it discovers this through testing. This is a fundamental shift from difficulty-conditioned allocation (where a separate classifier predicts difficulty) to evidence-conditional allocation (where the allocation decision is driven by the same data used to construct the baseline).

---

Innovation 2: Establishing That Bias Is Systematically Cheaper Than Variance at LLM Scale

Theorem 3.1, which bounds the policy gradient covariance by $\text{Var}_{\text{oracle}} + \Phi_{\text{score}} \cdot \text{MSE}(b) + L \cdot |\text{Bias}(b)|$, provides what is arguably the paper's most important theoretical insight — one that changes how practitioners should think about baseline design for large models. The insight is not the bound itself (bias-variance decompositions of gradient estimators are standard in the REINFORCE literature dating to Greensmith et al., 2004) but the asymmetric scaling that occurs when $\Phi_{\text{score}}$ is enormous.

For a model with $B$ parameters, $\Phi_{\text{score}} = \mathbb{E}_\pi[\|\nabla_\theta \log \pi_\theta(y|x)\|^2]$ scales roughly as $O(B)$ (each parameter contributes an independent gradient component). This means that a baseline MSE of, say, 0.1 (modest by any standard) gets multiplied by a factor proportional to billions, producing gradient variance that can dominate the optimisation signal. In contrast, the bias term is multiplied by $L$, a constant that does not scale with model size. The structural implication is that at LLM scale, variance is categorically more dangerous than bias. A method that reduces MSE by 50% while introducing a bias of 0.1 is almost certainly improving gradient stability, because the variance reduction gets multiplied by $\Phi_{\text{score}}$ while the bias penalty stays constant.

This asymmetry has been implicitly recognised in practice — GRPO's use of the group mean is explicitly justified by unbiasedness — but the paper provides the first formal quantification of how much bias can be traded for how much variance reduction. Theorem 3.4 provides the concrete answer: V0.5's fusion introduces bias bounded by $1/\sqrt{k}$ (at most 0.5 for $k = 4$), and this bias decays as $O(1/k)$ (faster than the $O(1/\sqrt{k})$ standard deviation of the empirical mean). In exchange, it reduces MSE from $\sim 1/k$ (pure empirical, when the prior is accurate but discarded) to near zero (when the hypothesis test accepts the prior). The tradeoff is massively favourable at LLM scale because of the $\Phi_{\text{score}}$ multiplier on the variance side.

This framing challenges the dominant assumption in the RLVR baseline literature that unbiasedness is paramount. Methods like ReMax (Li et al., 2024) and OPO (Hao et al., 2025) have pursued unbiased or approximately unbiased baselines, implicitly treating bias as the primary enemy. V0.5's analysis suggests this instinct, while correct for small models or low-dimensional problems, is backward for billion-parameter LLMs — a small controlled bias is vastly preferable to the variance amplification that unbiased methods cannot escape under sparse rollouts.

Evidence: Figure 3 shows the gradient norm under V0.5 remaining lower and more stable than GRPO throughout training. This is the empirical signature of the asymmetric tradeoff: V0.5's intentionally biased baseline produces cleaner gradient signals than GRPO's strictly unbiased group mean. Figure 4's entropy curves provide corroborating evidence — GRPO's high-variance gradients drive rapid entropy collapse (the policy overfits to noisy advantage signals), while V0.5's lower-noise gradients sustain exploration. These are not marginal improvements; they reflect the qualitative difference between training dynamics dominated by $\Phi_{\text{score}} \cdot \text{MSE}$ (GRPO) and dynamics where that term has been suppressed (V0.5).

---

Innovation 3: Baseline Estimation as a Sequential Decision Problem With Formal Regret Bounds

The field's approach to rollout budgeting has been almost entirely static: GRPO fixes $G$, PPO uses a single rollout per update, and even adaptive methods like best-of-N sampling treat the budget as a hyperparameter. V0.5 reframes baseline estimation as an online sequential decision problem where each additional rollout is an action whose expected value can be computed in real-time.

This is a genuine conceptual shift, not an optimisation trick. The key intellectual move is defining the total risk $R(k) = \widehat{\text{MSE}}(k) + c \cdot k$ — a loss function that trades off estimation quality against compute cost in a single currency — and then deriving a stopping rule by comparing the marginal reduction in $\widehat{\text{MSE}}$ to the marginal cost $c$. The One-Step-Look-Ahead (OSLA) structure is critical because it makes the decision myopically optimal: at each step, the system asks only whether one more rollout is expected to help, and stops when the answer is no. This avoids the full dynamic programming complexity of optimising over the entire future trajectory while still providing formal guarantees (the $O(c)$ regret bound in Appendix A.7).

What makes this more than a standard sequential testing procedure (Wald, 2004, provides the classical framework) is the closed-loop coupling between the bias estimate and the stopping rule. The stopping threshold $k \geq 1/\sqrt{c} - 1/\hat{\Delta}^2_k$ depends on $\hat{\Delta}^2_k$, which itself depends on the rollouts collected so far. This creates a feedback loop: as more data is collected, $\hat{\Delta}^2_k$ updates, which changes the stopping threshold, which determines whether more data is collected. A prior that initially appears wrong may be vindicated with a few more samples (if the initial discrepancy was due to random noise), causing $\hat{\Delta}^2_k$ to shrink, the threshold to drop, and the system to halt early. Conversely, a prior that initially appears correct may be exposed as wrong with more data, causing the budget to expand. The system updates its own stopping criterion as it gathers evidence.

The practical significance goes beyond compute savings (though the paper demonstrates that V0.5 with $k_{\text{init}} = 4$ outperforms GRPO with $G = 16$). It establishes that prompt-level compute allocation can be automated via statistical principles rather than heuristics. The cost factor $c$ is the single tunable parameter that controls the aggressiveness of the allocation — set $c$ low, and the system spends more compute per prompt; set $c$ high, and it is more frugal. This provides a principled interface for deploying the same system in different cost regimes (batch processing vs. interactive serving, GPU-rich vs. GPU-poor environments) without changing the underlying algorithm.

Evidence: Figure 1 shows V0.5 with OSLA enabled ($k_{\text{init}} = 4$) substantially outperforming GRPO ($G = 16$) across all six benchmarks. The key comparison is not just the accuracy gap but the compute efficiency: V0.5 achieves higher accuracy with an average rollout budget well below 16 (most prompts stop at $k = 4$ when V0 is accurate, only hallucination-triggering prompts expand to the maximum). Figure 5 isolates the fusion mechanism without OSLA: at fixed $k = 4, 8$, performance already matches or exceeds GRPO at $G = 16$, demonstrating that the prior alone provides substantial variance reduction. The OSLA mechanism adds the adaptive scheduling on top — it determines which prompts need more than the minimum budget, rather than giving all prompts the same allocation.

---

Innovation 4: Diagnosing and Quantifying the Catastrophic Interaction Between Extreme Sparsity and Discrete Reward Spaces

Appendix A.8 contains what might be the paper's most practically instructive theoretical result, even though it is presented as a "base group size analysis" rather than a main contribution. The analysis reveals that the hypothesis test's validity depends on a structural condition — $\text{Threshold}(k) \geq \text{Gap}(k)$, or $1/\sqrt{k} \geq 2/k$ — that fails catastrophically for $k < 4$ in the binary reward setting $\{-1, 1\}$. This is not an empirical observation about performance degradation; it is a mathematical proof that the framework structurally breaks when $k = 1, 2, 3$.

The diagnostic value of this result extends beyond V0.5. It identifies a general phenomenon: any method that tests for distributional shift using discrete observations must respect the quantisation gap of the observation space. With binary rewards, the empirical mean can only take $k+1$ discrete values separated by gaps of $2/k$. A hypothesis test with a continuous rejection boundary will inevitably misclassify sampling noise as systematic bias when the boundary is finer than the observation granularity. This affects GRPO as well — although GRPO does not perform explicit hypothesis testing, its baseline (the group mean) is implicitly "testing" whether individual rewards deviate from the group average, and with $G = 1$ or $G = 2$, the same quantisation pathology would manifest as pathological advantage sign flips (a correct answer in a group of 2 where the other is also correct receives zero advantage; the same correct answer in a group where the other is wrong receives positive advantage — a purely noise-driven outcome).

The paper's Figure 5 confirms this diagnosis empirically: V0.5 fails to converge at $k = 1, 2$ (training collapses) but succeeds at $k = 4, 8$. The boundary at $k = 4$ is not an arbitrary engineering choice — it is where the gap $2/4 = 0.5$ exactly equals the noise tolerance threshold $1/\sqrt{4} = 0.5$, providing the minimum statistical buffer needed for the test to function. Below this threshold, the system exhibits exactly the predicted pathology: normal sampling variation frequently triggers false rejections of the prior, causing the system to discard a stabilising influence and rely on an extremely noisy empirical mean, which the gradient amplification from Theorem 3.1 renders catastrophic.

This result has immediate practical implications for the design of sparse-rollout RL systems beyond V0.5: binary-reward RLVR should never use group sizes below 4 for empirical baseline estimation, regardless of the specific algorithm. Below this threshold, the baseline is not merely high-variance — it is structurally incapable of distinguishing signal from noise. This is a stronger claim than the standard "small group sizes increase variance" advice, and it follows directly from the discrete structure of the reward space, making it applicable to any method that computes empirical statistics over binary outcomes.

Evidence: Figure 5, bottom rows: the $k = 1$ and $k = 2$ curves flatline near zero accuracy across all benchmarks, while $k = 4$ and $k = 8$ curves track closely with GRPO at $G = 16$ or outperform it. The failure at $k = 1, 2$ is not gradual degradation — it is a sharp phase transition at $k = 4$, exactly as the structural analysis predicts.

---

Innovation 5: Decoupling Value Estimation From Policy Training Without Sacrificing Adaptivity

The paper's most architecturally significant contribution is demonstrating that a completely frozen, independently trained value model can serve as an effective baseline for RL training — but only when wrapped in a verification layer that compensates for its lack of synchronous adaptation. This resolves a tension that has shaped the RLVR literature since PPO: either you co-train the value model with the policy (high cost, potential OOD bias) or you eliminate it entirely (GRPO's approach, high variance). V0.5 shows there is a third option: deploy a pre-trained generalist model whose predictions are verified against live data, with verification failures triggering targeted compute expenditure rather than model retraining.

The significance of this decoupling is practical, not merely theoretical. Training a synchronous value model for a 4B-parameter policy roughly doubles the GPU memory and compute requirements. V0, once pre-trained (40 hours on 128 GPUs), incurs only inference cost during RL — a single forward pass per prompt through an embedding model and a lightweight adapter. The verification cost (4 initial rollouts) is already incurred by any method that samples from the policy, so the prior adds negligible overhead. This means V0.5 achieves the variance reduction benefits of a value model without the training cost — a genuinely new point in the design space.

However, the paper is careful not to claim that V0 entirely replaces the need for adaptation. The verification layer (hypothesis test + OSLA allocation) is the mechanism that provides adaptivity — it detects when the frozen model's predictions are stale or wrong and compensates by gathering more empirical data. This is a form of runtime adaptation rather than training-time adaptation: the value model itself never changes, but the system's reliance on it changes per-prompt based on evidence. This is a fundamentally different approach to handling distribution shift than the standard solution of retraining the value model on the policy's current outputs.

The limitation is that V0's prior quality bounds the system's efficiency. If V0 were perfectly accurate on all prompts, V0.5 would achieve the theoretical minimum variance with only $k = 4$ rollouts everywhere. If V0 were completely wrong on all prompts, V0.5 would degenerate to pure empirical sampling at the maximum budget (~16) — no worse than GRPO but no better. The practical value lies between these extremes: V0 is accurate enough on enough prompts that the average compute budget is well below the maximum, while the verification layer prevents the inaccurate predictions from corrupting training.

Evidence: The expanded V0 training described in Section 4.1.2 — using 424k training pairs across the full Qwen3 series (0.6B to 30B) — is what makes the prior accurate enough for the verification layer to be a net win. The paper does not ablate V0 quality directly (e.g., comparing a weaker V0 to the enhanced version), but the overall results (Figure 1) demonstrate that the enhanced V0 + V0.5 combination outperforms both GRPO and DAPO by substantial margins. The gap between V0.5 and GRPO is the empirical measure of how much the frozen prior contributes beyond what pure empirical sampling can achieve with matched total compute.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All experiments use six mathematical reasoning benchmarks: AIME 2024, AIME 2025, Olympiad Bench (He et al., 2024), MATH500 (Hendrycks et al., 2021), Minerva Math (Lewkowycz et al., 2022), and AMC 2023 (Math-AI, 2025). These constitute the standard evaluation suite for post-training reasoning RL, spanning competition-level (AIME, Olympiad Bench, AMC) and general mathematical problem-solving (MATH500, Minerva Math) domains. The base policy is fine-tuned on DAPO-Math-17k (Yu et al., 2025) prior to RL training.

• Base model(s). All RL training uses Qwen3-4B-Instruct-2507 as the base policy model, chosen as a mid-scale representative of modern LLM capabilities. The policy is first fine-tuned on the DAPO-Math-17k dataset before entering the RL phase. For the V0 prior, an enhanced generalist value model is pre-trained separately (128 GPUs, ~40 hours) on GRPO training trajectories sampled from LLMs spanning the Qwen3 series (0.6B to 30B parameters, including Base, Instruct, and Thinking variants), Qwen2.5-7B-Instruct, and DeepSeek-R1-Distill-Qwen-1.5B, yielding approximately 424k training pairs covering diverse policy scales and training stages.

• Metrics. The primary metric is mean@16 accuracy — the average dataset accuracy when sampling 16 responses per prompt at evaluation time. All evaluation uses rule-based reward verification for correctness (no formatting rewards), with inference temperature 1.0 and Top-p of 0.7. A response is scored as correct (reward = 1) or incorrect (reward = -1) based on exact answer matching against ground truth. Training progress is tracked via accuracy on held-out benchmarks, policy gradient norm (Figure 3), and policy entropy (Figure 4).

• Baselines. Two primary baselines are compared:

  1. GRPO (Shao et al., 2024; DeepSeek-AI et al., 2025): Uses global prompt batch size 512, fixed group size G = 16, KL penalty coefficient 0.001, and the intra-group empirical mean as the advantage baseline. No value model is used.
  2. DAPO (Yu et al., 2025): Same batch size and group size (512 prompts × G = 16), with DAPO-specific advantage filtering and asymmetric clipping. The rollout generation batch size is 512 × 3. Both baselines operate under identical hardware (4 nodes, 32 GPUs total, sglang engine) and software frameworks. The product of global prompt batch size and group size per prompt is held constant across all settings to ensure fair per-step computational comparison.

• Generation budget / compute accounting. Compute is measured in number of rollouts per prompt. For GRPO and DAPO, this is fixed at G = 16. For V0.5, the initial budget is $k_{\text{init}} = 4$, with the OSLA mechanism dynamically expanding up to a maximum of approximately 16 rollouts per prompt (controlled by the cost factor $c = 0.0039$ giving $1/\sqrt{c} \approx 16$). The global prompt batch size is adjusted so that the product (batch size × group size) remains constant across all methods, ensuring matched per-step computational overhead. The V0 prior's inference cost (a single forward pass through the frozen V0 model per prompt) is marginal compared to the cost of even a single rollout from the 4B policy model.

• Cross-validation / statistical protocol. No cross-validation over dataset splits is applied for strategy selection (unlike the reference paper). The V0.5 framework's hyperparameters ($c = 0.0039$, $k_{\text{init}} = 4$, support buffer size 512, support batch size 256) are fixed across all experiments. The V0 model is pre-trained once on the 424k training pairs and frozen. All RL training runs are single-seed based on the presented figures. The paper does not report confidence intervals or standard deviations across multiple seeds. Evaluation accuracy is reported as the mean@16 score at the final training step and at intermediate checkpoints.

---

Main Quantitative Results

Overall Performance and Convergence Speed (Figure 1)

The central result appears in Figure 1, which displays accuracy vs. training steps for V0.5 (with full OSLA dynamic budget allocation, $k_{\text{init}} = 4$) against GRPO (G = 16) and DAPO (G = 16) across all six benchmarks.

Headline numbers. The paper states that V0.5 "achieves faster convergence and some over 10% performance improvement" compared to GRPO and DAPO. The curves in Figure 1 visually support this claim: V0.5's accuracy trajectory consistently rises faster and plateaus higher than both baselines on all six benchmarks.

Convergence speed. On AIME 2024, AIME 2025, and Olympiad Bench, V0.5 reaches its final accuracy plateau substantially earlier (in fewer training steps) than GRPO and DAPO. The precise step counts at which each method plateaus are not numerically reported, but the visual gap in convergence rate is clear across the first ~50–100 training steps where V0.5's accuracy rises more steeply.

Final accuracy improvement. The "over 10%" figure refers to absolute accuracy improvement at the final training step. The paper does not provide a per-benchmark breakdown table of final accuracies, making it difficult to verify the over-10% claim on each individual benchmark from the text alone. Figure 1 shows qualitatively that V0.5's final accuracy sits above both baselines on all six benchmarks, with the gap appearing largest on AIME 2024, AIME 2025, MATH500, and AMC 2023. On Minerva Math, the gap appears somewhat smaller but V0.5 still maintains an advantage. On Olympiad Bench, V0.5 and DAPO converge to similar plateaus but V0.5 reaches it faster.

DAPO vs. GRPO. DAPO generally outperforms GRPO in final accuracy, consistent with prior findings. However, V0.5 surpasses both. The relative ordering is consistently V0.5 > DAPO > GRPO on all benchmarks at the final training step.

Computational efficiency caveat. The claim of over 10% improvement uses V0.5 with an average rollout budget per prompt well below 16 (since most prompts stop at $k = 4$ when V0 is accurate), compared to GRPO's fixed $G = 16$. The paper does not report the average rollout budget achieved by the OSLA allocator, making it impossible to quantify the exact compute multiplier. The maximum budget is ~16 (matching GRPO), so V0.5's per-step compute cost is at most equal to GRPO's and typically lower. The improvement is therefore both an accuracy gain and a compute efficiency gain, but the paper presents only the accuracy dimension numerically.

---

Gradient Stability: Norm and Variance Reduction (Figure 3)

Figure 3 plots the evolution of the policy gradient norm during training for V0.5 vs. GRPO. The paper claims this validates Theorem 3.1's prediction that baseline MSE reduction translates to gradient variance suppression.

Observation. V0.5 maintains a lower and more stable gradient norm than GRPO throughout training. The GRPO curve shows larger fluctuations and generally higher magnitude, consistent with the theoretical prediction that GRPO's empirical mean baseline (high MSE under G = 16, and especially relative to what an even noisier G = 4 would produce) amplifies gradient variance through the $\Phi_{\text{score}}$ multiplier. V0.5's curve is visibly smoother with smaller oscillations.

Interpretation. The paper attributes this to the Shrinkage Fusion mechanism "strategically trading a strictly bounded minor bias for a reduction in variance." When the prior is verified as accurate (discrepancy within noise bound), the baseline inherits the prior's zero variance, producing a near-oracle advantage signal. When the prior is rejected, the fusion weight shifts toward the empirical mean but with more samples than the minimum (due to OSLA allocation), still producing lower variance than a fixed small group size. The result is gradient updates that are directionally more consistent across steps.

What is not shown. The gradient norm is plotted without units on the y-axis, making quantitative comparison difficult. The paper does not report the actual MSE of the baseline estimator during training (which would directly validate Theorem 3.2 and the MSE decomposition), nor does it report the measured bias of the fused estimator (which would validate Theorem 3.4's $1/\sqrt{k}$ bound). The gradient norm reduction is consistent with the theory but does not constitute a direct empirical validation of the specific mathematical bounds.

---

Policy Entropy Maintenance and Exploration (Figure 4)

Figure 4 shows the evolution of policy entropy over training steps for V0.5 vs. GRPO.

Observation. GRPO's policy entropy decays rapidly, especially in the early-to-mid training phase. V0.5 sustains a higher entropy level throughout training, with a much slower decay rate.

Interpretation. The paper explains this through the gradient noise mechanism: GRPO's high-variance baseline produces noisy advantage signals that act as a random forcing term on the policy distribution, pushing it toward low-entropy, over-confident modes (a form of "noise-induced collapse"). V0.5's lower-variance gradients provide cleaner learning signals that allow the policy to maintain broader exploration — the policy is reinforced for genuine capability improvements rather than for random fluctuations in the baseline.

Significance. This is an important qualitative difference. Rapid entropy collapse is a known failure mode in GRPO training that limits final performance by trapping the policy in local optima. V0.5's ability to sustain entropy without explicit entropy regularisation (the paper uses no KL penalty during V0.5 training) suggests that baseline quality alone is sufficient to preserve exploration in mathematical reasoning tasks.

What is not shown. The paper does not report whether the sustained entropy translates to higher solution diversity (e.g., pass@k metrics). It also does not compare V0.5's entropy against DAPO, which might have different entropy dynamics due to its asymmetric clipping and advantage filtering.

---

Performance Under Extreme Sparsity Without Dynamic Allocation (Figure 5)

Figure 5 isolates the Shrinkage Fusion mechanism by testing V0.5 without OSLA dynamic allocation — using fixed group sizes $k \in \{1, 2, 4, 8\}$ — and comparing against standard GRPO with $G = 16$. To ensure fair comparison, prompt batch sizes are adjusted to maintain constant per-step computational overhead (batch size × group size held constant).

Headline result. With $k = 4$ or $k = 8$, V0.5 (fusion only, no dynamic allocation) matches or outperforms GRPO at G = 16 across all six benchmarks. This is the strongest evidence that the prior alone provides substantial variance reduction — even when spending only 4 rollouts per prompt (25% of GRPO's compute per prompt, since $4/16 = 0.25$), the fused baseline produces superior training outcomes.

$k = 4$ performance. Across AIME 2024, AIME 2025, Olympiad Bench, MATH500, Minerva Math, and AMC 2023, the $k = 4$ V0.5 curve closely tracks GRPO (G = 16) and in several cases (AMC 2023, MATH500) visually outperforms it. The paper does not provide numerical final accuracies for this comparison, but the curves show consistent parity or advantage.

$k = 8$ performance. With $k = 8$, V0.5 outperforms GRPO (G = 16) more clearly, achieving higher final accuracy and faster convergence across most benchmarks. The gap is most pronounced on AMC 2023 and MATH500.

$k = 1, 2$ failure. At $k = 1$ and $k = 2$, V0.5 fails to converge — accuracy flatlines near zero across all benchmarks. This is the empirical confirmation of the structural analysis in Appendix A.8: with binary rewards in $\{-1, 1\}$, the hypothesis test's tolerance radius ($1/\sqrt{k}$) is smaller than the discrete quantisation gap ($2/k$) for $k < 4$, causing normal sampling variation to frequently trigger false rejections of the prior. The system then discards the stabilising prior in favour of an extremely noisy empirical mean (with only 1 or 2 samples), and gradient variance explodes (via Theorem 3.1's $\Phi_{\text{score}} \cdot \text{MSE}$ amplification), preventing any learning.

Comparison across benchmarks. The relative ordering of V0.5 ($k = 4$), V0.5 ($k = 8$), and GRPO ($G = 16$) varies slightly across benchmarks but follows a consistent pattern: $k = 8$ is generally best, $k = 4$ is competitive, and the gap between V0.5 ($k = 4$) and GRPO tends to widen in later training (suggesting sustained benefits from lower-variance gradients). On Minerva Math, all three curves converge to similar plateaus. On AIME 2024 and 2025, V0.5 ($k = 8$) shows a clear advantage.

What this comparison does not show. Figure 5 compares fixed sparse rollouts plus fusion against GRPO's fixed G = 16. It does not compare against GRPO with the same small group sizes ($G = 4, 8$), which would isolate the benefit of the prior from the benefit of the group size. Such a comparison would directly answer: "Is V0.5 at $k = 4$ better than GRPO at $G = 4$?" The paper's implicit claim is that GRPO at $G = 4$ would perform worse (due to high variance without any prior), but this is not shown experimentally.

---

V0.5 With OSLA vs. Fixed Sparsity (Figures 1 vs. 5, Implicit Comparison)

The paper presents V0.5 with full OSLA dynamic allocation in Figure 1 ($k_{\text{init}} = 4$, dynamic expansion up to ~16) and V0.5 with fixed sparsity in Figure 5 ($k = 4, 8$ fixed). Comparing across these figures (acknowledging that Figure 1 also includes DAPO while Figure 5 does not):

• V0.5 with OSLA (Figure 1) achieves higher final accuracy than V0.5 with fixed $k = 4$ (Figure 5) on most benchmarks, though the improvement is modest. This is expected: OSLA allocates additional rollouts to prompts where the prior is unreliable, improving the baseline for those prompts without wasting compute on prompts where the prior is already accurate.
• The gap between V0.5 (OSLA) and V0.5 (fixed $k = 8$) is narrower, suggesting that the OSLA mechanism recovers most of the benefit of a larger fixed budget while spending less compute on average.
• The paper does not report the average rollout budget per prompt achieved by OSLA, which is the key efficiency metric. Without this, one cannot quantify how much compute OSLA saves relative to the fixed $k = 8$ or $G = 16$ baselines.

---

Ablation Studies and Robustness Checks

The paper contains fewer explicit ablation experiments than might be expected. The key ablations are distributed across the main results and the appendix, often implicit in the comparisons already described. I identify the explicit and implicit ablations below.

Ablation of OSLA dynamic allocation (Figures 1 vs. 5): The comparison between V0.5 with OSLA (Figure 1, $k_{\text{init}} = 4$, dynamic) and V0.5 with fixed $k = 4$ (Figure 5) constitutes an ablation of the dynamic budget allocation mechanism. The OSLA version achieves higher final accuracy, validating that selectively expanding the rollout budget for prompts where the prior is unreliable provides measurable benefit over a uniform minimal budget. However, the gap is not dramatic — the fixed $k = 4$ fusion already performs well — suggesting that the fusion mechanism contributes more of the total gain than the dynamic allocation.

Ablation of group size (Figure 5, $k = 1, 2, 4, 8$): This sweeps the fixed rollout budget and validates the theoretical $k_{\text{min}} = 4$ bound from Appendix A.8. The key finding is the phase transition at $k = 4$: convergence fails completely at $k = 1, 2$ but succeeds at $k = 4, 8$. This is an unusually sharp ablation result that provides strong evidence for the discrete gap analysis.

Ablation of the prior (implicit in baseline comparison): The comparison of V0.5 against GRPO constitutes an implicit ablation of the V0 prior itself. Since both methods use the same base policy and same verifier rewards, the performance gap isolates the contribution of the prior-mediated baseline estimation. The paper does not run an explicit "V0.5 without the prior" condition (which would be equivalent to GRPO at the same group size), but the fixed sparsity results in Figure 5 provide partial evidence: at $k = 4$, V0.5 outperforms GRPO at $G = 16$, and GRPO at $G = 4$ would almost certainly perform worse than GRPO at $G = 16$ (since variance scales as $1/G$). So the gap between V0.5 at $k = 4$ and a hypothetical GRPO at $G = 4$ would be even larger than what Figure 5 shows.

Baseline configurations as ablation: The main results already compare against two distinct baselines (GRPO, DAPO) and two V0.5 variants (fixed sparsity, OSLA). The fixed sparsity results additionally vary group size (1, 2, 4, 8). This provides reasonable coverage of the design space without dedicated ablation sections.

Missing ablations that would strengthen the paper:

1. V0 quality ablation: The paper uses an enhanced V0 trained on 424k pairs across the Qwen3 series. No results are reported for the original V0 (trained on fewer models) or for a degraded V0 (e.g., trained on less data or fewer model scales). Such an ablation would quantify how prior quality affects downstream RL performance — a critical question for practitioners deciding whether to invest in training their own generalist value model.

2. Cost factor $c$ sensitivity: The OSLA stopping rule uses $c = 0.0039$, which sets the maximum budget to ~16 rollouts. The paper does not sweep $c$ to show how performance varies with the cost tolerance. A sweep (e.g., $c = 0.01$ giving max ~10 rollouts, $c = 0.001$ giving max ~32) would demonstrate the robustness of the framework to this parameter and provide guidance for practitioners with different compute budgets.

3. Support buffer size and context set size: The system uses a support buffer of 512 and samples 256 pairs for the V0 context. No ablation varies these numbers. The context set size in particular affects V0's prediction quality — larger contexts provide more information but increase inference cost.

4. Initial group size $k_{\text{init}}$: The paper fixes $k_{\text{init}} = 4$ based on the theoretical analysis in Appendix A.8 and does not test $k_{\text{init}} = 6, 8$ or other values. Would starting with more rollouts improve performance by reducing false rejections, or does the overhead outweigh the benefit?

5. GRPO at matched group sizes: Figure 5 compares V0.5 at $k = 4, 8$ against GRPO at $G = 16$. Comparing against GRPO at $G = 4, 8$ would directly isolate the prior's contribution at equal compute.

6. Prior-only and empirical-only endpoints: A direct comparison of "pure prior as baseline" ($w = 0$ always) and "pure empirical mean as baseline" ($w = 1$ always) against the adaptive fusion would demonstrate that the fusion mechanism is genuinely necessary, not just that V0 provides useful information.

7. Multiple training seeds: All figures show single training runs. Without error bars or multiple seeds, it is impossible to assess whether the reported gaps are statistically significant or within run-to-run variation. RL training is notoriously sensitive to random seeds, and the 10% absolute accuracy improvement could in principle be partially attributable to seed effects.

8. Compute cost reporting for OSLA: The paper does not report the average number of rollouts per prompt under OSLA allocation, the distribution of stopping times, or the frequency with which the hypothesis test rejects the prior. These metrics are essential for evaluating the practical efficiency of the framework.

---

Critical Assessment

Claim 1: V0.5 significantly outperforms GRPO and DAPO, achieving faster convergence and over 10% performance improvement.

What was tested: Figure 1 shows V0.5 (with OSLA, $k_{\text{init}} = 4$) against GRPO ($G = 16$) and DAPO ($G = 16$) on six mathematical reasoning benchmarks using a single base model (Qwen3-4B-Instruct-2507).

Assessment: Supported with qualifications. The curves in Figure 1 visually demonstrate superior convergence speed and higher final accuracy for V0.5 across all six benchmarks. The "over 10%" figure is stated in the abstract and introduction but is not broken down per benchmark in the text. The gap appears largest on AMC 2023 and MATH-500, somewhat smaller on AIME benchmarks and Olympiad Bench, and smallest on Minerva Math. The single-seed nature of the experiments means the magnitude of improvement is uncertain — a multi-seed study with confidence intervals would substantially strengthen this claim.

Qualification: The improvement is measured against baselines that themselves use fixed compute budgets. GRPO at $G = 16$ is a strong baseline, but the comparison is not fully compute-matched because V0.5's average rollout budget is unknown (and likely less than 16). The paper's claim of "over 10% performance improvement" should be understood as "over 10% absolute accuracy improvement at lower average compute cost," but the compute savings are not quantified. This makes it a claim about both accuracy and efficiency, only one dimension of which is numerically reported.

What would strengthen this: Multi-seed experiments with standard deviations; a per-benchmark table of final accuracy for V0.5, GRPO, and DAPO; reporting the average rollout budget per prompt under OSLA.

---

Claim 2: The Empirical Shrinkage Fusion mechanism minimises baseline MSE while bounding induced bias.

What was tested: Figure 3 (gradient norm) and Figure 4 (entropy) provide indirect evidence. The fixed sparsity results (Figure 5) demonstrate that the fusion mechanism with $k = 4, 8$ works, which is consistent with MSE reduction but does not directly measure MSE.

Assessment: Supported by indirect evidence, but the core MSE and bias claims are theoretically proven (Theorems 3.2–3.4) rather than empirically validated. The paper does not measure or report baseline MSE or bias during training. Figure 3 shows lower gradient norm, which is consistent with lower baseline MSE (via Theorem 3.1's bound), but is not a direct measurement. The theoretical bounds ($|Bias(\hat{\mu}^*)| \leq 1/\sqrt{k}$, $\text{MSE}(w)$ decomposition) are proven mathematically but never verified against empirical estimates during actual training.

What would strengthen this: Logging $\hat{\Delta}^2_k$ (the estimated prior bias) and $\widehat{\text{MSE}}(k)$ (the empirical baseline MSE) during training, and comparing them against the theoretical bounds. Showing that the actual bias remains within the $1/\sqrt{k}$ envelope and that the actual MSE approaches the theoretical minimum would directly validate the core claims.

---

Claim 3: Sequential OSLA Allocation enables on-demand compute scheduling that balances statistical precision with marginal costs.

What was tested: Figure 1 (V0.5 with OSLA) vs. Figure 5 (V0.5 with fixed sparsity) implicitly tests this by showing that OSLA improves over fixed $k = 4$, but the improvement is not isolated as a dedicated ablation. The OSLA mechanism's behaviour (how often it expands the budget, how many additional rollouts are typically allocated, whether the stopping rule triggers appropriately) is not empirically characterised.

Assessment: Supported with significant gaps. The theoretical derivation of the stopping rule (Theorem 3.6) and the regret bound (Appendix A.7) provide mathematical justification, but the empirical validation is thin. The paper does not report:

• The average number of rollouts per prompt under OSLA
• The distribution of stopping times across prompts
• How often the hypothesis test rejects the prior ($\hat{\Delta}^2_k > 0$)
• Whether the stopping decisions correlate with actual prior accuracy (does the system correctly identify when V0 is wrong?)
• The $O(c)$ regret bound is proven but not empirically measured

Without these metrics, the reader cannot assess whether OSLA is doing what the theory claims — adaptively expanding compute for hard/OOD prompts while saving compute for easy ones — or whether the improvement over fixed sparsity is due to some other effect (e.g., simply having a larger average budget).

What would strengthen this: A detailed analysis of OSLA behaviour: histograms of stopping times, correlation between $\hat{\Delta}^2_k$ and actual prior error, the fraction of prompts that trigger additional allocation, and the achieved MSE as a function of stopping time.

---

Claim 4: The framework safely integrates a prior that may hallucinate, neutralising high variance while actively safeguarding against prior bias.

What was tested: The fixed sparsity results (Figure 5, $k = 4, 8$) vs. the $k = 1, 2$ failure mode. The hypothesis test's safety is demonstrated by the phase transition at $k = 4$ (Appendix A.8). The OSLA results (Figure 1) demonstrate that the full system works.

Assessment: Supported with qualifications. The strongest evidence for safety is the sharp phase transition at $k = 4$ (Figure 5) — when the test has sufficient statistical power, the system succeeds; when it doesn't, it fails catastrophically. This validates the structural analysis and demonstrates that the hypothesis test is genuinely doing work. The success of the full system across six benchmarks (Figure 1) shows that the prior's hallucinations are not severe enough to derail training, which is evidence that the verification layer is effective.

However, the paper does not directly demonstrate that the system actually detects and isolates prior hallucinations. There is no experiment where a deliberately corrupted prior (e.g., V0 predictions artificially biased by a known amount) is used, and the system's response (detection via $\hat{\Delta}^2_k$, budget expansion, and eventual override) is measured. Without such a controlled experiment, the "safeguarding against hallucinations" claim rests on the theoretical properties of the hypothesis test rather than on empirical demonstration that the test works as designed when confronted with known hallucinations.

---

Overarching Strengths

1. Multi-benchmark evaluation: Six diverse mathematical reasoning benchmarks provide reasonable coverage of the domain. The consistency of V0.5's advantage across all six benchmarks is a strong signal that the method is robust to task variation within the mathematical reasoning domain.

2. Phase transition at $k = 4$: The sharp failure at $k = 1, 2$ and success at $k = 4$ in Figure 5 is one of the most convincing results in the paper. It cleanly validates the theoretical analysis in Appendix A.8 and provides a clear, actionable guideline ($k_{\text{min}} = 4$ for binary-reward RLVR).

3. Fixed sparsity results match or beat GRPO at higher compute: V0.5 at $k = 4$ (Figure 5) matching GRPO at $G = 16$ is a strong result because it uses 4× fewer rollouts per prompt. Even with the fixed batch size × group size constraint making the wall-clock comparison subtle, the per-prompt sample efficiency improvement is clear.

4. Gradient norm and entropy evidence: Figures 3 and 4 provide mechanistic insight that supports the theoretical narrative — V0.5 does not just achieve higher accuracy but does so through the specific pathway (lower gradient variance, sustained exploration) that the theory predicts.

---

Overarching Weaknesses

1. Single model family, single domain: All experiments use Qwen3-4B-Instruct-2507 on mathematical reasoning benchmarks. The paper makes no claims about generalisation to other model families (e.g., LLaMA, DeepSeek-R1), other model scales (the framework should work better at larger scales where $\Phi_{\text{score}}$ is larger, but may be less necessary if larger models already have higher pass@1), or other domains (code generation, scientific QA, multi-modal reasoning). The reader cannot assess whether V0.5's benefits are specific to the Qwen3 architecture's training dynamics, to mathematical reasoning tasks, or to the particular scale of the base model.

2. No seed variation: All figures show single training runs. RL training with LLMs is notoriously high-variance; the lack of error bars or multiple seeds means the reported accuracy gaps could be partially attributable to run-to-run variation. The over-10% claim is difficult to evaluate without knowing the typical variance across training runs for the same configuration.

3. The V0 model is not described in sufficient detail to replicate: Section 4.1.2 provides architecture details but the training procedure (data construction across 424k pairs, exact training objective, validation methodology) is sketched rather than specified. A practitioner wishing to reproduce the results would need to consult the original V0 paper (Zhang et al., 2026) and fill in substantial gaps. The paper does not release the trained V0 model weights.

4. Missing efficiency metrics for OSLA: The core value proposition of V0.5 is compute-efficient baseline estimation, yet the paper never quantifies the actual compute used per prompt under OSLA. The average rollout budget, the distribution of stopping times, and the fraction of prompts requiring additional allocation are all absent. This makes it impossible to evaluate the framework's efficiency claims numerically.

5. No comparison against GRPO at matched sparse group sizes: Figure 5 compares V0.5 at $k = 4$ against GRPO at $G = 16$, not against GRPO at $G = 4$. This is the most direct ablation of the prior's contribution — what does the prior add beyond what the empirical mean alone can do at the same sample size? The paper implicitly assumes GRPO at $G = 4$ would perform poorly, which is reasonable given the $1/G$ variance scaling and the structural issues identified in Appendix A.8, but it is not demonstrated.

6. The $c = 0.0039$ choice is not justified empirically: The cost factor controls the maximum rollout budget and the aggressiveness of the OSLA stopping rule. The paper selects $c = 0.0039$ to make the maximum budget ~16 (matching GRPO's $G = 16$), but does not explore whether this is optimal. A smaller $c$ (allowing larger budgets) might further improve V0.5's accuracy at higher compute cost; a larger $c$ (restricting budgets) might provide similar accuracy with even greater efficiency. The absence of a sensitivity analysis around this parameter is a significant gap for a method whose central contribution is principled budget allocation.

7. No direct measurement of baseline MSE or bias: For a paper whose theoretical contribution centres on MSE minimisation and bias bounding (Theorems 3.1–3.4), the absence of empirical MSE and bias measurements is a notable gap. The gradient norm reduction (Figure 3) is consistent with the theory but does not directly validate the claimed decomposition or bounds.

8. The "over 10% improvement" claim is not broken down: The abstract and introduction state this figure without specifying whether it is the average across benchmarks, the maximum across benchmarks, or something else. A per-benchmark table of final accuracies with confidence intervals would make this claim verifiable and allow readers to assess which types of reasoning tasks benefit most.

---

What the Experiments Demonstrate vs. What They Claim

What is demonstrated convincingly:

• A frozen generalist value model (V0) can serve as a useful prior for advantage baseline estimation in RLVR, improving training outcomes when combined with sparse empirical rollouts via a convex fusion mechanism.
• The fusion mechanism enables a policy model to achieve comparable or superior performance with fewer rollouts per prompt than standard GRPO (Figure 5: V0.5 at $k = 4$ vs. GRPO at $G = 16$).
• There exists a minimum viable group size ($k = 4$ for binary rewards) below which the fusion mechanism fails structurally, validating the theoretical analysis of discrete quantisation effects.
• The fused baseline produces lower and more stable gradient norms during training (Figure 3) and sustains higher policy entropy (Figure 4), consistent with the theoretical variance reduction argument.
• Adding dynamic budget allocation (OSLA) on top of fixed-sparsity fusion further improves performance (Figures 1 vs. 5), though the magnitude of this improvement is not isolated.

What is claimed but not rigorously demonstrated:

• That the hypothesis test actively detects prior hallucinations and triggers appropriate corrective action (no controlled hallucination experiment exists).
• That the OSLA mechanism achieves near-optimal compute scheduling with bounded regret (the $O(c)$ regret bound is proven theoretically but not measured empirically).
• That the baseline MSE is actually reduced by the amount predicted by the theoretical decomposition (no MSE measurements are reported).
• That the induced bias remains within the $O(1/\sqrt{k})$ bound in practice (no bias measurements are reported).
• That the "over 10% improvement" is statistically reliable across multiple training seeds (single-seed results).
• That the framework's benefits generalise beyond the specific model (Qwen3-4B), domain (math reasoning), and V0 training procedure used.

6. Limitations and Trade-offs

Single Model Family, Single Domain — Generalisation Is Unvalidated

The assumption or constraint. All experiments use exactly one base model (Qwen3-4B-Instruct-2507) fine-tuned on one dataset (DAPO-Math-17k) and evaluated exclusively on mathematical reasoning benchmarks (AIME 2024, AIME 2025, Olympiad Bench, MATH500, Minerva Math, AMC 2023). The paper makes no claims about other model families, domains, or reward structures — and provides no evidence that the framework transfers. The V0 prior itself is trained on trajectories from the Qwen3 series (0.6B–30B), Qwen2.5-7B, and DeepSeek-R1-Distill-Qwen-1.5B (Section 4.1.2), all from a closely related architectural lineage. The paper does not test with, e.g., LLaMA-family models, larger policy scales (beyond 4B), or non-math domains such as code generation, scientific reasoning, or multi-modal tasks.

The consequence. A practitioner using a different model family (e.g., LLaMA-3, Gemma, DeepSeek-R1) cannot assess whether V0.5's benefits transfer. The V0 prior's quality is architecture-dependent — it was trained to recognise capability patterns in Qwen-family models specifically. A policy from a different family may produce output distributions (token-level style, error patterns, chain-of-thought length, calibration) that V0 has never seen, potentially causing systematic prior hallucinations at a much higher rate than observed in the paper's experiments. If the prior is wrong on a large fraction of prompts, the OSLA allocator expands the budget to the maximum (~16 rollouts) for most prompts, and V0.5 degrades to approximately GRPO with extra overhead — not worse, but also not better. The paper's 10% accuracy improvement could shrink or vanish entirely. Additionally, mathematical reasoning has specific properties (binary verifiable rewards, clear correctness criteria, structured chain-of-thought) that make it a favourable domain for value model training — the V0 prior benefits from clean outcome signals and well-defined difficulty gradients. In domains with noisy, subjective, or multi-dimensional rewards (dialogue quality, creative writing, open-ended generation), both V0 training and the hypothesis-testing mechanism (which assumes binary $\{-1, 1\}$ rewards) would require non-trivial adaptation.

What evidence exists in the paper. None that addresses generalisation. The paper does not acknowledge this as a limitation in the main text or conclusion. The evaluation section (Section 4.2) discusses results only on the six math benchmarks with the single base model. No experiments vary the base model architecture or the task domain.

Mitigation status. Not addressed. The paper does not suggest cross-model or cross-domain evaluation as future work. The conclusion (Section 6) mentions future work on "Process-level Generalist Value Models" for "increasingly complex, long-horizon tasks" but does not frame this as addressing the current single-domain limitation. A practitioner must assume that V0.5's efficacy is currently validated only for Qwen-family models on mathematical reasoning; any other deployment scenario requires independent validation.

---

The Cost of Difficulty Estimation Is Fully Externalised — V0 Pre-Training Cost Is Never Amortised in Headline Comparisons

The assumption or constraint. The V0.5 framework relies on a pre-trained generalist value model (V0) that is described as having been trained on "128 GPUs for approximately 40 hours" using "approximately 424k high-quality training pairs" collected from GRPO training trajectories across multiple model scales (Section 4.1.2). This is a substantial upfront computational investment that is never included in any cost accounting. The paper compares V0.5's runtime efficiency (number of rollouts per prompt) against GRPO and DAPO but does not amortise the 5,120 GPU-hours of V0 pre-training across the RL training runs that benefit from it. Similarly, the OSLA dynamic allocation mechanism uses a support buffer of 512 recent prompt-performance pairs and queries V0 in batches during RL training (Section 4.1.3). The inference cost of these V0 forward passes — while described as "marginal compared to the cost of even a single rollout from the 4B policy model" — is never quantified.

The consequence. The headline claim that V0.5 achieves "over 10% performance improvement" with lower per-prompt compute (4 initial rollouts vs. GRPO's 16) is true only if one ignores the cost of producing the prior. A fair total-cost comparison would ask: given a fixed total compute budget (including V0 pre-training), is it better to (a) train V0 and then run V0.5, or (b) skip V0 entirely and run GRPO with a larger group size or more training steps? The paper provides no data to answer this question. For a single RL training run of, say, 200 steps on 32 GPUs, the amortised V0 pre-training cost (5,120 GPU-hours / ~6,400 GPU-hours for training) could nearly double the effective compute cost per unit of accuracy improvement. The framework becomes more favourable if V0 is reused across many RL training runs (amortising the pre-training cost), but the paper does not discuss this amortisation scenario or provide a break-even analysis.

Similarly, the OSLA mechanism's V0 inference overhead adds latency per training step. Each prompt requires a V0 forward pass before any rollouts are generated (Step 1 in Section 4.1.3), and the support buffer must be maintained. For the 4B policy model, one V0 forward pass through Qwen3-Embedding-0.6B plus the adapter and TabPFN head is indeed cheap relative to generating a 4096-token chain-of-thought rollout — but it is not zero, and it serialises before the rollout generation can begin.

What evidence exists in the paper. Section 3.2 and the introduction frame V0 as a "frozen" model that eliminates "synchronous training overhead," implicitly contrasting it with PPO's co-trained value model. But the paper never quantifies V0's pre-training cost or its inference latency. The cost factor $c = 0.0039$ in the OSLA stopping rule (Theorem 3.6) accounts only for the marginal cost of an additional rollout, not for V0 inference. The paper acknowledges implicitly that V0 training is expensive by describing the enhanced training data construction in detail (Section 4.1.2), but does not include this cost in any efficiency calculation.

Mitigation status. Not addressed. The paper does not discuss amortisation, break-even analysis, or total-cost comparisons. The V0.5 framework is presented as an inference-time improvement, not as a joint training-inference optimisation. A practitioner evaluating whether to adopt V0.5 must independently assess whether the cost of training a generalist value model is justified by the downstream RL efficiency gains for their specific use case and deployment scale. The paper provides no tools or analysis to support this assessment.

---

No Direct Empirical Validation of the Core Theoretical Claims — MSE and Bias Are Never Measured

The assumption or constraint. The paper's primary theoretical contributions are theorems that decompose baseline MSE (Theorem 3.2), derive the optimal fusion weight (Theorem 3.3), bound the induced bias of the empirical estimator (Theorem 3.4: $|\text{Bias}(\hat{\mu}^*)| \leq 1/\sqrt{k}$, $O(1/k)$ asymptotic decay), and bound the policy gradient variance in terms of baseline MSE (Theorem 3.1). These theorems provide the mathematical justification for why V0.5 should work. However, the experimental section never directly measures baseline MSE, baseline bias, or policy gradient variance during training. The empirical evidence for the theoretical claims is entirely indirect.

The consequence. The paper cannot distinguish between the following two scenarios:

• Scenario A (theory-confirming): The fused baseline $\hat{\mu}^*$ actually achieves the predicted MSE reduction (near zero when $\hat{\Delta}^2_k = 0$, bounded when $\hat{\Delta}^2_k > 0$), the bias remains within the $1/\sqrt{k}$ bound, and the gradient variance reduction follows quantitatively from Theorem 3.1's decomposition. The observed performance improvement is a direct causal consequence of the mechanisms described in the theorems.

• Scenario B (alternative explanation): The fused baseline provides some statistical regularisation that helps training, but the actual MSE and bias do not closely track the theoretical predictions. The performance improvement arises from a combination of factors — the prior acting as a form of reward smoothing, the OSLA mechanism sometimes increasing the effective group size, the elimination of the KL penalty (V0.5 uses no KL penalty; GRPO uses 0.001, per Section 4.1.4), or other implementation differences — rather than from the specific MSE-minimising properties claimed in the theorems.

Without direct MSE and bias measurements, the paper's theoretical narrative remains a plausible explanation rather than a validated mechanism. Figure 3 (gradient norm) and Figure 4 (entropy) are consistent with the theory but do not constitute a quantitative test of the specific bounds or decompositions. A sceptical reader could interpret the gradient norm reduction as a generic consequence of adding any form of baseline regularisation, not as evidence for the specific shrinkage estimator's optimality.

What evidence exists in the paper. Figure 3 shows that V0.5 produces lower and more stable gradient norms than GRPO. Theorem 3.1 predicts this should happen if baseline MSE is reduced, but the figure does not measure MSE directly. Figure 4 shows that V0.5 sustains higher entropy, which the paper attributes to lower gradient variance, but entropy is also affected by many other factors (reward scale, clipping, learning rate). The fixed sparsity results (Figure 5) show that V0.5 at $k = 4$ works, which implies that the fused baseline is somehow better than a pure empirical mean at $k = 4$, but does not demonstrate that it is better because of the specific MSE-optimal weighting rather than simply because any convex combination with a decent prior helps. The phase transition at $k = 4$ validates the structural hypothesis-testing analysis (Appendix A.8) but does not validate the MSE or bias bounds specifically.

Mitigation status. Not addressed. The paper does not log $\hat{\Delta}^2_k$, $\widehat{\text{MSE}}(k)$, or $\text{Bias}(\hat{\mu}^*)$ during training, nor does it compare these quantities against the theoretical bounds. The theorems stand as mathematical statements (proved in the appendix) rather than as empirically validated descriptions of the actual training dynamics. The paper does not acknowledge this gap between theoretical claims and empirical measurement.

---

The OSLA Allocator's Behaviour Is a Black Box — No Empirical Characterisation of Dynamic Budget Allocation

The assumption or constraint. The Sequential OSLA Allocation mechanism is presented as a core contribution: it dynamically adjusts the rollout budget per prompt based on the statistical conflict between the prior and the empirical observations, using the stopping rule $K^* = \inf\{k \geq k_{\min} : k \geq 1/\sqrt{c} - 1/\hat{\Delta}^2_k\}$ (Theorem 3.6). The paper claims this enables "on-demand budget scheduling" that "balances statistical precision with marginal costs." However, the experiments provide zero empirical characterisation of this mechanism's actual behaviour during training.

The consequence. The reader cannot answer any of the following basic operational questions:

• What is the average rollout budget per prompt under OSLA? Is it close to 4 (indicating the prior is trusted most of the time), close to 16 (indicating frequent hallucinations), or somewhere in between? This is the single most important number for evaluating V0.5's compute efficiency claim.

• What fraction of prompts trigger additional allocation? Does the hypothesis test reject the prior on 5% of prompts, 50%, or 95%? This determines whether OSLA is an occasional safety valve or a routine expenditure.

• How often does the hypothesis test make correct decisions? When $\hat{\Delta}^2_k = 0$ (prior accepted), is the prior actually accurate? When $\hat{\Delta}^2_k > 0$ (prior rejected, budget expanded), does the expanded budget actually correct a genuine hallucination? Without ground-truth $\mu_{\text{true}}$, this is difficult to measure, but the paper does not even attempt an approximate validation (e.g., comparing against final pass@1 rates).

• Does the OSLA stopping rule actually minimise the risk $R(k) = \widehat{\text{MSE}}(k) + c \cdot k$ as claimed? The $O(c)$ regret bound (Appendix A.7) is proven mathematically, but the actual regret during training is never measured.

• What is the distribution of stopping times? Are there prompts where the system oscillates — expanding the budget, then revising the bias estimate, then expanding again — before stabilising? The paper's OSLA description (Step 4 in Section 4.1.3) says the system allocates "2 additional rollouts" per iteration, but the trajectory of $k$ and $\hat{\Delta}^2_k$ across iterations is never shown.

Without this characterisation, the OSLA mechanism is a black box that contributes to V0.5's performance for unknown reasons. It could be that OSLA genuinely allocates compute efficiently (more rollouts to hard/OOD prompts, fewer to easy ones), or it could be that OSLA simply increases the average budget above 4 (making the comparison against fixed $k = 4$ unfair) while contributing little marginal benefit over, say, fixed $k = 8$. The paper's own results in Figure 5 show that fixed $k = 8$ already matches or outperforms GRPO at $G = 16$, and the improvement from fixed $k = 8$ to OSLA (Figure 1) could be driven by the maximum budget of ~16 on a small fraction of prompts rather than by the adaptive scheduling logic.

What evidence exists in the paper. None. The paper reports no statistics about OSLA allocation behaviour — no average budget, no stopping time distribution, no hypothesis test rejection rate, no correlation between $\hat{\Delta}^2_k$ and actual prior error. The only empirical evidence that OSLA works is the comparison between Figure 1 (V0.5 with OSLA) and Figure 5 (V0.5 with fixed sparsity), which shows that OSLA improves performance, but without isolating how much of the improvement is due to the adaptive logic versus simply having a higher effective budget on some fraction of prompts.

Mitigation status. Not addressed. The paper describes the OSLA mechanism in detail (Section 3.4, Section 4.1.3) and proves its theoretical properties (Theorems 3.5, 3.6, Appendix A.7), but provides no empirical window into its operation. The theoretical analysis is rigorous; the empirical validation is absent. A practitioner cannot assess whether the OSLA mechanism would behave similarly under their prompt distribution, reward structure, or prior quality.

---

No Sensitivity Analysis Around the Cost Factor c — the Framework's Only Tunable Parameter

The assumption or constraint. The OSLA stopping rule (Theorem 3.6) depends on exactly one tunable parameter: the marginal compute cost $c$, which controls the tradeoff between statistical precision and rollout expenditure. The paper sets $c = 0.0039$ for all experiments, motivated by the observation that this gives $1/\sqrt{c} \approx 16$, matching GRPO's fixed group size $G = 16$ (Section 4.1.3). No sensitivity analysis around this value is performed.

The consequence. The choice of $c$ directly controls three critical properties of the V0.5 system:

1. Maximum rollout budget: $k_{\max} \approx 1/\sqrt{c}$. At $c = 0.0039$, $k_{\max} \approx 16$. At $c = 0.01$, $k_{\max} = 10$; at $c = 0.001$, $k_{\max} \approx 32$. A practitioner with a different compute budget (tighter or looser) has no guidance on how to set $c$ appropriately.

2. Stopping aggressiveness: The discount term $1/\hat{\Delta}^2_k$ in the stopping condition $k \geq 1/\sqrt{c} - 1/\hat{\Delta}^2_k$ means that a smaller $c$ (larger $1/\sqrt{c}$) makes the system more willing to expand the budget for a given level of observed bias. A larger $c$ makes it more conservative. The paper provides no evidence about whether the specific choice $c = 0.0039$ is near-optimal, or whether performance is robust to variations within a reasonable range.

3. The regret bound: Appendix A.7 proves that the expected excess cost relative to an oracle is $O(c)$. This means that smaller $c$ yields tighter regret bounds (less suboptimality from estimation noise), but the bound is asymptotic and its constant factors are unknown. Empirically, the relationship between $c$ and actual performance could be non-monotonic or exhibit threshold effects.

The absence of a sensitivity analysis means the paper provides a point solution ($c = 0.0039$) without demonstrating that the framework is robust to this choice or that the claimed benefits depend specifically on the OSLA logic rather than on simply capping the budget at a well-chosen value. If V0.5 at $c = 0.0039$ (max budget 16) outperforms GRPO at $G = 16$, but V0.5 at $c = 0.01$ (max budget 10) performs similarly to V0.5 at fixed $k = 8$, then the OSLA mechanism's added value over fixed-sparsity fusion is small.

What evidence exists in the paper. None. The paper does not sweep $c$, does not provide an ablation comparing different cost factors, and does not discuss the sensitivity of results to this parameter. The choice $c = 0.0039$ is justified only by the alignment of $1/\sqrt{c} \approx 16$ with GRPO's $G = 16$, not by any empirical optimisation or sensitivity analysis.

Mitigation status. Not addressed. The cost factor $c$ is presented as a fixed constant in the experimental setup (Section 4.1.3) without discussion of how it was selected or whether different values were explored. The theoretical framework (Theorem 3.6) provides the functional form of the dependence on $c$, which is elegant, but the empirical validation of this dependence is entirely missing. A practitioner wishing to deploy V0.5 in a different compute regime must treat $c$ as an untuned hyperparameter and perform their own sensitivity analysis from scratch.

---

The Framework Depends on a High-Quality Generalist Prior, but Prior Quality Is Neither Ablated Nor Guaranteed

The assumption or constraint. V0.5's entire value proposition rests on the availability of a generalist value model (V0) that provides reasonably accurate prior predictions $V$ for the policy being trained. The Shrinkage Fusion mechanism can handle occasional hallucinations (detecting and overriding them via the hypothesis test), but if the prior is systematically wrong across a large fraction of prompts, the system degenerates: the hypothesis test rejects the prior for most prompts, the OSLA allocator expands the budget to the maximum (~16 rollouts), and V0.5 effectively becomes GRPO with extra V0 inference overhead. The paper trains an enhanced V0 specifically for these experiments (Section 4.1.2) but provides no ablation studying how downstream RL performance depends on V0 quality.

The consequence. A practitioner cannot answer the following deployment-critical questions:

• How accurate does the prior need to be for V0.5 to outperform GRPO? If V0 has, say, 30% hallucination rate (systematic errors on 30% of prompts), does V0.5 still provide a net benefit? What about 50%? 70%? There exists some break-even point beyond which the overhead of maintaining and querying V0, plus the occasional false rejections of an actually-correct prior, outweighs the variance reduction benefit on the prompts where the prior is accurate. The paper provides no data to locate this point.

• Can a weaker/cheaper V0 still work? Training the enhanced V0 required 128 GPUs for 40 hours and carefully curated data from the full Qwen3 series (Section 4.1.2). A practitioner with limited resources might train a smaller V0 on fewer model trajectories or use a V0 trained on a different model family entirely. The paper provides no evidence about whether such a degraded prior would still be useful, or whether the framework catastrophically fails below some quality threshold.

• Is the prior's benefit due to the V0 architecture specifically, or could any reasonable value estimator work? The paper does not ablate V0 against a simpler prior — e.g., a lightweight linear model trained on prompt embeddings, or a fixed prior of $\mu_{\text{true}} \approx 0$ (the midpoint of the reward range) that provides zero-variance but biased predictions. Such an ablation would help distinguish whether V0's specific architecture (in-context learning via TabPFN) is essential, or whether the framework succeeds primarily because any zero-variance anchor, even a weak one, helps when combined with the hypothesis test.

• How does prior quality evolve as the policy trains? The V0 prior is frozen, but the policy's capability distribution shifts during RL training. If the policy's later checkpoints produce outputs that V0 has never seen (because V0 was trained on trajectories from specific model checkpoints), the hallucination rate may increase over the course of training. The paper does not track whether the hypothesis test's rejection rate changes over training steps.

What evidence exists in the paper. The paper describes the enhanced V0 training pipeline in Section 4.1.2 but does not ablate V0 quality. The fixed sparsity results (Figure 5) demonstrate that the enhanced V0 is good enough to make V0.5 at $k = 4$ competitive with GRPO at $G = 16$. However, this is a single data point — one prior of one quality, tested on one policy. The paper does not provide multiple V0 variants (e.g., trained on subsets of the 424k pairs, trained on fewer model scales, trained on shorter trajectories) to map out the performance-vs-prior-quality curve.

Mitigation status. Not addressed. The paper treats V0 as a given and does not discuss how its quality affects V0.5's performance, what minimum quality is required, or how to diagnose when the prior is too weak to be useful. The conclusion mentions future work on "Process-level Generalist Value Models" (Section 6) but frames this as an enhancement, not as an investigation of the current framework's sensitivity to prior quality. A practitioner who cannot replicate the exact V0 training pipeline (due to data, compute, or model availability constraints) has no guidance on whether a substitute prior would work.

7. Implications and Future Directions

How This Work Changes the Landscape

V0.5 introduces a new category of baseline estimation that sits between the two established paradigms — parameterized value models (PPO-style) that reduce variance at the cost of synchronous training, and empirical group sampling (GRPO-style) that eliminates training cost at the cost of high variance under sparsity — by demonstrating that a frozen, independently trained generalist prior can achieve the benefits of both when wrapped in a real-time verification layer. This is not merely an engineering compromise; it is a conceptually distinct approach that reframes the baseline estimation problem from "estimate μ_true as accurately as possible" to "use a zero-variance anchor that you actively test for correctness, and gather more data only when the test fails."

The magnitude of this shift is methodological rather than paradigmatic. V0.5 does not replace GRPO or PPO — it provides a new option in the design space that changes how practitioners should think about the value model's role. Prior work treated the value model as either indispensable (PPO) or dispensable (GRPO). V0.5 shows that a value model can be indispensable for variance reduction but dispensable for adaptation — the adaptation is handled by the hypothesis test and dynamic allocation, not by gradient updates to the value model itself. This is a genuine reframing: the value model transitions from being a continuously updated critic to being a static oracle whose predictions are interrogated rather than trusted.

What this work makes less attractive. The paper's Theorem 3.1 — which establishes that at LLM scale, baseline MSE gets multiplied by an enormous Φ_score while baseline bias is only scaled by a constant L — provides formal justification for abandoning the pursuit of strict unbiasedness in sparse-rollout RLVR. Methods that insist on unbiased baselines (e.g., the standard GRPO group mean, ReMax's greedy-decoding baseline, OPO's gradient-orthogonality-derived estimator) pay a variance penalty that is structurally amplified by model scale. This theoretical point, combined with the empirical demonstration that V0.5's intentionally biased estimator outperforms GRPO's unbiased one at 4× fewer rollouts (Figure 5), suggests that the field's default preference for unbiasedness in baseline estimation was appropriate for small models but is counterproductive for billion-parameter LLMs. Future work on RLVR baselines should prioritise MSE reduction — even at the cost of controlled bias — over unbiasedness as the primary design objective. This is a specific, actionable shift in research priorities, not a vague suggestion.

What this work reconciles. The paper implicitly reconciles two apparently contradictory positions in the literature: (1) generalist models like V0 can provide useful zero-shot value estimates across diverse policies and prompts, and (2) generalist models inevitably hallucinate on out-of-distribution inputs, making them unsafe for deployment in high-stakes optimisation loops. V0.5 demonstrates that these positions are not contradictory — they are complementary descriptions of a single object's strengths and weaknesses, and the solution is not to choose between trusting or discarding the prior but to build a verification mechanism that exploits the strengths while bounding the weaknesses. This resolves the tension that might otherwise prevent practitioners from deploying generalist value models in RL pipelines: the fear of silent corruption from hallucinated baselines is legitimate, but it has a principled solution in the form of real-time hypothesis testing with formal bias bounds.

What this work makes newly attractive as a research direction. The paper opens up generalist model deployment under uncertainty as a distinct research problem within RLVR. Prior work on generalist models (V0, and more broadly the in-context learning paradigm for value estimation) focused on improving prediction accuracy — making the generalist model better. V0.5 shifts the focus to making the deployment of generalist models safe — building verification layers, uncertainty quantification mechanisms, and adaptive compute allocators that can extract value from imperfect generalist predictions without being corrupted by their errors. This is a different research question ("how do we use a model we know to be fallible?") from the standard one ("how do we make the model less fallible?"), and it has broader relevance beyond RLVR to any setting where frozen generalist models are deployed in feedback loops (e.g., LLM-based evaluators, reward models, or safety classifiers).

---

Follow-Up Research This Work Enables

Direct measurement of baseline MSE and bias during training to validate the theoretical decomposition. The paper proves that the fused baseline's MSE decomposes as w²σ²_noise + (1−w)²Δ² (Theorem 3.2) and that the empirical estimator's bias is bounded by 1/√k (Theorem 3.4), but never measures these quantities during actual RL training. A strong follow-up would instrument the V0.5 training loop to log ˆΔ²_k, the fused baseline ˆμ*, the empirical mean ¯v_k, and the V0 prior V at each step, then compute the empirical MSE as E[(ˆμ* − μ_true)²] where μ_true is approximated by the pass@1 rate estimated from a large number of evaluation rollouts on a held-out set of prompts. This would directly test whether the MSE follows the predicted w²σ²_noise + (1−w)²Δ² trajectory, whether the empirical bias |E[ˆμ*] − μ_true| stays within the 1/√k envelope, and whether the O(1/k) asymptotic decay rate for bias (property 2 of Theorem 3.4) is observed as k increases under OSLA allocation. A negative result — e.g., the bias systematically exceeding the bound — would indicate that the nonlinear dependence of ˆw_k on ¯v_k introduces stronger bias than the proof accounts for, potentially due to the max operator in ˆΔ²_k creating a non-differentiable threshold effect at (¯v_k − V)² = 1/k.

Controlled hallucination experiments with artificially corrupted priors to validate the hypothesis test's detection and isolation logic. The paper claims that the positive-part truncation in ˆΔ²_k = max(0, (¯v_k − V)² − 1/k) functions as a hypothesis test that detects prior hallucinations (Appendix A.4) and that the OSLA allocator expands the budget to override detected hallucinations (Theorem 3.6). However, no experiment demonstrates this mechanism working in a controlled setting where the ground-truth prior bias is known. A follow-up would inject artificial bias into the V0 prior — e.g., by adding a known offset δ to V for a random subset of prompts, where δ ∈ {0.1, 0.3, 0.5, 0.7} — and measure (1) whether the hypothesis test correctly rejects the prior more frequently as δ increases (the rejection rate should rise with δ), (2) whether the OSLA allocator expands the budget proportionally to δ (the average k should increase with δ), and (3) whether the fused baseline's actual MSE converges toward the theoretical minimum given the known Δ² = δ². A failure to detect large injected biases (e.g., the test accepting a prior with δ = 0.5 as often as it accepts an unbiased prior) would indicate that the 1/k noise bound is too conservative or that the k_min = 4 initial budget is insufficient to distinguish moderate bias from sampling noise. This experiment would directly validate the paper's central "safety" claim.

Cross-model-family and cross-domain stress-testing to establish the boundary conditions for generalisation. All experiments in the paper use Qwen3-4B-Instruct-2507 on mathematical reasoning benchmarks, with a V0 prior trained on trajectories from Qwen-family models. A critical stress-test would replicate the V0.5 framework with (1) a policy model from a different architectural family (e.g., LLaMA-3-8B, Gemma-2-9B, or DeepSeek-R1-Distill-Llama-8B) using the same V0 prior trained on Qwen trajectories, and (2) a non-math domain such as code generation (HumanEval, MBPP) or scientific reasoning (GPQA). The prediction from the paper's framework is that V0.5 should still outperform GRPO when the V0 prior retains some predictive accuracy despite the domain/model shift, but that the benefit should shrink as the prior's accuracy degrades — and that the hypothesis test and OSLA allocator should prevent catastrophic failure even when the prior is largely wrong (by expanding the budget to the maximum and defaulting to empirical estimation). Measuring the correlation between V0's out-of-distribution accuracy (estimated via pass@1 on the new model/domain) and V0.5's performance improvement over GRPO would map out the deployment envelope — the range of prior qualities for which V0.5 provides net benefit. If the benefit vanishes at even modest domain shifts (e.g., moving from AIME to GPQA), this would indicate that the enhanced V0 training data (Qwen-family math trajectories) is too narrow for the framework to generalise, and that generalist value models need substantially broader pre-training to support V0.5-style deployment.

Sensitivity analysis around the cost factor c to characterise the OSLA Pareto frontier. The paper uses a single value c = 0.0039 (giving maximum budget ≈16) without sweeping or justifying this choice empirically. A systematic follow-up would run V0.5 training with c ∈ {0.01, 0.005, 0.0039, 0.002, 0.001, 0.0005} (giving maximum budgets from 10 to ~45) on a single benchmark (e.g., MATH500 for faster iteration) and measure both final accuracy and average rollout budget per prompt. This would produce a Pareto curve showing the accuracy-vs-compute tradeoff achievable by varying the single cost parameter, allowing practitioners to select c based on their specific compute budget rather than defaulting to the paper's alignment with GRPO's G = 16. Additionally, measuring the empirical regret (the gap between OSLA's risk R(K*) and the oracle risk R(k*_oracle) computed from ground-truth Δ²) across different c values would empirically validate the O(c) regret bound from Appendix A.7. If the regret grows faster than O(c) — e.g., superlinearly for small c due to accumulated estimation errors over long allocation sequences — this would indicate a practical limit on how aggressively the system can reduce the rollout budget before estimation noise in ˆΔ²_k causes the stopping rule to make systematically poor decisions.

Extending the framework to continuous or multi-dimensional reward spaces. The paper's hypothesis test and OSLA stopping rule rely on the binary reward structure r ∈ {−1, 1} to derive the variance bound σ²_noise ≤ 1/k (Equation 6) and the minimum group size k_min = 4 via the discrete gap analysis (Appendix A.8). Many practical RLVR applications use continuous rewards (e.g., partial credit on math problems, rubric-based scoring, learned reward models) or multi-dimensional rewards (e.g., correctness + formatting + helpfulness). A direct extension would generalise the variance estimate ˆσ²_noise from the conservative bound 1/k to an empirical estimate based on the observed reward variance (e.g., the sample variance of the k rollout rewards), which would require addressing the small-sample instability of variance estimates when k is small. The hypothesis test would need to be reformulated: with continuous rewards, the noise distribution is no longer bounded by a simple 1/k envelope, and the positive-part truncation logic max(0, (¯v_k − V)² − ˆσ²_noise) would depend on an estimated rather than guaranteed variance bound. The minimum group size analysis would also change — the quantisation gap argument disappears for continuous rewards, so the framework might function at k < 4, but the statistical power of the test would degrade. A strong follow-up would implement these generalisations and benchmark on RLVR tasks with continuous rewards (e.g., training a coding model with test-case pass rate as the reward signal), measuring whether the core benefits of V0.5 (variance reduction via prior fusion, adaptive budget allocation) persist when the clean binary-reward assumptions are relaxed.

Ablation of the V0 architecture to determine whether the prior's specific properties matter or whether any zero-variance anchor suffices. The paper uses a sophisticated generalist value model (V0 with embedding backbone, residual query adapter, and TabPFN in-context head) trained on 424k pairs across multiple model scales. But the theoretical framework (Theorems 3.2–3.4) does not depend on how the prior is produced — only on its properties (zero variance, unknown bias). A revealing ablation would replace V0 with simpler priors of varying quality: (1) a constant prior V = 0 (the midpoint of the reward range, representing "no information"), (2) a linear model trained on prompt embeddings from a frozen sentence encoder to predict difficulty, (3) a smaller V0 trained on fewer trajectories (e.g., 10k pairs instead of 424k), and (4) the enhanced V0 used in the paper. Running V0.5 with each prior on MATH500 would map out the performance-vs-prior-quality curve and answer the question: does the framework require a genuinely informative prior, or does it succeed primarily because any zero-variance anchor, even a weak one, helps stabilise the empirical mean under extreme sparsity? If the constant prior V = 0 still outperforms GRPO at G = 16, this would indicate that the hypothesis test and fusion mechanism are providing a form of statistical regularisation (James-Stein-style shrinkage toward zero) that is beneficial regardless of the prior's accuracy. If only the enhanced V0 provides benefit, this confirms that the framework crucially depends on a high-quality prior and that investment in generalist value model training is the binding constraint for adoption.

---

Practical Applications and Downstream Use Cases

Cost-efficient RL post-training for small-to-medium model deployments on consumer hardware. The paper demonstrates that V0.5 with k_init = 4 rollouts per prompt matches or exceeds GRPO with G = 16 rollouts (Figure 5), representing a 4× reduction in per-prompt generation cost. For an organisation fine-tuning a 4B-parameter model on mathematical reasoning tasks using 32 GPUs, this translates directly to either (1) training ~4× faster at matched per-step accuracy, or (2) achieving higher final accuracy in the same wall-clock time by using the saved compute for more training steps. The V0 prior's inference cost (a single forward pass through Qwen3-Embedding-0.6B plus a lightweight adapter) is marginal compared to generating a 4096-token chain-of-thought rollout from the 4B policy — perhaps 1–2% of the per-rollout cost — making the 4× rollout reduction nearly pure savings. The practical barrier is access to a pre-trained generalist value model. If the V0 authors release model weights, or if an organisation trains its own V0 once and amortises the 5,120 GPU-hours across many downstream fine-tuning runs, the economics become strongly favourable. The OSLA mechanism's adaptive budget allocation (spending more compute only when the prior appears unreliable) prevents the worst-case scenario where a systematically inaccurate prior silently degrades training — the system detects and compensates automatically, making deployment safer even without exhaustive prior validation.

RL training with severely compute-constrained rollout budgets, such as on-device or edge fine-tuning. The paper's demonstration that V0.5 converges successfully at k = 4 (Figure 5) while GRPO fails structurally at group sizes below 4 (the phase transition identified in Appendix A.8, visible in the k = 1, 2 failure in Figure 5) establishes k = 4 as the minimum viable rollout budget for binary-reward RLVR with a generalist prior. This has direct relevance for scenarios where generating even a handful of rollouts is expensive — on-device fine-tuning where each rollout runs locally on a mobile processor, or iterative self-improvement where the model generates and evaluates its own responses in a tight loop. In these settings, the difference between requiring 4 rollouts per prompt (V0.5) versus 16 (GRPO) or a co-trained value model (PPO, which doubles memory and compute) is the difference between feasible and infeasible deployment. The framework's theoretical guarantee that the induced bias from fusion is bounded by 1/√k ≤ 0.5 even at k = 4 (Theorem 3.4) provides a formal safety certificate that the baseline will not catastrophically mislead the policy — a property that pure empirical estimation at k = 4 cannot provide.

Batch RL inference pipelines where prompt difficulty varies widely and uniform compute allocation wastes resources. The OSLA mechanism's core behaviour — allocate minimal compute to prompts where the prior is accurate, expand compute only when statistical conflict is detected — naturally lends itself to production batch processing where different prompts require vastly different amounts of reasoning effort. In a deployment generating training data for self-improvement (e.g., STaR-style or ReST-style pipelines), a batch of 10,000 math problems might include 7,000 easy problems (where the policy has high pass@1 and the prior is accurate), 2,500 medium problems (where the prior is slightly off and needs some additional rollouts), and 500 hard problems (where the prior hallucinates and needs the full budget). A uniform GRPO allocation of G = 16 rollouts per prompt spends 7,000 × (16 − 4) = 84,000 unnecessary rollouts on easy problems. V0.5 with OSLA automatically concentrates the budget on the 3,000 prompts where the prior is unreliable, achieving better baseline quality on hard problems without wasting compute on easy ones. The paper's regret bound (O(c), Appendix A.7) guarantees that the excess compute spent due to imperfect stopping decisions is bounded by a small constant — meaning the system does not accidentally overspend on prompts where the prior appears wrong due to sampling noise. This provides a principled solution to the exploration-exploitation tradeoff in compute allocation that currently requires hand-tuned heuristics.

Deployment of generalist evaluation models in production RL pipelines without risking silent training corruption. A recurring challenge in production RL systems is that auxiliary models (reward models, value models, safety classifiers) drift out of distribution as the policy evolves, silently corrupting the training signal. V0.5 provides a template for deploying such models with active monitoring: instead of trusting the auxiliary model's output unconditionally, treat it as a hypothesis to be tested against live rollouts, and allocate additional ground-truth data (in this case, verifier rewards) when the hypothesis is rejected. This pattern generalises beyond value models. For example, a learned reward model that scores open-ended responses on a 1–5 scale could be deployed in a V0.5-style framework: the reward model's prediction serves as the prior, a sparse sample of human evaluations provides the empirical signal, and the hypothesis test detects when the reward model's prediction diverges from the human evaluations beyond the expected inter-annotator noise. The OSLA allocator then triggers additional human evaluations only when the reward model appears unreliable — concentrating expensive human labour on the prompts where it is most needed. The specific mechanism (shrinkage fusion, bias estimation via truncated squared discrepancy, stopping rule based on marginal return) would need adaptation to continuous rewards and non-Bernoulli noise models, but the architectural pattern — frozen prior + hypothesis test + adaptive ground-truth allocation — is directly transferable.

---

When to Prefer This Method

The paper articulates a clear tradeoff against the two dominant baseline estimation paradigms (GRPO-style empirical sampling and PPO-style parameterized value models), making a conditional decision framework appropriate.

Prefer V0.5 when:

• You have access to a pre-trained generalist value model (or the resources to train one) that achieves non-trivial predictive accuracy on your policy's output distribution. The paper's enhanced V0 was trained on 424k pairs from Qwen-family models on math reasoning; a prior trained on different models or domains may require independent validation.
• Your per-prompt rollout budget is severely constrained (e.g., you want to use 4–8 rollouts per prompt rather than 16+) due to generation length, hardware limitations, or throughput requirements. V0.5 at k = 4 matches GRPO at G = 16 (Figure 5), and at k = 8 outperforms it.
• Your reward structure is binary (r ∈ {−1, 1}) or can be binarised without losing critical signal. The hypothesis test's noise bound (1/k) and the minimum group size analysis (k_min = 4) both assume Bernoulli-distributed rewards. Continuous or multi-dimensional rewards require adapting the variance estimation and test threshold.
• You are training at a scale where baseline MSE amplification (Theorem 3.1) is the dominant source of gradient instability — i.e., your model has billions of parameters, making Φ_score enormous. For very small models, the asymmetric bias-variance tradeoff that V0.5 exploits (bias is less costly than variance) may not hold, and a standard unbiased estimator may be preferable.
• You need the value model to be frozen and reusable across multiple training runs (amortising pre-training cost). V0.5's V0 prior is trained once and never updated, unlike PPO's value model which must be re-trained for each new policy.

Prefer GRPO (or its empirical-sampling variants) when:

• You do not have access to a generalist value model and cannot afford the upfront cost of training one (5,120 GPU-hours for the enhanced V0 in this paper). GRPO requires no auxiliary model.
• Your training budget can comfortably accommodate group sizes of G ≥ 16, making the empirical mean's variance acceptably low without external priors. The paper's results show GRPO at G = 16 performs competitively, and at even larger G (e.g., 32–64) the variance reduction from a prior becomes less impactful.
• Simplicity of implementation is the primary concern — GRPO eliminates an entire model component (the value model) and the associated engineering complexity (V0 API integration, support buffer maintenance, OSLA allocation logic with its batch-padding and global-stop conditions).

Prefer PPO (or its coupled-value-model variants) when:

• Your policy model is small enough that training a synchronous value model does not dominate the compute budget. For a 1B-parameter policy, co-training a value model of comparable size may be more cost-effective than pre-training a generalist V0 that must generalise across many policies.
• You need the value model to continuously adapt to the policy's evolving output distribution without the detection-lag inherent in V0.5's hypothesis test (which only detects hallucinations after k ≥ 4 rollouts have been generated and the discrepancy computed). PPO's value model updates every step via gradient descent, providing tighter tracking of non-stationary returns at the cost of synchronous training.
• Your reward structure is continuous or multi-dimensional and you have not implemented the generalisations to V0.5's hypothesis test that would be required. PPO's value model natively handles continuous value prediction without the Bernoulli-distribution assumptions that V0.5's current formulation relies on.

Do not prefer V0.5 when:

• Your rollout budget per prompt is below 4 (as established by the structural phase transition in Appendix A.8 and Figure 5).
• You are operating in a domain where no reasonably accurate generalist prior is available and training one is infeasible. V0.5 with a systematically inaccurate prior degenerates to GRPO with extra overhead — it does not fail catastrophically (the hypothesis test rejects the prior and the budget expands to the maximum), but it provides no benefit over simply running GRPO at the maximum budget directly.