ArXiv: 2510.11683
🎯 Pitch
Training diffusion LLMs with RL blows up GPU memory because the nonlinear advantage weighting forces storing every Monte Carlo sample’s computation graph, capping sample counts at ≤4 and crippling likelihood approximation. BGPO rewrites the ELBO-based objective into a linear lower bound that decouples samples, enabling thousands of gradient-accumulation steps at constant memory while exactly recovering the original objective’s value and gradient on-policy—unlocking 15-point accuracy gains on Sudoku by simply allowing accurate likelihoods.
1. Executive Summary
This paper introduces Boundary-Guided Policy Optimization (BGPO), a memory-efficient reinforcement learning algorithm for diffusion large language models (dLLMs) that addresses the GPU memory bottleneck that previously constrained Monte Carlo sample sizes during likelihood approximation. Working with LLaDA-8B-Instruct across math problem solving (MATH500, GSM8K), code generation (HumanEval, MBPP), and planning tasks (Sudoku, Countdown), BGPO constructs a linear lower bound of the ELBO-based RL objective — satisfying both linearity (enabling gradient accumulation across MC samples for constant memory, illustrated by supporting nt = 16 vs. previous methods' nt = 4) and equivalence (matching the value and gradient of the original objective in on-policy training) — allowing substantially larger sample sizes that reduce gradient bias and variance. BGPO outperforms prior RL algorithms for dLLMs (diffu-GRPO and VRPO-OL) on all tasks, achieving state-of-the-art results including a 14.9-point improvement on Sudoku and 68.0-point improvement on Countdown over the supervised-fine-tuned baseline, with only marginal increases in per-step training time despite using 4× the MC sample size of VRPO-OL, establishing that memory-efficient RL for dLLMs is feasible and that larger MC sample sizes directly translate to more accurate policy optimization.
2. Context and Motivation
The Core Problem: Reinforcement Learning for dLLMs Is Bottlenecked by Memory, Not by Algorithm Design
The fundamental question this paper tackles is deceptively practical: what is preventing reinforcement learning from being effectively applied to diffusion large language models, and what minimal change can fix it? This matters because the answer uncovers not an algorithmic insight about RL itself, but a hardware constraint — GPU memory limits — that distorts the RL training signal in a way that has nothing to do with what the optimizer should be learning, yet dominates final performance.
To understand why this is a bottleneck requires understanding what makes dLLMs different from conventional autoregressive language models. In an autoregressive model like GPT-4 or LLaMA, the probability of generating a sequence of tokens y = (y₁, y₂, ..., y_n) given a prompt x factorizes cleanly as a product of conditional probabilities:
Each term is a single forward pass through the model, the computation is cheap, and — critically for RL — you can compute the exact log-likelihood of any generated sequence. This log-likelihood is the fundamental quantity that appears in policy gradient objectives (Eq. 1 in the paper), specifically inside the importance sampling ratio π_θ(y|x) / π_θold(y|x). Without it, you cannot compute the advantage-weighted update that tells the model how much to reinforce a generated response relative to others.
Diffusion language models break this clean factorization entirely. Instead of generating tokens left-to-right, they start from a fully masked sequence and iteratively unmask tokens in parallel across the whole sequence. At each denoising step, the model predicts all currently-masked tokens simultaneously. To compute log π_θ(y|x) — the probability that this specific sequence y would be generated through this entire iterative process — you would need to integrate over all possible trajectories of partial masking/unmasking that could lead to y. As the paper states in Section 2.2:
"the iterative denoising generation makes the exact computation of log π_θ(y|x) intractable."
This is not a minor inconvenience. The log-likelihood appears inside an exponential in the RL objective (Eq. 7), and small errors in approximating it get amplified nonlinearly. Worse, the gradient of that exponential depends on every intermediate computation used to estimate the log-likelihood, and automatic differentiation frameworks must retain all those intermediate tensors in GPU memory to backpropagate through them.
Why This Problem Matters: RL Is the Missing Piece for dLLMs
The paper is motivated by a genuine asymmetry in the LLM landscape. Autoregressive models have undergone a revolution through RL fine-tuning: from RLHF for instruction following to GRPO-based reasoning enhancements in DeepSeek-R1. These methods demonstrably improve mathematical reasoning, code generation, and instruction adherence. But dLLMs — despite their compelling advantage of parallel generation and the potential for dramatic inference speedups demonstrated by commercial systems like Mercury and Gemini Diffusion — have been largely left out of this RL revolution. The paper notes (Section 1):
"existing works primarily focus on pre-training and supervised fine-tuning of dLLMs, while leveraging reinforcement learning (RL) to further enhance dLLMs remains a challenging problem, even though RL has demonstrated significant efficacy in improving various capabilities of LLMs"
This gap is not academic. If dLLMs cannot be effectively fine-tuned with RL, they remain permanently disadvantaged relative to autoregressive models on tasks where RL provides the crucial "last mile" of capability — tasks requiring multi-step reasoning, self-verification, or alignment with nuanced human preferences. The dLLM's inference speed advantage means little if the model's reasoning quality can't match what RL provides to autoregressive competitors.
Prior Approaches: Two Approximation Strategies, Two Different Failure Modes
Prior to BGPO, the field had developed two approaches to the log-likelihood problem, each with a distinct and severe limitation.
Single-pass estimation (diffu-GRPO). The simplest approach, introduced by Zhao et al. (2025a) in the d1 work, is to pretend the diffusion process is factorized: approximate log π_θ(y|x) by a single forward pass where the prompt x is randomly partially masked, and the log-likelihood is the sum of the model's token-level predictions for each position in y. The paper calls this "single-pass mean-field estimation" in Section 2.2. This is computationally efficient — one forward pass, minimal memory — but introduces what the paper's gradient analysis reveals to be substantial and irreducible bias relative to the true log-likelihood. Figures 3 and 4 quantify this: diffu-GRPO's gradient exhibits both high variance and high bias compared to the golden gradient (BGPO with nt=256). The bias doesn't shrink with more computation because the approximation itself is structurally wrong — it ignores the dependencies between denoising steps that make the diffusion process non-factorized.
ELBO-based estimation with limited Monte Carlo (VRPO-OL). A more principled approach, introduced by Zhu et al. (2025) in LLaDA 1.5, uses the evidence lower bound (ELBO) of the log-likelihood as a proxy (Eq. 3). The ELBO is an expectation over random masking times t and partially masked sequences y_t, and it can be estimated via Monte Carlo sampling: sample nt independent (t, y_t) pairs, compute each ℓ_πθ(y_t, t, y|x), and average. When nt is large enough, this ELBO-based estimate approaches the true log-likelihood, eliminating the structural bias problem of single-pass estimation.
The mechanism is elegant. For each MC sample at time t, you partially mask the response y according to the forward process (each token masked independently with probability t), then ask the model to predict the original tokens at the masked positions. The term ℓ_πθ (Eq. 4) measures how well the model's predictions match the ground-truth tokens, normalized by 1/t. Averaging over many random t values approximates the integral over the continuous time range [0,1] that defines the ELBO in Eq. 3.
However, there is a catch — and it is the entire motivation for BGPO. When this ELBO estimate gets plugged into the RL objective (Eq. 6-7), it appears inside an exponential:
where each is the average of nt terms (Eq. 5). To compute the gradient of with respect to the model parameters θ, automatic differentiation must retain the entire forward computational graph for all nt MC samples simultaneously. This is because the gradient flows through the exponential, which is a nonlinear function of the sum — you cannot decompose the gradient into independent per-sample contributions and accumulate them. The paper explains this directly in Section 2.2:
"Each time is computed, nt forward passes of need to be executed (i.e., Eq. 4 and 5), and all the nt computational graphs must be retained in the memory for calculating the gradient of the exponential function in Eq. 7."
The consequence is brutally practical. On an H800 GPU (80GB), the maximum feasible nt is 4 for math and planning tasks, and only 2 for code generation tasks (where prompts are longer). The left panel of Figure 1 shows the memory explosion: VRPO-OL's memory usage scales linearly with nt, exceeding the H800's 80GB limit at nt=8 with a 512-token response length. Practitioners are forced to use nt=4, which means each ELBO estimate is based on only 4 random (t, y_t) pairs. The paper emphasizes this is not just imprecise — it distorts the RL objective:
"the sample size can only remain small (e.g., nt = 4), which results in inaccurate approximations for the likelihoods as well as the final objective, seriously affecting the final performance."
This is the central tension of the paper: the principled method (ELBO estimation) would work well if you could use large nt, but the hardware prevents it. The efficient method (single-pass) works within memory constraints but is structurally biased. Neither approach delivers what dLLMs need for effective RL.
How This Paper Positions Itself: Changing the Computation Graph, Not the Objective
BGPO's contribution is not a new RL objective, a better ELBO estimator, or a more sophisticated advantage function. Those components remain essentially unchanged from VRPO. Instead, BGPO reformulates the computation graph so that the gradient can be decomposed into independent per-MC-sample contributions, enabling gradient accumulation and constant memory usage regardless of nt.
The key insight is in Section 3. Instead of maximizing directly (Eq. 6), BGPO maximizes a constructed lower bound (Eq. 8) that is carefully designed to have two specific mathematical properties that lacks:
-
Linearity (Section 3.1): The lower bound is a linear sum of per-MC-sample terms. In Eq. 14, , where each depends only on the j-th MC sample (Eq. 13). This means you can compute the gradient contribution of each independently, accumulate them across samples, and update parameters — all while only keeping one MC sample's computational graph in memory at a time. Algorithm 1 makes this explicit: the inner loop (lines 9-10) computes and backpropagates each separately, then aggregates.
-
Equivalence (Section 3.2): The lower bound is not arbitrary — it is constructed so that when training is on-policy (π_θ = π_θold), both the value and gradient of exactly match those of the original . The proof (Eqs. 15-17) relies on the fact that on-policy, , which collapses the lower bound constructions (Taylor expansion for A ≥ 0, Jensen's inequality for A < 0) to exactly the original objective. This ensures that optimizing the lower bound is not a compromise — it provides the same signal as the original objective during on-policy training, just delivered through a memory-efficient path.
The construction itself is mathematically straightforward. When advantages are non-negative, the paper applies a first-order Taylor expansion: , which converts the exponential of an average into an average of linear terms (Eq. 9). When advantages are negative, Jensen's inequality is applied in the opposite direction to produce a convex combination that again factorizes (Eq. 12). The two cases are necessary because the direction of the inequality in the bound must be consistent with maximizing the objective — you need a lower bound, and the construction depends on the sign of A(x,y) to ensure the inequality points the right way.
This positions BGPO not as a competitor to VRPO's mathematical approach, but as an enabler of that approach at scale. BGPO lets VRPO's ELBO-based method actually use the large nt values that the theory says it needs, without being blocked by hardware limits. The paper's experiments confirm this interpretation: VRPO-OL at nt=4 performs worse than BGPO at nt=16 (Table 1), not because BGPO's objective is fundamentally better, but because nt=4 introduces too much noise in the gradient signal.
The Memory-Performance Tradeoff: A Quantitative Picture
Figure 1 (left panel) provides the key visual motivation. At nt=1, both VRPO-OL and BGPO use negligible memory. At nt=2, VRPO-OL's memory jumps to roughly 20GB. At nt=4 (VRPO-OL's practical maximum), it reaches roughly 37GB. At nt=8, VRPO-OL exceeds 70GB — approaching the H800's limit. At nt=16, VRPO-OL is far beyond feasible memory, while BGPO's memory usage remains flat at approximately 15-20GB regardless of nt.
This is not a minor efficiency improvement. The middle and right panels of Figure 1 show that this memory advantage translates directly to performance: BGPO achieves 45.7% on MATH500 and 84.3% on GSM8K, compared to VRPO-OL's 44.1% and 83.3%, and diffu-GRPO's 43.1% and 82.1%. All three methods start from the same LLaDA-8B-Instruct baseline (39.6% / 79.3%), so the gaps directly measure the quality of the RL signal.
The Gradient Quality Argument
The paper makes a specific causal claim: larger nt → lower gradient variance and bias → more accurate optimization direction → better final performance. This is not just assumed — it is empirically demonstrated.
Table 2 shows monotonic improvement on MATH500 as nt increases from 1 to 16 under BGPO: 43.5% (nt=1) → 44.1% (nt=2) → 43.7% (nt=4) → 45.3% (nt=8) → 45.7% (nt=16). There is a slight non-monotonicity at nt=4, but the overall trend is clear.
Figures 3 and 4 provide the mechanistic explanation. Using BGPO with nt=256 as the "golden gradient" (the best available approximation of the true RL gradient), the paper measures bias (systematic deviation from this golden signal) and variance (random fluctuation around it) for each method at different nt values. The findings are:
- diffu-GRPO (nt=1): Both gradient bias and variance are the highest — this method has the worst optimization signal, which aligns with its lower final performance in Table 1.
- VRPO-OL (nt=1, 2, 4): Both bias and variance decrease as nt increases, but VRPO-OL cannot go beyond nt=4 due to memory constraints, leaving significant residual noise.
- BGPO (nt=1, 2, 4, 8, 16): Bias and variance continue to decrease monotonically, reaching substantially lower levels at nt=16. This is the direct evidence that the memory constraint was the bottleneck, not the ELBO-based approach itself.
The gradient standard deviation (Figure 3) for BGPO at nt=16 is approximately 40-50% lower than VRPO-OL at nt=4, and the gradient bias (Figure 4) is reduced by a similar proportion. These are large reductions — a gradient with half the noise and half the systematic error will point more reliably toward the true optimum, requiring fewer steps to converge and arriving at a better final point.
Practical Significance: Why Memory Efficiency Matters More Than It Sounds
The training speed comparison in Section 4.5 and Figure 5 addresses a natural concern: if BGPO uses 4× more MC samples, doesn't it take 4× longer per training step? The answer is no — BGPO at nt=16 averages 151.5 seconds per step, compared to VRPO-OL at nt=4 at 130.2 seconds, and diffu-GRPO at nt=1 at 128.8 seconds. The increase is only about 16% over VRPO-OL, despite 4× the MC samples.
Why? Because the dominant time cost is response rollout: sampling G=8 full responses from the current policy for each prompt in the batch. Each response generation involves multiple denoising steps (128-256, per Table 4), and each step requires a full model forward pass. The MC sampling for ELBO estimation occurs on already-generated responses and is relatively cheap by comparison. This means BGPO's constant-memory gradient accumulation adds negligible overhead — the paper has effectively gotten the higher-quality RL signal "for free" in terms of wall-clock time, since the expensive part (response generation) is identical across methods.
This is a crucial practical point. If BGPO required 4× the training time, the benefit of larger nt might not justify the cost. But because the memory bottleneck was the only constraint — not compute time — BGPO unlocks performance that was already theoretically achievable with VRPO's approach, just previously blocked by hardware limits that had nothing to do with algorithm quality.
The Broader Implication: Enabling the RL Revolution for dLLMs
The paper's framing in Section 5 places this work in a larger narrative. Autoregressive models have seen explosive progress through RL: from basic RLHF to sophisticated reasoning enhancements like DeepSeek-R1's GRPO training. dLLMs — despite their compelling architecture for fast inference — have been excluded from this progress because the basic machinery of RL (computing and differentiating through log-likelihoods) doesn't work cleanly on them.
BGPO is positioned as the foundational infrastructure that removes this barrier. It doesn't propose a new RL algorithm or a new dLLM architecture — it solves the specific engineering problem (memory-efficient gradient computation for ELBO-based log-likelihood estimates) that was preventing existing RL algorithms from being applied effectively. The paper's experiments span math, coding, and planning tasks specifically to show this is not domain-specific — any task where RL benefits autoregressive models can now potentially benefit dLLMs through BGPO.
The limitations section (Section 7) acknowledges that experiments are limited to 8B-parameter models, since larger open-source dLLMs do not yet exist. But the theoretical argument — that the memory savings scale linearly with nt while performance improves — suggests the approach should generalize to larger scales, where the memory constraint would be even more binding.
3. Technical Approach
3.1 Reader Orientation
This paper develops a drop-in replacement for the RL objective computation in diffusion language model training — not a new RL algorithm, not a new model architecture, but a specific mathematical reformulation of how the training loss is constructed and its gradient computed. The system solves a GPU memory bottleneck: previous methods had to store the computational graphs of all Monte Carlo samples simultaneously to differentiate through the nonlinear exponential in the RL objective, which capped sample sizes at nt=4; BGPO rewrites the objective as a linear sum of per-sample terms so that gradients can be accumulated one sample at a time, keeping memory constant regardless of how many Monte Carlo samples are used.
3.2 Big-Picture Architecture (Diagram in Words)
The BGPO training loop (Algorithm 1) has five major components arranged in a nested loop structure:
-
Old Policy (π_θold): A snapshot of the current model parameters, frozen at the start of each training iteration. This generates G complete responses per prompt via the full iterative denoising process, and also provides the baseline log-likelihood estimates against which the current policy's improvements are measured.
-
Reward Model / Advantage Estimator: An external function (could be a learned reward model, a verifier, or rule-based scoring) that assigns a scalar reward r(x, y^(i)) to each generated response. These rewards are normalized within each prompt's group of G responses into advantages A(x, y^(i)) using group-based z-score normalization (Eq. 18).
-
Monte Carlo ELBO Sampler: For each generated response, this component draws nt independent random masking times t^(j) ~ Uniform[0,1] and applies the forward masking process to produce partially masked versions y^(i)_t^(j). Each masked sample is then fed through the current policy p_θ to compute the per-sample ELBO term ℓ_πθ (Eq. 4).
-
Boundary-Guided Lower Bound Constructor: This is BGPO's core innovation. Instead of averaging the nt ELBO terms and then exponentiating (which creates the memory bottleneck), it computes a per-sample quantity g_j for each Monte Carlo sample j (Eq. 13), using different construction rules depending on whether the advantage is positive (Taylor expansion) or negative (Jensen's inequality). These g_j terms are summed linearly to form (x, y) (Eq. 14).
-
Gradient Accumulator: The gradient of each g_j is backpropagated independently through the model, accumulated across all j for all G responses, and then used to update the policy parameters. Because each g_j depends only on its own MC sample's computational graph, memory can be freed after each backpropagation.
Information flows as follows: prompts enter → old policy generates G responses per prompt → reward model scores each response → advantages computed via group normalization → for each response, nt random masking times are sampled → each partially masked response passes through the CURRENT policy to compute ℓ_πθ and ℓ_πθold → per-sample g_j terms constructed using the boundary-guided rules → each g_j's gradient backpropagated and accumulated → parameters updated → old policy refreshed.
3.3 Roadmap for the Deep Dive
- First, the VRPO baseline's ELBO-based objective (Eqs. 3-7), since BGPO constructs a lower bound of this exact objective. Understanding what is being bounded is prerequisite to understanding how the bound works.
- Second, the memory bottleneck in detail: why the exponential in Eq. 7 forces all nt computational graphs to coexist in memory, and how this creates the nt=4 ceiling. This is the problem BGPO solves, so the problem must be crystal clear before the solution.
- Third, the two-sided lower bound construction (Eqs. 8-14): the core mathematical innovation, including why different bounding techniques are needed for positive vs. negative advantages, and how linearity emerges in both cases.
- Fourth, the on-policy equivalence proof (Eqs. 15-17): why the lower bound is not a compromise but an exact match to the original objective during on-policy training.
- Fifth, the algorithm pseudocode walkthrough (Algorithm 1) and the group-based advantage estimation (Eq. 18), which together specify how the mathematical construction maps to an executable training loop.
- Sixth, the hyperparameter and implementation choices, including block-wise decoding, response lengths, diffusion steps, and the specific nt values used per task.
3.4 Detailed, Sentence-Based Technical Breakdown
This is primarily a systems paper whose core idea is that GPU memory constraints, not algorithmic limitations, are the bottleneck for applying RL to diffusion language models, and that a specific reformulation of the training objective — trading a nonlinear computation graph for a linear one via a mathematically equivalent lower bound — removes this bottleneck with no performance penalty.
The VRPO Baseline: ELBO-Based Log-Likelihood Approximation
Before explaining BGPO, we must understand the objective it bounds. The paper builds directly on VRPO (Zhu et al., 2025), which addresses the fundamental intractability of log π_θ(y|x) in diffusion language models.
Why the log-likelihood is intractable. In a masked diffusion language model, generating a response y from a prompt x involves an iterative denoising process. Starting from a fully masked sequence at time t=1, the model progressively unmasks tokens as t decreases from 1 to 0, with each unmasking step conditioned on the currently visible tokens. The probability of ending up with a specific sequence y is an integral over all possible paths through this denoising process — all possible sequences of partial masks that could lead to y. This integral cannot be computed in closed form. The paper states this succinctly in Section 2.2:
"the iterative denoising generation makes the exact computation of log π_θ(y|x) intractable."
The ELBO as a variational proxy. Rather than computing log π_θ(y|x) exactly, VRPO uses its evidence lower bound (ELBO). The ELBO for masked diffusion models has a specific structure given in Eq. 3:
where $B_{\pi_\theta}(y|x)$ is the ELBO serving as a lower bound on the true log-likelihood $\log \pi_\theta(y|x)$, $t \sim U[0,1]$ means the masking time is sampled uniformly from the continuous interval 0 to 1, $y_t \sim q(\cdot|t,y,x)$ denotes sampling a partially masked version of the response y according to the forward masking process at time t, and $\ell_{\pi_\theta}(y_t, t, y|x)$ is a per-sample loss measuring how well the model predicts the original tokens.
What it computes: the ELBO averages the model's prediction quality across all possible masking times and all possible partially masked versions of the response. The expectation integrates over the continuous time range [0,1] with uniform weighting, meaning the model is evaluated equally at early denoising stages (high t, many tokens masked) and late stages (low t, few tokens masked). Each evaluation asks: "given this partially masked version of the response, how well does the model recover the original tokens at the masked positions?"
Why this form: the ELBO is the standard variational inference objective for latent variable models where the latent variables are the masking trajectories. Using the ELBO as a proxy for the log-likelihood is justified by the theoretical property that as the model becomes better trained, the gap between the ELBO and the true log-likelihood shrinks. The paper cites Ho et al. (2020) and Song et al. (2021) for this property:
"when the sample size nt is large enough, the bias of for a well-trained model relative to will become negligible"
The per-sample loss ℓ_πθ in detail. Eq. 4 specifies the term inside the expectation:
where $|y_t|$ is the length of the partially masked sequence, $\mathbb{1}[y_i^t = M]$ is an indicator that evaluates to 1 if position i is masked (containing the special mask token M) and 0 otherwise, $\log p_\theta(y_i|y_t, x)$ is the model's log-probability prediction for the true token y_i at position i given the partially masked context y_t and the prompt x, and the factor $\frac{1}{t}$ is a time-dependent normalization.
What it computes: for a given partially masked response y_t, the model makes predictions at every masked position. At each masked position i, the model outputs a probability distribution over the vocabulary, and we extract the log-probability it assigns to the correct token y_i. These log-probabilities are summed only over masked positions, then divided by t. The division by t is crucial: at early times (t close to 1), most tokens are masked, so the sum would be naturally larger — dividing by t normalizes across time scales so that contributions from different t values are comparable.
Why this form: the 1/t normalization is specific to the masked diffusion formulation used in LLaDA (Nie et al., 2025b) and derives from the continuous-time diffusion loss. Without it, the loss at high t (many masked tokens) would dominate the expectation, biasing the approximation. The indicator 1[y_i^t = M] ensures that positions where the token was already unmasked (and thus visible to the model as context) are not used for evaluation — the model should not get credit for "predicting" a token it can already see.
Monte Carlo estimation of the ELBO. Because the expectation in Eq. 3 cannot be computed analytically (it integrates over the continuous uniform distribution over t and the combinatorial space of possible masks), VRPO estimates it via Monte Carlo sampling (Eq. 5):
where $n_t$ is the number of Monte Carlo samples, $t^{(j)}$ is the j-th independently sampled masking time from Uniform[0,1], and $y_t^{(j)}$ is the j-th independently sampled partially masked response conditioned on t^(j).
What it computes: rather than integrating over all possible t and y_t, this approximates the ELBO by averaging ℓ_πθ over nt randomly sampled (t, y_t) pairs. As nt → ∞, the law of large numbers guarantees this average converges to the true expectation. For finite nt, there is variance in the estimate, and this variance scales as O(1/√nt).
Why this form: Monte Carlo is the only tractable way to estimate high-dimensional expectations when the sampling distribution is known (we can sample t and y_t but cannot integrate analytically). The uniform distribution over t is particularly convenient — no importance weights are needed because we are sampling directly from the distribution inside the expectation.
Plugging the ELBO estimate into the RL objective. The standard RL objective for language models maximizes the expected advantage-weighted log-ratio (Eq. 1). With the ELBO-based log-likelihood approximation substituted in, the objective becomes Eq. 6-7:
where
and $\hat{B}_{\pi_\theta}(y|x)$ is the Monte Carlo ELBO estimate for the current policy, $\hat{B}_{\pi_{old}}(y|x)$ is the same estimate computed under the frozen old policy (using the same nt random samples for a paired comparison), and $A(x, y)$ is the sequence-level advantage.
What it computes: for each generated response y, the exponentiated difference between the current and old policy's ELBO estimates serves as an importance weight (the ratio π_θ/π_θold approximated via ELBOs). This weight is multiplied by the advantage A(x,y) — if the response was better than average (positive advantage), the weight encourages the policy to increase its probability of generating similar responses; if worse (negative advantage), the policy is discouraged from generating such responses.
Why this form: this is the standard policy gradient objective (Schulman et al., 2017; Shao et al., 2024) with the log-likelihood ratio approximated by the ELBO ratio. The exponential converts the log-ratio into a ratio in probability space. The advantage weighting ensures that responses are weighted by how much better or worse they are than the baseline, not by their absolute reward — this reduces variance in the gradient estimate.
The Memory Bottleneck: Why nt Must Be Small
The critical technical problem that BGPO addresses lies in how automatic differentiation handles Eq. 7. This subsection explains the mechanism in detail, since the entire paper's contribution depends on understanding this one engineering constraint.
The computation graph for . To compute , the following operations occur in sequence:
- For j = 1 to nt: sample t^(j) and y_t^(j)
- For j = 1 to nt: run the current model p_θ forward on (y_t^(j), x) to compute log p_θ(y_i|y_t^(j), x) at each masked position
- For j = 1 to nt: compute ℓ_πθ(y_t^(j), t^(j), y|x) using Eq. 4
- Average the nt ℓ_πθ values to get (y|x)
- Do the same for the old policy to get (y|x)
- Compute the difference d = -
- Exponentiate: e^d
- Multiply by A(x,y): = e^d * A(x,y)
Where the memory cost comes from. Step 7 — the exponential — is the critical operation. The exponential function e^d is nonlinear in d, and d is a function of all nt forward passes through the current policy model (steps 1-4). When automatic differentiation computes the gradient of with respect to the model parameters θ, it must apply the chain rule through this nonlinear function:
Since d = (1/nt) Σ_j ℓ^(j), the gradient ∇_θ d requires gradients through each of the nt forward passes. The automatic differentiation engine (PyTorch in practice) must retain the intermediate activations — the hidden states, attention weights, and logits — from all nt forward passes simultaneously because they are all needed to compute ∇_θ d, and therefore all needed to compute ∇_θ .
The paper states this explicitly in Section 2.2:
"all the nt computational graphs must be retained in the memory for calculating the gradient of the exponential function in Eq. 7"
Quantifying the memory cost. Each forward pass through an 8B-parameter model with a 512-token sequence produces tensors whose total size is proportional to (model parameters × sequence length). For modern transformer architectures, the activation memory (intermediate values that must be stored for backpropagation) typically dominates the parameter memory. With nt forward passes, the activation memory scales linearly with nt — each additional MC sample adds a full set of intermediate tensors.
The left panel of Figure 1 quantifies this for the H800 GPU (80GB memory). VRPO-OL's memory usage climbs from approximately 15GB at nt=1 to roughly 37GB at nt=4, and exceeds 70GB at nt=8 before hitting the 80GB hardware ceiling. At nt=16 (which the paper's gradient analysis shows would provide substantially better approximation quality), VRPO-OL would require far more than 80GB.
The practical consequence, stated in Section 2.2:
"in practice, the sample size can only remain small (e.g., nt = 4), which results in inaccurate approximations for the likelihoods as well as the final objective, seriously affecting the final performance."
Why this matters for RL training quality. The Monte Carlo estimate (y|x) has variance that decreases with nt, but with nt=4, the standard deviation of the estimate around the true ELBO is roughly 1/√4 = 50% of the per-sample standard deviation. This noise propagates through the exponential (which amplifies errors nonlinearly) and into the gradient, meaning the optimization signal the model receives is substantially corrupted. The model may take gradient steps in directions that reflect sampling noise rather than genuine policy improvement — especially early in training when the policy is changing rapidly and the ELBO estimates are less reliable.
Figures 3 and 4 quantify this: VRPO-OL at nt=4 has approximately 2-2.5× the gradient bias of BGPO at nt=16, and approximately 40-50% higher gradient standard deviation. These are not small differences — a gradient vector with twice the noise will require roughly four times as many steps to converge with the same reliability.
The Core Innovation: Boundary-Guided Lower Bound Construction
BGPO's solution is to replace (x,y) with a constructed lower bound (x,y) that satisfies two properties: (1) it can be expressed as a linear sum of per-MC-sample terms (enabling gradient accumulation), and (2) it is exactly equal to (x,y) in value and gradient during on-policy training (so the optimization signal is not compromised). The construction uses different mathematical techniques for positive and negative advantages because the direction of the inequality in a lower bound must be consistent with the optimization objective.
The paper defines the overall lower-bound objective in Eq. 8:
where the expectation over prompts and responses is identical to the original objective, but the per-response term (x,y) is a constructed lower bound of (x,y): $\hat{R}_{lb}(x,y) \leq \hat{R}(x,y)$.
What it computes: the same expected advantage-weighted quantity as the original RL objective, but using a surrogate per-response term that is guaranteed to be less than or equal to the original. Maximizing this lower bound guarantees improvement in a lower bound on the true objective, which is a standard technique in variational inference and EM-style algorithms.
Why this form: by constructing the lower bound to have specific structural properties (linearity), the computational graph becomes decomposable, enabling memory-efficient gradient computation. The fact that it is a lower bound (rather than an upper bound or an approximation of unknown direction) ensures that maximizing it cannot over-optimize relative to the true objective — improvements in the lower bound are genuine improvements.
The dependent variable: the ELBO difference d_j. The construction centers on a per-sample quantity introduced in Eq. 10:
where $\ell_{\pi_\theta}(y_t^{(j)}, t^{(j)}, y|x)$ is the ELBO contribution for the j-th MC sample under the current policy, and $\ell_{\pi_{old}}(y_t^{(j)}, t^{(j)}, y|x)$ is the same under the frozen old policy, evaluated on the identical (t^(j), y_t^(j)) pair.
What it computes: for a given partially masked sample, d_j measures how much better (or worse) the current policy is at predicting the correct tokens compared to the old policy. If the current policy assigns higher probability to the correct tokens, d_j is positive; if lower, d_j is negative. This per-sample difference is the atomic unit from which the importance weight is built.
Why this form: using the same random (t^(j), y_t^(j)) for both policies provides paired comparison that reduces variance (common random numbers in Monte Carlo). The difference d_j isolates the policy change signal — the randomness of the masking process affects both policies equally, so differencing them cancels out shared noise.
The full ELBO difference. The average of d_j across all nt samples gives the ELBO difference that appears in the exponent of Eq. 7:
This is the quantity d from step 6 of the computation graph description above. The exponential in Eq. 7 is e^{(1/nt) Σ_j d_j}.
Case 1: Positive advantage — Taylor expansion lower bound. When A(x,y) ≥ 0, the paper applies Lemma 1 (first-order Taylor expansion of the exponential):
"For any δ ∈ R, the exponential function satisfies e^δ ≥ 1 + δ."
Starting from Eq. 7 with δ = (1/nt) Σ_j d_j and multiplying both sides by the non-negative advantage A(x,y):
The right-hand side can be rewritten as a sum of per-sample terms (Eq. 9):
What it computes: each MC sample contributes a term (1 + d_j)A(x,y)/nt to the lower bound. The factor 1 + d_j is a first-order approximation of e^{d_j} (which would be exact if d_j were zero), and the summation over j with division by nt mirrors the structure of a sample average. Since the advantage A(x,y) is non-negative, the inequality direction is preserved when multiplying through.
Why this form — linearity: each term g_j = (1 + d_j)A(x,y)/nt depends only on d_j, which depends only on the j-th MC sample's forward pass through the current policy. The gradient of g_j with respect to θ can be computed by backpropagating through only the j-th forward pass — there is no cross-term coupling between different j. After computing ∇_θ g_j, the intermediate tensors from the j-th forward pass can be freed, and the process continues to j+1. The total gradient is the sum of per-sample gradients: ∇_θ = Σ_j ∇_θ g_j.
Why this form — the Taylor expansion choice: the first-order Taylor expansion e^δ ≥ 1 + δ is the tightest linear lower bound on the exponential function for δ in a neighborhood of zero. During on-policy training, d_j is close to zero (since the current and old policies are similar early in training, and identical at the moment the old policy is refreshed), so the approximation error is small. The linearity of 1 + δ is what enables the decomposition — a higher-order expansion (e.g., 1 + δ + δ²/2) would introduce terms like d_j · d_k that couple different MC samples together, defeating the purpose.
Case 2: Negative advantage — Jensen's inequality lower bound. When A(x,y) < 0, the Taylor expansion approach would not produce a valid lower bound because multiplying an inequality by a negative number reverses its direction. If we applied e^{(1/nt) Σ d_j} ≥ 1 + (1/nt) Σ d_j and then multiplied by A(x,y) < 0, we would get e^{(1/nt) Σ d_j} · A(x,y) ≤ (1 + (1/nt) Σ d_j) · A(x,y), which is an upper bound — maximizing it would not guarantee improvement relative to the true objective.
Instead, the paper applies Lemma 2 (Jensen's inequality):
"For a convex function f and a finite set {x_i}ⁿ_{i=1}, we have f((1/n) Σ_i x_i) ≤ (1/n) Σ_i f(x_i)."
The exponential function is convex, so applying Jensen's inequality with x_j = d_j and f(x) = e^x yields:
Now multiply both sides by the negative advantage A(x,y) < 0. Multiplying by a negative number reverses the inequality:
The right-hand side simplifies to Eq. 12:
What it computes: each MC sample contributes a term e^{d_j} A(x,y)/nt. Unlike the Taylor case where the exponential is approximated linearly, here the exponential is applied per-sample before averaging. Since A(x,y) is negative, each term e^{d_j} A(x,y)/nt is negative (or zero if d_j is very negative, making e^{d_j} very small). The sum weights each sample by how much the current policy improves over the old policy at that particular (t, y_t) pair — samples where the current policy is much better (d_j large positive, e^{d_j} large) contribute more strongly to reducing the loss (remember A is negative, so this term is more negative, which when negated in the loss function becomes a larger penalty for deviating from the old policy).
Why this form — linearity: as with the positive-advantage case, each g_j = e^{d_j} A(x,y)/nt depends only on the j-th MC sample. The gradient backpropagates through e^{d_j}, which involves only the j-th forward pass. Memory can be freed after each j.
Why this form — the Jensen choice: Jensen's inequality is the natural tool for converting an exponential of an average into an average of exponentials while maintaining a bound in the needed direction. For convex f, f(average) ≤ average of f, meaning the exponential of the average d_j is less than or equal to the average of the per-sample exponentials. When multiplied by a negative advantage, this gives a lower bound on the original term, which is what we need for a lower bound on the full objective. The per-sample exponentials e^{d_j} are individually computed and then averaged, achieving the linear decomposition.
Unified per-sample term g_j. Eq. 13 consolidates both cases into a single definition:
where $g_j$ is the per-MC-sample contribution to the lower bound, $d_j$ is the per-sample ELBO difference from Eq. 10, $A(x,y)$ is the sequence-level advantage, and $n_t$ is the total number of MC samples.
What it computes: g_j is the atomic unit of the BGPO objective — the contribution of one MC sample to the per-response lower bound. The form depends on the advantage sign to ensure the overall bound direction is preserved: Taylor expansion for non-negative advantages, Jensen-based exponential for negative advantages.
Why this form: this case split is the key design choice that enables both linearity and correctness. Without splitting on advantage sign, a single bounding technique could not work: Taylor expansion fails for negative advantages (inequality reverses), and Jensen alone for positive advantages would give an upper bound (since multiplying Jensen's inequality e^{average} ≤ average of e by a positive number preserves the ≤ direction, resulting in an upper bound rather than lower bound). The sign-dependent construction is necessary for the lower bound property.
Final linear lower bound. Eq. 14 assembles the per-sample terms:
What it computes: the per-response lower bound is simply the sum of the nt independent g_j terms. Each g_j requires only one forward pass through the current policy and one through the old policy (or equivalently, reusing the old policy's cached values), and their gradients are independent.
Why this form:
The sum-of-per-sample-terms structure is what enables Algorithm 1's memory-efficient implementation. In the inner loop (lines 9-10), each g_j is computed (Lj = -g_j/G for the loss), its gradient is backpropagated, and the intermediate tensors from that iteration are freed before proceeding to j+1. The accumulated gradient across all j for all G responses is then used for the parameter update in line 14. Memory usage is proportional to the cost of one forward+backward pass, not nt forward passes, making it constant in nt.
The On-Policy Equivalence Proof
A natural concern with maximizing a lower bound rather than the original objective is that the optimum of the lower bound might not match the optimum of the true objective — you might converge to a policy that maximizes the bound but not the actual expected reward. BGPO avoids this concern because the lower bound is designed to be exactly equal to the original objective during on-policy training.
The key insight of the proof. When training is on-policy — meaning the current policy π_θ is identical to the old policy π_θold used for generating responses — the per-sample ELBO difference d_j is exactly zero for all j:
This is because π_θ and π_θold are literally the same model with the same parameters, so they produce identical predictions for every input. With d_j = 0, both the Taylor- and Jensen-based constructions simplify to the same value.
Value equivalence. When d_j = 0 for all j, Eq. 7 gives:
For the lower bound with A(x,y) ≥ 0 (Eq. 9): g_j = (1 + 0)A(x,y)/nt = A(x,y)/nt, so Σ_j g_j = nt · A(x,y)/nt = A(x,y).
For A(x,y) < 0 (Eq. 12): g_j = e^0 · A(x,y)/nt = A(x,y)/nt, so Σ_j g_j = nt · A(x,y)/nt = A(x,y).
In both cases, (x,y) = A(x,y) = (x,y). The lower bound and the original objective have exactly the same value when evaluated on-policy.
Gradient equivalence. The paper proves gradient equivalence in Eqs. 15-17, demonstrating that not only the values match, but the optimization direction is identical.
For the original objective (Eq. 15):
Evaluating at d_j = 0: e^0 = 1, so:
For the Taylor-based lower bound with A(x,y) ≥ 0 (Eq. 16):
For the Jensen-based lower bound with A(x,y) < 0 (Eq. 17):
Evaluating at d_j = 0 (so e^{d_j} = 1):
All three expressions reduce to the identical sum Σ_j A(x,y) ∇_θ d_j / nt. The gradient is exactly the same.
What this equivalence means in practice. At the start of each training iteration (Algorithm 1, line 2), the old policy π_θold is refreshed to match the current policy π_θ. Immediately after this refresh, training is exactly on-policy — the responses in the current batch were just generated from π_θold = π_θ, and the gradient computed through matches the gradient that would be computed through exactly. As training proceeds within the iteration (multiple gradient steps on the same batch of responses), π_θ diverges slightly from π_θold, and the lower bound becomes a genuine approximation with a small gap. But because the old policy is refreshed every iteration, the policy never drifts far from on-policy, and the approximation error introduced by the bounding construction remains small.
Why this property is critical: it means BGPO is not sacrificing optimization quality for memory efficiency. The lower bound is not a crude approximation that might lead to suboptimal policies. It is exactly the original objective when the policy matches the data distribution, and a provably tight bound when the policy is nearby. The only change is in how the gradient is computed — via independent per-sample contributions rather than a coupled nonlinear graph — not in what gradient is ultimately delivered to the optimizer.
Algorithm Pseudocode Walkthrough
Algorithm 1 specifies the complete BGPO training loop. Each line maps to a concrete computational operation.
Line 2: "Update the old policy π_θold ← π_θ and sample a batch D_b from D." This copies the current model parameters to create a frozen snapshot. All response generation and advantage computation in the current iteration will use this frozen model, ensuring the responses are on-policy at the start of the iteration. The batch D_b contains a set of prompts (batch size 16 in the paper's experiments).
Lines 3-4: Response generation. For each prompt x in the batch, sample G = 8 responses from π_θold using the full iterative denoising process with block-wise decoding. The response length is capped at 512 tokens for math and coding tasks, 256 for planning tasks. The denoising uses 256 steps during training for math tasks, 512 for coding, and 128 for planning (Table 4). Compute advantages for each response using Eq. 18.
Lines 5-11: Per-response MC sampling and gradient accumulation. This is the core of BGPO's memory efficiency. For each response y^(i):
- Line 6: Sample nt independent masking times from Uniform[0,1]. These determine how much of the response is masked for each MC sample.
- Lines 7-10: For each MC sample j:
- Line 8: Apply the forward masking process to y^(i) at time t^(j): each token independently becomes masked with probability t^(j), remains unchanged with probability 1-t^(j). This produces y^(i)_t^(j).
- Line 9: Compute g_j using Eq. 13. This requires one forward pass through the current policy p_θ (to get ℓ_πθ) and one lookup of the cached ℓ_πold value (computed and stored during or immediately after response generation). Let L_j = -g_j / G (negating for gradient descent on the loss, dividing by G to average over the group).
- Line 10: Backpropagate the gradient of L_j through the current policy. After backpropagation completes, the computational graph for this j can be discarded. Only the accumulated gradient buffers in the model parameters persist.
Line 14: "Update the policy θ ← θ - η ∇_θ." After all G·nt gradient contributions have been accumulated, take one optimizer step using learning rate η = 5 × 10⁻⁷. The gradient ∇_θ is the sum of all per-sample contributions, and the AdamW optimizer applies its standard moment estimates and weight decay.
Output: π_θ. The updated policy becomes the starting point for the next iteration.
Why this loop structure: the triple-nested loop (prompts → responses → MC samples) ensures that at any moment, only one MC sample's forward computational graph exists in GPU memory. The inner loop (lines 7-10) is the key innovation — it replaces the coupled computation of Eq. 7 with independent per-sample gradient contributions that can be sequentially computed and accumulated.
Group-Based Advantage Estimation
Eq. 18 defines the advantage function used in BGPO:
where $r(x, y^{(i)})$ is the scalar reward for the i-th response to prompt x, $\text{mean}(\{r(x, y^{(j)})\}_{j=1}^G)$ is the average reward across all G responses generated for the same prompt, and $\text{std}(\{r(x, y^{(j)})\}_{j=1}^G)$ is the standard deviation of those G rewards.
What it computes: for each response to a given prompt, the advantage is the z-score — how many standard deviations above or below the group mean the response's reward falls. Responses with above-average rewards get positive advantages; below-average get negative advantages. The normalization (subtracting mean, dividing by standard deviation) ensures that advantages are zero-mean and unit-variance within each prompt group.
Why this form: group-based advantage normalization (used in GRPO; Shao et al., 2024) eliminates the need for a learned value function baseline, which would be additional model complexity. Within each prompt group, the relative quality of responses is measured against each other, which is appropriate for tasks where the absolute reward scale varies across prompts. The standardization prevents prompts with large reward variance from dominating the gradient signal and prompts with small variance from being ignored. This is the same advantage formulation used in diffu-GRPO and VRPO-OL, ensuring fair comparison.
Final Loss Formulation
Eq. 19 defines the training loss:
where $\mathcal{L}_{BGPO}$ is the scalar loss to be minimized (negated expected reward), $x \sim \mathcal{D}$ means prompts are drawn from the training dataset, $\{y^{(i)}\}_{i=1}^G \sim \pi_{\theta_{old}}(\cdot|x)$ means G responses are sampled from the old policy for each prompt, and the inner average $\frac{1}{G} \sum_i \hat{R}_{lb}(x, y^{(i)})$ is the mean lower bound over the G responses.
What it computes: the loss is the negative of the average lower bound across all G responses for all prompts in the batch. Minimizing this loss is equivalent to maximizing the expected lower bound objective (θ). The G responses provide multiple samples per prompt, enabling the group-based advantage normalization within each prompt group.
Why this form: the expectation is approximated by the empirical average over the current batch. The averaging over G responses per prompt provides a more stable gradient estimate than using a single response. The negation converts the maximization of reward into the standard gradient descent framework used by deep learning optimizers.
Hyperparameter and Implementation Details
The paper uses consistent hyperparameters across tasks (from Table 4 and the main text):
Model and optimizer:
- Base model: LLaDA-8B-Instruct (Nie et al., 2025b), a pre-trained and supervised-fine-tuned masked diffusion language model
- Optimizer: AdamW (specific betas not explicitly stated, but following the VeRL framework's defaults)
- Learning rate η = 5 × 10⁻⁷
- Batch size = 16 prompts per iteration
- Rollout group size G = 8 responses per prompt
- Training framework: VeRL (Sheng et al., 2025)
Task-specific settings:
- Mathematics (MATH + GSM8K training): max response length 512 tokens (both training and evaluation), 256 diffusion steps (training), 512 steps (evaluation), block size 32, nt = 16 for BGPO
- Coding (DeepCoder training, MBPP/HumanEval evaluation): max response length 512 tokens, 512 diffusion steps, block size 32, nt = 16 for BGPO, nt = 2 for VRPO-OL (lower due to longer prompts)
- Sudoku: max response length 256 tokens, 128 diffusion steps (training), 256 steps (evaluation), block size 32, nt = 32 for BGPO, nt = 4 for VRPO-OL
- Countdown: max response length 256 tokens, 128 diffusion steps (training), 256 steps (evaluation), block size 32, nt = 16 for BGPO, nt = 4 for VRPO-OL
Block-wise decoding: both training and evaluation use a block-wise decoding strategy where the denoising process generates blocks of 32 tokens in parallel rather than individual tokens. This is standard for dLLMs following previous works (Zhu et al., 2025; Zhao et al., 2025a) and provides a tradeoff between generation speed and quality.
Evaluation protocol: models are evaluated every 20 training steps, and results are reported from the best-performing checkpoint (following Zhao et al., 2025a). The evaluation uses slightly more diffusion steps (512 vs. 256 for math, 256 vs. 128 for planning) to ensure high-quality generation at test time, while training uses fewer steps for efficiency.
Hardware: all experiments use 8 × H800 GPUs (80GB each).
Why these choices:
- The higher nt values for BGPO (16-32) compared to VRPO-OL's maximum (2-4) are the key experimental variable — BGPO's memory efficiency is what enables these larger sample sizes.
- Sudoku receives nt=32 because its shorter responses (256 tokens) leave more memory headroom, and the discrete combinatorial nature of the task may benefit from even more accurate likelihood approximation.
- The block-wise decoding and diffusion step counts follow established practice from prior dLLM work to ensure fair comparison — the innovation is in the RL objective formulation, not in the generation process.
- The learning rate 5×10⁻⁷ is relatively small, appropriate for fine-tuning a pre-trained model where large updates could destabilize the policy.
Summary of Design Choices and Their Justifications
- Lower bound over direct optimization: BGPO maximizes a bound rather than the original objective because the bound's linear structure enables memory-efficient gradient computation. The on-policy equivalence proof ensures this is not a compromise — at the point where gradients are computed (on-policy, immediately after policy refresh), the bound is exact.
- Taylor expansion for positive advantages, Jensen for negative: the sign-dependent construction is mathematically necessary because a single bounding technique cannot maintain the lower bound property across both sign regimes of the advantage. Taylor expansion gives a valid lower bound only when multiplied by a non-negative advantage; Jensen's inequality provides the correct bound direction when multiplied by a negative advantage.
- Gradient accumulation over gradient averaging: by computing and backpropagating each g_j independently and accumulating gradients, memory usage is O(1) in nt rather than O(nt). This is the enabling mechanism for large nt.
- Group-based advantage estimation: z-score normalization within each prompt's response group eliminates the need for a value function and provides a relative quality signal robust to per-prompt reward scale variation.
- Block-wise decoding throughout: consistent with prior dLLM training recipes, this generates text efficiently while maintaining quality sufficient for the RL training signal.
4. Key Insights and Innovations
Innovation 1: Diagnosing the Memory Bottleneck as the True Limiter of RL for dLLMs — Not Algorithm Quality
The paper's most intellectually distinctive move is not proposing a new algorithm, but correctly identifying what kind of problem this is. Prior work on RL for diffusion language models framed the challenge as fundamentally algorithmic: "approximating the intractable log-likelihood is hard, and we need better approximations." This framing led to two parallel research tracks — one pursuing cheap but biased single-pass estimates (diffu-GRPO; Zhao et al., 2025a), the other pursuing principled but expensive ELBO estimates with necessarily small Monte Carlo sample sizes (VRPO; Zhu et al., 2025). Both accepted their respective tradeoffs as inherent to the method: bias for diffu-GRPO, noise for VRPO.
BGPO's diagnostic reframes the problem entirely. The true bottleneck is not that ELBO-based approximation is algorithmically unsound at small nt — the theory says it works if nt is large enough. The bottleneck is that GPU memory constraints prevent nt from being large enough for the theory's conditions to hold. This is a systems problem disguised as an algorithmic problem: the RL objective gradient computation couples all nt forward passes into a single nonlinear computational graph, forcing memory to scale linearly with nt and capping feasible nt at 4 (or even 2 for longer prompts).
This reframing matters because it changes the research direction. If the problem were algorithmic — "ELBO approximation is inherently noisy" — the solution would be to find a fundamentally different likelihood estimator. But once the problem is recognized as a memory bottleneck in gradient computation, the solution is to decouple the computational graph, preserving the same ELBO-based approximation but computing its gradient through a different path. This is a much more tractable target, and it explains why BGPO achieves substantial gains without changing the underlying likelihood estimator: VRPO's approach was correct all along; it just couldn't afford enough samples to make it work.
The quantitative evidence for this diagnosis is in Figures 3 and 4. When VRPO-OL raises nt from 2 to 4, gradient bias and variance both decrease — showing the estimation would improve with more samples if memory permitted. BGPO simply continues this trajectory to nt=16, where bias and variance drop substantially further. The performance gap in Table 1 between VRPO-OL (nt=4, MATH500: 44.1%) and BGPO (nt=16, MATH500: 45.7%) is not a difference in algorithmic quality — it is the difference between running the same algorithm at the suboptimal sample size the hardware allows versus the sample size the theory needs.
This diagnostic contribution is fundamental: it reclassifies the bottleneck from algorithmic (requiring new mathematical ideas about likelihood approximation) to systems-level (requiring a reformulation of gradient computation), opening a different and more immediately actionable solution space.
Innovation 2: The Boundary-Guided Lower Bound as a General Strategy for Linearizing Nonlinear Reinforcement Learning Objectives
The specific mathematical construction — splitting on the sign of the advantage to apply different bounding techniques (Taylor expansion for positive, Jensen's inequality for negative) — is an instance of a more general design pattern that the paper establishes: when an RL objective contains a nonlinear function of a Monte Carlo average, you can construct a linear lower bound that decomposes across MC samples, provided the bound remains tight (exact) on-policy.
This is not a standard technique in the RL literature. Policy gradient methods typically handle nonlinear objectives by either (a) accepting the coupled computation graph and paying the memory cost, (b) using score function estimators (REINFORCE) that avoid backpropagating through the nonlinearity but introduce high variance, or (c) constructing surrogate objectives (like PPO's clipped objective) that are structurally simpler than the original. BGPO's approach — using mathematically grounded inequalities to produce a decomposable lower bound that is exact at the current policy — does not fit neatly into any of these categories. It is closest to variational EM, where a lower bound is constructed and maximized, but the sign-dependent construction and the emphasis on gradient decomposability (rather than tractability of the bound integral) are specific to the memory-constrained deep learning setting.
The design requirements the paper makes explicit — linearity and on-policy equivalence — define a template for solving similar problems in other domains where an RL objective depends on an expensive Monte Carlo estimate inside a nonlinearity. As long as you can identify: (1) the nonlinear function you need to linearize, (2) a bounding inequality that produces a valid lower bound in both sign regimes of the multiplier (here, the advantage), and (3) the condition under which the bound becomes exact (here, on-policy, where d_j = 0), you can adapt the BGPO pattern. The paper does not claim or explore this generality, but the structure is there: exponential of a sample average, sign-dependent bounding, on-policy collapse to exactness.
This is a fundamental contribution at the level of technique design patterns, not merely an application of known inequalities. The specific inequalities (Taylor first-order, Jensen) are standard; the recognition that splitting on advantage sign enables both to produce a valid lower bound with the right linearity properties is the creative step. A naïve application of Jensen alone would produce an upper bound for positive advantages (multiplying e^{average} ≤ average of e by a positive number preserves ≤), failing the lower-bound requirement. The Taylor expansion alone would reverse direction for negative advantages. The combination, gated by advantage sign, is what makes the construction work — and this gating is not an obvious consequence of either inequality alone.
Innovation 3: Establishing That Larger Monte Carlo Sample Sizes Directly Improve Policy Optimization Quality in dLLMs
Prior to BGPO, the relationship between nt (MC sample size for ELBO estimation) and final model quality in dLLM reinforcement learning was assumed but not demonstrated. Theory from the diffusion model literature (Ho et al., 2020; Song et al., 2021) established that the ELBO converges to the true log-likelihood as nt → ∞, but this is a property of the likelihood estimator, not of the full RL training loop. It was unclear whether improving this approximation from nt=4 to nt=16 would produce a measurable improvement in final task performance, since RL training involves many interacting factors: advantage estimation noise, policy drift, reward signal sparsity, and the inherent stochasticity of mini-batch training.
BGPO provides the first clean experiment to test this question. Because its memory usage is constant in nt, the same algorithm can be run at nt = 1, 2, 4, 8, and 16 without any other changes to the training procedure — same batch size, same learning rate, same response length, same number of GPUs. This isolates the effect of nt from all confounds. Table 2 demonstrates the result: MATH500 accuracy rises monotonically from 43.5% at nt=1 to 45.7% at nt=16. The improvement is not dramatic in absolute terms (2.2 percentage points from nt=1 to nt=16), but it is consistent, and the trajectory shows no sign of saturated returns — suggesting further increases in nt might yield further gains if memory and computation permitted.
More important than the performance numbers is the mechanistic evidence in Figures 3 and 4. By using BGPO at nt=256 as a proxy for the "true" gradient, the paper shows that both bias (systematic deviation from the true gradient direction) and variance (random fluctuation) decrease monotonically as nt increases. This closes the causal chain: larger nt → better gradient signal → better optimization → better final performance. It converts the theoretical claim ("ELBO improves with more samples") into an empirically validated engineering fact ("RL training for dLLMs improves with more MC samples, and the mechanism is gradient quality, not something else").
This is an incremental but practically important finding. It doesn't change the theory of RL for dLLMs, but it provides the empirical foundation for arguing that memory-efficient gradient computation (like BGPO's) is worth investing in — because the gains from larger nt are real and measurable, not merely theoretical asymptotic guarantees.
Innovation 4: Demonstrating That Memory-Efficient Gradient Accumulation Costs Nearly Zero Wall-Clock Time Relative to Response Generation
A concern that any practitioner reading Section 3 would have is: "If BGPO processes each MC sample sequentially with separate backpropagation, doesn't that serialize what was previously parallel, making each training step much slower?" This is a legitimate worry because gradient accumulation — computing per-sample gradients sequentially and summing them — trades parallelism (all samples processed simultaneously in one large batch) for memory (only one sample's graph in memory at a time), and serialization typically costs wall-clock time.
Figure 5 provides the counterintuitive result: BGPO at nt=16 takes 151.5 seconds per training step versus VRPO-OL at nt=4 taking 130.2 seconds — only a 16% increase despite 4× more MC samples. diffu-GRPO at nt=1 takes 128.8 seconds. The paper's explanation (Section 4.5) identifies the cause: the dominant cost of each training step is response rollout — generating G=8 full responses per prompt through the full iterative denoising process (128-512 diffusion steps, each requiring a model forward pass). This cost is identical across methods. The MC sampling and gradient computation cost is small relative to generation, so even serializing it across 16 samples adds minimal overhead.
This result is not merely a pleasant surprise — it is diagnostically significant because it reveals where the true computational bottleneck in dLLM RL training lies. If gradient computation were the dominant cost, BGPO's sequential accumulation would be a serious practical limitation despite its memory advantages. But because generation dominates, BGPO's approach is essentially "free" in wall-clock terms — the paper has found a way to improve gradient quality (via larger nt) at negligible time cost, by targeting the part of the computation that was not the bottleneck anyway.
This finding has implications beyond BGPO. It suggests that any technique that improves the efficiency of the RL objective computation (gradient variance reduction, better advantage estimation, more accurate likelihood approximation) is likely to have minimal wall-clock overhead as long as it doesn't increase the number or cost of response rollouts. The response generation step is the computational bottleneck, and optimizations elsewhere in the loop are of secondary importance to total training time. This reframes how researchers should think about algorithmic improvements to dLLM RL: generation cost dominates, so focus algorithmic effort there; memory efficiency in the objective computation is about feasibility (can you run the method at all?) rather than speed.
This is an incremental insight about system dynamics, not a new algorithm or theory, but it has practical consequences for research prioritization in this area.
5. Experimental Analysis
Evaluation Methodology
-
Datasets. The paper conducts RL experiments across three domains. For math problem solving, training uses a mix of MATH (Hendrycks et al., 2021) and GSM8K (Cobbe et al., 2021) training splits, with evaluation on the respective test sets (MATH500 and GSM8K). For code generation, training uses 16K medium-difficulty problems filtered from DeepCoder (Luo et al., 2025), with evaluation on MBPP (Austin et al., 2021b) and HumanEval (Chen et al., 2021). For planning, both training and evaluation use Countdown (Pan et al., 2025) and Sudoku (Arel, 2025), adopting the same train/test splits as d1 (Zhao et al., 2025a).
-
Base model. All experiments use LLaDA-8B-Instruct (Nie et al., 2025b), an 8-billion-parameter masked diffusion language model that has undergone both pre-training and supervised fine-tuning. The paper states this model is chosen because it represents the state-of-the-art among open-source dLLMs and sits at a scale where RL fine-tuning can meaningfully improve performance. The 8B scale is also a practical constraint: larger open-source dLLMs did not exist at the time of the work (Section 7).
-
Metrics. The primary metric is task-specific accuracy: for MATH500 and GSM8K, exact match with ground-truth answers; for HumanEval and MBPP, pass@1 on coding problems; for Sudoku and Countdown, task-completion accuracy. During training, the paper also reports reward (Figure 2), which is the task-specific scoring function used to compute advantages. Training reward dynamics are monitored as a proxy for optimization progress, but final evaluation uses only test-set accuracy from the best-performing checkpoint (checked every 20 steps).
-
Baselines. Three baselines are compared directly. diffu-GRPO (Zhao et al., 2025a) approximates log-likelihoods with single-pass mean-field estimation — it computes log π_θ(y|x) by summing token-level log-probabilities from a single forward pass on a randomly partially masked prompt. The paper notes this is "efficient" but "introduces notable bias relative to the exact policy" (Section 2.2). VRPO-OL, the online version of VRPO (Zhu et al., 2025), uses ELBO-based likelihood approximation with Monte Carlo sampling, directly maximizing the objective in Eq. 6. The MC sample size is set to the maximum the H800 GPU can support: nt = 4 for math and planning tasks, nt = 2 for coding tasks (where longer prompts consume more memory). LLaDA-8B-Instruct itself serves as the pre-RL baseline. The paper also reports results from several prior works as references (d1-LLaDA, wd1, LLaDA-IGPO, LLaDA-1.5), though these used partially different training settings and are not direct experimental comparisons — they appear in Table 1 as context for where BGPO sits relative to the broader literature.
-
Generation budget and compute accounting. The paper measures computational cost in two complementary ways. MC sample size nt is the primary experimental variable: it directly controls the quality of the log-likelihood approximation, with larger nt providing lower bias and variance at the cost of increased memory (for VRPO-OL) or increased gradient accumulation steps (for BGPO). Response rollout cost (the number of full-response generations per prompt, G = 8, each requiring 128-512 diffusion steps) is held constant across all methods, making the comparison fair: any differences in final performance come from the quality of the RL objective approximation, not from generating more or better candidate responses. Wall-clock training step time is reported in Figure 5 to address the concern that BGPO's sequential gradient accumulation might increase per-step latency. GPU memory usage is measured directly in Figure 1 (left panel) as a function of nt, confirming that VRPO-OL's memory scales linearly with nt while BGPO's remains flat.
-
Cross-validation and statistical protocol. The paper does not employ cross-validation in the traditional sense — there is no hyperparameter tuning on held-out validation sets. Instead, all three methods (diffu-GRPO, VRPO-OL, BGPO) use the same hyperparameters (batch size 16, G = 8, learning rate 5 × 10⁻⁷, same block sizes and diffusion steps per task) to ensure a controlled comparison. Models are evaluated on the standard test sets every 20 training steps, and the best-performing checkpoint for each method is reported (following Zhao et al., 2025a). This checkpoint selection is applied identically to all methods. The absence of a held-out validation set for hyperparameter tuning means the reported numbers reflect a single training run per method per task, which is standard practice in the RL fine-tuning literature given computational constraints but limits statistical conclusions about method reliability. The gradient bias and variance analyses (Figures 3-4) use BGPO with nt = 256 as a "golden gradient" proxy, computed once and used as a fixed reference for all comparisons.
Main Quantitative Results
Overall Performance Comparison Across All Tasks
Table 1 presents the headline results. BGPO improves over the LLaDA-8B-Instruct baseline and outperforms both diffu-GRPO and VRPO-OL on every task. The quantitative breakdown:
- MATH500: BGPO achieves 45.7%, compared to 44.1% (VRPO-OL), 43.1% (diffu-GRPO), and 39.6% (LLaDA-8B-Instruct baseline). The absolute improvement over the SFT baseline is 6.1 percentage points; the improvement over VRPO-OL is 1.6 points.
- GSM8K: BGPO achieves 84.3%, compared to 83.3% (VRPO-OL), 82.1% (diffu-GRPO), and 79.3% (baseline). The improvement over baseline is 5.0 points; over VRPO-OL is 1.0 point.
- HumanEval: BGPO achieves 47.6%, compared to 44.8% (VRPO-OL — notably below the 45.1% baseline for this task), 47.0% (diffu-GRPO), and 45.1% (baseline). This is the only task where VRPO-OL underperforms the SFT baseline, and where diffu-GRPO is competitive with BGPO (47.0% vs. 47.6%).
- MBPP: BGPO achieves 41.7%, compared to 41.5% (VRPO-OL), 40.3% (diffu-GRPO), and 39.1% (baseline). The gaps are narrower here, with BGPO and VRPO-OL nearly tied.
- Sudoku: BGPO achieves 26.9%, compared to 26.1% (VRPO-OL), 26.7% (diffu-GRPO), and 12.0% (baseline). The improvement over baseline is 14.9 points — the largest relative gain — but the gap over the next-best method (diffu-GRPO) is only 0.2 points.
- Countdown: BGPO achieves 87.5%, compared to 84.8% (VRPO-OL), 53.1% (diffu-GRPO), and 19.5% (baseline). This is the most dramatic result: a 68.0-point improvement over the SFT baseline, and a 34.4-point gap over diffu-GRPO. The gap over VRPO-OL is 2.7 points.
A critical pattern emerges when reading across tasks: the advantage of BGPO over VRPO-OL is largest on Countdown (2.7 points) and smallest on MBPP (0.2 points) and Sudoku (0.8 points). The paper does not explicitly analyze this variation, but it aligns with the gradient quality argument: on tasks where the baseline model is already strong and the optimization landscape is smoother (MBPP at 39.1% baseline, coding tasks with clear correctness signals), the additional gradient precision from larger nt provides diminishing returns. On tasks where the baseline is weak and exploration is critical (Countdown at 19.5%), having a more accurate gradient signal matters substantially more.
The comparisons to prior works in the upper section of Table 1 are informative but not controlled experiments. BGPO's MATH500 score of 45.7% exceeds all prior LLaDA-based models (d1-LLaDA: 40.2%, LLaDA-1.5: 42.6%, LLaDA-IGPO: 42.8%). Similarly, BGPO's GSM8K score of 84.3% exceeds all prior works (d1-LLaDA: 82.1%, LLaDA-1.5: 83.3%, wd1: 82.3%, LLaDA-IGPO: 83.6%). However, these prior works used different base models, different training data mixtures, and different training protocols — the comparison establishes that BGPO achieves state-of-the-art results but does not isolate the contribution of BGPO's specific innovation.
Training Dynamics: Reward Curves
Figure 2 plots training reward versus optimization steps for all three methods across the four task domains. The key observations:
- Mathematics (left panel): All three methods follow similar upward trajectories, with BGPO's reward slightly but consistently above VRPO-OL and diffu-GRPO after approximately step 200. The final reward for BGPO reaches roughly 0.70, compared to approximately 0.67 for VRPO-OL and 0.64 for diffu-GRPO. The curves are relatively tight, consistent with the modest accuracy gaps in Table 1 (1-2 points on MATH500/GSM8K).
- Coding (second panel): The reward curves nearly overlap for all three methods throughout training, with final rewards around 0.67-0.70. This mirrors the narrow performance gaps in Table 1 (BGPO: 47.6% on HumanEval vs. diffu-GRPO: 47.0%). On coding tasks, the choice of RL algorithm appears to matter less than on other domains — possibly because the supervised-fine-tuned baseline is already reasonably strong (45.1% on HumanEval), leaving limited room for RL-driven improvement regardless of gradient quality.
- Sudoku (third panel): BGPO and VRPO-OL track closely, both showing gradual improvement from approximately 0.05 to 0.27-0.28 over 400 steps. diffu-GRPO follows a similar trajectory, reaching roughly 0.27. All three methods converge to similar rewards, consistent with the near-identical final accuracies in Table 1 (26.9%, 26.1%, 26.7%).
- Countdown (right panel): This shows the starkest divergence. BGPO's reward rises rapidly from approximately 0.10 to 0.65 over the first 100 steps, while VRPO-OL reaches roughly 0.50 and diffu-GRPO reaches only about 0.30. By step 200, BGPO achieves roughly 0.72, VRPO-OL approximately 0.65, and diffu-GRPO approximately 0.40. This dramatic separation is the most visually compelling evidence in the paper: when the exploration space is relatively simple and the baseline model is weak, the quality of the gradient signal (determined by nt) cascades into substantially faster learning and higher asymptotic performance.
The paper draws specific attention to Countdown in Section 4.2:
"BGPO exhibits a notably faster reward increase and significantly higher reward on the Countdown task, where the exploration space is relatively simple. These phenomena demonstrate that the larger MC sample size of BGPO brings a more accurate optimization direction, which aims to maximize the expectation of rewards."
This interpretation is plausible but deserves scrutiny. The "more accurate optimization direction" is supported by the gradient bias/variance evidence in Figures 3-4, but Countdown's particular sensitivity might also reflect the nature of its reward function — if Countdown rewards are sparse or binary, noisy advantage estimates (from small-nt ELBO approximations) could cause the policy to reinforce incorrect responses that happen to receive lucky ELBO estimates, whereas dense or smoothly varying rewards would be more robust to this noise. The paper does not analyze the reward function properties across tasks, so this remains a plausible but untested hypothesis.
Effect of MC Sample Size on Final Performance
Table 2 provides the most direct test of the paper's central claim: larger nt improves model performance. Training LLaDA-8B-Instruct on math problem solving with BGPO at different nt values yields:
- nt = 1: 43.5% MATH500 / 83.5% GSM8K
- nt = 2: 44.1% / 82.5%
- nt = 4: 43.7% / 82.7%
- nt = 8: 45.3% / 83.9%
- nt = 16: 45.7% / 84.3%
The MATH500 trend is broadly monotonic (with a slight dip at nt=4: 43.7% vs. 44.1% at nt=2). The improvement from nt=1 to nt=16 is 2.2 percentage points on MATH500 and 0.8 points on GSM8K. These are modest absolute gains but represent meaningful relative improvements (5.1% and 1.0% relative, respectively) given that the baseline model already achieves 39.6%/79.3%.
The GSM8K trend is less clean: nt=1 achieves the second-highest score (83.5%), followed by a drop at nt=2 (82.5%) and nt=4 (82.7%), then recovery at nt=8 (83.9%) and nt=16 (84.3%). This non-monotonicity, which the paper does not discuss, suggests that the benefit of larger nt is not purely a smooth function — there may be interactions between gradient noise and the optimization landscape that cause temporary regressions at intermediate noise levels. However, the best performance is achieved at the largest nt for both benchmarks, supporting the paper's directional claim.
A limitation of this analysis is that it only varies nt for BGPO, not for VRPO-OL (which cannot go beyond nt=4). A stronger demonstration would be to show that VRPO-OL at nt=4 underperforms BGPO at nt=4 (same sample size, different objective formulation), which would isolate the effect of the lower bound construction from the effect of sample size. This comparison is partially available in Table 1: on math tasks, BGPO at nt=16 (45.7%) outperforms VRPO-OL at nt=4 (44.1%), but this conflates sample size and algorithm. The paper does not report BGPO at nt=4 in Table 1, though Table 2 shows BGPO at nt=4 achieves 43.7% on MATH500 — which is lower than VRPO-OL's 44.1%. This is a very interesting result: at the same nt=4, BGPO's lower-bound objective underperforms VRPO-OL's direct objective (43.7% vs. 44.1%). This suggests that at small nt, the bound gap (the difference between the lower bound and the true objective when off-policy) may be non-negligible, and the benefit of BGPO only materializes when nt is large enough that the gradient quality improvement outweighs the bound gap. The paper does not discuss this tradeoff explicitly.
Gradient Quality Analysis: Bias and Variance
Figures 3 and 4 quantify the mechanism linking larger nt to better performance: reduced gradient noise. The experimental design uses BGPO at nt=256 as a proxy for the "true" RL gradient (the best available approximation, given that the exact gradient of the intractable objective cannot be computed). For each method and nt setting, the gradient is computed on the same batch 8 times with different MC samples, and statistics are computed across these 8 replicates.
-
Gradient standard deviation (Figure 3): diffu-GRPO (single-pass, effectively nt=1) shows the highest standard deviation, with a normalized value of approximately 4.0 on the y-axis. VRPO-OL's standard deviation decreases from approximately 2.5 at nt=1 to roughly 1.0-1.5 at nt=4, but cannot be measured at higher nt due to memory constraints. BGPO's standard deviation continues to decrease, reaching approximately 0.5 at nt=16. The trend is monotonic for all methods. At nt=4 (the maximum VRPO-OL can reach), BGPO's standard deviation is approximately equal to VRPO-OL's, which is expected — at the same nt, the gradient computation should have similar variance regardless of whether it's computed through the direct or lower-bound path (since the bound is exact on-policy, as proved in Section 3.2).
-
Gradient bias (Figure 4): Using the BGPO nt=256 gradient as the reference, bias is computed as the normalized deviation from this golden signal. diffu-GRPO shows the highest bias (approximately 4.5-5.0), consistent with its structural approximation error that does not shrink with more computation. VRPO-OL at nt=1 shows bias around 2.5, decreasing to roughly 1.5 at nt=4. BGPO's bias decreases from approximately 2.0 at nt=1 to roughly 0.5 at nt=16. As with variance, the bias curves for BGPO and VRPO-OL nearly overlap at the nt values where both can be measured (1, 2, 4), confirming that the lower-bound construction does not introduce additional systematic error when evaluated on-policy.
These figures provide the mechanistic explanation for why BGPO outperforms VRPO-OL despite maximizing a lower bound of the same objective: the objective being maximized is effectively the same (due to on-policy equivalence), but the gradient used for optimization has substantially lower noise when nt=16 versus nt=4. The optimizer takes more accurate steps, converges to a better solution, and does so in fewer steps (as evidenced by the Countdown reward curves in Figure 2).
A nuanced point: the paper uses BGPO at nt=256 as the golden gradient for bias measurement, which means BGPO is being used as its own reference. This is reasonable — nt=256 should provide an ELBO estimate very close to the true expectation, and BGPO's lower bound at nt=256 is near-exact on-policy — but it means the bias numbers for BGPO at lower nt are measured relative to BGPO at higher nt, which could slightly understate absolute bias if BGPO's nt=256 gradient itself has residual bias relative to the true (uncomputable) gradient. This is a minor concern: the qualitative conclusion (bias and variance decrease with nt) would hold regardless of the reference choice.
Out-of-Domain Generalization
Table 3 evaluates whether RL training on one domain transfers to other domains. The base LLaDA-8B-Instruct is evaluated on all six benchmarks, then two models are compared: one trained with BGPO on math tasks only (MATH + GSM8K training sets), and one trained with BGPO on coding tasks only (DeepCoder training set).
- Math → Planning: The math-trained model improves Sudoku from 6.3% (baseline) to 8.6% (+2.3 points) and Countdown from 14.5% to 21.1% (+6.6 points). These are non-trivial gains suggesting that mathematical reasoning skills transfer partially to structured planning tasks, even without domain-specific RL.
- Math → Coding (negative transfer): The math-trained model slightly degrades on HumanEval (44.2% vs. 45.1% baseline, -0.9 points) and MBPP (38.6% vs. 39.1%, -0.5 points). The drops are small but consistent, suggesting mild negative transfer — optimizing for mathematical reasoning may shift the model's generation distribution away from coding patterns.
- Coding → Math: The coding-trained model improves MATH500 from 39.6% to 40.8% (+1.2 points) and GSM8K from 79.3% to 80.4% (+1.1 points). The gains are smaller than in-domain RL (+6.1 and +5.0 points, respectively) but show some transfer.
- Coding → Planning: The coding-trained model improves Sudoku from 6.3% to 9.2% (+2.9 points) and Countdown from 14.5% to 21.5% (+7.0 points) — gains comparable to the math-trained model.
The key finding is asymmetry: coding-trained models transfer positively to both math and planning, while math-trained models transfer to planning but slightly negatively to coding. The paper does not analyze this asymmetry, but it is a meaningful result: it suggests that the skills learned through RL on code (likely structured reasoning, step-by-step decomposition, and algorithmic thinking) are more universally applicable than skills learned through mathematical problem-solving, which may be more domain-specific. This has practical implications for designing multi-task RL curricula for dLLMs.
Training Speed
Figure 5 reports average training step time on math problem solving with 512-token maximum response length:
- diffu-GRPO (nt=1): 128.8 seconds per step
- VRPO-OL (nt=4): 130.2 seconds per step
- BGPO (nt=16): 151.5 seconds per step
The key comparison: BGPO at 4× the MC sample size of VRPO-OL takes only 16.4% longer per step (151.5 vs. 130.2 seconds). Measured in terms of wall-clock time per unit of gradient quality improvement, BGPO is substantially more efficient — it achieves a ~50% reduction in gradient variance and bias (Figures 3-4) for a 16% time increase.
The paper attributes this efficiency to the dominant cost of response rollout:
"the dominant time cost of each step lies in the response rollout phase (i.e., sampling G responses for each prompt) rather than the objective computation and policy updating phases" (Section 4.5)
This is an important practical finding that is not obvious from the algorithm description: gradient accumulation across nt samples is cheap because generating the responses (which all methods must do) is far more expensive than processing those responses through the ELBO estimator and backpropagating. The practical upshot is that practitioners can use BGPO with large nt without worrying about training slowdown.
Ablation Studies and Robustness Checks
MC sample size sweep (Table 2): The ablation on nt values (1, 2, 4, 8, 16) for BGPO on math tasks demonstrates that performance generally improves with larger nt, though the relationship is not perfectly monotonic — nt=4 achieves 43.7% on MATH500, lower than nt=2 at 44.1%. The overall trend from nt=1 (43.5%) to nt=16 (45.7%) is positive, with the best result at the maximum tested nt. This supports the claim that larger nt improves RL training, though the non-monotonicity at nt=4 suggests there may be complex interactions between gradient noise and optimization dynamics.
Advantage sign-dependent bound construction (implicit in Figures 3-4): The equivalence of BGPO and VRPO-OL at the same nt values in both gradient bias and variance (their curves overlap at nt=1, 2, 4 in both figures) serves as an ablation confirming that the boundary-guided lower bound does not introduce additional error when evaluated on-policy. If the bound construction introduced systematic bias, BGPO's gradient bias at nt=4 would diverge from VRPO-OL's. The fact that they match validates the theoretical equivalence proof in practice.
Task-wise hyperparameter sensitivity (Table 4): The paper varies nt by task — 32 for Sudoku, 16 for math, coding, and Countdown — but does not systematically ablate these choices. The nt=32 for Sudoku likely reflects that Sudoku's shorter responses (256 tokens vs. 512) leave more GPU memory for MC sampling. No experiment compares nt=16 vs. nt=32 on the same task, so the marginal benefit of increasing nt beyond 16 is unknown.
Reward function type (Figure 2): Across the four task panels in Figure 2, the separation between methods varies dramatically — nearly overlapping on coding, tight on Sudoku, moderately separated on math, and widely separated on Countdown. This is not a controlled ablation of reward function properties, but it serves as an implicit sensitivity analysis: BGPO's advantage over baselines is largest when the base model is weakest (Countdown at 19.5% SFT accuracy) and smallest when the base model is already strong (coding at 45.1%). This suggests that gradient quality matters most when the policy has substantial room for improvement, consistent with the intuition that noisy gradients are more damaging when the optimization landscape requires larger parameter movements.
Out-of-domain evaluation (Table 3): This tests robustness to distribution shift between training and evaluation domains. The results show non-trivial transfer, but also reveal negative transfer from math training to coding tasks (HumanEval drops 0.9 points). This is an important robustness check: RL with BGPO does not catastrophically overfit to the training domain — it largely preserves or improves out-of-domain performance — but domain-specific optimization does narrow the model's strengths toward the training distribution.
Missing ablation: direct comparison of BGPO vs. VRPO-OL at identical nt. The paper does not systematically compare BGPO and VRPO-OL at the same nt values across all tasks. Table 2 reports BGPO at different nt, and VRPO-OL appears only at its maximum feasible nt (4 for math, 2 for coding, 4 for planning). A direct head-to-head at nt=4 on MATH500 would test whether the bound gap in BGPO (when off-policy after gradient steps within an iteration) creates a meaningful disadvantage compared to the exact objective in VRPO-OL. The suggestive evidence from Table 2 (BGPO at nt=4 reaches 43.7% vs. VRPO-OL at nt=4 reaching 44.1% in Table 1) hints that VRPO-OL might be slightly better at equal nt, but this is not rigorously tested — the numbers come from different tables with potentially different evaluation runs. This is the single most informative ablation the paper could have run but did not.
Missing ablation: BGPO at VRPO-OL's maximum nt on all tasks. The paper could have run BGPO at nt=4 on math, nt=2 on coding, and nt=4 on planning, exactly matching VRPO-OL's sample sizes, to isolate whether the algorithm (bound construction vs. direct optimization) or the sample size (larger nt) drives the performance gains. Since the results would reveal whether BGPO's benefits come from the lower bound being a better objective or simply from enabling larger nt, this is a notable omission that makes it difficult to fully attribute the gains.
Critical Assessment
Claim: BGPO significantly outperforms previous RL algorithms for dLLMs.
Does the evidence fully support this? Yes, but with important nuance about why. Table 1 shows BGPO outperforming diffu-GRPO and VRPO-OL on all six benchmarks. However, the mechanism is not that BGPO's lower-bound objective is inherently better than VRPO-OL's direct objective — it is that BGPO enables larger nt, and larger nt improves gradient quality. The performance gap over VRPO-OL (1.6 points on MATH500, 1.0 on GSM8K, 0.2-0.8 on planning) is consistent with the hypothesis that both methods would perform similarly at the same nt, and BGPO's advantage comes from being able to use larger nt. The paper would be stronger if it explicitly demonstrated this: for example, showing that VRPO-OL at nt=16 (run on hardware with more memory or with gradient checkpointing) matches BGPO at nt=16. Without this demonstration, the claim "BGPO outperforms VRPO-OL" is correct but incomplete — it outperforms because of memory efficiency enabling larger sample sizes, not because of a fundamentally better optimization objective.
The comparison to diffu-GRPO is cleaner: BGPO substantially outperforms it on most tasks, with the Countdown gap being dramatic (87.5% vs. 53.1%). This establishes that ELBO-based likelihood approximation (whether via BGPO or VRPO-OL) is superior to single-pass mean-field estimation for RL of dLLMs, particularly on tasks requiring exploration.
Testing conditions: All experiments use a single base model (LLaDA-8B-Instruct), a single hardware configuration (8× H800), and standard RL hyperparameters from the VeRL framework. The performance ordering is consistent across tasks, strengthening the reliability claim, but the single-model limitation (acknowledged in Section 7) means we don't know whether these results generalize to other dLLM architectures (Dream, DiffuLLaMA, SDAR) or scales.
Claim: Larger MC sample sizes effectively reduce bias and variance of gradients and improve model performance.
Does the evidence fully support this? Yes, this is the most thoroughly validated claim in the paper. Figures 3 and 4 provide direct measurement of gradient statistics at different nt, showing monotonic decreases in both bias and variance. Table 2 shows the downstream effect on task performance, with generally monotonic improvement from nt=1 to nt=16. The causal chain — larger nt → lower gradient noise → better optimization → higher accuracy — is convincingly demonstrated.
Weaknesses: The bias measurement uses BGPO at nt=256 as ground truth, which is a pragmatic choice but means we cannot measure whether even nt=256 has residual bias relative to the true gradient. The performance improvements from nt=1 to nt=16 (2.2 points on MATH500, 0.8 on GSM8K) are real but modest, and the non-monotonicity at nt=4 (43.7%, lower than nt=2 at 44.1%) suggests the relationship is not perfectly smooth. The paper does not explore whether further increases beyond nt=16 would yield continued gains or whether saturation sets in. Given that Figure 3's variance curve appears to be flattening at nt=16, further gains might be diminishing.
Claim: BGPO achieves these improvements with only marginal increases in average training step time.
Does the evidence fully support this? The data in Figure 5 (151.5 seconds for BGPO at nt=16 vs. 130.2 seconds for VRPO-OL at nt=4) supports a 16% time increase for 4× the sample size — a favorable tradeoff. However, this is measured only on math problem solving with 512-token responses. The relative overhead might differ on tasks with shorter responses (where response generation is faster, making the MC sampling fractionally more expensive) or with larger models (where the forward/backward passes for MC sampling become more costly). The paper's explanation — that response rollout dominates — is plausible but should be verified across task configurations.
A more subtle point: the training time comparison in Figure 5 measures per-step time, but it does not account for whether all methods require the same number of steps to converge. Figure 2 shows that on Countdown, BGPO reaches high rewards much faster than baselines — it might need fewer total steps to reach a given performance threshold. This means the effective training-time-to-performance ratio could be even more favorable than the per-step comparison suggests, but the paper does not compute this metric.
Claim: BGPO achieves state-of-the-art results.
Does the evidence fully support this? Against the prior works listed in Table 1 (d1-LLaDA, wd1, LLaDA-IGPO, LLaDA-1.5), BGPO's numbers are higher on all overlapping benchmarks. However, these comparisons are not controlled — the prior works used different training data, different base model checkpoints, different hyperparameters, and different reward functions. The "state-of-the-art" claim is correct in the narrow sense of "best reported numbers on these benchmarks for dLLMs," but it overstates the degree to which BGPO's specific innovation (the lower bound construction) is responsible for the improvement. The model trained with BGPO benefits from all the engineering of the VeRL framework, the specific training data mixtures, the hyperparameter choices, and the base model quality — not just the BGPO algorithm. A fairer statement would be: "the LLaDA-8B-Instruct model fine-tuned with BGPO achieves the highest reported accuracy among dLLMs on these benchmarks."
Missing Experiments That Would Strengthen the Paper
- VRPO-OL at larger nt on high-memory hardware: If the paper could demonstrate VRPO-OL at nt=8 or nt=16 (using A100 80GB or H100 with more memory, or gradient checkpointing to trade compute for memory), it would directly test whether the performance gap in Table 1 is due to sample size or algorithm. This is the cleanest experiment the paper does not run.
- BGPO at VRPO-OL's nt: Running BGPO at nt=2 (coding) and nt=4 (math, planning) and comparing directly to VRPO-OL would reveal whether the lower bound introduces any approximation penalty at small sample sizes where VRPO-OL is feasible. This would clarify the performance crossover point — at what nt does BGPO's sample size advantage overcome any bound gap?
- Scaling beyond nt=16: Figures 3 and 4 show decreasing but still non-zero bias and variance at nt=16. Extending to nt=32, 64, or 128 (even on a subset of tasks) would reveal whether the benefits saturate or continue, informing practical recommendations for practitioners.
- Different base models: All experiments use LLaDA-8B-Instruct. Testing BGPO on Dream, DiffuLLaMA, or SDAR would establish whether the approach generalizes across dLLM architectures or whether it depends on specific properties of LLaDA's training (e.g., its ELBO-based pre-training objective that makes ELBO-based RL a natural fit).
- Statistical reliability: The paper reports single-run results. Multiple runs with different random seeds would provide confidence intervals and reveal whether the performance differences between methods (especially the narrow gaps on MBPP and Sudoku) are statistically significant or within run-to-run variance.
- Ablation on the bound construction: The paper could test alternative lower bounds — for example, using only the Taylor expansion (which would produce an upper bound for negative advantages, potentially causing instability) or only Jensen (upper bound for positive advantages). This would demonstrate that the sign-dependent construction is necessary, not merely one of many viable choices.
Conditional Claims and Their Boundaries
-
"Larger nt improves performance" holds in the tested range (nt=1 to nt=16) on math tasks with LLaDA-8B-Instruct, but the non-monotonicity at nt=4 and the flattening gradient curves at nt=16 suggest the relationship may be more complex at the extremes. At very small nt (1-2), gradient noise may be so high that the occasional "lucky" noisy gradient step provides beneficial exploration, partially offsetting the noise cost. At very large nt, diminishing returns likely set in as the ELBO estimate converges to the true expectation. The paper does not characterize these regimes.
-
"BGPO outperforms VRPO-OL" holds when comparing BGPO at nt=16 to VRPO-OL at nt=4, but likely would not hold at equal nt. The available evidence (Table 2 BGPO nt=4: 43.7% MATH500 vs. Table 1 VRPO-OL nt=4: 44.1%) suggests VRPO-OL may be slightly superior when sample sizes are equal, which makes sense — the direct objective has no bound gap, while BGPO's lower bound becomes a genuine (if tight) approximation once the policy drifts off-policy within an iteration.
-
"Training step time increases only marginally" holds for math tasks with 512-token responses, but the relative overhead would increase on tasks with shorter responses (where generation is cheaper relative to MC sampling) or with larger models (where forward/backward passes are more expensive relative to generation). The paper's general claim should be qualified: the overhead is marginal in the tested configuration, not universally.
-
"State-of-the-art results" holds relative to prior published dLLM results, but the comparison is uncontrolled and should not be interpreted as a pure measurement of BGPO's algorithmic contribution. The gains come from the full training pipeline (data, hyperparameters, base model, RL algorithm), and BGPO's specific contribution is best isolated by the direct comparisons to diffu-GRPO and VRPO-OL under identical conditions.
Summary
The experimental section is methodical and well-structured, with a clear logical progression: overall performance (Table 1) → training dynamics (Figure 2) → mechanistic explanation via gradient analysis (Figures 3-4) → sample size ablation (Table 2) → generalization (Table 3) → practical efficiency (Figure 5). The gradient bias/variance measurements are the strongest evidence, directly validating the paper's central claim that BGPO's memory efficiency enables higher-quality optimization signals through larger nt. The main interpretive gap is the inability to fully disentangle the effects of sample size from the lower-bound construction, since VRPO-OL cannot be run at BGPO's nt values. This gap is acknowledged implicitly in the paper's framing (BGPO is presented as enabling larger nt, not as a better objective), but it leaves open the question of whether a hybrid approach — VRPO's direct objective with BGPO's gradient accumulation trick — might perform even better, or whether the lower bound is necessary to achieve the linear decomposition in the first place.
6. Limitations and Trade-offs
6.1 Single Model Family, Single Scale: No Evidence Beyond LLaDA-8B-Instruct
The assumption or constraint. All experiments in the paper — every result in Tables 1–3, every gradient analysis in Figures 3–4, every reward curve in Figure 2 — uses exactly one base model: LLaDA-8B-Instruct (Nie et al., 2025b). The paper is transparent about this constraint in Section 7:
"we only conduct experiments on 8B-level models, since there are no larger open-source dLLMs, and our computational resources are also limited. Nonetheless, we believe our BGPO algorithm can be well applied to larger dLLMs due to its solid theoretical foundation."
The authors state a belief that the method will generalize, but this belief is untested. No experiments are conducted on any other dLLM architecture — not Dream (Ye et al., 2025), not DiffuLLaMA (Gong et al., 2025a), not SDAR (Cheng et al., 2025) — and no ablation varies model scale within the LLaDA family (e.g., a 1B or 3B variant).
The consequence. There are at least three distinct ways in which BGPO's effectiveness could fail to generalize, and the paper provides no evidence to rule out any of them:
-
Architecture dependence. LLaDA uses a specific masked diffusion formulation trained from scratch with bidirectional attention by maximizing ELBOs. The fact that BGPO's RL objective also uses ELBO-based likelihood approximation means there is a natural alignment between pre-training and RL — the quantity being optimized during RL is closely related to the quantity the model was pre-trained to maximize. Other dLLM architectures, particularly those adapted from autoregressive pre-trained weights (DiffuLLaMA, Dream), may have different relationships between their generation process and the ELBO, potentially changing how well the ELBO approximates the true log-likelihood and thus how effective BGPO's optimization signal is. Additionally, models with different attention patterns, different numbers of diffusion steps, or different block-wise decoding strategies might exhibit different memory scaling behavior in the forward passes, potentially altering the memory-efficiency advantage BGPO provides.
-
Scale dependence. The memory bottleneck that BGPO solves is a function of model size — larger models have larger activation tensors, and the linear memory scaling with nt that VRPO-OL suffers from becomes more severe at larger scales. This suggests BGPO's relative advantage over VRPO-OL should increase with model size, which would strengthen the paper's claims. However, the absolute memory cost of even a single forward/backward pass also increases with model size, and BGPO still requires storing the full computational graph for one MC sample at a time. At some sufficiently large model scale, even nt=1 might approach memory limits, and BGPO's constant-memory property relative to nt would become irrelevant because the baseline cost dominates. The paper does not characterize where this crossover occurs for LLaDA-family models, making it difficult for practitioners to predict whether BGPO will be useful at their target scale.
-
Training dynamics dependence. The gradient bias and variance curves in Figures 3–4 are measured on one model checkpoint. Different model architectures might have different gradient noise characteristics — models with smoother loss landscapes (lower curvature) would be less sensitive to gradient noise, reducing BGPO's advantage; models with sharper landscapes would be more sensitive, potentially increasing BGPO's advantage. No characterization of loss landscape properties is provided.
What evidence exists in the paper. None directly. The paper's theoretical argument — that the lower bound construction is mathematically valid for any masked diffusion model parameterized by a mask predictor — is correct as far as it goes, but it establishes feasibility, not empirical effectiveness at different scales or for different architectures. The "solid theoretical foundation" the authors cite ensures that BGPO can be applied; it does not guarantee that the performance benefits observed at 8B with LLaDA will transfer.
Mitigation status. The limitation is acknowledged in Section 7 but not addressed. The authors frame it primarily as a computational resource constraint ("no larger open-source dLLMs") rather than as a genuine uncertainty about generalization, and they express belief rather than caution about transferability. No experiments using smaller LLaDA variants are conducted to establish scaling trends. No experiments adapting BGPO to other dLLM architectures are attempted, even at small scale. The limitation remains entirely open.
6.2 The Bound Gap at Finite nt Is Not Characterized — and May Explain Why BGPO Underperforms VRPO-OL at Equal nt
The assumption or constraint. BGPO maximizes a lower bound $\hat{J}_{lb}(\theta)$ of the ELBO-based objective $\hat{J}(\theta)$ rather than maximizing $\hat{J}(\theta)$ directly. The on-policy equivalence proof (Section 3.2) guarantees that $\hat{J}_{lb}(\theta) = \hat{J}(\theta)$ and $\nabla_\theta \hat{J}_{lb}(\theta) = \nabla_\theta \hat{J}(\theta)$ at the exact moment when $\pi_\theta = \pi_{\theta_{old}}$, which occurs immediately after the old policy is refreshed (Algorithm 1, line 2). However, within a training iteration, the policy takes a gradient step and $\pi_\theta$ diverges from $\pi_{\theta_{old}}$. At that point, the lower bound becomes a genuine inequality — $\hat{J}_{lb}(\theta) < \hat{J}(\theta)$ — and the gradient of the lower bound may differ from the gradient of the true objective.
The paper does not characterize the size of this "bound gap" — the difference between $\hat{J}_{lb}$ and $\hat{J}$ as a function of how far the policy has moved from $\pi_{\theta_{old}}$. The on-policy equivalence proof only covers the boundary case; it says nothing about off-policy behavior within an iteration.
The consequence. This limitation matters because it directly affects the interpretation of BGPO's performance relative to VRPO-OL. When nt is small (e.g., nt=4), VRPO-OL can feasibly train with the direct objective $\hat{J}(\theta)$ — there is no bound gap. BGPO at the same nt=4 would be maximizing a lower bound that is exact only at the start of the iteration and becomes inexact after gradient steps. The tradeoff is: BGPO sacrifices some objective fidelity (the bound gap) for the ability to use larger nt, which improves gradient quality. The performance crossover — at what nt does the gradient quality gain from larger samples outweigh the loss from the bound gap? — is never established.
The empirical evidence in the paper hints that this tradeoff is real. Table 2 shows BGPO at nt=4 achieves 43.7% on MATH500, which is lower than nt=2 at 44.1% and also lower than VRPO-OL at nt=4 achieving 44.1% (Table 1). While the comparison between tables is not a controlled head-to-head, the pattern is suggestive: at the same nt=4, BGPO may underperform VRPO-OL because the bound gap penalty outweighs any benefit from the lower bound construction. If this interpretation is correct, then BGPO's advantage over VRPO-OL in Table 1 is entirely attributable to larger nt, and at equal nt, the direct objective would be preferable.
If this interpretation is incorrect (i.e., BGPO at nt=4 is genuinely comparable to VRPO-OL at nt=4, and the 43.7% vs. 44.1% difference is noise), the paper should have demonstrated it with a controlled experiment. The absence of this experiment leaves the performance attribution ambiguous.
What evidence exists in the paper. The on-policy equivalence proof (Eqs. 15–17) is rigorous for the boundary case d_j = 0. The gradient bias and variance measurements in Figures 3–4 show BGPO and VRPO-OL matching at nt=1, 2, 4 — but these measurements are taken on-policy (since the gradient is computed immediately after a policy refresh for the analysis), so they only validate the boundary case, not the off-policy regime where the bound gap would appear. Table 2 provides performance at different nt for BGPO but not for VRPO-OL. The paper does not measure the bound gap magnitude, does not analyze how it grows with the number of gradient steps per iteration, and does not compare BGPO and VRPO-OL at equal nt in a controlled experiment.
Mitigation status. The paper does not address this limitation. There is no discussion of the bound gap in Section 3 (where the lower bound is constructed), Section 4 (experimental results), or Section 7 (limitations). The authors appear to treat the on-policy equivalence as sufficient justification, implicitly assuming that the policy drift within an iteration is small enough that the bound gap is negligible. Whether this assumption holds in practice — particularly at larger learning rates, on tasks with sharper loss landscapes, or when training for many steps per iteration — is not investigated. A natural mitigation would be to refresh the old policy more frequently (every gradient step rather than every iteration), which would keep training closer to on-policy and shrink the bound gap, potentially at the cost of increased computational overhead from more frequent response generation. This tradeoff is not explored.
6.3 Difficulty Estimation via Large MC Sampling Is Computationally Expensive and Not Amortized in the Reported Numbers
The assumption or constraint. The paper does not discuss or account for the computational cost of the MC sampling used in the ELBO estimation during training. The reported training step times in Figure 5 (151.5 seconds for BGPO at nt=16, 130.2 seconds for VRPO-OL at nt=4) include the cost of sampling nt random masking times, applying the forward masking process to each response, running forward passes through the current and old policies to compute $\ell_{\pi_\theta}$ and $\ell_{\pi_{old}}$, and backpropagating through the lower bound. However, these costs do not include any equivalent of the "difficulty estimation" step that prior work on compute-optimal test-time scaling (e.g., the reference paper in this analysis, which generates 2048 samples per question to estimate difficulty) found to be a dominant fraction of total computation.
In BGPO's case, the analogous cost is the nt-dependent ELBO estimation itself — while the memory cost is the bottleneck the paper focuses on, the computational cost of running nt forward passes through an 8B-parameter model per response per gradient step is non-trivial, and this cost scales linearly with nt. The paper's claim that training step time increases "only marginally" (Section 4.5) from nt=4 to nt=16 rests on the observation that response generation dominates — but this dominance depends on the ratio of response generation cost to MC sampling cost, which in turn depends on the number of diffusion steps used for generation versus the cost of a single ELBO forward pass.
The consequence. The efficiency claim — "only marginal increases in average training step time" — is configuration-dependent in ways the paper does not explore. Specifically:
-
Shorter response lengths reduce the generation-to-MC-sampling cost ratio. For Sudoku and Countdown, responses are 256 tokens rather than 512. The denoising process uses 128 steps in training rather than 256–512. This means response generation is roughly 2–4× cheaper (fewer tokens × fewer steps), while the MC sampling cost (forward passes on partially masked responses of the same length) remains proportional to the full response length. The fraction of total step time spent on MC sampling would be higher for these tasks than for math/coding. The paper does not report training step times for planning tasks, so the claimed efficiency may not hold there.
-
Longer evaluation-time diffusion steps widen the gap. The paper uses more diffusion steps at evaluation (e.g., 512 for math vs. 256 for training). If evaluation were as expensive as training, the efficiency claim would be stronger — but evaluation cost is not amortized over many gradient steps the way training cost is.
-
The nt=4 vs. nt=16 comparison conflates sample count with algorithm. The 16% time increase (151.5 vs. 130.2 seconds) is not exclusively due to the larger nt — it also includes the overhead of BGPO's gradient accumulation loop (the inner loop in Algorithm 1 that processes each MC sample sequentially) compared to VRPO-OL's batched computation. A fair comparison would measure VRPO-OL at nt=4 vs. VRPO-OL at nt=16 (if memory permitted) to isolate the marginal cost of MC sampling, then compare to BGPO at nt=16 to measure the accumulation overhead. Neither comparison is possible because VRPO-OL cannot run at nt=16.
What evidence exists in the paper. Figure 5 provides the only direct timing comparison, limited to math problem solving with 512-token responses. The paper's explanation (Section 4.5) attributes the small time increase to response rollout dominating, but does not break down the time into generation, MC sampling, and gradient computation components, making it impossible to verify this attribution or to extrapolate to other task configurations.
Mitigation status. Not addressed. The paper does not report per-component timing, does not analyze how the generation-to-MC-sampling cost ratio varies across tasks, and does not discuss the scalability of the efficiency claim to larger models (where forward/backward passes through a larger model would increase the MC sampling cost relative to generation, since generation and MC sampling both scale with model size but generation additionally scales with the number of diffusion steps).
6.4 The Lower Bound Construction Depends on the Advantage Sign — and Fails Gracefully in Ways the Paper Does Not Analyze
The assumption or constraint. The boundary-guided lower bound construction in Eq. 13 splits on the sign of the advantage A(x,y): Taylor expansion (e^δ ≥ 1 + δ) is applied when A(x,y) ≥ 0, and Jensen's inequality (e^{average} ≤ average of e) is applied when A(x,y) < 0. This sign-dependent construction is mathematically necessary to maintain the lower bound property, as discussed in Section 3.1 and analyzed in the Technical Approach section.
However, the tightness of the two bounds is not symmetric, and their behavior as the policy moves off-policy is not characterized. The Taylor expansion bound (1 + δ vs. e^δ) has error that grows quadratically with δ: e^δ - (1 + δ) ≈ δ²/2 for small δ. The Jensen bound (average of e^{d_j} vs. e^{average of d_j}) has error that depends on the variance of the d_j values: by Jensen's inequality, the gap is larger when the d_j values are more dispersed. These two error sources behave differently as training progresses and as nt varies, but the paper provides no analysis of which dominates or how they interact.
The consequence. The asymmetry between the two bound constructions could create systematic biases in the optimization:
-
Responses with positive advantages (A ≥ 0) are optimized using the Taylor bound. When d_j is positive (current policy improves over old), the bound understates the improvement (1 + d_j < e^{d_j}), meaning the model receives a weaker positive reinforcement signal than the true objective would provide. When d_j is negative (current policy degrades), the bound is 1 + d_j, which can become negative if d_j < -1, while the true e^{d_j} remains positive — the bound can flip the sign of the per-sample contribution. The paper does not discuss whether d_j < -1 occurs in practice or what effect it has on training stability.
-
Responses with negative advantages (A < 0) are optimized using the Jensen bound. The gap
$\frac{1}{n_t}\sum e^{d_j} - e^{\frac{1}{n_t}\sum d_j}$is larger when the d_j values are more variable. Since d_j measures the per-sample difference between current and old policy predictions, and this difference varies across masking times (some masking times are more informative than others), the Jensen bound gap is inherently tied to the variance of the ELBO estimation — which is exactly what larger nt reduces. This creates a complex interaction: larger nt reduces the variance of d_j, which tightens the Jensen bound, which improves the optimization signal, which improves the policy, which may change the variance of d_j. The paper treats these effects as uniformly positive ("larger nt → better"), but the interaction between nt, bound tightness, and policy improvement rate is not characterized. -
The discontinuity at A(x,y) = 0: The construction switches between two different functional forms at exactly A = 0. While both forms reduce to A(x,y) when d_j = 0 (on-policy), their gradients with respect to d_j differ: for the Taylor bound, ∂g_j/∂d_j = A/nt; for the Jensen bound, ∂g_j/∂d_j = A · e^{d_j}/nt. At d_j = 0 and A near zero, both approach 0, but for non-zero d_j and A near zero, the behavior differs. This could cause instability for responses near the advantage decision boundary, where small changes in the advantage (due to slight changes in the reward model or the group normalization) flip the construction method.
What evidence exists in the paper. Figures 3 and 4 measure gradient bias and variance empirically, which partially captures the aggregate effect of these bound asymmetries. The fact that BGPO and VRPO-OL show matching gradient statistics at equal nt (their curves overlap) suggests that, at least on-policy, the net effect of the bound construction does not introduce systematic error beyond what VRPO-OL already has. However, this measurement is taken immediately after policy refresh (on-policy, d_j ≈ 0), where both bounds are exact. The off-policy regime where the asymmetries would manifest is not measured.
The performance results in Table 2 (BGPO at different nt) provide indirect evidence: if the bound asymmetry were severely distorting optimization, we might expect non-monotonic or erratic performance as nt increases, since larger nt also changes the d_j variance and thus the bound gap. The generally monotonic improvement from nt=1 to nt=16 (with the nt=4 dip) suggests the asymmetry is not catastrophic, but does not quantify its effect.
Mitigation status. Not addressed. The paper proves that both bounds are valid lower bounds and that they are exact on-policy, but does not analyze their off-policy tightness, their relative error magnitudes, the potential for sign flipping in the Taylor bound, or the discontinuity at A = 0. A natural mitigation — using a smoothed transition between the two bounds near A = 0, or using a single bounding technique with importance sampling to handle sign issues — is not explored. The limitation is implicit in the mathematical construction but not acknowledged as a potential source of training instability or bias.
6.5 The Evaluation Protocol Uses Best-Checkpoint Selection Without Confidence Intervals, Limiting Reliability of Performance Comparisons
The assumption or constraint. All results in Table 1 and Table 2 are reported from a single training run per method per task. The models are evaluated every 20 training steps, and "results from the best-performing checkpoint" are reported, following Zhao et al. (2025a). The paper does not report confidence intervals, standard deviations across multiple runs with different random seeds, or any statistical test comparing methods. The test sets are standard benchmarks with fixed sizes — MATH500 has 500 questions, GSM8K has approximately 1,300 test questions — meaning performance estimates have inherent sampling variability.
The consequence. Without multiple runs, we cannot distinguish genuine algorithmic differences from run-to-run variance due to random seed effects, batch ordering, or the inherent stochasticity of RL training (response sampling, MC sampling for ELBO estimation, dropout, etc.). This is particularly concerning for the narrow performance gaps in Table 1:
- MBPP: BGPO (41.7%) vs. VRPO-OL (41.5%) — a 0.2-point gap that is almost certainly within the noise floor of single-run evaluation.
- Sudoku: BGPO (26.9%) vs. diffu-GRPO (26.7%) vs. VRPO-OL (26.1%) — all three methods cluster within 0.8 points, and the ordering could easily change with a different random seed.
- HumanEval: BGPO (47.6%) vs. diffu-GRPO (47.0%) vs. baseline (45.1%) — the gap between the two best methods (0.6 points) is small relative to typical HumanEval variance.
The larger gaps — particularly Countdown (87.5% vs. 84.8% vs. 53.1%) and MATH500 (45.7% vs. 44.1% vs. 43.1%) — are more likely to be robust, but we have no way to quantify how likely. The paper makes strong claims about BGPO "significantly outperforming" previous algorithms, but without statistical backing, "significantly" is being used colloquially, not in the statistical sense.
The best-checkpoint selection also introduces a subtle bias: each method gets to pick its luckiest evaluation from the entire training trajectory. If one method has higher variance in evaluation performance (oscillating more between good and bad checkpoints) while another has lower variance (steadier performance), the high-variance method benefits more from best-checkpoint selection, even if its average performance is no better. The paper provides training reward curves (Figure 2) that show some methods exhibiting more oscillation than others (e.g., diffu-GRPO on Countdown appears more variable than BGPO), but evaluation accuracy curves are not shown, so we cannot assess this effect.
What evidence exists in the paper. The training curves in Figure 2 provide trajectory-level information, but these show training reward, not test accuracy. The relationship between training reward and test accuracy is not necessarily monotonic — especially when using best-of-Group advantage normalization, the model could be optimizing to produce responses that score well under the reward function without improving ground-truth accuracy (reward hacking). The paper does not plot test accuracy versus training steps, which would reveal whether the best-checkpoint selection masks differences in training stability.
Mitigation status. Not addressed. The single-run, best-checkpoint protocol is standard in the RL fine-tuning literature (cited as following Zhao et al., 2025a), but it represents a concession to computational cost rather than a principled evaluation choice. The paper does not acknowledge the limitation, does not report any measure of statistical uncertainty, and does not discuss the potential for best-checkpoint selection to favor methods with higher variance. Given that the paper uses 8 × H800 GPUs and the per-step time is ~150 seconds, even 3 independent runs per method would multiply the computational cost by 3× — but for the narrower performance gaps (MBPP, Sudoku), this cost is arguably necessary to support the comparative claims. The limitation is compounded by the single-model-family constraint (Section 6.1): with only one base model and one run each, the entire experimental section rests on a single degree of freedom at the model level.
6.6 No Demonstration That BGPO's Gains Persist or Compound Over Multiple RL Iterations or with Iterated Self-Improvement
The assumption or constraint. The paper trains models with BGPO for a fixed number of steps (evidenced by the x-axes in Figure 2, which range from 0–600 steps for math, 0–400 for planning) and reports final performance. The training is a single stage of RL fine-tuning: start from the supervised-fine-tuned LLaDA-8B-Instruct, run BGPO, evaluate. The paper does not explore whether the benefits of larger nt compound over multiple rounds of RL — for example, training with BGPO, using the resulting model to generate new training data, and training again (iterated RL or self-improvement loops, analogous to STaR or ReST^EM as discussed in the prior paper's analysis framework).
The consequence. This matters because the paper's central claim is that BGPO provides a "more accurate optimization direction" (Section 4.2). In a single RL stage, this manifests as faster reward improvement and higher final accuracy. But in iterated RL — where the model's outputs from one stage become the training data for the next — the consequences of gradient quality could be very different:
-
Error amplification across iterations. If BGPO's more accurate gradients lead to a better policy after one stage, that policy generates slightly better training data for the next stage. The improved data quality could interact with BGPO's gradient accuracy to produce compounding gains — the gap between BGPO and VRPO-OL might widen over multiple iterations as the data distribution improves. Alternatively, if all methods eventually converge to similar policies given enough iterations (the optimization landscape has a single basin of attraction), the initial advantage might wash out — BGPO might reach the same final performance but in fewer iterations, which is still valuable but less transformative.
-
Distribution shift effects on the lower bound. In iterated RL, the training data in later stages is generated by a policy that has been RL-fine-tuned, not the original SFT model. This data is likely to have different properties — different response lengths, different error patterns, different diversity — that could affect the ELBO estimation quality and the bound gap. The paper's gradient bias/variance measurements (Figures 3–4) are taken relative to a single model checkpoint; they do not characterize how these statistics evolve as the policy improves.
-
Saturation of the nt benefit. Table 2 shows improvement from nt=1 to nt=16, but the gains appear to be flattening (the jump from nt=8 to nt=16 is 0.4 points on MATH500, smaller than the 1.6-point jump from nt=4 to nt=8). If the benefit of larger nt saturates, then BGPO's advantage over VRPO-OL is a one-time gain in the first RL stage, and further iterations would see diminishing returns. The paper does not test whether nt=32 or nt=64 would yield continued improvements, or whether the saturation point depends on the base model's quality (i.e., a weaker base model might benefit from larger nt into higher ranges).
-
Practical deployment of self-improvement pipelines. One of the most compelling use cases for RL on dLLMs is automated self-improvement — using the model to generate solutions, filtering for correctness, and retraining. BGPO's memory efficiency is directly relevant here because self-improvement loops require running RL repeatedly. If BGPO's advantage is primarily in the first stage, practitioners might use BGPO for initial fine-tuning and switch to a cheaper method (diffu-GRPO) for subsequent iterations. If the advantage compounds, BGPO would be essential throughout. The paper provides no guidance.
What evidence exists in the paper. Table 3 (out-of-domain evaluation) provides some indirect evidence about generalization but does not test iterated training. The training reward curves in Figure 2 show BGPO maintaining an advantage over baselines throughout the training trajectory, but these are single-stage curves — they terminate at a single checkpoint, and the model is not used to generate new training data for a second stage. The paper does not discuss iterated RL, self-improvement loops, or multi-stage training.
Mitigation status. Not addressed. The paper frames BGPO as a general RL algorithm for dLLMs and demonstrates its effectiveness in single-stage fine-tuning. The iterated RL setting — which is where the field is headed (cf. DeepSeek-R1's multi-stage training, self-improvement in prior work on autoregressive models) — is not discussed, and no experiments probe whether the benefits of larger nt are front-loaded or persistent. This is a significant gap for practitioners considering adopting BGPO in multi-stage training pipelines, which are increasingly standard for reasoning-focused LLM development.
7. Implications and Future Directions
How This Work Changes the Landscape
This paper causes a reframing rather than a paradigm shift. The core contribution is diagnostic: it identifies that the primary obstacle to applying reinforcement learning to diffusion language models is not an algorithmic problem of log-likelihood approximation, but a systems problem of GPU memory constraints forcing suboptimal Monte Carlo sample sizes. This reframing reclassifies the bottleneck from "we need better likelihood estimators" to "we need memory-efficient ways to compute gradients through existing estimators at larger sample sizes," which opens a different solution space entirely.
The practical consequence of this reframing is that BGPO keeps the same ELBO-based likelihood estimator that VRPO pioneered, but changes only the gradient computation path — and gains 1.6 points on MATH500 (44.1% to 45.7%) and 2.7 points on Countdown (84.8% to 87.5%) as a result. These gains do not come from a new mathematical insight about diffusion likelihoods. They come from recognizing that the memory bottleneck was preventing the existing method from working at the sample sizes its theory requires. The paper's gradient bias and variance measurements (Figures 3 and 4) are the linchpin evidence: they show that VRPO-OL's gradient quality would improve with larger nt if memory permitted, and BGPO simply enables that improvement by linearizing the computation graph.
This reframing changes the research landscape in several specific ways. First, it democratizes ELBO-based RL for dLLMs by removing the requirement for specialized high-memory hardware. On an H800 GPU (80GB), VRPO-OL caps at nt=4; BGPO reaches nt=16 with memory to spare (Figure 1, left panel), and the flat memory curve suggests nt=64 or nt=128 would be feasible without hardware upgrades. This is not a minor efficiency improvement — it is the difference between running the method at a sample size where gradient noise dominates versus one where the ELBO estimate is reliable.
Second, it redirects research attention away from single-pass likelihood approximations. The paper shows that diffu-GRPO's approach — approximating log π_θ(y|x) from a single forward pass — introduces irreducible bias in both the gradient (Figures 3-4) and the final performance (e.g., 53.1% vs. 87.5% on Countdown, Table 1). The dramatic Countdown gap is particularly telling: on a task where the base model is weak and exploration matters, the structural bias of single-pass estimation prevents the policy from ever finding good solutions. BGPO's results suggest that future work on RL for dLLMs should use principled ELBO-based (or tighter variational) likelihood bounds rather than heuristic single-pass estimates, and should focus research effort on making those bounds computationally feasible rather than on developing new heuristics.
Third, it establishes the response rollout as the dominant computational cost and the RL objective computation as a secondary concern for training time. Figure 5 shows that quadrupling the MC sample size from nt=4 to nt=16 increases per-step time by only 16% (130.2 to 151.5 seconds). This means that improvements to the RL objective — better advantage estimation, lower-variance gradient estimators, more sophisticated bounds — are essentially "free" in wall-clock terms as long as they don't increase the number or cost of response generations. This finding should shift research priorities: invest effort in better training signals (which are cheap to compute) rather than in faster generation (which already dominates and benefits all methods equally). It also means that the practical adoption barrier for BGPO is low — existing VRPO training pipelines can switch to BGPO with minimal code changes and a negligible time penalty, while gaining the benefit of larger nt.
Finally, the paper implicitly resolves a tension in the dLLM RL literature between the "cheap but biased" single-pass approach and the "principled but memory-constrained" ELBO approach. The resolution is not a compromise — it is the observation that the ELBO approach was never fundamentally flawed, just practically constrained. BGPO removes the constraint, making the principled approach both more accurate (larger nt) and comparably efficient to the biased approach. The question "should we use single-pass or ELBO-based likelihood approximation?" now has a clear answer for practitioners with BGPO: use ELBO-based, with as large an nt as your time budget allows, because the time cost is marginal and the accuracy benefit is measurable.
Follow-Up Research This Work Enables
Directly measuring the bound gap and its impact on training dynamics. The paper proves that BGPO's lower bound is exact on-policy (when d_j = 0) but does not characterize how large the bound gap becomes as the policy drifts off-policy within a training iteration. A targeted follow-up would instrument the training loop to measure both $\hat{J}(\theta)$ and $\hat{J}_{lb}(\theta)$ at each gradient step within an iteration, computing the bound gap $\hat{J}(\theta) - \hat{J}_{lb}(\theta)$ as a function of the number of steps since the last policy refresh. If the gap remains small (e.g., <1% of the objective value) for the first few gradient steps and then grows, a natural mitigation is adaptive policy refresh frequency: refresh the old policy after k gradient steps rather than after a fixed number of iterations, where k is chosen to keep the bound gap below a threshold. If the gap is large even after a single step, this would indicate that BGPO's on-policy equivalence proof, while mathematically correct, is too optimistic in practice due to the variance of the ELBO estimate causing the policy to take steps that move it substantially off-policy even within one gradient update. Either outcome — small gap or large gap — would provide actionable guidance for tuning the refresh frequency, which the current paper leaves unspecified.
Combining BGPO with PRM-guided tree search during response generation. The paper treats response generation as a fixed process (block-wise denoising from the old policy) and focuses exclusively on improving the RL objective computation. However, the response rollout is the dominant cost (Section 4.5), and the quality of generated responses directly determines the quality of the RL training signal. A natural extension is to replace the standard denoising process with a search procedure — for example, using a process reward model to guide which tokens to unmask at each denoising step, or generating multiple candidate completions per step and selecting the best. BGPO's memory efficiency would be critical here because tree search multiplies the number of forward passes already, and adding RL objective computation on top could easily exceed memory limits. The experiment would be: train a PRM on the base dLLM's outputs (following the Monte Carlo rollout procedure from the reference paper on test-time compute), use it to guide response generation during RL training, and compare (a) standard generation + BGPO, (b) PRM-guided generation + VRPO-OL (likely infeasible due to memory), and (c) PRM-guided generation + BGPO. The hypothesis is that the combination of better proposals (via search) and better gradient signals (via larger nt) would yield compounding gains, particularly on hard exploration problems where both proposal quality and gradient accuracy matter.
Scaling BGPO to larger models and characterizing the memory crossover point. The paper acknowledges that experiments are limited to 8B-parameter models due to the lack of larger open-source dLLMs (Section 7). As larger dLLMs become available (either through scaling LLaDA-style training or through ongoing commercial development like Gemini Diffusion and Mercury), the first-priority experiment is to run BGPO at increasing model scales and measure: (a) the absolute memory usage of BGPO at nt=16 as model size increases, (b) the maximum feasible nt at each model scale given fixed GPU memory (80GB), and (c) the performance benefit of that maximum nt compared to VRPO-OL's maximum nt at the same scale. The paper hypothesizes that BGPO's advantage should increase with model size because VRPO-OL's memory constraint becomes more binding (larger models → larger activations → even smaller feasible nt). But there is a counter-hypothesis: at sufficiently large scales, a single forward/backward pass may consume most of the GPU memory, making even BGPO's nt=1 infeasible without model parallelism — at which point BGPO's constant-memory property relative to nt becomes irrelevant because the baseline memory dominates. This crossover point — the model size at which BGPO's advantage disappears — should be empirically characterized, not assumed away.
Stress-testing BGPO with on-policy data collection and multi-turn RL. The paper's training uses offline data collected once per iteration: the old policy generates G=8 responses per prompt, and these are reused for all gradient steps in that iteration. This is standard for GRPO-style algorithms but introduces a growing off-policy gap as the policy updates. A strong stress test for BGPO would be to run fully on-policy training: after each gradient step, discard the old responses, generate new ones from the updated policy (now the new old policy), and recompute advantages. This would keep d_j ≈ 0 at all times, making the bound gap zero by construction — but it would dramatically increase the response generation cost (one round of generation per gradient step instead of per iteration). The experiment would compare: (a) standard BGPO with per-iteration response collection and multiple gradient steps per iteration, (b) fully on-policy BGPO with per-step response collection, and (c) a hybrid with adaptive refresh based on a KL-divergence threshold between current and old policy. The key measurement is the effective gradient quality per unit of wall-clock time: does the improved gradient signal from on-policy training (zero bound gap) compensate for the dramatically increased generation cost? This experiment would directly inform the practical tradeoff that practitioners face when deploying BGPO in production RL pipelines.
Extending the boundary-guided construction to other nonlinear RL objectives. The paper's lower bound construction — splitting on the advantage sign and applying different inequalities — is specific to the exponential function in the policy gradient objective (Eq. 7). But the general pattern — linearize a nonlinear function of a Monte Carlo average by constructing a sign-dependent bound that is exact at the current policy — could apply to other RL objectives that involve nonlinear transformations of estimated quantities. Concrete candidates include: (a) PPO-style clipped objectives where the clipping function creates a nonlinearity that couples samples, (b) objectives with KL penalties where the KL divergence is estimated via Monte Carlo and appears inside a nonlinearity, and (c) multi-step RL objectives (e.g., GAE) where advantage estimates involve nested expectations that compound Monte Carlo noise. For each, the research question is: can we construct a linear lower bound with on-policy equivalence that preserves the original objective's optimization properties while enabling memory-efficient gradient accumulation? A negative result — finding objectives where no such bound exists or where the bound gap is unacceptably large — would be equally valuable, as it would delineate the limits of the boundary-guided approach.
Testing whether the benefit of larger nt saturates and characterizing the saturation point per task. Table 2 shows monotonic improvement from nt=1 to nt=16, but the gradient variance curve in Figure 3 appears to be flattening — suggesting diminishing returns. A systematic follow-up would run BGPO at nt = 1, 2, 4, 8, 16, 32, 64, 128 on a representative subset of tasks (e.g., MATH500, Countdown, and HumanEval) to map the full curve of performance versus nt. The key questions: (a) at what nt does performance saturate for each task? (b) does the saturation point depend on task difficulty (earlier saturation for easy tasks where gradient noise matters less) or base model strength (earlier saturation for stronger base models)? (c) can we predict the saturation point from the ELBO variance at a given nt, enabling practitioners to choose nt without a full sweep? If saturation occurs at nt=32 for most tasks, the practical recommendation is clear: use nt=32 and stop. If saturation does not occur even at nt=128 (i.e., continued gains), this would indicate that ELBO estimation noise is a fundamental bottleneck that even large nt cannot fully resolve, and the field should invest in fundamentally lower-variance likelihood estimators rather than merely enabling larger sample sizes.
Practical Applications and Downstream Use Cases
Fine-tuning dLLMs for reasoning tasks with limited GPU memory. The most immediate practical application is straightforward: any team currently using VRPO or diffu-GRPO to fine-tune LLaDA (or similar dLLMs) on math, coding, or planning tasks can switch to BGPO, increase nt to 16 (or higher if memory permits), and expect a ~1-2 point accuracy improvement on standard benchmarks with only a ~16% increase in per-step training time. This is a drop-in replacement — BGPO uses the same VeRL framework, the same hyperparameters, and the same training data as VRPO-OL, requiring only a change in the objective computation (Algorithm 1 replacing the direct objective computation). For teams with limited hardware (single 8-GPU H800 node, as used in the paper), this is the difference between running a suboptimal RL algorithm and running one that can use sufficient MC samples for reliable gradient estimates. The 4× increase in effective sample size (from nt=4 to nt=16) translates directly to the gradient quality improvements documented in Figures 3-4, without requiring hardware upgrades.
Enabling RL for dLLMs on long-context tasks. The paper notes that VRPO-OL's feasible nt drops from 4 to 2 for coding tasks because longer prompts consume more GPU memory. For tasks with even longer contexts — document summarization, multi-turn dialogue, repository-level code generation — VRPO-OL might be restricted to nt=1, effectively reducing it to a noisy single-sample estimate that loses the benefit of Monte Carlo averaging entirely. BGPO's constant memory profile is especially valuable here: since memory usage is independent of nt, the same nt=16 can be used regardless of prompt length, as long as a single forward/backward pass fits in memory. This opens RL fine-tuning for dLLMs on long-context tasks that were previously inaccessible to ELBO-based methods. A concrete deployment scenario: fine-tuning a dLLM for legal document analysis where prompts are 4K+ tokens. VRPO-OL at nt=1 would suffer from the same gradient bias as diffu-GRPO (Figure 4); BGPO at nt=16 provides the ELBO-based method's theoretical advantages at a memory cost that fits on standard hardware.
Iterated self-improvement pipelines for dLLMs. One of the most promising use cases for RL on language models is automated self-improvement: generate solutions with the current model, filter for correctness, and retrain on the successful trajectories, repeating for multiple rounds. Each round requires running RL fine-tuning, and the quality of the RL signal in early rounds determines the quality of the data in later rounds — errors compound. BGPO's improved gradient quality (lower bias and variance at larger nt) means that each round of self-improvement produces a better policy, which generates better data for the next round, potentially leading to compounding gains across rounds. The paper's out-of-domain results (Table 3) provide preliminary evidence that BGPO-trained models generalize reasonably, which is a prerequisite for self-improvement (the model must generate correct solutions on problems it wasn't explicitly trained on). A practical self-improvement loop would: (1) use BGPO to fine-tune LLaDA-8B-Instruct on a seed dataset, (2) use the resulting model to generate solutions on a larger unlabeled dataset, (3) filter for correctness using a verifier or execution feedback, (4) add the correct solutions to the training set, and (5) repeat from step 1. BGPO's memory efficiency means each round of fine-tuning can use large nt, maintaining gradient quality even as the training data distribution shifts across rounds.
When to Prefer This Method
The paper implicitly positions BGPO against VRPO-OL (same ELBO-based estimator, different gradient computation) and diffu-GRPO (different estimator, single-pass). The tradeoffs that emerge from the experiments are:
-
Prefer BGPO over VRPO-OL when GPU memory is constrained and you cannot run VRPO-OL at your target nt, but you can fit a single forward/backward pass. This is the H800 scenario documented in Figure 1: VRPO-OL caps at nt=4; BGPO reaches nt=16 in the same memory budget. The performance benefit of the larger nt (Table 2) outweighs any bound gap introduced by the lower-bound construction. If you have access to hardware with substantially more memory (e.g., H100 with 188GB HBM3, or multi-node model parallelism that distributes activation storage), VRPO-OL at equally large nt should be equally effective, since the gradient statistics match at the same nt (Figures 3-4) — but at that point, BGPO's additional constant-memory property provides no advantage, and VRPO-OL's direct objective avoids any off-policy bound gap.
-
Prefer BGPO over diffu-GRPO when your task requires exploration and the base model is weak. The Countdown results (87.5% vs. 53.1%) demonstrate that single-pass estimation catastrophically fails when the policy needs to discover new solution strategies that are far from its initial distribution. On tasks where the base model is already strong (e.g., HumanEval at 45.1% baseline, where diffu-GRPO achieves 47.0% vs. BGPO's 47.6%), the gap narrows because even biased gradients can guide the policy to nearby optima. The gradient bias in Figure 4 suggests diffu-GRPO will always have a systematic error floor that no amount of training can overcome; BGPO's bias can be driven arbitrarily low by increasing nt. For production systems where final performance matters more than the last 1-2 percentage points, BGPO is the safer choice regardless of task.
-
Prefer diffu-GRPO over BGPO when training speed is the absolute priority and the base model is already near the performance ceiling. diffu-GRPO's per-step time (128.8 seconds) is ~15% faster than BGPO (151.5 seconds), and if the task is one where the structural bias of single-pass estimation does not prevent convergence (e.g., the coding tasks in Figure 2 where all methods converge similarly), the time savings could be worthwhile. However, this is a narrow window: the paper shows that even on coding tasks, BGPO achieves the best final numbers (47.6% vs. 47.0% on HumanEval), so the speed advantage only matters if you are willing to sacrifice final quality for iteration speed — a reasonable tradeoff during early-stage experimentation but not for final model releases.
