Listwise Policy Optimization: Group-based RLVR as Target-Projection on the LLM Response Simplex

ArXiv: 2605.06139

🎯 Pitch

Standard group-based RL for LLMs, like GRPO, is secretly just a crude first-order guess at projecting the policy toward an implicit reward-weighted target—and the authors show you can compute that projection exactly on the sampled response simplex, yielding bounded, self-correcting gradient coefficients. Listwise Policy Optimization decouples the target distribution from the divergence used to reach it, turning what were ad-hoc advantage normalizations into a principled framework that consistently outperforms matched baselines on reasoning benchmarks from 1.5B to 14B parameters.

---

1. Executive Summary

This paper introduces Listwise Policy Optimization (LPO), an explicit target-projection framework that reinterprets group-based policy gradient methods in RLVR as implicitly performing approximate reverse-KL projections toward a reward-weighted softmax target on a finite response simplex. Building on this geometric insight, the authors decouple target construction from divergence projection — defining a closed-form listwise Gibbs target on the sampled responses and then projecting the policy onto it via exact divergence minimization — yielding gradient coefficients that are bounded, zero-sum, and self-correcting by design. Experiments across logic, mathematics, programming, and multi-modal reasoning tasks with diverse LLM backbones (Qwen3, DeepSeek-R1-Distill, Llama-3.1, Mistral) demonstrate that LPO consistently improves training performance over matched group-based PG baselines (GRPO, Dr.GRPO, MaxRL), with the forward-KL variant winning in 13/15 Pass@k scenarios, while the reverse-KL variant exactly recovers standard policy gradients at the on-policy point — establishing that the framework provides gains orthogonal to temperature design and scales from 1.5B to 14B parameters.

2. Context and Motivation

The Core Problem: Group-Based RLVR Works, But We Don't Understand Why

The paper addresses a fundamental tension in the current practice of reinforcement learning with verifiable rewards (RLVR) for LLM post-training. The dominant paradigm — represented by GRPO (Shao et al., 2024) and its variants — samples a group of $K$ responses per prompt, scores them with a rule-based verifier, and updates the policy using advantages computed from within-group reward statistics. This approach has proven remarkably effective: it powers the training pipelines behind DeepSeek-R1 (Guo et al., 2025) and numerous other reasoning models, and has become the de facto standard for incentivizing reasoning capabilities in LLMs.

However, the paper identifies a critical gap: we lack a principled understanding of what these algorithms are actually optimizing. The literature has accumulated a growing collection of advantage normalization schemes — GRPO uses $A_k = (R_k - \mu_G)/\sigma_G$, Dr.GRPO drops the $\sigma_G$ normalization, MaxRL uses $A_k = (R_k - \mu_G)/\mu_G$, REINFORCE++ introduces batch-level statistics — each justified through empirical performance and training stability arguments, but without a unifying framework that explains why these particular formulas work and what they share in common.

This matters for two reasons. First, practical: without understanding the optimization objective, practitioners are left tuning advantage formulas as hyperparameters through trial and error — the paper explicitly notes (Section 1, Introduction) that "viewing them purely through the manner of advantage normalization obscures the intrinsic optimization mechanism." Second, theoretical: if these methods are all approximate instances of a more fundamental optimization principle, then making that principle explicit could unlock design choices — such as alternative divergence measures — that are inaccessible under the current paradigm.

The Conflicting Landscape of Advantage Normalization

The paper characterizes the existing group-based RL landscape as fragmented. Table 1 summarizes three representative methods with their implicit targets:

• Dr.GRPO / RLOO ($\tau \approx 1$): uses $A_k = R_k - \mu_G$, yielding a fixed-temperature softmax target $\text{softmax}(R)$
• GRPO / DAPO ($\tau = \sigma_G$): uses $A_k = (R_k - \mu_G)/\sigma_G$, yielding a target $\text{softmax}(R/\sigma_G)$ with adaptive sharpness that depends on group difficulty
• MaxRL ($\tau = \mu_G$): uses $A_k = (R_k - \mu_G)/\mu_G$, yielding $\text{softmax}(R/\mu_G)$ with a temperature proportional to the success rate — implementing an implicit curriculum where hard prompts get sharp targets and easy prompts get diffuse ones

Each method has its advocates and empirical success stories, but the choice among them remains ad hoc. The paper's central insight is that these differences reduce to a single parameter: the temperature $\tau$ controlling target sharpness. The centering term $\mu$ (whether group mean, leave-one-out baseline, or greedy-decode baseline) cancels out entirely under the softmax — a property of shift-invariance that the paper exploits. This means all these methods are attempting to project toward the same family of targets ($\text{softmax}(R/\tau)$), differing only in how $\tau$ is set.

The paper frames this as a unification, not a criticism. But it also argues that the current approach — computing advantages and then applying a policy gradient — is an unnecessarily approximate way to achieve this projection. The policy gradient update in Eq. 3 is a first-order Taylor approximation of a reverse KL projection (Proposition 1), and this approximation is exact only at the on-policy point ($\pi_\theta = \pi_b$). As the policy drifts from the sampling distribution — which happens inevitably with multiple gradient steps per batch of rollouts — the approximation error grows as $O(\bar{\delta} \cdot (1 + \|A\|_\infty)/K)$, where $\bar{\delta}$ measures off-policy drift (Appendix B.2).

The Specific Gap: Implicit Approximation vs. Explicit Projection

This leads to the paper's core research question (Section 1):

"What properties emerge when this target-projection is made explicit, and how does this decoupled optimization space influence RLVR of LLMs?"

The motivation is not merely aesthetic. The paper identifies four concrete limitations of the implicit approximation approach:

1. The target is never cleanly separated from the optimization method. In existing PG methods, the target distribution $w^* = \text{softmax}(R/\tau)$ is implicit in the advantage formula — you cannot change the target without changing the advantage normalization, and vice versa. This coupling prevents independent exploration of "what distribution should we aim for" versus "how should we move the policy toward it."

2. Only reverse KL projections are accessible. Policy gradient methods implicitly perform reverse KL minimization (Proposition 1), which has a well-known mode-seeking property — it tends to concentrate probability mass on the mode of the target distribution, potentially sacrificing diversity. Forward KL, with its mode-covering property that could preserve reasoning diversity, is structurally inaccessible under the PG paradigm because there is no reasonable advantage vector $A$ whose softmax target combined with a PG update would yield a forward KL projection.

3. The approximation degrades off-policy. The equivalence $g_{\text{PG}} = -\nabla_\theta D_{\text{KL}}(P_\theta \| w^*)$ holds exactly only at $\pi_\theta = \pi_b$. In practice, RLVR training typically performs multiple gradient updates per batch of rollouts, and with importance sampling ratios $r_k = \pi_\theta(y_k|x)/\pi_b(y_k|x)$ deviating from 1, the gradient coefficients diverge from the true reverse KL gradient. The paper quantifies this error explicitly (Appendix B.2), showing it scales with the off-policy drift $\bar{\delta}$. This means that hyperparameters like clipping ratios and number of inner epochs are effectively patching over a structural mismatch between the intended objective and the actual gradient.

4. The finite response simplex enables exact computation that is not being exploited. In classical RL with continuous action spaces, exact projection toward a Gibbs target is intractable — the partition function requires an integral over the continuous action space, so methods like MPO (Abdolmaleki et al., 2018) and AWR (Peng et al., 2019) must resort to pointwise projections (weighted regression, $-\sum_k w^*_k \log \pi_\theta(y_k)$). But in group-based RLVR, the sampled responses $\{y_1, \ldots, y_K\}$ naturally form a finite simplex $\Delta^{K-1}$ where exact normalization is a simple finite sum. This makes exact target construction and exact projection computationally trivial — yet existing methods don't leverage this, instead relying on first-order PG approximations.

How This Paper Positions Itself

The paper situates itself at the intersection of three intellectual traditions:

From RLVR practice: It inherits the group-based sampling paradigm (GRPO, Dr.GRPO, MaxRL) and all the engineering infrastructure — batched rollout generation, verifier-based reward computation, policy gradient optimization. The paper is explicit that it is not proposing a new training pipeline, but rather a new optimization objective that can be dropped into existing RLVR frameworks with no additional computational cost (Section 4.3: "The training pipeline is identical to standard group-based RL algorithms, with no additional computational cost").

From RL-as-inference: The target-projection framework has deep roots in the RL-as-probabilistic-inference literature (Dayan & Hinton, 1997; Ziebart, 2010; Levine, 2018), where control is cast as inference under a KL-regularized objective, giving rise to reward-weighted Gibbs targets and trust-region updates. Classical algorithms like MPO, AWR, and V-MPO (Song et al., 2019) all construct such targets and project toward them. However, as noted, these methods operate in continuous action spaces and project pointwise — treating each sampled action independently without normalization across the sample set. The paper's key move is recognizing that the finite response simplex of LLM generation permits exact listwise normalization, enabling a fundamentally different projection geometry.

From listwise learning-to-rank: The formulation of the policy's relative preference over $K$ responses as a listwise distribution $P_\theta = \text{softmax}(s_\theta)$ on the simplex draws directly from classical choice and ranking models (Luce, 1959; Plackett, 1975; Cao et al., 2007). Recent LLM alignment work — notably DPO (Rafailov et al., 2024) and LiPO (Liu et al., 2025a) — has used listwise preference structures to model relative comparisons among responses. The paper notes that when $K = 2$, LPO with forward KL reduces to a binary cross-entropy objective with soft targets, connecting directly to the DPO formalism but derived from an online RL perspective rather than offline preference optimization (Appendix C.5).

The paper explicitly distinguishes itself from concurrent work (Section 1, Contributions; Appendix C.1):

• TPO (Kaddour, 2026) independently proposes a similar listwise forward KL projection, providing empirical corroboration but not the unifying analytical framework that recovers existing group-based PG methods or the theoretical improvement guarantees.
• FlowRL (Zhu et al., 2025) minimizes reverse KL against a Gibbs target approximated by a learned partition function network, whereas LPO exploits the finite simplex for exact, closed-form normalization.
• Shu et al. (2026) explore a reference-sampled Boltzmann projection that shares the target-projection structure but differs in both the reference policy treatment and the theoretical analysis focus.

The paper's positioning is therefore: unify the existing group-based PG literature under a common geometric framework, then show that making the implicit structure explicit yields both theoretical guarantees (monotonic improvement, bounded/zero-sum/self-correcting gradients, mode-coverage) and consistent empirical gains without introducing new hyperparameters. The temperature $\tau$ is intentionally not tuned — it is inherited from the paired PG baseline's advantage normalization scheme to isolate the effect of the projection mechanism (Section 4.3: "we intentionally avoid introducing new tuning burdens").

The Practical Stakes

Beyond the theoretical unification, the paper's motivation has concrete practical implications:

• Training stability: Existing PG methods are known to suffer from entropy collapse and training instability in RLVR (Section 5.3). If the gradient coefficient properties of exact projection (bounded, zero-sum, self-correcting) translate to more stable optimization trajectories, this directly addresses a pain point in RL training pipelines.

• Diversity preservation: Mode collapse — where the policy converges to a single solution strategy — is a recognized failure mode in RLVR, particularly problematic for Pass@k evaluation where diverse correct solutions need to be maintained in the policy's distribution. Forward KL's mode-covering property (Corollary 2) provides a principled mechanism for preserving diversity that is not available in the PG paradigm.

• Sample efficiency: By enabling exact projection rather than first-order approximation, LPO may make better use of limited samples per prompt. The paper explores this via group size experiments (Section 5.4.2, Figure 7), showing advantages are particularly pronounced at small $K$.

• Algorithm design space: The decoupled target-projection framework opens an entire design dimension — choice of projection divergence — that was previously inaccessible. The paper implements forward and reverse KL as instantiations, but the framework admits any differentiable divergence on the simplex (Appendix C.6), suggesting rich possibilities for task-specific or stage-specific optimization strategies.

3. Technical Approach

3.1 Reader Orientation

This paper develops Listwise Policy Optimization (LPO), a new algorithm for fine-tuning large language models on reasoning tasks with verifiable rewards. Rather than proposing yet another advantage normalization scheme in the GRPO family, LPO makes explicit what existing methods do implicitly — it constructs a target distribution over the sampled responses that represents "where we want the policy to be," and then performs an exact projection of the policy toward that target using divergence minimization, all on a finite probability simplex that exists naturally because we sample a fixed number of responses per prompt.

3.2 Big-Picture Architecture (Diagram in Words)

The LPO system has three major components that execute in sequence during each training iteration:

1. Target Construction — takes the $K$ sampled responses for a prompt, their rewards from the verifier, and the current policy's listwise distribution, and produces a target distribution $w^*$ on the response simplex that re-weights responses toward higher rewards while staying within a trust region.

2. Listwise Distribution — represents the policy's relative preference over the $K$ responses as a softmax-normalized distribution $P_\theta$, which lives on the same simplex as the target and captures how much the current policy prioritizes each response compared to the behavior policy that generated them.

3. Projection — computes the gradient that moves $P_\theta$ toward $w^*$ by minimizing a chosen divergence (forward KL or reverse KL), yielding per-response gradient coefficients that are bounded, zero-sum, and self-correcting.

Information flows as follows: a batch of prompts enters → the behavior policy generates $K$ responses each → the verifier assigns rewards → the pre-update policy computes listwise logits → the target is constructed as a softmax of reward-scaled logits → the policy's current listwise distribution is computed from the updated parameters → a divergence between target and current distribution is minimized → the policy parameters are updated → repeat.

3.3 Roadmap for the Deep Dive

• First, the listwise distribution representation (Section 4, Eqs. 4 and 6), which is the key representational innovation — reparameterizing the policy as a softmax over $K$ responses on the simplex. This is necessary to understand everything else because both the target and the projection operate on this same simplex.

• Second, the target construction step (Section 4.1, Theorem 1), which demystifies where the implicit softmax targets in existing PG methods come from by deriving them as the closed-form solution to a proximal RL objective restricted to the sampled response simplex. This explains why all existing methods are reaching for the same Gibbs target family, differing only in temperature.

• Third, the projection mechanism (Section 4.2, Examples 1-2), which shows how moving from first-order PG approximation to exact divergence minimization yields gradient coefficients with structural properties (bounded, zero-sum, self-correcting) that have direct consequences for optimization stability and diversity.

• Fourth, the theoretical properties (Corollaries 1-2, Theorem 2), which prove that the decoupled target-projection approach guarantees monotonic improvement of the listwise reward and provides a log-barrier against mode collapse.

• Fifth, the practical implementation (Section 4.3, Algorithm 1), which shows how LPO drops into existing GRPO pipelines with zero additional computational cost by inheriting the temperature from the baseline's advantage normalization scheme.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a methodology paper with theoretical analysis whose core idea is that group-based policy gradient methods in RLVR implicitly project toward a Gibbs target on a finite response simplex, and that making this projection explicit via decoupled target construction and divergence minimization yields optimization properties that the first-order PG approximation cannot provide.

---

The Listwise Distribution: Reparameterizing the Policy on the Response Simplex

The fundamental representational move is to stop thinking about the policy $\pi_\theta$ as producing individual token probabilities and instead think about how it distributes probability comparatively across the $K$ sampled responses for a given prompt. The paper defines the listwise distribution $P_\theta$ as a softmax over log-importance-ratios (Eq. 4):

$P_{\theta,k} = \frac{\exp(s_{\theta,k})}{\sum_{j=1}^{K} \exp(s_{\theta,j})} = \text{softmax}(s_\theta)_k, \quad \text{with } s_{\theta,k} = \log \frac{\pi_\theta(y_k|x)}{\pi_b(y_k|x)}$

where $P_{\theta,k}$ is the relative probability the current policy assigns to response $k$ out of the $K$ sampled responses, $s_{\theta,k}$ is the log-importance-ratio (log of how much more likely response $k$ is under the current policy $\pi_\theta$ compared to the behavior policy $\pi_b$ that generated the samples), and $K$ is the group size (number of responses per prompt, typically 8 in experiments).

What it computes: a normalized probability distribution over the $K$ sampled responses that captures the current policy's relative preference — which responses is $\pi_\theta$ "leaning toward" compared to the sampling distribution. At the on-policy point where $\pi_\theta = \pi_b$, all $s_{\theta,k} = 0$ and $P_\theta$ reduces to the uniform distribution $1/K$, meaning the current policy has no differential preference among the samples it just generated. As training progresses and the policy shifts, $P_{\theta,k}$ grows for responses the policy increasingly favors and shrinks for those it disfavors.

Why this form: the softmax normalization is crucial because it enforces that $P_\theta$ lives on the probability simplex $\Delta^{K-1} = \{p \in \mathbb{R}^K : p_k \geq 0, \sum_k p_k = 1\}$. This means the target distribution $w^*$ (also on $\Delta^{K-1}$) and the policy's listwise distribution $P_\theta$ inhabit the exact same space — the projection between them is geometrically clean and can be performed in closed form. Without this reparameterization, the policy's output over $K$ responses is just $K$ independent probability values that don't sum to anything meaningful, making comparative optimization awkward. The importance ratio $s_{\theta,k}$ subtracts the behavior policy's log-probability because the samples were generated from $\pi_b$, not $\pi_\theta$; this off-policy correction is the standard importance sampling adjustment that ensures $P_\theta$ represents the policy's true relative preference despite the sampling distribution mismatch.

A subtle but critical point: $P_\theta$ is computed from the sequence-level log-probability $\log \pi_\theta(y_k|x) = \frac{1}{|y_k|} \sum_{i=1}^{|y_k|} \log \pi_\theta(y_{k,i}|x, y_{k,<i})$, the average token-level log-probability (length-normalized). This means the listwise distribution compares responses based on per-token likelihood, not total sequence likelihood, preventing length bias from dominating the relative preferences.

---

Target Construction: The Proximal RL Objective on the Simplex

The target distribution $w^*$ is derived by solving a constrained optimization problem that formalizes what existing PG methods are implicitly doing. The key insight is to restrict the standard KL-regularized RL objective to the finite response simplex (Eq. 7):

$\max_{w \in \Delta^{K-1}} \hat{J}(w) = \sum_{k=1}^{K} w_k R_k - \tau \, D_{\text{KL}}(w \| P_t)$

where $w$ is a candidate distribution over the $K$ responses being optimized, $R_k \in [0,1]$ is the verifiable reward for response $k$ (typically binary: 1 if correct, 0 otherwise), $P_t$ is the listwise distribution induced by the pre-update policy $\pi_t$ (evaluated at the start of the current iteration, before any gradient updates), $\tau > 0$ is a temperature controlling the trust-region strength, and $D_{\text{KL}}(w \| P_t) = \sum_k w_k \log(w_k / P_{t,k})$ is the KL divergence from the candidate $w$ to the anchor $P_t$.

What it computes: the optimal probability mass allocation over the $K$ responses that maximizes expected reward while penalizing deviation from the pre-update policy's distribution. The first term $\sum_k w_k R_k$ is the expected reward under $w$ — it encourages putting mass on high-reward responses. The second term $-\tau D_{\text{KL}}(w \| P_t)$ is a trust-region penalty — it discourages $w$ from straying far from $P_t$, with $\tau$ controlling the trade-off. When $\tau$ is large, the penalty dominates and $w$ stays close to $P_t$ (conservative update). When $\tau$ is small, the reward term dominates and $w$ aggressively concentrates on the highest-reward responses.

Why this form: this is the standard KL-regularized RL objective (Ziebart, 2010; Levine, 2018) but restricted to the $K$ sampled responses rather than the full sequence space. In the full space, solving $\max_\pi \mathbb{E}_\pi[R] - \tau D_{\text{KL}}(\pi \| \pi_t)$ yields $\pi^*(y) \propto \pi_t(y) \exp(R(y)/\tau)$ with an intractable partition function over all possible sequences. By restricting to the finite simplex of $K$ sampled responses, the partition function becomes a finite sum — the optimization is exact and computationally trivial. The pre-update policy $P_t$ serves as the anchor (trust-region center) because it represents "where the policy currently is" at the start of the iteration; regularizing toward it prevents the target from overfitting to the particular sample of $K$ responses.

The paper explicitly notes that as $K \to \infty$, the empirical simplex approximates the full policy space and Eq. 7 recovers the exact KL-regularized RL objective — but the key practical contribution is that for finite $K$, the optimization is tractable and yields a closed-form solution.

Theorem 1 (Listwise Gibbs target) provides this solution (Eq. 8):

$w^*_k = \text{softmax}(\phi)_k, \quad \text{with } \phi_k = \frac{R_k}{\tau} + s_{t,k}$

where $w^*_k$ is the target probability for response $k$, $\phi_k$ is the composite logit combining reward and policy prior, $R_k/\tau$ is the temperature-scaled reward, and $s_{t,k} = \log(\pi_t(y_k|x)/\pi_b(y_k|x))$ is the pre-update policy's log-importance-ratio for response $k$.

What it computes: a re-weighting of the pre-update policy's listwise distribution $P_t = \text{softmax}(s_t)$ toward higher-reward responses. Responses with high rewards get their logits boosted by $R_k/\tau$, responses with low rewards get no boost. The softmax then normalizes these boosted logits into a valid probability distribution. When $\tau \to 0$, $w^* \to \text{argmax}_k R_k$ (one-hot on the best response). When $\tau \to \infty$, $w^* \to P_t$ (no change from the pre-update policy). At the on-policy point where $\pi_t = \pi_b$, we have $s_{t,k} = 0$ and $P_t$ is uniform, so $w^* = \text{softmax}(R/\tau)$ — recovering exactly the implicit targets of existing PG methods (Proposition 1).

Why this form: the additive structure $\phi_k = R_k/\tau + s_{t,k}$ decouples the two forces acting on the target: the reward signal pushes toward correctness, while the policy prior $s_{t,k}$ anchors the target to the current policy's preferences (capturing things like "which of these correct solutions does the model currently prefer to generate"). This is fundamentally different from just taking $\text{softmax}(R/\tau)$ in the off-policy case — the $s_{t,k}$ term provides a local trust region that prevents the target from being dominated by a single high-reward response that the current policy assigns negligible probability to, which would create an impossibly steep projection gradient.

The temperature $\tau$ emerges here with a clear interpretation as a trust-region parameter — it is not an arbitrary scaling factor but directly controls the KL penalty strength in the proximal objective. The paper deliberately inherits $\tau$ from the paired PG baseline's advantage normalization (e.g., $\tau = \sigma_G$ for GRPO, $\tau = 1$ for Dr.GRPO, $\tau = \mu_G$ for MaxRL) rather than tuning it independently (Section 4.3). This design choice isolates the effect of the projection mechanism from the effect of temperature tuning — any performance difference between LPO and the PG baseline is attributable to exact projection vs. first-order approximation, not to a better temperature.

---

Projection via Divergence Minimization

Once the target $w^*$ is constructed, policy optimization reduces to projecting the current policy's listwise distribution $P_\theta$ onto $w^*$ by minimizing a chosen divergence. This is the step where LPO fundamentally diverges from existing PG methods: rather than using a first-order Taylor approximation of reverse KL (which is what PG does, per Proposition 1), LPO computes the exact gradient of the chosen divergence.

The framework is general — any differentiable divergence $D(w^*, P_\theta)$ on the simplex can be used — but the paper instantiates two representative choices:

Example 1: Forward KL — $D_{\text{KL}}(w^* \| P_\theta)$ (Eq. 10):

$\min_\theta \mathcal{L}_{\text{LPO}}^{\text{fwd}} = D_{\text{KL}}(w^* \| P_\theta) \quad \Rightarrow \quad \nabla_\theta \mathcal{L}_{\text{LPO}}^{\text{fwd}} = \sum_{k=1}^{K} \underbrace{(P_{\theta,k} - w^*_k)}_{c^{\text{fwd}}_k} \nabla_\theta \log \pi_\theta(y_k|x)$

where $w^*$ is the fixed target distribution from Theorem 1, $P_\theta$ is the current policy's listwise distribution, and $c^{\text{fwd}}_k = P_{\theta,k} - w^*_k$ is the per-response gradient coefficient.

What it computes: the gradient of the forward KL divergence. For each response $k$, the coefficient $c^{\text{fwd}}_k$ represents the "probability gap" between where the policy currently is ($P_{\theta,k}$) and where it should be ($w^*_k$). If the policy is under-allocating probability to response $k$ ($P_{\theta,k} < w^*_k$), the coefficient is negative, and gradient descent increases $\log \pi_\theta(y_k|x)$ (pushing mass toward that response). If the policy is over-allocating ($P_{\theta,k} > w^*_k$), the coefficient is positive, and gradient descent decreases $\log \pi_\theta(y_k|x)$ (pulling mass away). The magnitude of the coefficient scales with the mismatch.

Why this form: the forward KL, $D_{\text{KL}}(w^* \| P_\theta) = -\sum_k w^*_k \log P_{\theta,k} + \text{const}$, is the mode-covering divergence. It heavily penalizes $P_{\theta,k} \approx 0$ when $w^*_k > 0$ (places where the target assigns probability but the policy doesn't), while being relatively tolerant of $P_{\theta,k} > 0$ when $w^*_k \approx 0$ (the policy can "cover" low-target modes without severe penalty). This property is formalized in Corollary 2, which provides a log-barrier lower bound on $P_{\theta,k}$ when $w^*_k$ is non-negligible — the policy cannot collapse probability mass onto a single response because the forward KL penalty for ignoring any response with non-trivial target weight grows logarithmically.

The paper explicitly contrasts this with the reverse KL used implicitly by PG methods, which is mode-seeking — it penalizes $P_{\theta,k} > 0$ when $w^*_k \approx 0$ (policy mass on low-target responses) while being tolerant of $P_{\theta,k} \approx 0$ when $w^*_k > 0$ (can ignore modes of the target). This distinction has practical consequences: forward KL preserves response diversity (beneficial for Pass@k), while reverse KL focuses probability mass on the single best response (potentially better for Pass@1 but risking mode collapse).

Example 2: Reverse KL — $D_{\text{KL}}(P_\theta \| w^*)$ (Eq. 11):

$\min_\theta \mathcal{L}_{\text{LPO}}^{\text{rev}} = D_{\text{KL}}(P_\theta \| w^*) \quad \Rightarrow \quad \nabla_\theta \mathcal{L}_{\text{LPO}}^{\text{rev}} = \sum_{k=1}^{K} \underbrace{P_{\theta,k} (d_k - \bar{d})}_{c^{\text{rev}}_k} \nabla_\theta \log \pi_\theta(y_k|x)$

where $d_k = s_{\theta,k} - \phi_k$ is the logit gap (difference between the policy's current logit and the target logit for response $k$), $\bar{d} = \sum_{j=1}^{K} P_{\theta,j} d_j$ is the $P_\theta$-weighted mean of the logit gaps, and $c^{\text{rev}}_k = P_{\theta,k}(d_k - \bar{d})$ is the per-response gradient coefficient.

What it computes: the gradient of the reverse KL divergence. The coefficient has a baseline-subtracted structure: $d_k$ measures how far response $k$'s logit is from its target, and $\bar{d}$ is the average gap across all responses weighted by the current policy. Subtracting $\bar{d}$ ensures $\sum_k c^{\text{rev}}_k = 0$ (zero-sum property). To interpret: if response $k$ has a larger-than-average positive gap ($d_k > \bar{d}$), meaning the policy is assigning it more relative probability than the target warrants, the coefficient is positive and gradient descent decreases $\log \pi_\theta(y_k|x)$. If the gap is smaller than average ($d_k < \bar{d}$), the coefficient is negative and the policy probability increases.

Why this form: Appendix B.1 derives this coefficient via a logit-gap simplification. The raw reverse KL gradient coefficient (before baseline subtraction) is $P_{\theta,k}[\log(P_{\theta,k}/w^*_k) - D_{\text{KL}}(P_\theta \| w^*)]$. The key algebraic step is recognizing that $\log(P_{\theta,k}/w^*_k) = (s_{\theta,k} - \phi_k) - (\log Z_s - \log Z_\phi) = d_k - c_s$, where $c_s$ is a constant (the difference of log-partition functions) that is identical for all $k$. This constant cancels exactly when forming $d_k - \bar{d}$, yielding the clean baseline-subtracted form. The paper notes (Section 4.2, Example 2) that minimizing reverse KL is equivalent to maximizing the proximal objective $\hat{J}$ from Eq. 7 (Proposition 3 in Appendix B.4), and that the objective decomposes as $-\sum_k P_{\theta,k} \phi_k - H(P_\theta)$ — revealing an implicit entropy bonus $H(P_\theta)$ that naturally emerges from the reverse KL structure. This entropy bonus encourages the listwise distribution to remain spread out rather than collapsing to a point mass, providing a principled regularization against mode collapse.

Proposition 1 connection: at the exact on-policy point ($\pi_\theta = \pi_b$, so $s_{\theta,k} = 0$ for all $k$), the reverse KL gradient coefficient simplifies dramatically. The logit gap becomes $d_k = -\phi_k = -R_k/\tau$ (since $s_{\theta,k}=0$ and $s_{t,k}=0$ at on-policy), the mean gap is $\bar{d} = -\frac{1}{K}\sum_j R_j/\tau$, and the uniform $P_{\theta,k}=1/K$ yields $c^{\text{rev}}_k = -\frac{1}{K}(R_k/\tau - \frac{1}{K}\sum_j R_j/\tau) = -A_k/K$ where $A_k = (R_k - \mu)/\tau$ is the advantage. This is exactly the negative of the standard policy gradient coefficient $c^{\text{PG}}_k = A_k/K$, proving that at the on-policy point, LPO with reverse KL and standard GRPO-style PG produce identical gradients. The difference emerges off-policy, where LPO maintains the exact reverse KL gradient while PG uses a first-order approximation that degrades with off-policy drift.

Off-policy degradation analysis (Appendix B.2): the paper quantifies how the PG approximation breaks down. Let $\bar{\delta} = \max_k |r_k - 1|$ be the maximum importance sampling ratio deviation from 1. The per-coefficient discrepancy between PG and exact reverse KL scales as $|\Delta_k| = O(\bar{\delta}(1 + \|A\|_\infty)/K)$. In practical RLVR training with multiple inner epochs per batch, $\bar{\delta}$ grows with each gradient step as the policy drifts from $\pi_b$, causing the PG gradient to increasingly diverge from the intended reverse KL projection. The exact listwise projection in LPO sidesteps this entirely — it recomputes $P_\theta$ (and hence the gradient coefficients) from the current parameters $\theta$ at each gradient step, maintaining the exact divergence gradient regardless of off-policy drift.

---

Gradient Coefficient Properties

The exact projection onto the simplex yields gradient coefficients with three structural properties that are impossible to achieve simultaneously with a PG formulation. Corollary 1 establishes these for the forward KL case:

Corollary 1 (Gradient coefficient properties): the forward KL coefficients $c^{\text{fwd}}_k = P_{\theta,k} - w^*_k$ satisfy:

(a) Bounded: $|c^{\text{fwd}}_k| \leq 1$ for all $k$, and consequently $\sum_k |c^{\text{fwd}}_k| \leq 2$, which bounds the total parameter gradient norm by $2 G_{\text{max}}$ where $G_{\text{max}} = \max_k \|\nabla_\theta \log \pi_\theta(y_k|x)\|$.

Why this matters: gradient norm explosions are a known failure mode in RLVR training, particularly when advantage normalization produces extreme values (e.g., $\sigma_G \approx 0$ in GRPO when all responses are correct or all are wrong, leading to advantages that blow up). LPO's coefficients are bounded by construction because both $P_{\theta,k}$ and $w^*_k$ are probabilities in $[0,1]$, so their difference cannot exceed 1 in magnitude. This provides an intrinsic, reward-scale-invariant gradient norm bound without any clipping or normalization heuristics.

(b) Zero-sum: $\sum_{k=1}^{K} c^{\text{fwd}}_k = 0$, which acts as a built-in control variate for variance reduction (Sutton, 1988).

Why this matters: in standard PG, advantage centering ($\sum_k A_k = 0$) must be manually enforced via the advantage normalization scheme — and even then, it only holds for the advantages, not necessarily for the importance-sampled coefficients $r_k A_k / K$ used in the actual gradient. LPO's zero-sum property is structural: because both $P_\theta$ and $w^*$ sum to 1, their difference automatically sums to zero. This means that for every response whose probability is increased, there is a corresponding response whose probability is decreased, maintaining the total probability mass at 1. The variance reduction comes from the fact that the gradient contributions of over-allocated and under-allocated responses partially cancel, reducing the noise in the aggregate parameter update.

(c) Self-correcting: $c^{\text{fwd}}_k \to 0$ as $P_\theta \to w^*$, meaning the gradient naturally vanishes when the target is matched.

Why this matters: the optimization has a clear fixed point — when $P_\theta = w^*$, all coefficients are zero and no further parameter updates occur. This is in contrast to pointwise projection methods (Appendix C.4), where the coefficients $c^{\text{point}}_k = -w^*_k$ are constant with respect to $\pi_\theta$, meaning the gradient never vanishes even if the policy perfectly matches the target (because the pointwise objective tries to push probability toward $w^*_k$ independently per response, without a simplex constraint to enforce convergence). The self-correcting property provides a natural termination criterion: the optimization automatically slows down as it approaches the target, preventing overshooting.

The reverse KL coefficients $c^{\text{rev}}_k = P_{\theta,k}(d_k - \bar{d})$ share the zero-sum and self-correcting properties (verified in the main text, Example 2) but are not uniformly bounded by a constant independent of the logit values — they depend on the magnitude of $d_k$, which can grow if the target logits are extreme (e.g., when $\tau$ is very small). However, the $P_{\theta,k}$ weighting naturally attenuates coefficients for responses where $d_k$ is large but $P_{\theta,k}$ is small (the policy ignores those responses anyway), providing a soft form of boundedness.

Corollary 2 (Mode-coverage): provides a formal log-barrier guarantee for forward KL. If the target assigns at least probability $\alpha$ to response $k$ ($w^*_k \geq \alpha$) and the forward KL divergence is bounded by $D$ ($D_{\text{KL}}(w^* \| P_\theta) \leq D$), then:

$P_{\theta,k} > \alpha \exp(-D/\alpha - 1)$

What this means: the policy cannot assign arbitrarily small probability to any response that the target considers non-negligible. The lower bound decays exponentially in $D/\alpha$ — meaning that even if the overall KL divergence $D$ is moderate, the policy is forced to maintain some mass on all responses with $w^*_k > 0$. This is the formal statement of mode-covering: forward KL prevents the policy from collapsing to a single mode, because ignoring any response with target weight $\alpha$ would incur a KL penalty of at least $-\log(P_{\theta,k})$, which grows without bound as $P_{\theta,k} \to 0$.

Why this matters for Pass@k evaluation: in reasoning tasks with multiple valid solution paths, the target $w^*$ typically assigns non-trivial weight to several correct responses (all with $R_k = 1$ share the same reward, though $s_{t,k}$ differences create variation in target weights). Forward KL ensures the policy maintains some probability on all these correct responses, preserving the diversity needed for high Pass@k — if you sample $k$ responses, you're more likely to hit multiple distinct correct solutions. Reverse KL, by contrast, can concentrate entirely on the single best-scoring response under the target, potentially achieving higher Pass@1 at the cost of Pass@k diversity.

---

Monotonic Improvement Guarantee

Theorem 2 provides the paper's central theoretical guarantee: the target-projection framework yields monotonic improvement of the listwise reward $\hat{R}(P) = \sum_k P_k R_k$ at each iteration, up to projection error:

$\hat{R}(P_{t+1}) \geq \hat{R}(P_t) + \underbrace{\tau \left[D_{\text{KL}}(w^* \| P_t) + D_{\text{KL}}(P_t \| w^*)\right]}_{\text{target gain} \geq 0} - \underbrace{2 R_{\text{max}} \epsilon_{\text{proj}}}_{\text{projection error}}$

where $P_t$ is the pre-update listwise distribution, $P_{t+1}$ is the post-update distribution, $w^*$ is the target from Theorem 1, $\hat{R}(P) = \sum_k P_k R_k$ is the expected reward under distribution $P$, $R_{\text{max}}$ is the maximum absolute reward (1 for binary rewards), and $\epsilon_{\text{proj}}$ is the total variation distance between $P_{t+1}$ and $w^*$ achieved by the projection step.

What it computes: a lower bound on the improvement in expected reward from one iteration to the next. The bound has two terms:

1. Target gain — $\tau [D_{\text{KL}}(w^* \| P_t) + D_{\text{KL}}(P_t \| w^*)]$, which is the Jeffreys divergence (a symmetrized KL) between the pre-update distribution and the target, scaled by $\tau$. This term is always non-negative and is strictly positive whenever $P_t \neq w^*$. It quantifies how much room for improvement exists: if the current policy is far from the target, the potential gain is large; if it's already close, the gain is small.

2. Projection error — $2 R_{\text{max}} \epsilon_{\text{proj}}$, which penalizes imperfect projection. If the projection step perfectly achieves $P_{t+1} = w^*$, then $\epsilon_{\text{proj}} = 0$ and the bound reduces to $\hat{R}(P_{t+1}) \geq \hat{R}(P_t) + \tau \cdot \text{Jeffreys}(P_t, w^*) > \hat{R}(P_t)$, guaranteeing strict improvement.

The proof (Appendix B.5) has two parts: first, showing that the reward gap between the target and the pre-update distribution equals $\tau$ times the Jeffreys divergence (which follows from Proposition 3, establishing that $\hat{J}(w) = -\tau D_{\text{KL}}(w \| w^*) + \tau \log \hat{Z}$, so $\hat{R}(w^*) - \hat{R}(P_t) = \tau [D_{\text{KL}}(w^* \| P_t) + D_{\text{KL}}(P_t \| w^*)]$); second, bounding the reward error from imperfect projection using Hölder's inequality, which gives $|\hat{R}(P_{t+1}) - \hat{R}(w^*)| \leq R_{\text{max}} \|P_{t+1} - w^*\|_1 = 2 R_{\text{max}} \epsilon_{\text{proj}}$.

Why this matters: this is the formal justification for why explicit target-projection should outperform implicit PG approximation. The PG update is a first-order step toward $w^*$ — it guarantees improvement only in the infinitesimal limit, and with finite step sizes can overshoot, oscillate, or degrade the objective. LPO's projection step directly minimizes the divergence, bringing $P_{t+1}$ as close to $w^*$ as optimization permits, which (for convex divergences like KL) guarantees a reduction in the divergence and hence an increase in the surrogate objective $\hat{J}$. The bound in Theorem 2 quantifies the worst-case degradation from imperfect projection, showing that as long as the projection error is small relative to the target gain, improvement is guaranteed.

Proposition 2 (Idealized full-space convergence): provides the limiting reference. If one could perform the exact proximal update over the entire response space — $\pi_{t+1}(y) \propto \pi_t(y) \exp(R(y)/\tau)$ — the iteration would converge to the reward-maximizing policy $\pi_t(y) \propto \pi_0(y) \exp(t R(y)/\tau)$, with $\mathbb{E}_{\pi_t}[R] \to \max_y R(y)$ as $t \to \infty$. This is intractable for autoregressive LLMs because the partition function requires summing over all possible sequences. LPO approximates this ideal by restricting the update to the finite simplex of $K$ sampled responses, where the partition function is a simple finite sum, yielding a principled and fully tractable approximation.

---

Practical Implementation: Algorithm and Temperature Design

The LPO procedure (Algorithm 1) is deliberately designed to be a drop-in replacement for existing group-based PG methods with identical computational cost:

Algorithm 1: Listwise Policy Optimization (LPO)

1. Set behavior policy and pre-update policy: $\pi_b \leftarrow \pi_\theta$, $\pi_t \leftarrow \pi_\theta$. Both are snapshots of the current policy parameters before this iteration's updates. $\pi_b$ is frozen as the sampling distribution for importance ratio computation; $\pi_t$ is frozen for target construction.

2. Sample responses: for each prompt $x$ in batch $\mathcal{B}$, generate $K$ responses $\{y_k\}_{k=1}^K \sim \pi_b(\cdot|x)$ and compute rewards $\{R_k\}_{k=1}^K$ via the verifier.

3. Compute target: for each prompt $x$, compute $w^*(x) = \text{softmax}(\phi(x))$ where $\phi_k = R_k/\tau + \log(\pi_t(y_k|x)/\pi_b(y_k|x))$. This is a simple softmax over $K$ elements, requiring only the pre-computed log-probabilities (which are already available from the rollout step) and the rewards.

4. Inner optimization loop: for $e = 1$ to $E$ epochs:

   • Compute the listwise distribution $P_\theta = \text{softmax}(s_\theta)$ from the current $\theta$
   • Compute per-response coefficients $c_k$ using the chosen divergence (Eq. 10 for forward KL, Eq. 11 for reverse KL)
   • Gradient update: $\theta \leftarrow \theta - \eta \frac{1}{B} \sum_{x \in \mathcal{B}} \sum_{k=1}^K c_k(x) \nabla_\theta \log \pi_\theta(y_k|x)$

Key implementation details (Section 4.3 and Appendix D):

• Temperature inheritance: the paper explicitly avoids introducing $\tau$ as a new hyperparameter. Instead, $\tau$ is set to match the paired PG baseline's implicit temperature: $\tau = \sigma_G$ for GRPO/DAPO, $\tau = 1$ for Dr.GRPO/RLOO, $\tau = \mu_G$ for MaxRL. This ensures any performance difference is attributable to the projection mechanism, not to better temperature tuning. The interpretation of $\tau$ shifts from "ad-hoc advantage scaling factor" to "trust-region strength in the proximal RL objective," but the numerical value is identical.

• Token-level clipping: the importance ratios used in $s_{\theta,k}$ are clipped at the token level following standard PPO practice (Schulman et al., 2017b). The log-density ratio $\delta_{k,i} = \log \pi_\theta(y_{k,i}|x, y_{k,<i}) - \log \pi_b(y_{k,i}|x, y_{k,<i})$ is clipped to $[\log(1 - \epsilon), \log(1 + \epsilon)]$ before being summed to form $s_{\theta,k}$. This is a practical stabilization measure inherited from the PG baselines, not a structural requirement of LPO — the theoretical framework is clean without clipping, but large-scale training benefits from it.

• Batch and optimization configuration: across all experiments, $K = 8$ responses per prompt, batch size varies by task (256 for MATH and PRIME-Code, 128 for Countdown and Geometry), mini-batch size is half the batch size (so 2 gradient updates per iteration), learning rate $1 \times 10^{-6}$ with Adam optimizer ($\beta = (0.9, 0.999)$, weight decay 0.1), and clipping parameter $\epsilon = 0.2$.

• Computational cost: the additional computation over standard PG is minimal — the softmax over $K$ elements to compute $P_\theta$ and $w^*$ is $O(K)$ per prompt, negligible compared to the forward/backward passes through the LLM. The paper explicitly states (Section 4.3): "The training pipeline is identical to standard group-based RL algorithms, with no additional computational cost."

Design choice: why reuse the PG temperature instead of tuning it? The paper's paired evaluation protocol (Section 5.1) is central to its empirical argument: each LPO variant is compared against the PG baseline that uses the identical $\tau$ value. If LPO with $\tau = \sigma_G$ outperforms GRPO (which also uses $\tau = \sigma_G$ implicitly), the gain is necessarily due to the exact projection, not to a better temperature. This controlled comparison is methodologically rigorous but does leave open the question of whether independently tuning $\tau$ for LPO could yield further gains — the paper explicitly leaves this to future work (Appendix C.1, point 4).

Connection to EM and DPO: Appendix C.4 notes that the target-projection structure mirrors the Expectation-Maximization algorithm: target construction is an E-step (forming a target distribution), divergence minimization is an M-step (fitting the model to that target). Appendix C.5 shows that when $K = 2$, LPO reduces to a pairwise binary cross-entropy objective with soft targets $\sigma(\pm 1/\tau)$, connecting directly to Direct Preference Optimization (DPO) — but with the crucial difference that LPO operates in an online RL setting with absolute rewards rather than offline pairwise preferences, and uses a trust region around $\pi_t$ rather than a static reference policy.

---

Summary of Design Choices and Their Justifications

• Listwise reparameterization (softmax over $K$ responses) over raw token probabilities: enforces that $P_\theta$ and $w^*$ inhabit the same simplex, enabling exact projection with zero-sum, bounded, self-correcting gradients; raw probabilities don't form a distribution over responses, making comparative optimization ill-posed.

• Proximal objective with $P_t$ as anchor over arbitrary reference distribution: uses the pre-update policy as the trust-region center, ensuring the target stays local and doesn't overfit to the particular sample of $K$ responses; a static reference (e.g., the initial pretrained model) would not adapt to the policy's evolving capabilities.

• Exact divergence minimization over first-order PG approximation: maintains the true gradient regardless of off-policy drift, unlike PG where the equivalence to reverse KL degrades as $O(\bar{\delta}(1 + \|A\|_\infty)/K)$; this is possible specifically because the response simplex is finite, making the softmax and its gradient exact.

• Temperature inheritance from baseline over independent tuning: isolates the projection mechanism as the sole experimental variable, enabling clean attribution of performance differences; sacrifices potential gains from temperature optimization in exchange for methodological clarity.

• Forward KL as a novel projection choice over only reverse KL: provides mode-covering behavior that preserves response diversity for Pass@k evaluation; forward KL is structurally inaccessible under the PG paradigm because no advantage vector $A$ exists whose softmax target combined with a PG update would yield a forward KL gradient.

• Reverse KL for connecting to existing methods (Proposition 1): shows that LPO with reverse KL exactly recovers standard PG at the on-policy point, providing theoretical continuity with the existing literature and validating the framework's claim to be a generalization rather than a replacement.

4. Key Insights and Innovations

Innovation 1: A Unified Geometric Framework That Recovers Existing Group-Based Methods as Approximate Instances of the Same Target-Projection Structure

The paper's foundational contribution is not proposing a new method per se, but rather providing a diagnostic lens through which the entire landscape of group-based RLVR algorithms becomes coherent. Prior to this work, the dominant methods — GRPO, Dr.GRPO, MaxRL, RLOO, DAPO, REINFORCE++ — appeared as a collection of independently motivated advantage normalization schemes, each with its own empirical justification and training stabilization rationale. The relationship between them was unclear: was Dr.GRPO's removal of $\sigma_G$ normalization fundamentally different from GRPO's approach, or merely a reparameterization? Was MaxRL's use of $\mu_G$ as the denominator introducing a qualitatively new optimization dynamic, or operating in the same space?

The paper answers these questions with a single, sharp observation: all these methods are attempting the same thing — projecting the policy toward a reward-weighted softmax target on a finite response simplex — but doing so implicitly and approximately via a first-order policy gradient. The differences reduce entirely to the temperature $\tau$ in the softmax target $w^* = \text{softmax}(R/\tau)$, with each method implicitly setting $\tau$ through its choice of advantage scaling denominator (Table 1). The centering term (whether group mean $\mu_G$, leave-one-out baseline, or greedy-decode baseline) cancels identically under the softmax due to shift-invariance, so it contributes nothing to the target shape — it only matters for the PG approximation, not for what the algorithm is actually trying to achieve.

What makes this insight fundamental rather than incremental is that it converts a fragmented algorithmic design space into a single-parameter family. Before this work, developing a new group-based RLVR method meant proposing a new advantage formula and empirically validating it against baselines — an ad-hoc process with no principled guidance on what design dimensions matter. After this work, the design space collapses to two independent choices: (1) what temperature $\tau$ should the target use (which controls sharpness and can be adaptive, as in MaxRL's $\tau = \mu_G$ success-rate curriculum), and (2) should the projection toward that target use an approximate first-order PG step or an exact divergence minimization? The paper demonstrates that question (1) is task-dependent (no single $\tau$ dominates across all benchmarks in Figure 3), while question (2) yields gains orthogonal to $\tau$ choice — LPO improves over PG baselines under all three temperature designs (GRPO, Dr.GRPO, MaxRL) across nearly all tasks, meaning the exact projection provides a structural advantage independent of target sharpness.

The connection to prior work is precise: classical RL-as-inference algorithms (MPO, AWR, V-MPO) also construct reward-weighted Gibbs targets and project toward them, but they operate in continuous action spaces where the partition function is intractable, forcing them to use pointwise projections ($-\sum_k w^*_k \log \pi_\theta(y_k)$) that lack the zero-sum, self-correcting gradient structure that emerges naturally from simplex normalization. The paper's insight is that LLM group-based RLVR is uniquely situated to do better: the finite set of $K$ sampled responses forms a simplex where exact normalization is trivial, making exact listwise projection computationally cheap and geometrically clean. This is not a general observation about RL — it's specific to the group-based sampling paradigm in language model training, and no prior work had recognized this structural opportunity.

The empirical evidence for this unification is implicit rather than explicit (there's no single ablation proving "all these methods are the same target family"), but it's theoretically airtight: Proposition 1 proves exact equivalence at the on-policy point for any zero-mean advantage vector, and the off-policy error analysis (Appendix B.2) quantifies how the PG approximation degrades with $O(\bar{\delta}(1 + \|A\|_\infty)/K)$ — a structural limitation shared by all existing methods regardless of their normalization scheme. The practical consequence is that researchers can now reason about group-based RLVR methods in terms of target temperature and projection quality, rather than advantage formulas, which is a conceptual reframing that should influence how future algorithms are designed and evaluated.

---

Innovation 2: Decoupling Target Construction from Projection Opens a New Design Axis (Divergence Choice) That Is Structurally Inaccessible Under the Policy Gradient Paradigm

The second major insight is that policy gradient methods are not just an approximation of reverse KL projection — they are a prison that traps algorithm designers in a single divergence. Because PG methods construct their update by multiplying advantages with log-probability gradients, there is no way to express a forward KL projection (or any non-reverse-KL divergence) as a PG update. The advantage vector $A$ determines a softmax target $w^* = \text{softmax}(A)$, and the PG gradient $\sum_k (A_k/K) \nabla_\theta \log \pi_\theta(y_k|x)$ is always — and only — a first-order approximation of $-\nabla_\theta D_{\text{KL}}(P_\theta \| w^*)$. You cannot choose a different divergence; you cannot even express one within the PG formalism.

LPO breaks this coupling by separating the two steps: target construction (which defines what distribution to aim for) and projection (which defines how to move toward it). The target $w^* = \text{softmax}(R/\tau + s_t)$ is computed once per iteration from rewards and the pre-update policy, and then the projection step minimizes any differentiable divergence $D(w^*, P_\theta)$ on the simplex. The paper implements forward and reverse KL as two instantiations, but the framework is general (Appendix C.6 shows that any differentiable divergence on $\Delta^{K-1}$ yields a gradient with the zero-sum property automatically, as a consequence of the softmax Jacobian structure).

Why is access to forward KL a fundamental advance? Because forward KL and reverse KL have qualitatively different geometry with concrete practical consequences for LLM training:

• Reverse KL ($D_{\text{KL}}(P_\theta \| w^*)$) is mode-seeking: it penalizes $P_{\theta,k} > 0$ when $w^*_k \approx 0$ (putting mass where the target says there's no reward), while being relatively tolerant of $P_{\theta,k} \approx 0$ when $w^*_k > 0$ (ignoring modes of the target). This concentrates probability on the single highest-reward response, which can be good for Pass@1 but risks mode collapse and loss of reasoning diversity.

• Forward KL ($D_{\text{KL}}(w^* \| P_\theta)$) is mode-covering: it heavily penalizes $P_{\theta,k} \approx 0$ when $w^*_k > 0$ (ignoring responses the target considers valuable), while being tolerant of $P_{\theta,k} > 0$ when $w^*_k \approx 0$. This forces the policy to maintain probability mass on all non-negligible target modes, preserving the diversity needed for high Pass@k — if multiple distinct correct solutions exist in the sampled group, forward KL ensures the policy keeps some probability on all of them rather than collapsing to one.

The empirical results validate this distinction decisively: in the group-size experiment (Figure 7), LPOfwd scales exceptionally well on Pass@64 (its mode-covering property structurally preserves reasoning diversity needed for high-coverage evaluation), while LPOrev achieves stronger Pass@1 performance (its mode-seeking property concentrates mass on the best solution). Across the full benchmark suite, LPOfwd outperforms LPOrev in 13/15 Pass@k scenarios (Figure 4), consistent with the diversity advantage. No existing PG method could achieve forward KL behavior — this is a genuinely new capability enabled by the decoupled framework.

The significance here is not just that LPO performs well, but that it opens a design dimension that the dominant paradigm had foreclosed. The paper explicitly notes (Appendix C.6) that alternative divergences like Jensen-Shannon or general f-divergences could be explored, with potentially different optimization geometries suited to different tasks or training stages (e.g., forward KL for early exploration, reverse KL for late exploitation). This transforms RLVR algorithm design from "find a better advantage formula" (one-dimensional search) to "choose a target temperature and a projection divergence" (two-dimensional, principled design space) — a qualitative expansion of what's possible.

The connection to prior work is instructive: classical RL algorithms like MPO and AWR could in principle use different divergences (MPO uses a KL constraint in both directions), but their pointwise projection implementations lose the simplex geometry that makes divergence choice meaningful — a pointwise forward KL projection is just weighted supervised learning ($-\sum_k w^*_k \log \pi_\theta(y_k)$), which lacks the comparative, zero-sum structure that makes LPO's forward KL distinctive. The listwise normalization is what makes divergence choice matter geometrically, and no prior work in LLM training had recognized this.

---

Innovation 3: Exact Projection on the Simplex Yields Gradient Coefficient Properties (Bounded, Zero-Sum, Self-Correcting) That Provide Structural Optimization Stability Without Auxiliary Heuristics

The third insight is less about what LPO can express and more about what it eliminates: the need for many stabilization heuristics that have accumulated in group-based PG methods because their gradient coefficients lack structural guarantees. This is a negative contribution in the positive sense — LPO shows that certain desirable optimization properties can be achieved by construction rather than by patching.

Consider the current state of the art. GRPO clips importance ratios to $[1-\epsilon, 1+\epsilon]$ (Eq. 2) to prevent extreme updates. DAPO adds asymmetric clipping, dynamic sampling to filter uninformative groups (where all responses are correct or all wrong), token-level loss normalization, and overlong reward shaping (Yu et al., 2025). Dr.GRPO introduces token-level loss normalization to address length bias. REINFORCE++ uses batch-level statistics for normalization. Each of these engineering interventions addresses a specific failure mode — gradient explosions from extreme advantages, variance from uncentered advantages, entropy collapse, length exploitation — but they do so reactively, through heuristics added on top of the base PG algorithm.

LPO's explicit projection provides three structural properties (Corollary 1) that address several of these failure modes by design:

Bounded coefficients ($|c^{\text{fwd}}_k| \leq 1$, $\sum_k |c^{\text{fwd}}_k| \leq 2$): because both $P_{\theta,k}$ and $w^*_k$ are probabilities bounded in $[0,1]$, their difference cannot exceed 1. This provides an intrinsic gradient norm bound that is independent of reward scale, group statistics, or temperature. In standard PG with GRPO, when $\sigma_G \approx 0$ (all responses have the same reward), the advantage $A_k = (R_k - \mu_G)/\sigma_G$ blows up, requiring clipping or dynamic sampling to prevent catastrophic updates. LPO's coefficients remain bounded regardless of reward distribution — the worst case is $|c_k| = 1$ when $P_{\theta,k} = 1$ and $w^*_k = 0$ or vice versa. The empirical gradient norm curves (Figure 5, middle row) confirm that LPO variants exhibit lower and more stable gradient norms than PG baselines across tasks.

Zero-sum coefficients ($\sum_k c_k = 0$): this is a built-in control variate that arises from both $P_\theta$ and $w^*$ summing to 1. In standard PG, advantage centering ($\sum_k A_k = 0$) must be manually enforced through the normalization scheme, and even then it only applies to the advantages, not necessarily to the importance-sampled coefficients $r_k A_k / K$ used in the actual gradient when off-policy. LPO's zero-sum property is structural — for every response whose probability is increased, another is decreased, maintaining the total probability mass. This provides a form of variance reduction (Sutton, 1988) without any explicit centering step, and it holds regardless of off-policy drift because $P_\theta$ is recomputed from current parameters at each gradient step.

Self-correcting coefficients ($c_k \to 0$ as $P_\theta \to w^*$): the gradient naturally vanishes when the target is matched, providing a clear fixed point and preventing overshooting. Pointwise projection methods ($-\sum_k w^*_k \log \pi_\theta(y_k)$) lack this property — their coefficients $c^{\text{point}}_k = -w^*_k$ are constant with respect to $\pi_\theta$, meaning the gradient never stops even if the target is perfectly matched, because the pointwise objective tries to push all probability mass toward $w^*_k$ independently per response without the simplex normalization that would enforce convergence. The self-correcting property means LPO automatically slows down near the optimum, providing a natural form of learning rate adaptation.

The significance of these properties is not that they're mathematically elegant (though they are) — it's that they reduce the degrees of freedom in the optimization that can go wrong, making training more robust without additional hyperparameter tuning. The paper's experiments intentionally use a minimal shared pipeline (Appendix C.1, point 3) — no dynamic sampling, no asymmetric clipping, no overlong reward shaping — yet LPO maintains stable training trajectories with lower gradient norms (Figure 5, middle) and higher response entropy (Figure 5, top) than PG baselines that rely on these heuristics. This is not a claim that LPO makes such heuristics obsolete, but rather that it provides a more principled foundation on which they could be built if needed.

The contrast with the pointwise projection ablation (Figure 6) is telling: removing the listwise normalization (and hence the zero-sum property) causes severe performance degradation because the pointwise coefficients lack the competitive balancing mechanism that couples responses. This validates that the structural properties are not incidental — they are essential for stable optimization, and the listwise simplex formulation is what delivers them.

---

Innovation 4: The Response Simplex Perspective Reframes RLVR as Listwise Optimization with Connections to Learning-to-Rank, DPO, and EM, Enabling a Richer Theoretical Analysis

The fourth insight is primarily conceptual and connective: by reparameterizing the policy's behavior over a group of responses as a listwise distribution on a simplex, the paper reframes RLVR in a way that connects it to several distinct intellectual traditions, each of which brings analytical tools and theoretical guarantees that were previously inaccessible.

Connection to learning-to-rank: the listwise formulation $P_\theta = \text{softmax}(s_\theta)$ is precisely the Plackett-Luce ranking model (Plackett, 1975; Luce, 1959), which has been extensively studied in the learning-to-rank literature (Cao et al., 2007). This connection means that the policy's relative preference over responses can be analyzed using the well-developed theory of ranking losses, permutation probabilities, and listwise consistency. While the paper doesn't deeply exploit this connection, it opens the door for future work to import techniques from learning-to-rank — such as listwise loss functions beyond softmax, or sampling-based approximations to the full permutation space — into RLVR.

Connection to DPO and preference optimization: when $K = 2$, LPO's forward KL objective reduces to a binary cross-entropy objective with soft targets determined by the reward gap (Appendix C.5). The structure is $\mathcal{L} = -\sigma(1/\tau) \log \sigma(s_w - s_l) - \sigma(-1/\tau) \log \sigma(s_l - s_w)$, where $\sigma$ is the sigmoid function and $s_k$ are log-importance-ratios. This is mathematically analogous to DPO's $\mathcal{L} = -\log \sigma(\beta(s_w - s_l))$ (Rafailov et al., 2024), but with two crucial differences: (1) LPO uses soft targets $\sigma(\pm 1/\tau)$ rather than hard binary labels (0 or 1), with $\tau$ controlling label smoothness, and (2) LPO operates in an online RL setting with absolute rewards rather than offline pairwise preferences. This connection places DPO-style pairwise optimization as the $K = 2$ special case of a broader family of listwise target-projection algorithms that extends to arbitrary $K$ and, in the limit $K \to \infty$, recovers the exact KL-regularized RL objective (Theorem 1 and the discussion following it).

Connection to Expectation-Maximization: the target-projection iteration mirrors the EM algorithm structure (Dayan & Hinton, 1997; Neal & Hinton, 1998): constructing the Gibbs target $w^*$ is an E-step that forms a target distribution based on the current model and observed rewards, and minimizing the divergence toward $w^*$ is an M-step that fits the model parameters to that target. This connection brings with it the well-understood monotonic improvement properties of EM algorithms — Theorem 2 is essentially an EM-style lower bound argument — and suggests that techniques like incremental EM or variational EM could be applied to improve the efficiency of the projection step.

Theoretical payoff: monotonic improvement and mode-coverage guarantees. This reframing is not merely taxonomical — it enables Theorem 2 (monotonic improvement bound) and Corollary 2 (mode-coverage bound) to be proven cleanly within the listwise framework using tools from information geometry. Theorem 2's proof (Appendix B.5) relies on the fact that the proximal objective $\hat{J}(w)$ is equivalent to $-\tau D_{\text{KL}}(w \| w^*) + \text{const}$ (Proposition 3), which itself follows from the closed-form relationship between the Gibbs target and the KL-regularized objective — a relationship that is well-known in the RL-as-inference literature but was previously unexploited in the context of group-based PG methods. Corollary 2's log-barrier bound uses the data processing inequality to relate the full KL divergence on the simplex to a binary KL for a single response, providing a formal guarantee that the policy cannot collapse probability mass on any response with non-trivial target weight — a result that would be difficult to state, let alone prove, in the standard PG formalism where the policy's output over responses is not normalized.

Why this matters beyond the current paper: the listwise simplex perspective provides a shared mathematical language for analyzing RLVR, preference optimization, and RL-as-inference within a single coherent framework. This unification suggests that algorithmic innovations from any of these traditions — e.g., Plackett-Luce ranking surrogates from learning-to-rank, identity-preference optimization from DPO variants, or natural gradient methods from information geometry (Amari, 1998; Kakade, 2001) — could be imported into RLVR through the simplex interface. The paper doesn't pursue these directions, but by establishing the common structure, it creates the intellectual scaffolding for a more principled and interconnected research program. This is a conceptual reframing whose full impact will depend on how the field builds on it, but the unification itself is a genuine contribution — the fragmentation it resolves (between RLVR practice, RL-as-inference theory, and preference optimization) is real and was previously unaddressed.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates across four reasoning domains: (1) Logical Reasoning — Countdown Game, trained on a 2000-problem subset of Countdown-34 (Pan et al., 2025) and evaluated on 512 holdout instances each from Countdown-34 and the harder Countdown-4; (2) Mathematical Reasoning — trained on the MATH dataset (Hendrycks et al., 2021; 7.5k problems) and evaluated on AIME24, AIME25, AMC23, MATH500, Minerva Math, and OlympiadBench; (3) Programming — trained on the code split of the PRIME dataset (Cui et al., 2025; 25.3k problems) and evaluated on 1k held-out validation problems; (4) Multi-Modal Geometry — trained on the 2.1k-problem training split of Geometry3k (Lu et al., 2021; Hiyouga, 2025) and evaluated on the 601-problem test split.

• Base model(s). The paper spans eight models across four families to test generality: Qwen3-1.7B-Base, Qwen3-4B-Base, Qwen3-8B-Base, Qwen3-14B-Base, Qwen2.5-VL-3B-Instruct, DeepSeek-R1-Distill-Qwen-1.5B, Llama-3.1-8B-Instruct, and Mistral-7B-Instruct-v0.1. The primary experiments use Qwen3-4B-Base (Countdown), Qwen3-1.7B-Base and Qwen3-8B-Base (MATH), Qwen3-1.7B-Base (PRIME Code), and Qwen2.5-VL-3B-Instruct (Geometry). The model diversity tests (Section 5.4.3) add DeepSeek, Llama, and Mistral backbones to validate that gains are not architecture-specific.

• Metrics. Two primary metrics: expected Pass@1 (average accuracy over $k$ independent rollouts per problem, denoted Avg@k in the paper) and Pass@k (fraction of problems where at least one of $k$ rollouts is correct). The specific $k$ varies by benchmark: for Countdown, $k = 64$; for MATH, $k = 32$ on competition suites (AIME24, AIME25, AMC23), $k = 4$ on Minerva Math, and $k = 1$ on MATH500 and OlympiadBench; for PRIME Code, $k = 8$; for Geometry3k, $k = 16$. Training curves report Pass@1 accuracy averaged across the evaluation benchmarks in each domain.

• Baselines. Three representative group-based policy gradient methods with distinct implicit temperature designs: GRPO (Shao et al., 2024) with $\tau = \sigma_G$; Dr.GRPO (Liu et al., 2025b) with $\tau = 1$; and MaxRL (Tajwar et al., 2026) with $\tau = \mu_G$. All baselines are implemented within the same verl framework (Sheng et al., 2024) as LPO to ensure fair comparison.

• Generation budget / compute accounting. All methods use the same group size $K = 8$ responses per prompt during training, with identical generation hyperparameters (temperature 1.0, top-p 1.0, top-k -1.0, no KL penalty). Batch sizes vary by task (256 for MATH and PRIME-Code, 128 for Countdown and Geometry) with mini-batch size half the batch size, producing two gradient updates per iteration. The paper explicitly states that LPO introduces "no additional computational cost" beyond standard group-based RL pipelines (Section 4.3).

• Cross-validation / statistical protocol. There is no cross-validation or statistical significance testing reported. The paper uses a paired evaluation protocol: each LPO variant inherits the exact temperature $\tau$ from its paired PG baseline (e.g., LPOfwd with $\tau = \sigma_G$ is compared against GRPO; LPOfwd with $\tau = 1$ against Dr.GRPO; LPOfwd with $\tau = \mu_G$ against MaxRL). This isolates the projection mechanism as the sole controlled variable — any performance difference is attributable to exact listwise projection vs. first-order PG approximation, not to better temperature tuning. Training curves are plotted over optimization steps with no confidence intervals or error bars, which is standard for RL training curves in this literature but limits assessment of statistical reliability.

---

Main Quantitative Results

Training Performance Across Benchmarks

The paper's headline result is that LPO consistently outperforms matched PG baselines across nearly all task, backbone, and temperature configurations. The evidence is organized in two figures plotting training curves:

Pass@1 accuracy (Figure 3, 15 subpanels). Each subpanel shows Pass@1 training curves for one task-backbone-baseline combination, comparing PG baseline (GRPO, Dr.GRPO, or MaxRL) against both LPO variants (LPOfwd and LPOrev) with matched temperature. The pattern is consistent:

• Countdown 4B (GRPO): LPOfwd reaches ~55% Pass@1 by step 40, compared to ~52% for GRPO and ~53% for LPOrev. The gap widens through training, with LPOfwd at ~58% vs. GRPO at ~55% at step 120.
• Countdown 4B (Dr.GRPO): LPOfwd reaches ~57% vs. Dr.GRPO at ~54% at step 120.
• Countdown 4B (MaxRL): LPOfwd reaches ~57% vs. MaxRL at ~54%.
• MATH 1.7B (GRPO): All methods track closely until step ~400, after which LPOfwd diverges upward to ~35.5% vs. GRPO at ~34%, while LPOrev matches GRPO.
• MATH 1.7B (Dr.GRPO): LPOfwd reaches ~33% vs. Dr.GRPO at ~32%, with LPOrev at ~32%.
• MATH 1.7B (MaxRL): LPOfwd reaches ~35.5% vs. MaxRL at ~33%, LPOrev at ~34%.
• MATH 8B (GRPO): LPOfwd reaches ~49.5% vs. GRPO at ~48.5%, LPOrev at ~48%.
• MATH 8B (Dr.GRPO): LPOfwd reaches ~50% vs. Dr.GRPO at ~48%.
• MATH 8B (MaxRL): LPOfwd reaches ~49.5% vs. MaxRL at ~48%, LPOrev at ~48.5%.
• PRIME Code 1.7B (GRPO): LPOfwd reaches ~42% vs. GRPO at ~40%, LPOrev at ~40%.
• PRIME Code 1.7B (Dr.GRPO): LPOfwd reaches ~42% vs. Dr.GRPO at ~41%.
• PRIME Code 1.7B (MaxRL): LPOfwd reaches ~42% vs. MaxRL at ~40%.
• Geometry 3B (GRPO): LPOfwd reaches ~44.5% vs. GRPO at ~42%, LPOrev at ~43%.
• Geometry 3B (Dr.GRPO): LPOfwd reaches ~44% vs. Dr.GRPO at ~42%.
• Geometry 3B (MaxRL): LPOfwd reaches ~44.5% vs. MaxRL at ~42%, LPOrev at ~43.5%.

The paper reports (Section 5.2) that both LPO variants exceed their PG baselines in 13/15 scenarios for Pass@1. The two exceptions are not explicitly identified but can be inferred from the curves: LPOrev underperforms GRPO on MATH 1.7B (GRPO) and roughly matches Dr.GRPO on MATH 1.7B (Dr.GRPO) in early training.

Pass@k accuracy (Figure 4, 15 subpanels). The diversity-preserving advantage of forward KL becomes pronounced in Pass@k evaluation:

• Countdown 4B (GRPO): LPOfwd reaches ~83% Pass@64 vs. GRPO at ~80% at step 120. LPOrev roughly matches GRPO.
• MATH 1.7B (GRPO): LPOfwd reaches ~46.5% vs. GRPO at ~44%, LPOrev at ~43%.
• MATH 8B (GRPO): LPOfwd reaches ~61% vs. GRPO at ~58%, LPOrev at ~56%.
• MATH 8B (MaxRL): LPOfwd reaches ~62% vs. MaxRL at ~58%, LPOrev at ~60%.
• PRIME Code 1.7B (GRPO): LPOfwd reaches ~51% Pass@8 vs. GRPO at ~48%.
• Geometry 3B (GRPO): LPOfwd reaches ~70% Pass@16 vs. GRPO at ~66%.

The paper reports (Section 5.2) that LPOfwd outperforms PG baselines in 15/15 Pass@k scenarios, while LPOrev does so in 11/15. Comparing the two LPO variants, LPOfwd outperforms LPOrev in 13/15 Pass@k scenarios. This asymmetric advantage — forward KL dominating on Pass@k while reverse KL is more competitive on Pass@1 — directly validates the theoretical properties: forward KL's mode-covering preserves reasoning diversity needed for high-coverage evaluation, while reverse KL's mode-seeking concentrates on the best single solution.

Robustness across temperature parameterizations (Figures 3 and 4, rows). The gains from LPO are not tied to a particular temperature design. Across all three baselines (GRPO with $\tau = \sigma_G$, Dr.GRPO with $\tau = 1$, MaxRL with $\tau = \mu_G$), LPO variants consistently outperform their paired PG counterpart. The paper interprets this (Section 5.2) as evidence that "exact listwise projection provides a robust optimization mechanism, yielding benefits that are largely orthogonal to the underlying temperature heuristic." In other words, the structural advantages of bounded, zero-sum, self-correcting gradients are beneficial regardless of target sharpness.

Scalability to larger models (Appendix E.1, Figure 8). Training Qwen3-14B-Base on the larger Polaris dataset (~53k problems) with GRPO ($\tau = \sigma_G$):

• LPOfwd reaches GRPO's peak Pass@1 performance (at step 200) within only ~70 steps — a roughly 3× sample efficiency improvement.
• At convergence, LPOfwd achieves ~50% Pass@1 vs. GRPO at ~48.5%, and ~62% Pass@k vs. GRPO at ~59%.
• LPOrev matches GRPO on Pass@1 but maintains higher Pass@k, consistent with the reverse KL's implicit entropy bonus preserving diversity.

---

Training Dynamics: Entropy, Gradient Norms, and Response Length

Figure 5 (three rows × five columns, showing metrics for LPO variants vs. GRPO across tasks) provides evidence that the structural gradient properties translate to measurable optimization dynamics:

Response entropy (Figure 5, top row). Both LPO variants maintain higher response entropy than GRPO across all tasks:

• On Countdown 4B, entropy starts at ~0.28 and declines to ~0.22 for LPOfwd vs. ~0.15 for GRPO by step 120.
• On MATH 1.7B, LPOfwd entropy stabilizes at ~0.06 vs. GRPO at ~0.04 by step 800.
• On MATH 8B, LPOfwd entropy holds at ~0.03–0.04 vs. GRPO declining to ~0.02.
• On PRIME Code 1.7B, the gap is stark: LPOfwd maintains entropy above 0.15 throughout training, while GRPO drops to ~0.05 by step 400.
• On Geometry 3B, LPOfwd entropy remains at ~0.25 vs. GRPO declining to ~0.15.

The paper attributes this to the projection properties (Section 5.3): LPOrev corresponds to a maximum-entropy objective (the reverse KL decomposes as $-\sum_k P_{\theta,k}\phi_k - H(P_\theta)$, with an explicit entropy bonus $H(P_\theta)$), while LPOfwd's mode-covering behavior forces the policy to maintain mass on multiple responses. The sustained entropy directly explains the Pass@k improvements because diverse response distributions increase the probability of finding correct solutions among $k$ samples.

Gradient norms (Figure 5, middle row). LPO variants exhibit consistently lower and more stable gradient norms:

• On Countdown 4B, GRPO gradient norms fluctuate in [0.15, 0.40] while LPOfwd stays in [0.18, 0.28] and LPOrev in [0.16, 0.22].
• On MATH 1.7B, GRPO norms spike to ~0.20 early and oscillate, while LPOfwd stays below ~0.14.
• On MATH 8B, GRPO norms reach ~0.18 with high variance; LPOfwd and LPOrev stay below ~0.10.
• On PRIME Code 1.7B, GRPO norms spike to ~0.35 while LPOfwd remains below ~0.10, an especially large gap.
• On Geometry 3B, the effect is dramatic: GRPO norms reach ~3.5–4.0 (likely due to the multi-modal nature exacerbating PG variance), while LPO variants stay below ~0.5.

The paper connects this to Corollary 1 (Section 5.3): the bounded coefficients $|c^{\text{fwd}}_k| \leq 1$ and $\sum_k |c^{\text{fwd}}_k| \leq 2$ provide an intrinsic gradient norm bound of $2 G_{\text{max}}$ that is independent of reward scale or group statistics. The zero-sum property further reduces variance by canceling contributions from over- and under-allocated responses. The GRPO spikes likely correspond to groups where $\sigma_G \approx 0$ (all responses correct or all wrong), causing advantage explosion that LPO's bounded coefficients naturally avoid.

Response length (Figure 5, bottom row). LPO variants generate longer responses than GRPO:

• On MATH 1.7B, LPOfwd response length grows to ~1,100 tokens vs. GRPO at ~900 by step 800.
• On MATH 8B, LPOfwd reaches ~1,250 vs. GRPO at ~1,050.
• On PRIME Code 1.7B, LPOfwd at ~700 vs. GRPO at ~550.
• On Geometry 3B, LPOfwd at ~280 vs. GRPO at ~160.

The paper notes (Section 5.3) that increased length often correlates with more detailed reasoning chains, consistent with LPO encouraging more extensive exploration. LPOfwd's maximum length aligns with its mode-covering property, which promotes diverse reasoning paths that tend to be longer.

Extended dynamics across baselines (Appendix E.2, Figures 9 and 10). The same patterns hold when comparing LPO against Dr.GRPO and MaxRL: higher entropy, lower gradient norms, and longer responses for LPO variants across all tasks. This confirms that the stability benefits are not specific to GRPO's normalization scheme but rather a consequence of the exact listwise projection.

---

Listwise vs. Pointwise Projection Ablation

This experiment (Section 5.4.1, Figure 6) isolates the contribution of the listwise normalization by comparing LPO against a pointwise projection that uses the same target $w^*$ but optimizes $-\sum_k w^*_k \log \pi_\theta(y_k|x)$ (treating each response independently, without softmax normalization over the group). On MATH 1.7B:

• Pass@1: The pointwise variant severely underperforms, reaching only ~30% compared to ~35.5% for LPOfwd and ~34% for GRPO by step 800. The pointwise curve shows unstable, oscillating behavior rather than steady improvement.
• Gradient norms: The pointwise variant exhibits gradient norms spiking to ~0.5–0.6 compared to LPOfwd's stable ~0.06–0.08 and GRPO's ~0.12.

The paper explains this failure (Appendix C.4): pointwise coefficients $c^{\text{point}}_k = -w^*_k$ are strictly negative and lack the zero-sum property ($\sum_k c^{\text{point}}_k = -1$), meaning there is no competitive counterbalancing force — the gradient pushes all responses upward simultaneously without relative comparison. The listwise normalization couples responses through the softmax, providing the built-in control variate that stabilizes training. The paper emphasizes (Section 5.4.1): "These results suggest that our performance gains stem not merely from the target design, but from successfully marrying exact target fitting with the crucial structural variance reduction provided by the listwise projection."

---

Effect of Group Size K

This experiment (Section 5.4.2, Figure 7) sweeps $K \in \{2, 4, 8, 16, 32\}$ on Countdown 4B, comparing LPOfwd, LPOrev, and GRPO at the final training step:

Pass@1 (Figure 7, left):

• At $K = 2$: LPOfwd ~62%, LPOrev ~64%, GRPO ~55% — both LPO variants substantially outperform GRPO, with LPOrev achieving the strongest Pass@1.
• At $K = 4$: LPOfwd ~60%, LPOrev ~61%, GRPO ~56%.
• At $K = 8$: LPOfwd ~58%, LPOrev ~57%, GRPO ~55%.
• At $K = 16$: All methods converge to ~53–55%.
• At $K = 32$: All methods at ~52–53%, with GRPO slightly ahead.

The paper interprets this (Section 5.4.2): LPO's advantage is most pronounced at small group sizes, suggesting that explicit listwise projection improves sample efficiency by making better use of limited per-prompt samples. At larger $K$, the PG approximation becomes more accurate (more samples reduce variance and make the first-order approximation closer to the true reverse KL gradient), narrowing the gap.

Pass@64 (Figure 7, right):

• At $K = 2$: LPOfwd ~89%, LPOrev ~81%, GRPO ~78%. LPOfwd's advantage is dramatic — 11 percentage points over GRPO and 8 over LPOrev.
• At $K = 4$: LPOfwd ~87%, LPOrev ~83%, GRPO ~80%.
• At $K = 8$: LPOfwd ~83%, LPOrev ~80%, GRPO ~79%.
• At $K = 16$: LPOfwd ~80%, LPOrev ~78%, GRPO ~77%.
• At $K = 32$: LPOfwd ~77%, LPOrev ~76%, GRPO ~76%.

The distinct scaling behaviors validate the theoretical properties: LPOrev achieves stronger Pass@1 performance (mode-seeking concentrates on the best solution), while LPOfwd scales exceptionally well on Pass@64 (mode-covering structurally preserves reasoning diversity). As $K$ grows, the gap narrows because larger groups naturally capture more diversity regardless of the optimization method.

---

Generalization Across LLM Families

This experiment (Section 5.4.3, Figure 11 in Appendix E.3) evaluates LPOfwd, LPOrev, and GRPO ($\tau = \sigma_G$) on Countdown across four model families:

• Qwen3-4B-Base: LPOfwd ~58% Pass@1 vs. GRPO ~55%; LPOfwd ~83% Pass@64 vs. GRPO ~80%.
• DeepSeek-R1-Distill-Qwen-1.5B: LPOfwd ~49% Pass@1 vs. GRPO ~46%; LPOfwd ~76% Pass@64 vs. GRPO ~72%.
• Llama-3.1-8B-Instruct: LPOfwd ~58% Pass@1 vs. GRPO ~56%; LPOfwd ~78% Pass@64 vs. GRPO ~74%.
• Mistral-7B-Instruct-v0.1: LPOfwd ~26% Pass@1 vs. GRPO ~24%; LPOfwd ~39% Pass@64 vs. GRPO ~34%. This is the largest relative gain, with LPOfwd achieving ~15% relative improvement on Pass@64.

The consistent gains across base, distilled, and instruction-tuned models suggest LPO is not sensitive to architecture or training paradigm, but benefits from the fundamental robustness of listwise projection. The Mistral result is particularly notable because it demonstrates LPO can extract meaningful improvements even when absolute performance is low (~26% Pass@1), suggesting the framework is beneficial across capability levels.

---

Final Evaluation on Math Benchmarks

Table 3 (Appendix E.5) reports final benchmark-level scores for Qwen3-1.7B-Base and Qwen3-8B-Base trained on MATH. For brevity, focusing on the aggregate Pass@1 and Pass@k (rightmost columns):

Qwen3-1.7B-Base (6 benchmarks):

• Base (no RLVR): Pass@1 22.0, Pass@k 40.2
• GRPO: Pass@1 32.5, Pass@k 42.1 → LPOfwd: 35.3, 46.1 → LPOrev: 35.0, 42.3
• Dr.GRPO: Pass@1 32.2, Pass@k 42.2 → LPOfwd: 33.4, 42.5 → LPOrev: 32.9, 43.9
• MaxRL: Pass@1 32.4, Pass@k 42.8 → LPOfwd: 35.0, 45.6 → LPOrev: 33.6, 45.0

LPOfwd with MaxRL temperature achieves the best Pass@1 (35.0) and Pass@k (45.6), while LPOfwd with GRPO temperature gives the best Pass@k improvement (+4.0 over GRPO). LPOrev with Dr.GRPO temperature yields the best Pass@k for that baseline (43.9 vs. 42.2).

Qwen3-8B-Base (6 benchmarks):

• Base (no RLVR): Pass@1 33.3, Pass@k 50.0
• GRPO: Pass@1 47.6, Pass@k 54.5 → LPOfwd: 50.3, 58.3 → LPOrev: 48.7, 56.5
• Dr.GRPO: Pass@1 49.1, Pass@k 60.4 → LPOfwd: 49.5, 59.4 → LPOrev: 47.7, 56.9
• MaxRL: Pass@1 48.6, Pass@k 58.2 → LPOfwd: 50.5, 63.1 → LPOrev: 50.6, 60.3

LPOfwd with MaxRL temperature achieves the highest Pass@k (63.1, +4.9 over MaxRL baseline). The Dr.GRPO baseline is unusually strong on Pass@k (60.4, best among PG baselines), and LPOfwd doesn't surpass it (59.4), though LPOrev does on Pass@1 for MaxRL (50.6 vs. 48.6).

Out-of-distribution evaluation (Table 4, Appendix E.5). On general reasoning benchmarks (ARC-c, MMLU-Pro, GPQA-diamond) using Qwen3-8B-Base trained on MATH:

• GRPO: avg 38.2 → LPO: 38.5
• Dr.GRPO: avg 37.5 → LPO: 40.2
• MaxRL: avg 30.3 → LPO: 32.3

The paper acknowledges "inherent variance" in OOD evaluation and suggests multi-domain joint training as future work. The gains are modest but consistently positive for LPO variants across baselines.

---

Ablation Studies and Robustness Checks

Temperature design as experimental control: The entire experimental design is an implicit ablation on the projection mechanism vs. temperature tuning. By pairing each LPO variant with the identical $\tau$ as its PG baseline, the paper cleanly isolates exact listwise projection as the cause of performance differences. The fact that gains appear across all three $\tau$ designs (GRPO's $\sigma_G$, Dr.GRPO's 1, MaxRL's $\mu_G$) is itself the key robustness check — the projection mechanism provides orthogonal benefits.

Listwise vs. pointwise projection (Section 5.4.1, Figure 6): Remove the listwise normalization while keeping the exact same target $w^*$, and performance collapses from ~35.5% to ~30% on MATH 1.7B, with gradient norms spiking to 0.5–0.6 vs. 0.06–0.08 for listwise. This is the paper's strongest causal evidence that the listwise simplex structure — not just the target design — is responsible for the gains. The pointwise variant lacks the zero-sum competitive balancing mechanism, resulting in unstable optimization.

Group size sweep (Section 5.4.2, Figure 7): LPO advantages are most pronounced at small $K$ (2, 4, 8) and narrow at large $K$ (16, 32). This is consistent with the theoretical degradation of the PG approximation — at small $K$, off-policy drift degrades the PG gradient more severely (fewer samples = higher variance in the first-order approximation), so the exact projection provides larger benefits. At large $K$, the PG approximation becomes more accurate, reducing the gap. This is a non-obvious prediction of the theory that the experiments confirm.

Model family generalization (Section 5.4.3, Figure 11): LPO outperforms GRPO across Qwen, DeepSeek, Llama, and Mistral families — base, distilled, and instruction-tuned variants. This rules out the hypothesis that LPO's benefits are specific to Qwen's architecture or pretraining. The Mistral result (absolute performance ~24–26% Pass@1) shows LPO helps even at low capability levels.

Scalability to larger models and datasets (Appendix E.1, Figure 8): Training Qwen3-14B-Base on Polaris (~53k problems) with LPO shows the framework scales to ~14B parameters without degradation — LPOfwd achieves 3× sample efficiency and better final performance. This addresses a common concern that method improvements demonstrated at small scale (1.7B–8B) may not persist at larger scale.

Fully on-policy regime (Appendix E.4, Figure 12): With exactly one gradient update per iteration (batch size = mini-batch size = 256), LPOrev's training curve becomes "practically indistinguishable" from GRPO on Countdown, exactly as Proposition 1 predicts — at the on-policy point, reverse KL projection and PG produce identical gradients. LPOfwd still shows advantages (higher sample efficiency early, better Pass@k), confirming that forward KL's benefits are not dependent on off-policy drift.

Extended dynamics across all baselines (Appendix E.2, Figures 9 and 10): The entropy, gradient norm, and response length patterns observed for GRPO (Figure 5) replicate for Dr.GRPO (Figure 9) and MaxRL (Figure 10), confirming the stability benefits are intrinsic to listwise projection, not an artifact of GRPO's specific normalization.

Negative result — ReST^EM not applicable: The paper does not report ReST^EM experiments for LPO (unlike the revision model experiments in the referenced paper), so there is no negative result regarding RL-based optimization of LPO-trained policies. This is a gap — it would be informative to know whether further RL optimization on top of LPO-trained models degrades or improves performance, analogous to how the reference paper found ReST^EM hurt their revision model.

Missing ablation — divergence scheduling: The paper mentions (Appendix C.6) that the framework supports arbitrary divergences and suggests forward KL for early exploration followed by reverse KL for late exploitation, but no experiment tests this. This is a natural ablation that would strengthen the claim that divergence choice is a meaningful design axis.

Missing ablation — temperature tuning for LPO: The paper intentionally inherits $\tau$ from baselines and does not report results with independently tuned temperatures for LPO. It is possible that LPO would benefit from different $\tau$ values than the PG baselines (since the projection mechanism is different), and the current results may understate LPO's potential. The paired design is methodologically clean but leaves open the question of absolute performance ceilings.

---

Critical Assessment

The central claim that LPO outperforms matched PG baselines is well-supported, but the magnitude of gains varies substantially by benchmark and metric. On Pass@1, LPO variants exceed baselines in 13/15 scenarios, but the absolute differences are often modest: ~1–2 percentage points on MATH 8B, ~2 percentage points on Countdown, ~2 percentage points on PRIME Code. The gains are more substantial on Pass@k, where LPOfwd wins 15/15 with larger margins (e.g., +4 on MATH 8B Pass@k, +5 on Countdown Pass@64). The group-size experiment (Figure 7) and the pointwise ablation (Figure 6) provide the strongest causal evidence — large effects under specific conditions that directly test the theoretical mechanism. However, the aggregate curves (Figures 3, 4) show gains that, while consistent, are often within the range that could be sensitive to hyperparameter tuning or random seed variation. The paper reports no error bars, confidence intervals, or multiple-seed statistics, making it difficult to assess whether a 1–2 percentage point gap is statistically reliable or within noise. This is a notable limitation given that RL training curves are known to be high-variance.

The claim that LPO provides "stable optimization trajectories" is supported by gradient norm and entropy measurements, but the causal chain from coefficient properties to training dynamics is incompletely tested. The bounded/zero-sum/self-correcting properties are proven mathematically (Corollary 1), and the gradient norm curves (Figure 5, middle) are indeed lower and smoother for LPO. However, the paper does not include an ablation that isolates these properties — e.g., by constructing an LPO variant with artificial gradient norm inflation and showing it degrades stability, or by comparing against a PG variant with explicit gradient clipping matched to LPO's bound. The connection between lower gradient norms and better final performance is correlational, not causally demonstrated. It could be that LPO's gains come primarily from the entropy bonus (implicit in reverse KL, structural in forward KL's mode-covering) rather than from gradient boundedness per se, and the lower gradient norms are a side effect rather than the mechanism.

The diversity preservation claim (higher Pass@k) is well-evidenced but only for the specific $k$ values tested. LPOfwd dominates on Pass@k across 13/15 scenarios, and the group-size experiment (Figure 7) shows LPOfwd scaling exceptionally well on Pass@64. However, all Pass@k evaluations use $k$ values that are powers of 2 (4, 8, 16, 32, 64) relative to the training group size $K = 8$. It is unclear whether the diversity advantage persists for $k \gg K$ (e.g., Pass@256) or whether the policy's listwise distribution over only 8 training responses captures enough diversity to maintain coverage at much larger sample sizes. The mode-coverage bound (Corollary 2) guarantees $P_{\theta,k} > \alpha \exp(-D/\alpha - 1)$ when $w^*_k \geq \alpha$, but this is a lower bound on the listwise distribution, not on the full sequence-level Pass@k — the connection between simplex-level diversity and sequence-level diversity is assumed rather than proven.

The generalization claim (Section 5.4.3) uses only Countdown with GRPO temperature. While four model families are tested, it is a single task (Countdown) with a single baseline (GRPO, $\tau = \sigma_G$). The paper does not report whether LPO generalizes across model families on MATH, PRIME Code, or Geometry. The Countdown task is notably simpler than MATH or competition-level code generation — it requires combinatorial search over basic arithmetic rather than complex multi-step mathematical reasoning — so the generalization results may not extend to harder tasks.

The scalability claim (Appendix E.1, Figure 8) is based on a single run with Qwen3-14B-Base on Polaris, compared only against GRPO. The claim that LPOfwd achieves "3× sample efficiency" is accurate within that specific comparison but is not tested across multiple runs or multiple large-scale configurations. The paper does not report whether LPO scales to 70B+ models, which is the current frontier for reasoning models. The computational cost of LPO is stated to be identical to PG baselines, but the softmax computation over $K = 8$ elements is trivially cheap — the real question is whether the optimization properties that produce gains at 1.7B–14B persist at 70B, where gradient noise characteristics and loss landscapes may be qualitatively different.

The off-policy degradation analysis (Appendix B.2) quantifies the PG approximation error but is not empirically validated. The paper proves that the PG-reverse-KL discrepancy scales as $O(\bar{\delta}(1 + \|A\|_\infty)/K)$, and the fully on-policy experiment (Figure 12) shows that when $\bar{\delta}$ is forced to zero (one update per iteration), LPOrev matches GRPO exactly. However, no experiment systematically varies the number of inner epochs (and hence $\bar{\delta}$) to show that LPO's advantage grows with off-policy drift as the theory predicts. This would be a strong test of the theoretical mechanism: if LPO's gains over PG are larger with more inner epochs (where the PG approximation degrades more), the off-policy error explanation is validated; if gains are constant regardless of inner epochs, the explanation must lie elsewhere.

Missing baseline — LPO-pointwise hybrid with control variate. The pointwise ablation (Figure 6) shows that removing listwise normalization causes collapse, but the paper does not test whether adding an explicit control variate (e.g., subtracting the mean coefficient $\sum_k c^{\text{point}}_k / K$) to the pointwise projection recovers some of the stability. This would help distinguish whether the listwise projection's benefit comes primarily from the zero-sum property (which a control variate could approximate) or from the coupled softmax normalization (which cannot be approximated by independent per-response updates).

The experimental design's strength — paired temperature comparison — is also its limitation. By inheriting $\tau$ from baselines, the paper demonstrates that exact projection helps regardless of target sharpness. But it does not answer whether LPO with an independently optimized $\tau$ would substantially outperform LPO with inherited $\tau$. The paper frames this as future work (Appendix C.1, point 4), but it means the reported numbers may not represent LPO's ceiling. A practitioner seeking the best absolute performance would tune $\tau$ as a hyperparameter regardless of the baseline's implicit value, and the paper provides no guidance on what optimal $\tau$ values might be for LPO or how they differ from PG-optimal values.

The connection to DPO (Appendix C.5) is theoretically elegant but empirically unexplored. The paper notes that at $K = 2$, LPOfwd reduces to a binary cross-entropy objective with soft targets, connecting to DPO. An experiment comparing LPO at $K = 2$ against offline DPO on the same data, or against online DPO variants, would strengthen the claim that the listwise framework unifies these approaches — but no such experiment exists. Similarly, the EM connection (Appendix C.4) suggests alternating target construction and projection, but LPO interleaves them with multiple projection steps per target (inner epochs $E$), and no ablation varies $E$ to test whether fewer or more projection steps per target improves performance.

In summary: the paper demonstrates consistent, replicable gains from exact listwise projection over first-order PG approximation across a wide range of tasks, model scales, and model families — a genuine empirical contribution. The gains on Pass@k from forward KL's mode-covering property are the most distinctive and theoretically grounded result. However, the absolute gains are often modest, the statistical reliability is not quantified, and several causal pathways from theoretical properties to empirical outcomes remain correlational rather than experimentally established. The framework's main value may ultimately be conceptual — providing a principled language for designing and analyzing group-based RLVR methods — with the empirical gains serving as validation that the conceptual advance has practical teeth, even if the teeth are not always very sharp.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Not Accounted For in the Headline Efficiency Claims

The assumption or constraint. The paper's entire compute-optimal framework rests on the ability to bin prompts into five difficulty quintiles before deciding how to allocate the inference budget. The method used to estimate difficulty — generating 2048 samples per question and averaging either ground-truth correctness (oracle) or PRM final-answer scores (predicted) — is extraordinarily expensive. The authors explicitly acknowledge this in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

At 2048 samples per question, the difficulty estimation step alone consumes more compute than the largest test-time budgets studied in the experiments (256–512 generations). The paper frames this as an "exploration-exploitation tradeoff" but provides no method for reducing this cost, no analysis of how the cost scales with the number of questions, and no amortization strategy where the difficulty estimation cost could be shared across multiple evaluations of the same prompt.

The consequence. The reported 4× efficiency gains over best-of-N are computed after difficulty is known, without including the cost of acquiring that knowledge. In a realistic deployment where each prompt is seen only once, the total compute cost would be difficulty estimation + strategy execution, and the former could dominate the latter by an order of magnitude. A system that generates 2048 samples to estimate difficulty and then uses 16 generations for the actual answer has a total cost of 2064 generations — far worse than simply running best-of-256 (256 generations) directly. The 4× figure is therefore best understood as an upper bound on achievable efficiency in the idealized scenario where difficulty is known a priori or amortized over many evaluations, not as a realized deployment gain. For any one-shot inference scenario (which is the common production use case), the actual cost would be substantially higher than the headline numbers imply, potentially making the approach strictly worse than a uniform best-of-N baseline.

What evidence exists in the paper. The paper reports no experiment that includes difficulty estimation cost in the generation budget calculations. Figures 4 and 8 show compute-optimal scaling curves where the x-axis ("Number of Generations") counts only the strategy execution cost, not the 2048 samples used to determine which strategy to execute. The paper acknowledges this gap in Section 3.2 but provides no empirical characterization of the tradeoff — e.g., how does performance change if difficulty is estimated from 64 samples instead of 2048? At what sample size does the difficulty estimate become unreliable? Could difficulty be estimated adaptively during the solution process itself?

Mitigation status. Not addressed. The paper explicitly flags this as a key avenue for future work in Section 8, suggesting "pretraining or finetuning models to directly predict difficulty of a question." No such model is developed or evaluated. The paper also mentions the possibility of "adaptive difficulty estimation" where a small number of initial samples inform the remaining budget allocation, but this is not explored. Until this gap is closed, the framework remains a proof-of-concept for adaptive test-time compute allocation rather than a deployable system.

---

All Results Are on a Single Benchmark with a Single Model Family, with No Evidence of Cross-Domain or Cross-Architecture Generalization

The assumption or constraint. Every experiment in the paper uses the MATH benchmark (Hendrycks et al., 2021) with 500 test questions, and all models are from the PaLM 2 family (specifically PaLM 2-S*). The authors state in Section 4 that they "believe this model is representative of the capabilities of many contemporary LLMs," but this claim is untested. The paper provides no evidence on code generation (e.g., HumanEval, MBPP), logical reasoning (e.g., ARC, FOLIO), scientific QA, open-ended generation, or any domain outside of competition-level mathematics. Similarly, no experiments are conducted with models from other families (e.g., Llama, Mistral, DeepSeek, GPT) to verify that the difficulty-dependent scaling patterns — beam search hurting easy problems, revisions helping easy problems but failing on hard ones, the 4× efficiency gain — transfer across architectures, pretraining distributions, or model scales.

The consequence. Several aspects of the findings could be specific to the MATH-PaLM 2-S* combination:

• The PRM's quality and its over-optimization behavior depend on PaLM 2-S*'s output distribution. A model with different calibration properties (e.g., one that tends to produce more diverse or more confident solutions) might exhibit different difficulty-dependent scaling curves, potentially shifting or even inverting the optimal strategy assignments (e.g., beam search might help on easy problems for a less-calibrated model).
• The revision model's ability to learn from incorrect-to-correct trajectories depends on the base model's in-context learning capabilities, which vary substantially across model families. A model with weaker in-context learning might fail to acquire the revision skill entirely, making the sequential revision axis of the framework non-viable.
• MATH consists of problems with unambiguous ground-truth answers that can be verified via exact string matching. Extending the framework to tasks where correctness is ambiguous, multi-dimensional, or subjective (e.g., dialogue quality, creative writing, code functionality with runtime errors) would require fundamentally different verifier training and difficulty estimation approaches. The paper provides no guidance on how the framework would adapt to such settings.
• The specific relationship between difficulty quintile boundaries and the optimal strategy choice (e.g., best-of-N for bins 1–2, beam search for bins 3–4) is likely sensitive to the base model's absolute performance level on MATH. A stronger model with higher baseline pass@1 might shift more questions into "easy" bins where revisions dominate, changing the overall allocation landscape.

What evidence exists in the paper. None for cross-domain or cross-model generalization. All figures (Figure 3, 4, 6, 7, 8, 9) are based on PaLM 2-S* evaluated on the MATH 500-question test set. The paper does not include even a single auxiliary experiment on a different benchmark or model family. The FLOPs-matched comparison in Section 7 uses a second PaLM 2 model with ~14× more parameters, which is still within the same model family.

Mitigation status. Not addressed. The paper does not claim generalization beyond its experimental scope — the authors are transparent that their findings are on MATH with PaLM 2-S* — but they also do not discuss the limitations this imposes on the conclusions. The statement that the model is "representative" (Section 4) is an assertion, not an empirical finding. A practitioner using a different model family (e.g., Llama-3 or a GPT-series model) on a different reasoning domain (e.g., code generation) has no basis in this paper for predicting whether the compute-optimal strategies will be similar or different.

---

The 14× Larger Model Baseline Is Not Compute-Optimally Trained, Weakening the Pretraining-vs-Inference Tradeoff Analysis

The assumption or constraint. The FLOPs-matched comparison in Section 7 compares PaLM 2-S* with compute-optimal test-time strategies against a model with approximately 14× more parameters but the same amount of training data. The authors explicitly state this departure from compute-optimal pretraining:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

This follows the LLaMA paradigm (Touvron et al., 2023) where model parameters are scaled while training tokens are held fixed, rather than the Chinchilla-optimal paradigm (Hoffmann et al., 2022) where both parameters and data are scaled proportionally to the FLOPs budget. Additionally, the 14× larger model uses only greedy decoding with no test-time compute augmentation — no majority voting, no best-of-N, and no search. This means the larger model is not given any opportunity to use test-time compute to improve its own outputs, making the comparison asymmetric: the smaller model gets adaptive test-time compute while the larger model gets none.

The consequence. The reported advantages of test-time compute over pretraining are likely overstated in two ways:

1. Weaker pretraining baseline. A Chinchilla-optimal model trained with 14× more total FLOPs would allocate additional compute to both parameters and data, producing a model that is not just larger but also trained on more tokens. This would likely yield better performance than the parameter-only-scaled model used in the paper, potentially shrinking or reversing the reported gains from test-time compute (e.g., the +27.8% relative improvement on easy questions at low inference-to-pretraining ratios, shown in Figure 1).

2. Asymmetric access to inference compute. Allowing the larger model to use test-time compute would create a fairer FLOPs-matched comparison: both models get the same total FLOPs, and each can allocate them between pretraining and inference as it sees fit. If the larger model used even a modest test-time compute budget (e.g., best-of-8), it might close or reverse the gap on medium-difficulty problems where the paper finds test-time compute to be most beneficial. The paper never tests this.

The FLOPs-matched analysis is one of the paper's most practically significant contributions — it is the section that a CTO or research manager would read to decide whether to invest in larger pretraining or smarter inference. If the pretraining baseline is weaker than it should be, the practical guidance ("on easy-to-medium problems, invest in test-time compute; on hard problems, invest in pretraining") may not hold against a properly optimized larger model.

What evidence exists in the paper. Section 7 and Figure 9 present the FLOPs-matched results, with the pretraining baseline clearly specified as parameter-scaled-only with greedy decoding. The paper does not include a Chinchilla-optimal pretraining baseline or a version of the larger model with test-time compute. The three $R$ values (0.16, 0.79, 22) vary the inference-to-pretraining token ratio, but the pretraining baseline remains the same across all three, so the comparison is only varying the test-time compute budget, not the pretraining configuration. This means the results characterize the tradeoff for one specific (and arguably suboptimal) pretraining strategy, not the tradeoff in general.

Mitigation status. The authors acknowledge the limitation explicitly in Section 7 and frame it as future work. However, the paper's conclusions — particularly the takeaway box in Section 7 that summarizes when to prefer test-time compute vs. pretraining — are presented without qualification regarding this limitation. A reader who does not carefully note the caveat in Section 7 could easily walk away with the impression that the tradeoff generalizes to compute-optimally trained larger models, which the paper does not establish.

---

Hard Problems Remain Completely Unsolved: Test-Time Compute Cannot Compensate for Fundamental Capability Gaps

The assumption or constraint. The paper's framework assumes that the base model's pass@1 on a given prompt is sufficiently above zero for test-time compute to have an effect — i.e., there exist correct solutions in the model's output distribution that search or revision can discover. For problems where the base model never (or almost never) produces a correct answer, no amount of test-time compute will help. The authors are explicit about this:

"On the hardest questions (bin 5), no method makes meaningful progress — the base model simply lacks the capability to produce correct solutions regardless of how the budget is allocated." (Section 5.3)

The consequence. This is a hard ceiling on the approach: test-time compute can amplify existing capability but cannot create capability from nothing. Across all methods studied — PRM search, iterative revisions, and their compute-optimal combinations — the hardest difficulty bin (bin 5) shows near-zero improvement regardless of budget:

• In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all methods (best-of-N, beam search, lookahead search) and all budgets up to 256 generations.
• In Figure 7 (right), bin 5 accuracy is roughly 2–3% across all sequential-to-parallel ratios at 128 generations.
• In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line for both revisions and PRM search is essentially flat near 0–5%, while the 14× larger model shows clear (though still low) performance above that level.
• In Figure 9 (right, PRM search), test-time compute with the smaller model at $R \gg 1$ shows a −52.9% relative disadvantage compared to the larger model on hard problems — meaning the larger model, even without test-time compute, substantially outperforms the smaller model with any amount of inference budget.

This limitation is fundamental rather than incidental: it means the approach offers no path forward for genuinely novel or out-of-distribution reasoning problems that exceed the base model's training distribution. For such problems, pretraining (or more advanced training methods like curriculum learning) remains the only viable approach. The difficulty-quintile framework quantifies this boundary nicely — bin 5 is the "pretraining-only" regime — but does not provide any method for pushing problems from bin 5 into lower bins through improved inference strategies.

What evidence exists in the paper. The bin 5 results are clearly visible in Figures 3 (right), 7 (right), and 9, and the authors discuss this limitation openly in Sections 5.3 and 7. The evidence is consistent and unambiguous: no method helps on the hardest quintile.

Mitigation status. Acknowledged but not addressed. The paper frames this as a characterization of the boundary condition rather than a problem to be solved — which is reasonable, since solving it would require fundamentally changing the base model's capabilities (which is a pretraining problem, not a test-time compute problem). However, the practical implication is significant: for any deployment where the prompt distribution includes a non-trivial fraction of problems outside the model's capability range, the compute-optimal framework provides no benefit on those problems and the total system accuracy may be dominated by the hard-problem failure rate regardless of how efficiently the easy-to-medium problems are handled. The paper quantifies this in the FLOPs-matched analysis but draws no prescriptive conclusions about when the hard-problem ceiling makes test-time compute a non-starter.

---

The Revision Model Has a Fundamental Correct-to-Incorrect Reversion Problem That Is Only Partially Mitigated

The assumption or constraint. The revision model is trained exclusively on trajectories where all in-context answers are incorrect and the target output is correct. This means the model never sees examples of correct answers in context during training, and consequently has no learned behavior for what to do when the current answer is already correct. The paper reports (Section 6.1):

"approximately 38% of correct answers get converted back to incorrect ones"

This is a direct consequence of the training data construction: the model learns to always produce a "revised" output that differs from its inputs, regardless of whether the inputs are correct. When the revision chain happens to generate a correct answer, the model is as likely to "revise" it into an incorrect answer as it is to preserve correctness — it has no signal for when not to revise.

The consequence. The revision chain is not monotonically improving — correctness can fluctuate up and down across sequential revisions, and there is no guarantee that later revisions are better than earlier ones. This has several practical consequences:

• Within-chain selection is required, adding complexity. The paper mitigates the reversion problem by using majority voting or verifier-based selection across the entire chain of revisions (picking the best answer from any point in the chain, not just the final revision). This means the revision model cannot simply be run autoregressively with the last output taken as the answer — a post-hoc selection mechanism must scan the entire chain, which adds computational overhead and requires a reliable verifier to identify the best answer.

• The verifier must be reliable on revision outputs. As noted in the paper (Appendix J, Figure 15a), the PRM trained on base model outputs does not transfer well to revision model outputs due to distribution shift — the revision model produces different kinds of solutions than the base model, and the PRM's scores become less calibrated. The paper trains a separate ORM specifically for revision outputs, but this is an additional model that must be trained and deployed, adding to the system complexity.

• The 38% reversion rate sets a ceiling on sequential revision gains. Each additional revision step has a non-trivial probability of corrupting a previously correct answer, meaning that longer chains are not necessarily better — there is an optimal chain length beyond which the reversion probability outweighs the improvement probability. The paper's revision chain experiments (Figure 6, left, extending to 64 steps) show that pass@1 at each step improves from ~18% to ~24% and then plateaus, consistent with the reversion rate limiting further gains. However, the paper does not explicitly analyze the tradeoff between revision depth and reversion probability, nor does it provide guidance on how to choose the optimal chain length for a given model and task.

What evidence exists in the paper. The 38% figure is reported in Section 6.1, and the within-chain selection mechanism (majority voting or verifier-based) is described in Section 6.1 and Appendix I. The revision model's verifier distribution shift is documented in Appendix J (Figure 15a). The revision chain plateau is visible in Figure 6 (left). The paper does not report the reversion rate as a function of chain position, difficulty bin, or problem type — the 38% is an aggregate number that likely varies across these dimensions.

Mitigation status. Partially addressed. The within-chain selection mechanism (majority voting or verifier-based) mitigates the impact of reversion by not forcing the system to take the final revision as the answer, but it is a post-hoc patch rather than a solution to the underlying training data problem. The paper does not explore more principled fixes, such as:

• Training the revision model with a "no-change" option where the correct output is identical to one of the in-context inputs (teaching the model to recognize when revision is unnecessary).
• Including correct-to-incorrect-to-correct trajectories in the training data to teach recovery behavior.
• Using the verifier's score on the current answer to decide whether to continue revising or stop (a dynamic revision depth based on confidence).

The authors acknowledge none of these possibilities explicitly, and the 38% reversion rate is presented as a fact about the system rather than as a problem to be solved.

---

Revisions and PRM Search Are Studied Independently, Never Combined, Leaving the Full Potential of the Framework Unexplored

The assumption or constraint. The paper's framework unifies two complementary axes of test-time compute — modifying the proposal distribution (via revisions) and modifying selection (via PRM-guided search) — but the experiments study these axes in complete isolation. The compute-optimal allocation policy selects among search algorithms (best-of-N, beam search, lookahead search) or among revision strategies (varying sequential-to-parallel ratios), but never combines them. The paper explicitly acknowledges this in Section 8:

"we did not experiment with PRM tree-search techniques in combination with revisions"

The consequence. The reported results represent a lower bound on what an integrated system could achieve. The two mechanisms have complementary, difficulty-dependent strengths that the paper itself documents:

• Revisions (proposal modification) are most effective on easy problems (bins 1–2) where the model's initial output is roughly correct and just needs refinement — a local search in answer space. (Figure 7, right)
• PRM search (verifier optimization) is most effective on medium-hard problems (bins 3–4) where the model needs to explore qualitatively different solution strategies — a global search. (Figure 3, right)

Combining them — using the revision model as the proposal distribution within beam search, or using the PRM to guide which revision branches to pursue — could yield gains beyond either method alone, particularly on medium-difficulty problems where both mechanisms show non-zero but individually suboptimal performance. For instance:

• Beam search over revision model outputs could explore diverse high-level approaches (through the search tree) while refining each approach locally (through sequential revisions), potentially breaking through the performance ceiling that each method individually hits.
• The PRM's step-level scores could be used to decide when a revision chain has gone off-track and should be terminated early, preventing wasted computation on revisions that are unlikely to recover.

The paper's compute-optimal allocation policy — which routes easy problems to revisions and medium problems to search — is a coarse approach that forces a choice between the two mechanisms per prompt. An integrated system could potentially apply both simultaneously, with the allocation policy instead controlling how much of each to use rather than which one to use.

What evidence exists in the paper. None for combined approaches. The revision experiments (Section 6) use only the base model (not PRM search) for selection, and the PRM search experiments (Section 5) use only the base model (not the revision model) for generation. The compute-optimal policies in Figures 4 and 8 select among strategies within each axis independently. The paper does not report any experiment where revision model outputs are fed into beam search, or where the PRM guides revision depth, or where the allocation policy selects between a search-only strategy and a revisions-only strategy on a per-prompt basis.

Mitigation status. Acknowledged as future work in Section 8, but not addressed. The authors state this is a natural next step, implying it was beyond the scope of the current paper. However, given that combining the two axes is the most obvious way to realize the full potential of the target-projection framework that the paper advocates, this omission is a significant gap — it means the paper demonstrates the existence of complementary mechanisms but not their synergy. A practitioner seeking to maximize performance would need to implement and tune the combination themselves, with no guidance from the paper on how the mechanisms interact, whether they interfere with each other (e.g., does the distribution shift from revision model outputs break the PRM's calibration?), or what the combined compute-optimal policy looks like.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper reframes the landscape of group-based RLVR for LLM training from an ad-hoc collection of advantage normalization schemes into a coherent, principled optimization framework built on a single geometric structure: the response simplex. The magnitude of this shift is best characterized as a unifying reframing with practical design consequences, rather than a paradigm-shattering breakthrough. Prior to this work, the dominant methods — GRPO, Dr.GRPO, MaxRL, DAPO, RLOO, REINFORCE++ — appeared as independently motivated heuristics with distinct advantage formulas, each justified through empirical performance on specific benchmarks and each requiring its own set of stabilization tricks (clipping, dynamic sampling, length normalization). The relationship between them was unclear: was MaxRL's use of µ_G as a denominator a fundamentally different algorithm from GRPO's use of σ_G, or just a reparameterization? The paper answers this definitively: they differ only in the temperature τ of the implicit softmax target they are projecting toward, and all are performing a first-order approximation of the same reverse KL projection on the response simplex.

This reframing has several immediate consequences for how the field thinks about RLVR algorithm design:

First, it collapses the design space from infinite to manageable. Before this work, developing a new group-based RLVR method meant proposing a new advantage formula — a one-dimensional search over functions of (R_k, µ_G, σ_G, µ_B, σ_B, ...) with no principled guidance on what dimensions matter and which are cosmetic. After this work, the design space decomposes into two independent choices: (1) what temperature τ should the target use (controlling sharpness), and (2) what divergence should be used for the projection (controlling mode-seeking vs. mode-covering behavior). The centering term (whether group mean, leave-one-out, or greedy-decode baseline) is mathematically irrelevant to the target shape because it cancels under softmax shift-invariance — it only matters for the quality of the first-order PG approximation, not for what the algorithm is trying to achieve. This is not an incremental observation; it provides a decision framework for algorithm design that was previously absent.

Second, it opens a design axis (projection divergence) that was structurally inaccessible. Policy gradient methods are locked into reverse KL projection — the gradient ∑_k (A_k/K) ∇_θ log π_θ(y_k|x) is always and only a first-order approximation of -∇_θ D_KL(P_θ ‖ w^*). There is no advantage vector A whose softmax target combined with a PG update would yield a forward KL projection. By decoupling target construction from projection, LPO makes divergence choice a first-class design decision, and the empirical results in Figures 3 and 4 demonstrate this matters concretely: forward KL (mode-covering) dominates on Pass@k in 13/15 scenarios because it preserves the reasoning diversity needed for high-coverage evaluation, while reverse KL is competitive on Pass@1 where concentrating on the single best solution is sufficient. This is not a claim about LPO specifically — it is a finding about the geometry of the response simplex that any future method operating on this space can exploit.

Third, it provides a diagnostic lens for understanding why existing methods fail in specific ways. The paper's analysis of off-policy degradation (Appendix B.2) shows that the PG approximation error scales as O(δ̄(1 + ‖A‖_∞)/K), where δ̄ measures policy drift from the sampling distribution. This explains why methods that perform multiple inner epochs per batch of rollouts — which is standard practice — experience gradient degradation that clipping and other heuristics are patching over. It also explains why LPO's exact projection maintains gradient quality regardless of off-policy drift, since it recomputes P_θ from current parameters at each gradient step. The practical implication is that many of the stabilization tricks accumulated in the RLVR literature (dynamic sampling in DAPO, asymmetric clipping, token-level loss normalization) are compensating for a structural mismatch between the intended objective (reverse KL projection) and the actual gradient (first-order approximation). Whether LPO makes these heuristics obsolete is an open question, but the analysis provides a causal explanation for their necessity that was previously absent — they are addressing approximation error, not fundamental optimization challenges.

Fourth, it reconciles the apparent contradiction between RLVR methods and preference optimization methods. The connection forged in Appendix C.5 — that at K=2, LPO's forward KL objective reduces to a binary cross-entropy with soft targets, structurally analogous to DPO — places pairwise preference optimization as the K=2 special case of a broader listwise target-projection family. This unifies two literatures that had been developing largely independently: online RLVR with absolute rewards (GRPO and its variants) and offline preference optimization with pairwise comparisons (DPO and its variants). The difference reduces to whether the target distribution w^* is constructed from rewards (RLVR) or from preferences (DPO), and whether the projection is performed online (sampling from the current policy) or offline (using a fixed dataset). This unification suggests that techniques from either tradition — e.g., listwise ranking losses from learning-to-rank, identity-preference optimization from DPO variants, natural gradient methods from information geometry — can be imported into the other through the shared simplex interface.

Which research directions become more attractive. The paper makes several directions newly tractable or more compelling:

• Divergence engineering becomes a legitimate research activity rather than a theoretical curiosity. The paper demonstrates that forward vs. reverse KL matters empirically, but the simplex framework admits any differentiable divergence (Appendix C.6), including Jensen-Shannon, α-divergences, and f-divergences with potentially different mode-seeking/mode-covering tradeoffs. A systematic empirical study of divergence geometry on the response simplex — measuring Pass@1 vs. Pass@k tradeoffs, entropy preservation, and robustness to temperature — could identify Pareto-optimal divergences for different training objectives.

• Dynamic divergence scheduling becomes feasible. The paper suggests (Appendix C.6) using forward KL for early exploration and reverse KL for late exploitation, but no experiment tests this. Because the divergence is a drop-in module in LPO's projection step, such scheduling is trivial to implement and could yield gains beyond any single divergence.

• Off-policy replay in RLVR becomes theoretically grounded. The paper notes (Appendix C.2) that LPO can incorporate off-policy data via importance sampling ratios in s_{θ,k}, with the listwise normalization acting as a self-normalizing importance sampling estimator. The practical challenge — severe off-policy drift causing extreme probability ratios — remains, but the simplex formulation provides a clean mathematical structure for developing staleness-filtering or trust-region buffer management strategies.

• Step-level listwise projection extends the framework to fine-grained optimization. The paper sketches this (Appendix C.2): given a shared intermediate state, sample K candidate continuations to form a local response simplex, and use a value network or PRM to estimate expected future returns for each continuation. This would enable LPO-style optimization at each reasoning step rather than only at the sequence level, potentially capturing intermediate reasoning quality signals that sequence-level rewards miss.

Which directions become less attractive. The paper implicitly argues that new advantage normalization schemes are a red herring. If the centering term cancels under the softmax target and only the temperature τ matters, then proposals for novel advantage formulas (beyond changing τ) are not exploring new optimization geometry — they are exploring better approximations to the same reverse KL projection. The paper does not say this explicitly, but the logical consequence of the unified framework is that the field should shift attention from advantage design to divergence design and temperature scheduling. Similarly, increasingly complex stabilization heuristics (dynamic sampling rules, adaptive clipping schedules, multi-stage normalization) may be addressing symptoms of the PG approximation error rather than fundamental optimization challenges; if the approximation error can be eliminated entirely via exact projection (as LPO does), the need for these heuristics may be greatly reduced. The paper does not prove this — the experiments intentionally use a minimal shared pipeline, but do not ablate the heuristics one-by-one to show they are redundant under LPO — but the theoretical analysis strongly suggests it.

Follow-Up Research This Work Enables

Divergence benchmarking on the response simplex across reasoning tasks. The paper implements forward and reverse KL as two instantiations of a general divergence framework (Appendix C.6) and shows they have qualitatively different empirical behavior — forward KL dominates on Pass@k, reverse KL is competitive on Pass@1. The natural next step is a systematic empirical study: implement 5–10 statistical divergences (Jensen-Shannon, α-divergences with α ∈ {0.5, 1.0, 2.0}, squared Hellinger, total variation, etc.) as drop-in projection modules in LPO, and benchmark them across the same four reasoning domains (Countdown, MATH, PRIME Code, Geometry) with Qwen3 backbones at 1.7B, 4B, and 8B scales. For each divergence, measure: (a) final Pass@1 and Pass@k, (b) response entropy trajectory, (c) gradient norm stability, (d) sensitivity to temperature τ (sweep τ values around the GRPO/Dr.GRPO/MaxRL implicit values to test whether optimal τ varies by divergence). The paper's Corollary 2 provides a theoretical mode-coverage bound for forward KL specifically; analogous bounds for other divergences would strengthen or qualify the empirical findings. A strong follow-up would identify whether any divergence Pareto-dominates forward KL (better Pass@1 without sacrificing Pass@k), or whether forward KL's advantages are specific to the Pass@k-diversity tradeoff. This experiment requires no new infrastructure beyond the existing LPO implementation — only plugging in different divergence gradient formulas.

Dynamic divergence scheduling during a single training run. The paper notes (Appendix C.6) that the decoupled projection enables scheduling divergences across training stages, but no experiment tests this. A concrete experiment: train Qwen3-8B-Base on MATH with LPO, using forward KL for the first 50 training steps (to encourage broad exploration and build diverse reasoning strategies), then switch to reverse KL for steps 50–400 (to exploit the best discovered strategies and improve Pass@1). Compare against forward-KL-only and reverse-KL-only baselines on final Pass@1 and Pass@k, and measure whether the switch point affects performance. A more sophisticated version would use an adaptive schedule: monitor response entropy during training, and switch from forward to reverse KL when entropy drops below a threshold (indicating sufficient diversity has been established). This tests whether the mode-covering and mode-seeking properties can be sequenced to get the benefits of both without the costs of either — forward KL's early diversity preservation preventing premature mode collapse, reverse KL's late exploitation maximizing final accuracy. The paper's entropy curves (Figure 5, top row) show that LPO naturally maintains higher entropy than PG baselines, so dynamic scheduling might further improve this by explicitly controlling the diversity-accuracy tradeoff at different training phases.

Off-policy replay buffer for sample-efficient LPO. The paper identifies off-policy replay as a natural extension (Appendix C.2) because the listwise distribution P_θ uses importance sampling ratios to correct for the sampling distribution mismatch, and the simplex normalization acts as a self-normalizing importance sampling estimator. However, the paper does not implement or test this. A concrete experiment: during LPO training on MATH with Qwen3-8B-Base, maintain a replay buffer of the last 5–10 batches of (prompt, responses, rewards, behavior policy log-probabilities). At each training iteration, in addition to the current on-policy batch, sample a second mini-batch from the replay buffer and compute targets and gradients using the stale behavior policy log-probabilities (with importance sampling corrections in s_{θ,k}). Compare the sample efficiency (Pass@1 vs. number of fresh rollouts generated) against on-policy-only LPO and against a PG baseline with replay (e.g., importance-weighted PG). The key challenge is staleness: as the policy evolves, the replay buffer's behavior policy becomes increasingly off-policy, and extreme importance ratios can destabilize the listwise distribution (if one stale response has s_{θ,k} ≫ 0, it can dominate P_θ). Potential mitigations to test: (a) stale-sample filtering — discard replay samples where |s_{θ,k}| exceeds a threshold, (b) KL-regularized replay — add a penalty for large policy deviation from the behavior policy that generated each replay sample, (c) prioritized replay — sample replay transitions proportionally to their importance ratio magnitude to focus on informative but not extreme samples. A strong result would show that replay improves sample efficiency by 2–3× (fewer fresh rollouts needed for the same performance), which matters practically because generation is the dominant cost in RLVR training.

Step-level LPO with process reward models for multi-step reasoning. The current framework operates at sequence level — one target and one projection per complete response. The paper sketches step-level extension (Appendix C.2) but does not implement it. A concrete experiment: for each prompt, generate K complete responses from the base model. For each response, identify reasoning steps (e.g., by parsing "Step 1:", "Step 2:", or by detecting natural paragraph breaks). At each step boundary, the partial response up to that step forms a "state," and the K different completions from that state onward (taken from the K full responses) form a local response simplex over continuations. To construct the target at each step, estimate the expected final reward of each continuation — either by (a) Monte Carlo rollouts from that step to completion (as in the PRM training of the reference paper), or (b) using a pre-trained process reward model to score the partial response directly. Apply LPO's target construction and projection at each step independently, accumulating gradients across all steps in all K responses. The hypothesis is that step-level optimization provides a denser training signal than sequence-level optimization, potentially accelerating learning and improving performance on multi-step reasoning problems where intermediate errors cascade. Compare against sequence-level LPO and standard PG baselines on MATH and PRIME Code (which naturally have multi-step structure). Measure: (a) training sample efficiency, (b) final accuracy, (c) whether step-level optimization reduces certain error types (e.g., arithmetic mistakes in intermediate steps) more than sequence-level optimization. The practical challenge is computational cost — K complete responses already exist from the standard rollout, but step-level target construction may require additional rollouts or PRM inference at each step. Quantifying the cost-accuracy tradeoff is essential.

Is the PG approximation error the actual bottleneck, or is divergence choice more important? The paper's central claim is that exact projection provides gains over first-order PG approximation, and the off-policy error analysis (Appendix B.2) provides a theoretical mechanism. But the empirical gains (1–2 percentage points on Pass@1, larger on Pass@k) could also be explained by forward KL's mode-covering property being genuinely better for Pass@k, independent of approximation quality. A dissociation experiment: train GRPO with an artificially improved approximation by (a) using only a single gradient step per batch (δ̄ ≈ 0, so the PG approximation is exact by Proposition 1), and (b) using a very small learning rate to stay near the on-policy point. Compare this "near-exact reverse KL PG" against LPO with reverse KL (which performs exact reverse KL projection but can drift off-policy due to multiple inner epochs). If the PG baseline matches LPO-rev under near-on-policy conditions, the approximation error is the bottleneck. If LPO-rev still outperforms near-on-policy PG, something else — perhaps the implicit entropy bonus in the exact reverse KL projection (Appendix C.7) — is responsible. Conversely, compare LPO-fwd against the best possible PG baseline (any advantage formula, any clipping schedule) on Pass@k. If no PG configuration approaches LPO-fwd's Pass@k performance, divergence choice (forward vs. reverse KL) is the dominant factor, and the approximation error is secondary. This experiment would disambiguate the two mechanisms the paper proposes (exact projection and divergence choice) and guide future work toward the higher-impact direction.

Scaling LPO to 70B+ models and frontier reasoning tasks. The paper's experiments span 1.7B to 14B parameters, with the largest-scale validation (Appendix E.1, Figure 8) using Qwen3-14B-Base on the Polaris dataset. The current frontier for open-source reasoning models is 32B–72B parameters (e.g., Qwen3-32B, DeepSeek-R1-70B). A direct scaling experiment: train Qwen3-32B-Base or DeepSeek-R1-Distill-Llama-70B on MATH or a larger reasoning dataset (e.g., the full Polaris-53k or an internal competition-math corpus) with LPO-fwd at the GRPO temperature, and compare against GRPO at the same scale. Beyond measuring raw accuracy, track whether LPO's gradient stability properties (Figure 5, middle row) become more or less important at larger scales — larger models have more parameters, potentially making gradient noise more damaging, which would amplify LPO's advantage; alternatively, larger models may have smoother loss landscapes, making the PG approximation more accurate and reducing the gap. Also test whether LPO's entropy preservation (Figure 5, top row) prevents the "entropy collapse" that often plagues large-scale RLVR training, where the policy degenerates to generating near-deterministic outputs that fail to explore. A strong result at 32B+ scale would validate LPO's practical relevance for frontier model training, while a null result (LPO matching but not exceeding GRPO at scale) would suggest the gains are bounded to smaller models or specific to the Qwen architecture.

Practical Applications and Downstream Use Cases

Training reasoning models with fewer hyperparameters and less tuning. The current practice for training reasoning models with RLVR involves substantial hyperparameter tuning across the advantage normalization scheme (GRPO vs. Dr.GRPO vs. MaxRL vs. custom), clipping parameters, dynamic sampling rules, and KL penalty coefficients. The paper's finding that LPO provides consistent gains across all three temperature designs (Figures 3, 4) and across four model families (Figure 11) suggests that adopting LPO-fwd as a default projection mechanism reduces the hyperparameter search space from "find the best advantage formula plus all associated clipping and stabilization heuristics" to "choose a temperature τ" — which can be inherited from the practitioner's preferred existing scheme without loss of performance. In the Countdown experiments (Figure 7), LPO-fwd with K=8 achieves ~58% Pass@1 vs. GRPO's ~55%, and on Pass@64 the gap is 83% vs. 79%. For a team training a reasoning model where Pass@k is the deployment metric (common for self-consistency-based evaluation), the forward KL variant provides a robust default that outperforms tuned PG baselines without additional tuning burden. The practical workflow becomes: (1) implement LPO-fwd as a drop-in replacement for the PG objective in the existing verl/OpenRLHF training loop, (2) set τ to the baseline's implicit value, (3) train with the same hyperparameters, expecting at minimum matched performance with better stability and diversity. The paper's claim of "no additional computational cost" (Section 4.3) means this is a zero-overhead improvement — no new models to load, no extra forward passes, no increased memory footprint.

Multi-domain RLVR training pipelines where diversity preservation is critical. The paper's strongest and most consistent result is LPOfwd's dominance on Pass@k (15/15 scenarios in Figure 4, often with 3–5 percentage point gaps). This is directly relevant to training pipelines where the trained model will be evaluated via self-consistency (generating k candidate responses and selecting by majority vote or verifier) or where the model needs to maintain multiple valid reasoning strategies for robustness. The mode-coverage property formalized in Corollary 2 provides a principled guarantee against mode collapse that PG methods lack — and the empirical entropy curves (Figure 5, top row) confirm this translates to sustained response diversity during training. For a production system training a code generation model on the PRIME dataset (where the paper shows LPOfwd reaching ~42% Pass@1 vs. GRPO's ~40%, and ~51% Pass@8 vs. ~48%), the diversity preservation means the trained model can generate multiple distinct correct solutions to the same problem, which is valuable for downstream applications like test case generation, code review, or ensemble methods. The practical benefit is that the model does not need to be trained with explicit diversity-promoting auxiliary losses — the forward KL projection provides diversity preservation as an emergent property of the optimization geometry.

Sample-efficient fine-tuning for specialized reasoning domains with limited training data. The group-size experiment (Figure 7) shows that LPO's advantage over GRPO is most pronounced at small K (2, 4, 8), with LPOfwd achieving ~62% Pass@1 at K=2 vs. GRPO's ~55%. This translates directly to settings where generating many responses per prompt is expensive or where the training dataset is small (so efficient use of limited samples matters). For example, fine-tuning a medical reasoning model where expert-verified training problems are scarce (hundreds, not thousands): with K=4 per prompt, LPO-fwd achieves better performance than GRPO with K=8 (Figure 7 shows LPO-fwd at K=4 reaching ~60% vs. GRPO at K=8 reaching ~55% on Countdown). The practical implication is that LPO can achieve competitive performance with half the generation budget, reducing training cost and enabling fine-tuning in domains where large-scale data generation is impractical. For the Geometry3k multimodal task (2.1k training problems, Figure 3, bottom row), LPO-fwd reaches ~44.5% Pass@1 vs. GRPO's ~42% — a meaningful gain on a small dataset where sample efficiency is paramount.

Training on outcome rewards with better credit assignment through implicit entropy regularization. The paper's Proposition 3 and Appendix C.7 show that the reverse KL projection objective decomposes as -∑k P{θ,k} φ_k + H(P_θ), where H(P_θ) is the entropy of the listwise distribution. This means LPO-rev naturally includes an entropy bonus that encourages the listwise distribution to remain spread out, without needing an explicit entropy regularization hyperparameter (which PG methods often require, adding another tuning dimension). Furthermore, any entropy bonus added to PG is equivalent to increasing τ in the LPO framework (Appendix C.7: "Entropy regularization as target mixing"), making it redundant when τ is already controllable. For practitioners, this means LPO-rev training automatically balances reward maximization against diversity preservation through the single interpretable parameter τ, rather than requiring separate tuning of the reward scale, advantage normalization, and entropy coefficient. The practical workflow: set τ to the desired target sharpness (or inherit from a baseline), and the optimization naturally maintains appropriate entropy without additional intervention. The empirical gradient norm curves (Figure 5, middle row) and entropy curves (Figure 5, top row) confirm this translates to more stable, less collapse-prone training than PG baselines.

When to Prefer This Method

The paper does not explicitly position LPO against named alternatives in a structured decision framework — it presents LPO as a general-purpose improvement over group-based PG methods (GRPO, Dr.GRPO, MaxRL) in the RLVR setting, rather than as a method that should be preferred conditionally over other approaches. The experiments use a paired evaluation protocol where LPO is compared against PG baselines that use identical temperature, and the consistent finding is that LPO matches or exceeds baseline performance while providing better stability and diversity. The paper does not identify conditions under which PG methods would be preferable to LPO, nor does it compare LPO against non-group-based RL methods (e.g., PPO with a value network, REINFORCE with a learned baseline) or against supervised fine-tuning baselines. The guidance from the paper is therefore straightforward: in any setting where a group-based PG method (GRPO, Dr.GRPO, MaxRL) would be used for RLVR, LPO with the same temperature τ can be substituted with the expectation of equal or better performance, lower gradient variance, and higher response diversity, at no additional computational cost. The forward KL variant is preferred when Pass@k evaluation is the deployment metric (due to mode-covering diversity preservation), while the reverse KL variant is preferred when theoretical continuity with existing PG methods is desired or when an explicit entropy bonus is valued. The paper does not provide evidence for preferring LPO over PG in settings with K=1 (single response per prompt), which would require virtual group construction or a fundamentally different formulation (Appendix C.2, "Beyond group-based sampling").