REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization

ArXiv: 2501.03262

🎯 Pitch

Local advantage normalization causes catastrophic overfitting: GRPO scores 95.0% on training data but crumbles to 0.0% on out-of-distribution test sets, while REINFORCE++ achieves robust generalization using global normalization across the entire batch, completely eliminating the bias inherent in prompt-level statistics.

---

1. Executive Summary

This paper introduces REINFORCE++, a critic-free policy optimization framework for RLHF that replaces the prompt-level (local) advantage normalization used in prior methods like GRPO and RLOO with Global Advantage Normalization (normalizing advantages across the entire global training batch rather than within small prompt-specific groups). The authors prove that local normalization is a theoretically biased estimator and propose two variants—REINFORCE++ (k ≥ 1) for general-domain RLHF and reasoning, and REINFORCE++_w/_Baseline (k > 1) for complex agentic tasks—that substitute global batch statistics for local group statistics and, in the latter case, replace GRPO's unstable k3 KL estimator with a theoretically sound k2 estimator. On Chat-Arena-Hard using Llama-3-8B-SFT, single-sample REINFORCE++ achieves comparable reward scores to group-sampling GRPO while producing shorter responses and exhibiting lower KL divergence; on complex tool-use benchmarks with Qwen 2.5 Base 7B, REINFORCE++_w/_Baseline achieves 24.10 average accuracy, outperforming both GRPO (22.58) and full-critic PPO (21.85). The paper establishes that global normalization prevents catastrophic overfitting on low-data reasoning regimes, with REINFORCE++ achieving 2.5 Pass@1 and 40.0 Pass@16 on out-of-distribution AIME-25 evaluation after training on only 30 AIME-24 problems, while GRPO scores 95.0 on the training set but collapses to 0.0 Pass@1 on the test set—demonstrating that local normalization encourages prompt-specific exploitation rather than robust generalization.

2. Context and Motivation

The Core Problem: Critic-based RLHF Is Computationally Prohibitive

The central problem this paper addresses is deceptively straightforward: the dominant RLHF algorithm, Proximal Policy Optimization (PPO), is too expensive to deploy at scale for LLM alignment. PPO (Schulman et al., 2017) employs an "Actor-Critic" architecture where, alongside the policy model being trained, a separate critic network (also called a value network) must be trained to estimate the expected future reward from any given state. This critic is essential for computing the advantage function — a measure of how much better a particular action is compared to the average action in a given situation, which guides the policy gradient update.

The computational burden is severe. The critic network is typically of comparable size to the policy model itself — for a 7B-parameter LLM, the critic adds roughly another 7B parameters that must be stored in memory, forward-passed during training, and backward-passed during optimization. This doubles the memory footprint and increases the computational cost of each training step significantly. The paper notes that this "makes training very expensive and limits large model alignment in small-scale clusters" (Section 1). In practical terms, an organization that could train a 7B model with critic-free RL might need twice the GPU memory to run PPO, effectively halving the model size they could train on affordable hardware.

This financial and hardware barrier has real consequences. RLHF is not an academic curiosity — it is the technique that makes models like ChatGPT, Claude, and Gemini behave helpfully rather than generating toxic, unhelpful, or dangerous outputs. If PPO's cost prevents smaller teams from performing alignment, the field consolidates around a few well-resourced players. Making RLHF accessible to smaller compute budgets is therefore both a practical deployment concern and a matter of scientific democratization.

The Emergence of Critic-Free Methods — and Their Own Problems

In response to PPO's cost, a family of critic-free algorithms has emerged over the past few years. These methods eliminate the separate critic network entirely and instead estimate advantages directly from the rewards of multiple sampled responses:

• ReMax (Li et al., 2023): Samples one response stochastically and one greedily, using the difference between their rewards as the advantage. This is the simplest approach, requiring only two samples per prompt.

• RLOO (Ahmadian et al., 2024): Samples $k$ responses per prompt and uses the mean reward of all other responses as a baseline for each individual response: $A_{q, o^{(i)}} = r(o^{(i)}) - \frac{1}{k-1} \sum_{j \neq i} r(o^{(j)})$.

• GRPO (Group Relative Policy Optimization; Shao et al., 2024): Samples $k$ responses per prompt and normalizes advantages using the mean and standard deviation of that small local group: $A_{q, o^{(i)}} = \frac{r(o^{(i)}) - \text{mean}(\{r(o^{(j)})\}_{j=1}^k)}{\text{std}(\{r(o^{(j)})\}_{j=1}^k) + \epsilon}$

These methods substantially reduce computational cost by discarding the critic. However, the paper argues that this cost reduction comes at a hidden price: these methods rely on prompt-level (local) advantage normalization, which introduces a distinct set of problems that prior work has not adequately addressed.

Three Critical Flaws in Local Normalization

The paper identifies three specific, interrelated problems with local normalization — each of which represents a gap in our understanding of critic-free RLHF that the paper aims to fill.

Flaw 1: Theoretical Bias (The Mathematical Problem)

The paper provides, in Appendix Section A, what it claims is the first formal proof that the GRPO-style advantage estimator is mathematically biased. Understanding why requires carefully examining the structure of the estimator:

$A_i = \frac{\epsilon_i - \bar{\epsilon}}{D}$

where $\epsilon_i \sim \mathcal{N}(0, \sigma^2)$ are the centered rewards (true advantages), $\bar{\epsilon}$ is the local group mean, and $D = \sqrt{\frac{1}{N}\sum_{j=1}^N (\epsilon_j - \bar{\epsilon})^2}$ is the local group standard deviation.

The bias arises from a dependency between numerator and denominator. The proof proceeds in four steps:

1. Numerator bias: $E[\epsilon_i - \bar{\epsilon} \mid \epsilon_i] = (1 - \frac{1}{N})\epsilon_i$ — the conditional expectation of the numerator is proportional to the true advantage, scaled by a factor that approaches 1 as group size $N$ grows.

2. Denominator depends on $\epsilon_i$: The conditional expectation of $D^2$ given $\epsilon_i$ is $E[D^2 \mid \epsilon_i] = \alpha + \beta\epsilon_i^2$, where $\alpha = \frac{(N-1)^2}{N^2}\sigma^2$ and $\beta = \frac{N-1}{N^2}$. Since $\beta > 0$, the expected denominator depends on the square of the true advantage — larger-magnitude advantages lead to larger expected standard deviations, which deflates the normalized output.

3. The reciprocal is not constant: Through a Taylor expansion of $f(x) = 1/\sqrt{x}$ around $\mu(\epsilon_i) = E[D^2 \mid \epsilon_i]$, the paper shows that $g(\epsilon_i) = E[1/D \mid \epsilon_i]$ is a non-constant function of $\epsilon_i^2$.

4. The bias conclusion: If the estimator were unbiased, we would require $(1 - 1/N) \cdot g(\epsilon_i) = 1$ for all $\epsilon_i$, implying $g(\epsilon_i)$ is constant. Since $g(\epsilon_i)$ depends on $\epsilon_i^2$, this cannot hold. Therefore $E[A_i \mid \epsilon_i] \neq \epsilon_i$ for any finite $N \geq 2$.

The practical implication is subtle but important: GRPO systematically under-estimates the magnitude of large advantages and over-estimates small ones because large $\epsilon_i$ values inflate the denominator, pulling the normalized output toward zero. This is not just theoretical pedantry — it means the policy receives systematically distorted gradient signals, potentially learning suboptimal behavior.

Flaw 2: Practical Instability (The Engineering Problem)

The standard deviation term in local normalization creates a vulnerability that manifests in practice. GRPO and RLOO typically use small group sizes — $k = 4$ or $k = 8$ — because generating more responses per prompt is expensive (each response requires a full forward pass through a multi-billion-parameter model).

With such small groups, a pernicious edge case emerges: if all $k$ responses to a prompt happen to receive similar rewards, the local standard deviation approaches zero. This might happen because the prompt is genuinely easy (all responses are good), genuinely hard (all responses are equally bad), or the reward model is insensitive to the variations present. When $\text{std}(\cdot)$ becomes tiny, the denominator in Eq. (3) from the original paper approaches $\epsilon$ (a small constant added for numerical stability), and the advantage estimate becomes:

$A \approx \frac{r - \text{mean}(r)}{\epsilon}$

This fraction can be arbitrarily large, causing the advantage to explode. The policy then takes an enormous gradient step, potentially destroying learned behaviors. The paper characterizes this as causing "high variance and unstable training" (Section 2.2), but the underlying mechanism is more specific: it is a catastrophic failure mode of division by near-zero that becomes more likely as $(k, \text{reward variance})$ decreases.

This is not a hypothetical concern. In complex reasoning tasks where rewards are often binary (0/1 for correct/incorrect), it is entirely common for all $k = 4$ samples to receive the same reward. The local standard deviation is then either exactly zero (all same) or driven by a single different sample, creating extreme variance in the advantage estimate depending on whether one response happened to differ.

Flaw 3: Task Overfitting (The Generalization Problem)

The third flaw is perhaps the most consequential for real-world performance. Local normalization creates a relative evaluation metric: a response's advantage depends not on how good it is in absolute terms, but on how good it is compared to other responses to the same prompt.

The paper states this precisely: "The policy is rewarded for being 'better than other samples from the same prompt,' not for being 'globally good'" (Section 2.2). This incentivizes a form of within-group competition rather than absolute improvement.

Consider two prompts:

• Prompt A (simple): The model can easily generate responses of varying quality — some nearly perfect, some mediocre. The standard deviation of rewards is large, making it easy for the best response to earn a high normalized advantage and strongly influence the policy update.
• Prompt B (complex): All responses cluster around a similarly low reward. The standard deviation is small, so even the relatively-best response earns a modest normalized advantage. The policy receives weak gradient signals for improving on this prompt, even though it desperately needs improvement.

The result is that locally-normalized training overfits to easy prompts where reward variance is naturally high, while underfitting on hard prompts where all responses are similarly mediocre. The policy learns to "win" within its local group on simple problems rather than developing the capabilities needed to succeed on difficult ones.

This is not just a training artifact — it manifests catastrophically in out-of-distribution generalization, as the paper demonstrates in Section 4.2.1. When GRPO trains on only 30 AIME-24 math problems, it achieves 95.0% accuracy on the training set but 0.0 Pass@1 on the held-out AIME-25 set. The local normalization signal has driven the model to memorize solution patterns specific to those 30 training problems (where it can "beat" other samples from the same prompt) rather than learning generalizable reasoning skills. The global normalization variant (REINFORCE++) achieves only 71.0% on the training set but generalizes to 2.5 Pass@1 and 40.0 Pass@16 on the test set.

A Gap in the Literature: No Principled Fix for Local Normalization

When the paper was written, the RLHF literature had recognized that critic-free methods were cheaper than PPO, but had not systematically analyzed the statistical properties of local advantage normalization. Several prior works had identified practical issues:

• GRPO's own paper (Shao et al., 2024) added an $\epsilon$ term to the denominator to prevent division by zero, acknowledging numerical instability without analyzing its root cause.
• VAPO (Yue et al., 2025) discussed the benefits of group sampling for complex reasoning but did not propose an alternative normalization strategy.
• Various implementations used reward clipping, learning rate annealing, and other heuristics to manage instability — treating symptoms rather than causes.

What was missing was (1) a theoretical understanding of why local normalization is problematic, and (2) a principled, drop-in replacement that preserves the computational benefits of critic-free training while eliminating the bias, instability, and overfitting. The paper positions itself as filling exactly this gap.

Known but Unresolved: The KL Estimator Problem

A secondary but important gap the paper identifies concerns the KL divergence penalty used to prevent the policy from drifting too far from the reference model during RL training. This is a standard component of RLHF — without it, the policy can "reward hack" by producing degenerate outputs that score well under the reward model but are no longer coherent language.

Critic-free methods that sample multiple responses per prompt ($k > 1$) typically add the KL penalty as a separate loss term rather than incorporating it into the reward (as PPO does). The paper identifies that GRPO uses the k3 estimator:

$k_3(y) = \delta(y) - 1 - \log \delta(y)$

where $\delta(y) = \pi_{\text{ref}}(y|\cdot) / \pi_\theta(y|\cdot)$. Through analysis in Appendix B.1, the paper shows this estimator has two critical problems:

1. It estimates the wrong KL direction. The k3 estimator approximates the Forward KL $D_{KL}(\pi_{\text{ref}} \| \pi_\theta)$, but since training samples come from $\pi_\theta$, the correct objective is the Reverse KL $D_{KL}(\pi_\theta \| \pi_{\text{ref}})$. Using k3 creates a distribution mismatch: the estimator requires samples from $\pi_{\text{ref}}$ (which we don't have) but is computed using samples from $\pi_\theta$.

2. It has extremely high variance. When $\pi_\theta(y)$ becomes small for a sample where $\pi_{\text{ref}}(y)$ is moderate, the importance ratio $\pi_{\text{ref}}(y)/\pi_\theta(y)$ explodes, leading the paper to characterize k3 as having "infinite variance." This is why GRPO-based implementations frequently reset the reference model — to prevent the policy from diverging too far and the k3 estimator from becoming numerically unstable (with $\exp(\log p - \log q)$ prone to overflow).

The paper's analysis shows that the k2 estimator,

$k_2(y) = \frac{1}{2}\left(\log \frac{\pi_\theta(y|\cdot)}{\pi_{\text{ref}}(y|\cdot)}\right)^2$

provides a gradient that exactly matches the theoretical Reverse KL gradient, making it the "theoretically correct and stable choice" (Appendix B.1). This is not a new mathematical discovery — the relationship between k2 and Reverse KL is known in the literature (Liu et al., 2025a) — but the paper contributes by showing that GRPO's choice of k3 is actively harmful and that switching to k2 is a necessary complement to global normalization.

How This Paper Positions Itself

The paper's framing is distinctive: it does not claim credit for inventing global normalization per se — the technique of normalizing advantages across a batch rather than within groups is known from standard reinforcement learning practice (Andrychowicz et al., 2020). What the paper contributes is the explicit identification of local normalization as the root cause of three distinct failure modes in critic-free RLHF (bias, instability, overfitting), the mathematical proof of the bias, and the design of complete algorithm variants that apply global normalization appropriately for different training regimes.

The positioning is that REINFORCE++ is not a fundamentally new algorithm but rather a correction to the existing critic-free family. It fixes what was broken (local normalization) while preserving what was valuable (critic-free efficiency, group sampling where beneficial). The two variants reflect a pragmatic acknowledgment that different training regimes have different needs:

• General-domain RLHF benefits from maximum prompt diversity and minimal computation, making $k=1$ sampling with pure global normalization ideal.
• Complex reasoning tasks benefit from group sampling ($k > 1$) because reward signals are sparse (often binary) and having multiple attempts per prompt helps distinguish capability from luck. But the group sampling should be used for reward reshaping (subtracting the group mean) rather than for computing the standard deviation — the standard deviation should always come from the global batch.

This is a more nuanced position than "global normalization is always better" — it is "local normalization's bias and instability come specifically from using local standard deviation; local mean subtraction has its place for reward reshaping."

The paper also indirectly positions itself in the broader debate about whether RLHF is even necessary. Direct Preference Optimization (DPO; Rafailov et al., 2023) and its variants offer a simpler alternative by directly optimizing from preference pairs without an explicit reward model or RL loop. By making RLHF more stable and accessible through critic-free training, the paper implicitly argues that RL-based alignment has advantages worth preserving — it points to state-of-the-art systems like ChatGPT/GPT-4, Claude, and Gemini all using RL algorithms as evidence that "non-RL alternatives" have not fully displaced RLHF in practice.

Why This Matters Beyond the Technical Details

The practical stakes of this work extend beyond the specific algorithms. When GRPO overfits on 30 training problems to 95% accuracy but gets 0% on a held-out set, this is not just a benchmark failure — it represents a fundamental risk in RLHF deployment. If alignment training causes models to memorize reward-pleasing patterns rather than learning robust behaviors, the resulting models may appear well-aligned during evaluation but fail catastrophically on novel inputs. This is the alignment analog of the overfitting problem that has plagued supervised learning for decades, and the paper's demonstration that global normalization substantially mitigates it suggests a path toward more genuinely robust alignment.

Similarly, the instability caused by division-by-near-zero in local normalization is not merely an inconvenience. In production training pipelines where thousands of GPU-hours are at stake, a single training run destabilized by an unlucky batch can waste enormous resources. The paper's global normalization approach eliminates this failure mode entirely, since a global batch of 1024 samples is effectively guaranteed to have non-zero standard deviation in all realistic scenarios.

3. Technical Approach

3.1 Reader Orientation

REINFORCE++ is a training recipe for aligning language models using reinforcement learning without the expensive critic network that PPO requires. It solves the problem that existing critic-free methods (GRPO, RLOO) produce biased, unstable advantage estimates by replacing their prompt-level normalization—which computes means and standard deviations from tiny groups of 4–8 responses to the same question—with Global Advantage Normalization, where these statistics are computed across the entire training batch of hundreds or thousands of samples, making the estimator effectively unbiased and preventing the division-by-near-zero failures and prompt-overfitting that plague local methods.

3.2 Big-Picture Architecture (Diagram in Words)

The REINFORCE++ system has five major components, forming a pipeline that flows from prompts through policy sampling to advantage estimation to policy updates:

1. Policy Model ($\pi_\theta$) — the LLM being trained. It takes prompts as input and generates responses (token sequences). Initially copied from a supervised-fine-tuned or base model.

2. Reference Model ($\pi_{\text{ref}}$) — a frozen copy of the initial policy. Used only to compute KL divergence penalties that prevent the policy from drifting too far from its starting point during training.

3. Reward Model ($\mathcal{R}$) — scores each generated response with a scalar reward indicating quality. For general RLHF this is a learned Bradley-Terry model trained on human preferences; for reasoning tasks it can be rule-based (e.g., 1 if the math answer is correct, 0 otherwise).

4. Global Batch Normalizer — the core innovation. Collects all rewards (or raw advantages) from every prompt in the current training batch, computes their mean and standard deviation across the entire batch, and normalizes each individual advantage using these global statistics rather than per-prompt statistics.

5. KL Penalty Module — adds a regularization term to the loss that prevents the policy model's output distribution from diverging too far from the reference model. In REINFORCE++, this uses the k1 estimator embedded in the reward; in REINFORCE++_w/_Baseline, it uses the k2 estimator as a separate loss term.

Information flows as follows: a batch of prompts is sampled → the policy model generates one response per prompt (REINFORCE++, k=1) or multiple responses per prompt (REINFORCE++_w/_Baseline, k>1) → the reward model scores each response → raw advantages are computed by incorporating KL penalties → all raw advantages from the entire batch are collected and global mean/std are computed → each advantage is normalized using these global statistics → the normalized advantages drive a PPO-style clipped policy gradient update → the KL penalty loss is added (for the Baseline variant) → model parameters are updated.

3.3 Roadmap for the Deep Dive

• First, the global normalization mechanism itself — how means and standard deviations are computed across the full batch, the mathematical form of the normalized advantage, and why large batch sizes make this estimator effectively unbiased. This is the central claim of the paper and everything else depends on it.

• Second, the REINFORCE++ variant for $k=1$ (Algorithm 1) — how single-sample-per-prompt training works, how the k1 KL penalty is embedded in the reward, and why this configuration maximizes prompt diversity for general-domain RLHF.

• Third, the REINFORCE++_w/_Baseline variant for $k>1$ — how group sampling is retained but the local standard deviation is replaced with global batch statistics, the two-step process of group-mean subtraction followed by global normalization, and the k2 KL estimator's theoretical justification.

• Fourth, the KL penalty design choice — the mathematical analysis showing why the k1, k2, and k3 estimators differ, why GRPO's k3 estimates the wrong KL direction with unstable variance, and why k2 is the correct choice for Reverse KL when sampling from $\pi_\theta$.

• Fifth, the implementation details — token-level advantage assignment, batch construction, mini-batch updates, and reward clipping/scaling, since these practical engineering choices affect reproducibility and stability.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a methodology paper whose core idea is that global batch normalization of advantages eliminates the theoretical bias, practical instability, and task overfitting caused by prompt-level local normalization in critic-free RLHF, while a properly chosen KL estimator prevents the policy from reward-hacking, and that these two changes together produce a critic-free training framework that matches or exceeds PPO's performance at substantially lower computational cost.

---

Global Advantage Normalization: The Core Mechanism

Global Advantage Normalization is a statistical technique that replaces per-prompt normalization with per-batch normalization during advantage estimation. Understanding why this matters requires understanding what advantage normalization does and how the choice of normalization group affects the resulting estimates.

What advantage normalization is and why it's needed. In policy gradient methods, the raw advantage $A_{q, o_t}$ tells the model how much better taking token $o_t$ in context $q, o_{<t}$ is compared to some baseline. However, raw advantages can have arbitrary scale: a reward model might output values in $[0, 1]$, $[-1, 1]$, or $[0, 100]$. If these raw values are used directly in the PPO objective, the effective learning rate varies dramatically across training runs and reward models. Normalization—subtracting the mean and dividing by the standard deviation—standardizes advantages to approximately zero mean and unit variance, making the optimization landscape more consistent and the learning rate more interpretable.

The key question is: over which population do we compute the mean and standard deviation?

Local normalization (GRPO's approach). GRPO computes these statistics over the $k$ responses generated for a single prompt (typically $k = 4$ or $k = 8$). The formula from the original paper's Equation (3) is:

$A_{q, o^{(i)}_t} = \frac{r(o^{(i)}) - \text{mean}(\{r(o^{(j)})\}_{j=1}^k)}{\text{std}(\{r(o^{(j)})\}_{j=1}^k) + \epsilon}$

where $r(o^{(i)})$ is the reward for the $i$-th response to prompt $q$, $\text{mean}(\cdot)$ is the average reward across all $k$ responses to that same prompt, $\text{std}(\cdot)$ is the standard deviation of those $k$ rewards, and $\epsilon$ is a small constant (added to prevent division by zero).

What this computes: for each response $i$ to prompt $q$, subtract the average reward of all responses to the same prompt, then divide by the standard deviation of those same rewards. The output is a z-score indicating how many within-group standard deviations this response is above or below the group mean.

Why this form was chosen historically: it provides automatic reward scaling per prompt—if one prompt consistently gets high rewards from the reward model and another gets low rewards, local normalization centers each independently, preventing the model from ignoring low-reward prompts. It also requires no coordination across prompts, making it simple to implement in distributed training where different prompts might be processed on different devices.

Global normalization (REINFORCE++'s approach). The paper's central formula, Equation (5), is:

$A^{\text{norm}}_{q, o_t} = \frac{A_{q, o_t} - \text{mean}(A \mid A \in \mathcal{D}_{\text{batch}})}{\text{std}(A \mid A \in \mathcal{D}_{\text{batch}}) + \epsilon}$

where $A_{q, o_t}$ is the raw (un-normalized) advantage for token $o_t$ in response to prompt $q$, $\mathcal{D}_{\text{batch}}$ is the set of all raw advantages from all prompts in the current training batch, $\text{mean}(A \mid A \in \mathcal{D}_{\text{batch}})$ is the average raw advantage across the entire batch (potentially hundreds or thousands of samples from many different prompts), and $\text{std}(A \mid A \in \mathcal{D}_{\text{batch}})$ is the standard deviation across the same global population.

What this computes: for every individual advantage in the batch, regardless of which prompt it came from, subtract the global mean advantage and divide by the global standard deviation. All samples are normalized using the same two numbers (the global mean and global standard deviation).

Why this form solves the local normalization problems:

• Theoretical bias is eliminated (as $N \to \infty$). As proven in Appendix A.2, when the normalization group size $N \to \infty$, the sample mean $\mu_b$ and sample standard deviation $\sigma_b$ converge to their true population constants $\mu$ and $\sigma$. At this limit, the denominator $D$ in the bias analysis becomes independent of any individual $\epsilon_i$, and the conditional expectation $E[A_i \mid \epsilon_i] = \epsilon_i / \sigma$ — an unbiased estimator (up to the constant scaling factor $1/\sigma$, which is absorbed into the learning rate). In practice, batch sizes of 1024 are large enough that the residual bias is negligible compared to the inherent noise in policy gradient estimation.

• Practical instability is prevented. With 1024 samples in the global batch, the standard deviation is effectively guaranteed to be non-zero in all realistic scenarios. Even if every response to a particular prompt receives identical rewards, the global standard deviation is driven by the variance across all prompts. The edge case where $\text{std} \to 0$ and advantages explode cannot occur because it would require every sample from every prompt in the entire batch to have exactly the same reward, which is vanishingly unlikely with diverse prompts.

• Task overfitting is mitigated. Because every sample is normalized against the same global baseline, the policy update is driven by absolute performance relative to the global distribution, not by relative performance within a prompt's local group. A response that is the best among four mediocre attempts to a hard problem will receive a modest normalized advantage (it's near the global mean), rather than an inflated one (it's far above the local mean). Conversely, a response that is slightly better than three excellent attempts to an easy problem will receive a normalized advantage that reflects its genuinely high quality relative to the global distribution, not a suppressed one due to a small local standard deviation.

The role of batch size. The paper explicitly ties the effectiveness of global normalization to batch size: "As the global batch size grows (e.g., $N = 1024$), the batch mean and standard deviation converge to constants, resulting in an effectively unbiased (bias vanishes as $N \to \infty$) and low-variance advantage estimator" (Section 3.4). This is the key insight that justifies the approach—global normalization with a small batch (e.g., 32) would still suffer from some of the same finite-sample bias as local normalization, but at the batch sizes typical in LLM training (hundreds to thousands), the convergence is sufficient.

What "effectively unbiased" means operationally. The paper does not claim the estimator is exactly unbiased at finite $N$. Rather, it claims the bias becomes negligible compared to other sources of noise in the training process. Since policy gradient estimates are inherently high-variance (they are Monte Carlo estimates over discrete token sequences), a small residual normalization bias is dominated by sampling noise. The practical test is whether training is stable and generalizes, not whether the estimator passes a formal unbiasedness test at specific $N$.

The normalization timing. The global mean and standard deviation are computed fresh for each training batch from that batch's samples. They are not running averages or learned parameters—they are batch statistics computed from the raw advantages generated during the current forward pass. This means the normalization adapts naturally as the policy improves and reward distributions shift, without requiring any persistent state across batches.

Comparison to batch normalization in deep learning. The technique is analogous to batch normalization (Ioffe & Szegedy, 2015) but applied to scalar advantages rather than layer activations. The same principle applies: using batch statistics during training provides stable, normalized signals that accelerate learning, even though the statistics themselves are stochastic estimates of the true population moments. Unlike batch normalization, there is no need for running averages or learned affine parameters, because the normalized values are intermediate training signals rather than network activations that must be consistent between training and inference.

---

REINFORCE++ for General-Domain RLHF ($k = 1$)

The primary REINFORCE++ variant uses $k = 1$ (a single response sampled per prompt) and is designed for general-domain RLHF where prompt diversity and computational efficiency are the priorities.

Motivation for $k = 1$. When training on diverse instruction-following prompts, generating multiple responses per prompt is wasteful: the model sees fewer distinct prompts per training step (since the generation budget is split among $k$ responses per prompt rather than $k$ distinct prompts), and the additional responses provide diminishing returns because general-domain reward models produce informative scalar signals from single responses. The paper's experiments (Table 1) confirm this empirically—single-sample REINFORCE++ achieves a score of 46.7 on Chat-Arena-Hard, statistically tied with GRPO's 46.8 using $k = 4$, while generating shorter responses (832 vs. 860 tokens) and maintaining lower KL divergence.

Raw advantage computation. The raw (pre-normalization) advantage in REINFORCE++ is computed as stated in Equation (4):

$A_{q, o_t} = r(o_{1:T}, q) - \beta \cdot \sum_{i=t}^T \text{KL}(i)$

where $r(o_{1:T}, q)$ is the scalar reward assigned by the reward model to the complete response $o_{1:T}$ for prompt $q$, $\beta$ is a hyperparameter controlling the strength of KL regularization, and $\text{KL}(t) = \log\left(\frac{\pi_{\theta_{\text{old}}}(o_t \mid q, o_{<t})}{\pi_{\text{ref}}(o_t \mid q, o_{<t})}\right)$ is the per-token KL divergence between the old policy (frozen at the start of the batch) and the reference model.

What this computes: for each token position $t$, sum the per-token KL divergences from that position to the end of the sequence, multiply by $\beta$, and subtract from the final reward. This means tokens earlier in the sequence have more KL penalty subtracted (they accumulate penalties from more future tokens), while the final token has only its own KL penalty subtracted. The result is a token-level advantage that rewards high-reward responses while penalizing deviation from the reference model—but with earlier tokens penalized more heavily because they commit the model to a trajectory.

Why this form (k1 KL in reward): embedding the KL penalty directly into the reward (rather than as a separate loss term) means the PPO objective automatically balances reward maximization against KL minimization. The clipped surrogate objective optimizes token-level probability ratios, and subtracting $\beta \cdot \sum \text{KL}(i)$ from the reward ensures that increasing the probability of a token that deviates from the reference incurs a cost proportional to $\beta$, preventing the policy from exploiting reward model blind spots by generating out-of-distribution text. This is the standard PPO reward formulation (what the paper calls the "k1-style KL penalty")—the REINFORCE++ innovation is not in this formula but in how the resulting advantages are subsequently normalized.

The full algorithm (Algorithm 1). The training loop for $k = 1$ proceeds as follows:

1. Initialize the policy model $\pi_\theta$ as a copy of the reference model $\pi_{\text{ref}}$. (The reference model is frozen for the entire training run.)

2. For each of $M$ training steps:

   • Sample a batch $\mathcal{D}_{\text{batch}}$ of prompts from the training dataset. The batch size for $k = 1$ is simply the number of distinct prompts (the paper does not specify exact batch sizes in the general RLHF experiments, but global batches are "typically large, e.g., 1024 or more" per Section 3.1).
   • Freeze the old policy $\pi_{\text{old}} \leftarrow \pi_\theta$. (This provides the reference point for the PPO probability ratio clipping.)
   • For each prompt $q \in \mathcal{D}_{\text{batch}}$, sample one response $o \sim \pi_{\text{old}}(\cdot \mid q)$. (Since $k = 1$, only one response is generated per prompt.)
   • Compute the scalar reward $r_i$ for each response $o_i$ using the reward model $\mathcal{R}$.
   • Compute the raw advantage $A_{q, o_t}$ for each token in each response using Equation (4).
   • Collect all raw advantages from all tokens in all responses across the entire batch, compute their global mean and standard deviation, and normalize each advantage using Equation (5) to obtain $A^{\text{norm}}_{q, o_t}$.
   • Perform $k$ iterations of mini-batch updates (the variable $k$ here is the number of updates per batch, not to be confused with the group size): for each iteration, maximize the PPO clipped surrogate objective using the normalized advantages.

Token-level advantage assignment. The normalized advantage $A^{\text{norm}}_{q, o_t}$ is assigned at the token level with a specific pattern: for tokens $t < T$ (all tokens except the final one in the sequence), the advantage is set to 0. For the final token $t = T$, the advantage is the normalized reward $A^{\text{norm}}_{q, o_T}$. The paper describes this as "standard practice in RLHF for LLMs" (Appendix B.2). The rationale is that language model RL is a bandit problem: the model generates a sequence, receives a single reward at the end, and the policy gradient distributes credit backward through the sequence via the probability ratio term in the PPO objective. The advantage being zero for non-final tokens means only the final-token advantage signal drives the gradient.

Mini-batch updates within a batch. The paper implements mini-batch updates (Appendix B.2) to improve training efficiency. After computing normalized advantages for the full batch, the data is split into smaller chunks, and multiple parameter updates are performed on these chunks rather than a single full-batch update. This "introduces beneficial randomness for better generalization" and "improves convergence rates." The specific mini-batch size and number of updates are not specified—they appear to be implementation details of the OpenRLHF framework rather than carefully tuned hyperparameters.

Why k=1 with global normalization works at all. A reader might reasonably ask: with only one sample per prompt, there is no local comparison group, so how do we even define an advantage? The answer is that the advantage does not come from comparing responses to the same prompt—it comes from comparing this response to the global distribution of all responses. The global mean acts as a learned baseline (albeit a very simple one: the average reward of the current batch). A response that scores above the global mean earns a positive normalized advantage; one below earns a negative one. This is a valid policy gradient signal—it tells the model to increase the probability of responses that are better than average and decrease the probability of responses worse than average—without requiring multiple responses to the same prompt.

The cost is that the baseline is less precise than one conditioned on the specific prompt (some prompts are naturally harder, so "above average for this specific prompt" is a more informative signal than "above average across all prompts"). The paper argues that this cost is outweighed by the benefits: more distinct prompts per batch, no bias from local normalization, and no instability from small-group statistics. The empirical results in Table 1 support this argument.

---

REINFORCE++_w/_Baseline for Complex Tasks ($k > 1$)

The second variant, REINFORCE++_w/_Baseline, is designed for complex reasoning and agentic tasks where sampling multiple responses per prompt is beneficial—but where GRPO's local normalization causes overfitting and instability.

Why group sampling helps for complex tasks. In mathematical reasoning and tool-use tasks, reward signals are often sparse and binary (0 for wrong answer, 1 for correct). A single sample provides only one bit of information. Multiple samples per prompt provide richer information: if the model generates four responses and two are correct, we learn that the prompt is within the model's capability but requires multiple attempts; if zero are correct, the prompt may be genuinely hard; if all four are correct, it's trivially easy. Additionally, the group mean reward provides a prompt-specific baseline that accounts for intrinsic prompt difficulty—a response scoring 1 on a prompt where the average is 0.25 is more informative than a response scoring 1 where the average is 0.9.

However, the paper argues that while the group mean is useful, the group standard deviation is harmful when used for normalization. The REINFORCE++_w/_Baseline approach separates these two: keep the group mean for reward reshaping, but replace the group standard deviation with the global batch standard deviation.

Two-step advantage computation. The baseline variant computes advantages in two distinct stages, described in Equations (6) and (7):

Step 1 — Group Mean Subtraction (Reward Reshaping):

$A'_{q, o_t} = \mathcal{R}_{q, o_t} - \text{mean}_{\text{group}}(\mathcal{R}_{q, o_t})$

where $\mathcal{R}_{q, o_t}$ is the raw reward for the response containing token $o_t$ to prompt $q$, and $\text{mean}_{\text{group}}(\mathcal{R}_{q, o_t})$ is the average reward across all $k$ responses generated for that same prompt.

What this computes: subtract the prompt's average reward from each individual response's reward. If the prompt is easy (average reward 0.9), a correct response gets $A' = 1 - 0.9 = 0.1$; if the prompt is hard (average reward 0.1), a correct response gets $A' = 1 - 0.1 = 0.9$. The transformed advantage $A'$ is no longer confounded by prompt difficulty—it reflects how much better this response is than the typical response to this prompt.

Why this step: reward reshaping makes the signal robust to different reward scales. The paper notes that this "reduces the need for designing complex reward structures" and "supports both 0/1 and -1/1 reward schemes" (Section 5.1). Without this subtraction, a reward model that consistently outputs values near 1 would produce small advantages even for excellent responses, while a reward model outputting values near 0 would produce large advantages even for mediocre ones. The group mean subtraction centers each prompt's rewards around zero, making the subsequent global normalization meaningful across prompts with different reward level baselines.

Step 2 — Global Batch Normalization (Stability):

$A^{\text{norm}}_{q, o_t} = \frac{A'_{q, o_t} - \text{mean}_{\text{batch}}(A')}{\text{std}_{\text{batch}}(A') + \epsilon}$

where $A'_{q, o_t}$ is the group-mean-subtracted advantage from Step 1, $\text{mean}_{\text{batch}}(A')$ is the mean of all group-mean-subtracted advantages across the entire training batch (all prompts, all responses), and $\text{std}_{\text{batch}}(A')$ is the standard deviation of the same global population.

What this computes: after each response's advantage has been centered relative to its prompt's group mean, normalize all advantages globally. A response that is $+0.9$ above its group mean on a hard prompt and another that is $+0.1$ above its group mean on an easy prompt are both placed on the same scale by dividing by the global standard deviation. The result is a normalized advantage that reflects both (a) how much better this response is than average for its prompt, and (b) how that improvement compares to the typical improvement seen across all prompts.

Why this form over GRPO's approach: GRPO subtracts the group mean AND divides by the group standard deviation in one step. REINFORCE++_w/_Baseline decouples these—the group mean subtraction handles reward reshaping, while global standard deviation provides stable scaling. The critical difference is the denominator: GRPO divides by $\text{std}_{\text{group}}(\mathcal{R}_{q, o_t})$, which is estimated from only $k$ samples, while REINFORCE++_w/_Baseline divides by $\text{std}_{\text{batch}}(A')$, estimated from potentially thousands of samples. The group standard deviation is noisy, biased, and can explode (when all $k$ samples have similar rewards), while the global standard deviation is stable by construction.

The formal relationship to PPO. The paper explicitly draws a parallel in Section 3.3: "REINFORCE++_w/_Baseline can be viewed as a simplified and more stable variant of PPO. It is formally equivalent to a PPO agent where: (1) The critic network is removed; (2) The GAE parameters are set to $\lambda = 1$ and $\gamma = 1$; and (3) A two-step global batch normalization is used as the baseline instead of a learned value function."

Setting $\lambda = 1$ and $\gamma = 1$ in Generalized Advantage Estimation (Equation 2 in the original paper) means the advantage reduces to $A_{q, o_t} = \sum_{l=0}^\infty \delta_{t+l} = \sum_{l=0}^\infty (r_{t+l} + V(o_{t+l+1}) - V(o_{t+l}))$, which telescopes to the total return minus the value baseline. Removing the critic ($V(\cdot) = 0$) further reduces this to the raw cumulative reward. The two-step normalization then replaces the learned value baseline with a combination of group-mean and global-mean statistics. This is a significant simplification—no GAE computation, no value network training, no temporal difference learning—while the paper claims it performs better, not just more efficiently.

Batch construction with group sampling. For $k > 1$, a global batch of size $N = 1024$ might consist of $N/k$ unique prompts (e.g., 256 prompts with $k = 4$). The global mean and standard deviation are computed over all $N$ individual advantages (not averaged per prompt). This means prompts with larger $k$ contribute proportionally more to the global statistics, which is the intended behavior—each response is an independent sample from the policy for the purpose of advantage estimation.

---

KL Penalty Design: Why k2 Replaces k3

The KL divergence penalty is a critical regularization mechanism in RLHF that prevents the policy from "reward hacking"—generating text that scores highly under the reward model but has drifted so far from natural language that it is incoherent or exploits reward model blind spots. The penalty encourages the policy $\pi_\theta$ to remain close to the reference model $\pi_{\text{ref}}$ (typically the supervised-fine-tuned starting point).

The paper's analysis in Appendix B.1 distinguishes three common estimators of the KL divergence, each constructed from the same building blocks but with importantly different properties. The estimators are all functions of the log-ratio between the policy and reference distributions.

The building block. Define the per-token log probability ratio:

$\delta(y) = \frac{\pi_{\text{ref}}(y \mid \cdot)}{\pi_\theta(y \mid \cdot)}$

where $\pi_\theta(y \mid \cdot)$ is the policy model's probability of generating token $y$ given the context, and $\pi_{\text{ref}}(y \mid \cdot)$ is the reference model's probability. Note the numerator/denominator convention: $\delta(y) > 1$ when the reference model assigns higher probability than the current policy (the policy has moved away), and $\delta(y) < 1$ when the policy assigns higher probability (the policy has reinforced this token beyond the reference).

The three estimators.

• k1: $k_1(y) = -\log \delta(y) = \log \frac{\pi_\theta(y|\cdot)}{\pi_{\text{ref}}(y|\cdot)}$
• k2: $k_2(y) = \frac{1}{2}(\log \delta(y))^2 = \frac{1}{2}\left(\log \frac{\pi_{\text{ref}}(y|\cdot)}{\pi_\theta(y|\cdot)}\right)^2$
• k3: $k_3(y) = \delta(y) - 1 - \log \delta(y)$

What each one estimates. The expected value of each under samples from $\pi_\theta$ estimates a different divergence:

• $E_{y \sim \pi_\theta}[k_1(y)] = -D_{\text{KL}}(\pi_\theta \| \pi_{\text{ref}})$ — the negative Reverse KL. The k1 estimator, when averaged, directly gives the (negative) KL divergence. This is useful when embedded in the reward (as REINFORCE++ does) because subtracting it from the reward term $r - \beta \cdot k_1$ is equivalent to adding $+\beta \cdot D_{\text{KL}}$ to the objective.

• $E_{y \sim \pi_\theta}[k_2(y)] = \frac{1}{2}E[(\log(\pi_{\text{ref}}/\pi_\theta))^2]$ — this is not a standard divergence in expectation, but its gradient with respect to $\theta$ equals the gradient of the Reverse KL, making it a valid loss term for constraining the policy.

• $E_{y \sim \pi_\theta}[k_3(y)]$ — this estimates the Forward KL $D_{\text{KL}}(\pi_{\text{ref}} \| \pi_\theta)$ only when $\pi_\theta$ is absolutely continuous with respect to $\pi_{\text{ref}}$. The paper argues this is fundamentally the wrong target.

Why k3 fails (the GRPO problem). The paper identifies two specific failure modes of k3 when used as a separate loss term in GRPO:

1. Wrong KL direction. GRPO is training with samples from $\pi_\theta$ (the policy being optimized), so any estimator computed on these samples estimates properties of $\pi_\theta$, not $\pi_{\text{ref}}$. The Reverse KL $D_{\text{KL}}(\pi_\theta \| \pi_{\text{ref}})$ is the natural objective because it penalizes the policy for placing probability mass where the reference does not—it is "mode-seeking," encouraging the policy to stay within the reference's support. The Forward KL $D_{\text{KL}}(\pi_{\text{ref}} \| \pi_\theta)$ is "mass-covering," penalizing the policy for failing to cover the reference's entire distribution. When training an LLM to follow instructions, mode-seeking (stay close to what the reference does) is the correct behavior; mass-covering (try to do everything the reference might do) would encourage the policy to spread probability across implausible outputs.

2. Extreme variance. The k3 estimator involves the term $\delta(y) = \pi_{\text{ref}}(y)/\pi_\theta(y)$. When the policy assigns very low probability to a token that the reference assigns moderate probability (which happens routinely in the early stages of RL training when the policy is exploring), this ratio explodes. The paper states this "leads to 'infinite variance'" and that the computation $\exp(\log p - \log q)$ is "prone to overflow." In practice, this manifests as sudden, massive gradient spikes that destabilize training, which is why GRPO implementations frequently reset the reference model to the current policy—effectively restarting the KL constraint from zero to prevent the ratio from growing too large.

Why k2 is correct. The paper proves that the k2 estimator, when used as a loss term, produces a gradient that exactly matches the theoretical Reverse KL gradient:

$\nabla_\theta J_{\text{k2 as loss}}(\theta) = E_{y \sim \pi_\theta}\left[\left(\log \frac{\pi_\theta(y|x)}{\pi_{\text{ref}}(y|x)}\right) \nabla_\theta \log \pi_\theta(y|x)\right]$

What this computes: the gradient of the k2 loss is the log-ratio between policy and reference probabilities, multiplied by the policy score function (the gradient of log-probability). This is exactly the standard policy gradient for the Reverse KL divergence. When the policy assigns higher probability than the reference ($\pi_\theta > \pi_{\text{ref}}$), the log-ratio is positive, and the gradient pushes the policy to decrease probability (because the loss gradient is positive and we minimize the loss). When the policy assigns lower probability ($\pi_\theta < \pi_{\text{ref}}$), the log-ratio is negative, and the gradient pushes the policy to increase probability.

Why this form: the k2 estimator is symmetric in log-space—it penalizes the squared log-ratio, so deviations in either direction (higher or lower policy probability than reference) incur a penalty. The gradient is proportional to the log-ratio itself, providing a linear penalty signal whose magnitude scales with the degree of deviation. This avoids the instability of k3 (where the penalty involves the raw probability ratio, not its log) while correctly targeting the Reverse KL gradient.

Why k1 cannot stand alone as a loss term. The paper explicitly excludes k1 as a separate loss term because its gradient "does not depend on $\pi_{\text{ref}}$ and thus provides no constraining effect." When $J_{\text{k1}} = E[\log \pi_\theta - \log \pi_{\text{ref}}]$, differentiating gives $\nabla_\theta J_{\text{k1}} = E[\nabla_\theta \log \pi_\theta]$, because the $\pi_{\text{ref}}$ term vanishes (the reference is frozen). The gradient only depends on the policy's own log-probability, pushing it uniformly toward higher probability everywhere. It does not compare against the reference, so it provides no constraint.

However, k1 is appropriate inside the reward, as REINFORCE++ does. When $r_{\text{effective}} = r_{\text{RM}} - \beta \cdot \text{KL}(t)$, the KL term subtracts from the reward, and the PPO objective balances reward maximization against this penalty. The policy gradient of this combined objective includes terms from both the reward model signal and the KL penalty, with the latter providing the constraining effect. The key distinction is that k1 inside the reward constrains the policy because the reward enters the advantage, which multiplies the policy gradient, while k1 as a separate loss does not constrain because its gradient relies only on the policy's own probabilities.

The final objective for REINFORCE++_w/_Baseline. Combining the PPO surrogate with the k2 KL penalty, the full objective (Equation 8) is:

$\mathcal{L} = \mathcal{L}_{\text{PPO}}(A^{\text{norm}}) - \lambda \cdot J_{\text{k2 as loss}}(\theta)$

where $\mathcal{L}_{\text{PPO}}(A^{\text{norm}})$ is the standard PPO clipped surrogate objective using the globally normalized advantages, $J_{\text{k2 as loss}}(\theta) = E\left[\frac{1}{2}\left(\log \frac{\pi_\theta}{\pi_{\text{ref}}}\right)^2\right]$ is the k2 loss term, and $\lambda$ is a hyperparameter controlling the KL penalty strength (analogous to $\beta$ in the k1 reward-embedded approach).

What this computes: the PPO term maximizes the expected reward (via the normalized advantages) while keeping updates conservative (via clipping). The KL term separately penalizes the policy for diverging from the reference, with strength $\lambda$. Minimizing this combined loss increases reward while staying close to the reference.

Why the minus sign: the KL term is subtracted because the overall objective is maximized (PPO convention), and the KL term is a cost to be minimized. So the objective reads: "maximize PPO reward, minus a penalty for KL divergence."

---

Implementation Details

The paper describes several engineering details that, while not algorithmic innovations, are necessary for reproducing the results.

Token-level advantage assignment (Appendix B.2). As noted earlier, the normalized advantage is applied only at the final token position. For all $t < T$, $A^{\text{norm}}_{q, o_t} = 0$. For $t = T$, $A^{\text{norm}}_{q, o_T}$ is the globally normalized reward. This follows standard RLHF practice: the language generation task is treated as a contextual bandit where the model receives a single reward at sequence completion, and the policy gradient propagates credit to earlier tokens through the probability ratio $s_t(\theta) = \pi_\theta(o_t|q, o_{<t}) / \pi_{\text{old}}(o_t|q, o_{<t})$ in the PPO objective.

Why zero advantages for non-final tokens: if non-zero advantages were assigned at every token, the PPO objective would be optimizing for intermediate states as though they were terminal, which is incorrect for a task where only the final output matters. The probability ratio term in the PPO loss handles temporal credit assignment—if a token early in the sequence leads to sequences that ultimately receive high rewards, increasing that token's probability will increase the expected PPO objective because it increases the probability ratio for that token, multiplied by the final-token advantage.

Mini-batch updates. The paper implements mini-batch processing where "data is processed in smaller, manageable chunks rather than full-batch updates," with "multiple parameter updates" per collected batch. This is a standard optimization technique in RL—rather than computing one gradient update on the full batch and discarding the data, split the batch into mini-batches and perform multiple gradient steps, which increases sample efficiency. The number of mini-batch iterations per batch is denoted $k$ in Algorithm 1 (confusingly, the same letter used for group size in the Baseline variant; these are different variables in different contexts).

Reward clipping and scaling (Appendix B.2). Beyond the global normalization, the paper implements "comprehensive reward processing":

• Z-score normalization — global standardization (the main method) already handles this.
• Clipping — "constrains reward values within predefined bounds to avoid instability." Exact clipping thresholds are not specified.
• Scaling — "applies appropriate scaling factors for numerical stability during updates." Again, exact factors are not specified.

These are standard defensive measures in RL implementations and likely follow defaults from the OpenRLHF framework.

Advantage clipping in PPO objective. The paper uses the standard PPO clipped surrogate objective (Equation 1):

$\mathcal{L}^{\text{PPO}}(\theta) = E_{q \sim P(Q), o \sim \pi_{\theta_{\text{old}}}(O|q)}\left[\frac{1}{|o|}\sum_{t=1}^{|o|} \min\left(s_t(\theta) A_t, \text{clip}(s_t(\theta), 1 - \epsilon, 1 + \epsilon) A_t\right)\right]$

where $s_t(\theta) = \frac{\pi_\theta(o_t|q, o_{<t})}{\pi_{\theta_{\text{old}}}(o_t|q, o_{<t})}$ is the probability ratio between the current and old policy for token $t$, $A_t$ is the (globally normalized) advantage, and $\epsilon$ is the clipping parameter.

What this computes: for each token, compute the probability ratio $s_t(\theta)$ (how much more/less likely the current policy is to generate this token compared to when the batch was sampled). If the advantage $A_t$ is positive, the objective encourages increasing the ratio, but clips it at $1 + \epsilon$ to prevent the policy from changing too much. If $A_t$ is negative, the objective encourages decreasing the ratio, but clips at $1 - \epsilon$. Taking $\min(\text{unclipped}, \text{clipped})$ ensures the objective is a pessimistic bound—it only counts improvements that stay within the trust region defined by $\epsilon$.

Why this form: PPO's clipped objective prevents destructive large policy updates. Without clipping, a single batch with unusually high or low advantages could drive the probability ratio $s_t(\theta)$ far from 1, effectively erasing previously learned behaviors. The clipping parameter $\epsilon$ (typically 0.1 or 0.2 in practice, though not specified in this paper) defines a "trust region" in probability-ratio space. The paper does not modify this standard PPO mechanism—its innovation is entirely in how the advantages $A_t$ are computed and normalized before being plugged into this existing objective.

No GAE, no value function. A notable absence in both REINFORCE++ variants is Generalized Advantage Estimation (GAE). In standard PPO, the critic network's value estimates are combined with rewards through GAE to produce a low-variance advantage estimate that balances bias and variance via parameters $\gamma$ (discount) and $\lambda$ (trace decay). REINFORCE++ completely dispenses with this machinery: there is no temporal discounting ($\gamma = 1$), no eligibility traces ($\lambda = 1$), and the "baseline" is the global mean advantage rather than a learned value function. The paper's Section 3.3 explicitly notes this equivalence: setting $\gamma = 1$ and $\lambda = 1$ in GAE with zero value function telescopes the advantage to the raw reward, which is then normalized. This is a dramatic simplification—the entire temporal credit assignment infrastructure of PPO is reduced to "use the final reward, normalized globally."

The reference model is never updated. Throughout training, $\pi_{\text{ref}}$ remains frozen at the initial policy checkpoint. This is standard in RLHF but worth noting because GRPO implementations sometimes reset $\pi_{\text{ref}}$ to the current policy periodically to prevent k3 estimator instability. The paper's choice of k1 (embedded in reward) and k2 (as loss) makes such resets unnecessary—both estimators are well-behaved even as the policy diverges, because they operate in log-space rather than probability-ratio space.

4. Key Insights and Innovations

Innovation 1: Diagnosing Local Normalization as a Root Cause, Not Just a Minor Instability

The paper's most consequential intellectual move is not proposing global normalization—the technique is well-known from standard RL practice (Andrychowicz et al., 2020)—but rather identifying prompt-level normalization as the single root cause of three distinct failure modes that prior work had treated as separate problems requiring separate fixes. GRPO added an epsilon term to prevent division by zero (treating it as a numerical edge case). VAPO discussed the value of group sampling without questioning the normalization itself. Various implementations used learning rate schedules, reward clipping, and reference model resets to manage instability—each treating a symptom rather than the underlying disease.

The paper's diagnostic contribution is the formal proof in Appendix A that the GRPO advantage estimator is mathematically biased, combined with empirical demonstrations that this bias produces catastrophic real-world consequences: complete overfitting to 30 training problems (95% training accuracy → 0% test accuracy), explosion of KL divergence during training (Figure 2), and collapse of logical reasoning performance on out-of-distribution inputs (Figure 4). This is not a marginal improvement—it is a root-cause analysis showing that the core statistical mechanism of the dominant open-source RLHF method is fundamentally flawed.

What distinguishes this from prior observations of instability is the unified explanation. Local normalization's three problems—theoretical bias (proven mathematically), practical instability (small-group standard deviation approaching zero), and task overfitting (relative rather than absolute evaluation)—are not independent issues. They all arise from the same source: using statistics computed from 4–8 correlated samples to normalize advantages. The paper's framing converts a collection of seemingly disparate engineering headaches into a single, tractable design flaw. This changes the conversation from "how do we patch GRPO to be more stable?" to "local normalization should never be used for advantage scaling."

Innovation 2: Separating Reward Reshaping from Variance Normalization as Distinct Operations

Prior critic-free methods conflated two operations that the paper argues should be kept separate: removing prompt-level baseline effects (reward reshaping) and scaling advantages to unit variance (stabilization). GRPO performs both simultaneously by dividing by the local standard deviation after subtracting the local mean. RLOO performs only baseline subtraction without explicit scaling.

The paper's key reframing, embodied in REINFORCE++_w/_Baseline, is that group-mean subtraction for reward reshaping is valuable, but group-standard-deviation division for normalization is harmful. The two-step advantage computation (Equation 6 → Equation 7) is not merely a different formula—it is a conceptual decomposition of what had been a monolithic operation into its constituent purposes with different optimal solutions.

This decomposition has practical significance beyond the specific formulas. It provides a design principle for future critic-free algorithms: local statistics are appropriate for removing prompt-specific bias (because "better than average for this prompt" is a meaningful signal), but global statistics should govern scaling (because the variance of a small group is an unreliable measure of the advantage distribution's width). This principle explains the paper's empirical finding that REINFORCE++ (k=1) works well for general RLHF—when prompt diversity is high enough, even prompt-specific baseline subtraction is unnecessary because the global mean serves as a reasonable baseline across diverse prompts. It also explains why the Baseline variant uses group sampling at all: not for variance normalization, but to obtain the prompt-conditioned mean for reward reshaping on tasks where prompt difficulty varies dramatically.

The field previously lacked this distinction. Implementations treated normalization as a single operation whose purpose was "make the numbers well-behaved." The paper shows that the choice of normalization group depends on which statistical purpose the normalization serves, and that using the wrong group for the wrong purpose (local for scaling) is actively destructive.

Innovation 3: Formal Proof that Local Advantage Normalization is a Biased Estimator

While the empirical demonstration of overfitting is compelling, the paper's theoretical contribution in Appendix A gives the argument permanence—it proves that the problem is inherent to the estimator's mathematical structure, not a hyperparameter tuning issue or a quirk of specific reward models. The bias emerges from a dependency between the numerator (centered reward) and denominator (local standard deviation) that cannot be eliminated by any tuning of k, epsilon, learning rate, or reward clipping.

The proof's structure is elegant in its simplicity. The key step is showing that the conditional expectation of the denominator $E[1/D \mid \epsilon_i]$ depends on $\epsilon_i^2$—larger-magnitude true advantages produce larger expected standard deviations, which partially cancel the numerator's scaling. This creates a systematic compression effect: large advantages are underestimated, small advantages are overestimated. The estimator is not just noisy (which would average out over many updates) but systematically wrong (which accumulates bias in the learned policy).

This matters beyond GRPO specifically. The proof shows that any normalization scheme that divides by a statistic computed from the same small group of samples whose mean was subtracted will exhibit this bias. This includes RLOO when implemented with variance scaling, any future group-based method, and even global normalization with small batch sizes. The theoretical result thus provides a criterion for evaluating future proposals: if the normalization denominator depends on the same samples as the numerator, and the sample size is small, the estimator will be biased. The fix—increasing the denominator's sample size until the dependency becomes negligible—is the theoretical justification for global normalization with large batches.

The proof also provides the asymptotic guarantee that the paper relies on: as $N \to \infty$, the sample standard deviation converges to the true population $\sigma$, which is independent of any individual $\epsilon_i$, eliminating the bias. This connects the engineering choice of batch size 1024 to a mathematical property: at that scale, the residual bias is dominated by other noise sources. The paper does not claim exact unbiasedness at finite N, but rather that the bias becomes practically negligible relative to the inherent variance of policy gradient estimation.

Innovation 4: Identifying the KL Estimator as a Complementary Failure Mode Requiring Simultaneous Fix

The paper's analysis of KL estimators (Appendix B.1) identifies a secondary but critical problem: GRPO's instability comes not only from local advantage normalization but also from using the k3 estimator for the KL divergence penalty. The insight is that both failures must be fixed together—switching to global normalization without fixing the KL estimator would leave training vulnerable to a different instability mechanism.

The k3 analysis is more than an implementation detail because it reveals a mismatch between mathematical objective and computational estimator that pervades the critic-free RLHF literature. GRPO's k3 estimator approximates the Forward KL $D_{KL}(\pi_{\text{ref}} \| \pi_\theta)$, but training samples come from $\pi_\theta$, making the Reverse KL $D_{KL}(\pi_\theta \| \pi_{\text{ref}})$ the correct target. This is not a minor approximation error—it estimates the wrong quantity, with the wrong variance properties, using samples from the wrong distribution. The paper's characterization of k3 as having "infinite variance" is technically informal but captures the practical reality: the importance ratio $\pi_{\text{ref}}/\pi_\theta$ explodes when the policy explores tokens it assigns low probability, which is exactly what happens during RL training.

The k2 solution is elegant because it sidesteps the importance ratio entirely. By working in log-space, k2 avoids the division that causes k3's instability, and its gradient exactly matches the Reverse KL gradient without requiring importance sampling corrections. The paper frames this not as inventing k2 (the estimator was known) but as diagnosing that GRPO's choice of k3 is actively harmful and that the correct estimator is both theoretically sound and practically stable enough to eliminate the need for reference model resets—a common GRPO practice that the paper's approach renders unnecessary.

This innovation is significant because it shows that the two algorithmic changes (global normalization + k2 KL) are complementary rather than alternative. Fixing only the advantage normalization while keeping k3 would leave the KL penalty as a source of gradient spikes. Fixing only the KL estimator while keeping local normalization would leave the overfitting problem intact. The paper's experimental design implicitly tests this complementarity—REINFORCE++_w/_Baseline uses both changes simultaneously and the results (Table 4, 24.10 vs. 22.58 for GRPO) reflect their combined effect.

5. Experimental Analysis

Evaluation Methodology

• Datasets. The paper evaluates across several distinct benchmarks, each chosen for a specific experimental purpose:

  • Chat-Arena-Hard (Li et al., 2024): Used for general-domain RLHF experiments (Section 4.1). This benchmark contains diverse instruction-following prompts and uses an LLM-based judge to score response quality. The paper does not specify the number of evaluation prompts used.
  • AIME-24 and AIME-25 (Jia, 2024; math ai, 2025): American Invitational Mathematics Examination problems. AIME-24 (30 questions) is used as a small training set for the overfitting analysis (Section 4.2.1); AIME-25 serves as the out-of-distribution test set. AIME-24 also appears as an evaluation benchmark in the multi-step RL experiments (Section 4.3).
  • MATH-500: A 500-question subset of the MATH benchmark (Hendrycks et al., 2021), used as an in-distribution evaluation in the RL-from-zero reasoning experiments (Section 4.2, Table 3).
  • AMC-23: Used alongside AIME-24 as an OOD evaluation in the RL-from-zero setting (Section 4.2, Table 3). No size or source details are specified beyond the name.
  • Knights and Knaves (K&K) Puzzles (Xie et al., 2025, 2024): Logic puzzles where difficulty increases with the number of characters ("people") involved. Used to test OOD generalization by training on simpler puzzles (2-3 people) and evaluating on harder ones (up to 8 people).
  • Multi-step tool-use benchmarks (Section 4.3): AIME 2024, AIME 2025, HMMT FEB-2024, HMMT FEB-2025, and CMIMC. These are math competition datasets used in the ZeroTIR environment (Mai et al., 2025) for evaluating agentic tool-use capabilities. The paper reports average@32 across these five benchmarks.

• Base models. The paper uses two different base models depending on the experiment:

  • Llama-3-8B-SFT: An 8B-parameter instruction-following model for the general-domain RLHF experiments (Section 4.1). The paper states it was "refined using a Bradley-Terry reward model trained on ~700K human preference pairs."
  • Qwen 2.5 Base 7B: A 7B-parameter base model used for both the reasoning experiments (RL-from-zero setting in Section 4.2) and the multi-step tool-use experiments (Section 4.3). In the reasoning setting, a math-specific variant "Qwen2.5-Math-Base" is used. The choice of 7-8B scale models reflects a practical regime where critic-free methods' memory savings are most impactful—large enough to be capable, small enough that doubling memory for a critic is painful.

• Metrics. The paper reports several metrics across experiments:

  • Score (Chat-Arena-Hard): An LLM-judged quality score, with higher values indicating better instruction-following. The exact scale is not specified, but scores cluster around 44-47 in Table 1.
  • Pass@N: The probability that at least one of N sampled responses is correct, standard for math reasoning benchmarks. Reported as Pass@1, Pass@8, Pass@16, or Pass@32 depending on the experiment. This is computed by sampling N responses, checking each for correctness against ground-truth answers, and reporting the fraction of problems where at least one response is correct.
  • Average@32 (tool-use benchmarks): The average accuracy across five benchmarks when sampling 32 responses per problem. This is effectively Pass@32 averaged across benchmarks.
  • Response Length (Table 1): Average number of tokens per generated response, used to compute per-token efficiency (Score / Length).
  • Per-Token Score (Table 1): Score divided by average response length in tokens, measuring the quality achieved per unit of generation cost.
  • Training Reward and KL Divergence (Figure 2): Smoothed curves showing the reward model score and KL divergence from the reference model during training. These are monitoring metrics, not evaluation metrics—they track training dynamics rather than final performance.

• Baselines. The paper compares against four established critic-free methods and one critic-based method:

  • GRPO (Shao et al., 2024): Group Relative Policy Optimization with local advantage normalization using $k = 4$ or unspecified $k > 1$ depending on the experiment.
  • RLOO (Ahmadian et al., 2024): REINFORCE Leave-One-Out with $k = 4$, using the mean of other responses as a per-prompt baseline.
  • ReMax (Li et al., 2023): Uses a greedy decoding response as baseline, with $k = 1+1$ (one stochastic sample, one greedy sample).
  • PPO (Schulman et al., 2017): Full critic-based Proximal Policy Optimization, included only in the multi-step tool-use experiments (Table 4) as a heavyweight baseline.
  • For the RL-from-zero reasoning experiments (Table 3), only GRPO is compared against, using unspecified $k > 1$.

• Generation budget / compute accounting. The paper uses several different approaches to quantifying compute, which creates some inconsistency across experiments:

  • Per-token efficiency (Table 1): For general RLHF, efficiency is measured as Score / Length (tokens), recognizing that longer responses cost more compute to generate and evaluate. This is a practical metric for deployment where generation cost scales with output length.
  • Group size $k$: For critic-free methods, the primary compute lever is the number of responses sampled per prompt. GRPO and RLOO use $k = 4$ in general RLHF experiments, costing $4\times$ more generations per prompt than REINFORCE++ at $k = 1$.
  • Training iterations and batch size: The paper describes batch sizes of "typically 1024 or more" for global normalization, but does not systematically compare different batch sizes or report total training FLOPs. For the multi-step experiments (Section 4.3), training runs use the OpenRLHF framework with datasets from ORZ and DAPO, but specific compute budgets are not reported.
  • Pass@N budget: For evaluation, Pass@N measures accuracy when allowed N samples. This is a test-time compute metric separate from training compute. Pass@32 in Table 4 means 32 samples were drawn per problem at evaluation time.
  • The paper does NOT provide FLOPs-matched comparisons (unlike the compute-optimal scaling paper in the example). There are no experiments comparing, e.g., REINFORCE++ with extra training steps against GRPO with fewer steps at the same total FLOPs.

• Cross-validation / statistical protocol. The paper does not report any cross-validation, statistical significance testing, or confidence intervals. All results appear to be single-run evaluations with no error bars. This is a notable weakness—in the overfitting experiment (Table 2), the extreme result (95.0% vs. 0.0%) is dramatic enough that statistical noise is unlikely to be the explanation, but for closer comparisons (e.g., REINFORCE++ 46.7 vs. GRPO 46.8 in Table 1, or REINFORCE++ 21.04 vs. GRPO 18.96 in Table 3), the lack of error bars makes it impossible to assess whether differences are meaningful or within run-to-run variance. The small training set experiment (30 AIME-24 problems) is particularly vulnerable to noise—a different random 30-problem subset might yield meaningfully different overfitting behavior.

Main Quantitative Results

General-Domain RLHF: Single-Sample REINFORCE++ vs. Group-Sampling Methods

The headline result from Table 1 is that REINFORCE++ ($k = 1$) achieves a score of 46.7 on Chat-Arena-Hard, statistically tied with GRPO's 46.8 ($k = 4$), while using $4\times$ fewer generations per training step and producing shorter responses (832 vs. 860 tokens). The per-token efficiency metric highlights this advantage: REINFORCE++ achieves 0.0561 per token versus GRPO's 0.0544 and RLOO's 0.0515.

Several details in Table 1 deserve attention. First, all critic-free methods cluster within a narrow range (44.6 to 46.8), suggesting the benchmark or reward model may have limited discriminative power at this performance level. The paper does not report scores for the base SFT model before RL training, making it impossible to assess the absolute improvement from RLHF. Second, response length varies systematically: REINFORCE++ produces the shortest responses (832 tokens), ReMax produces shorter responses still (805 tokens), while GRPO (860) and RLOO (866) are longer. The per-token metric partially controls for this, but the causal direction is unclear—does global normalization produce more concise responses, or does it avoid the length exploitation that inflates GRPO's scores? Figure 2 provides suggestive evidence: GRPO's KL divergence rises rapidly during training (reaching approximately 0.5+ by step 300), while REINFORCE++'s KL remains much lower (under 0.1), consistent with the interpretation that GRPO achieves its score partially through reward hacking (lengthening responses to exploit the reward model) rather than genuine quality improvement.

The training dynamics in Figure 2 reveal a more nuanced story than the final scores suggest. GRPO's training reward rises faster and higher (reaching approximately 2.0-2.2 by step 300) compared to REINFORCE++ (approximately 1.6-1.8). But this higher reward comes at the cost of rapidly increasing KL divergence—GRPO's KL approximately doubles between steps 100 and 300, while REINFORCE++'s remains nearly flat. The paper interprets this as evidence of "higher KL-to-reward conversion efficiency" for REINFORCE++—each unit of KL divergence buys more genuine reward improvement. This is a more sophisticated argument than simply "we match GRPO's score"—it claims that GRPO's score is inflated by reward model exploitation that would likely be penalized by a stronger reward model or human evaluation.

An important limitation: the paper evaluates only on Chat-Arena-Hard, which uses an LLM judge rather than human evaluation. LLM judges are known to have their own biases (e.g., favoring longer, more verbose responses), and it is possible that both methods are exploiting judge biases in ways that would not transfer to human preference. Direct human evaluation or a broader benchmark suite would strengthen this result substantially.

Reasoning with Group Sampling ($k > 1$): Preventing Catastrophic Overfitting

The small-dataset overfitting experiment (Table 2) provides the paper's most dramatic result. Trained on only 30 AIME-24 problems and evaluated on AIME-25:

• GRPO (local normalization): Achieves 95.0% Pass@1 on the training set but 0.0% Pass@1 and 0.4% Pass@16 on the held-out AIME-25 test set.
• REINFORCE++ (global normalization): Achieves only 71.0% Pass@1 on training but generalizes to 2.5% Pass@1 and 40.0% Pass@16 on AIME-25.

This is a stark demonstration of the overfitting problem described in Section 2.2: GRPO's local normalization creates a training signal that rewards performing well relative to other responses to the same training prompt, which is trivially achievable through memorization. The model learns to solve the 30 specific problems in its training set (95% accuracy) but acquires zero transferable reasoning capability (0% on held-out problems). REINFORCE++, by normalizing against a global batch, receives a training signal that rewards absolute performance rather than within-group competition, leading to more modest training-set accuracy but meaningful generalization.

Figure 3's training curves visually reinforce this: GRPO's training-set performance shoots up almost immediately (the left plot shows a steep near-vertical rise in the first few steps), while REINFORCE++ learns gradually over many more steps (the right plot shows a slow, steady increase). The paper describes GRPO as "immediately overfits, mastering the training questions in just a few steps," which is precisely what one would expect from a method that incentivizes memorization of prompt-specific patterns.

However, several caveats are necessary. The Pass@16 result for REINFORCE++ (40.0%) suggests that while the model does not reliably find the correct answer on the first attempt, it does possess some generalizable reasoning capability that emerges with multiple samples. This is a common pattern in reasoning models—the model learns heuristics that sometimes work rather than robust algorithms. The paper does not compare against a PPO baseline trained on the same 30 problems, which would help disambiguate whether the overfitting is specific to local normalization or a general property of RLHF with small datasets. Additionally, AIME problems are known to be extremely difficult (typical Pass@1 for strong models is in the single digits), so 2.5% Pass@1 and 40% Pass@16 on AIME-25 from only 30 training problems is arguably an impressive result—the paper frames it as evidence of generalization, but an alternative framing is that even global normalization cannot extract much generalizable reasoning from 30 examples.

Logical Reasoning: Robustness to Increasing Difficulty

The K&K puzzles experiment (Figure 4) tests how well each method's learned reasoning transfers to harder versions of the same task type. The x-axis shows the number of "people" in the puzzle (more people = harder logical deduction), and the y-axis shows Pass@1 score.

Key observations from Figure 4:

• Easy regime (2-3 people): Both methods perform similarly, with scores around 70-80%. GRPO is actually slightly better at 3 people (approximately 75% vs. 73%).
• Medium regime (4-6 people): REINFORCE++ pulls ahead. At 4 people, REINFORCE++ scores approximately 68% vs. GRPO's 60%; at 5 people, approximately 63% vs. 55%; at 6 people, approximately 58% vs. 52%.
• Hard/OOD regime (7-8 people): The gap widens further. At 7 people, REINFORCE++ scores approximately 55% vs. GRPO's 47%; at 8 people (the hardest, most OOD setting), REINFORCE++ scores approximately 48% vs. GRPO's 38%.
• Average across all difficulties: REINFORCE++ achieves 62.1 vs. GRPO's 55.7.

The pattern is consistent with the paper's thesis: local normalization overfits to patterns that work on simpler problems (where within-group variance is higher and the relative-advantage signal is stronger), while global normalization produces more robust reasoning strategies that degrade more gracefully as difficulty increases. The performance at 8 people is particularly notable—this represents a substantial extrapolation from simpler puzzles, and REINFORCE++'s near-50% accuracy suggests it has learned general logical deduction principles rather than pattern-matched shortcuts.

A limitation: the paper does not specify the training setup for this experiment. How many training puzzles? What reward signal? Is this using the REINFORCE++ ($k > 1$) variant or the Baseline variant? The lack of these details makes it difficult to assess whether the results reflect the algorithm's properties or specific training choices. The paper also does not report Pass@N for N > 1, which would help distinguish between "the model sometimes knows the answer" and "the model always knows the answer but sometimes fails to express it."

RL from Zero: OOD Generalization in Mathematical Reasoning

Table 3 compares REINFORCE++ against GRPO when training a Qwen2.5-Math-Base model from scratch on MATH dataset splits. The key finding is that REINFORCE++ achieves superior OOD generalization while being competitive in-distribution:

• AIME-24 (OOD): REINFORCE++ 21.04 vs. GRPO 18.96 at Pass@8 — a 2.08 percentage point advantage.
• AMC-23 (OOD): REINFORCE++ 60.47 vs. GRPO 59.22 at Pass@8 — a 1.25 percentage point advantage.
• MATH-500 (ID): REINFORCE++ 72.00 vs. GRPO 73.00 at Pass@1 — a 1.00 percentage point disadvantage.

This pattern—slightly worse in-distribution but better out-of-distribution—is exactly what the paper's overfitting thesis predicts. Local normalization (GRPO) optimizes more aggressively for the training distribution, squeezing out slightly higher ID performance at the cost of generalization. Global normalization (REINFORCE++) produces a more conservative policy that transfers better to harder, OOD problems.

The Pass@8 metric on AIME-24 and AMC-23 versus Pass@1 on MATH-500 introduces an inconsistency that complicates comparison. Pass@8 measures the probability of getting the correct answer within 8 attempts, which is inherently higher than Pass@1. The paper does not explain why different Pass@N values are used for different benchmarks—it may reflect standard practice in the respective benchmark communities, but it prevents direct cross-benchmark comparison.

More critically, the paper does not report Pass@1 for the OOD benchmarks or Pass@8 for MATH-500, which would provide a more complete picture. If REINFORCE++'s OOD advantage primarily manifests at higher N (i.e., its solutions are more diverse, so more samples increase the chance of finding a correct one), that would suggest a different mechanism than if the advantage appears at Pass@1 (i.e., its most-likely solution is genuinely more often correct).

Multi-Step Tool-Use: Surpassing Full-Critic PPO

Table 4 reports results on the most challenging evaluation: the ZeroTIR environment where a Qwen 2.5 Base 7B model must learn to use Python tools to solve mathematical problems. This is a multi-step agentic setting where the model generates code, executes it, observes results, and iterates—a far more complex RL problem than single-response generation.

The headline result is that REINFORCE++_w/_Baseline achieves the highest average@32 (24.10) across five benchmarks, outperforming both GRPO (22.58, local normalization) and full-critic PPO (21.85, critic-based). Breaking down by benchmark:

• AIME 2024: GRPO leads at 31.66, REINFORCE++_w/_Baseline at 30.83, PPO at 30.20. All three are within 1.5 points.
• AIME 2025: REINFORCE++_w/_Baseline achieves 27.18, substantially outperforming both GRPO (21.87) and PPO (21.66). This is a 5.3-point gap—the largest single-benchmark advantage in the table.
• HMMT 2025: REINFORCE++_w/_Baseline at 17.91, GRPO at 16.97, PPO at 15.00.
• HMMT 2024: REINFORCE++_w/_Baseline at 18.95, PPO at 18.43, GRPO at 17.70.
• CMIMC: REINFORCE++_w/_Baseline at 25.62, GRPO at 24.68, PPO at 23.95.

Several aspects of these results merit scrutiny:

The PPO comparison is the most significant. The paper's central claim is that critic-free methods can match or exceed PPO at lower computational cost. Table 4 is the only experiment where PPO is included as a baseline, making it the sole direct evidence for this claim. REINFORCE++_w/_Baseline beats PPO on every benchmark except AIME 2024 (where it trails by 0.37 points), with an average advantage of +2.25 points. Given that PPO uses a separate critic network roughly doubling the model parameters, this is a strong result—but it comes from a single experiment on a single model family with a single PPO implementation. The paper does not describe PPO hyperparameters, training budget, or whether the PPO implementation is well-tuned. A poorly-tuned PPO baseline would inflate the apparent advantage of REINFORCE++.

The AIME 2025 result drives the average. REINFORCE++_w/_Baseline's largest advantage (+5.31 over GRPO, +5.52 over PPO) occurs on AIME 2025, which is the most temporally OOD benchmark (AIME problems change each year). This aligns with the overfitting narrative—local normalization (GRPO) and possibly the critic (PPO) overfit to patterns in the training distribution that don't transfer to future AIME problems, while global normalization produces more robust strategies. If AIME 2025 were excluded, the average gap narrows to REINFORCE++_w/_Baseline 23.33 vs. GRPO 22.75 vs. PPO 22.89—a much smaller advantage (0.6 points over GRPO, 0.4 over PPO).

The benchmarks are math competitions. All five benchmarks are mathematical problem-solving tasks. The paper does not evaluate on non-math agentic tasks (e.g., web navigation, software engineering, multi-step question answering), limiting the generality of the claim that REINFORCE++_w/_Baseline is "highly effective for complex agentic tasks" broadly.

Absolute scores are low. The average@32 scores (21-24%) mean that even with 32 attempts, the model solves only about one in four problems correctly. The paper does not report Pass@1, which would likely be in the single digits. This suggests these tasks are near the frontier of the model's capability, where test-time compute scaling (multiple samples) is essential, but also where small algorithmic improvements can have outsized effects on the Pass@32 metric.

Ablation Studies and Robustness Checks

Global vs. local normalization for overfitting (Table 2, Figure 3): The small-dataset experiment is itself an ablation—the only difference between the two conditions is the normalization strategy (global for REINFORCE++, local for GRPO). The dramatic performance divergence (95.0% train / 0.0% test vs. 71.0% train / 2.5% test Pass@1) provides the cleanest evidence that local normalization is the causal factor in overfitting. However, this is not a pure ablation because the two methods likely differ in other implementation details (KL estimator, exact reward formulation) inherited from their respective algorithm designs.

Group mean subtraction vs. full local normalization (Table 4 vs. Table 2-3): The paper's two variants implicitly ablate the role of the group standard deviation. REINFORCE++ ($k > 1$, used in Table 2-3) does not use group mean subtraction—it applies only global normalization to raw rewards. REINFORCE++_w/_Baseline (Table 4) adds group mean subtraction before global normalization. Both outperform GRPO (full local normalization, including local standard deviation), suggesting the local standard deviation is the harmful component. However, the paper never directly compares REINFORCE++ ($k > 1$) against REINFORCE++_w/_Baseline on the same task, so the marginal benefit of group mean subtraction is not isolated.

KL estimator analysis (Appendix B.1): The paper provides a theoretical analysis of k1, k2, and k3 estimators, arguing k2 is correct for Reverse KL and k3 is problematic. However, there is no empirical ablation comparing k2 vs. k3 performance within the same algorithm. The paper uses k1 (embedded in reward) for REINFORCE++ and k2 (as separate loss) for REINFORCE++_w/_Baseline, but never shows that replacing k3 with k2 in GRPO would improve GRPO's performance independent of the normalization change. This makes it impossible to attribute the Table 4 improvements specifically to the KL estimator change versus the normalization change.

$k = 1$ vs. $k > 1$ for general RLHF (Table 1 vs. Table 2-3): By comparing REINFORCE++ ($k = 1$) on general RLHF against GRPO ($k = 4$), the paper implicitly ablates the value of group sampling for diverse instruction-following tasks. The near-identical scores (46.7 vs. 46.8) with $4\times$ fewer generations suggests group sampling adds little for this task type. However, this is confounded by the normalization strategy—it's equally possible that group sampling would help GRPO-style tasks if paired with global normalization (i.e., the REINFORCE++_w/_Baseline approach on general RLHF). The paper does not test this combination.

Third-party validation (Section 5.2): The paper cites independent validations from LitePPO, ScaleRL, and DLER as evidence for global normalization's benefits. LitePPO found "global standard deviation is superior to local," ScaleRL found batch-level normalization "slightly superior in both compute efficiency and final performance" at 16,000 GPU-hour scale, and DLER found global normalization remains stable under length truncation while local normalization degrades. These are not the authors' own experiments, but they provide corroborating evidence from independent groups at different scales. However, the paper does not describe these experiments in sufficient detail to assess their methodology—readers must consult the original papers (Liu et al., 2025c; Khatri et al., 2025; Liu et al., 2025b) for specifics.

Response length effects (Table 1, Figure 2): The paper observes that REINFORCE++ produces shorter responses than GRPO (832 vs. 860 tokens) and exhibits much lower KL divergence during training. This suggests global normalization may indirectly reduce length exploitation—a common failure mode in RLHF where models learn to produce verbose but low-quality outputs that score well under length-biased reward models. However, there is no ablation that controls for response length (e.g., by adding a length penalty to the reward), so the causal mechanism remains speculative.

Missing ablation: Batch size sensitivity. The paper argues that global normalization's bias vanishes as batch size grows, and uses batch sizes of "typically 1024 or more." However, there is no experiment showing how performance varies with batch size. At what batch size does global normalization become stable? Does the bias reappear with smaller batches (e.g., 128 or 256)? This is a practically important question because large global batches require more GPU memory and may limit training throughput. The paper's claim about asymptotic unbiasedness would be strengthened by showing that the estimator's practical behavior improves with batch size and plateaus at a reasonable value.

Missing ablation: $k$ sensitivity for REINFORCE++_w/_Baseline. The Baseline variant uses $k > 1$ for group mean subtraction, but the paper does not systematically vary $k$. Does performance improve with $k = 8$ versus $k = 4$? Does the group mean estimate become more reliable with larger groups, and does this interact with the global normalization? Since GRPO's local normalization becomes more biased with smaller $k$, and REINFORCE++_w/_Baseline uses $k$ only for the mean (not the standard deviation), the sensitivity to $k$ may differ substantially between methods.

Critical Assessment

The experiments provide strong support for the paper's central diagnostic claim—that local advantage normalization causes catastrophic overfitting in low-data regimes—but the evidence for the positive claim that REINFORCE++_w/_Baseline outperforms full PPO is considerably thinner and rests on a single experiment.

The overfitting claim (local normalization causes poor generalization) is well-supported. Table 2's 95.0% → 0.0% result, Figure 3's training curve divergence, Figure 4's widening performance gap as OOD difficulty increases, and Table 3's consistent OOD advantage for REINFORCE++ across multiple benchmarks all point in the same direction. The theoretical proof in Appendix A provides a mechanistic explanation that makes the empirical results more credible—the bias is not an accidental implementation detail but a predictable consequence of the estimator's mathematical structure. The consistency across three different task types (mathematical reasoning on AIME, logical reasoning on K&K, and RL-from-zero on MATH/AMC) further strengthens the claim.

However, the experiments do not establish precisely which aspect of local normalization causes the problem. GRPO does three things simultaneously: subtracts the group mean, divides by the group standard deviation, and uses k3 for KL estimation. REINFORCE++ changes all three (global mean, global standard deviation, k2 KL). The overfitting experiments compare the full GRPO recipe against the full REINFORCE++ recipe—this is a system-level comparison, not a controlled ablation. It is possible that fixing only the KL estimator (from k3 to k2) while keeping local normalization would substantially reduce overfitting, or that fixing only the standard deviation (from local to global) while keeping the k3 KL would suffice. Without component-level ablations, the paper's claim that local normalization is the root cause is supported by theory but not isolated empirically.

The PPO-beating claim has fragile support. Table 4 is the only experiment comparing against PPO, and it shows REINFORCE++_w/_Baseline beating PPO by 2.25 points on average. However:

• This is a single model family (Qwen 2.5 Base 7B) on a single task type (math tool-use) with a single PPO implementation whose tuning is not described. The paper does not report PPO's KL divergence, training stability, or whether the PPO run suffered from any implementation issues. A fair comparison would require demonstrating that the PPO baseline was reasonably well-tuned, not just that one PPO run underperformed.

• The average advantage (+2.25) is driven heavily by AIME 2025 (+5.52 over PPO). On the other four benchmarks, the advantage is +0.63 (AIME 2024), +2.91 (HMMT 2025), +0.52 (HMMT 2024), +1.67 (CMIMC)—a more modest but still positive margin. The sensitivity to AIME 2025's inclusion raises the question of whether the result would replicate on a different set of benchmarks or whether AIME 2025 happened to be particularly favorable.

• The paper does not compare training cost. REINFORCE++_w/_Baseline is claimed to be cheaper (no critic), but no FLOPs, GPU-hours, or wall-clock time comparisons are provided. If REINFORCE++_w/_Baseline requires more training steps to converge (as Figure 3 suggests—global normalization learns more gradually), the per-step savings from removing the critic might be offset by requiring more steps. A proper comparison would match total compute budgets.

• Missing baseline: PPO with global normalization. If global normalization is the key innovation, a natural question is whether adding it to PPO would improve PPO's performance rather than replacing PPO entirely. This ablation is not tested.

The general RLHF results are suggestive but incomplete. Table 1 shows REINFORCE++ ($k = 1$) matching GRPO ($k = 4$) on Chat-Arena-Hard with better per-token efficiency and lower KL divergence. This supports the claim that $k = 1$ with global normalization is competitive with group-sampling methods for diverse instruction-following tasks. However:

• The absolute score differences are tiny (46.7 vs. 46.8) and reported without confidence intervals. It is entirely possible that these results are within run-to-run noise, especially given that LLM-judge benchmarks like Chat-Arena-Hard are known to have non-trivial variance.
• The paper does not compare against a $k = 1$ baseline with local normalization—which is, admittedly, impossible since local normalization requires $k \geq 2$. The comparison is therefore between fundamentally different training regimes (single-sample vs. group-sampling) rather than between normalization strategies at fixed $k$.
• The per-token efficiency metric (Score / Length) is a creative way to penalize length exploitation, but it assumes a linear relationship between length and quality that may not hold. A response that is 10% longer but 20% better would be penalized by this metric.

The KL estimator claim is not empirically tested. Appendix B.1 provides a careful theoretical argument for k2 over k3, but zero experiments compare them. The paper never shows GRPO-with-k2, REINFORCE++_w/_Baseline-with-k3, or any other permutation. The claim that k3's variance is problematic and k2 is stable is plausible given the theory, but its practical impact on training outcomes is unquantified. Given that the paper's REINFORCE++_w/_Baseline uses both global normalization AND k2, it is impossible to attribute performance improvements to either change individually.

Missing: Sensitivity to reward model quality. The theoretical bias of local normalization depends on the distribution of rewards within a group. If the reward model is highly accurate and produces well-separated scores, local standard deviations will be larger and the bias less severe. If the reward model is noisy or poorly calibrated, local standard deviations will be smaller and the bias worsens. The paper uses different reward setups (Bradley-Terry model for general RLHF, rule-based binary rewards for math reasoning), but does not systematically vary reward model quality to test whether global normalization's advantage grows as reward quality degrades. This would be a practically important interaction—in real deployments, reward model quality varies substantially.

Missing: Scalability to larger models. All experiments use 7-8B parameter models. The paper's motivation emphasizes that critic-free methods are important for enabling RLHF on smaller hardware, but it does not test whether global normalization remains effective at larger scales (e.g., 70B+ parameters) where PPO's critic overhead is proportionally smaller (the critic can be a smaller model or use weight sharing). It is plausible that at very large scales, the critic's value estimate becomes accurate enough that PPO's advantage estimation outperforms the simple global normalization baseline, and the paper's claimed advantage over PPO would narrow or reverse.

The paper's strongest result is the overfitting prevention, not the PPO comparison. Table 2 (95% → 0% vs. 71% → 2.5%/40%) and Figure 4 (K&K scaling) demonstrate a qualitative difference in training dynamics that is unlikely to be explained by hyperparameter tuning or benchmark variance. Even if the PPO-beating claim weakens under scrutiny, the demonstration that local normalization causes catastrophic overfitting in low-data regimes is a genuine contribution with immediate practical implications: practitioners training on small curated datasets (a common scenario in specialized domain adaptation) should avoid GRPO-style local normalization. The paper's recommendation to use global normalization for such settings is well-supported by the empirical evidence, even if the precise mechanism (standard deviation vs. KL estimator vs. both) is not fully isolated.

6. Limitations and Trade-offs

6.1 The PPO-Beating Claim Rests on a Single Experiment Without Documented PPO Tuning

The assumption or constraint. The paper's most consequential positive claim—that REINFORCE++_w/_Baseline outperforms full-critic PPO—is supported by exactly one experiment (Table 4, Section 4.3) using Qwen 2.5 Base 7B on math tool-use benchmarks in the ZeroTIR environment. The paper does not describe the PPO implementation: no hyperparameters (learning rate, clipping ϵ, GAE λ, γ, number of PPO epochs per batch, value loss coefficient), no training budget (steps, total tokens, GPU-hours), no monitoring metrics (value loss, explained variance, KL divergence), and no evidence that the PPO run was well-tuned rather than using default settings from OpenRLHF.

The consequence. A reader cannot determine whether REINFORCE++_w/_Baseline genuinely outperforms a well-implemented PPO baseline or whether the PPO run in Table 4 was suboptimal. PPO is notoriously sensitive to hyperparameters—the critic learning rate, number of critic updates per policy update, value function clipping, and GAE parameters all significantly affect performance. If the PPO baseline used poorly-tuned defaults, the reported +2.25 average advantage (24.10 vs. 21.85) may overstate REINFORCE++'s relative performance. Conversely, if PPO was well-tuned, the result is genuinely impressive but not reproducible without documentation. The sensitivity is compounded by the fact that the largest advantage (+5.52 over PPO) occurs on AIME 2025—a single benchmark whose exclusion reduces the average gap to +0.44 points, making the claim fragile to benchmark selection.

What evidence exists in the paper. Table 4 reports PPO's per-benchmark scores alongside GRPO and REINFORCE++_w/_Baseline. Section 4.3 states that training "adhere[s] to the training and evaluation protocols of the ZeroTIR environment" and uses OpenRLHF with datasets from ORZ and DAPO. No PPO-specific details are provided. The paper does not report PPO training curves (reward, KL, value loss), making it impossible to assess whether the PPO run converged properly or suffered from critic underfitting/overfitting. Section 3.3's claim that REINFORCE++_w/_Baseline is "formally equivalent to a PPO agent where [...] the critic network is removed" is a theoretical equivalence, not an empirical demonstration that the simplified version performs better.

Mitigation status. Not addressed. The paper does not acknowledge the thinness of the PPO comparison, does not discuss PPO tuning, and does not report confidence intervals or multiple PPO runs with different hyperparameters. No ablation tests whether PPO with global normalization would close the gap. The paper's Section 8 (Conclusion) states REINFORCE++_w/_Baseline "outperform[s] both GRPO and full-critic PPO in complex, long-horizon tool-use tasks" as a headline finding without caveats about the single-experiment basis.

6.2 The KL Estimator Claim Lacks Empirical Validation

The assumption or constraint. Appendix B.1 provides a theoretical argument that the k3 KL estimator (used in GRPO) estimates the wrong KL direction (Forward KL instead of Reverse KL) and has "infinite variance" due to the importance ratio π_ref/π_θ exploding, while the k2 estimator is "the theoretically correct and stable choice." This analysis is used to justify REINFORCE++_w/_Baseline's use of k2 as a separate loss term. However, no experiment in the paper isolates the KL estimator's effect. Every comparison between REINFORCE++ variants and GRPO changes multiple components simultaneously: global vs. local standard deviation, global vs. local mean, k2 vs. k3 KL, and possibly other implementation details (reward clipping, batch construction).

The consequence. It is impossible to determine how much of REINFORCE++'s improved stability and generalization comes from global normalization versus the k2 estimator change. A practitioner deciding whether to adopt REINFORCE++_w/_Baseline or simply modify their existing GRPO implementation cannot assess whether switching only the KL estimator (a simpler change than redesigning the normalization pipeline) would capture most of the benefit. Conversely, if k2 is genuinely crucial, the paper's REINFORCE++ variant (k=1), which uses k1 embedded in the reward rather than k2 as a loss, may be vulnerable to different KL-related failure modes that are not explored.

The paper's statement that k3's variance explains "why methods using k3 require frequent resetting of π_ref to prevent the policies from diverging" is a plausible hypothesis but is not tested—the paper does not show that switching to k2 eliminates the need for reference model resets in a controlled comparison.

What evidence exists in the paper. The KL analysis is purely theoretical (Appendix B.1). There is no ablation comparing GRPO-with-k2 against GRPO-with-k3, REINFORCE++_w/_Baseline-with-k3 against REINFORCE++_w/_Baseline-with-k2, or any other permutation. Table 4 compares full GRPO (local norm + k3) against full REINFORCE++_w/_Baseline (global norm + k2 + group mean subtraction), so the 1.52-point average improvement (24.10 vs. 22.58) is the combined effect of all changes. Figure 2 shows KL divergence curves only for REINFORCE++ (k=1, which uses k1 in reward, not k2 as loss) versus GRPO (which uses k3 as loss)—a comparison that confounds normalization strategy and KL estimator.

Mitigation status. Not addressed. The paper presents the KL analysis as a contribution ("we adopt the k2 estimator [...] the k2 estimator provides a stable, unbiased gradient") but never validates it empirically. A reader must take on faith that the theoretical variance problem with k3 manifests in practice and that k2 solves it. The paper also does not cite empirical evaluations of k2 from prior work (the Liu et al., 2025a reference is listed in the appendix but its empirical content is not discussed).

6.3 Batch Size Requirements Are Not Characterized, and Small-Batch Behavior Is Unknown

The assumption or constraint. The entire theoretical justification for global normalization rests on asymptotic behavior: "As the global batch size grows (e.g., N = 1024), the batch mean and standard deviation converge to constants, resulting in an effectively unbiased (bias vanishes as N → ∞) and low-variance advantage estimator" (Section 3.4). The paper uses batch sizes of "typically 1024 or more" (Section 3.1) but never varies batch size experimentally to test sensitivity. The proof in Appendix A.2 states that "as N_global → ∞, the sample statistics μ_b and σ_b converge to true constants," but does not quantify the bias at finite N.

The consequence. Practitioners with limited GPU memory who cannot fit batch size 1024 receive no guidance. At what batch size does global normalization become "stable enough"? The paper's own bias proof shows that local normalization with N = 4 to 8 is biased because the denominator depends on the same small set of samples as the numerator. Global normalization with N = 64 or N = 128 uses more samples, but the same dependency exists—the bias does not magically disappear at some threshold, it decays continuously as N increases. Without empirical characterization, a practitioner using batch size 128 (common for 7B models on consumer GPUs) does not know whether they are getting the "effectively unbiased" benefits claimed in the paper or whether residual bias still degrades training.

This is particularly important because the paper's efficiency argument for REINFORCE++ (k=1) is that it uses fewer generations per step than GRPO (k=4). But REINFORCE++ with k=1 and batch size 1024 processes 1024 prompts per step, while GRPO with k=4 and batch size 256 processes the same 1024 responses (256 prompts × 4 responses). If a practitioner running GRPO with k=4 uses batch size 256, they get 1024 samples for global normalization—but the paper doesn't test whether global normalization at this effective sample size (coming from fewer unique prompts) is equivalent.

What evidence exists in the paper. None. No experiment varies batch size. No figure shows training dynamics or final performance as a function of batch size. No table reports results at different batch sizes. The paper's third-party validation citations (LitePPO, ScaleRL, DLER in Section 5.2) are summarized qualitatively without batch size details. The theoretical analysis provides no finite-sample bias bounds—only the asymptotic claim that bias "vanishes as N → ∞."

Mitigation status. Not addressed or acknowledged as a limitation. The paper treats "batch size 1024" as a given without discussing the practical constraints that might prevent its use or the expected degradation at smaller scales.

6.4 All Experiments Use a Single Model Scale (7-8B Parameters)

The assumption or constraint. Every experiment in the paper uses models in the 7-8B parameter range: Llama-3-8B-SFT for general RLHF (Section 4.1), Qwen 2.5 Base 7B for multi-step tool-use (Section 4.3), and Qwen2.5-Math-Base for reasoning (Section 4.2). The paper's motivation emphasizes that critic-free methods are valuable because PPO's critic "makes training very expensive and limits large model alignment in small-scale clusters" (Section 1). This motivation implies that the methods should work at scales where the critic's overhead is most painful—but the paper never tests at those scales.

The consequence. It is unknown whether the findings generalize to larger models (e.g., 70B parameters) or smaller models (e.g., 1-3B). Several mechanisms could cause the relative performance of global vs. local normalization to change with scale:

• At larger scales, the policy is more capable and may produce more diverse responses per prompt, increasing the local standard deviation and reducing local normalization's bias. If a 70B model naturally generates varied responses to the same prompt (unlike a 7B model that may produce similar answers), the k = 4 local standard deviation might be more informative and the bias less severe.
• At larger scales, PPO's critic overhead is proportionally smaller if weight-sharing or a smaller critic architecture is used. The paper's claimed advantage over PPO might narrow or reverse if the critic can be made efficient at scale.
• At smaller scales, the policy is less capable and may produce near-identical responses per prompt (all equally bad), making local normalization's division-by-near-zero problem more acute. Global normalization's benefits might be even larger at small scales, which would strengthen the paper's practical value but is untested.

Additionally, the interaction between model scale and batch size is unexplored. Larger models require more GPU memory per sample, forcing smaller batch sizes, which may weaken global normalization.

What evidence exists in the paper. None at other scales. All tables and figures report results from 7-8B models. The paper does not justify the choice of this scale beyond stating that Qwen 2.5 Base 7B is used "in the zero-shot agent setup" (Section 4.3). The general RLHF experiments use Llama-3-8B-SFT without comment on why 8B was chosen. No scaling trends are reported.

Mitigation status. Not acknowledged. The paper does not discuss model scale as a limitation or suggest that results should be validated at other scales. Section 8 (Conclusion) makes general claims ("state-of-the-art generalization in general RLHF," "dramatic improvements in stability") without qualifying the model scale at which these claims were demonstrated.

6.5 The Computational Cost of Global Normalization Relative to Local Normalization Is Not Quantified

The assumption or constraint. The paper claims REINFORCE++ "retains the critic-free architecture, significantly reducing computational and memory overhead compared to PPO" (Section 3.4). This claim compares against PPO, not against other critic-free methods like GRPO. The paper does not quantify the cost of global normalization relative to GRPO's local normalization within the critic-free family. Global normalization requires gathering all advantages from the entire batch (potentially across multiple GPUs in distributed training) to compute global mean and standard deviation, while local normalization computes statistics independently per prompt (requiring no cross-prompt communication).

The consequence. In distributed training settings where different prompts are processed on different GPUs, global normalization introduces an all-reduce communication step that local normalization avoids. For each training batch, every GPU must share its local advantage sums and counts with all other GPUs to compute the global mean and standard deviation, then receive the global statistics back to normalize its local advantages. This communication overhead is typically small relative to the forward/backward passes through a 7B-parameter model, but it is non-zero and scales with the number of GPUs. The paper provides no measurements of this overhead, no comparison of wall-clock time per training step between REINFORCE++ and GRPO, and no analysis of how the overhead scales with distributed training configuration.

Additionally, global normalization may require larger batch sizes to be effective (see Limitation 6.3), which increases memory usage per training step. If GRPO can train stably with batch size 256 (64 prompts × 4 responses) and REINFORCE++ requires batch size 1024, the latter requires 4× more GPU memory, potentially forcing the use of gradient accumulation or reducing the model size that fits on the same hardware.

What evidence exists in the paper. None. The paper reports only model quality metrics (scores, accuracies, Pass@N), never training throughput, memory usage, or wall-clock time. Table 1's "Per Token" metric (Score / Length) measures generation efficiency at inference time, not training efficiency. The third-party ScaleRL citation (Section 5.2) is summarized as finding REINFORCE++ "slightly superior in both compute efficiency and final performance," but the paper does not report ScaleRL's efficiency numbers, and "compute efficiency" in that context may refer to sample efficiency (reaching a given performance with fewer training steps) rather than wall-clock or FLOPs efficiency.

Mitigation status. Partially addressed in the paper's framing but not empirically. Section 3.1 states that REINFORCE++ in its k = 1 configuration is "designed for maximum efficiency and prompt diversity," and Section 3.4 claims "it retains the critic-free architecture, significantly reducing computational and memory overhead compared to PPO." These statements are about the absence of a critic, not about the cost of global vs. local normalization. The paper does not discuss the communication cost of all-reduce for global statistics, does not measure training throughput, and does not provide FLOPs comparisons between REINFORCE++ variants and GRPO.

6.6 Experiments Are Confined to Mathematical and Logical Reasoning; No Evidence in Other RLHF Domains

The assumption or constraint. All experiments demonstrating REINFORCE++'s advantages over baselines involve tasks with objective, verifiable correctness: mathematical problem-solving (AIME, MATH, AMC, HMMT), logical puzzles (K&K), or tool-use for math (ZeroTIR). The only non-math experiment is the general RLHF evaluation on Chat-Arena-Hard (Section 4.1), where REINFORCE++ essentially ties GRPO (46.7 vs. 46.8) rather than demonstrating a clear advantage. The paper's strongest results—catastrophic overfitting prevention (Table 2), OOD generalization (Table 3, Figure 4), and PPO-beating (Table 4)—all come from math/logic domains.

The paper does not evaluate on other important RLHF domains: instruction following with nuanced quality criteria, safety alignment (refusal of harmful requests), multi-turn dialogue, creative writing, summarization, code generation with functional correctness, or any task where the reward signal is a learned neural network rather than a rule-based binary checker.

The consequence. The paper's findings may not transfer to domains where reward signals have fundamentally different statistical properties. Mathematical reasoning with binary (0/1) rewards creates a specific distribution: for any given prompt, rewards are often all-zero (all responses wrong) or mixed (some right, some wrong). The local standard deviation in the all-zero case is exactly zero, triggering the worst-case failure mode of local normalization. In contrast, learned reward models (for instruction following, helpfulness, safety) typically produce continuous, roughly Gaussian-distributed scores where the local standard deviation is rarely near zero. Local normalization's bias and instability might be much less severe in these domains, potentially eliminating REINFORCE++'s advantage.

Conversely, the math-focused results may overstate REINFORCE++'s generalization benefits. The overfitting from local normalization (Table 2) may be particularly acute for math problems because the model can memorize problem-specific solution templates (solution structure for "find the number of ways to arrange X" transfers poorly to "solve this geometry problem"). In more open-ended domains like dialogue, the distinction between memorization and generalization is blurrier—learning to be polite or helpful may generalize well even if the training signal is prompt-specific.

What evidence exists in the paper. Table 1 (Chat-Arena-Hard) is the only non-math evaluation, and it shows REINFORCE++ (46.7) statistically tied with GRPO (46.8) and barely ahead of ReMax (45.1). This is a much weaker demonstration than the math results. The paper does not evaluate on standard RLHF benchmarks like AlpacaEval, MT-Bench, or human preference evaluations. The paper does not test safety alignment, where the reward signal (refusal vs. compliance) is binary but the consequences of overfitting (learning to always refuse or always comply) differ from math overfitting.

Mitigation status. The paper does not acknowledge the domain restriction as a limitation. Section 4.1's title "General RLHF" implies broader applicability than the single Chat-Arena-Hard benchmark supports. The paper's recommendation in Section 5.1 that REINFORCE++ "(k = 1) is designed for general-domain RLHF" is based on a single experiment that shows parity, not superiority. The paper does not discuss how reward model characteristics (binary vs. continuous, sparse vs. dense, rule-based vs. learned) might interact with the global vs. local normalization choice.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a fundamentally new algorithmic paradigm—it does not propose a novel policy gradient method or a new RL framework. Instead, it makes a more specific and arguably more immediately actionable contribution: it identifies a single design flaw (local advantage normalization) as the root cause of three seemingly distinct failure modes in critic-free RLHF, and shows that a simple, known technique (global batch normalization) resolves all three simultaneously. This is best understood as a diagnostic contribution that reframes the conversation around critic-free methods from "how do we patch GRPO's instability?" to "local normalization should not be used for advantage scaling, and here is why."

The magnitude of this shift becomes clear when examining the paper's most dramatic result: GRPO trained on 30 AIME-24 problems achieves 95.0% training accuracy and 0.0% Pass@1 on AIME-25, while REINFORCE++ on the same data achieves 71.0% training and 40.0% Pass@16 on the test set (Table 2). This is not a marginal improvement—it is the difference between a method that catastrophically memorizes and one that learns transferable reasoning skills. If this result generalizes (and the consistent pattern across K&K puzzles in Figure 4, AIME/AMC in Table 3, and the tool-use benchmarks in Table 4 suggests it does), then the dominant open-source RLHF algorithm (GRPO) has a fundamental flaw that makes it unsuitable for small-dataset fine-tuning, and REINFORCE++ provides a drop-in fix.

The paper also resolves a tension that has existed implicitly in the RLHF literature. On one hand, GRPO has been widely adopted because it is cheaper than PPO and has produced strong results on reasoning benchmarks (notably in DeepSeekMath and related work). On the other hand, practitioners have observed that GRPO training can be unstable, requiring frequent reference model resets, reward clipping, and careful learning rate tuning. Prior work treated these as separate engineering issues—the KL estimator is unstable (hence the resets), the reward scale varies (hence the clipping), the advantage variance is high (hence the small learning rates). This paper provides a unified explanation: all these symptoms stem from local normalization. The frequent reference model resets are necessary because the k3 KL estimator's variance explodes as the policy diverges (Appendix B.1). The reward clipping is necessary because local standard deviations approach zero, causing advantage explosion. The sensitivity to learning rate is necessary because the bias in local normalization systematically distorts the advantage signal, requiring careful tuning to prevent the policy from overfitting to the distorted signal.

By providing this unified explanation and a simple fix (global normalization + k2 KL), the paper effectively moots a class of engineering patches that the field had accumulated. Reference model resets, aggressive reward clipping, and learning rate annealing to manage GRPO's instability become unnecessary when the root cause is addressed. This is a meaningful contribution to engineering practice even if the theoretical insights were partially anticipated—it tells practitioners which knobs they can stop worrying about.

The paper also indirectly strengthens the case for RL-based alignment over direct preference optimization (DPO). DPO has gained traction partly because it avoids RL's instability entirely—no reward model, no policy gradient, no advantage estimation. By making RLHF more stable through critic-free training with global normalization, the paper reduces one of the key practical arguments for DPO. If REINFORCE++ can match or exceed PPO's performance at substantially lower computational cost (as Table 4 suggests, with REINFORCE++_w/_Baseline at 24.10 vs. PPO at 21.85 on the tool-use benchmarks), then the efficiency gap between RL and DPO narrows, and the claimed benefits of RL (online exploration, reward model flexibility, compatibility with iterative self-improvement) become more accessible.

However, the paper's influence on the PPO vs. critic-free debate is more nuanced than "critic-free methods are now strictly better." The paper demonstrates that REINFORCE++_w/_Baseline beats PPO on one specific task configuration (math tool-use with Qwen 2.5 Base 7B), but acknowledges (Section 3.3) that this variant is "formally equivalent to a PPO agent where: (1) The critic network is removed; (2) The GAE parameters are set to λ = 1 and γ = 1; and (3) A two-step global batch normalization is used as the baseline instead of a learned value function." Setting γ = 1 and λ = 1 in GAE means there is no temporal discounting and no eligibility trace—the advantage reduces to the raw reward (minus baseline). This simplification works for the math problems tested (where only the final answer matters) but may fail for tasks with intermediate rewards or long horizons where temporal credit assignment is nontrivial. The paper does not test tasks where GAE with γ < 1 or λ < 1 would be beneficial, so the claimed equivalence to PPO is domain-conditional.

What the paper convincingly demonstrates is that for tasks where only the final output matters, global normalization can replace the entire critic apparatus, and that local normalization actively harms generalization. This is a significant narrowing of the regime where PPO's complexity is justified. Prior to this work, the default assumption was that PPO was the "gold standard" and critic-free methods were approximations that traded performance for efficiency. This paper suggests that for a broad class of alignment tasks (instruction following, mathematical reasoning, tool use where reward is outcome-based), the critic-free approach with global normalization may actually be the better choice, not just the cheaper one.

Follow-Up Research This Work Enables

Stress-testing the overfitting claim across reward model types. The paper's most dramatic result—GRPO's 95% training / 0% test overfitting on 30 AIME problems—uses binary (0/1) rule-based rewards. The theoretical bias proof (Appendix A) depends on the reward distribution through the conditional variance of $D^2$, but does not assume binary rewards. A crucial stress-test would replicate the small-dataset overfitting experiment with continuous learned reward models (e.g., a Bradley-Terry preference model, or a fine-grained rubric-based scorer) to determine whether local normalization's bias causes catastrophic overfitting only when rewards are sparse and discrete, or whether the phenomenon generalizes. The prediction from the paper's theory is that any reward distribution where the local standard deviation is small relative to the mean (i.e., where within-prompt reward variance is low) will amplify the bias, but the catastrophic 0% test accuracy may be specific to binary rewards where "all wrong" is a common case. An experiment training on 30-50 Chat-Arena-Hard prompts with a learned reward model, evaluating on held-out prompts, and comparing GRPO vs. REINFORCE++ generalization would directly test this.

Varying batch size to map the bias-variance tradeoff of global normalization. The paper proves that the GRPO estimator's bias vanishes as the normalization group size $N \to \infty$ (Appendix A.2) and uses batch sizes of "typically 1024 or more," but provides no empirical characterization of finite-sample behavior. A systematic experiment training REINFORCE++ (k = 1) on a fixed task (e.g., MATH) while varying the global batch size from 32 to 2048, measuring both final accuracy and training stability (KL divergence, reward variance, number of NaN/inf occurrences), would establish the practical minimum batch size for "effectively unbiased" behavior. This is directly actionable for practitioners with GPU memory constraints. A negative result—finding that batch size 128 performs nearly as well as 1024—would actually strengthen the paper's contribution by showing the method is more robust than the asymptotic theory requires. Conversely, finding a sharp degradation below some threshold would provide a concrete deployment guideline.

Isolating the KL estimator change from the normalization change. The paper's REINFORCE++_w/_Baseline changes both the normalization strategy (local standard deviation → global standard deviation) and the KL estimator (k3 → k2), but never ablates them independently. A controlled experiment comparing four conditions on the same math reasoning task (e.g., AIME training with rule-based rewards): (a) GRPO (local norm + k3), (b) GRPO-with-k2 (local norm + k2 KL), (c) REINFORCE++_w/_Baseline-with-k3 (global norm + group mean subtraction + k3 KL), (d) REINFORCE++_w/_Baseline (global norm + group mean subtraction + k2 KL). This would decompose the 1.52-point average improvement in Table 4 into components attributable to each change. The paper's theory predicts that (c) should improve over (a) primarily on generalization (less overfitting), while (b) should improve primarily on stability (lower KL divergence, fewer reference model resets needed), and (d) should combine both benefits. If (b) alone captures most of the improvement, the paper's emphasis on global normalization would need recalibration. This experiment requires only implementing the KL estimator variants within an existing GRPO codebase and is logistically straightforward.

Testing REINFORCE++ with process reward models (PRMs). The paper mentions in Section 5.1 that REINFORCE++ (k = 1) is recommended "for tasks where obtaining multiple reward signals for distinct responses is challenging—such as training with Process-Supervised Reward Models (PRMs)." PRMs assign per-step correctness scores rather than a single outcome reward, creating a fundamentally different advantage structure where multiple tokens receive non-zero advantages (unlike the bandit setting where only the final token gets the normalized advantage, as described in Appendix B.2). The interaction between global normalization and per-step rewards is non-obvious: should the global mean and standard deviation be computed over all token-level advantages (mixing early-step and late-step advantages with potentially different scales), or should normalization be stratified by step position? An experiment training on a PRM-annotated math dataset (e.g., PRM800k) with REINFORCE++ (k = 1) and comparing against GRPO (k = 4) with the same PRM would test whether global normalization's benefits extend to process-level rewards or whether the bandit assumption (advantages only at final tokens) is load-bearing for the paper's results.

Scaling REINFORCE++ to larger models and comparing against well-tuned PPO at matched compute. The paper's PPO-beating claim (Table 4) uses a single 7B model with undocumented PPO tuning. A rigorous follow-up would train REINFORCE++_w/_Baseline and PPO on the same task (e.g., ZeroTIR math tool-use) at multiple model scales (1B, 7B, 30B, 70B) with systematic PPO hyperparameter tuning (sweeping GAE λ, critic learning rate, value loss coefficient) and matched training FLOPs budgets, measuring final performance and wall-clock time. The key question: does REINFORCE++'s advantage over PPO grow, shrink, or stay constant with model scale? The paper's theory predicts that as models get larger and more capable, within-prompt response diversity increases (the model can generate more varied solutions to the same problem), which would increase the local standard deviation and reduce local normalization's bias—potentially narrowing the gap. Conversely, if larger models are also more prone to reward hacking (exploiting reward model blind spots in more sophisticated ways), global normalization's stability benefit might grow. This experiment would also address whether the γ = 1, λ = 1 simplification (no temporal discounting) becomes more costly at larger scales where long-horizon credit assignment matters more.

Dynamic difficulty estimation for adaptive normalization. The paper shows that local normalization causes overfitting on easy prompts (where within-group variance is high and the relative-advantage signal is strong) while global normalization prevents it. This suggests a hybrid approach: use the global batch to estimate prompt difficulty (via the variance of rewards each prompt receives) and adaptively mix local and global normalization based on that estimate. For prompts with high within-prompt reward variance (easy prompts where the model can generate varied responses), use more global normalization weight to prevent overfitting. For prompts with low within-prompt variance (hard prompts where all responses are similarly poor), a small amount of local rescaling might help amplify the signal that distinguishes the least-bad response. This connects to the concept of difficulty-conditioned compute allocation from the prior example paper, but applied to normalization rather than search strategy. The specific experiment would compare fixed global normalization against this adaptive scheme on a dataset with mixed difficulty (e.g., MATH with all five difficulty levels) and measure both overall accuracy and per-difficulty-bin performance.

Practical Applications and Downstream Use Cases

Fine-tuning on small curated datasets for specialized domains. The paper's overfitting result (Table 2) has direct implications for organizations that collect small, high-quality datasets of expert demonstrations or preferences in specialized domains (medical diagnosis, legal reasoning, scientific research). With only 30-100 examples, GRPO's local normalization drives the model to memorize domain-specific patterns rather than learn transferable reasoning—exactly the opposite of what fine-tuning on expert data is intended to achieve. REINFORCE++ with global normalization enables training on these small datasets while preserving generalization. The practical recipe: when training on fewer than ~200 examples per task, use REINFORCE++ with k = 1 (to maximize prompt diversity per batch) and batch size ≥ 512. The 40.0% Pass@16 on AIME-25 from only 30 training examples demonstrates that meaningful generalization from tiny datasets is achievable when the normalization strategy is correct. Without global normalization, the same data produces a model that performs worse than random guessing on held-out problems.

Self-improvement pipelines with iterative data generation. In self-play or iterative self-improvement setups (where a model generates its own training data, filters for quality, and retrains), the reward distribution across generations fundamentally changes over iterations. Early iterations produce mostly incorrect responses with occasional successes, creating the low-variance local groups that trigger GRPO's worst failure modes. Later iterations may produce more consistently correct responses, shifting the reward distribution. REINFORCE++'s global normalization adapts automatically to these distribution shifts without requiring per-iteration tuning of clipping thresholds or reference model resets. The paper's finding that REINFORCE++ requires no reference model resets (unlike GRPO, where the k3 estimator forces periodic resets to prevent explosion) is particularly valuable here—in a multi-iteration self-improvement loop, resetting the reference model conflates progress from the current iteration with regression to the original policy, making it difficult to track genuine improvement. A concrete deployment scenario: training a code generation model via RLHF where each iteration generates solutions, executes tests, and uses test pass/fail as rewards. REINFORCE++ with k = 1 handles the sparse binary reward distribution without exploding gradients, and the global normalization provides stable training signals across iterations as the model improves.

On-device or edge deployment where the critic network is infeasible. The paper's primary practical contribution is enabling RLHF without a critic network, which roughly halves the memory required for training. For organizations deploying models on edge devices or consumer GPUs (where even a 7B model strains memory limits), this is the difference between being able to perform alignment training and not. REINFORCE++ (k = 1) with batch size 512-1024 on a 7B model can be done on hardware that cannot fit a 7B policy + 7B critic simultaneously. The general RLHF results in Table 1 show that this memory savings does not come at a performance cost—REINFORCE++ matches GRPO's score (46.7 vs. 46.8) on Chat-Arena-Hard while using fewer generations and producing more token-efficient outputs. For a practitioner with a single 8-GPU node training a 7B model, switching from GRPO (k = 4) to REINFORCE++ (k = 1) frees approximately 4× of the generation budget to process more diverse prompts per step, and the global normalization stabilizer ensures these additional prompts contribute to generalization rather than noise.

Alignment training for math and code reasoning models. The paper's results on mathematical reasoning (Tables 2, 3, 4; Figures 3, 4) are directly applicable to the rapidly growing field of reasoning-focused LLM training. The standard recipe for training reasoning models (as in DeepSeek-R1, Seed-Thinking, and related work) involves RL fine-tuning with rule-based binary rewards on math and code problems—exactly the setting where the paper demonstrates GRPO's overfitting and REINFORCE++'s generalization benefits. The practical recommendation is clear: when training reasoning models with outcome-based rewards on math or code, replace GRPO's local normalization with REINFORCE++_w/_Baseline (if using k > 1 group sampling) or REINFORCE++ (if using k = 1 single-sample training). The expected benefit is not just higher final accuracy (the +1.52 average improvement over GRPO in Table 4) but more importantly, better out-of-distribution generalization to harder or future problems (as demonstrated by the +5.31 advantage on AIME 2025, the temporally OOD benchmark). For teams training reasoning models where evaluation on future competition problems is the true test of capability (not just training-set performance), the generalization improvement alone justifies the switch.

When to Prefer This Method

The paper explicitly articulates the tradeoff between its two variants in Section 5.1 ("General Principle"), and also positions both against GRPO and PPO. The decision rule it recommends is:

• Prefer REINFORCE++ (k = 1) when prompt diversity and efficiency are paramount, or when obtaining multiple reward signals per prompt is difficult. Specific scenarios include: general-domain instruction-following RLHF (where the reward model provides informative continuous scores from single responses), training with process-supervised reward models (where per-step rewards already provide rich signals and generating multiple complete trajectories is expensive), and online real-time sampling (where latency constraints prevent batching multiple responses). The paper's Table 1 results (46.7 score on Chat-Arena-Hard with 4× fewer generations than GRPO) directly support this recommendation. The method also requires symmetric rewards (e.g., -1/1 in RLVR tasks) for best performance, per Section 5.1.

• Prefer REINFORCE++_w/_Baseline (k > 1) when group sampling is beneficial for the task structure, specifically: complex multi-step reasoning where reward signals are sparse (binary 0/1) and group mean subtraction provides automatic reward reshaping across prompts with different difficulty baselines; tasks with a high proportion of void or non-informative samples (as in multi-turn tool-calling, where the group mean subtraction "effectively filter[s] out void samples" by centering rewards); and scenarios supporting both 0/1 and -1/1 reward schemes, since the group mean subtraction makes the method robust to reward scale. The Table 4 results (24.10 average across tool-use benchmarks, outperforming both GRPO at 22.58 and PPO at 21.85) support this for complex agentic tasks.

• Prefer REINFORCE++_w/_Baseline over PPO when the critic overhead is prohibitive and the task does not require sophisticated temporal credit assignment. The paper explicitly frames the Baseline variant as "a simplified and more stable variant of PPO" with γ=1, λ=1, and no learned value function (Section 3.3). This equivalence holds for tasks where only final outcomes matter (math problem correctness, tool-use success, instruction following quality). For tasks with intermediate rewards or long horizons where discounting is beneficial (e.g., multi-turn dialogue with turn-level feedback, game playing with intermediate scoring), PPO's GAE machinery may still be necessary, and the paper does not test REINFORCE++ in such settings.

• Avoid GRPO's local normalization (with k3 KL) for small-dataset fine-tuning or any setting where out-of-distribution generalization is critical. The paper's overfitting demonstration (Table 2: 95% training → 0% test) and theoretical bias proof (Appendix A) provide both empirical and mathematical justification for this recommendation. Even when group sampling is used (k > 1), the local standard deviation should be replaced with global batch statistics, as in REINFORCE++_w/_Baseline.

The paper does not provide guidance on choosing between REINFORCE++ (k > 1) and REINFORCE++_w/_Baseline (k > 1) when group sampling is already decided. Both use global normalization, but the Baseline variant adds explicit group mean subtraction before global normalization. The paper's experiments never compare these two directly on the same task—Table 2-3 use REINFORCE++ (k > 1, no group mean subtraction) for reasoning, while Table 4 uses REINFORCE++_w/_Baseline (with group mean subtraction) for tool-use. The implicit recommendation from the task assignments appears to be: use group mean subtraction when reward scales vary dramatically across prompts or when void sample filtering is needed; omit it when the reward scale is already consistent (as in binary 0/1 math rewards, where the global mean already provides a reasonable baseline). But this is inference from the experimental design, not an explicit paper claim.