ArXiv: 2507.20673

🎯 Pitch

Swapping one mean for another boosts stability and exploration in LLM reinforcement learning—simply replacing the arithmetic with the geometric mean of token-level rewards increases reasoning accuracy by up to 4.1% by naturally suppressing outliers that destabilize training, eliminating the trade-off between aggressive exploration and safe clipping.

1. Executive Summary

This paper proposes Geometric-Mean Policy Optimization (GMPO), a plug-and-play variant of Group Relative Policy Optimization (GRPO) that replaces the arithmetic mean of token-level rewards with the geometric mean to suppress outlier-driven instability during reinforcement learning fine-tuning of large language models. Across five mathematical reasoning benchmarks (AIME24, AMC, MATH500, Minerva, OlympiadBench) using Qwen2.5-Math and DeepSeek-R1-Distill-Qwen architectures, GMPO improves average Pass@1 over GRPO by up to 4.1% (63.4% vs. 59.3% with the 7B R1-Distill model) while maintaining higher token entropy and lower KL divergence from the reference model throughout training, indicating sustained exploration and greater policy stability. The geometric mean's robustness to extreme importance sampling ratios (individual token probability ratios between current and old policies) enables GMPO to use a substantially wider clipping range—(e⁻⁰·⁴, e⁰·⁴) versus GRPO's (0.8, 1.2)—establishing that stable policy optimization can coexist with aggressive exploration only when the underlying objective is itself outlier-resistant rather than relying solely on clipping constraints for stability.

2. Context and Motivation

The Core Problem: GRPO Training Exhibits Destabilizing Outlier Tokens

The central problem this paper addresses is deceptively specific but operationally consequential: during GRPO training (Shao et al., 2024; Guo et al., 2025a), individual tokens can develop extreme importance sampling ratios that destabilize the entire policy optimization process. To understand why this matters, we need to unpack what an "importance sampling ratio" is in this context and why extreme values are harmful.

In GRPO, the objective function weights each token's contribution to the loss by a ratio ρt(θ)=πθ(ot∣q,o<t)πθold(ot∣q,o<t)\rho_t(\theta) = \frac{\pi_\theta(o_t|q, o_{<t})}{\pi_{\theta_{\text{old}}}(o_t|q, o_{<t})} — the probability the current model assigns to that token divided by the probability the old model assigned to it when the training data was generated. This ratio serves as a correction factor: since training rollouts were sampled from the old policy, the model must adjust its updates as if the data came from the current policy, and ρt(θ)\rho_t(\theta) performs that adjustment. When ρt(θ)≈1\rho_t(\theta) \approx 1, the old and current policies agree on that token's likelihood, and the update is well-calibrated. When ρt(θ)\rho_t(\theta) deviates substantially from 1 — say, 0.01 or 100 — the token exerts vastly disproportionate influence on the gradient, either vanishingly small or explosively large.

The paper's Figure 1 (right panel) makes this concretely visible: during GRPO training, the range of ρt(θ)\rho_t(\theta) expands dramatically as training progresses, with individual tokens reaching ratios far from 1. These are "outlier tokens" — words or sub-words whose probabilities changed drastically between the old and current policies. Because GRPO's objective uses the arithmetic mean of token-level rewards (Equation 2), and the arithmetic mean is sensitive to extreme values (Figure 2), a single outlier token can dominate the per-sequence loss and drive an aggressive, poorly-justified parameter update. This manifests as:

  • Unstable gradient norms (Figure 4c): the magnitude of parameter updates fluctuates wildly rather than smoothly converging.
  • Rapid entropy collapse (Figure 4a): the model prematurely becomes overconfident in its token predictions, losing the diversity needed to explore alternative reasoning paths.
  • Large KL divergence from the reference model (Figure 4d): the fine-tuned policy drifts far from the pre-trained base, risking catastrophic forgetting and degraded generalization.

The paper is fundamentally about breaking a stability-exploration tradeoff that prior work accepted as inevitable. The standard mitigation in GRPO is clipping — constraining ρt(θ)\rho_t(\theta) to a narrow range, typically (0.8,1.2)(0.8, 1.2), so that extreme ratios are truncated. But clipping has a dark side: it also caps legitimate exploration. When the model wants to meaningfully shift its probability distribution to explore new reasoning strategies, clipping prevents those shifts from being reflected in the loss, effectively locking the policy into conservative updates. As the paper cites from DAPO (Yu et al., 2025), "the clipping operation can limit exploration and cause early deterministic policy, which can hinder the scaling process." The core challenge, then, is: how can we achieve stable training without sacrificing exploration, when the instability arises from the objective function itself rather than from insufficient constraints?

Why This Problem Matters: GRPO Is a Dominant Paradigm with Fragile Stability

The significance of this problem extends far beyond a technical quirk in one RL algorithm. GRPO has become a foundational post-training method for LLM reasoning, particularly after DeepSeek-R1 (Guo et al., 2025a) demonstrated that reinforcement learning with verifiable rewards can induce sophisticated chain-of-thought reasoning behaviors — including self-verification, backtracking, and reflection — without explicit supervision on intermediate steps. The GRPO algorithm specifically is attractive because it eliminates the need for a separate value model (the "critic" in actor-critic RL), instead estimating advantages by comparing rewards within a group of sampled rollouts. This makes it computationally cheaper and simpler to implement than full PPO (Schulman et al., 2017), which has driven its rapid adoption across the open-source community (Section 2.1 catalogs well over a dozen GRPO variants: DAPO, Dr.GRPO, GPG, SRPO, OPO, EMPO, AAPO, BNPO, Seed-GRPO, GRPO-lead, CPPO, S-GRPO, Ada-GRPO, GVPO, GRPO-λ, PODS, RePO, each addressing different limitations).

However, this widespread adoption has surfaced a consistent pain point: GRPO training is finicky. Practitioners report that runs can collapse — entropy drops to near-zero, validation performance plateaus or declines, gradient norms spike — without clear diagnostic signals. The paper makes this concrete in Appendix B (Figure 5e): on the CountDown task with a Mixture-of-Experts model, GRPO's validation score "collapses after about 250 steps." This is not a hypothetical concern; it represents wasted GPU-hours and failed experiments in the many labs now running GRPO-style RL on their models.

The problem is especially acute in two practically important settings:

  1. Mixture-of-Experts (MoE) models (Section 4.2, Table 2 right, and Appendix B): MoE architectures route different tokens to different expert sub-networks, creating sparse activation patterns. These models are inherently more sensitive to training instability because individual expert modules can become over-specialized or under-utilized if gradient signals are noisy. The paper's results on Qwen3-32B (MoE) and on smaller MoE models in Appendix B demonstrate that GMPO's stability advantage is most pronounced precisely where GRPO is most fragile.

  2. Long-chain reasoning tasks: As language models are pushed toward longer chain-of-thought generation (3,000-token maximum response length in the paper's setup), the number of tokens per sequence grows, and with it, the probability that at least one token will have an extreme ρt(θ)\rho_t(\theta). The arithmetic mean's vulnerability to outliers means that a single problematic token in a 3,000-token reasoning chain can corrupt the gradient for the entire sequence, making long-form reasoning training particularly brittle.

The theoretical significance is equally important. The paper identifies a fundamental tension in the arithmetic mean as a training objective for sequence-level tasks. The arithmetic mean treats every token as equally influential on the loss. In reinforcement learning, this means tokens where the old and current policies agree strongly (benign tokens contributing stable gradients) get the same per-token weight as tokens where the policies diverge dramatically (potentially destabilizing tokens). The geometric mean, by contrast, is a "smoother" aggregator that naturally suppresses outlier influence because the nn-th root of the product reduces the impact of any single extreme factor. This insight — that the choice of mean matters not just for numerical stability but for the fundamental dynamics of policy optimization — has implications beyond GRPO to any RL algorithm that aggregates per-token contributions into a sequence-level objective.

Prior Approaches and Their Shortcomings

The paper situates itself within a rapidly expanding landscape of GRPO variants, each targeting different aspects of the training pipeline. Understanding these prior approaches is essential to appreciating what GMPO contributes that is genuinely novel rather than incremental.

The GRPO baseline (Shao et al., 2024; Guo et al., 2025a). GRPO's core innovation was eliminating the value model by computing advantages as normalized rewards within a group: A^i=ri−mean({r1,...,rG})std({r1,...,rG})\hat{A}_i = \frac{r_i - \text{mean}(\{r_1, ..., r_G\})}{\text{std}(\{r_1, ..., r_G\})}. This means a response's advantage depends on how it performed relative to other responses to the same question, not relative to some learned value estimate. This is computationally efficient and conceptually elegant, but it inherits the importance-sampling instability from PPO without the critic's stabilizing influence. GRPO applies clipping at the token level as its primary defense against instability, with standard thresholds of (0.8,1.2)(0.8, 1.2) (in the original DeepSeek-Math formulation; Shao et al., 2024).

DeepSeek-R1's sequence-level approach (Guo et al., 2025a). A notable variant — and one the paper explicitly critiques — is DeepSeek-R1's shift to sequence-level importance sampling and clipping. Rather than clipping each token's ρt(θ)\rho_t(\theta) individually, DeepSeek-R1 computes the product ∏t=1∣oi∣ρi,t(θ)\prod_{t=1}^{|o_i|} \rho_{i,t}(\theta) for the entire sequence and clips that single product. The paper identifies two problems with this approach (Section 3, design point (i)):

  • Sequence-level clipping is less stable than token-level (Figure 3): The GMPO-seq-clip variant shows a substantially wider range of importance sampling ratios compared to token-level GMPO, making it "more prone to create extreme gradients during optimization."
  • Sequence-level clipping is too aggressive when triggered: When the cumulative product exceeds the threshold, the gradients for all tokens in the sequence are set to zero, "potentially discarding valuable update signals from informative parts of rollouts." This is a sledgehammer solution — one extreme token at position 500 in a 3,000-token sequence nullifies the learning signal from the other 2,999 tokens that may have been providing useful gradient information.

DAPO's clip-higher strategy (Yu et al., 2025). Recognizing that narrow clipping limits exploration, DAPO proposed slightly expanding the clipping range from (0.8,1.2)(0.8, 1.2) to (0.8,1.28)(0.8, 1.28). This is a step in the right direction, but the paper's Figure 1 (right) shows why it's insufficient: during GRPO training, the importance sampling ratio range dynamically expands far beyond what even a (0.8,1.28)(0.8, 1.28) clip can contain. The problem isn't just the clipping threshold — it's that the objective function itself generates extreme ratios that require clipping in the first place. DAPO treats the symptom (narrow exploration) without addressing the cause (the arithmetic mean's sensitivity to outliers).

Dr.GRPO's length-aware approach (Liu et al., 2025). This variant, which the paper uses as its primary comparison baseline, addresses length bias in GRPO and removes the KL regularization term "for simplicity and memory saving" (Section 2.2). The paper follows Dr.GRPO in omitting the explicit KL penalty, which makes the comparison to GMPO cleaner — any stability advantage GMPO demonstrates cannot be attributed to differences in KL regularization, since neither method uses it. Dr.GRPO's contributions are orthogonal to GMPO's: it addresses what signal drives the reward (length-aware scoring), while GMPO addresses how that signal is aggregated across tokens (geometric vs. arithmetic mean).

The broader landscape of GRPO variants (Section 2.1). The paper's literature review is notably comprehensive, cataloging over 20 GRPO extensions. These can be grouped into several clusters, each targeting a different limitation:

  • Rollout selection and bias correction: SRPO (history resampling), DAPO (dynamic sampling), Dr.GRPO (length bias), OPO (optimal baseline). These improve which data the model learns from.
  • Reward shaping: EMPO (semantic entropy), AAPO (advantage momentum), BNPO (Beta normalization), Seed-GRPO (uncertainty scaling), GRPO-lead (length-dependent accuracy). These improve what signal drives the updates.
  • Efficiency: CPPO (pruning low-advantage completions), S-GRPO (early exit), Ada-GRPO (adaptive reasoning formats), GVPO (analytical KL weighting), GRPO-λ (dynamic length penalty switching), PODS (training on informative rollouts), RePO (replay buffers). These reduce how much computation is needed.
  • Exploration: The 80/20 rule (emphasizing high-entropy minority tokens), entropy-based advantage augmentation. These target what tokens receive emphasis during updates.

Critically, none of these prior methods address the fundamental instability that arises from the arithmetic mean itself. They all accept GRPO's core aggregation function — the arithmetic mean of token-level rewards — as given and work around its limitations through data filtering, reward rescaling, clipping, or other peripheral modifications. This is the gap GMPO fills. It is not another data selection or reward shaping method; it is a objective-function-level intervention that changes the fundamental mathematics of how token contributions are combined.

How GMPO Positions Itself: An Objective-Level Fix, Not a Constraint-Level Patch

The paper's positioning is clear and well-motivated: prior work tried to stabilize GRPO by constraining the optimization process (clipping, reward normalization, data filtering), while GMPO stabilizes it by changing what is being optimized. This is the conceptual distinction between treating symptoms and treating causes.

The geometric mean's key property — and the reason it justifies this positioning — is that it is inherently less sensitive to outliers than the arithmetic mean. This is not an empirical claim; it's a mathematical fact that the paper visualizes in Figure 2. Given a set of numbers {x1,...,xn}\{x_1, ..., x_n\}, the arithmetic mean 1n∑xi\frac{1}{n}\sum x_i can be pulled arbitrarily far from the typical values by a single extreme xix_i. The geometric mean (∏xi)1/n(\prod x_i)^{1/n}, by contrast, only grows as the nn-th root of any outlier, dramatically dampening its influence. In the context of GRPO, where xt=∣ρt(θ)A^i∣x_t = |\rho_t(\theta) \hat{A}_i| is the absolute importance-weighted reward at token tt, this means a token with ρt(θ)=100\rho_t(\theta) = 100 affects the geometric mean roughly as 100∣oi∣\sqrt[|o_i|]{100}, not 100/∣oi∣100/|o_i| as in the arithmetic mean. For sequences of hundreds or thousands of tokens, this suppression is dramatic.

The paper formalizes this advantage through multiple theoretical lenses:

  1. Value range analysis (Section 3): The absolute value of the GMPO objective is bounded above by the absolute value of the GRPO objective, via the inequality geometric mean ≤ arithmetic mean. This means GMPO's loss cannot reach the extreme magnitudes that GRPO's loss can, providing an inherent regularization effect.

  2. Gradient analysis (Equations 5-6, Appendix A): Both GRPO and GMPO gradients are weighted sums of policy gradients ∇θlog⁡(πθ(oi,t∣q,oi,<t))\nabla_\theta \log(\pi_\theta(o_{i,t}|q, o_{i,<t})), but with different weights. GRPO weights token tt by its individual ρt(θ)\rho_t(\theta), which can be extreme. GMPO weights all tokens in a sequence by the geometric mean of all ρ\rho values in that sequence, (∏k=1∣oi∣ρi,k(θ))1/∣oi∣(\prod_{k=1}^{|o_i|} \rho_{i,k}(\theta))^{1/|o_i|}. This shared weight is inherently more stable because it averages over the entire sequence. A single outlier ρ\rho at one position is diluted across all positions.

  3. Exploration-stability decoupling (Section 4.3, Figure 4): Because GMPO's objective is inherently stable, it can afford a much wider clipping range — (e−0.4,e0.4)(e^{-0.4}, e^{0.4}) versus GRPO's (0.8,1.2)(0.8, 1.2). Note that e0.4≈1.49e^{0.4} \approx 1.49, significantly wider than 1.2. The ablation in Table 5 confirms that GMPO with (e−0.4,e0.4)(e^{-0.4}, e^{0.4}) achieves 52.7% average performance, while narrowing to (e−0.2,e0.2)(e^{-0.2}, e^{0.2}) drops to 52.4% and removing clipping entirely (−∞,+∞-\infty, +\infty) drops to 52.3%. This demonstrates that the wide clipping range is necessary for full exploration, but it only works because GMPO's geometric mean keeps the importance sampling ratios naturally within bounds.

The paper is careful not to overclaim. It does not argue that GRPO is fundamentally broken — GRPO achieves strong results, as shown in the baseline row of Table 1. Rather, it argues that GRPO's stability depends on tight clipping, which creates a ceiling on exploration and performance. GMPO raises that ceiling by making stability a property of the objective rather than a property of the constraints.

One subtle aspect of the positioning deserves attention: GMPO is described as "plug-and-play" and the pseudo-code in Algorithm 1 shows that implementing it requires changing only a few lines in the loss computation. This is a strategic choice — the paper targets the large community of practitioners already running GRPO who could switch to GMPO with minimal engineering effort. The "plug-and-play" framing emphasizes that the contribution is not a complex new system requiring infrastructure changes (new models, new reward functions, new data pipelines) but a surgical modification to the loss function that preserves the rest of the GRPO workflow.

Finally, the paper explicitly connects its approach to a broader theme in the RL-for-LLMs literature: the tension between exploration and stability. By citing work on entropy collapse (Cui et al., 2025b), the 80/20 rule for token importance (Wang et al., 2025), and the exploration-limiting effects of clipping (Yu et al., 2025), the paper situates GMPO within an emerging consensus that sustained exploration is the bottleneck for scaling RL-based post-training. GMPO's contribution to this conversation is demonstrating that the right aggregation function can maintain exploration without sacrificing stability — a finding that suggests future RL algorithms should pay as much attention to their mean as to their clipping.

3. Technical Approach

3.1 Reader Orientation

This paper proposes a new loss function for reinforcement learning fine-tuning of language models — the system being built is not a deployable application but a training algorithm that replaces a single mathematical operation (arithmetic mean) with another (geometric mean) inside the GRPO objective function. The problem it solves is specific yet operationally critical: during GRPO training, individual tokens can develop extreme importance sampling ratios (probability ratios between current and old policies) that destabilize the entire optimization process, and the solution is to change how token-level contributions are aggregated into a sequence-level loss so that no single outlier token can dominate the gradient update.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four interacting components, only one of which GMPO actually modifies:

  1. Rollout Generator — takes a training question $q$, samples $G$ complete response sequences $\{o_1, ..., o_G\}$ from the old policy $\pi_{\theta_{\text{old}}}$ (the model frozen at the start of the current training round), and receives a binary reward $r_i \in \{0, 1\}$ for each based on whether the final answer matches the ground-truth solution.

  2. Advantage Computer — normalizes the $G$ rewards within each group into scalar advantages $\hat{A}_i = \frac{r_i - \text{mean}(\{r_1, ..., r_G\})}{\text{std}(\{r_1, ..., r_G\})}$, producing positive values for above-average responses and negative for below-average. This is identical in GRPO and GMPO.

  3. Importance Sampling Ratio Tracker — for each token position $t$ in each sequence $i$, computes $\rho_{i,t}(\theta) = \frac{\pi_\theta(o_{i,t}|q, o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t}|q, o_{i,<t})}$, the ratio of the current model's probability assigned to that token to the old model's probability. This ratio measures how much the policy has shifted since the data was generated; values near 1 indicate agreement, while extreme values indicate instability. This is identical in GRPO and GMPO.

  4. Loss Aggregator (the only modified component) — combines the per-token clipped importance-weighted rewards into a single scalar per sequence, then averages across sequences. GRPO uses the arithmetic mean: $\frac{1}{|o_i|}\sum_{t=1}^{|o_i|} \min[\rho_{i,t}(\theta)\hat{A}_i, \text{clip}(\rho_{i,t}(\theta), \epsilon_{\text{low}}, \epsilon_{\text{high}})\hat{A}_i]$. GMPO replaces this with the geometric mean: $\left(\prod_{t=1}^{|o_i|} |\min[\rho_{i,t}(\theta)\hat{A}_i, \text{clip}(\rho_{i,t}(\theta), \epsilon_{\text{low}}, \epsilon_{\text{high}})\hat{A}_i]|\right)^{1/|o_i|} \cdot \text{sgn}(\hat{A}_i)$. This single replacement is the entire contribution.

Information flows sequentially: a batch of questions enters → the old policy generates $G$ rollouts per question → rewards are computed by checking final answers → advantages are computed via group normalization → the current policy $\pi_\theta$ computes token probabilities for each rollout → importance sampling ratios are computed → the loss aggregator (GRPO or GMPO) combines everything into a scalar → backpropagation updates $\theta$ → the process repeats with a new old policy after $K$ updates.

3.3 Roadmap for the Deep Dive

  • First, the GRPO objective decomposed into its mathematical components, establishing the exact mechanism by which outlier importance ratios destabilize training — this is necessary because GMPO's design is directly motivated by GRPO's failure mode, and understanding the failure requires seeing where the arithmetic mean sits in the loss function.
  • Second, GMPO's replacement objective (the geometric mean version), including the sign-multiplication trick for handling negative advantages, the numerical-stability implementation in log-space, and the inequality proof that GMPO's objective has a narrower value range than GRPO's.
  • Third, the gradient analysis (Equations 5-6 and Appendix A), showing that both objectives are weighted policy gradients but GMPO assigns all tokens in a sequence the same stable weight (the geometric mean of all importance ratios) instead of per-token weights that can be extreme.
  • Fourth, the clipping design choices — why token-level clipping is preferred over sequence-level clipping, and why GMPO can use a much wider clipping range $(e^{-0.4}, e^{0.4})$ compared to GRPO's $(0.8, 1.2)$.
  • Fifth, the normalization factor $1/|o_i|$ inside the geometric mean — why it matters (preventing the product from exploding with sequence length) and the ablation showing its importance (Table 4, row 4 vs. row 5).
  • Sixth, the complete training loop and hyperparameters (Algorithm 1, training dataset, generation budget, update frequency) that instantiate GMPO in practice.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an algorithmic modification paper whose core idea is that swapping the arithmetic mean for the geometric mean in GRPO's loss function suppresses outlier-driven instability, enabling wider clipping and sustained exploration without degrading policy stability.

The GRPO Objective: Where the Arithmetic Mean Causes Instability

The GRPO objective function (Equation 1, reproduced from Shao et al., 2024) defines what the model optimizes during each training round:

JGRPO(πθ)=Eq∼Q,{oi}i=1G∼πθold(⋅∣q)[1G∑i=1G1∣oi∣∑t=1∣oi∣{min⁡[ρi,t(θ)A^i,clip(ρi,t(θ),ϵlow,ϵhigh)A^i]−βDKL(πθ∥πref)}]\mathcal{J}_{\text{GRPO}}(\pi_\theta) = \mathbb{E}_{q \sim \mathcal{Q}, \{o_i\}_{i=1}^G \sim \pi_{\theta_{\text{old}}}(\cdot|q)} \left[ \frac{1}{G} \sum_{i=1}^G \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \left\{ \min\left[\rho_{i,t}(\theta) \hat{A}_i, \text{clip}(\rho_{i,t}(\theta), \epsilon_{\text{low}}, \epsilon_{\text{high}}) \hat{A}_i\right] - \beta D_{\text{KL}}(\pi_\theta \parallel \pi_{\text{ref}}) \right\} \right]

where $q$ is a training question sampled from the question set $\mathcal{Q}$, $G$ is the number of rollouts (responses) generated per question, $o_i$ is the $i$-th rollout with length $|o_i|$ tokens, $\pi_{\theta_{\text{old}}}$ is the frozen policy that generated the rollouts, $\pi_\theta$ is the current policy being optimized, $\pi_{\text{ref}}$ is the reference (pre-RL) model for KL regularization, $\rho_{i,t}(\theta) = \frac{\pi_\theta(o_{i,t}|q, o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t}|q, o_{i,<t})}$ is the importance sampling ratio for token $t$ in rollout $i$, $\hat{A}_i = \frac{r_i - \text{mean}(\{r_1, ..., r_G\})}{\text{std}(\{r_1, ..., r_G\})}$ is the group-normalized advantage, $\epsilon_{\text{low}}$ and $\epsilon_{\text{high}}$ are clipping thresholds, and $\beta$ is the KL penalty coefficient.

What it computes: for each question, the model generates $G$ rollouts using the old policy. Each rollout receives a binary reward $r_i$ (1 if the final mathematical answer matches the ground truth, 0 otherwise). These $G$ rewards are normalized into advantages $\hat{A}_i$ by subtracting the group mean and dividing by the group standard deviation — this makes the advantage positive for rollouts that scored above the group average and negative for those below. For each token in each rollout, the importance-weighted reward $\rho_{i,t}(\theta) \hat{A}_i$ is computed (multiplying the token's probability ratio by the rollout's advantage). This product is then clipped to $[\epsilon_{\text{low}}, \epsilon_{\text{high}}]$ times $\hat{A}_i$ to prevent extreme updates, and the minimum of the clipped and unclipped versions is taken (the PPO-style pessimistic clipping). These clipped token-level values are averaged (arithmetic mean) over all tokens in the sequence (the $\frac{1}{|o_i|}\sum_{t=1}^{|o_i|}$ term), then averaged over the $G$ rollouts. A KL penalty term $\beta D_{\text{KL}}(\pi_\theta \parallel \pi_{\text{ref}})$ encourages the updated policy to stay close to the pre-trained model.

Why this form: GRPO adapts PPO's clipped surrogate objective to the group-relative setting. The clipping $\min[\rho \hat{A}, \text{clip}(\rho, \epsilon_{\text{low}}, \epsilon_{\text{high}}) \hat{A}]$ is the standard PPO mechanism for preventing excessively large policy updates: when the importance ratio $\rho$ moves outside the clipping range, the gradient of the clipped term becomes zero, preventing further movement in that direction. The group-based advantage normalization $\hat{A}_i$ replaces the learned value function (critic) that PPO requires, reducing computational cost. The arithmetic mean $\frac{1}{|o_i|}\sum_{t=1}^{|o_i|}$ is the natural default for combining per-token contributions — it treats every token as equally important to the loss.

The paper simplifies GRPO in two ways for fair comparison with GMPO. First, following Dr.GRPO (Liu et al., 2025), it drops the KL regularization term ($\beta = 0$) "for simplicity and memory saving" (Section 2.2). Second, it rewrites the objective without the clipping, for analytical clarity, as the arithmetic mean of token-level importance-weighted rewards:

JGRPO∗(πθ)=Eq∼Q,{oi}i=1G∼πθold(⋅∣q)[1G∑i=1G1∣oi∣∑t=1∣oi∣ρi,t(θ)A^i]\mathcal{J}^*_{\text{GRPO}}(\pi_\theta) = \mathbb{E}_{q \sim \mathcal{Q}, \{o_i\}_{i=1}^G \sim \pi_{\theta_{\text{old}}}(\cdot|q)} \left[ \frac{1}{G} \sum_{i=1}^G \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \rho_{i,t}(\theta) \hat{A}_i \right]

The failure mode: this arithmetic mean $\frac{1}{|o_i|}\sum_{t=1}^{|o_i|} \rho_{i,t}(\theta) \hat{A}_i$ is what causes instability. Consider a sequence of 500 tokens where 499 have $\rho_{i,t}(\theta) \approx 1$ (the policy hasn't changed much for those tokens) and one token has $\rho_{i,t}(\theta) = 50$ (the current policy assigns 50× higher probability to that token than the old policy did). The arithmetic mean of the importance-weighted rewards would be $\approx \frac{499 \cdot 1 \cdot \hat{A}_i + 50 \cdot \hat{A}_i}{500} = 1.098 \cdot \hat{A}_i$. The single outlier contributes roughly $10\%$ of the total sequence loss. If the sequence length were 3,000 (the paper's maximum response length), a single $\rho = 50$ token would contribute only $50/3000 \approx 1.7\%$ of the arithmetic mean — still linearly dependent on the outlier magnitude. The key insight is that the variance of the per-token weights determines gradient stability, and the arithmetic mean preserves each token's individual $\rho$ as its weight, meaning outlier ratios directly translate into outlier gradient contributions.

The GMPO Objective: Replacing Arithmetic Mean with Geometric Mean

GMPO replaces the arithmetic mean in Equation 2 with the geometric mean, producing the simplified objective:

JGMPO∗(πθ)=Eq∼Q,{oi}i=1G∼πθold(⋅∣q)[1G∑i=1G(∏t=1∣oi∣∣ρi,t(θ)A^i∣)1∣oi∣⋅sgn(A^i)]\mathcal{J}^*_{\text{GMPO}}(\pi_\theta) = \mathbb{E}_{q \sim \mathcal{Q}, \{o_i\}_{i=1}^G \sim \pi_{\theta_{\text{old}}}(\cdot|q)} \left[ \frac{1}{G} \sum_{i=1}^G \left( \prod_{t=1}^{|o_i|} \big| \rho_{i,t}(\theta) \hat{A}_i \big| \right)^{\frac{1}{|o_i|}} \cdot \text{sgn}(\hat{A}_i) \right]

where $\text{sgn}(\hat{A}_i)$ returns $+1$ when the advantage is positive and $-1$ when negative, and the absolute value $|\rho_{i,t}(\theta) \hat{A}_i|$ inside the product ensures we can take the $|o_i|$-th root of a non-negative number.

What it computes: instead of summing token-level rewards and dividing by sequence length (arithmetic mean), this multiplies all token-level absolute importance-weighted rewards together, takes the $|o_i|$-th root (geometric mean), then multiplies by the sign of the advantage to restore the correct optimization direction. For a sequence of $|o_i| = 500$ tokens where 499 have $|\rho \hat{A}| = 1$ and one has $|\rho \hat{A}| = 50$, the arithmetic mean gives $(499 \cdot 1 + 50) / 500 \approx 1.098$, while the geometric mean gives $(1^{499} \cdot 50)^{1/500} \approx 50^{0.002} \approx 1.008$. The outlier's influence is exponentially dampened by the root operation because $50^{1/500}$ is essentially 1. This is the mathematical mechanism by which GMPO suppresses outlier influence.

Why this form — the sign trick: the geometric mean is only defined for non-negative numbers. Since $\hat{A}_i$ can be negative (for below-average rollouts), the product $\prod \rho_{i,t}(\theta) \hat{A}_i$ could be negative if an odd number of tokens have negative $\hat{A}_i$. However, $\hat{A}_i$ is a per-rollout scalar, not a per-token scalar — all tokens in the same rollout share the same $\hat{A}_i$. Therefore, the sign of the product is simply $\text{sgn}(\hat{A}_i)$, and the magnitude is $\prod |\rho_{i,t}(\theta) \hat{A}_i|$. By factoring out the sign and taking the geometric mean of the absolute values, we get a non-negative magnitude that we then re-sign with $\text{sgn}(\hat{A}_i)$. This preserves the correct gradient direction (positive advantage → increase token probabilities; negative advantage → decrease them) while using the geometric mean for magnitude.

The inequality proof — narrower value range: the paper establishes that the absolute value of GMPO's objective is bounded above by the absolute value of GRPO's objective:

∣JGMPO∗(πθ)∣=E[1G∑i=1G(∏t=1∣oi∣∣ρi,t(θ)A^i∣)1∣oi∣]≤E[1G∑i=1G1∣oi∣∑t=1∣oi∣∣ρi,t(θ)A^i∣]=∣JGRPO∗(πθ)∣|\mathcal{J}^*_{\text{GMPO}}(\pi_\theta)| = \mathbb{E} \left[ \frac{1}{G} \sum_{i=1}^G \left( \prod_{t=1}^{|o_i|} |\rho_{i,t}(\theta) \hat{A}_i| \right)^{\frac{1}{|o_i|}} \right] \leq \mathbb{E} \left[ \frac{1}{G} \sum_{i=1}^G \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} |\rho_{i,t}(\theta) \hat{A}_i| \right] = |\mathcal{J}^*_{\text{GRPO}}(\pi_\theta)|

This is a direct application of the AM-GM inequality: the geometric mean of non-negative numbers is always less than or equal to their arithmetic mean, with equality only when all numbers are identical. In the GMPO context, $\left( \prod_{t=1}^{|o_i|} |\rho_{i,t}(\theta) \hat{A}_i| \right)^{1/|o_i|} \leq \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} |\rho_{i,t}(\theta) \hat{A}_i|$ for every rollout. Since this holds for each rollout, it holds in expectation. The practical implication: GMPO's loss cannot reach the extreme magnitudes that GRPO's loss can, providing inherent regularization. When an outlier token drives $|\rho_{i,t} \hat{A}_i|$ to a very large value, GRPO's loss can spike proportionally, but GMPO's loss only increases as the $|o_i|$-th root of that spike — a dramatically smaller change for long sequences.

What the inequality physically means for training: the loss magnitude directly determines gradient magnitude since $\nabla_\theta \mathcal{J}$ scales with $\mathcal{J}$. A narrower loss range means narrower gradient range, which means fewer spikes in the gradient norm. Figure 4c confirms this: GMPO's gradient norm remains "stable" throughout training while GRPO's exhibits large fluctuations. This stability is not achieved through external constraints (clipping) but through the mathematical properties of the objective itself.

Why this form — geometric mean vs. harmonic mean or median: the paper chooses the geometric mean specifically because it (1) is a well-defined mean (satisfying the standard properties of internality and homogeneity), (2) has a natural interpretation as the arithmetic mean in log-space, making implementation straightforward via log-domain computation, (3) satisfies the AM-GM inequality that provides the theoretical value-range bound, and (4) has a smooth gradient that can be derived analytically (Appendix A). Alternatives like the median would be non-differentiable, and the harmonic mean would be even more sensitive to values near zero (creating a different instability). The geometric mean hits the sweet spot of being outlier-resistant, differentiable, and analytically tractable.

The Complete GMPO Objective with Clipping

The full GMPO objective (Equation 4) incorporates the PPO-style token-level clipping into the geometric mean:

JGMPO(πθ)=Eq∼Q,{oi}i=1G∼πθold(⋅∣q)[1G∑i=1G{∏t=1∣oi∣∣min⁡[ρi,t(θ)A^i,clip(ρi,t(θ),ϵlow,ϵhigh)A^i]∣}1∣oi∣⋅sgn(A^i)]\mathcal{J}_{\text{GMPO}}(\pi_\theta) = \mathbb{E}_{q \sim \mathcal{Q}, \{o_i\}_{i=1}^G \sim \pi_{\theta_{\text{old}}}(\cdot|q)} \left[ \frac{1}{G} \sum_{i=1}^G \left\{ \prod_{t=1}^{|o_i|} \Big| \min \big[ \rho_{i,t}(\theta) \hat{A}_i, \text{clip}(\rho_{i,t}(\theta), \epsilon_{\text{low}}, \epsilon_{\text{high}}) \hat{A}_i \big] \Big| \right\}^{\frac{1}{|o_i|}} \cdot \text{sgn}(\hat{A}_i) \right]

where $\min[\rho \hat{A}, \text{clip}(\rho, \epsilon_{\text{low}}, \epsilon_{\text{high}}) \hat{A}]$ is the standard PPO clipped surrogate, identical in form to GRPO's clipping but applied per-token inside the geometric mean rather than inside the arithmetic mean.

What it computes: for each token $t$ in rollout $i$, compute the clipped importance-weighted reward $\min[\rho_{i,t} \hat{A}_i, \text{clip}(\rho_{i,t}, \epsilon_{\text{low}}, \epsilon_{\text{high}}) \hat{A}_i]$. Take the absolute value of this clipped term. Multiply all these absolute clipped terms together across all tokens in the sequence. Take the $|o_i|$-th root (the geometric mean). Multiply by $\text{sgn}(\hat{A}_i)$ to restore the sign. Average across the $G$ rollouts.

Numerical stability implementation: computing the product of hundreds or thousands of numbers near 1 directly would cause floating-point underflow or overflow. Algorithm 1 shows that GMPO is implemented entirely in log-space. The key steps (lines 10-18 of the pseudo-code):

  1. new_log_probs and old_log_probs are the log-probabilities from the current and old policies, respectively.
  2. The difference new_log_probs - old_log_probs equals $\log(\rho_{i,t}(\theta))$ since $\rho = \pi_\theta / \pi_{\theta_{\text{old}}}$ implies $\log \rho = \log \pi_\theta - \log \pi_{\theta_{\text{old}}}$.
  3. This log-ratio is multiplied by $\text{sgn}(\hat{A})$ (line 12): sgn_A_log_probs_diff = sgn_A * (new_log_probs - old_log_probs). This cleverly bakes the sign into the log-space computation.
  4. The signed log-ratio is clipped to $[-\epsilon, \epsilon]$ (line 13) where $\epsilon = 0.4$ corresponds to the clipping range $(e^{-0.4}, e^{0.4})$.
  5. The pessimistic minimum of the unclipped and clipped signed log-ratios is taken (line 14), mimicking PPO's $\min$ operation.
  6. The result is multiplied back by $\text{sgn}(\hat{A})$ to recover the correct sign for the final exponentiation (line 15): log_probs_diff_min = sgn_A * sgn_A_log_probs_diff_min.
  7. The per-token clipped log-ratios are summed over valid tokens (using the mask to ignore padding), then divided by the number of valid tokens to get the mean log-ratio (line 17): log_probs_diff_min[mask].sum() / mask.sum().
  8. This mean log-ratio is exponentiated to get the importance sampling ratio for the sequence (line 17): importance_sampling_ratio = torch.exp(log_probs_diff_min[mask].sum() / mask.sum()). This exponentiation of the mean log corresponds exactly to the geometric mean in probability space: $\exp(\frac{1}{|o_i|}\sum_t \log |x_t|) = (\prod_t |x_t|)^{1/|o_i|}$.
  9. The final loss (line 18) is -advantage * importance_sampling_ratio, which matches the objective $\hat{A}_i \cdot (\prod_t |\min[\cdots]|)^{1/|o_i|} \cdot \text{sgn}(\hat{A}_i)$ (the - sign converts maximization to minimization for gradient descent).

Why the sign multiplication back-and-forth (lines 12 and 15): the clipping operation torch.clamp(sgn_A_log_probs_diff, -epsilon, epsilon) expects a single real number and constrains it to a symmetric range. If the advantage is positive, sgn_A = 1, and sgn_A_log_probs_diff = new_log_probs - old_log_probs, which is clipped to [-epsilon, epsilon]. This correctly implements $\text{clip}(\rho, e^{-\epsilon}, e^{\epsilon})$ in log-space. If the advantage is negative, sgn_A = -1, and sgn_A_log_probs_diff = -(new_log_probs - old_log_probs), which is also clipped to [-epsilon, epsilon]. The multiplication by sgn_A again on line 15 correctly flips the sign back. This algebraic trick avoids branching logic for positive vs. negative advantages while keeping the implementation entirely in log-space.

Gradient Analysis: Why GMPO Updates Are More Stable

The paper provides a gradient-level explanation for GMPO's stability (Equations 5-6, with full derivations in Appendix A, Lemmas 1-3). The gradient of the simplified GRPO objective (without clipping) for a single rollout $o_i$ with respect to model parameters $\theta$ is:

∇θJGRPO∗(πθ)∣q,oi=1G⋅∣oi∣∑t=1∣oi∣ρi,t(θ)⋅A^i⋅∇θlog⁡(πθ(oi,t∣q,oi,<t))\nabla_\theta \mathcal{J}^*_{\text{GRPO}}(\pi_\theta)\big|_{q, o_i} = \frac{1}{G \cdot |o_i|} \sum_{t=1}^{|o_i|} {\color{red}{\rho_{i,t}(\theta)}} \cdot \hat{A}_i \cdot \nabla_\theta \log(\pi_\theta(o_{i,t}|q, o_{i,<t}))

where $\nabla_\theta \log(\pi_\theta(o_{i,t}|q, o_{i,<t}))$ is the standard policy gradient for token $o_{i,t}$ — it points in the direction that increases the log-probability of that token — and ${\color{red}{\rho_{i,t}(\theta)}}$ is the per-token importance sampling weight.

The gradient of the simplified GMPO objective for the same rollout is:

∇θJGMPO∗(πθ)∣q,oi=1G⋅∣oi∣∑t=1∣oi∣(∏k=1∣oi∣ρi,k(θ))1∣oi∣⋅A^i⋅∇θlog⁡(πθ(oi,t∣q,oi,<t))\nabla_\theta \mathcal{J}^*_{\text{GMPO}}(\pi_\theta)\big|_{q, o_i} = \frac{1}{G \cdot |o_i|} \sum_{t=1}^{|o_i|} {\color{blue}{\left( \prod_{k=1}^{|o_i|} \rho_{i,k}(\theta) \right)^{\frac{1}{|o_i|}}}} \cdot \hat{A}_i \cdot \nabla_\theta \log(\pi_\theta(o_{i,t}|q, o_{i,<t}))

where ${\color{blue}{(\prod_{k=1}^{|o_i|} \rho_{i,k}(\theta))^{1/|o_i|}}}$ is the geometric mean of all importance ratios in the sequence, used as a shared weight for every token.

What it computes — both are weighted policy gradients: the term $\hat{A}_i \cdot \nabla_\theta \log(\pi_\theta(o_{i,t}|q, o_{i,<t}))$ is the raw policy gradient signal: it says "if the advantage is positive, increase the probability of token $t$; if negative, decrease it." Both GRPO and GMPO weight this raw gradient by a scalar that reflects how much the policy has changed since the data was generated. The difference is entirely in what weight is used.

Why GRPO's weight causes instability: in GRPO, each token $t$ is weighted by its individual importance ratio $\rho_{i,t}(\theta)$. If token 47 in a 500-token sequence has $\rho = 100$ (the current model is 100× more confident in that token than the old model was), then the gradient update for that token specifically gets 100× the weight it would have under $\rho = 1$. Meanwhile, other tokens in the same sequence get weight near 1. The result is an unbalanced gradient where one token dominates the per-sequence update, potentially pushing the model parameters in a direction that overfits to that token's context rather than improving the overall sequence quality. This is directly visible in Figure 1 (right): GRPO's $\rho$ distribution develops a long tail of extreme values as training progresses.

Why GMPO's weight is more balanced: in GMPO, every token in the sequence gets exactly the same weight: the geometric mean $(\prod_{k=1}^{|o_i|} \rho_{i,k}(\theta))^{1/|o_i|}$. This weight aggregates information from the entire sequence. If 499 tokens have $\rho \approx 1$ and one has $\rho = 100$, the geometric mean is $(1^{499} \cdot 100)^{1/500} \approx 1.009$ — barely above 1. The shared weight means that outlier tokens cannot dominate the gradient; their influence is diluted across the entire sequence. This is the gradient-level mechanism that produces the stable gradient norms observed in Figure 4c.

The full derivation (Appendix A, Lemma 3): the gradient of the geometric mean involves a three-step chain rule: (1) derivative of $(\prod \rho)^{1/|o|}$ with respect to the product, giving $\frac{1}{|o|}(\prod \rho)^{1/|o| - 1}$; (2) derivative of the product $\prod \rho$ with respect to each $\rho_k$, giving $\sum_k \left(\prod_{t \neq k} \rho_t\right) \nabla_\theta \rho_k$; (3) derivative of each $\rho_k$ with respect to $\theta$, giving $\rho_k \nabla_\theta \log \pi_\theta(o_k)$. The algebra simplifies to the clean form in Equation 6 where the shared weight emerges as the geometric mean. This is not an approximation — it is the exact gradient of the geometric mean objective.

Token-Level vs. Sequence-Level Clipping

The paper makes a deliberate design choice to apply clipping at the token level rather than the sequence level, contrasting with DeepSeek-R1's approach (Guo et al., 2025a). DeepSeek-R1 computes the product of all per-token importance ratios $\prod_{t=1}^{|o_i|} \rho_{i,t}(\theta)$ and clips this single product to $[\epsilon_{\text{low}}, \epsilon_{\text{high}}]$. This is "sequence-level clipping."

GMPO's Equation 4 instead applies the $\min[\rho \hat{A}, \text{clip}(\rho, \epsilon_{\text{low}}, \epsilon_{\text{high}}) \hat{A}]$ operation at the individual token level — each token gets its own clipped importance-weighted reward, and then the geometric mean aggregates these already-clipped token-level values.

The rationale for token-level clipping (Section 3, design point (i)):

  1. Stability: Figure 3 provides the empirical evidence. The solid blue line labeled "GMPO $(e^{-0.4}, e^{0.4})$" (token-level clip) maintains a substantially narrower range of importance sampling ratios throughout training compared to the purple line labeled "GMPO-seq-clip-$(e^{-0.4}, e^{0.4})$" (sequence-level clip). The sequence-level clip produces a "larger importance sampling range," which makes it "more prone to create extreme gradients during optimization." This is because sequence-level clipping can only intervene when the cumulative product exceeds the threshold — by that point, individual tokens may already have quite extreme ratios, and the clipping only truncates the final product rather than constraining individual token contributions.

  2. Granularity — not discarding information: when sequence-level clipping is triggered, "it sets the gradients of all tokens in the sequence to zero, potentially discarding valuable update signals from informative parts of rollouts." Consider a 1,000-token reasoning chain where the model makes a subtle logical error at token 500 (causing $\rho_{500}$ to spike because the current policy would assign very different probability to that token) but produces excellent reasoning for the other 999 tokens. With token-level clipping, only token 500's gradient contribution is clipped; the other 999 tokens still provide learning signals. With sequence-level clipping, if $\prod_{t=1}^{1000} \rho_t$ exceeds the threshold, the entire sequence is zeroed out, losing the 999 valid gradient contributions. This is particularly damaging in long-form reasoning where most tokens are stable and only a few are problematic.

Table 4 ablation — row 3 (seq-clip) vs. row 5 (GMPO, token-clip): both achieve similar average performance (52.6% vs. 52.7%), but Figure 3 shows that the token-level version has a tighter $\rho$ range throughout training. The paper chooses token-level clipping as the default because it achieves equivalent final performance with better training stability (narrower importance ratio range), which is the primary design goal.

The Wider Clipping Range: $(e^{-0.4}, e^{0.4})$

A critical design choice is GMPO's clipping thresholds of $(e^{-0.4}, e^{0.4})$, which translates to approximately $(0.67, 1.49)$. This is substantially wider than standard GRPO's $(0.8, 1.2)$ and even wider than DAPO's expanded $(0.8, 1.28)$. The paper justifies this through a combination of empirical and theoretical reasoning.

Table 5 sweeps four clipping ranges, all tested on Qwen2.5-Math-7B:

  • $(e^{-0.2}, e^{0.2}) \approx (0.82, 1.22)$: average 52.4%
  • $(e^{-0.4}, e^{0.4}) \approx (0.67, 1.49)$: average 52.7% (best)
  • $(e^{-0.8}, e^{0.8}) \approx (0.45, 2.23)$: average 52.1%
  • $(-\infty, +\infty)$ (no clipping): average 52.3%

The inverted-U pattern is revealing: too narrow $(e^{-0.2}, e^{0.2})$ limits exploration (matching DAPO's observation), too wide or no clipping introduces instability from extreme $\rho$ values, and $(e^{-0.4}, e^{0.4})$ hits the sweet spot. The key insight is that this sweet spot is only accessible because GMPO's geometric mean naturally constrains the $\rho$ distribution. If GRPO tried to use clipping thresholds of $(0.67, 1.49)$, the importance sampling ratio range would explode (Figure 3: GRPO $(0.8, 1.2)$ already shows a wide expanding range). GMPO's inherent stability makes the wider clipping range safe to use.

Why wider clipping enables exploration: the clipping operation truncates gradients when $\rho$ moves outside the range. Narrow clipping means that even moderate policy shifts (which are necessary for exploring new reasoning strategies) get clipped, preventing the model from meaningfully changing its token probabilities. This is the "early deterministic policy" problem DAPO identified. GMPO's wider clipping allows the model to make larger probability shifts — exploring alternative token choices — without triggering clipping. The exploration is reflected in Figure 4a-b: GMPO maintains consistently higher token entropy than GRPO throughout training, meaning the model retains uncertainty and continues exploring the token space rather than collapsing to deterministic predictions.

The connection to entropy: higher entropy means the model's probability distribution over next tokens is more spread out — it is considering multiple possible continuations rather than being locked into a single high-confidence prediction. In the context of mathematical reasoning, this means the model can explore alternative proof strategies, try different algebraic manipulations, or backtrack from dead ends. GRPO's arithmetic mean, by creating aggressive outlier-driven updates, sharpens the distribution too quickly (entropy collapse), locking the model into whichever reasoning patterns it first discovers. GMPO's geometric mean, by damping outlier influence, preserves entropy longer, allowing the model to discover superior reasoning strategies later in training. This is confirmed by Figure 4e-g: GMPO's validation scores continue improving after GRPO's have plateaued.

The Normalization Factor $1/|o_i|$: Why It Matters

In Equation 4, the geometric mean includes the exponent $1/|o_i|$, which normalizes the product by the sequence length. Row 4 of Table 4 ablates this by removing the normalization, testing the objective $\left\{ \prod_{t=1}^{|o_i|} \big| \min[\cdots] \big| \right\} \cdot \text{sgn}(\hat{A}_i)$ (without the $1/|o_i|$ root). This drops average performance from 52.7% to 52.0% — a 0.7% decrease.

Why normalization matters — the exploding product problem (Appendix C, Figure 6): without the $1/|o_i|$ root, the objective becomes the product of per-token values rather than their geometric mean. As Figure 6 visualizes, the "sequence-level importance sampling ratios from trajectories that yield positive rewards during GRPO training" can reach extremely large values, and "without normalization, these ratios can become highly unstable, especially as the response length increases." The product of 1,000 values each around 1.01 is $1.01^{1000} \approx 20,959$, while the geometric mean $(1.01^{1000})^{1/1000} = 1.01$ remains well-behaved. Since GMPO training involves sequences up to 3,000 tokens, the product without normalization would create extreme loss values and gradients for long sequences, making training unstable. The $1/|o_i|$ exponent is not just a mathematical formality — it is essential for making GMPO work with variable-length sequences typical of chain-of-thought reasoning.

The connection to DeepSeek-R1: DeepSeek-R1 also computes $\prod_{t=1}^{|o_i|} \rho_{i,t}(\theta)$ without the normalization root, applying sequence-level clipping to keep it bounded. GMPO's addition of the $1/|o_i|$ root means that it computes the geometric mean rather than the raw product, which is inherently length-independent. This eliminates the need for aggressive sequence-level clipping and allows the token-level clipping with wider thresholds.

Training Loop and Hyperparameters (Algorithm 1 Instantiation)

The paper adopts the training setup from Dr.GRPO (Liu et al., 2025) with specific hyperparameter choices:

Training data: for models under 7B parameters, the paper uses MATH Levels 3-5 (Hendrycks et al., 2021), which contains 8,523 mathematical problems. For Mixture-of-Experts models (Table 2, right column), the paper uses DeepScaleR (Luo et al., 2025; ~40,000 problems from AIME, AMC, Omni-MATH, and Still datasets) and CountDown (Pan, 2024; arithmetic puzzles requiring step-by-step problem solving). Training details for MoE models are in Appendix B, Table 6.

Rollout generation: for each training question, the old policy $\pi_{\theta_{\text{old}}}$ generates $G = 8$ rollouts. The maximum response length is capped at 3,000 tokens. The reward $r_i$ is binary and verifiable: 1 if the final mathematical answer (extracted from the model's response) matches the ground-truth answer, 0 otherwise.

Training rounds and updates: in each RL training round, the old policy produces 1,024 rollouts (across multiple questions), and the current policy $\pi_\theta$ is updated 8 times with a batch size of 128. This means each round processes $1024 / 8 = 128$ questions (since each question produces 8 rollouts), and each of the 8 update steps uses a mini-batch of 128 rollouts.

Hardware: all models under 7B are trained on a server with 8× A800 GPUs.

Evaluation: Pass@1 accuracy on five mathematical reasoning benchmarks (AIME24: 30 problems, AMC: 83 problems, MATH500: 500 problems, Minerva: 272 problems, OlympiadBench: 675 problems) and one multimodal benchmark (Geometry3K: 601 problems). For language tasks, temperature is 0.0 with one generation per question (greedy decoding). For multimodal tasks, temperature is 0.5 with 16 generations per question.

Kl regularization: following Dr.GRPO, the KL penalty term $\beta D_{\text{KL}}(\pi_\theta \parallel \pi_{\text{ref}})$ is omitted ($\beta = 0$). This means that any stability advantage GMPO demonstrates over GRPO comes entirely from the geometric mean, not from differences in KL regularization strategy. The KL divergence from the pre-trained model is still measured as a diagnostic (Figure 4c-d), but it is not used in the loss.

The complete Algorithm 1 in operational terms: for each mini-batch of rollouts, the gmpo_loss function (Algorithm 1) receives token probabilities from both the current policy new_probs and the old policy old_probs (each of shape [L, 1] where $L$ is the padded sequence length), a binary mask indicating valid (non-padding) tokens, a single scalar advantage advantage for the entire rollout, and the clipping threshold epsilon (set to 0.4, corresponding to the range $(e^{-0.4}, e^{0.4})$). It returns a scalar loss value that is averaged across the mini-batch and used for backpropagation. The function is called once per rollout, and the per-rollout losses are aggregated (averaged) across the $G = 8$ rollouts per question and across questions in the batch.

Summary of Design Choices and Their Justifications

  • Geometric mean over arithmetic mean: inherent outlier resistance, mathematically guaranteed narrower value range via AM-GM inequality, produces shared (stable) per-token gradient weights rather than individual (potentially extreme) weights.
  • Token-level clipping over sequence-level clipping: more granular control (individual tokens clipped, not entire sequences), preserves learning signals from non-problematic tokens when one token triggers clipping, empirically tighter importance sampling ratio range (Figure 3).
  • Wide clipping range $(e^{-0.4}, e^{0.4})$ over standard $(0.8, 1.2)$: enables exploration by allowing larger policy shifts, made safe by GMPO's inherent stability which prevents $\rho$ from exploding even with wide thresholds, ablation-confirmed optimum (Table 5).
  • Normalization exponent $1/|o_i|$ over raw product: prevents the loss from exploding with sequence length (Appendix C, Figure 6), ensures length-independence of the objective, ablation shows 0.7% performance improvement (Table 4, row 4 vs. row 5).
  • Log-space implementation over naive product/exponentiation: numerical stability for sequences up to 3,000 tokens where direct product would overflow or underflow floating-point representation.
  • $\text{sgn}(\hat{A}_i)$ trick over case-splitting on advantage sign: allows the geometric mean (defined for non-negative numbers) to handle negative advantages cleanly, implemented efficiently in log-space as sign-multiplication before and after clipping (Algorithm 1, lines 12 and 15).
  • Omission of KL penalty following Dr.GRPO: simplifies implementation, isolates GMPO's contribution (stability comes from geometric mean, not from KL regularization), memory savings.

4. Key Insights and Innovations

Innovation 1: The Aggregation Function Is a First-Class Design Axis in RL for LLMs

The paper's most fundamental contribution is not the specific choice of the geometric mean — it is the reframing of the aggregation function itself as a primary design dimension in reinforcement learning objectives for language models. Prior to this work, the RL-for-LLMs literature treated the aggregation of per-token contributions into a sequence-level loss as a mechanical default: sum them up and divide by length (the arithmetic mean). Every GRPO variant cataloged in Section 2.1 — from DAPO's dynamic sampling to Dr.GRPO's length-aware rewards to OPO's optimal baselines — accepted this aggregation function as given and worked around its consequences through data filtering, reward reshaping, clipping strategies, or exploration heuristics.

GMPO demonstrates that this default is not neutral. The choice of mean carries substantial consequences for optimization dynamics: it determines how outlier tokens influence the gradient, whether exploration can coexist with stability, and what clipping ranges are viable. By showing that a single mathematical operation — swapping arithmetic mean for geometric mean — can achieve what dozens of prior methods attempted through complex peripheral modifications, the paper makes a conceptual point that transcends its algorithmic contribution: the loss function's internal structure matters as much as the reward signal it optimizes. This is a shift in where researchers should look when RL training becomes unstable — not just at the data, the rewards, or the constraints, but at the mathematics of how token contributions are combined.

The AM-GM inequality proof in Section 3 (showing |J*_GMPO| ≤ |J*_GRPO|) is not merely a technical justification — it introduces a design principle: an objective function can be "self-regularizing" if its mathematical form bounds its own magnitude. This principle stands in contrast to the dominant paradigm in PPO/GRPO where stability is achieved entirely through external constraints (clipping, KL penalties). GMPO shows that geometric aggregation provides what the authors effectively argue is "free regularization" — stability that emerges from the aggregation formula itself rather than from imposed limits. This distinction matters for future algorithm design: rather than adding more constraints when training is unstable, one might instead ask whether the aggregation function is amplifying noise.

The conceptual distinction between treating symptoms (clipping outlier gradients) and treating causes (preventing outlier gradients from existing in the first place) is clean and generalizable. Any RL objective that aggregates per-token or per-step contributions into a sequence-level loss could potentially benefit from re-examining its aggregation function through this lens. The paper does not explore this — it focuses narrowly on GRPO — but the framing invites future work on what other statistical summaries (harmonic mean, trimmed mean, median-like differentiable approximations) might offer for other RL algorithms and domains.

Innovation 2: Diagnosing and Breaking the Stability-Exploration Coupling

The paper identifies and exploits a previously diagnosed but never-resolved coupling between stability and exploration in GRPO. The coupling works as follows: GRPO's arithmetic mean is sensitive to outlier importance ratios, so practitioners apply tight clipping to prevent instability; tight clipping limits how much token probabilities can shift during updates; limited probability shifts mean the model cannot explore alternative token choices; premature entropy collapse follows, and performance plateaus (Figure 4a,e). The result is a forced tradeoff: you can have stable training or sustained exploration, but not both.

Prior work recognized pieces of this puzzle. DAPO (Yu et al., 2025) identified that clipping "can limit exploration and cause early deterministic policy, which can hinder the scaling process" and responded by slightly expanding the clipping range from (0.8, 1.2) to (0.8, 1.28). This is an incremental relaxation of the tradeoff — still operating within the paradigm where clipping is the primary stabilization mechanism and exploration is constrained by how wide you can make the clip without causing collapse. The 80/20 rule work (Wang et al., 2025) addressed a different angle, emphasizing high-entropy minority tokens during training to counteract entropy collapse, but left the underlying instability mechanism intact.

GMPO's key insight is that the coupling is an artifact of the arithmetic mean, not a fundamental property of policy optimization. By replacing the outlier-sensitive arithmetic mean with the outlier-resistant geometric mean, GMPO decouples the two concerns: stability becomes a property of the objective function (via the geometric mean's inherent dampening of extreme values), while exploration becomes a property of the clipping range (which can now be widened substantially without risking instability). The evidence for this decoupling is in Figure 4: GMPO simultaneously maintains higher entropy (a-b, sustained exploration) and lower KL divergence from the reference model (c-d, greater stability) — two metrics that are inversely correlated under GRPO. Under GRPO with wider clipping, entropy is temporarily higher but the model drifts further from the reference (instability). Under GMPO, both metrics improve together because they are no longer competing.

This is fundamentally different from DAPO's approach. DAPO asked "how much wider can we make the clip before things break?" GMPO asks "how can we change the objective so that wider clipping no longer breaks things?" The answer — use an outlier-resistant aggregation function — is a qualitative shift from constraint-tuning to objective-redesigning. The distinction matters because constraint-tuning approaches will always hit a ceiling: at some clipping width, even the best-tuned arithmetic mean will encounter destabilizing outliers. Objective-redesigning raises that ceiling substantially, which is why GMPO can use (e^{-0.4}, e^{0.4}) ≈ (0.67, 1.49) — a range ~2.7× wider than standard GRPO — without the importance ratio explosion visible in Figure 3.

The decoupling also explains a subtle empirical result that might otherwise seem contradictory: Table 5 shows that completely removing clipping (-∞, +∞) with GMPO performs worse (52.3%) than the optimal clipping (52.7%), meaning some clipping is still beneficial. This is because exploration without any bounds eventually encounters instability even with the geometric mean — the decoupling is substantial but not absolute. The geometric mean reduces outlier influence exponentially (by taking the |o_i|-th root) but does not eliminate it entirely, and for very long training runs, accumulated small instabilities can still matter. The paper's contribution is not that GMPO makes clipping unnecessary, but that it makes the optimal clipping range dramatically wider, which in turn enables exploration that GRPO cannot safely achieve.

Innovation 3: The Gradient-Weighting Perspective on Sequence-Level RL Stability

While the geometric mean's outlier resistance is a standard statistical property, the paper's gradient-weighting analysis (Equations 5-6, Appendix A) provides a novel diagnostic lens for understanding why certain aggregation functions produce stable training and others do not. The paper shows that both GRPO and GMPO produce gradients that are weighted sums of per-token policy gradients ∇_θ log(π_θ(o_t|q, o_<t)), but with fundamentally different weighting schemes:

  • GRPO: each token t is weighted by its individual importance ratio ρ_t(θ). This creates "token-level variance" in gradient contributions — some tokens get 100× the weight of others within the same sequence, and which tokens dominate changes as training progresses and ρ_t values shift.

  • GMPO: every token in a sequence receives the same weight — the geometric mean of all importance ratios in that sequence, (∏ ρ_k)^(1/|o|). This shared weight means that outlier tokens cannot dominate; their influence is diluted across the entire sequence's tokens.

This gradient-weighting insight is significant beyond GMPO because it provides a diagnostic framework for evaluating any sequence-level RL objective. Rather than reasoning about abstract properties like "stability" or "variance," one can examine the gradient decomposition: does the objective assign widely varying per-token weights that can become extreme, or does it assign more uniform weights that remain bounded? This framework explains why arithmetic-mean-based objectives are inherently vulnerable (per-token weights equal individual ρ_t values, which can be extreme) and why sequence-level clipping in DeepSeek-R1 is insufficient (it produces identical gradients for all tokens — zero — when triggered, which is uniform but destructive).

The analysis also reveals something non-obvious about GMPO: it does not eliminate the influence of importance sampling ratios on the gradient — it redistributes that influence uniformly across tokens. A sequence where the old and current policies disagree strongly overall (high geometric mean of ρ) still receives proportionally larger gradient updates. But the update is applied evenly to all tokens rather than concentrated on the tokens where disagreement is highest. This matters because in long reasoning chains, the tokens where policies disagree most may not be the ones that need the most gradient signal — they may simply be tokens where the model happened to randomly sample an unusual continuation. The uniform weighting prevents the optimization from overfitting to these noisy tokens at the expense of the broader reasoning pattern.

The paper uses this gradient analysis primarily as explanation rather than as a separate contribution, but its implications extend further. Future work could use this framework to design new objectives by explicitly engineering their gradient-weighting properties: what distribution of per-token weights is optimal for a given task and model architecture? Should weights be uniform (GMPO), proportional to individual ρ_t (GRPO), or perhaps something in between (trimmed mean, Winsorized mean)? The gradient-weighting lens transforms what appears to be a simple mean-substitution into a principled design space.

Innovation 4: Empirical Evidence That Objective-Level Changes Beat Constraint-Level Patches at Scale

The paper provides systematic empirical evidence supporting a claim that is often gestured at but rarely demonstrated with controlled comparisons: modifying the optimization objective itself can achieve what an extensive ecosystem of constraint-level patches (clipping strategies, reward normalization schemes, data filtering heuristics) has been trying to accomplish, and does so more elegantly. This is not merely a performance claim — it is an argument about algorithm design philosophy.

The evidence for this comes from the structured comparison against the GRPO variant landscape (Table 3). GMPO-7B achieves 52.7% average accuracy, outperforming not just vanilla GRPO (51.2%) but also methods that add substantial complexity: PRIME-Zero (48.0%, which uses implicit reward modeling), GPG (51.0%, which eliminates surrogate losses and critics), OpenReasoner-Zero (45.9% at 8k context, which curates 129k diverse samples with curriculum learning), and Eurus (48.0%, which uses preference trees and novel reward modeling). Each of these methods invests significant engineering effort into data curation, reward design, or training procedures — and each is outperformed by a three-line change to the loss function (replacing arithmetic mean with geometric mean in the aggregation step).

The significance of this comparison is not primarily about the specific accuracy numbers — GMPO's margin over some methods is modest (52.7% vs. 51.4% for Oat-Zero), and the 7B model comparisons involve different training data and hyperparameters that limit direct comparability. Rather, the significance is in what the comparison implies about where research effort should be directed. The GRPO variant explosion documented in Section 2.1 (20+ extensions in roughly a year) represents a collective bet that GRPO's limitations can be addressed through increasingly sophisticated peripheral modifications. GMPO's results suggest that at least some of those limitations — specifically, the instability-exploration coupling — can be addressed more directly and with less complexity by revisiting the core optimization objective.

This finding has practical implications for the RL-for-LLMs community. The cost of implementing a new GRPO variant is not just the algorithmic complexity — it includes the engineering effort to maintain custom training code, the hyperparameter tuning required for new components (dynamic sampling schedules, reward shaping coefficients, curriculum thresholds), and the reproducibility challenges when methods interact in unexpected ways. GMPO's "plug-and-play" nature (Algorithm 1 is 19 lines, a drop-in replacement for the GRPO loss) means it can be adopted with minimal engineering overhead. If the field's goal is to make RL training more accessible and reliable, objective-level interventions like GMPO offer a higher ratio of benefit to complexity than the constraint-level patches that have dominated recent work.

The paper is careful not to overstate this. The results in Table 3 are on a specific task distribution (mathematical reasoning) with specific model architectures (Qwen2.5-Math and R1-Distill). The claim is not that GMPO makes all other GRPO innovations obsolete — Dr.GRPO's length-aware rewards, for instance, address a different problem (length bias) that GMPO does not target. Rather, the claim is that for the specific problem of outlier-driven training instability and its consequences (entropy collapse, limited exploration, performance plateaus), modifying the aggregation function is more direct and effective than layering on additional constraints.

The MoE results (Appendix B, Figure 5) provide the strongest evidence for this claim because MoE models are where GRPO's instability is most acute — GRPO's validation score on CountDown "collapses after about 250 steps" while GMPO maintains stable improvement. The fact that GMPO's advantage is largest precisely where GRPO is most fragile supports the argument that the geometric mean is addressing a fundamental limitation rather than providing a generic performance boost that could be achieved through other means.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. For language-only tasks with models under 7B parameters, training uses MATH Levels 3–5 (Hendrycks et al., 2021), containing 8,523 mathematical problems. For Mixture-of-Experts models (Table 2, right column), training uses DeepScaleR (Luo et al., 2025; approximately 40,000 unique mathematics problem-answer pairs compiled from AIME, AMC, Omni-MATH, and Still datasets) and CountDown (Pan, 2024; arithmetic puzzles where models combine given numbers using basic operations to reach a target). For the multimodal task, training uses Geometry3K (Lu et al., 2021) following the EasyR1 setup (Zheng et al., 2025). Evaluation is conducted on five mathematical reasoning benchmarks — AIME24 (30 problems), AMC (83 problems), MATH500 (500 problems), Minerva (272 problems), OlympiadBench (675 problems) — and one multimodal benchmark, Geometry3K (601 problems). A held-out subset of CountDown is reserved for MoE model evaluation (Appendix B).

  • Base model(s). The paper evaluates across four language model families spanning different scales and architectures: Qwen2.5-Math-1.5B and Qwen2.5-Math-7B (Yang et al., 2024), DeepSeek-R1-Distill-Qwen-7B (Guo et al., 2025b), and the Mixture-of-Experts Qwen3-32B (Yang et al., 2025). For multimodal experiments, Qwen2.5-VL-Instruct-7B (Bai et al., 2025) is used. The 1.5B and 7B dense models establish baseline comparisons at different capacity levels, the R1-Distill variant tests GMPO on a model already optimized for reasoning, and the 32B MoE model probes stability in the architecture class where GRPO is known to be most fragile (Appendix B demonstrates GRPO collapse on CountDown with MoE models). Appendix B also includes a small-scale Qwen2.5-200M MoE model (8 experts, 1 active per token) for controlled stability experiments.

  • Metrics. The primary metric throughout is Pass@1 accuracy — the fraction of test problems for which a single generated response produces the correct final answer. For language tasks, evaluation uses temperature 0.0 (greedy decoding) with one generation per question following Dr.GRPO (Liu et al., 2025). For the multimodal Geometry3K task, temperature is set to 0.5 with 16 answers generated per question, and Pass@1 is computed from these 16 samples. Rewards during training are binary and verifiable: 1 for responses whose extracted final answer matches the ground truth, 0 otherwise. The paper also reports training diagnostics — mean token entropy, KL divergence between the current policy and the pre-RL reference model, gradient norm — as secondary metrics for analyzing training dynamics (Figure 4).

  • Baselines. The paper's primary comparison is against GRPO (Shao et al., 2024), trained under identical data and hyperparameter conditions (same training dataset, 8 rollouts per question, 3,000-token maximum response length, 1,024 rollouts per RL round with 8 update steps at batch size 128). The Dr.GRPO variant (Liu et al., 2025) serves as an additional baseline in Tables 1 and 3, with both methods sharing the design choice of omitting the KL regularization term. For the broader comparison in Table 3, the paper includes results from Qwen2.5-Math base and instruct models (Qwen et al., 2025), Oat-Zero (Liu et al., 2025), SimpleRL-Zero-7B (Zeng et al., 2025), PRIME-Zero-7B (Cui et al., 2025a), OpenReasoner-Zero-7B at 3k and 8k context lengths (Hu et al., 2025), Eurus-7B (Yuan et al., 2024), and GPG-7B (Chu et al., 2025). These baselines encompass the major GRPO variants and reasoning-focused RL approaches available at the time of writing, with all 7B-scale numbers reported from the respective papers.

  • Generation budget / compute accounting. The unit of test-time compute during training is the number of rollouts per question (G = 8 throughout) and the number of optimization steps per RL round (8 updates on batches of 128 rollouts). Training compute is reported in terms of the hardware used (8× A800 GPUs for models under 7B) rather than FLOP counts. During evaluation, Pass@1 with temperature 0.0 uses exactly one generation per question, making the inference budget identical for GMPO and all compared methods — any accuracy difference reflects training quality, not differential inference-time expenditure. For the multimodal task, 16 generations per question are used for both GRPO and GMPO evaluation, again keeping inference budgets matched.

  • Cross-validation / statistical protocol. The paper does not employ cross-validation or statistical significance testing. Results are reported as point estimates (Pass@1 accuracy on fixed test sets) without confidence intervals or error bars. The training process involves multiple RL rounds (each with 1,024 rollouts produced by the frozen old policy, followed by 8 mini-batch updates to the current policy), and the training curves in Figure 4 show validation scores evaluated periodically during training, providing some indication of run-to-run consistency through the smoothness of the curves. The ablation in Table 5 sweeps four clipping threshold values systematically, establishing the optimum through direct comparison rather than statistical testing. The MoE experiments in Appendix B use separate training datasets (DeepScaleR and CountDown) from the main experiments, providing an informal robustness check across data distributions.

Main Quantitative Results

Language-Only Mathematical Reasoning: GMPO vs. GRPO Across Model Scales

Table 1 presents the core head-to-head comparison between GRPO and GMPO across three model configurations. The headline result is that GMPO improves average Pass@1 over GRPO for every model tested, with gains increasing at larger scales and with reasoning-specialized base models:

  • Qwen2.5-Math-1.5B: GMPO achieves 43.9% average across the five benchmarks versus GRPO's 42.5%, a gain of 1.4 percentage points. Individual benchmark performance includes 20.0% on AIME24 (vs. 23.3% for GRPO — GMPO underperforms here), 53.0% on AMC (vs. 49.4%), 77.6% on MATH500 (vs. 75.2%), 30.1% on Minerva (vs. 25.7%), and 38.7% on OlympiadBench (vs. 39.0%).

  • Qwen2.5-Math-7B: GMPO achieves 52.7% average versus GRPO's 51.2%, a gain of 1.5 points. Individual results: 43.3% on AIME24 (vs. 40.0%), 61.4% on AMC (vs. 59.0%), 82.0% on MATH500 (vs. 83.4% — GMPO underperforms slightly here), 33.5% on Minerva (vs. 32.4%), and 43.6% on OlympiadBench (vs. 41.3%).

  • DeepSeek-R1-Distill-Qwen-7B: GMPO achieves 63.4% average versus GRPO's 59.3%, a gain of 4.1 percentage points — the largest improvement. Individual results: 46.6% on AIME24 (vs. 43.3%), 78.3% on AMC (vs. 67.5%), 91.4% on MATH500 (vs. 89.0%), 37.9% on Minerva (vs. 39.7% — GMPO underperforms here), and 62.5% on OlympiadBench (vs. 56.7%). The AMC improvement of 10.8 points (78.3% vs. 67.5%) is particularly striking and accounts for the bulk of the 4.1% average gain.

The pattern of gains is not uniform across benchmarks. GMPO consistently outperforms on AMC and OlympiadBench across all three model scales. On AIME24, GMPO underperforms GRPO at 1.5B (20.0% vs. 23.3%) but outperforms at 7B (43.3% vs. 40.0%) and R1-Distill-7B (46.6% vs. 43.3%). On MATH500, results are mixed — GMPO wins at 1.5B (77.6% vs. 75.2%) and R1-Distill-7B (91.4% vs. 89.0%) but loses at 7B (82.0% vs. 83.4%). On Minerva, GMPO wins at 1.5B and 7B but loses at R1-Distill-7B (37.9% vs. 39.7%). This benchmark-dependent pattern suggests GMPO's benefits are not uniform across difficulty levels or problem types, though the paper does not analyze this variation in detail.

Mixture-of-Experts Models: Stability Advantage Amplified

Table 2 (right column) reports results on Qwen3-32B, a Mixture-of-Experts model with 128 experts and 8 active per token (Table 6). GMPO achieves 96.7% accuracy on MATH500 versus GRPO's 94.6%, a gain of 2.1 percentage points. This is notable because (1) the absolute performance is already very high, making further improvement difficult, and (2) the improvement margin is larger than for the 7B dense model on the same benchmark (GMPO lost 1.4 points to GRPO on MATH500 with Qwen2.5-Math-7B).

Appendix B provides detailed training dynamics for MoE models on two additional datasets. On CountDown with a small Qwen2.5-200M MoE model (8 experts, 1 active per token), Figure 5e shows that GRPO's validation score collapses after approximately 250 training steps — a clear training failure mode — while GMPO maintains stable improvement throughout the training run. Figure 5a confirms that GMPO maintains smaller KL divergence from the reference model, and Figure 5c shows GMPO maintains a steadier gradient norm. On DeepScaleR with Qwen3-32B (Figure 5b,d,f), GMPO maintains both higher entropy and more stable gradients than GRPO, achieving higher validation scores throughout training.

These MoE results are the paper's strongest evidence that GMPO's stability advantage is not merely a marginal improvement but addresses a genuine failure mode of GRPO in architectures that are sensitive to gradient instability.

Multimodal Reasoning: Generalization Beyond Text-Only Math

Table 2 (left column) shows results on Geometry3K using Qwen2.5-VL-Instruct-7B as the base model. GMPO achieves 54.7% Pass@1 versus GRPO's 53.3%, a gain of 1.4 percentage points. This is a single-benchmark result but demonstrates that GMPO's benefits extend beyond text-only mathematical reasoning to visual geometry problem-solving, where the model must interpret diagrams and reason about spatial relationships. The gain is smaller than for text-only math (1.4 points vs. 1.5-4.1 points for 7B language models), which may reflect differences in the training data size (Geometry3K is smaller than MATH Levels 3-5), the base model's multimodal pretraining, or the nature of the task.

Figure 4g tracks validation scores on Geometry3K throughout training, showing that GMPO consistently outperforms GRPO after the initial training steps. Unlike some language-only experiments where GRPO initially matches or exceeds GMPO before plateauing, the multimodal validation curves show GMPO pulling ahead early and maintaining the lead.

Comparison Against the Broader GRPO Variant Landscape

Table 3 situates GMPO within the ecosystem of GRPO variants and reasoning-focused RL methods at the 7B scale. GMPO-7B achieves 52.7% average accuracy, outperforming all listed methods on this aggregate metric:

  • Versus vanilla GRPO-7B (51.2%): +1.5 points (Table 1 already established this).
  • Versus Dr.GRPO / Oat-Zero-7B (51.4%): +1.3 points. This is the most directly comparable baseline since both methods share the same training setup and omit KL regularization.
  • Versus GPG-7B (51.0%): +1.7 points. GPG simplifies optimization by eliminating surrogate losses, critics, and KL constraints, yet GMPO's geometric mean modification alone outperforms this more radical simplification.
  • Versus PRIME-Zero-7B (48.0%): +4.7 points. PRIME uses implicit reward modeling rather than the binary verifiable rewards GMPO uses; the large gap suggests that reward modeling overhead may not be worth the complexity for tasks with clean verifiable rewards.
  • Versus OpenReasoner-Zero-7B @ 8k (45.9%): +6.8 points. OpenReasoner-Zero invests in curating 129k diverse training samples with curriculum learning; GMPO's simpler approach on MATH Levels 3-5 substantially outperforms it.
  • Versus Eurus-7B (48.0%): +4.7 points. Eurus uses large-scale alignment datasets and preference trees for reward modeling.
  • Versus SimpleRL-Zero-7B (46.6%): +6.1 points.

For the R1-Distill variant, GMPO-7B (R1-Distill) achieves 63.4% average versus Dr.GRPO / Oat-Zero-7B (R1-Distill)'s 61.5%, a gain of 1.9 points. This is a more modest gain than the 4.1-point improvement over vanilla GRPO on the same architecture, suggesting that Dr.GRPO's length-aware rewards already address some of the instability that GMPO also mitigates — the combination of GMPO with Dr.GRPO-style length correction (which the paper does not explore) might yield further improvements.

At the 1.5B scale, GMPO-1.5B achieves 43.9% average versus Oat-Zero-1.5B's 42.1% (+1.8 points) and substantially above the Qwen2.5-Math-1.5B base model (33.1%, +10.8 points) and instruct variant (39.8%, +4.1 points).

The per-benchmark breakdown in Table 3 for 7B models reveals that GMPO's advantage is not uniformly distributed. On AIME24, GMPO-7B ties with Oat-Zero-7B (43.3% each) and GPG-7B (33.3% is lower). On AMC, GMPO's 61.4% is slightly below GPG's 65.0%, PRIME-Zero's 62.7%, Oat-Zero's 62.7%, and Eurus's 62.7%. On MATH500, GMPO's 82.0% is below PRIME-Zero's 83.8% and Eurus's 83.8%. On Minerva, GMPO's 33.5% is below PRIME-Zero's 36.0% and Eurus's 36.0%. On OlympiadBench, GMPO's 43.6% is the highest among all listed methods. This pattern — GMPO dominating on OlympiadBench but lagging on AMC, MATH500, and Minerva — is consistent with the hypothesis that GMPO's exploration advantage is most valuable on harder problems, though the paper does not explicitly analyze difficulty-stratified performance within benchmarks.

Ablation Studies and Robustness Checks

All ablations in this section use Qwen2.5-Math-7B as the base model with MATH Levels 3-5 training data, unless otherwise specified. The pre-RL model (no fine-tuning) achieves 26.5% average accuracy across the five benchmarks (Table 4, row 0).

  • Geometric mean vs. arithmetic mean (core contribution): Table 4, row 1 (GRPO, arithmetic mean) achieves 51.2% average; row 5 (GMPO, geometric mean) achieves 52.7%, a gain of 1.5 percentage points. This is the fundamental comparison that isolates the effect of the aggregation function, with all other training parameters held constant. GMPO outperforms GRPO on AIME24 (43.3% vs. 40.0%), AMC (61.4% vs. 59.0%), and OlympiadBench (43.6% vs. 41.3%), underperforms on MATH500 (82.0% vs. 83.4%), and slightly outperforms on Minerva (33.5% vs. 32.4%).

  • Token-level clipping vs. sequence-level clipping: Table 4, row 5 (GMPO with token-level clipping) achieves 52.7%; row 3 (GMPO with sequence-level clipping, as in DeepSeek-R1's approach) achieves 52.6%. The final accuracy is nearly identical, but Figure 3 shows that sequence-level clipping produces a substantially wider range of importance sampling ratios throughout training (the purple "GMPO-seq-clip-(e⁻⁰·⁴, e⁰·⁴)" line vs. the blue "GMPO (e⁻⁰·⁴, e⁰·⁴)" line). The paper chooses token-level clipping as the default because it achieves equivalent performance with tighter importance ratio control — a stability advantage that could matter for longer training runs or more challenging tasks where the wider ratio range of sequence-level clipping might eventually cause problems.

  • Clipping range completely removed: Table 4, row 2 (GMPO without any clipping, equivalent to setting thresholds to (-∞, +∞)) achieves 52.3%, a 0.4-point drop from the clipped version (52.7%). Figure 3 shows that the no-clip variant ("GMPO (-∞, +∞)") exhibits wider importance sampling ratio fluctuations than the clipped version, confirming that even with the geometric mean's inherent outlier resistance, some clipping is still beneficial for controlling extreme policy shifts. The 0.4-point gap between no-clip (52.3%) and optimal-clip (52.7%) indicates that GMPO's stability is partially inherent (geometric mean) and partially achieved through the wider-but-still-present clipping.

  • Normalization term 1/|o_i| in the geometric mean: Table 4, row 4 (GMPO without the 1/|o_i| root — computing the raw product rather than the geometric mean) achieves 52.0%, a 0.7-point drop from the correct GMPO (52.7% in row 5). This ablation confirms that the normalization is not cosmetic; removing it causes the loss magnitude to grow with sequence length (Appendix C, Figure 6), creating instability for longer reasoning chains. The 0.7-point gap is substantial relative to the 1.5-point total GMPO advantage over GRPO, meaning the normalization accounts for nearly half of GMPO's benefit.

  • Clipping threshold sweep (exploration-stability tradeoff): Table 5 tests four clipping ranges on GMPO, all with Qwen2.5-Math-7B. Narrow clipping (e^{-0.2}, e^{0.2}) ≈ (0.82, 1.22) — comparable to DAPO's expanded range — achieves 52.4%. The paper's chosen range (e^{-0.4}, e^{0.4}) ≈ (0.67, 1.49) achieves 52.7% (the optimum). Wider clipping (e^{-0.8}, e^{0.8}) ≈ (0.45, 2.23) drops to 52.1%. Complete removal of clipping (-∞, +∞) drops to 52.3%. The inverted-U shape confirms that too-narrow clipping limits exploration (52.4% → 52.7% when widening from e^{-0.2} to e^{-0.4}), while too-wide or absent clipping introduces instability (52.7% → 52.3% → 52.1% as width increases further). The optimum at (e^{-0.4}, e^{0.4}) is substantially wider than GRPO's standard (0.8, 1.2), supporting the claim that GMPO's inherent stability enables wider exploration-friendly clipping. Figure 3 visualizes the importance sampling ratio distributions corresponding to these thresholds, showing that wider clipping indeed produces wider ratio ranges, but GMPO's ranges remain tighter than GRPO's even at the widest thresholds.

  • Entropy dynamics throughout training: Figure 4a-b visualizes mean token entropy over training steps for GMPO and GRPO. On MATH Level 3-5 (Figure 4a), GRPO's entropy drops rapidly and continues declining; applying wider clipping to GRPO ("GRPO-wide-clip") temporarily elevates entropy but the same decline pattern resumes. GMPO maintains consistently higher entropy that declines more gradually. On the harder DeepScaleR dataset (Figure 4b), the same pattern holds — GMPO sustains higher entropy throughout training. This provides direct evidence for the paper's claim that GMPO enhances exploration: higher entropy means the policy distribution is less peaked, preserving the model's ability to sample diverse reasoning paths rather than collapsing to deterministic predictions. The entropy curves also explain the validation score dynamics in Figure 4e-h: GRPO's validation scores plateau or decline after entropy collapse, while GMPO's continue improving as exploration persists.

  • KL divergence from reference model: Figure 4c-d shows that GMPO maintains a smaller KL divergence from the pre-RL model than GRPO throughout training. On MATH Level 3-5 (Figure 4c), GMPO's KL grows slowly and stabilizes at a low value; GRPO's KL grows faster and to a higher level. On CountDown with an MoE model (Figure 5a), the contrast is starker — GMPO's KL remains small while GRPO's increases substantially. This is evidence for the paper's stability claim: smaller KL divergence means the fine-tuned policy stays closer to the pretrained model, reducing the risk of catastrophic forgetting and preserving general capabilities while improving at the target task.

  • Gradient norm stability: Figure 4c-d and Figure 5c-d show that GMPO maintains a more stable gradient norm than GRPO. GRPO exhibits larger fluctuations and occasional spikes. This is a direct consequence of the outlier-suppression mechanism: when extreme importance sampling ratios arise in GRPO, they create extreme gradient contributions that manifest as gradient norm spikes. GMPO's geometric mean dampens these outliers, producing smoother gradient norms, which in turn enables more reliable convergence. The gradient norm stability is most pronounced on MoE models (Figure 5c-d), where GRPO's gradient norm shows substantial volatility that correlates with the eventual training collapse on CountDown (Figure 5e).

  • Validation score trajectories: Figure 4e-h tracks validation scores over training steps across four settings. For Qwen2.5-Math-7B on language benchmarks (Figure 4e), GRPO's validation score plateaus approximately halfway through training while GMPO continues improving, ending with a visible gap. For Qwen3-32B on DeepScaleR (Figure 4f) and Qwen2.5-VL-Instruct-7B on Geometry3K (Figure 4g), GMPO maintains a consistent lead over GRPO after the initial steps. For the small MoE model on CountDown (Figure 5e), GRPO collapses catastrophically at ~250 steps while GMPO continues stable improvement. For Qwen3-32B on DeepScaleR validation (Figure 5f), GMPO tracks above GRPO throughout. These trajectory comparisons are important because they show that GMPO's final-accuracy advantage is not merely from a better final checkpoint but from consistently superior training dynamics — GMPO would outperform GRPO at most intermediate stopping points as well.

  • Dr.GRPO integration and length normalization: Table 4, row 4 ablates whether removing the 1/|o_i| normalization — which is conceptually similar to the sequence-level product used in DeepSeek-R1 — affects GMPO performance. The 0.7-point drop (52.0% vs. 52.7%) demonstrates that length normalization through the geometric mean's root is important. This is not simply Dr.GRPO's length-bias correction applied to GMPO; it is a distinct property of the geometric mean that prevents long sequences from dominating the loss. The paper does not ablate combining GMPO with Dr.GRPO's explicit length-aware reward modifications, which would disentangle these two length-related effects.

  • Cross-modal generalization: Table 2 (left) tests GMPO on the multimodal Geometry3K benchmark using Qwen2.5-VL-Instruct-7B, achieving 54.7% vs. GRPO's 53.3%. While this is a single-benchmark result, it demonstrates that GMPO's mechanism (geometric mean aggregation) does not depend on text-only architectures or mathematical symbol manipulation — the same loss modification transfers to vision-language models processing geometry diagrams. Figure 4g shows that the validation score advantage is maintained throughout training, not just at the final checkpoint.

Critical Assessment

Claim 1: GMPO improves stability by suppressing token reward outliers

What was demonstrated: The paper provides converging evidence across multiple diagnostic metrics that GMPO training exhibits greater stability than GRPO. The importance sampling ratio range in Figure 1 (right) is narrower for GMPO than GRPO. The gradient norm in Figure 4c-d is more stable. The KL divergence from the reference model in Figure 4c-d and Figure 5a is smaller. The entropy in Figure 4a-b remains higher (avoiding collapse). The CountDown experiment in Figure 5e shows that GRPO can catastrophically collapse while GMPO does not.

What was not demonstrated: The paper does not directly measure "stability" through run-to-run variance. All training curves in Figure 4 appear to come from single training runs (no error bars, no multiple seeds reported). The only evidence of run-to-run variability is indirect — the smoothness of the curves and the consistency of patterns across different models and datasets. A quantitative stability metric (e.g., standard deviation of validation accuracy across 3-5 random seeds, or frequency of training collapse) would substantially strengthen the claim. The CountDown collapse in Figure 5e is dramatic but is shown for one MoE architecture on one dataset; it is unclear whether GRPO collapses consistently across seeds or whether this particular run was selected to illustrate the failure mode.

Conditional strength: The stability evidence is strongest for MoE architectures (Figure 5), where GRPO's instability is most acute and GMPO's advantage is most visible. For dense models on standard math training (Figure 4e), the advantage is real but subtler — GRPO does not collapse, it plateaus.

Claim 2: GMPO enhances exploration relative to GRPO

What was demonstrated: Figure 4a-b shows that GMPO maintains higher mean token entropy than GRPO throughout training. This is a direct and convincing metric — higher entropy means the model is less certain about its token predictions, which implies it is considering more diverse continuations. The entropy curves show a clear and persistent gap. Figure 4e-h shows that this higher entropy translates into better validation scores, suggesting that the exploration is productive (finding better reasoning strategies) rather than aimless (merely adding noise).

What was not demonstrated: The paper does not provide qualitative evidence of enhanced exploration. There are no examples showing that GMPO-trained models generate more diverse reasoning chains, try different problem-solving approaches, or exhibit more backtracking/self-correction behavior. The entropy metric captures output distribution diversity, but this could in principle reflect the model being uncertain about superficial word choices rather than genuinely exploring alternative reasoning strategies. The connection from "higher token entropy" to "better reasoning exploration" is plausible but unverified.

Additionally, the paper does not compare GMPO's exploration to other exploration-enhancing methods. The 80/20 rule (Wang et al., 2025) and entropy-based advantage augmentation (Cheng et al., 2025) specifically target exploration through token-level reweighting. An experiment comparing GMPO's entropy preservation to these methods would contextualize whether GMPO's exploration benefit is unique or whether any method that avoids entropy collapse achieves similar gains.

Claim 3: GMPO-7B achieves 4.1% higher Pass@1 than GRPO-7B on five benchmarks

What was demonstrated: Table 1 reports this exact number for the DeepSeek-R1-Distill-Qwen-7B configuration: 63.4% GMPO average vs. 59.3% GRPO average, a difference of 4.1 percentage points.

What qualifies this claim: The 4.1% figure is an average across five benchmarks, and the per-benchmark improvements are highly uneven. GMPO gains 10.8 points on AMC (78.3% vs. 67.5%) and 5.8 points on OlympiadBench (62.5% vs. 56.7%), but only 3.3 points on AIME24 (46.6% vs. 43.3%), 2.4 points on MATH500 (91.4% vs. 89.0%), and actually loses 1.8 points on Minerva (37.9% vs. 39.7%). The average is driven disproportionately by the AMC result. If AMC were removed, the average gain would be roughly 2.4 points — still positive but substantially smaller. The paper does not analyze why GMPO's gains are concentrated on AMC and OlympiadBench, or why it underperforms on Minerva for this model configuration. This benchmark-level heterogeneity suggests that GMPO's benefits are task-dependent in ways the paper does not explore.

Furthermore, the 4.1% figure applies specifically to the R1-Distill variant. For Qwen2.5-Math-7B, the gain is 1.5 points (52.7% vs. 51.2%), and for Qwen2.5-Math-1.5B, it is 1.4 points (43.9% vs. 42.5%). The gain scales with the base model's reasoning capability — larger for the already-reasoning-optimized R1-Distill than for the base math models. This is an interesting pattern (GMPO helps more when the model already reasons well) that the paper notes but does not explain mechanistically.

Claim 4: GMPO is "plug-and-play" and implementation is straightforward

What was demonstrated: Algorithm 1 shows that the GMPO loss function can be implemented in 19 lines of PyTorch code, and the paper states that it can simply replace GRPO's loss computation. The experiments use the same training pipeline, data, and hyperparameters for both GRPO and GMPO (except clipping thresholds), confirming that no additional infrastructure is needed.

What qualifies this claim: While the loss function is indeed simple to implement, the paper does not discuss whether GMPO requires different hyperparameter tuning than GRPO. The clipping threshold sweep in Table 5 shows that GMPO's optimal threshold (e^{-0.4}, e^{0.4}) is different from GRPO's standard (0.8, 1.2). Other hyperparameters — learning rate, number of rollouts G, batch size, number of updates per round — were inherited from Dr.GRPO without modification. It is possible that GMPO would benefit from retuning these as well (e.g., perhaps GMPO's stability allows more updates per round or a higher learning rate), which would add engineering overhead not captured by the "plug-and-play" framing.

Additionally, the paper does not discuss whether GMPO introduces numerical stability concerns beyond what Algorithm 1 handles through log-space computation. The geometric mean involves taking the |o_i|-th root of a product; for very long sequences or for products that approach zero (all importance ratios near 0), the log-space implementation could encounter numerical issues at the extremes of floating-point range. The paper caps sequence length at 3,000 tokens and uses 32-bit floating point, which should be safe, but this is not discussed.

Missing Experiments That Would Strengthen the Paper

  • Multi-seed training runs with statistical comparisons: The paper reports point estimates from what appear to be single training runs. Given that RL training is known to be noisy and seed-dependent, reporting mean and standard deviation across 3-5 seeds (with the same hyperparameters) would establish whether the observed differences exceed run-to-run variance. This is particularly important for the 7B dense model where the gain is only 1.5 points — without variance estimates, it is unclear whether this difference is reliable.

  • Difficulty-stratified evaluation: The paper does not report performance broken down by problem difficulty within each benchmark. Given that the geometric mean suppresses outlier importance ratios, it is plausible that GMPO's benefits are concentrated on problems of specific difficulty levels (much as the compute-optimal scaling paper found that search strategies have difficulty-dependent efficacy). For instance, GMPO might help most on medium-difficulty problems where exploration matters, while very easy problems (where the model already gets them right) and very hard problems (where no amount of exploration helps) might show no difference. Such an analysis would provide mechanistic insight and practical guidance on when to use GMPO.

  • Comparison with DAPO's clip-higher strategy under GRPO: Table 5 shows that widening GRPO's clip range alone does not solve the stability problem (in fact, "GRPO-wide-clip" in Figure 4a only temporarily elevates entropy before it declines). However, the paper does not report a systematic comparison of GRPO performance across the full range of clipping thresholds tested for GMPO. Specifically, does GRPO with (e^{-0.4}, e^{0.4}) clipping collapse, or does it simply underperform GMPO with the same thresholds? This comparison would directly test the paper's central claim that the geometric mean is necessary for the wider clipping to be safe. The Figure 3 "GRPO (0.8, 1.2)" line suggests that even at GRPO's standard narrow clip, the importance ratio range expands during training — running GRPO at (e^{-0.4}, e^{0.4}) would likely be unstable, but this is not empirically verified.

  • Combination with Dr.GRPO's length-aware rewards: The paper uses Dr.GRPO's training setup (no KL penalty) but does not incorporate Dr.GRPO's core contribution — length-aware reward normalization. Since GMPO's 1/|o_i| normalization also addresses a length-related issue (preventing the product from exploding), combining GMPO with Dr.GRPO's explicit length-bias correction might be synergistic. The paper does not ablate this combination.

  • Direct measurement of outlier influence: The paper argues that GMPO works by suppressing outlier tokens' influence on the gradient, but it does not directly measure this. An informative ablation would track the distribution of per-token gradient contributions during GRPO and GMPO training, showing that GRPO has a fatter tail (more tokens with extreme gradient magnitudes) than GMPO. The importance sampling ratio range (Figure 1, right) is a proxy for this, but the gradient itself also includes the policy gradient term ∇_θ log(π_θ), which could amplify or dampen the importance ratio's effect.

  • Longer training horizons: All training runs appear to stop at a predetermined step count (visible in Figure 4 and Figure 5). It is unclear whether GRPO's plateau represents a permanent ceiling or whether additional training would eventually close the gap with GMPO. Running both methods for 2-3× longer would test whether GMPO's advantage is in convergence speed or in asymptotic performance.

  • Evaluation with non-zero temperature and multiple samples: The evaluation uses temperature 0.0 (greedy decoding) for language tasks, which measures the model's most likely output. If GMPO truly enhances exploration and preserves output diversity, evaluating with temperature > 0 and Pass@k (e.g., Pass@8 or Pass@16) might show larger gaps, since GMPO-trained models might have more diverse high-quality outputs in their distribution. This would provide convergent evidence for the exploration claim and test a practically important capability (ensemble or best-of-N evaluation).

6. Limitations and Trade-offs

Single Benchmark Family, Single Task Domain — Generalization Is Unestablished

The constraint: All of GMPO's language-only evaluations are on mathematical reasoning benchmarks (AIME24, AMC, MATH500, Minerva, OlympiadBench). All training uses either MATH Levels 3–5 (8,523 problems) or DeepScaleR (~40,000 mathematics problems). The single multimodal result on Geometry3K extends the domain to visual geometry but remains within the same task family (formal problem-solving with verifiable binary rewards). The paper does not evaluate on code generation, logical reasoning, open-ended QA, dialogue, summarization, or any task where rewards are non-binary, learned, or subjective.

The consequence: The mechanism by which GMPO works — suppressing outlier importance sampling ratios that arise when the old and current policies disagree sharply on token probabilities — is architecture-agnostic. However, when and how frequently such outliers arise is a function of the training data, the reward signal, and the nature of the task. Mathematical reasoning with binary verifiable rewards may be a particularly favorable setting: correct answers receive a reward of exactly 1, incorrect answers receive exactly 0, and the advantage normalization (r_i - mean)/std produces cleanly separated positive and negative advantage signals. In domains with continuous rewards (e.g., learned reward models), noisy rewards (e.g., human preference scores), or tasks where "partial correctness" matters (e.g., code generation where output is tested against multiple unit tests), the distribution of importance-weighted rewards ρ_t(θ)Â_i could differ substantially. It is possible that in those settings, the arithmetic mean's greater sensitivity is actually beneficial — allowing the model to strongly upweight tokens from high-reward sequences even if those tokens have extreme importance ratios. The paper provides no evidence either way.

What evidence exists in the paper: None for any non-math, non-geometry domain. The Geometry3K result (+1.4%) is directionally positive for multimodal generalization but is a single benchmark trained on a single dataset. The CountDown experiments in Appendix B are still mathematical reasoning (arithmetic puzzles), though they test a different reward structure (step-by-step operations to reach a target number).

Mitigation status: Not addressed. The paper does not claim generalization beyond mathematical reasoning and does not suggest that GMPO would work for other domains. However, the "plug-and-play" framing and the gradient analysis (Equations 5–6) implicitly suggest broad applicability, since the gradient-weighting argument does not depend on the reward type. Future work explicitly testing GMPO on code generation (HumanEval, MBPP), scientific reasoning, or RLHF-style preference optimization would be necessary to establish whether the geometric mean's benefits are universal or math-specific.

The 4.1% Headline Number Is Driven by a Single Benchmark's Outsize Gain

The constraint: The paper's most prominent empirical claim — "GMPO-7B improves the average Pass@1 of GRPO by up to 4.1%" — appears in the abstract and is repeated throughout the paper as the primary quantitative result. This figure applies to the DeepSeek-R1-Distill-Qwen-7B configuration, where GMPO achieves 63.4% average vs. GRPO's 59.3% across five benchmarks.

The consequence: The 4.1-point average gain is composed of highly unequal per-benchmark contributions. On AMC, GMPO gains 10.8 points (78.3% vs. 67.5%). On OlympiadBench, the gain is 5.8 points (62.5% vs. 56.7%). On AIME24, the gain is 3.3 points (46.6% vs. 43.3%). On MATH500, the gain is 2.4 points (91.4% vs. 89.0%). On Minerva, GMPO loses 1.8 points (37.9% vs. 39.7%). The AMC result alone accounts for more than half of the total gain across the five benchmarks. If AMC were excluded, the four-benchmark average gain would be roughly 2.4 points — still positive but less than 60% of the headline figure. If the claim were stated per-benchmark ("GMPO improves Pass@1 by 2.4–10.8 points depending on the benchmark"), a practitioner evaluating whether to adopt GMPO for their specific use case would have a more accurate picture.

Furthermore, the paper does not analyze why the AMC gain is so large. AMC consists of 83 intermediate-difficulty multiple-choice problems. One hypothesis: multiple-choice problems have a narrower space of correct reasoning paths than open-ended problems, so the entropy preservation GMPO provides is less beneficial for exploration (since there are fewer valid paths to discover) and more beneficial for avoiding premature convergence to a wrong answer. Alternatively, the result could be an artifact of the 83-problem test set size — a few lucky correct answers could swing the Pass@1 rate by several points. The paper provides no diagnostic to distinguish these possibilities.

What evidence exists in the paper: Table 1 provides the full per-benchmark breakdown. The abstract and introduction selectively quote the 4.1% figure without noting the Minerva underperformance or the AMC-driven composition of the average.

Mitigation status: The paper does not discuss this imbalance. It reports the per-benchmark numbers in Table 1, so a careful reader can reconstruct the composition, but the text narrative around the 4.1% figure treats it as a summary statistic without qualification. A sensitivity analysis (e.g., average with each benchmark removed, or weighted average by benchmark size) would make the claim's robustness more transparent.

No Statistical Replication — Single Training Runs Without Variance Estimates

The constraint: All training curves (Figure 4, Figure 5) and all final accuracy numbers (Tables 1–5) appear to come from single training runs. The paper reports point estimates without confidence intervals, error bars, or standard deviations across random seeds. The training curves are smooth enough to suggest stability within a run, but run-to-run variance — a well-known challenge in RL training — is completely unmeasured.

The consequence: RL fine-tuning of LLMs is noisy. Different random seeds can produce different learning trajectories due to stochasticity in rollout sampling, mini-batch composition, and optimization dynamics. The paper's central quantitative claims — GMPO outperforms GRPO by 1.4–4.1 percentage points depending on model configuration — are moderate effect sizes. For the Qwen2.5-Math-7B comparison (52.7% vs. 51.2%, a 1.5-point gap), it is entirely possible that this difference falls within the range of run-to-run variance. Without multi-seed statistics, a practitioner cannot assess whether adopting GMPO would reliably improve their training outcomes or whether the reported advantage might disappear under different random initializations.

This limitation is most acute for the smaller gains. The 1.4-point improvement at 1.5B scale (43.9% vs. 42.5%) and the 1.5-point improvement for Qwen2.5-Math-7B are small enough that a few-seed replication could reverse the conclusion. The 4.1-point gain for R1-Distill-7B is larger and more likely robust, but even there, the per-benchmark variability (losing on Minerva) suggests that individual benchmark results could be seed-sensitive. The CountDown collapse result (Figure 5e), while dramatic, is shown for one run — does GRPO always collapse on CountDown with that MoE architecture, or was this a particularly unstable seed?

What evidence exists in the paper: None. The paper does not mention running multiple seeds, does not report variance, and does not discuss statistical significance.

Mitigation status: Not addressed. The training curves in Figures 4 and 5 are presented as smooth lines without confidence bands, implying single runs. The ablation tables (Tables 4 and 5) similarly report point estimates. Standard practice in the RL-for-LLMs literature at the time of this paper's writing does not consistently require multi-seed reporting, but for a contribution whose primary claim is improved training stability (a property that manifests partly through reduced run-to-run variance), the absence of variance estimates is a significant gap. A simple 3-seed replication for the main 7B comparison would substantially strengthen the paper's central claims.

Difficulty Estimation Is Absent — No Analysis of Where GMPO Helps or Hurts

The constraint: The paper evaluates GMPO only on aggregate benchmark-level Pass@1 accuracy. It does not analyze performance stratified by problem difficulty within each benchmark, nor does it analyze which types of problems benefit most from the geometric mean. The five benchmarks span a range of difficulties (AIME24 and OlympiadBench are olympiad-level; AMC is intermediate; MATH500 and Minerva cover mixed difficulty), but the per-benchmark results (Table 1) do not form a clean difficulty gradient — GMPO gains 10.8 points on the intermediate-difficulty AMC but only 3.3 points on the olympiad-level AIME24, which is not monotonic with difficulty.

The consequence: Without difficulty-stratified analysis, a practitioner cannot predict whether GMPO will help on their specific problem distribution. If a deployment involves primarily easy problems (where the base model already performs well), the geometric mean might provide negligible benefit since there are few outlier tokens to suppress. If the problems are very hard (near the base model's capability frontier), GMPO's entropy preservation might help exploration find novel reasoning strategies — or, alternatively, the model might need the strong gradient signals that the arithmetic mean provides to make decisive progress on tokens where the policy must change dramatically. The paper's own results hint at this complexity: GMPO underperforms GRPO on Minerva with the R1-Distill model (37.9% vs. 39.7%), and Minerva contains graduate-level problems that are among the hardest in the evaluation suite. This single data point raises the possibility that for the hardest problems, the geometric mean's outlier suppression might be too aggressive, damping legitimate gradient signals from tokens where large policy shifts are genuinely necessary.

This limitation is particularly relevant given the paper's theoretical framing. The AM-GM inequality proof establishes that GMPO's loss magnitude is bounded above by GRPO's loss magnitude, but this bound applies uniformly — it does not distinguish between cases where reduced magnitude is beneficial (outlier-dominated sequences) and cases where it might be detrimental (sequences where large importance ratios reflect legitimate, necessary policy changes). A difficulty-stratified analysis would reveal whether there is a problem difficulty regime where GRPO's greater sensitivity is actually advantageous.

What evidence exists in the paper: The per-benchmark breakdown in Table 1 provides a coarse difficulty signal, but benchmarks mix difficulties internally (MATH500 includes problems from Level 1 through Level 5 of the MATH dataset). There is no within-benchmark difficulty stratification. The paper does not report Pass@1 broken down by whether the base model (pre-RL) could solve the problem, which would be a natural difficulty proxy.

Mitigation status: Not addressed. The paper does not discuss difficulty as a relevant variable for GMPO's efficacy. This is a missed opportunity given the paper's theoretical framework: the gradient analysis in Equations 5–6 suggests that GMPO's advantage is largest when the variance of per-token importance ratios is high (since the geometric mean suppresses the variance). Problem difficulty could systematically affect this variance — easy problems might have low variance (the model is confident in most tokens), while hard problems might have high variance (the model is uncertain and exploration produces diverse importance ratios). An analysis correlating GMPO's per-problem gain with some measure of token-level importance ratio variance would connect the theoretical mechanism to practical deployment guidance.

Training Cost of Difficulty-Aware Evaluation Is Not Assessed

The constraint: This limitation concerns not an explicit claim in the paper but a significant omission in the experimental design. The paper evaluates GMPO at the end of training using Pass@1 with temperature 0.0 (greedy decoding, one generation per problem). This measures the single most likely output. However, the paper's central argument is that GMPO improves training dynamics — higher entropy, more stable gradients, sustained exploration — which should produce a model whose output distribution is qualitatively different from GRPO's, not just one whose argmax is better.

The consequence: A model that maintains higher entropy during training might have a more diverse output distribution at inference time. If this diversity translates into multiple distinct correct reasoning paths (rather than redundant variations), then GMPO-trained models might benefit more from test-time compute strategies like majority voting or best-of-N sampling than GRPO-trained models. The paper's evaluation protocol (temperature 0.0, single generation) is blind to this potential advantage. Conversely, higher entropy could also mean the model retains unhelpful uncertainty — producing a broader distribution of both correct and incorrect answers. Without evaluating Pass@k (e.g., Pass@8, Pass@16) or measuring output diversity directly (e.g., number of distinct correct solutions generated), the paper cannot distinguish between "useful exploration that produces multiple valid reasoning strategies" and "residual uncertainty that adds noise without improving the best answer."

This is a practical concern: if a practitioner deploys GMPO-trained models with temperature 0.0, they may see only the 1.5–4.1% Pass@1 improvement reported in the tables. But if they deploy with best-of-N sampling (a common practice for reasoning tasks), the gap might be larger or smaller depending on whether GMPO's entropy preservation produces genuinely useful diversity. The paper provides no evidence on this point.

What evidence exists in the paper: The only multi-sample evaluation is for the multimodal Geometry3K task, which uses temperature 0.5 and 16 generations. The reported Pass@1 of 54.7% vs. 53.3% is computed from these 16 samples (the paper does not specify the exact Pass@1 computation method for multiple samples, but standard practice is to count a problem as correct if any of the 16 answers matches). This provides weak evidence that GMPO's advantage persists under non-greedy sampling, but it is a single result with a different temperature and task domain, so it does not isolate the effect of sampling budget.

Mitigation status: Not addressed. The paper does not evaluate Pass@k for any language-only benchmark and does not discuss whether GMPO's higher training entropy translates into more diverse or higher-quality output distributions at inference time.

The Geometric Mean's Downside: All Tokens Get the Same Weight Regardless of Informativeness

The constraint: The paper presents GMPO's shared per-token gradient weight (Equation 6, where every token in a sequence is multiplied by the same geometric mean of importance ratios) as a strict improvement over GRPO's per-token weights (Equation 5, where each token gets its individual ρ_t(θ)). The gradient analysis frames GRPO's per-token weighting as a source of instability and GMPO's uniform weighting as a source of stability.

The consequence: This uniform weighting discards potentially useful information about which tokens are driving the policy disagreement. In a long reasoning chain, different tokens play different roles. Some tokens are structural (formatting, transition words, equals signs) — these should have importance ratios near 1 and contribute little gradient signal regardless of weighting scheme. Some tokens are semantically critical (the key algebraic manipulation, the logical inference step) — these are precisely where the old and current policies should disagree strongly if the model is learning, and large importance ratios at these tokens may reflect genuine learning progress rather than noise. GMPO's uniform weighting dilutes the gradient contribution from these critical tokens by averaging their high importance ratios with the near-1 ratios of structural tokens. GRPO's per-token weighting, for all its instability, at least preserves the information that certain tokens are more "surprising" to the current policy and therefore more deserving of gradient emphasis.

This is a fundamental tradeoff the paper does not acknowledge: uniform weighting suppresses noise but also suppresses signal. The noise comes from outlier tokens where large importance ratios reflect sampling variance or unstable probability estimates; the signal comes from tokens where large importance ratios reflect genuine policy improvement. The geometric mean cannot distinguish between these two cases. The paper provides no analysis of whether GMPO's uniform weighting slows down learning on tokens that genuinely require large policy updates, or whether this effect matters compared to the stability benefits.

What evidence exists in the paper: Indirect and mixed. The fact that GMPO underperforms GRPO on Minerva with the R1-Distill model (37.9% vs. 39.7%) could reflect a case where GRPO's ability to strongly weight critical tokens outweighs its instability cost — Minerva problems are graduate-level and may require large policy shifts on specific decisive tokens. The fact that GMPO outperforms on AMC and OlympiadBench suggests that for those benchmarks, the noise-suppression benefit dominates. But this is post-hoc speculation; the paper does not analyze token-level importance ratios on correct vs. incorrect sequences to test whether GMPO's uniform weighting suppresses useful gradient signal.

Mitigation status: Not addressed. The gradient analysis in Section 3 and Appendix A presents the uniform weighting as an unambiguous advantage, with no discussion of the potential downside. The paper does not propose any variant that would selectively preserve informative high-importance tokens while suppressing noisy ones (e.g., a trimmed mean, a weighted geometric mean, or an adaptive weighting scheme based on token-level variance estimates). This is a clear direction for future work that the paper does not identify.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper establishes that the choice of aggregation function in sequence-level RL objectives is not a neutral implementation detail but a first-class design axis with substantial and measurable consequences for training dynamics. Prior to GMPO, the RL-for-LLMs community treated the arithmetic mean of per-token contributions as an implicit default — every GRPO variant cataloged in Section 2.1 (DAPO, Dr.GRPO, GPG, PRIME, OpenReasoner-Zero, and over a dozen others) accepted this aggregation as given and worked around its consequences through an expanding ecosystem of peripheral modifications: dynamic sampling schedules, reward reshaping heuristics, adaptive clipping strategies, entropy-based token reweighting, and curriculum learning over training data.

GMPO demonstrates that a single mathematical substitution — replacing the arithmetic mean with the geometric mean — achieves what this collective engineering effort has been pursuing through far more complex means: stable training that sustains exploration without sacrificing convergence. The 19-line PyTorch implementation (Algorithm 1) outperforms methods that introduce entire new training pipelines (OpenReasoner-Zero's 129k curated samples with curriculum learning: 52.7% vs. 45.9%), new reward modeling architectures (PRIME-Zero's implicit rewards: 52.7% vs. 48.0%), and preference-tree-based reward design (Eurus: 52.7% vs. 48.0%). This is not merely a performance result — it is an argument about where research effort should be directed: toward the mathematical structure of the optimization objective itself rather than toward increasingly elaborate constraints layered on top of it.

The magnitude of this shift should be understood precisely. This is not a paradigm shift in the sense of overturning RL fundamentals — the paper works entirely within the PPO/GRPO framework and preserves the clipped surrogate objective, group-based advantage estimation, and importance sampling correction that define that family. Rather, it is a reframing that changes what practitioners diagnose first when training becomes unstable. The dominant diagnostic impulse in the GRPO variant literature has been to ask: "What constraint should we add or relax? Should we clip differently? Normalize rewards differently? Filter the data differently?" GMPO reframes the question to: "Is the aggregation function amplifying noise? Would a different mean stabilize the updates without constraining exploration?" This is a shift from constraint-first to objective-first debugging of RL training instability, and it is likely to influence how future RL-for-LLMs algorithms are designed from scratch rather than patched post-hoc.

The paper also reconciles a tension that was latent in the GRPO literature but not explicitly articulated. DAPO (Yu et al., 2025) identified that clipping limits exploration and proposed widening the clipping range, but this widening is fundamentally unsafe under GRPO because the arithmetic mean's sensitivity to outliers means wider clipping directly translates into larger gradient variance and potential training collapse. The 80/20 rule work (Wang et al., 2025) identified that high-entropy minority tokens are important for effective learning, but GRPO's arithmetic mean drives entropy collapse precisely by overweighting outlier tokens that produce aggressive, entropy-reducing updates. These two lines of work were pulling in opposite directions — one wanted wider exploration, the other wanted to preserve specific tokens that exploration would surface — and neither could be fully realized within GRPO's constraints. GMPO resolves this tension by showing that stability and exploration are not fundamentally in tension; they are only in tension when the aggregation function is outlier-sensitive. By decoupling these two properties (Figure 4: GMPO simultaneously achieves higher entropy and lower KL divergence than GRPO), GMPO enables both the wider clipping DAPO advocates and the sustained high-entropy token exploration the 80/20 rule targets.

The research directions that become more attractive after this work:

  • Objective-function engineering for RL stability. Rather than designing new clipping schedules or reward normalization schemes, researchers should systematically explore alternative aggregation functions — trimmed means, Winsorized means, Huber-style robust estimators, median-like differentiable approximations — each with different gradient-weighting properties. The paper's gradient analysis (Equations 5–6) provides a template for evaluating any candidate: compute the per-token gradient weights and assess whether they suppress outliers while preserving signal.

  • Verifier and reward model robustness. The paper identifies verifier over-optimization as a parallel stability concern (citing the prior paper's discussion of reward hacking in Section 7). GMPO's insight — that stability should be a property of the objective, not just the constraints — applies equally to verifier design: reward models that produce smoothly varying scores rather than brittle binary judgments would combine naturally with outlier-resistant loss functions.

  • Long-horizon RL training for reasoning. GMPO's entropy preservation (Figure 4a-b) and avoidance of training collapse on MoE models (Figure 5e) suggest that geometric-mean-based objectives could enable substantially longer RL training runs. Current GRPO training often stops when entropy collapses and validation plateaus; GMPO's sustained entropy and continued validation improvement (Figure 4e-h) imply that the effective training horizon could be extended, potentially unlocking reasoning capabilities that require more optimization steps to emerge.

The research directions that become less attractive:

  • Incremental clipping-range tuning for GRPO. The paper's Figure 3 and Table 5 collectively show that even with GMPO's inherent stability, there is an optimal clipping range — too narrow limits exploration (52.4%), too wide or absent introduces instability (52.1–52.3%). But the optimal range under GMPO, (e^{-0.4}, e^{0.4}) ≈ (0.67, 1.49), is substantially wider than anything GRPO can safely use. This suggests that the returns to further clipping-range engineering under GRPO are fundamentally limited — the arithmetic mean will always produce outliers that require clipping, and widening the clip to enable exploration will always risk instability. Future work on clipping strategies should focus on algorithms whose objectives already constrain importance ratio variance.

  • Data filtering as a primary stability intervention. Methods like CPPO (pruning low-advantage completions), PODS (training on informative rollouts), and SRPO (history resampling) address stability by controlling what data the model sees. GMPO's results suggest that when the objective function is properly designed, the model can safely learn from noisier or more diverse data without elaborate filtering. This does not make data quality irrelevant, but it reduces the pressure to design sophisticated filtering heuristics.

Follow-Up Research This Work Enables

Multi-seed stability quantification with explicit variance metrics. The paper's central claim is improved stability, yet all training curves come from what appear to be single runs without error bars or confidence intervals. A high-priority follow-up would train GMPO and GRPO across 5–10 random seeds on the same MATH Levels 3–5 setup with Qwen2.5-Math-7B, reporting mean and standard deviation of final Pass@1, and more importantly, metrics that directly quantify stability: (a) the standard deviation of gradient norms across training steps, (b) the step at which entropy falls below a threshold (e.g., 0.1 nats), (c) the frequency of "collapse events" defined as validation accuracy dropping below 90% of the running maximum. If GMPO's stability advantage is genuine, it should manifest as reduced variance across seeds — not just a higher mean — and as later or absent entropy-collapse events. The CountDown collapse result (Figure 5e) is striking but shown for a single run; a multi-seed replication would establish whether GRPO's collapse on that task is deterministic or stochastic, and whether GMPO eliminates or merely reduces collapse probability.

Difficulty-stratified evaluation to identify when the geometric mean helps versus hurts. The paper's per-benchmark results (Table 1) show that GMPO's gains are highly uneven: +10.8 points on AMC, +5.8 on OlympiadBench, but -1.8 on Minerva for the R1-Distill model. This pattern is not analyzed. A follow-up study should bin problems by difficulty within each benchmark. The most natural difficulty proxy is the pre-RL model's Pass@1 on each problem (estimated via multiple samples if needed, following the compute-optimal scaling paper's oracle difficulty estimation approach). The hypothesis to test: GMPO helps most on medium-difficulty problems where exploration is valuable but the model has some initial traction, and may be neutral or harmful on very easy problems (where the model already gets them right and exploration adds noise) and very hard problems (where the model needs aggressive gradient signals on decisive tokens that the geometric mean's uniform weighting might suppress). This experiment would provide a decision rule for practitioners: "Use GMPO if your problem distribution has Pass@1 in the [X%, Y%] range under the base model; prefer GRPO outside that range."

Token-level gradient-weight analysis to measure signal suppression versus noise suppression. The paper argues that GMPO's uniform per-token weighting (Equation 6) is beneficial because it suppresses outlier-driven noise, but Section 6 identified the unexamined downside: uniform weighting may also suppress legitimate gradient signal on tokens where large importance ratios reflect genuine learning. A follow-up study should instrument the training loop to log, for each sequence in each update step: (a) the per-token GRPO weight (individual ρ_t), (b) the per-token GMPO weight (shared geometric mean), (c) the token's position in the sequence, and (d) whether the sequence is ultimately correct or incorrect. The analysis would ask: for sequences that are correct, are there specific token positions where GRPO's per-token weight is substantially higher than GMPO's shared weight, and do those tokens correspond to semantically meaningful reasoning steps (identified via a simple heuristic like presence of mathematical operators, logical connectives, or answer-extraction tokens)? If such "signal-carrying" tokens exist and are routinely suppressed by GMPO's uniform weighting, it would motivate hybrid approaches — e.g., a weighted geometric mean where token weights are proportional to some estimate of the token's informativeness — that could recover the lost signal while preserving outlier suppression.

Combination with Dr.GRPO's length-aware reward normalization. The paper uses Dr.GRPO's training setup (no KL penalty, identical data and hyperparameters) but does not incorporate Dr.GRPO's core contribution: length-dependent accuracy normalization that corrects for the bias that longer responses tend to receive higher rewards under verifiable evaluation. GMPO's 1/|o_i| normalization in the geometric mean addresses a different length issue — preventing the product from exploding with sequence length — but does not address reward bias. These two length-related mechanisms are complementary: Dr.GRPO corrects the reward signal for length, GMPO corrects the loss aggregation for length. A natural experiment would train GMPO with Dr.GRPO's length-aware reward normalization on MATH Levels 3–5 with Qwen2.5-Math-7B, comparing four conditions in a 2×2 design: (GRPO vs. GMPO) × (standard rewards vs. Dr.GRPO length-aware rewards). The hypothesis is that the combination would outperform either alone, since each addresses a distinct length-related pathology. This experiment is low-cost — it requires only modifying the reward computation, not the training infrastructure — and would directly inform practitioners who are already using or considering Dr.GRPO.

Extension to code generation and non-binary reward domains. All of GMPO's evaluations (except CountDown) use binary verifiable rewards: the answer is either exactly correct (reward 1) or not (reward 0). Code generation tasks offer a richer reward structure: unit tests provide partial-credit signals (7/10 tests passing vs. 3/10), and code execution feedback provides dense per-step correctness information. This changes the distribution of advantages Â_i — instead of two clusters (positive for correct, negative for incorrect), the advantage distribution becomes more continuous. The geometric mean's outlier-suppression properties may interact differently with continuous advantages. A concrete experiment: train GMPO and GRPO on a code generation dataset (e.g., APPS or CodeContests) with reward defined as the fraction of unit tests passed, and evaluate Pass@1 and Pass@k on HumanEval and MBPP. The hypothesis is that GMPO's stability advantage persists but may be smaller in magnitude because continuous advantages produce fewer "extreme" importance-weighted rewards (the reward signal itself is less discrete, so the advantage normalization produces less extreme Â_i values, meaning there are fewer outlier ρ_t Â_i products for the geometric mean to suppress). A null result (GMPO ≈ GRPO on code) would be informative: it would suggest that GMPO's benefits are specific to binary-reward domains and would motivate research into aggregation functions tailored to continuous-reward settings.

GMPO as a building block for iterative self-improvement loops. The paper does not explore combining GMPO with the self-improvement paradigm where models generate their own training data (STaR, ReST^EM, rejection sampling fine-tuning). In such loops, the model's outputs from one iteration become the training data for the next. Training stability is critical because instability in one iteration propagates: if the model overfits to noisy self-generated solutions in iteration k, it produces even noisier solutions in iteration k+1, creating a divergence cascade. GMPO's sustained entropy and lower KL divergence from the reference model (Figure 4c-d) are precisely the properties needed to prevent this cascade. A concrete experiment: run 3–5 iterations of self-improvement where the model generates solutions to MATH training problems, filters for correct solutions via answer verification, and fine-tunes on those solutions, comparing GMPO and GRPO as the optimizer in each iteration. Measure not just final accuracy but also the stability of the process — does GRPO's entropy collapse compound across iterations, causing later iterations to produce less diverse or lower-quality solutions? Does GMPO maintain solution diversity across iterations? This experiment would test whether GMPO's training-dynamics benefits translate into more robust self-improvement, which is a critical capability for scaling reasoning beyond available human-annotated data.

Practical Applications and Downstream Use Cases

Training Mixture-of-Experts models for reasoning, where GRPO is known to collapse. The paper's strongest stability evidence comes from MoE architectures. Figure 5e shows GRPO's validation score on CountDown collapsing at ~250 steps with a Qwen2.5-200M MoE model, while GMPO maintains stable improvement. Table 2 shows GMPO-32B (Qwen3, 128 experts, 8 active per token) achieving 96.7% on MATH500 versus GRPO's 94.6% — a 2.1-point gain at near-ceiling performance. For teams training large MoE models (which are increasingly common as the default architecture for frontier models), GMPO offers a direct replacement for GRPO that eliminates a known failure mode. The benefit is not speculative — the collapse in Figure 5e represents wasted GPU-hours on a failed training run. Adopting GMPO for MoE training would reduce the risk of such collapses and the associated engineering time spent diagnosing and restarting failed runs. The implementation cost is minimal: replacing the GRPO loss computation with Algorithm 1 requires changing fewer than 20 lines of code in most training frameworks.

Long-horizon chain-of-thought RL training where entropy collapse limits reasoning depth. The paper caps response length at 3,000 tokens, but frontier reasoning models are being pushed toward substantially longer chains of thought (10,000+ tokens). Longer sequences compound GRPO's instability: more tokens means a higher probability that at least one token will have an extreme importance ratio, and the arithmetic mean gives that token direct influence on the loss. GMPO's geometric mean suppresses this influence exponentially (the |o_i|-th root operation means that for a 10,000-token sequence, an outlier ρ = 100 contributes roughly 100^{1/10000} ≈ 1.0005 to the geometric mean — essentially invisible). For teams scaling chain-of-thought length, GMPO provides a loss function whose stability does not degrade with sequence length. The entropy preservation in Figure 4a-b (GMPO maintains higher entropy throughout training on both MATH Level 3-5 and the harder DeepScaleR dataset) is particularly important for long-form reasoning: if the model's entropy collapses early in training on a 10,000-token reasoning task, it will never explore the diverse reasoning strategies needed to solve harder problems. GMPO's sustained entropy directly enables the exploration that long-form reasoning requires.

Cost-efficient RL fine-tuning in resource-constrained settings. The paper's results show that GMPO consistently outperforms GRPO across model scales (1.5B, 7B, 32B) without requiring additional computational resources — the training setup (hardware, data, number of rollouts, update frequency) is identical for both methods. For academic labs, startups, or teams without access to large GPU clusters, this means achieving better final model performance from the same training budget. The 4.1% average gain on R1-Distill-7B or the 2.1% gain on Qwen3-32B MATH500 represent meaningful accuracy improvements at zero additional cost. Moreover, GMPO's reduced training instability means fewer failed runs that must be discarded and restarted — a direct reduction in wasted compute. For a small team running 8×A800 GPUs for several days per training run (the paper's setup), avoiding even one collapsed run saves significant time and money.

When to Prefer This Method

The paper itself articulates a clear tradeoff between GMPO and GRPO, but does not position GMPO against the broader landscape of GRPO variants as a prescriptive decision rule. The following is therefore grounded only in the paper's explicit comparisons:

  • Prefer GMPO over GRPO when training Mixture-of-Experts models. The CountDown collapse (Figure 5e) and the consistent stability advantage on MoE architectures (Figure 5a-d) make this the strongest recommendation. The failure mode is severe (complete training collapse) and GMPO eliminates it.

  • Prefer GMPO over GRPO when sustained exploration matters and entropy collapse is a concern. If the task requires discovering diverse reasoning strategies (complex math, multi-step planning, creative problem-solving), GMPO's higher maintained entropy (Figure 4a-b) directly supports the exploration needed. This is especially relevant for training runs where GRPO's validation score plateaus (Figure 4e) — switching to GMPO may unlock continued improvement.

  • Prefer GMPO over GRPO when training stability is more valuable than marginal per-benchmark optimization. The paper shows GMPO sometimes underperforms GRPO on individual benchmarks (Minerva with R1-Distill: 37.9% vs. 39.7%; MATH500 with Qwen2.5-Math-7B: 82.0% vs. 83.4%) while improving the average. If the deployment requires reliable training that does not collapse, and the cost of a failed run is high, GMPO's stability advantage may outweigh the small per-benchmark regressions.

The paper does not provide evidence to recommend GMPO over non-GRPO methods (PPO, GPG, PRIME) or over GRPO variants that address orthogonal concerns (Dr.GRPO's length bias, DAPO's dynamic sampling), and does not claim to. The "plug-and-play" framing suggests that GMPO can be combined with these methods rather than substituted for them — a practitioner using Dr.GRPO for length-bias correction could adopt GMPO as the loss function while preserving Dr.GRPO's reward normalization, potentially gaining the benefits of both.