GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization

ArXiv: 2601.05242

🎯 Pitch

GRPO silently collapses multiple rewards into identical advantage values, discarding critical training-signal resolution and causing early failure in multi-reward RL. The proposed GDPO decouples per-reward normalization, preserving an exponentially larger number of distinct advantage groups and consistently outperforming GRPO on tool-calling, math reasoning, and code reasoning.

---

1. Executive Summary

This paper introduces Group reward-Decoupled Normalization Policy Optimization (GDPO), a new policy optimization method for multi-reward reinforcement learning that decouples the group-wise normalization of individual rewards before aggregation, preserving fine-grained distinctions across reward combinations that GRPO collapses into identical advantage values. The authors demonstrate across three tasks—tool calling (Qwen2.5-Instruct), math reasoning (DeepSeek-R1 and Qwen3-4B-Instruct), and code reasoning (DeepSeek-R1-7B)—that GDPO consistently improves both correctness metrics (accuracy, pass rate) and constraint-adherence metrics (format compliance, length-exceeding ratio, bug ratio) compared to GRPO, with gains of up to 6.3% higher AIME accuracy and up to 4× better format compliance on tool-calling. The paper establishes that GRPO's summed-reward normalization is fundamentally unsuitable for multi-reward RL because it compresses the training signal to only two distinct advantage groups in common configurations, and that this collapse is only partially mitigated by removing the standard deviation term, while GDPO's per-reward decoupling preserves an exponentially larger number of distinct advantage groups across all reward and rollout counts.

2. Context and Motivation

The Core Problem: GRPO Wasn't Designed for Multiple Rewards, Yet Everyone Uses It That Way

The fundamental question this paper tackles is deceptively simple: are the optimization algorithms we use for multi-reward RL actually suitable for that purpose? Specifically, the paper questions whether Group Relative Policy Optimization (GRPO)—the dominant algorithm in modern LLM alignment pipelines—is appropriate when optimizing for multiple, potentially competing reward signals simultaneously.

This matters because the field has rapidly converged on a specific engineering pattern without examining its foundations. As language models grow more capable, practitioners increasingly want to optimize for diverse objectives simultaneously: correctness of answers, adherence to format constraints, efficiency of reasoning (response length), safety, code quality, and many others. These preferences are often heterogeneous—an ideal model should produce answers that are simultaneously correct, concise, properly formatted, and safe. Yet the standard approach in recent work has been to simply sum all reward components into a single scalar and apply GRPO exactly as it was originally designed for single-reward optimization, without asking whether this practice is theoretically sound.

The stakes are high for several reasons the paper demonstrates:

• Suboptimal convergence: The paper shows GRPO's training curves for correctness reward on math reasoning (Figure 5) begin declining after approximately 400 steps—a partial training collapse that GDPO eliminates entirely. On tool-calling, GRPO's format reward convergence is markedly lower than GDPO's (Figure 4).

• Collapsed training signals: As illustrated in Figure 2, GRPO maps six distinct reward combinations in a two-rollout, two-binary-reward setting into only two distinct advantage groups. This means the optimization algorithm cannot distinguish between a rollout that satisfies both rewards ($r=2$) versus only one ($r=1$) when the other rollout in the group scores zero—both produce identical normalized advantages of $(-0.7071, 0.7071)$. The model receives the same learning signal for qualitatively different behaviors.

• Training instability: The paper documents cases where GRPO training fails entirely. On the 1.5B tool-calling task with the standard deviation term removed (GRPO w/o std), the format reward never improves (Figure 1b), resulting in 0% format compliance on downstream evaluation (Table 2). This is a catastrophic failure mode that would be unacceptable in production deployments.

• Misleading reward weight semantics: When practitioners attempt to encode priorities by assigning different weights to rewards, the paper shows this doesn't work as intended when objectives differ substantially in difficulty (Section 4.2.1, Figure 6). Reducing the length reward weight from 1.0 to 0.75 or 0.5 has negligible impact on the length-exceeding ratio, meaning the model ignores the weighting signal and continues optimizing the easier objective. This is a practical failure of reward design that stems from how GRPO processes multiple rewards internally.

Why This Gap Has Been Overlooked

The multi-reward RL for language models has a peculiar history that explains why this gap persisted. The initial excitement around GRPO came from DeepSeek-R1 (Guo et al., 2025), which demonstrated that GRPO could elicit complex reasoning behaviors from LLMs using only a single accuracy-based reward. The success was dramatic—"aha moments" where models spontaneously learned to double-check their work—and the community rapidly adopted GRPO as the standard backbone for RL-based alignment.

As researchers began adding additional reward components (format rewards, length penalties, safety constraints), they inherited the GRPO optimization machinery wholesale. The focus was on reward design—what new reward functions to add, how to balance them, what thresholds to set—rather than on optimization fidelity—whether the underlying algorithm could actually optimize the multi-dimensional signal being created. This is exactly the pattern the paper identifies in its opening:

"recent work [1, 3, 5] has largely focused on the reward design itself and often directly relied on applying Group Relative Policy Optimization (GRPO) directly for multi-reward RL optimization, often without examining whether GRPO is well-suited for optimizing combinations of heterogeneous rewards."

This pattern is not unusual in machine learning: when a technique works well in one context, practitioners extend it to new contexts without revisiting assumptions. But GRPO was designed with an implicit assumption of a single, semantically coherent reward signal. When rewards are summed, the group-wise normalization in Equation 2 operates on the sum distribution, not on the individual reward distributions. This causes information loss because distinct reward combinations that happen to sum to the same total are indistinguishable after normalization.

The theoretical issue is best understood through combinatorics. When you have $n$ binary rewards and $G$ rollouts per group, the space of possible reward combinations is $2^{nG}$ (each rollout has $2^n$ possible reward vectors). After summing across rewards, you get a scalar total per rollout taking values in $\{0, 1, ..., n\}$, meaning the space of possible group configurations collapses to at most $(n+1)^G$ distinct patterns. But GRPO's mean-and-standard-deviation normalization further collapses this space because the normalized advantage depends only on the relative ordering and dispersion of the summed totals, not their absolute magnitudes. The paper demonstrates this collapse concretely in Figure 3: with 16 rollouts and 2 binary rewards, GRPO produces only ~10 distinct advantage groups, GRPO without standard deviation produces ~30, while GDPO produces over 175.

Prior Approaches and Their Shortcomings

GRPO (the status quo). GRPO (Shao et al., 2024) was introduced as a simplification of PPO that eliminates the need for a separately trained value function. It estimates advantages by computing group-relative statistics: for each question, the model generates a group of $G$ responses, computes the mean and standard deviation of their rewards, and normalizes each response's reward as $A = (r - \text{mean}) / \text{std}$. This is efficient and elegant for single-reward settings because it provides a natural baseline—responses better than the group average get positive advantages, worse ones get negative advantages. However, as the paper demonstrates, this normalization operates on the summed scalar when multiple rewards are present, destroying the multi-dimensional structure of the signal. The paper is the first to systematically characterize this collapse and its consequences.

GRPO without standard deviation normalization (Dr.GRPO, DeepSeek-v3.2). Some recent variants (Zichen Liu et al., 2025; DeepSeek-AI, 2025) remove the standard deviation denominator from Equation 2, computing $A = r - \text{mean}$. The paper acknowledges this was introduced to mitigate question-level difficulty bias, but also examines whether it addresses the signal collapse problem. The answer is a partial no: Figure 3 shows it increases the number of distinct advantage groups compared to standard GRPO, but only modestly (e.g., from ~10 to ~30 with 16 rollouts, while GDPO achieves ~175). More critically, the empirical results in Section 4.1.1 show that GRPO w/o std fails entirely to learn the format reward on tool-calling (0% format compliance in Table 2), despite converging to a similar correctness reward as GDPO. This reveals that simply increasing advantage diversity is insufficient—the underlying normalization strategy matters qualitatively, not just quantitatively. The paper hypothesizes that removing standard deviation normalization introduces training instability, consistent with the format reward training curve in Figure 1b showing zero progress.

Value-function-based methods (PPO, etc.). Proximal Policy Optimization (Schulman et al., 2017) uses a learned value function to estimate advantages rather than group-relative statistics. In principle, this avoids the information collapse issue because each reward component could have its own value head, and advantages could be computed in a decoupled manner. However, training separate value functions for each reward introduces computational overhead and complexity, particularly when the number of rewards grows. More importantly, the community's shift to GRPO was motivated precisely by the desire to eliminate value function training, which is notoriously unstable in language model settings. GDPO can be seen as taking the practical benefits of GRPO (no value function, simple implementation) while fixing its multi-reward pathology.

Reward-function engineering approaches. A parallel line of work has tried to address multi-reward optimization through reward design rather than optimization algorithms. The paper discusses two strategies:

1. Weight adjustment: Assigning different coefficients $w_k$ to each reward so that $r_{\text{sum}} = \sum_k w_k r_k$. The paper's experiments (Section 4.2.1, Figure 6) demonstrate this is unreliable when rewards differ in difficulty. The model optimizes the easiest reward regardless of weights until the weight differential becomes extreme (e.g., reducing length weight to 0.25 before the length constraint meaningfully relaxes). This is a fundamental issue: the optimization process is responding to the gradient signal, which is dominated by the easiest-to-improve objective, not the highest-weighted one.

2. Conditional rewards: Making easier rewards contingent on harder ones, e.g., only awarding the length bonus if the answer is correct: $\tilde{\mathcal{R}}_{\text{length}} = \mathbb{1}[\text{correct}] \times \mathcal{R}_{\text{length}}$. This addresses the difficulty disparity by forcing the model to master the harder objective before receiving any signal from the easier one. The paper shows this works well (Table 4), particularly when combined with GDPO, but notes it changes the semantics of the reward—the length reward no longer independently encourages conciseness, it only rewards concise correct answers. This may not always be the desired behavior.

The key insight is that these reward-design approaches are orthogonal to the optimization algorithm. You can use conditional rewards with GRPO (which the paper shows still underperforms conditional rewards with GDPO) or with GDPO (which yields the best results). The optimization algorithm's ability to faithfully process the reward structure is a separate axis of improvement that prior work had not examined.

How This Paper Positions Itself

The paper positions GDPO as a minimal, principled modification to GRPO's advantage computation that preserves the simplicity and efficiency of group-relative methods while fixing their fundamental failure mode in multi-reward settings. The verb "introduce" in the title accurately reflects the contribution: GDPO is not a new algorithm from scratch but rather a surgical change to one step of GRPO's pipeline—the normalization step.

The key intellectual move is to recognize that normalization should happen on the reward vectors, not on their sum. In Mathematical Expression (Equation 4), GDPO computes:

$A^{(i,j)}_k = \frac{r^{(i,j)}_k - \text{mean}\{r^{(i,1)}_k, ..., r^{(i,G)}_k\}}{\text{std}\{r^{(i,1)}_k, ..., r^{(i,G)}_k\}}$

for each reward $k$ independently, then sums the resulting per-reward advantages, then applies a batch-wide normalization (Equation 6) to maintain numerical stability. The batch-wide normalization step is empirically important (Appendix A, Figure 8 shows training failures without it) but theoretically secondary—it doesn't change the number of distinct advantage groups, just their scale.

This decoupling preserves the joint distribution of rewards. In the two-binary-reward, two-rollout example (Figure 2), GDPO assigns distinct advantage vectors to (0,1) vs. (0,2)—specifically, $(-0.7071, 0.7071)$ vs. $(-1.4142, 1.4142)$—because the per-reward normalizations preserve the fact that in the (0,2) case, rollout 2 is further above its group mean on each individual reward dimension compared to the (0,1) case.

The paper's positioning relative to existing work is careful and precise:

• Not a replacement for reward design: GDPO complements reward-function engineering (conditional rewards, weight tuning) rather than replacing it. The paper explicitly discusses how to combine GDPO with these techniques in Section 3.2.
• Not a value-function method: GDPO maintains GRPO's computational simplicity—no critic network, no GAE computation, no value loss. It only modifies the advantage calculation.
• Not specific to any particular reward structure: The paper demonstrates GDPO with 2 rewards (tool-calling, math reasoning) and 3 rewards (code reasoning), with binary, continuous, and sparse rewards, establishing generality.
• Compatible with existing GRPO infrastructure: GDPO is implemented in verl, HF-TRL, and Nemo-RL, the same frameworks used for GRPO, making adoption trivial.

Why This Work Matters Now

The timing of this paper is significant. The RL-for-LLMs landscape in early 2026 has several converging trends that make multi-reward optimization critical:

1. Proliferation of reward types: Models are expected to satisfy format requirements (tool-calling APIs expect specific XML/JSON structures), efficiency constraints (users and providers care about token costs), safety constraints (refusing harmful requests, avoiding toxic outputs), and correctness requirements (getting the right answer). Each of these dimensions requires a reward signal.

2. GRPO's dominance: GRPO has become the de facto standard, implemented in verl, TRL, and other widely-used training frameworks. A fix that integrates into this ecosystem without requiring infrastructure changes has immediate practical impact.

3. The sophistication ceiling: As the field pushes toward more complex agentic behaviors (multi-step tool use, long-horizon planning, self-correction), the number of simultaneous constraints grows. A method whose optimization fidelity degrades with more rewards has a progressively worsening failure mode.

4. Observability gap: The information collapse in GRPO is subtle—training curves may look reasonable (loss decreases, rewards increase) even as the optimization is inefficient because the model might be learning from a degraded signal. The paper's contribution of making this collapse visible (Figures 2, 3) and measurable (distinct advantage group count) is itself valuable for the community's understanding of what's happening during training.

The paper doesn't claim GDPO is a revolutionary departure from GRPO. Instead, it argues that a small, well-motivated change to the advantage computation—decoupling per-reward normalization—reliably fixes a pervasive and previously undiagnosed failure mode, and that the empirical benefits are consistent across diverse tasks, model sizes, and reward configurations. This positioning as a "better GRPO" rather than a fundamentally new paradigm is both honest and practically useful.

3. Technical Approach

3.1 Reader Orientation

GDPO is a policy optimization method for training language models to satisfy multiple objectives simultaneously using reinforcement learning—it takes GRPO, the dominant RL algorithm for LLMs, and fixes a specific information-destroying step in its advantage computation. The core problem is that when GRPO sums rewards from different objectives and then normalizes the sum, distinct combinations of individual reward scores collapse into identical advantage values, depriving the model of the fine-grained signal it needs to learn efficiently across multiple reward dimensions. GDPO solves this by normalizing each reward independently before aggregation, preserving the multi-dimensional structure of the training signal, and then applying a batch-level renormalization to maintain numerical stability regardless of how many rewards are being optimized.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components organized into a pipeline:

1. The base language model (e.g., Qwen2.5-Instruct, DeepSeek-R1, Qwen3-4B-Instruct) serves as the policy $\pi_\theta$—the model whose parameters are being updated. It generates responses to prompts drawn from a training dataset.

2. The rollout generation process samples a group of $G$ responses per prompt from the current policy. These $G$ responses form the "group" in "group-relative" advantage estimation—the fundamental unit across which relative comparisons are made.

3. The multi-reward scoring functions evaluate each generated response along $n$ independent objective dimensions, producing a vector of $n$ reward scores per response. These can be binary (0/1 for format compliance, length adherence), continuous (test case pass rates), or bounded real values (correctness scores in [-3, 3]).

4. The GDPO advantage computation is where the core contribution lives. It applies group-wise normalization to each reward dimension independently (Equation 4), sums the per-reward advantages (Equation 5), and then applies batch-wide normalization (Equation 6) to produce the final advantage $\hat{A}_{\text{sum}}^{(i,j)}$ for each rollout. This three-step process preserves the joint distribution of rewards while maintaining numerical stability.

5. The policy update uses the standard GRPO clipped objective (Equation 3) with these GDPO-computed advantages, updating model parameters $\theta$ via gradient descent. The KL-divergence penalty against a reference model is applied as in standard GRPO, though omitted from the main equations for clarity.

Information flows as follows: a training batch of prompts enters → the current policy generates $G$ rollouts per prompt → each rollout is scored by $n$ reward functions producing reward vectors → GDPO normalizes each reward dimension separately per group → per-reward advantages are summed → batch-wide normalization produces final scalar advantages → the PPO-style clipped surrogate objective updates the policy using these advantages → the next batch begins with the updated policy.

3.3 Roadmap for the Deep Dive

• First, the GRPO objective and its multi-reward extension (Equations 1–3) because GDPO is a modification of this objective, and understanding the baseline is prerequisite to understanding what GDPO changes.
• Second, the information collapse analysis using the two-binary-reward, two-rollout example (Figure 2) because this is the diagnostic that motivates GDPO's design—it shows exactly what goes wrong with GRPO's summed-reward normalization.
• Third, GDPO's decoupled normalization procedure (Equations 4–6) because this is the method itself, and understanding why each step exists (group-wise per-reward, summation, batch-wise renormalization) requires knowing what failure it addresses.
• Fourth, the combinatoric analysis of distinct advantage group counts (Figure 3) because this provides quantitative justification that GDPO preserves more information than GRPO or GRPO without standard deviation, scaling favorably with both rollout count and reward count.
• Fifth, reward priority mechanisms (weighting and conditional rewards, Equations 7–8) because these are practical tools for practitioners that interact with GDPO's advantage computation in specific ways—the paper shows they work more reliably under GDPO than under GRPO.

3.4 Detailed, Sentence-Based Technical Breakdown

This is fundamentally an algorithm modification paper whose core idea is that normalizing each reward dimension independently before aggregation preserves information that GRPO's summed-reward normalization destroys, and that this preserved information leads to more accurate policy updates across all multi-reward RL configurations.

---

The GRPO Multi-Reward Objective (Baseline)

GDPO modifies the advantage computation in GRPO's policy update, so understanding GRPO's multi-reward formulation is necessary before examining the modification. Given a question-answer pair $(q_i, o_j)$ where the behavior policy $\pi_{\theta_{\text{old}}}$ samples a group of $G$ responses $\{o_j\}_{j=1}^G$, and assuming $n$ objectives (each with its own reward function), the standard practice for multi-reward GRPO proceeds in three steps:

Step 1: Sum rewards.

$r_{\text{sum}}^{(i,j)} = r_1^{(i,j)} + \cdots + r_n^{(i,j)}$

where $r_k^{(i,j)}$ is the reward from the $k$-th objective for the $j$-th rollout of the $i$-th question, and $r_{\text{sum}}^{(i,j)}$ is the scalar total reward for that rollout.

What it computes: a single scalar per rollout by adding all individual reward scores together. If there are two binary rewards $r_1, r_2 \in \{0, 1\}$, the possible values for $r_{\text{sum}}$ are $\{0, 1, 2\}$. If there is a correctness reward in $[-3, 3]$ and a format reward in $\{0, 1\}$, the sum range is $[-3, 4]$.

Why this form: it is the simplest possible aggregation that reduces a multi-dimensional reward vector to a scalar that the GRPO advantage formula can process. The implicit assumption is that all rewards are expressed in compatible units and that their sum meaningfully represents overall response quality. The paper challenges exactly this assumption when it shows that the subsequent normalization step destroys information contained in the individual reward components.

Step 2: Compute group-relative advantages.

$A_{\text{sum}}^{(i,j)} = \frac{r_{\text{sum}}^{(i,j)} - \text{mean}\left\{r_{\text{sum}}^{(i,1)}, \ldots, r_{\text{sum}}^{(i,G)}\right\}}{\text{std}\left\{r_{\text{sum}}^{(i,1)}, \ldots, r_{\text{sum}}^{(i,G)}\right\}}$

where $\text{mean}\{\cdot\}$ is the arithmetic mean of the $G$ summed rewards within the group for question $i$, and $\text{std}\{\cdot\}$ is their standard deviation.

What it computes: for each rollout, how many standard deviations its total reward is above or below the group average. A response that scored better than the average gets a positive advantage (the policy should become more likely to produce similar responses); a response that scored worse gets a negative advantage (the policy should become less likely). The magnitude tells us how exceptional the response was relative to its peers.

Why this form: the group-relative normalization provides a natural baseline without training a value function. Because the rollouts are all responses to the same question, comparing them within the group factors out question difficulty—a response that scores perfectly on an easy question won't get an advantage if all other rollouts also scored perfectly, while a response that scores 0.3 correct on a hard question might get a positive advantage if the group mean is 0.1. However, this normalization operates on the summed rewards, meaning it sees only the scalar total, not the individual reward components that produced it. This is the step that GDPO changes.

Step 3: Optimize the GRPO objective.

$\mathcal{J}_{\text{GRPO}}(\theta) = \mathbb{E}_{(q_i, o_j) \sim D, \{o_j\}_{j=1}^G \sim \pi_{\theta_{\text{old}}}(\cdot|q)} \left[\frac{1}{G}\sum_{j=1}^G \frac{1}{|o_j|}\sum_{t=1}^{|o_j|} \min\left(s_{i,t}(\theta) A_{\text{sum}}^{(i,j)}, \text{clip}(s_{i,t}(\theta), 1-\epsilon, 1+\epsilon) A_{\text{sum}}^{(i,j)}\right)\right]$

where $s_{i,t}(\theta) = \frac{\pi_\theta(o_j^t | q, o_j^{<t})}{\pi_{\theta_{\text{old}}}(o_j^t | q, o_j^{<t})}$ is the per-token importance ratio comparing the new policy's probability of generating token $t$ to the old policy's probability, $\epsilon$ is the clipping threshold (preventing updates that would change the policy too drastically), $|o_j|$ is the number of tokens in response $o_j$, and $\text{clip}(x, a, b)$ clamps $x$ to the interval $[a, b]$.

What it computes: for each token in each rollout, we multiply its importance ratio $s_{i,t}$ by the rollout's advantage $A_{\text{sum}}^{(i,j)}$ and clip this product to stay within $[1-\epsilon, 1+\epsilon]$ times the advantage for stability. The $\min$ operator chooses the more conservative of the clipped and unclipped values—this is the standard PPO clipping mechanism that prevents the new policy from deviating too far from the old policy. The result is a scalar expectation that gradient descent maximizes with respect to $\theta$.

Why this form: the per-token formulation (averaging over all tokens in the response) means that every token contributes to the policy update proportionally to the rollout-level advantage, rather than only the final token or only the reward-bearing tokens. This is the standard GRPO/PPO formulation and is unchanged by GDPO—GDPO only modifies how $A_{\text{sum}}^{(i,j)}$ is computed before it enters this objective.

What GDPO changes and what it leaves alone. GDPO replaces Step 2 (Equation 2) with a three-stage computation (Equations 4–6) but uses the same Equation 3 for the policy update. All other components—rollout generation, reward scoring, KL-penalty against a reference model, optimizer, and hyperparameters—remain identical to standard GRPO. This is a surgical change: one function in the training loop is replaced, and everything else stays the same.

---

The Information Collapse Mechanism: Why Summed-Reward Normalization Fails

Before presenting GDPO's solution, the paper provides a minimal worked example that makes the information loss in GRPO's advantage computation visually obvious. This is not a minor diagnostic—it is the causal explanation for why GRPO underperforms in multi-reward settings.

The example setup. Consider a single question for which we generate $G = 2$ rollouts. The task has $n = 2$ binary rewards: $\mathcal{R}_1 \in \{0, 1\}$ (e.g., format correctness) and $\mathcal{R}_2 \in \{0, 1\}$ (e.g., answer correctness). The summed reward for a rollout is $r_{\text{sum}} = r_1 + r_2 \in \{0, 1, 2\}$. Within a group of two rollouts, there are $3^2 = 9$ possible ordered pairs $(r_{\text{sum}}^{(1)}, r_{\text{sum}}^{(2)})$. Ignoring order (since the group is symmetric), there are 6 distinct combinations:

• $(0, 0)$: both rollouts fail both rewards
• $(0, 1)$: one fails everything, one succeeds on exactly one reward
• $(0, 2)$: one fails everything, one succeeds on both rewards
• $(1, 1)$: both succeed on one reward each (possibly different ones)
• $(1, 2)$: one succeeds on one reward, one succeeds on both
• $(2, 2)$: both succeed on both rewards

What GRPO's normalization does to these combinations. The paper enumerates the normalized advantages for each case using Equation 2:

For $(0, 1)$: the group mean is $(0+1)/2 = 0.5$, standard deviation is $\sqrt{((0-0.5)^2 + (1-0.5)^2)/1} \approx 0.7071$, giving advantages $((0-0.5)/0.7071, (1-0.5)/0.7071) = (-0.7071, 0.7071)$.

For $(0, 2)$: the group mean is $(0+2)/2 = 1.0$, standard deviation is $\sqrt{((0-1)^2 + (2-1)^2)/1} \approx 1.4142$, giving advantages $((0-1)/1.4142, (2-1)/1.4142) = (-0.7071, 0.7071)$.

For $(1, 2)$: the group mean is $(1+2)/2 = 1.5$, standard deviation is $\sqrt{((1-1.5)^2 + (2-1.5)^2)/1} \approx 0.7071$, giving advantages $((1-1.5)/0.7071, (2-1.5)/0.7071) = (-0.7071, 0.7071)$.

The collapse. Despite representing qualitatively different situations—in $(0, 2)$ the better rollout satisfies both rewards while the worse satisfies none, while in $(0, 1)$ the better rollout satisfies only one—all three cases produce identical advantage pairs $(-0.7071, 0.7071)$. Similarly, $(0, 0)$, $(1, 1)$, and $(2, 2)$ all produce $(0, 0)$ because the standard deviation of identical values is zero, so the normalized advantage is zero regardless of the actual reward level.

The result is that six distinct group configurations map to only two distinct advantage pairs. The training signal cannot distinguish between a group where one response is perfect and the other is terrible versus a group where one response is mediocre and the other is slightly better—both get the same update. This is the mechanism by which information is destroyed.

The root cause. The collapse occurs because the standard deviation term in the denominator scales with the spread of the summed rewards. When both rollouts have rewards that differ by 1 (cases (0,1), (1,2)), the standard deviation is approximately 0.7071, and the normalized advantage magnitudes are approximately 0.7071. When the rewards differ by 2 (case (0,2)), the standard deviation doubles to approximately 1.4142, and the normalized advantage magnitudes remain 0.7071. The larger absolute difference is exactly cancelled by the larger standard deviation. This is a direct consequence of normalizing the sum: the normalization erases the magnitude of the difference between rewards, preserving only the relative ordering.

Why this matters for optimization. Consider what the policy update should do in case $(0, 2)$ versus $(0, 1)$. In $(0, 2)$, the better rollout achieved both objectives, while the worse achieved neither. The policy update should strongly favor generating responses like rollout 2 and strongly avoid responses like rollout 1. In $(0, 1)$, the better rollout achieved only one objective, and the worse achieved neither. The policy update should still favor rollout 2, but less strongly—the gap between the responses is smaller. Under GRPO, both cases produce identical updates, meaning the policy cannot learn to distinguish between responses that are comprehensively good and responses that are only partially good when contrasted against poor responses.

An even more subtle failure occurs with $(1, 1)$: both rollouts score exactly 1 on summed reward, so GRPO assigns advantage $(0, 0)$—no update at all. But the underlying reward vectors might be $(r_1=1, r_2=0)$ for rollout 1 and $(r_1=0, r_2=1)$ for rollout 2—they excel on different objectives. A desirable update would push the policy toward the specific behaviors that worked (rollout 1's way of getting format right, rollout 2's way of getting correctness right), but GRPO's zero advantage prevents any learning from this pair.

---

GDPO: Decoupled Group-Wise Normalization (the Core Algorithm)

GDPO replaces GRPO's single normalization of summed rewards with a three-stage computation that preserves the multi-dimensional structure of the reward signal. The stages are performed in sequence for each training batch.

Stage 1: Per-reward, group-wise normalization (Equation 4).

For each reward dimension $k \in \{1, \ldots, n\}$ and each rollout $j \in \{1, \ldots, G\}$ within the group for question $i$, compute:

$A_k^{(i,j)} = \frac{r_k^{(i,j)} - \text{mean}\left\{r_k^{(i,1)}, \ldots, r_k^{(i,G)}\right\}}{\text{std}\left\{r_k^{(i,1)}, \ldots, r_k^{(i,G)}\right\}}$

where $r_k^{(i,j)}$ is the raw reward for objective $k$ on rollout $j$ of question $i$, and the mean and standard deviation are computed over the $G$ rollouts within that group for that specific reward dimension $k$ only.

What it computes: for each reward dimension independently, a group-relative advantage that measures how many standard deviations above or below the group mean this rollout scored on that particular objective. The output is an $n$-dimensional advantage vector $A^{(i,j)} = (A_1^{(i,j)}, \ldots, A_n^{(i,j)})$ for each rollout, where each component is normalized using only the statistics of that reward dimension within the group.

Why this form: by normalizing each reward independently before summing, the normalization process has access to the per-dimension statistics. In the two-binary-reward, two-rollout example, this matters decisively. For the $(0, 2)$ case where the underlying rewards are $(r_1=0, r_2=0)$ and $(r_1=1, r_2=1)$, GDPO normalizes $r_1$ across the group (values $\{0, 1\}$ with mean 0.5, std 0.7071) giving $A_1 = (-0.7071, 0.7071)$, and normalizes $r_2$ across the group (also $\{0, 1\}$) giving $A_2 = (-0.7071, 0.7071)$. The total advantage vector is then $(-1.4142, 1.4142)$ after summation. For the $(0, 1)$ case where one rollout is $(r_1=0, r_2=0)$ and the other might be $(r_1=1, r_2=0)$ or $(r_1=0, r_2=1)$, at least one reward dimension has both rollouts scoring 0, producing a zero standard deviation. The paper's Figure 2 shows that GDPO assigns $(-0.7071, 0.7071)$ to $(0, 1)$ versus $(-1.4142, 1.4142)$ to $(0, 2)$—the advantage for $(0, 2)$ is twice as large, correctly reflecting that a rollout satisfying both rewards is more exceptional relative to a completely failed rollout than a rollout satisfying only one.

Handling zero standard deviation. When all $G$ rollouts receive the same reward on a dimension, the standard deviation is zero, and the normalization is undefined. The paper does not explicitly discuss this edge case in the main text, but it is implied by the empirical results (where such cases occur, e.g., when all rollouts satisfy the format constraint) that the implementation handles it, likely by assigning zero advantage when the standard deviation is zero (since no rollout is better or worse than the group average). This is consistent with the $(0, 0)$ and $(2, 2)$ cases in Figure 2 producing $(0, 0)$ advantages under both GRPO and GDPO.

Stage 2: Sum per-reward advantages across objectives (Equation 5).

$A_{\text{sum}}^{(i,j)} = A_1^{(i,j)} + \cdots + A_n^{(i,j)}$

What it computes: a scalar advantage for each rollout by summing its normalized per-reward advantages. This collapses the advantage vector back to a scalar, but unlike GRPO's summation of raw rewards, this summation occurs after normalization, so the per-reward structure has already influenced the individual components before they are combined.

Why this form: the summation across objectives is necessary because the downstream policy update (Equation 3) expects a scalar advantage per rollout. The key difference from GRPO is that each $A_k^{(i,j)}$ term has been normalized using only the distribution of reward $k$ within the group, preserving information about how unusual this rollout's performance on reward $k$ is relative to its peers. The magnitudes of different $A_k$ components are directly comparable because they are all expressed in units of standard deviations from their respective group means.

Stage 3: Batch-wise advantage normalization (Equation 6).

$\hat{A}_{\text{sum}}^{(i,j)} = \frac{A_{\text{sum}}^{(i,j)} - \text{mean}\left\{A_{\text{sum}}^{(i',j')} \mid i' \in D_{\text{Batch}}, j' = 1, \ldots, G\right\}}{\text{std}\left\{A_{\text{sum}}^{(i',j')} \mid i' \in D_{\text{Batch}}, j' = 1, \ldots, G\right\} + \epsilon}$

where $D_{\text{Batch}}$ is the set of all questions in the current training batch, $A_{\text{sum}}^{(i',j')}$ ranges over all $G$ rollouts for every question in the batch, and $\epsilon$ is a small constant added to prevent division by zero.

What it computes: a renormalization of the summed per-reward advantages across the entire batch, centering them to have mean approximately zero and standard deviation approximately one within the batch. The advantage $\hat{A}_{\text{sum}}^{(i,j)}$ for a particular rollout is how many batch-level standard deviations its summed per-reward advantage is above or below the batch mean.

Why this form: this step serves two critical functions. First, it ensures numerical stability: as the number of reward objectives $n$ increases, the magnitudes of $A_{\text{sum}}$ would naturally grow (since it sums $n$ terms each with standard deviation approximately 1), which could lead to exploding gradients. The batch normalization keeps the advantage scale constant regardless of $n$. Second, it provides an additional layer of baseline normalization across questions: even after per-reward group normalization, some groups might have uniformly high advantages (e.g., when all rollouts in a group perform well across all objectives) while others have uniformly low advantages. The batch normalization re-centers the entire batch so that advantages are comparable across questions, preventing questions where all rollouts happen to score well from dominating the gradient.

Empirical evidence for Stage 3. Appendix A (Figure 8) demonstrates that removing batch-wise normalization causes occasional training failures. The paper shows that without this step, some training runs fail to converge on the format reward in the tool-calling task, while runs that succeed still exhibit the correct behavior. This indicates that the batch normalization acts as a stabilizer, reducing variance across training runs and preventing edge cases where per-reward-normalized advantages accumulate to extreme values.

The GDPO policy update. The final advantage $\hat{A}_{\text{sum}}^{(i,j)}$ is plugged into the standard GRPO clipped objective (Equation 3), replacing $A_{\text{sum}}^{(i,j)}$. The optimization proceeds identically to GRPO from this point: the importance ratio $s_{i,t}$ is computed per token, the clipped surrogate objective is evaluated, and gradient descent updates $\theta$. The only difference is that the advantage feeding this update has been computed via Equations 4–6 instead of Equation 2.

Design rationale: why not just use per-reward advantages in separate policy updates? An alternative approach would be to compute separate policy updates for each reward dimension, each using its own advantage $A_k^{(i,j)}$, and sum the policy gradients. GDPO instead sums the advantages first and then does a single policy update. This choice preserves the standard GRPO optimization loop (single advantage, single update) while still benefiting from per-reward normalization. Doing separate updates would require multiple forward/backward passes or careful gradient accumulation, increasing computational cost, and would raise questions about how to weight the separate updates when objectives conflict. GDPO's approach of summing normalized advantages is simpler and, as the empirical results show, effective.

The $\epsilon$ in the batch normalization denominator. The paper includes this small constant for numerical stability but does not specify its value. In standard implementations, it is typically $10^{-8}$ or similar—large enough to prevent division by zero when all rollouts in a batch receive identical summed advantages, small enough not to meaningfully affect the normalization otherwise.

---

Combinatoric Analysis: Why Per-Reward Decoupling Scales Better

The information-preservation argument in Figure 2 is qualitative: GDPO preserves more distinct advantage states than GRPO. But does this advantage grow with problem scale, or is it only relevant for the minimal two-rollout, two-reward case? The paper provides a combinatoric analysis in Figure 3 that quantifies the information capacity of each method as a function of the number of rollouts $G$ and the number of rewards $n$.

The distinct advantage group count metric. For a given method (GRPO, GRPO without standard deviation, or GDPO), the paper enumerates all possible reward configurations across $G$ rollouts and $n$ binary rewards, computes the resulting advantage vectors for the $G$ rollouts, and counts how many distinct ordered $G$-tuples of advantages arise. A higher count means the advantage computation can express more fine-grained distinctions between different group reward configurations, providing a richer training signal.

As rollouts increase (left panel): With $n=2$ binary rewards and $G$ varying from 2 to 16:

• GRPO produces the fewest distinct groups, starting at approximately 2 for $G=2$ and growing slowly to approximately 10 at $G=16$.
• GRPO without standard deviation produces more, starting at approximately 3 for $G=2$ and reaching approximately 30 at $G=16$.
• GDPO produces dramatically more, starting at approximately 3 for $G=2$ and growing to over 175 at $G=16$.

The gap widens substantially as $G$ grows—at $G=16$, GDPO produces roughly 18× more distinct advantage groups than GRPO and roughly 6× more than GRPO without standard deviation. This is significant because most GRPO implementations use $G$ values between 4 and 16 (the paper uses $G=4$ for tool-calling and $G=16$ for math/code reasoning), precisely the regime where GDPO's advantage is largest.

As rewards increase (right panel): With $G=4$ fixed rollouts and $n$ varying from 2 to 8:

• GRPO starts at approximately 5 groups for $n=2$, grows to approximately 70 at $n=8$.
• GRPO without standard deviation starts at approximately 8, reaching approximately 120 at $n=8$.
• GDPO starts at approximately 15 for $n=2$ and grows explosively to over 650 at $n=8$.

Again, the gap widens with $n$. At $n=8$ rewards, GDPO produces approximately 9× more groups than GRPO and 5× more than GRPO without standard deviation. This matters because real-world multi-reward settings are trending toward more objectives (correctness, format, length, safety, coherence, bias, etc.), and a method whose information capacity degrades relative to the theoretically available information is increasingly suboptimal.

Why GDPO scales better (combinatoric intuition). GRPO and its variants reduce the multi-dimensional reward vector to a scalar sum before normalization, meaning the advantage depends only on the scalar total per rollout. The number of distinct group configurations is bounded by the number of possible patterns of $G$ scalar sums, which grows polynomially with $G$ (specifically, as $O(G^{n})$ for summed rewards taking $n+1$ possible values). GDPO normalizes each reward dimension independently, then sums, meaning the advantage is a function of the full $n \times G$ reward matrix. While the summation after normalization does collapse some information (different per-reward advantage matrices can sum to the same advantage vector), the combinatorial space before summation is much larger, and empirically much more of it survives. Formally, the number of possible per-reward-normalized advantage patterns grows exponentially with $n$ (each reward dimension independently produces $O(G)$ possible advantage values), and the summation preserves combinatorial variety because different reward dimension patterns can sum to different total advantage magnitudes.

Connection to empirical performance. The combinatoric analysis shows GDPO's advantage computation can express more distinctions, but it doesn't prove that those distinctions are useful for learning. The usefulness must be demonstrated empirically, which the paper does across three tasks (Sections 4.1–4.3). However, the combinatoric analysis provides a theoretical upper bound on the information GRPO can transmit: if two distinct reward configurations always produce identical advantages under GRPO, the model cannot possibly learn to distinguish them, regardless of how much data it sees. GDPO increases this upper bound, and the empirical results confirm the additional capacity translates to better optimization.

---

Reward Priority Mechanisms: Weights and Conditional Rewards

GDPO's advantage computation assumes all $n$ objectives initially have equal importance. In practice, users often have priorities—correctness may matter more than conciseness, format compliance may be non-negotiable, safety constraints may be hard requirements. The paper provides guidance on how to encode these priorities in combination with GDPO, covering both reward weighting and reward function modification.

Weighted reward aggregation (Equation 7). The simplest method is to assign different coefficients to each objective's normalized advantage before summation:

$A_{\text{sum}}^{(i,j)} = w_1 A_1^{(i,j)} + \cdots + w_n A_n^{(i,j)}$

where $w_k > 0$ is the weight for objective $k$. In GDPO, these weights are applied to the already-normalized per-reward advantages, ensuring that the weighting operates on comparable scales (each $A_k$ is in units of within-group standard deviations for that reward dimension).

What it computes: a weighted average (not exactly, since we don't divide by the sum of weights) of the per-reward advantages, where a higher weight means that reward dimension's relative standing within the group contributes more to the final advantage. If $w_{\text{correct}} = 2$ and $w_{\text{length}} = 1$, a rollout that is exceptional on correctness will get roughly twice the advantage boost compared to a rollout that is equally exceptional on length.

Why this is better under GDPO than GRPO: under GRPO, weights are applied to raw rewards before summation and normalization, creating a complex interaction. The weight changes the distribution of summed rewards, which changes the group mean and standard deviation, which changes the normalization. The mapping from weight ratio to effective priority is highly nonlinear and hard to predict. Under GDPO, the per-reward normalization happens before weighting, so each $A_k$ has stable, known statistics (mean 0, standard deviation 1 within the group), and the weight ratio directly controls the relative contribution of each objective to the final advantage.

The difficulty disparity problem. The paper identifies a critical practical issue (Section 4.2.1): when objectives differ substantially in difficulty, simply adjusting weights does not produce the intended behavior. In the math reasoning experiments (Figure 6), the length constraint (keeping responses under 4000 tokens) is much easier to satisfy than answer correctness—the model rapidly achieves near-perfect length scores regardless of weighting, while correctness takes many steps to improve. Reducing the length weight $w_{\text{length}}$ from 1.0 to 0.75 or 0.5 has negligible effect on the length-exceeding ratio, meaning the model continues prioritizing the easy objective over the harder one despite the weights indicating that correctness should be more important.

The mechanism is simple: the length reward produces a strong, consistent gradient because the model can easily learn to be concise (it just stops generating). The correctness reward produces a weaker, noisier gradient because getting math right is hard. Even if the correctness weight is higher, the optimization dynamics are dominated by the steep, reliable gradient from the length objective. Only when the length weight is reduced to 0.25—making it small enough that the weighted correctness gradient can compete—does the model meaningfully relax the length constraint.

Conditional rewards (Equation 8). To address the difficulty disparity, the paper adopts the conditional reward design from prior work (Liu et al., 2025; DLER, 2025):

$r_k = \begin{cases} r_k, & \text{if } r_\ell \geq t \\ 0, & \text{otherwise} \end{cases}$

where $r_\ell$ is a more important or more difficult reward, $t$ is a threshold (typically $t=1$ for binary rewards), and $r_k$ is an easier reward that is made contingent on satisfying $r_\ell$. In the math reasoning experiments, the length reward is conditioned on correctness:

$\tilde{\mathcal{R}}_{\text{length}} = \begin{cases} 1, & \text{if response length} \leq \ell \text{ and } \mathcal{R}_{\text{correct}} = 1 \\ 0, & \text{otherwise} \end{cases}$

A response that is concise but wrong receives zero length reward, eliminating the gradient signal that would otherwise push the model toward short-but-wrong answers.

Why conditional rewards work with GDPO. Conditional rewards change the reward structure so that the easy objective provides no signal at all unless the hard objective is satisfied. This forces the model to learn the hard objective first or at least simultaneously, because improving conciseness without improving correctness yields no reward gain. Under GDPO, the per-reward normalization then operates on these modified rewards, preserving the intended semantics: a rollout that is both correct and concise gets a positive advantage on both dimensions; a rollout that is correct but too long gets a positive advantage on correctness but zero on length; a rollout that is concise but wrong gets zero on both (because the length reward is gated by correctness).

The empirical results in Table 4 show that GDPO with conditional rewards ($\tilde{\mathcal{R}}_{\text{length}}$) achieves the best accuracy-efficiency trade-offs: on AIME, it improves accuracy to 57.7% (vs. 53.1% for GDPO with unconditional length reward) while maintaining a low 12.3% length-exceeding ratio (vs. 29.2% for GRPO with conditional rewards). This demonstrates that the combination of GDPO's faithful advantage computation and conditional reward design synergize: conditional rewards encode the priority structure, and GDPO ensures that structure is accurately reflected in the optimization signal.

Weight sensitivity after conditioning. An important finding is that after conditioning resolves the difficulty disparity, varying reward weights becomes more reliable for fine-grained preference tuning. In Figure 6 (right column, with $\tilde{\mathcal{R}}_{\text{length}}$), reducing the length weight from 1.0 to 0.25 produces a monotonic increase in the length-exceeding ratio for both GRPO and GDPO on both MATH and AIME. This is in contrast to the left column (unconditional $\mathcal{R}_{\text{length}}$) where weight changes produce non-monotonic or negligible effects. The mechanism: once the model cannot exploit the easy objective independently, the weight ratio directly controls the relative importance of the two objectives in the summed advantage, and GDPO's per-reward normalization ensures this weighting operates on comparable, well-behaved advantage components.

---

Implementation Details and Hyperparameters

GDPO is implemented in three frameworks (verl, HF-TRL, Nemo-RL) as a drop-in replacement for GRPO's advantage computation. The paper provides complete hyperparameter configurations for reproducibility.

Tool-calling (Section 4.1): training Qwen2.5-Instruct (1.5B and 3B) using verl for 100 steps. Group size $G=4$ rollouts per question. Training batch size 512 questions. Maximum response length 1024 tokens. AdamW optimizer with learning rate $1 \times 10^{-6}$. PPO mini-batch size 128. Maximum prompt length 2048 tokens. Total 15 epochs over the 4k training samples. KL-divergence coefficient 0.001. The reward functions are: format reward $\mathcal{R}_{\text{format}} \in \{0, 1\}$ (structural compliance) and correctness reward $\mathcal{R}_{\text{correct}} \in [-3, 3]$ (tool-name, parameter-name, and parameter-content matching against ground truth). The complete hyperparameter table is in Appendix D (Table 6).

Math reasoning (Section 4.2): training DeepSeek-R1-1.5B, DeepSeek-R1-7B, and Qwen3-4B-Instruct on the DeepScaleR-Preview dataset (40k competition-level math problems) for 500 steps using verl. Group size $G=16$ rollouts. Training batch size 512. Maximum response length 8000 tokens (training) and 32000 tokens (evaluation). Learning rate $1 \times 10^{-6}$. PPO mini-batch size 64. Single PPO epoch per batch. Clipping thresholds $\epsilon_{\text{low}} = 0.2$ and $\epsilon_{\text{high}} = 0.28$ (the higher upper clip is from the DAPO recipe). Dynamic sampling enabled, filtering groups by sequence-level reward. KL-divergence coefficient $5 \times 10^{-4}$ with MSE-type KL loss. Rollout temperature 1.0. The complete configuration is in Appendix E (Table 7). Rewards: correctness $\mathcal{R}_{\text{correct}} \in \{0, 1\}$ (exact match of extracted final answer) and length $\mathcal{R}_{\text{length}} \in \{0, 1\}$ (response within 4000 tokens).

Code reasoning (Section 4.3): training DeepSeek-R1-7B on the Eurus-2-RL dataset (24k coding problems) for 400 steps using the same hyperparameter configuration as math reasoning. Rewards: pass rate $\mathcal{R}_{\text{pass}} \in [0, 1]$ (proportion of test cases passed), conditioned length $\tilde{\mathcal{R}}_{\text{length}} \in \{0, 1\}$ (concise and all tests pass), and bug-free $\mathcal{R}_{\text{bug}} \in \{0, 1\}$ (no runtime or compilation errors).

Design choice: same hyperparameters as GRPO. GDPO intentionally uses identical hyperparameters to the GRPO baselines in each task. This ensures that any observed improvements are attributable to the advantage computation method, not to hyperparameter tuning. The paper is explicitly testing whether GDPO's advantage estimation is better, not whether a separately tuned GDPO configuration can outperform GRPO. This is a strong experimental design choice that makes the comparison conservative—any hyperparameter that works particularly well for GRPO but suboptimally for GDPO would disadvantage GDPO, yet GDPO still consistently outperforms.

Design choice: using established RL recipes. The math and code reasoning experiments follow the DLER recipe (Liu et al., 2025), which includes dynamic sampling, higher clipping thresholds, token-mean loss from DAPO, and overlong reward shaping. These are orthogonal improvements to the GRPO pipeline that GDPO inherits without modification, demonstrating that GDPO is compatible with existing best practices in RL for LLMs.

4. Key Insights and Innovations

Innovation 1: Diagnosing GRPO's Signal Collapse as a First-Class Failure Mode

The paper's most distinctive contribution is not GDPO itself but the diagnostic framing that makes the problem visible in the first place. Before this work, the field treated GRPO as a generic policy optimizer applicable to any scalar reward—the fact that the scalar was constructed by summing heterogeneous rewards was considered an implementation detail, not a potential source of failure. The paper flips this assumption by asking: what information is destroyed when you normalize a sum rather than a vector?

The answer, demonstrated in Figure 2, is stark: in a minimal two-rollout, two-binary-reward setting, GRPO maps six qualitatively distinct group configurations (e.g., one rollout perfect and the other terrible vs. one rollout mediocre and the other slightly better) into only two distinct advantage pairs. This is not a small inefficiency—it means the optimization algorithm cannot distinguish between situations where the model should learn strongly (a perfect response contrasted with a failed one) and situations where it should learn weakly (a partially correct response contrasted with a failed one). The training signal is quantized to a binary: either the rollouts differ (producing $(\pm 0.7071, \mp 0.7071)$) or they don't (producing $(0, 0)$).

What makes this intellectually significant is that it reveals a hidden coupling between reward design and optimization fidelity that the field had been ignoring. Researchers have invested enormous effort in designing sophisticated multi-reward functions (ToolRL's correctness decomposition, DLER's length penalties, adaptive reward shaping), implicitly assuming that the optimizer would faithfully translate these reward signals into appropriate policy updates. The paper shows this assumption is false: GRPO's normalization step acts as a non-linear compressor on the reward structure, and the compression is severe enough that the optimizer may be learning from a fundamentally different signal than the one the reward designer intended.

The combinatoric analysis in Figure 3 transforms this qualitative insight into a quantitative framing. By counting distinct advantage groups as a function of rollout count and reward count, the paper shows that GRPO's information capacity grows slowly (polynomially, roughly $O(G^n)$ for the summed scalar), while GDPO's grows much faster because it preserves per-dimension structure before summation. The gap widens precisely in the regimes where practitioners operate—$G=4$ to $G=16$ rollouts and $n=2$ to $n=8$ rewards. This analysis provides a complexity-theoretic argument for why GRPO degrades with more objectives: it's not just "harder to optimize multiple things," but specifically that GRPO's architecture for processing the reward vector is combinatorially impoverished.

This insight is fundamental rather than incremental. Prior work on GRPO variants (GSPO, DAPO, GFPO, DLER) focused on improving other aspects of the pipeline—clipping, sampling, loss formulation, length penalties—while leaving the advantage normalization step untouched. By identifying normalization as the bottleneck, the paper opens a new axis for algorithmic improvement that had been entirely overlooked.

Innovation 2: Decoupled Normalization as a Principle, Not Just a Trick

The specific algorithmic contribution—normalizing each reward dimension independently before summing (Equations 4–6)—is deceptively simple, but the paper's treatment elevates it from an implementation detail to a design principle: when aggregating heterogeneous signals for policy optimization, normalize the signals before mixing them, not after.

This principle matters because it addresses a structural problem that generalizes beyond GRPO. Any reinforcement learning pipeline that sums rewards from different sources and then normalizes the sum is vulnerable to the exact collapse analyzed in Figure 2—the more objectives added, the more combinatorially compressed the training signal becomes. The paper demonstrates this concretely for GRPO, but the underlying issue applies to any advantage estimation method that reduces multi-dimensional rewards to a scalar before applying group-relative or batch-relative normalization.

What distinguishes GDPO from a simple "fix" is the paper's analysis of why the decoupling works and which parts are necessary. The three-stage computation (per-reward group normalization → summation → batch-wide renormalization) is not arbitrary—each stage serves a specific function that the paper validates empirically:

• Per-reward group normalization (Stage 1) is the core innovation that preserves multi-dimensional structure. The paper shows this alone increases distinct advantage groups by 10–20× over GRPO (Figure 3).

• Summation (Stage 2) is the pragmatic bridge to GRPO's scalar-advantage policy update. The paper implicitly acknowledges that replacing GRPO's entire optimization loop with a multi-objective one would be more principled but less practical, and the summation-plus-renormalization approach achieves most of the benefit with minimal disruption.

• Batch-wide renormalization (Stage 3) is validated as necessary but not sufficient: Appendix A (Figure 8) shows removing it causes occasional training failures, indicating it serves as a stabilizer, not just an optional scaling step. The paper's willingness to include this empirical finding—that the method doesn't work as well without a seemingly minor normalization—is intellectually honest and practically useful.

The paper also draws a sharp distinction between GDPO and GRPO without standard deviation normalization (Dr.GRPO, DeepSeek-v3.2), which the field had already identified as a potential fix. The combinatoric analysis in Figure 3 shows GRPO w/o std modestly increases distinct advantage groups but falls far short of GDPO, and the empirical results in Table 2 demonstrate that on tool-calling, GRPO w/o std completely fails to learn the format reward (0% format compliance). This is a powerful negative result: simply increasing advantage diversity by removing the standard deviation term is not enough—the signal becomes noisy or unstable in ways that prevent convergence on objectives with subtle structure. GDPO's decoupled normalization preserves diversity while maintaining stability, a balance that removing the standard deviation alone cannot achieve.

This is a fundamental advance (a new design principle with broad applicability) rather than an incremental improvement (a tweak to an existing hyperparameter). It changes how practitioners should think about advantage computation in multi-reward settings: they should ask "are my rewards being normalized independently before I mix them?" rather than "what weights should I assign to my rewards?"

Innovation 3: The Difficulty-Disparity Problem in Multi-Reward Optimization

A substantial sub-contribution of the paper is the empirical characterization of what happens when multi-reward optimization involves objectives of very different difficulty levels. Section 4.2.1 demonstrates that simply adjusting reward weights—the natural first approach any practitioner would try—does not produce the intended prioritization when the easier objective's gradient dominates training dynamics.

This is not a theoretical claim but an empirical finding with rich evidence. In Figure 6 (left column), reducing the length reward weight $w_{\text{length}}$ from 1.0 to 0.75 or 0.5 on DeepSeek-R1-7B produces negligible changes in the length-exceeding ratio (<1% change on AIME). The model continues aggressively satisfying the length constraint, often at the expense of correctness, because the length reward provides a steep, reliable gradient signal while correctness provides a weak, noisy one. Only when $w_{\text{length}}$ is dropped to 0.25—making it small enough that the weighted correctness gradient can compete—does the length constraint meaningfully relax. Even then, the relationship is not monotonic: reducing $w_{\text{length}}$ from 0.75 to 0.5 sometimes decreases the length-exceeding ratio (as on MATH for GRPO), indicating the optimizer is responding to the weight change in ways that are opaque and hard to control.

The paper's framing of this as a difficulty disparity, not just a weight-tuning failure, is conceptually important. It explains why practitioners' intuition—"just lower the weight on the objective you care less about"—breaks: the optimizer responds to gradient magnitude, not weight magnitude, and gradient magnitude is driven by how easy the objective is to improve. An objective that can be satisfied by a simple behavioral change (stop generating after 4000 tokens) will dominate an objective that requires complex reasoning (solve the math problem correctly), regardless of how the weights are set, until the weight ratio becomes extreme enough to compensate for the gradient disparity.

The conditional reward design (Equation 8) the paper adopts from prior work is not novel, but the paper's contribution is demonstrating that it becomes far more reliable when combined with GDPO. Under GRPO, conditional rewards help (Table 4 shows GRPO with $\tilde{\mathcal{R}}_{\text{length}}$ achieves 53.3% AIME accuracy vs. 50.2% without conditioning), but the improvements are inconsistent—accuracy on MATH actually drops from 94.1% to 93.2%. Under GDPO, conditional rewards produce consistent improvements across all benchmarks: AIME accuracy jumps from 53.1% to 57.7%, AMC from 84.0% to 85.9%, while keeping length-exceeding ratios manageable (12.3% on AIME vs. 0.2% without conditioning, representing a deliberate relaxation rather than a loss of control).

Moreover, after conditioning resolves the difficulty disparity, the paper shows that weight tuning becomes more predictable. In Figure 6 (right column), reducing the conditioned length weight $\tilde{w}_{\text{length}}$ produces monotonic increases in length-exceeding ratios—the behavior practitioners intuitively expect from weight adjustments but couldn't achieve with unconditional rewards. This is an important practical insight: reward conditioning and weight tuning are complementary, not alternative, approaches, and applying them in sequence (first condition to remove difficulty disparity, then weight to fine-tune) is more effective than either alone.

This contribution is empirically grounded rather than theoretically novel, but it addresses a real pain point for practitioners deploying multi-reward RL systems. The paper provides a diagnostic framework (is one of your rewards much easier than the others?) and a remediation strategy (condition the easy reward on the hard one, then weight-tune) that is immediately actionable.

Innovation 4: Scaling the Number of Rewards Without Sacrificing Fidelity

The code reasoning experiments (Section 4.3) demonstrate that GDPO's advantage degrades less than GRPO's as the number of simultaneous objectives increases from two to three. This is not merely "GDPO works with more rewards"—it is evidence for a scaling property that the combinatoric analysis predicts: GDPO's information capacity grows faster with the number of rewards than GRPO's, so the performance gap should widen as more objectives are added.

The evidence is consistent with this prediction. In the two-reward code setting ($\mathcal{R}_{\text{pass}} + \tilde{\mathcal{R}}_{\text{length}}$), GDPO2-obj outperforms GRPO2-obj on pass rate across all four benchmarks (Table 5), with gains of 1.1–3.3 percentage points. In the three-reward setting ($\mathcal{R}_{\text{pass}} + \tilde{\mathcal{R}}_{\text{length}} + \mathcal{R}_{\text{bug}}$), the pass rates are comparable between GDPO3-obj and GRPO3-obj, but GDPO achieves substantially better scores on the secondary objectives: length-exceeding ratios are 2.7–5.7 percentage points lower, and bug ratios are 1.5–2.0 percentage points lower. This pattern—comparable primary objective performance with better constraint satisfaction—is exactly what one would expect if GDPO's richer advantage signal allows the optimizer to simultaneously improve multiple objectives without sacrificing any single one.

The significance of this finding extends beyond the specific reward counts tested. As language model capabilities advance, the number of simultaneous constraints on their behavior will continue to grow—correctness, efficiency, safety, format compliance, citation accuracy, bias mitigation, tone appropriateness, and many others. A method whose optimization fidelity degrades polynomially with reward count (as GRPO's combinatoric analysis suggests) will become increasingly inadequate, while one whose fidelity degrades more gracefully (as GDPO's does) will remain viable. The paper doesn't test beyond 3 rewards, but the combinatoric trend in Figure 3 and the empirical pattern from 2 to 3 rewards both point toward GDPO being the more scalable approach.

This contribution is best characterized as an empirical scaling result rather than a theoretical proof—the paper demonstrates the pattern for 2 and 3 rewards without extrapolating to larger numbers. But it establishes the critical property (performance gap widens with reward count) that future work can test at scale.

Innovation 5: Resolving the Tension Between Reward Design and Optimization Fidelity

The paper's meta-contribution is reframing multi-reward RL as a joint problem of reward design and optimization fidelity rather than treating these as independent concerns. The dominant paradigm in prior work was: (1) design good reward functions, (2) apply GRPO as a generic optimizer, (3) tune weights if needed. The paper shows that step (2) actively undermines step (1) when multiple rewards are present—GRPO's normalization compresses the carefully designed reward structure into a degraded signal that may not reflect the designer's intent.

This is most clearly illustrated by the comparison between unconditional and conditional length rewards (Tables 3 vs. 4). Under GRPO with unconditional rewards, the model learns to satisfy the length constraint but at the expense of correctness (the training curves in Figure 5 show correctness dropping as length reward rises). The reward designer's intent was presumably to encourage simultaneously correct and concise answers, but the optimizer interprets the summed reward as permission to trade off correctness for conciseness—a perfectly rational response to the signal it receives, but not what the designer wanted. Under GDPO with the same unconditional rewards, the correctness-recovery is better (the correctness curve in Figure 5 continues rising past step 400 where GRPO's declines), suggesting GDPO's richer advantage signal better preserves the multi-dimensional nature of the objective.

Under conditional rewards, the interaction becomes even clearer. The reward designer explicitly encodes "conciseness only matters when the answer is correct" into the reward function. GRPO processes this through its lossy normalization, achieving some improvement (Table 4: AIME 50.2% → 53.3%) but also degrading on MATH (94.1% → 93.2%). GDPO processes the same conditional reward, achieves larger improvements (AIME 53.1% → 57.7%) without regression (MATH 93.9% → 93.9%), and further reduces length violations (AIME exceed: 0.2% → 12.3% vs. GRPO's 2.1% → 29.2%). The reward structure is identical; the optimizer is the only variable. This demonstrates that reward design and optimization fidelity are complementary levers—improving either helps, but improving both together yields the best results.

This reframing has practical implications for how research in this area should proceed. Rather than treating "better reward functions" and "better optimizers" as separate research threads, the paper suggests they should be evaluated jointly: a new reward function tested only with GRPO might appear ineffective not because the reward is poorly designed, but because GRPO can't faithfully transmit it. Conversely, a new optimizer that appears to add little value on simple reward structures might shine on complex, multi-dimensional rewards. The paper's own ablation in Section 4.2.1—testing GDPO under four different weight configurations with two different reward structures—is an exemplar of this joint-evaluation methodology.

5. Experimental Analysis

Evaluation Methodology

Dataset(s). Three distinct task domains are evaluated: (1) Tool calling: training on 4k samples—2k from ToolACE (Weiwen et al., 2024), 1k from Hammar (Lin et al., 2024), and 1k from xLAM (Zhang et al., 2025)—with evaluation on the Berkeley Function Call Leaderboard v3 (BFCL-v3, Patil et al.), which covers single-step reasoning, multi-step tool use, real-time execution, irrelevant tool rejection, simultaneous multi-tool selection, and multi-tool execution. (2) Math reasoning: training on the DeepScaleR-Preview dataset (Luo et al., 2025, 40k competition-level math problems) with evaluation on AIME-24, AMC (2022 and 2023), MATH (Hendrycks et al., 2021, 500 test questions from Lightman et al., 2022 split), Minerva (Lewkowycz et al., 2022), and Olympiad Bench (He et al., 2024). (3) Code reasoning: training on Eurus-2-RL (Cui et al., 2025, 24k coding problems, each with multiple test cases) with evaluation on the PRIME validation set including Apps (Hendrycks et al., 2021), CodeContests (Li et al., 2022), Codeforces (MatrixStudio dataset), and Taco (Li et al., 2023).

Base model(s). Four model families across three scales: Qwen2.5-Instruct (1.5B and 3B, Yang et al., 2025) for tool-calling; DeepSeek-R1 (1.5B and 7B, Guo et al., 2025) and Qwen3-4B-Instruct (Yang et al., 2025) for math reasoning; and DeepSeek-R1-7B for code reasoning. The 1.5B models test whether GDPO works at small scales where training instability is most likely, while the 7B and 4B models test at production-relevant scales. The choice spans two model families (Qwen and DeepSeek) to demonstrate method generality.

Metrics. Tool calling: (1) Average accuracy (%) across BFCL-v3 subtasks (Live Overall, Multi Turn Overall, Non-Live Overall) measuring correct tool selection and parameterization; (2) Correct format ratio (%) measuring the fraction of model outputs satisfying the required XML structure with all fields in correct order. Math reasoning: (1) Pass@1 accuracy (%) measuring whether the extracted final answer matches ground truth, averaged over 16 samples per question with sampling temperature 0.6 and top_p=0.95; (2) Length-exceeding ratio (Exceed, %) measuring the percentage of generated responses exceeding the 4000-token length constraint. Code reasoning: (1) Test case pass rate (% of test cases passed); (2) Length-exceeding ratio (%, responses exceeding the 8000-token training length limit); (3) Bug ratio (%, generated code producing runtime or compilation errors).

Baselines. The paper compares against (1) GRPO (Shao et al., 2024), the standard multi-reward GRPO formulation with summed-reward normalization (Equations 1–3); (2) GRPO without standard deviation normalization (GRPO w/o std), following Dr.GRPO (Zichen Liu et al., 2025) and DeepSeek-v3.2 (DeepSeek-AI, 2025), which removes the denominator from Equation 2, computing $A^{(i,j)}_{\text{sum}} = r^{(i,j)}_{\text{sum}} - \text{mean}\{r^{(i,1)}_{\text{sum}}, \ldots, r^{(i,G)}_{\text{sum}}\}$. For code reasoning, the baselines are distinguished by objective count: GRPO2-obj and GDPO2-obj optimize two rewards ($\mathcal{R}_{\text{pass}} + \tilde{\mathcal{R}}_{\text{length}}$), while GRPO3-obj and GDPO3-obj optimize three rewards ($\mathcal{R}_{\text{pass}} + \tilde{\mathcal{R}}_{\text{length}} + \mathcal{R}_{\text{bug}}$). The original (pre-RL) model performance is reported as a reference point. All models use identical hyperparameters (learning rates, batch sizes, clipping thresholds, KL coefficients) drawn from established recipes—ToolRL's configuration for tool-calling and DLER's recipe for math/code reasoning—so any differences arise from the advantage computation method, not hyperparameter tuning.

Generation budget / compute accounting. The paper measures compute in training steps (100 steps for tool-calling, 500 for math reasoning, 400 for code reasoning) with a fixed group size ($G=4$ rollouts per question for tool-calling, $G=16$ for math and code reasoning) and fixed batch size (512 questions). Total FLOPs are not directly compared—the comparison is between GDPO and GRPO consuming identical compute with identical hyperparameters, so the metric of interest is convergence quality per training step, not total compute efficiency. The evaluation protocol generates 16 rollouts per question with temperature 0.6 and top_p=0.95 for math and code benchmarks.

Cross-validation / statistical protocol. For tool-calling, each method is run five times with different random seeds, and the paper reports average accuracy and format correctness across runs, along with median and interquartile range training curves (Figures 1b, 4). This five-run protocol provides direct evidence of training stability and variance. For math and code reasoning, the paper presents single-run training curves (Figures 5, 7, 9, 10 in appendix) but evaluates on multiple diverse benchmarks (5 for math, 4 for code) to assess robustness across evaluation distributions. No formal statistical significance testing is reported, but the consistency across tasks, model sizes, and reward configurations provides triangulating evidence.

---

Main Quantitative Results

Tool Calling: GDPO Improves Both Accuracy and Format Compliance

Headline numbers (Table 1). For Qwen2.5-Instruct-1.5B, GDPO achieves 32.81% average accuracy and 80.66% correct format ratio, compared to GRPO's 30.18% accuracy and 76.33% format—a 2.63 percentage point accuracy gain and 4.33 percentage point format improvement. The untrained base model achieves only 17.88% accuracy and 4.74% format compliance, meaning GDPO nearly doubles correctness while almost completely solving format adherence. For the 3B model, GDPO achieves 40.87% average accuracy and 82.23% format versus GRPO's 39.20% and 81.64%, with gains of 1.67 and 0.59 percentage points respectively.

Per-subtask breakdown. On the Live subtask (the most challenging category), GDPO-1.5B reaches 55.36% versus GRPO's 50.63%, a 4.73 percentage point improvement. On Multi Turn Overall (requiring sustained tool use across dialogue turns), GDPO achieves 2.50% versus 2.04%—both low, reflecting the difficulty of the task at this model scale, but GDPO maintains an edge. On Non-Live Overall, GDPO attains 40.58% versus 37.87% for GRPO. The 3B model shows similar patterns with Live subtask improving from 69.23% to 71.22% and Multi Turn from 3.14% to 4.59%.

Training dynamics (Figure 4). The median and IQR training curves over five runs reveal that GDPO converges to higher reward values on both objectives. On format reward, GDPO reaches near-perfect scores (~1.0) in most runs but with notably higher variance in the number of steps required to converge compared to GRPO—some GDPO runs achieve full format compliance by step 20, others by step 80, while GRPO converges more uniformly around step 40 but plateaus at a lower median (~0.8). This higher variance is expected from a method that preserves more signal: the richer advantage space allows different optimization trajectories, some faster and some slower, but all reaching higher final values. On correctness reward, GDPO shows faster early-stage improvement and ends at a higher median (approximately 1.5 vs. 1.2 for GRPO at step 100).

Key interpretation. The gains are largest on the 1.5B model, where training budget and model capacity are most constrained. GDPO's richer advantage signal appears most impactful when the optimizer has less margin for error—the 3B model, with more capacity, can partially compensate for GRPO's signal degradation. The consistent improvement on both accuracy and format, without any trade-off between them, suggests GDPO's decoupled normalization indeed preserves the multi-dimensional structure rather than just re-weighting the trade-off.

---

GRPO Without Standard Deviation Fails on Format Reward

Headline numbers (Table 2). GRPO w/o std achieves 29.26% average accuracy and 0% correct format ratio on BFCL-v3, compared to GDPO's 32.81% accuracy and 80.66% format. The format failure is catastrophic—the model never learns the required output structure despite the format reward being part of the training objective. The accuracy (29.26%) is also lower than standard GRPO (30.18%), suggesting that removing standard deviation normalization destabilizes the correctness optimization as well.

Training dynamics (Figure 1b, right panel). The format reward curve for GRPO w/o std (Figure 1b) is flat at zero throughout all 100 training steps—the model makes no progress whatsoever on format compliance. The correctness reward converges to a value similar to GDPO (around 1.5 at step 100), which superficially suggests the method works, but the downstream evaluation reveals this correctness was achieved by outputs that violate the required format and thus would be unusable in a real tool-calling deployment.

Interaction with GRPO w/o std's failure mode. The paper's combinatoric analysis (Figure 3) showed GRPO w/o std increases distinct advantage groups compared to standard GRPO (from ~10 to ~30 at G=16), so one might expect it to improve optimization. The empirical result contradicts this expectation: the additional advantage diversity introduces instability that prevents convergence on objectives with sparse or structured rewards. The format reward is binary and structural (all fields present and in correct order), meaning there are many equally bad ways to fail format but relatively few ways to succeed. The noisier advantage estimates from removing standard deviation normalization may cause the optimizer to oscillate between different failure modes without ever discovering the correct structure, while the correctness reward—being smoother (ranging from -3 to 3 with partial credit)—is more forgiving of noisy updates. This is a crucial negative result: increasing advantage expressiveness without maintaining stability can be worse than the original collapsed signal.

---

Math Reasoning: GDPO Improves Accuracy-Efficiency Trade-offs

Headline numbers (Table 3). For DeepSeek-R1-1.5B trained with GDPO versus GRPO: MATH accuracy 86.2% vs. 83.6% (+2.6 pp), AIME accuracy 29.4% vs. 23.1% (+6.3 pp), AMC accuracy 69.0% vs. 64.5% (+4.5 pp), Minerva accuracy 44.0% vs. 43.5% (+0.5 pp), Olympiad accuracy 46.6% vs. 44.3% (+2.3 pp). Simultaneously, length-exceeding ratios are lower under GDPO: on AIME, 6.5% vs. 10.8% (a 40% relative reduction), on MATH 0.8% vs. 1.5%, on AMC 2.3% vs. 3.2%. The model becomes simultaneously more accurate and more concise—the two objectives are not being traded off, they are being jointly improved.

For DeepSeek-R1-7B, GDPO achieves: MATH 93.9% vs. 94.1% (slightly lower, -0.2 pp), AIME 53.1% vs. 50.2% (+2.9 pp), AMC 84.0% vs. 83.8% (+0.2 pp), Minerva 53.8% vs. 53.2% (+0.6 pp), Olympiad 59.7% vs. 60.2% (-0.5 pp). The accuracy differences are smaller than for the 1.5B model, but the length-exceeding ratios show dramatic improvements: AIME 0.2% vs. 2.1% (10× reduction), MATH 0.1% vs. 0.5%, AMC 0.3% vs. 0.6%, Minerva 0.1% vs. 0.2%. The 7B model under GDPO achieves near-perfect length constraint adherence (0.1–0.4% exceed across all benchmarks) while matching or slightly exceeding GRPO's accuracy—a strict Pareto improvement.

For Qwen3-4B-Instruct, GDPO achieves AIME 56.9% vs. 54.6% (+2.3 pp), with length-exceeding ratio 0.1% vs. 2.5%. On Olympiad, 67.5% vs. 66.8% (+0.7 pp) with exceed 1.0% vs. 1.6%. The pattern holds across model families and scales.

Training dynamics (Figure 5). The training curves on DeepSeek-R1-1.5B reveal the mechanism behind GDPO's advantage. Both methods rapidly achieve full length reward (score 1.0) by step 100, with an accompanying dip in correctness reward as the model initially sacrifices accuracy for conciseness. However, GDPO subsequently recovers and surpasses its pre-dip correctness score, continuing to improve through step 500, while GRPO's correctness begins declining after approximately step 400—a partial training collapse. The maximum response length per batch (Figure 5, right panel) tells a complementary story: despite both methods maintaining near-perfect average length scores, GRPO's worst-case response lengths begin increasing sharply after step 400, indicating the model is losing control over length on outlier cases, while GDPO's maximum lengths continue decreasing throughout training. Figure 9 (DeepSeek-R1-7B) and Figure 10 (Qwen3-4B) in the appendix replicate this pattern: GDPO consistently shows improving correctness and tighter length control at later training stages where GRPO plateaus or degrades.

Effect of difficulty disparity on training (Figure 5, early phase). The initial correctness dip (from approximately 0.55 to 0.48 between steps 0–100) occurs because the length constraint is trivially satisfiable—the model can simply stop generating earlier—and the optimizer rapidly exploits this easy gradient. Under GRPO, the summed-reward normalization collapses the advantage signal during this phase, making it harder for the optimizer to distinguish between long-but-correct responses and short-but-wrong ones, leading to a persistent correctness penalty. Under GDPO, the per-reward normalization preserves the distinction: responses that are correct but long get positive advantages on the correctness dimension and negative on the length dimension, while responses that are wrong but short get the reverse. This richer signal allows the optimizer to eventually find responses that score well on both dimensions, explaining the correctness recovery after step 100.

---

Reward Priority Analysis: Weight Tuning Fails Without Conditioning

Headline finding (Figure 6, Table 8 in Appendix G). Reducing the length reward weight $w_{\text{length}}$ from 1.0 to 0.5 while keeping the correctness weight fixed at 1.0 has minimal impact. For GRPO on AIME: $w_{\text{length}} = 1.0$ yields 50.2% accuracy with 2.1% exceed, while $w_{\text{length}} = 0.5$ yields 52.1% accuracy with 2.3% exceed—the length-exceeding ratio actually increases slightly rather than relaxing as intended. For GDPO: $w_{\text{length}} = 1.0$ yields 53.1% accuracy with 0.2% exceed, while $w_{\text{length}} = 0.5$ yields 53.8% accuracy with 0.8% exceed—a tiny relaxation. The intended effect (lower weight → relaxed constraint → higher accuracy) is essentially absent. Only at $w_{\text{length}} = 0.25$ does the constraint meaningfully relax: GRPO achieves 53.3% AIME accuracy with 4.9% exceed, GDPO achieves 54.7% with 3.9% exceed. This demonstrates a highly non-linear relationship between weight ratio and behavior: a 4× reduction in weight (from 1.0 to 0.25) is required to produce noticeable effects, and even then the effects are modest.

Conditional rewards fix the difficulty disparity (Table 4). Switching to the conditioned length reward $\tilde{\mathcal{R}}_{\text{length}}$ (awarding length bonus only when answer is correct) produces fundamentally different dynamics. The training curves with $\tilde{\mathcal{R}}_{\text{length}}$ (Figure 7) show no initial correctness dip—the model cannot sacrifice correctness for conciseness because conciseness provides no reward without correctness. The length reward rises more gradually and reaches approximately 0.6–0.7 (not 1.0), reflecting that only a subset of correct answers are also concise.

Under GDPO with $\tilde{\mathcal{R}}_{\text{length}}$: AIME accuracy reaches 57.7% (vs. 53.1% with unconditional length reward), AMC reaches 85.9% (vs. 84.0%), Olympiad reaches 60.8% (vs. 59.7%), while length-exceeding ratios rise to 12.3%, 3.8%, and 6.2% respectively—a deliberate relaxation of the length constraint in exchange for accuracy, consistent with the reward design's intent (conciseness is secondary to correctness). Under GRPO with $\tilde{\mathcal{R}}_{\text{length}}$: AIME accuracy is 53.3% (vs. 50.2% unconditional), MATH actually drops to 93.2% (from 94.1% unconditional), and length-exceeding ratios rise substantially more—29.2% on AIME vs. 12.3% for GDPO, 8.6% on AMC vs. 3.8% for GDPO. This means GRPO with conditional rewards achieves less accuracy improvement while sacrificing more length control—a worse trade-off on both objectives simultaneously.

Weight sensitivity after conditioning (Figure 6, right column; Table 9 in Appendix G). With $\tilde{\mathcal{R}}_{\text{length}}$ in place, varying $\tilde{w}_{\text{length}}$ produces the monotonic behavior that weight tuning is supposed to provide. For GDPO on AIME: $\tilde{w}_{\text{length}} = 1.0$ gives 57.7% accuracy with 12.3% exceed, $\tilde{w}_{\text{length}} = 0.75$ gives 56.0% with 31.9% exceed, $\tilde{w}_{\text{length}} = 0.25$ gives 57.7% with 32.8% exceed. The length-exceeding ratio increases monotonically as the weight decreases, and the relationship is now interpretable: lower weight means the optimizer cares less about length, so it allows longer responses in exchange for higher accuracy.

Comparison of GDPO vs. GRPO under conditioning. Across all conditioned reward weight settings, GDPO consistently achieves higher accuracy with lower length-exceeding ratios than GRPO at comparable weights. For example, at $\tilde{w}_{\text{length}} = 1.0$, GDPO achieves 57.7% AIME with 12.3% exceed versus GRPO's 53.3% with 29.2% exceed—4.4 percentage points higher accuracy with less than half the length violations. This is the strongest evidence for GDPO's superiority: when the reward structure is well-designed (conditional rewards encoding correct priorities) and the optimizer faithfully transmits that structure (GDPO's decoupled normalization), the result is a substantively better model.

---

Code Reasoning: GDPO Scales to Three Rewards with Better Constraint Satisfaction

Two-reward results (Table 5, $\mathcal{R}_{\text{pass}} + \tilde{\mathcal{R}}_{\text{length}}$). GDPO2-obj achieves higher pass rates than GRPO2-obj across all four benchmarks: Apps 68.3% vs. 67.2% (+1.1 pp), CodeContests 65.8% vs. 63.2% (+2.6 pp), Codeforces 71.2% vs. 68.1% (+3.1 pp), Taco 48.4% vs. 45.1% (+3.3 pp). Simultaneously, bug ratios are lower: Apps 23.5% vs. 25.0% (-1.5 pp), CodeContests 13.2% vs. 14.1% (-0.9 pp), Codeforces 5.6% vs. 7.0% (-1.4 pp), Taco 36.2% vs. 37.7% (-1.5 pp). Length-exceeding ratios are comparable: Apps 5.0% vs. 5.2%, CodeContests 14.3% vs. 14.2%, Codeforces 18.4% vs. 18.1%, Taco 10.8% vs. 11.8%. The untrained base model performs substantially worse on all metrics, confirming that both methods successfully optimize the objectives.

Three-reward results (Table 5, $\mathcal{R}_{\text{pass}} + \tilde{\mathcal{R}}_{\text{length}} + \mathcal{R}_{\text{bug}}$). Adding the bug reward creates a three-objective optimization problem. GDPO3-obj and GRPO3-obj achieve comparable pass rates: Apps 67.8% vs. 68.1% (-0.3 pp), CodeContests 65.6% vs. 65.6% (tied), Codeforces 69.4% vs. 69.5% (-0.1 pp), Taco 45.1% vs. 44.4% (+0.7 pp). The pass rates are essentially equal—adding the third objective doesn't penalize pass rate under either method. However, GDPO3-obj achieves substantially better scores on the secondary objectives. Length-exceeding ratios: Apps 8.5% vs. 11.2% (-2.7 pp), CodeContests 15.8% vs. 19.3% (-3.5 pp), Codeforces 13.6% vs. 16.9% (-3.3 pp), Taco 10.6% vs. 14.7% (-4.1 pp). Bug ratios: Apps 18.8% vs. 20.3% (-1.5 pp), CodeContests 2.5% vs. 3.9% (-1.4 pp), Codeforces 1.8% vs. 2.5% (-0.7 pp), Taco 28.0% vs. 30.0% (-2.0 pp). GDPO produces code that is equally correct, more concise, and less buggy.

Comparing two-reward and three-reward GDPO. GDPO3-obj versus GDPO2-obj shows the effect of adding a third objective under the better optimizer. Pass rates drop slightly: Apps 67.8% vs. 68.3% (-0.5 pp), CodeContests 65.6% vs. 65.8% (-0.2 pp), Codeforces 69.4% vs. 71.2% (-1.8 pp), Taco 45.1% vs. 48.4% (-3.3 pp). The Taco drop is notable and warrants investigation, but on the other three benchmarks the cost of adding the bug objective is minimal (0.2–1.8 pp). Bug ratios improve substantially: Apps 18.8% vs. 23.5% (-4.7 pp), CodeContests 2.5% vs. 13.2% (-10.7 pp), Codeforces 1.8% vs. 5.6% (-3.8 pp), Taco 28.0% vs. 36.2% (-8.2 pp). Length-exceeding ratios worsen: Apps 8.5% vs. 5.0% (+3.5 pp), CodeContests 15.8% vs. 14.3% (+1.5 pp), Codeforces 13.6% vs. 18.4% (-4.8 pp decrease), Taco 10.6% vs. 10.8% (-0.2 pp). The pattern is mixed—the model trades some conciseness and pass rate for substantially less buggy code, which may be the intended trade-off depending on user priorities.

Comparing two-reward and three-reward GRPO. GRPO3-obj versus GRPO2-obj: pass rates are comparable or slightly higher (Apps 68.1% vs. 67.2%, CodeContests 65.6% vs. 63.2%), bug ratios improve (Apps 20.3% vs. 25.0%, CodeContests 3.9% vs. 14.1%), but length violations increase significantly (Apps 11.2% vs. 5.2%, CodeContests 19.3% vs. 14.2%). Under GRPO, adding the bug objective comes at a clearer cost to length constraint adherence, whereas under GDPO, the trade-off is more moderate.

Key interpretation. The three-reward results demonstrate a scaling property: as the number of objectives grows, GDPO's advantage signal remains richer than GRPO's, allowing the optimizer to improve secondary objectives (length, bugs) without sacrificing the primary objective (pass rate). GRPO's collapsed advantage signal forces a sharper trade-off—improving bug ratio degrades length adherence more severely. This is consistent with the combinatoric analysis in Figure 3: at $n=3$ rewards, GDPO produces substantially more distinct advantage groups than GRPO, and this additional expressiveness translates to better multi-objective optimization.

---

Ablation Studies and Robustness Checks

Batch-wise advantage normalization in GDPO (Appendix A, Figure 8). Removing the batch-wise normalization step (Equation 6) from GDPO causes occasional training failures. Figure 8 shows two representative training runs from the tool-calling task: one run succeeds (format reward converges to ~1.0, correctness reward improves to ~1.5), while another fails completely (format reward stays at 0, correctness reward oscillates). This demonstrates that the batch normalization serves as a stabilizer—without it, the per-reward-normalized advantages can accumulate to extreme values that destabilize training, particularly early in optimization when the policy is changing rapidly. The paper does not report how many of the five runs fail, only that failures occur "occasionally." The $\epsilon$ constant in the denominator provides additional numerical stability for edge cases where all batch advantages are identical.

GRPO without standard deviation normalization (Section 4.1.1, Table 2). As discussed in the main results, this ablation demonstrates that simply removing the standard deviation term from GRPO's normalization (as in Dr.GRPO and DeepSeek-v3.2) does not solve the multi-reward collapse problem and in fact can make it worse. The complete failure on format reward (0% compliance) while correctness converges indicates that the no-standard-deviation variant introduces instability that particularly harms objectives with sparse, structural rewards. This is a negative result that strengthens GDPO's positioning: the solution is not to remove normalization but to restructure it.

Conditional versus unconditional length reward (Tables 3 vs. 4). Under GDPO, switching from unconditional $\mathcal{R}_{\text{length}}$ to conditional $\tilde{\mathcal{R}}_{\text{length}}$ (awarding length bonus only when answer is correct) changes the optimization semantics. With unconditional reward (Table 3), GDPO achieves very low length-exceeding ratios (0.1–0.4% on 7B) while maintaining competitive accuracy. With conditional reward (Table 4), GDPO deliberately relaxes the length constraint (e.g., AIME exceed rises from 0.2% to 12.3%) to gain accuracy (53.1% → 57.7% on AIME). This demonstrates that GDPO responds faithfully to changes in reward semantics—the optimizer is accurately transmitting whatever trade-off the reward designer encodes, rather than imposing its own implicit trade-off as GRPO does.

Model scale robustness. GDPO improves performance across 1.5B, 3B, 4B, and 7B parameter models, spanning two model families (Qwen, DeepSeek). The gains are generally larger for smaller models (1.5B: +6.3 pp AIME, +2.6 pp MATH; 7B: +2.9 pp AIME, -0.2 pp MATH), which is expected—smaller models have less capacity to compensate for degraded training signals, so improving the signal quality matters more. GDPO never substantially underperforms GRPO at any scale, with the largest negative being MATH on 7B (-0.2 pp, within noise).

Task diversity. GDPO is evaluated on three qualitatively different task types: structured output generation (tool calling with specific XML formats), open-ended mathematical reasoning (with implicit length-accuracy trade-offs), and code generation (with test-case-based evaluation). The consistent improvement across all three suggests the method is not exploiting task-specific properties of the reward structure.

Number of rewards. GDPO is tested with 2 rewards (tool calling, math reasoning) and 3 rewards (code reasoning). The performance gap between GDPO and GRPO appears to widen with more rewards—the three-reward code setting shows the clearest advantage on secondary objectives (length and bug ratios) while maintaining comparable primary objective performance. This is consistent with the combinatoric prediction (Figure 3) that GDPO's advantage grows with reward count, but the paper does not test beyond 3 rewards.

Reward weight variation (Appendix G, Tables 8–9). The comprehensive sweep over length reward weights $w_{\text{length}} \in \{0.25, 0.5, 0.75, 1.0\}$ with both unconditional and conditional rewards, evaluated across all five math benchmarks, shows that GDPO maintains better accuracy-efficiency trade-offs at every weight setting. The full tables in Appendix G provide per-benchmark, per-weight results that confirm the patterns visible in the aggregated Figure 6.

---

Critical Assessment

Do the experiments support the central claim—that GRPO's summed-reward normalization causes signal collapse and GDPO fixes it?

The paper's primary claim is causal: GRPO's normalization of summed rewards destroys information in multi-reward settings, and GDPO's per-reward decoupling preserves it, leading to better optimization. The evidence for the mechanism is strong but mostly theoretical/combinatoric—Figures 2 and 3 demonstrate the collapse exists and quantify its severity. The evidence for the consequences—that this collapse causes worse optimization—is empirical and generally convincing, but with important caveats.

The training curves (Figures 4, 5) provide the most direct causal evidence. In Figure 5 (math reasoning), the correctness reward under GRPO begins declining after step 400 while GDPO continues improving. This is a qualitative difference in optimization behavior—GRPO's training partially collapses, GDPO's does not—that is consistent with the hypothesis that GDPO's richer advantage signal prevents the optimizer from getting stuck in a local optimum where it sacrifices correctness for length. However, the paper does not provide a mechanistic ablation showing that the collapse specifically (as opposed to some other property of GDPO) causes this. For instance, one could test whether artificially increasing GRPO's advantage granularity (by some other means) also prevents the collapse, or whether reducing GDPO's granularity (by post-hoc collapsing certain advantage groups) reintroduces it.

The ablation of GRPO w/o std (Table 2) is particularly revealing and partially addresses this concern. GRPO w/o std increases advantage granularity (Figure 3 shows it produces more distinct groups than GRPO) yet performs worse—format reward fails entirely. This demonstrates that granularity alone is not sufficient; the way advantages are disaggregated matters. But it also raises a question: if GDPO's advantage isn't just "more granular," what specific property makes it better? The paper's answer is that GDPO normalizes per-reward, preserving cross-reward structure, while GRPO w/o std still normalizes the sum. However, one could imagine intermediate ablations—for example, normalizing the sum but using a non-standard-deviation dispersion metric (e.g., range, IQR) that doesn't perfectly cancel magnitude differences, or applying different normalization schemes per reward without decoupling. The paper doesn't explore this design space, leaving some ambiguity about which aspect of GDPO's design is truly necessary.

Are the empirical gains large enough to matter in practice?

For tool calling (Table 1), the gains are modest in absolute terms—2.6 pp average accuracy on 1.5B, 1.7 pp on 3B—but the format compliance improvement (4.3 pp on 1.5B) is practically significant because format violations make outputs unusable regardless of accuracy. In a production setting, an 80.7% format compliance rate (GDPO) versus 76.3% (GRPO) means 4.4% fewer queries fail purely due to malformed output, reducing the need for fallback mechanisms or retries.

For math reasoning (Table 3), the gains are more substantial. A 6.3 pp improvement on AIME for the 1.5B model (23.1% → 29.4%) is a 27% relative improvement—large enough to change the qualitative assessment of the model's capability. For the 7B model, the 2.9 pp AIME gain (50.2% → 53.1%) is smaller in absolute terms but still meaningful at competition-level math, where small improvements are hard-won. The length-exceeding ratio reductions are dramatic—from 2.1% to 0.2% on AIME for 7B, a 10× reduction—and directly translate to lower inference costs (fewer tokens generated per query) and better user experience (no excessively long responses).

For code reasoning (Table 5), the three-reward setting shows the clearest practical advantage. GDPO3-obj reduces bug ratios by 1.5–2.0 pp across benchmarks while maintaining pass rates and improving length control. In a code generation deployment, a 2 pp reduction in bug ratio means 2% more generated code snippets run without errors, directly reducing debugging burden on users.

What experiments are missing that would strengthen the paper?

1. Beyond binary and bounded rewards. All tested rewards are binary (format, length, bug-free, answer correctness) or bounded continuous (correctness in [-3, 3], pass rate in [0, 1]). The combinatoric analysis assumes binary rewards. Real-world multi-reward settings often involve unbounded or heavy-tailed rewards (e.g., BLEU scores, user satisfaction ratings, engagement metrics). GDPO's per-reward normalization using standard deviation assumes roughly symmetric distributions—would it fail if one reward follows a power law while another is binary? The paper doesn't test this.

2. More than 3 rewards. The combinatoric analysis extends to 8 rewards and shows GDPO's advantage growing, but the empirical tests stop at 3. A 4+ reward experiment (e.g., adding safety or coherence rewards to the code reasoning setup) would validate the scaling argument. Without this, the claim that GDPO scales better with reward count is a prediction supported by combinatorics but not directly demonstrated.

3. Direct measurement of advantage signal quality. The paper infers advantage quality from downstream task performance and training curves, but doesn't directly measure properties of the advantage estimates themselves. For example: how correlated are GDPO's advantages with the "true" advantage (if we could compute it from a value function or from exhaustive rollouts)? How does the variance of advantage estimates compare between GDPO and GRPO? These measurements would provide mechanistic validation beyond empirical outcomes.

4. Comparison to value-function methods. The paper positions GDPO as a fixed GRPO, not a new paradigm. But a natural question is whether simply using PPO with separate value heads per reward would also solve the collapse problem (since per-reward value functions would naturally preserve per-reward structure). The paper doesn't compare to PPO, which weakens the claim that GDPO is the right fix rather than just a better fix than the broken baseline.

5. Statistical significance. The five-run protocol for tool-calling provides variance estimates, but the math and code experiments report single runs. For the math results where GDPO's accuracy improvements are small (e.g., 0.6 pp on Minerva for 7B), we cannot distinguish signal from noise without variance information. Given the high cost of RL training, running 5 seeds for all experiments may be impractical, but reporting at least 2–3 seeds for the main math results would substantially increase confidence.

6. Difficulty-stratified analysis. The paper's prior work (DLER) studied difficulty-dependent behavior extensively, showing that optimization strategies that help on easy problems can hurt on hard ones. The current paper doesn't break down math results by question difficulty. GDPO might provide most of its gains on easy-medium problems (where the base model already has some capability) and little on the hardest problems (where no method helps), which would be useful to know.

Do the experiments generalize beyond the tested configurations?

The paper tests two model families (Qwen, DeepSeek) and three scales (1.5B, 3–4B, 7B), which is reasonable breadth. However, all models are in the 1.5B–7B range—a ~14× larger model is never tested. Larger models might have more capacity to learn despite degraded signals, potentially reducing GDPO's relative advantage. Conversely, larger models might have more parameters that could be updated incorrectly by noisy advantage estimates, potentially increasing GDPO's advantage. We simply don't know from these experiments.

The tasks cover tool use, math, and code—all are reasoning-heavy domains with objective correctness criteria. Tasks with subjective or learned reward models (helpfulness, safety, style) are not tested. PRM-based training (as in the reference paper's example) is also not tested—GDPO operates on outcome-level rewards, not process-level rewards. Whether the signal collapse problem exists for process rewards and whether GDPO helps there is unknown.

Are there hidden confounds?

The hyperparameter settings are inherited from prior work (ToolRL for tool-calling, DLER for math/code) that were tuned for GRPO. GDPO might perform even better with GDPO-specific hyperparameter tuning, or GDPO's gains might partially reflect that these hyperparameters happen to work well for both methods. The paper's choice to use identical hyperparameters is methodologically clean (isolates the variable of interest) but means the reported gains are a lower bound—GDPO with tuned hyperparameters could be even stronger. However, the converse concern—that GRPO with different hyperparameters would close the gap—is not addressed by the paper's design. For instance, the learning rate or KL coefficient might interact with advantage granularity: richer advantages might require smaller updates to avoid overfitting to within-group comparisons.

The batch-wise normalization in GDPO (Equation 6) introduces a hyperparameter—the choice of normalization scope (batch-wise). The paper shows removing it causes failures (Appendix A), but doesn't explore alternatives like normalizing over a larger history buffer or using running statistics (as in batch normalization during supervised learning). This choice could interact with batch size—the paper uses batch size 512 throughout; at much larger or smaller batch sizes, the batch-wise normalization might behave differently.

6. Limitations and Trade-offs

6.1 No Comparison to Value-Function-Based Methods (PPO) That Could Inherently Avoid the Collapse Problem

The assumption or constraint. The paper positions GDPO as a fix for GRPO specifically, but does not compare against alternative policy optimization frameworks that could avoid the signal collapse problem entirely. Proximal Policy Optimization (PPO), which uses a learned value function to estimate advantages rather than group-relative normalization, could naturally preserve per-reward structure by maintaining separate value heads for each reward dimension. The paper acknowledges this distinction in Section 2 when introducing GRPO—"GRPO eliminates the need for a value model by leveraging group-relative advantage estimation"—but never evaluates whether this efficiency-motivated design choice is actually worth the optimization fidelity cost that GDPO only partially recovers.

The consequence. Without a PPO baseline, we cannot determine whether GDPO's improvements over GRPO represent the best possible multi-reward optimization, or merely a repair of a fundamentally suboptimal architecture. If PPO with per-reward value heads achieves substantially better multi-reward trade-offs than GDPO, then GDPO's advantage over GRPO would be a local improvement within a globally inferior framework. The paper's central claim—that GDPO is "a better alternative to GRPO for multi-reward RL optimization"—is supported, but the stronger implicit claim that practitioners should prefer GDPO over any alternative is not tested. This matters because the computational cost of training a value function (PPO's overhead vs. GRPO) might be justified if it yields substantially better multi-reward optimization, and practitioners need to make this cost-vs-performance trade-off decision with evidence.

What evidence exists in the paper. None. The paper never implements, trains, or evaluates a PPO baseline, nor does it discuss the expected behavior of value-function-based methods in multi-reward settings. Section 2 notes the existence of PPO only as context for why GRPO was developed. The entire experimental comparison is GRPO variants versus GDPO, leaving the GRPO family as the only evaluated framework.

Mitigation status. The paper does not acknowledge this as a limitation. It frames GRPO as the de facto standard ("GRPO has become the dominant RL algorithm for LLMs") and positions GDPO as improving that standard, which is defensible given GRPO's widespread adoption. However, the absence of a PPO comparison means the paper cannot make claims about GDPO's absolute optimality—only about its superiority within the GRPO ecosystem. This is a practical limitation for practitioners who have not already committed to GRPO-based training infrastructure.

---

6.2 Difficulty Estimation for Reward Priority Is Implicitly Assumed to Be Known in Advance

The assumption or constraint. Section 4.2.1 demonstrates that adjusting reward weights fails to produce intended prioritization when objectives differ substantially in difficulty—the model optimizes the easier objective regardless of weights until the weight differential becomes extreme (e.g., a 4× reduction in length weight from 1.0 to 0.25 before the constraint meaningfully relaxes). The paper's solution is to condition easier rewards on harder ones (Equation 8): the length reward is awarded only when the correctness reward is satisfied, forcing the model to prioritize the harder objective. However, this requires knowing in advance which objective is easier and which is harder, and encoding this knowledge into the reward function design. The paper states: "If one objective is much easier than the others, the model often focuses on maximizing the reward for that objective regardless of the assigned weights" (Section 3.2), but treats this difficulty disparity as an observable property the practitioner can identify.

The consequence. In many real-world multi-reward settings, the relative difficulty of objectives is not known a priori and may change during training. For example, a safety reward might be easy to satisfy initially (simple refusal) but become harder as the model learns to provide helpful responses within safety bounds—the difficulty ordering could invert during training. If the practitioner incorrectly identifies which objective is harder, the conditional reward design (making the wrong reward contingent on the other) could produce exactly the wrong behavior: forcing the model to master an easy objective before receiving any signal from a hard one, when the intended priority was the reverse. The paper provides no diagnostic for determining difficulty ordering and no adaptive mechanism that could adjust conditioning based on observed training dynamics.

The paper also shows that weight tuning becomes more reliable after conditioning resolves the difficulty disparity (Figure 6, right column), but this creates a chicken-and-egg problem: you need to correctly identify and condition on the harder objective before weight tuning works as intended, but you might not know the difficulty ordering until you've attempted training. A failed conditioning choice (conditioning the wrong reward) could waste substantial compute before the error is detected.

What evidence exists in the paper. Section 4.2.1 provides extensive evidence for the difficulty disparity problem itself. Figure 6 demonstrates that weight tuning fails to produce intended effects under unconditional rewards (the length-exceeding ratio barely changes when weight drops from 1.0 to 0.5), and Figure 7 and Table 4 show that conditional rewards fix this. However, the paper studies only one difficulty ordering (length is easier than correctness for math reasoning) and does not test scenarios where the difficulty ordering is unknown or changes during training. The paper's guidance in Section 3.2—"if one objective is much easier than the others" and "adjusting reward weights does not always yield the intended behavior when the difficulty levels of the underlying objectives differ substantially"—assumes the practitioner can identify this disparity, but provides no methodology for doing so beyond observing training curves post-hoc.

Mitigation status. The paper does not acknowledge this assumption as a limitation. The conditional reward strategy is presented as a solution rather than as a technique with its own prerequisite knowledge requirement. The paper suggests future work only implicitly through its analysis—an adaptive difficulty estimation mechanism that adjusts reward conditioning during training based on observed optimization dynamics would address this gap, but is not proposed.

---

6.3 The Advantage Granularity Analysis Shows Combinatoric Improvement Without Empirical Validation at Scale (More Than 3 Rewards)

The assumption or constraint. The paper's combinatoric analysis in Figure 3 demonstrates that GDPO produces substantially more distinct advantage groups than GRPO or GRPO w/o std as the number of rewards grows—over 650 distinct groups for GDPO versus approximately 70 for GRPO at n=8 rewards, an approximately 9× advantage. The paper uses this analysis to argue that GDPO's advantage grows with the number of objectives, stating in Section 3.1: "This demonstrate that the proposed decoupled normalization approach effectively increases the number of distinct advantage groups across all the RL settings and enables more precise advantage estimation." However, the empirical experiments test only 2 and 3 rewards (tool calling and math reasoning use 2, code reasoning uses 3), leaving the scaling claim to 4+, 8+, or more rewards as a combinatoric prediction without experimental validation.

The consequence. We cannot determine whether the combinatoric advantage translates to meaningful optimization improvements beyond 3 rewards. It is possible that the additional advantage granularity becomes redundant—perhaps the model only needs to distinguish a few dozen distinct advantage states to optimize effectively, and GDPO's hundreds of additional states provide no further benefit. It is also possible that training instability worsens as the number of distinct advantage groups grows, since the advantage landscape becomes more complex and the optimizer may struggle to navigate it. The paper's empirical finding that GRPO w/o std (which increases granularity from ~10 to ~30 groups at G=16) performs worse than standard GRPO on format reward (Table 2, 0% format compliance) is a cautionary tale: more granularity does not guarantee better optimization, and can introduce instability. Whether GDPO's decoupled per-reward normalization avoids this instability at higher reward counts is unknown.

More critically, the paper's claim in the abstract—"GDPO consistently outperforms GRPO, demonstrating its effectiveness and generalizability for multi-reward reinforcement learning optimization"—uses "multi-reward" to mean 2–3 rewards. A practitioner with 5, 8, or 10 simultaneous objectives (which is increasingly common as models are expected to satisfy correctness, conciseness, format, safety, citation accuracy, bias mitigation, tone, and other constraints) cannot extrapolate from these results.

What evidence exists in the paper. The combinatoric analysis (Figure 3) extends to 8 rewards and shows GDPO's advantage growing. The empirical results show a pattern consistent with the scaling claim: the two-reward math experiments show GDPO improvements on correctness and length, while the three-reward code experiments show GDPO maintaining pass rate while substantially improving length and bug ratios, a pattern that is more impressive than the two-reward results. The paper acknowledges this pattern implicitly in Section 4.3: "these results demonstrate that GDPO remains effective as the number of reward signals increases." However, this claim is based on going from 2 to 3 rewards—a single step—and the paper does not test 4+ rewards.

Mitigation status. The paper does not explicitly acknowledge the gap between combinatoric prediction and empirical validation as a limitation. The abstract and conclusion refer to "multi-reward reinforcement learning optimization" without qualifying the reward count. A more precise claim would be "GDPO is demonstrated effective for 2–3 reward settings; combinatoric analysis suggests benefits may extend to larger reward counts." This is a reasonable limitation to flag given the paper's emphasis on scalability.

---

6.4 GDPO Requires Batch-Wise Advantage Normalization, Introducing a Sensitivity to Batch Size and Composition

The assumption or constraint. GDPO's third stage—batch-wise advantage normalization (Equation 6)—is empirically necessary for training stability. Appendix A (Figure 8) demonstrates that removing this step causes occasional training failures on the tool-calling task: some runs succeed while others completely fail to learn the format reward. The paper states: "Empirically, we also find that this normalization step improves training stability, as shown in Appendix A, where removing batch-wise normalization occasionally leads to convergence failures." The batch normalization computes the mean and standard deviation of all summed per-reward advantages across the entire training batch (batch size 512 questions × G rollouts per question) and normalizes each advantage using these batch-level statistics.

The consequence. This introduces a dependency on batch composition that GRPO does not have in the same way. In standard GRPO, advantages are computed purely within each question's group of G rollouts—the statistics of other questions in the batch do not affect the advantage estimate for a given question. In GDPO, the batch-wide normalization means that the final advantage for a rollout depends on how that rollout's performance compares to all other rollouts in the batch, not just to rollouts for the same question. This has several practical implications:

• Batch size sensitivity: At very small batch sizes, the batch-level statistics become noisy, potentially destabilizing training. At very large batch sizes, the normalization might wash out meaningful differences—if most rollouts in a large batch perform similarly, the standard deviation shrinks and small differences get amplified. The paper tests only batch size 512 for math/code reasoning and does not ablate this choice.

• Batch composition effects: The advantage assigned to a particular rollout depends on what other questions happen to be in the same batch. If a batch happens to contain many very hard questions (where all rollouts score poorly), a mediocre response to an easy question might receive an inflated advantage because it looks good relative to the batch. If a batch contains many easy questions, a good response to a hard question might receive a deflated advantage. This introduces variance in the training signal that depends on data ordering and batch construction, not just on the reward function and policy.

• Distributed training complications: In large-scale distributed training where batches are split across many GPUs, computing batch-level statistics requires synchronization across devices, adding communication overhead or requiring approximations (e.g., normalizing within each device's micro-batch rather than globally). The paper does not discuss distributed training considerations.

What evidence exists in the paper. Appendix A (Figure 8) provides direct evidence that the batch-wise normalization is necessary—training can fail without it. However, the paper reports only two representative runs (one success, one failure) without quantifying the failure rate across multiple seeds, and does not ablate the batch size or test alternative normalization scopes (e.g., normalizing over a running history rather than the current batch, or normalizing across questions of similar difficulty). The main experiments all use the same batch sizes (512 for tool-calling, 512 for math/code) inherited from prior work's GRPO recipes, so the interaction between batch size and GDPO's batch normalization is unexplored.

Mitigation status. The paper does not acknowledge batch size sensitivity or batch composition effects as limitations. The need for batch-wise normalization is presented as an empirical finding that motivated its inclusion, not as a potential source of instability or a hyperparameter that requires tuning. The small epsilon added to the denominator ("+ε" in Equation 6) provides numerical stability against division by zero but does not address the broader sensitivity to batch statistics. Future work could explore alternative normalization strategies (exponential moving averages of mean and variance, difficulty-stratified normalization) that reduce this dependency.

---

6.5 All Evaluated Rewards Are Binary or Bounded Continuous; Unbounded and Heavy-Tailed Rewards Are Untested

The assumption or constraint. Every reward function in the paper's experiments produces values from a bounded, well-behaved set. In tool-calling (Section 4.1, Appendix C): format reward is in {0, 1}, correctness reward is in [-3, 3] (computed as a normalized matching score mapped to this interval). In math reasoning (Section 4.2): correctness reward is binary {0, 1} (exact match of extracted answer), length reward is binary {0, 1} (response within 4000 tokens). In code reasoning (Section 4.3): pass rate reward is in [0, 1] (proportion of test cases passed), conditioned length reward is binary {0, 1}, bug reward is binary {0, 1}. The combinatoric analysis in Section 3 (Figures 2, 3) explicitly assumes binary rewards to compute distinct advantage group counts.

The consequence. GDPO's per-reward normalization uses the sample mean and standard deviation within each group of G rollouts (Equation 4). For binary and bounded continuous rewards, these statistics are well-behaved—the mean is bounded within the reward range, and the standard deviation is bounded by half the range. For unbounded rewards (e.g., BLEU scores, user engagement time, text length in tokens) or heavy-tailed rewards (e.g., user satisfaction ratings where a few extremely positive or negative outliers dominate), the standard deviation can be dominated by outliers, causing the normalized advantages for typical rollouts to be compressed toward zero while outliers receive extreme values. This could create a training dynamic where the optimizer focuses on rare extreme outcomes at the expense of consistent good performance—a form of reward hacking that GDPO's design does not address.

The combinatoric analysis also depends on the reward space being discrete and small (binary values). For continuous rewards, the number of distinct advantage groups is effectively infinite, and the relevant question becomes whether GDPO preserves more of the information in the continuous reward vector than GRPO, not whether it preserves more distinct configurations. The paper's information-preservation argument relies on combinatorics that assume discrete rewards, and it is unclear whether the advantage carries over to continuous settings where the collapse mechanism is different (GRPO's normalization of summed continuous rewards doesn't produce exactly identical advantages for different reward vectors, but may still compress differences).

What evidence exists in the paper. The only continuous reward tested is the tool-calling correctness reward in [-3, 3]. This reward is bounded, symmetric, and designed to have a meaningful zero point (scores near 0 indicate partial correctness), making it relatively well-behaved. The code reasoning pass rate reward [0, 1] is also bounded. The paper provides no experiments with unbounded rewards, no ablation of the reward distribution shape (e.g., adding noise to make rewards heavy-tailed, scaling rewards to different magnitudes), and no analysis of how sensitive GDPO's per-reward normalization is to outliers or distribution shape. Appendix A shows that batch-wise normalization can fail without careful stabilization, suggesting sensitivity to distribution, but this is not explored systematically.

Mitigation status. The paper does not acknowledge the bounded-reward assumption. The combinatoric analysis in Figure 3 explicitly states it assumes binary rewards in the caption context ("GDPO consistently preserve a substantially larger number of distinct advantage groups"), and the derivation in Figure 2 uses binary rewards, but the paper's claims about GDPO's effectiveness are not qualified by reward type. A practitioner applying GDPO to rewards with different distributional properties (unbounded, heavy-tailed, multi-modal) has no guidance on whether the method will work or how to adapt it. Extending GDPO to handle heterogeneous reward distributions—e.g., using robust statistics (median and IQR instead of mean and standard deviation) or per-reward adaptive normalization—would be a natural next step.

---

6.6 Training Variance Is Not Quantified for the Primary Results (Math and Code Reasoning)

The assumption or constraint. The paper's strongest claimed improvements—up to 6.3% higher AIME accuracy for DeepSeek-R1-1.5B, 2.9% higher AIME accuracy for DeepSeek-R1-7B, 10× reduction in length-exceeding ratio—come from the math reasoning experiments (Table 3) and code reasoning experiments (Table 5). These experiments are run once per configuration (a single training run followed by evaluation). The tool-calling experiments (Tables 1–2) are run five times with different random seeds and report averages with training curves showing median and interquartile range (Figures 1b, 4). The paper states for tool-calling: "We finetune the models with GRPO and GDPO across five runs and report the average accuracy" (Section 4.1). No such statement appears for math or code reasoning.

The consequence. Without multiple runs, we cannot determine whether observed differences between GDPO and GRPO are statistically reliable or within typical run-to-run variance. RL training for LLMs is known to exhibit high variance—different random seeds can produce models with meaningfully different performance due to stochasticity in rollout sampling, parameter initialization, data ordering, and optimization dynamics. The paper's tool-calling results provide direct evidence of this variance: Figure 4 shows the IQR band for GDPO's format reward spans from approximately 0.7 to 1.0 at step 40, while GRPO's correctness reward IQR spans from approximately 0.5 to 1.3 at step 60. This is substantial variance—a single unlucky run could underperform the average by a meaningful margin.

For the math results, several of GDPO's accuracy advantages are small: on DeepSeek-R1-7B, GDPO outperforms GRPO by 0.2 pp on AMC and underperforms by 0.2 pp on MATH and 0.5 pp on Olympiad (Table 3). These differences are almost certainly within single-run noise. Even the larger gains—2.9 pp on AIME for 7B—might not be statistically significant if run-to-run variance is comparable to the 3–6 pp IQR ranges observed in tool-calling. The paper's headline claim of "up to 6.3% higher accuracy on AIME" (for the 1.5B model) is based on a single run and could overstate the true expected improvement if that run was unusually favorable to GDPO.

The length-exceeding ratio reductions face the same issue: GDPO achieves 0.2% vs. 2.1% on AIME for 7B—a 10× reduction—but if a single unlucky batch during evaluation caused GRPO to generate a few long responses, the difference might reflect sampling noise rather than a stable property of the trained policy. The maximum batch response length curves in Figure 5 (right panel) show that GRPO's max length begins increasing after step 400 and exhibits visible fluctuations, while GDPO's steadily decreases—this is more suggestive of a real effect since it reflects training dynamics over many steps, but is still a single-run observation.

What evidence exists in the paper. The paper provides strong variance evidence for tool-calling (5 runs, median + IQR training curves) and essentially none for math and code reasoning (single runs). The training curves for math reasoning (Figures 5, 9, 10) are single-run curves; the evaluation results (Tables 3, 4, 5) are single-model evaluations. The paper's claim in Section 4.2 that "GDPO consistently provides better alignment to the length constraint" is supported by the training dynamics in the single runs shown but not by statistical evidence across multiple runs.

Mitigation status. The paper does not acknowledge the absence of variance estimates for math and code experiments as a limitation. This is a significant gap given that these experiments contain the paper's headline results (AIME improvements, length-exceeding ratio reductions). The cost of running 5 seeds for 7B model RL training on 40k math problems for 500 steps would be substantial—likely hundreds of GPU-hours per run—which may explain the practical constraint. However, the paper could acknowledge this limitation explicitly and note that the reported numbers should be interpreted as point estimates from single training runs rather than expectations with quantified uncertainty. Even 2–3 runs would provide substantially more confidence than a single run, and could be feasible with smaller-scale experiments (e.g., 1.5B model with 3 runs).

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around multi-reward RL for language models from reward design to optimization fidelity—a reframing, not a paradigm shift, but one with substantial practical consequences. Before this work, the dominant mental model was: design good reward functions, sum them, apply GRPO, and tune weights if necessary. The paper demonstrates that this pipeline has a hidden bottleneck: GRPO's summed-reward normalization compresses the multi-dimensional reward structure into a degraded signal, and this compression is not a minor inefficiency but a qualitative failure mode that can cause training collapse (Figure 5), zero learning on certain objectives (Table 2, format reward under GRPO w/o std), and optimization behavior that contradicts the reward designer's intent (Section 4.2.1, weight tuning failing to produce intended prioritization).

The magnitude of this shift is best understood as making optimization fidelity a first-class design concern alongside reward engineering. Prior work on GRPO variants (GSPO, DAPO, GFPO, DLER) focused on the policy update mechanics—clipping, sampling, loss formulation, length penalties—while treating advantage computation as a solved problem inherited from single-reward GRPO. This paper shows that advantage computation is the bottleneck for multi-reward settings and that a surgical change (decoupling per-reward normalization) reliably improves outcomes across tasks, model scales, and reward configurations. The $4\times$ improvement in format compliance on tool-calling (76.3% → 80.7%), the 6.3 percentage point gain on AIME for the 1.5B model, and the 10× reduction in length-exceeding ratio for the 7B model are not achieved by better reward design—the rewards are identical across GDPO and GRPO—but by more faithfully transmitting the existing reward structure to the policy update.

The paper also partially reconciles a tension in the literature between reward-design-focused and optimizer-focused approaches. The finding that GRPO w/o std increases advantage granularity (Figure 3) yet fails catastrophically on format reward (Table 2, 0% compliance) demonstrates that "more signal" is not the same as "better signal"—the normalization strategy matters qualitatively, not just in terms of how many distinct advantage states it produces. GDPO succeeds where GRPO w/o std fails because decoupled per-reward normalization preserves the joint structure of the reward vector, while simply removing the standard deviation term still normalizes the sum and introduces instability. This resolves the apparent paradox of why GRPO w/o std was proposed as an improvement (for question-level difficulty bias) but performs worse in multi-reward settings: the two problems require different solutions, and a fix for one can exacerbate the other.

The paper also makes verifier over-optimization in multi-reward settings a more tractable problem, though indirectly. The training curves for math reasoning (Figure 5) show that under GRPO, the model initially over-optimizes the length reward (rapidly reaching perfect length scores at step 100) at the expense of correctness, and then correctness partially collapses after step 400 as length control degrades. Under GDPO, the same over-optimization dynamic occurs initially (length reward rises, correctness dips), but GDPO's richer advantage signal allows the optimizer to recover from this phase—correctness continues improving past step 400 and maximum response lengths continue decreasing (right panel). This suggests that GDPO's advantage structure provides the optimizer with a more accurate gradient signal for navigating the trade-off between competing objectives, reducing the tendency for over-optimization on one objective to permanently damage another. The conditional reward design (Section 4.2.1) addresses this more directly by structurally preventing over-optimization, but GDPO's improvement even with unconditional rewards (Table 3) indicates the advantage signal itself provides some protection.

Research directions that become more attractive after this work:

• Designing new advantage computation strategies for multi-objective policy optimization—the paper establishes that the specific normalization structure matters and provides a combinatorial framework (Figure 3) for evaluating alternatives.
• Combining reward conditioning (Equation 8) with improved advantage computation, since the paper shows these are complementary: GDPO with conditional rewards achieves the best results (57.7% AIME with 12.3% exceed vs. 53.1%/0.2% for unconditional GDPO), demonstrating that better optimizers amplify the benefits of better reward design rather than making reward design obsolete.
• Scaling to larger reward counts—the combinatoric analysis predicts GDPO's advantage grows with $n$, and empirical results for 2 and 3 rewards are consistent with this, but testing at 4+ rewards is now a well-motivated capability assessment rather than an open-ended exploration.
• Empirical studies of training dynamics under different advantage granularities—the GRPO w/o std result (Table 2) shows that the relationship between advantage expressiveness and optimization stability is non-monotonic, and GDPO's combination of high expressiveness with stability is a specific property worth understanding mechanistically.

Research directions that become less attractive:

• Blindly applying GRPO to multi-reward settings without examining the optimizer's suitability—the paper provides a clear diagnostic (Figure 2) and quantitative metric (distinct advantage group count) for assessing whether a given optimization algorithm is appropriate.
• Tuning reward weights as the primary mechanism for encoding priorities when objectives differ in difficulty—Section 4.2.1 shows this is unreliable without first resolving the difficulty disparity through conditional rewards, and even then, the optimizer's ability to faithfully transmit the weighted signal depends on advantage computation quality.
• Adding more reward components to GRPO-based training pipelines without improving the advantage computation—the combinatoric analysis shows that additional rewards under standard GRPO produce only modest increases in distinct advantage groups, so the marginal benefit of each new reward is degraded by the collapsed signal.

Follow-Up Research This Work Enables

Direct measurement of advantage signal quality through value function correlation. The paper infers advantage quality from downstream performance and training curves, but never directly measures whether GDPO's advantages are "better" in a quantifiable sense. A strong follow-up would train a separate value function (PPO-style critic) on the same rollouts and compute the correlation between GDPO's group-relative advantages and the critic's GAE advantages, compared to GRPO's. The hypothesis: GDPO's advantages should correlate more highly with the value-function estimates because they preserve more of the reward structure. If the correlation is similar, it would suggest GDPO's benefit comes from something other than advantage accuracy (perhaps reduced variance or better gradient alignment). This experiment would clarify the mechanism, which is currently attributed to signal preservation without direct evidence.

Intermediate ablation: per-reward normalization with different dispersion metrics. GDPO normalizes each reward using mean and standard deviation (Equation 4). An informative ablation would test alternative per-reward normalization strategies that preserve different aspects of the reward distribution: (a) range normalization ($A = (r - \min)/(\max - \min)$), (b) quantile-based normalization ($A = \Phi^{-1}(F(r))$ where $F$ is the empirical CDF), (c) median/median absolute deviation instead of mean/std. The rationale: GRPO's failure mode (Figure 2) arises because the standard deviation exactly cancels the magnitude of the reward difference. Range normalization would preserve relative magnitudes (a difference of 2 is always larger than a difference of 1), but might be more sensitive to outliers. Quantile normalization would handle arbitrary reward distributions (heavy-tailed, multi-modal) more robustly than standard deviation. The paper's tools—the distinct advantage group analysis (Figure 3) and the tool-calling benchmark with five-run variance estimation—provide a direct evaluation framework for these variants.

Stress-testing GDPO with deliberately adversarial reward structures. The paper demonstrates GDPO works when rewards are complementary or weakly competing (length vs. correctness). A strong test of GDPO's robustness would design reward functions that are strongly anti-correlated or mutually exclusive—for example, a "verbosity" reward that awards higher scores for longer responses alongside the length constraint reward, or a "creativity" reward that penalizes exact-match correctness. The question: does GDPO's richer advantage signal help the optimizer navigate genuinely conflicting objectives, or does it amplify destructive interference between them? The hypothesis from the current results: GDPO should preserve the conflict more faithfully, meaning training curves would show persistent tension between objectives rather than one dominating—but this might also mean the optimizer never converges on either, which could be worse than GRPO's implicit resolution (optimizing the easier one). This experiment would define the boundary conditions for GDPO's applicability.

Scaling to 4–8 rewards with controlled reward difficulty calibration. The combinatoric analysis (Figure 3, right panel) predicts GDPO's advantage granularity grows from ~15 groups at $n=2$ to ~650 at $n=8$, while GRPO grows from ~5 to ~70—a ~9× advantage that widens with $n$. A direct test would extend the code reasoning setup to 4+ rewards by adding objectives like code efficiency (runtime), code style (lint compliance), and documentation quality (docstring presence). Critically, this experiment should control for reward difficulty: add rewards of varying difficulty (easy: enforce specific import statements; medium: pass style checker; hard: achieve sub-quadratic complexity) and measure whether GDPO's advantage over GRPO grows with reward count, as the combinatorics predict. A flat or declining advantage would falsify the scalability claim, while a growing advantage would validate it and provide practical guidance for complex multi-objective deployments.

GDPO combined with learned reward models (PRMs, learned verifiers). All experiments in the paper use rule-based, deterministic reward functions (format compliance, exact match, test case pass rate). In many practical settings—helpfulness, safety, coherence—rewards come from learned models (PPO value functions, PRMs, LLM-as-judge) that are themselves noisy and potentially miscalibrated. The signal collapse problem might be more severe with learned rewards because the reward distributions have more entropy (smoother, less structured than binary rewards), giving GRPO's standardization more information to destroy. Conversely, GDPO's per-reward normalization might amplify noise from poorly calibrated learned rewards—if a learned reward model has high variance, per-reward normalization could inflate the importance of noisy reward dimensions. A follow-up should replicate the math reasoning setup but replace the binary length reward with a learned "conciseness quality" model (trained on human preferences) and measure whether GDPO still outperforms GRPO. The experiment would establish whether GDPO's advantages are specific to rule-based rewards or generalize to the learned-reward settings that dominate RLHF deployments.

Adaptive difficulty detection and dynamic reward conditioning. Section 4.2.1 identifies a critical practical problem: reward conditioning requires knowing which objective is harder before training. A follow-up could develop an online difficulty estimator that monitors per-reward training dynamics (e.g., the rate of reward improvement, the variance of within-group advantages) during the first ~50 steps and automatically configures reward conditioning based on detected difficulty ordering. Concretely: start training with equal weights and unconditional rewards, track the slope of each reward's moving average, and after a warmup period, condition the reward with the steeper slope (easier objective) on the reward with the shallower slope (harder objective). This would be evaluated against the paper's fixed-conditioning baseline (Table 4) to determine whether adaptive conditioning matches or exceeds the performance of correctly-chosen fixed conditioning, and whether it avoids the catastrophic failure mode of incorrectly-chosen fixed conditioning (conditioning the hard reward on the easy one). The paper's five-run tool-calling protocol provides a framework for measuring the variance and reliability of such an adaptive mechanism.

Practical Applications and Downstream Use Cases

Production RLHF pipelines with multiple simultaneous constraints. Organizations training language models for deployment (e.g., chatbot providers, coding assistant companies) typically optimize for helpfulness, safety, conciseness, and format compliance simultaneously. The paper's tool-calling results (Table 1) demonstrate that switching from GRPO to GDPO increases format compliance from 76.3% to 80.7% for the 1.5B model—a 4.4 percentage point absolute improvement that directly translates to fewer failed API calls and lower retry rates in production. For a service handling millions of queries daily, a 4.4% reduction in format failures represents substantial cost savings in error handling and improved user experience. The implementation cost is minimal: GDPO replaces one function in the training loop and is already implemented in verl, HF-TRL, and Nemo-RL, the same frameworks used for GRPO. The paper shows this gain comes with no accuracy trade-off—correctness also improves (30.2% → 32.8% average accuracy)—so the deployment decision is a strict improvement with no downside.

On-device model deployment with multi-objective fine-tuning. The paper's results on the 1.5B model scale (tool-calling: +2.6 pp accuracy and +4.3 pp format; math reasoning: +6.3 pp AIME accuracy) are particularly relevant for edge deployment, where model capacity is tightly constrained and every percentage point of performance matters. A mobile coding assistant running a 1.5B model fine-tuned with GDPO would produce code that is equally correct (comparable pass rates, Table 5) but substantially less buggy (Apps bug ratio 18.8% vs. 20.3% for GRPO3-obj) and more concise (Apps length-exceed 8.5% vs. 11.2%). For an on-device application where debugging support is limited and long responses cause poor UX, the bug ratio and length improvements are directly user-facing. The paper shows these gains are achieved with identical training compute to GRPO—no additional FLOPs, just a different advantage computation.

Self-improvement data generation pipelines with quality constraints. When using LLMs to generate training data for distillation or iterative self-improvement (as in the reference paper's STaR/ReST paradigm), the quality of generated solutions is critical, but so is their diversity, format correctness, and efficiency. A system generating math reasoning traces for knowledge distillation needs responses that are simultaneously correct, well-formatted, and concise. The paper's math results with conditioned length rewards (Table 4) show GDPO achieves 57.7% AIME accuracy with 12.3% length violations versus GRPO's 53.3% with 29.2% violations—significantly more correct answers that also respect length constraints, producing higher-quality training data. The three-reward code results (Section 4.3) extend this to code generation: GDPO reduces bug ratios by 1.4–2.0 percentage points while maintaining pass rates, meaning self-improvement pipelines would generate fewer buggy code examples, reducing the risk of the student model learning flawed patterns.

Multi-objective preference alignment with verified reward priorities. The paper's analysis of reward priority variation (Section 4.2.1) provides a practical recipe for organizations that need to encode explicit priorities into model behavior—for example, a medical QA system where correctness is non-negotiable and conciseness is secondary, or a customer service bot where tone appropriateness is prioritized over response speed. The paper demonstrates that: (1) simply weighting rewards fails when objectives differ in difficulty (Figure 6, left), (2) conditioning easier rewards on harder ones fixes the difficulty disparity (Figure 7), and (3) GDPO amplifies the benefits of conditioning while GRPO partially undermines them (Table 4). A practitioner deploying such a system would: first, identify which objective is most critical (e.g., correctness for medical QA); second, condition secondary rewards on the critical one (e.g., conciseness bonus only when answer is correct); third, use GDPO rather than GRPO as the optimizer to ensure the conditioning is faithfully transmitted to the policy. The paper provides the evidence (across 5 math benchmarks, 4 code benchmarks, and a tool-calling benchmark) that this recipe produces models that better reflect the intended priorities than either weight-tuning under GRPO or conditioning under GRPO.

When to Prefer This Method

The paper explicitly positions GDPO as a replacement for GRPO in multi-reward RL settings, backed by consistent empirical improvements across all tested configurations. The decision rule is straightforward based on the paper's evidence:

• Prefer GDPO over GRPO when optimizing for two or more objectives simultaneously, regardless of whether objectives are complementary, competing, or independent—the paper demonstrates improvements across all three task types (tool calling, math, code) and across all tested objective combinations. The implementation cost is zero (same training frameworks, same hyperparameters, different advantage function), making the switch a strict upgrade with no downside demonstrated in the paper.

• The case strengthens when: (a) objectives differ substantially in difficulty (GDPO's advantage granularity matters more when the optimizer is pulled in different directions); (b) one reward is structural or sparse (format, bug-free) while another is smooth (accuracy, pass rate)—GDPO preserves the distinction between reward types, while GRPO collapses them; (c) you are already using conditional rewards to encode priorities—GDPO more faithfully transmits the conditioned structure (Table 4); (d) you operate at smaller model scales (1.5B–3B) where training signal quality matters more because model capacity cannot compensate for degraded signals.

• Be cautious (the paper provides no direct evidence, but these are logical extensions of the limitations) when: (a) using rewards with extreme heavy-tailed or unbounded distributions where per-reward standard deviation normalization may be unstable; (b) operating at very small batch sizes (the batch-wise normalization in Equation 6 uses batch statistics and may become noisy); or (c) working with more than 3 rewards where the combinatoric advantage is predicted but not empirically validated. In these cases, the paper's results do not guarantee improvement, and monitoring training dynamics (as in Figures 4 and 5) is advisable when first adopting GDPO.