Fine-Grained GRPO for Precise Preference Alignment in Flow Models

ArXiv: 2510.01982

🎯 Pitch

Other flow-model RL methods inject noise across every denoising step, but G2RPO shows that restricting stochasticity to a single step and evaluating with ODE rollouts at multiple grains yields a per-step reward signal that is an order of magnitude more precise. This simple redesign flips credit assignment from hopelessly sparse to densely informative, boosting a Flux.1-dev model’s out-of-domain Image Reward from 1.13 to 1.48.

---

1. Executive Summary

This paper introduces Granular-GRPO (G2RPO), a fine-grained online reinforcement learning framework for aligning flow-based generative models with human preferences, evaluated on the Flux.1-dev model using the HPSv2 prompt dataset. The core contribution is a two-part mechanism: Singular Stochastic Sampling — confining stochastic SDE exploration to a single denoising step rather than injecting noise uniformly across all steps, which creates a dense, step-specific reward signal — and Multi-Granularity Advantage Integration — aggregating advantage estimates computed from ODE denoising at multiple interval scales (fine-grained vs. coarse-grained step counts) to produce a robust evaluation of each sampling direction. When trained jointly with HPS-v2.1 and CLIP Score rewards, G2RPO improves the out-of-domain Unified Reward from 3.569 to 3.783 and Image Reward from 1.128 to 1.483 over the DanceGRPO baseline, while also achieving 6.52% relative improvement on in-domain HPS-v2.1 under single-reward training, establishing that precise per-step credit assignment and multi-scale trajectory assessment can substantially raise the optimization ceiling — but only when the additional sampling overhead (~45.7% more denoising steps during training) is acceptable at training time, with inference latency remaining unchanged.

2. Context and Motivation

The Core Problem: We Can't Assign Credit for Exploration in Flow Model Reinforcement Learning

The fundamental challenge this paper tackles arises at the intersection of two powerful ideas that have recently been combined: flow-based generative models (specifically rectified flow models like Flux) and online reinforcement learning for preference alignment (specifically GRPO). The paper addresses a specific, previously underappreciated failure mode in that combination: when flow-based GRPO methods inject stochasticity across all denoising timesteps to explore, they create a credit assignment nightmare that no existing method properly resolves.

To understand why this matters, we need to step back and consider what makes this specific combination difficult. Flow models—unlike autoregressive language models where GRPO was originally developed—don't generate tokens one at a time in a natural left-to-right sequence. Instead, they iteratively denoise a sample through a trajectory of states, transforming pure noise into a coherent image. When we want to apply reinforcement learning to improve alignment with human preferences, we need the model to explore—to try different denoising paths and learn which ones lead to higher rewards. The standard approach, introduced by Flow-GRPO and refined by DanceGRPO and MixGRPO, is to inject Gaussian noise at every single denoising step via an ODE-to-SDE conversion, creating stochastic variation throughout the entire trajectory.

This creates a classic sparse reward problem, but one with a specific structure that the paper identifies as the central bottleneck. A final image reward is computed after the entire T-step denoising sequence completes. The reward signal then gets uniformly broadcast to every step: the exact same advantage value (computed from the final image's quality score) is assigned to the SDE sampling direction at step T, step T-1, step T-2, and so on—all the way back to the initial noise. The paper articulates this clearly in Section 3.2:

"the reward signal available only after multiple decision steps impedes the model's capability to link the final reward to each decision, thereby resulting in imprecise and sparse rewards."

This is not merely a conceptual inconvenience. The practical consequence is that the model receives the same optimization signal for a good early decision that set the overall composition on the right track as it does for a mediocre late decision that merely refined texture details. There's no mechanism to distinguish which specific perturbation at which specific step was responsible for the final outcome. In reinforcement learning terminology, this is a failure of temporal credit assignment—and it means the gradient updates are diluted across all timesteps rather than concentrated where they would be most informative.

Why This Matters: The Alignment Ceiling for Visual Generation

This problem is important for both practical and theoretical reasons. On the practical side, aligning text-to-image models with human preferences is increasingly critical as these models are deployed at scale. A model that produces technically high-quality images but misunderstands prompts, generates aesthetically unpleasing compositions, or fails to capture nuanced spatial and relational constraints (a "sad white king" to the right of a "proud black king" on a chessboard, as in one of the paper's examples) has limited utility. Reinforcement learning from human feedback (RLHF) has emerged as the dominant paradigm for alignment because it's adaptive—it can be tuned to arbitrary reward signals—and cost-effective compared to collecting massive supervised datasets. But the effectiveness ceiling of RLHF for flow models is directly determined by the quality of the optimization signal. If that signal is sparse and imprecise, the model's alignment will plateau at a suboptimal level regardless of how much training compute is expended.

The theoretical significance is even deeper. Flow-based models operate in a continuous state space with a fundamentally different structure than the discrete token spaces where GRPO was pioneered. The trajectory from noise to image is not a sequence of discrete choices with natural boundaries (like choosing the next word) but a smooth transformation where "steps" are an artifact of numerical discretization. The fact that existing methods treat every discretized timestep identically—injecting the same amount of noise and receiving the same reward signal—ignores the reality that early denoising steps (where the overall layout, color palette, and composition are determined) and late denoising steps (where fine details are refined) play qualitatively different roles in the generation process. The paper's approach implicitly acknowledges this by focusing GRPO training on the first half of the denoising timesteps ($M = \{T, T-1, \ldots, \lfloor T/2 \rfloor\}$):

"This selection prioritizes the early denoising steps, where the exploration space of the SDE is larger and significantly determines the overall direction of the entire denoising chain. In contrast, the improvements from GRPO in the later stages are relatively minor."

But even with this temporal restriction, the uniform reward broadcasting remains—every early step still gets the same advantage signal whether it was responsible for the good outcome or not.

Where Existing Approaches Fall Short

The paper identifies two specific, intertwined limitations in prior flow-based GRPO methods that the proposed G2RPO framework addresses:

1. Sparse Reward from Uniform Stochasticity Injection

DanceGRPO and Flow-GRPO both use what the paper calls "long-range stochasticity injection"—SDE sampling with noise added at every timestep. The reward signal, computed only after the full denoising trajectory completes, is then uniformly distributed across all steps via advantage broadcasting ($A^i_t = A^i_0$ for all $t$). This is mathematically precise but informationally impoverished. The paper identifies this as a sparse reward problem analogous to the challenge in multi-step RL where an agent receives a single terminal reward after a long sequence of actions—a well-known source of training instability and slow convergence.

The paper's Figure 1(b) illustrates this contrast clearly. In the "Sparse Reward" regime (existing methods), the "Final Reward" signal flows backward uniformly across all SDE steps (represented as a single bar of reward assigned identically at each perturbation point). In the proposed "Dense Reward" regime, the stochasticity is confined to a single step, creating a direct causal chain from perturbation to reward.

What makes this particularly problematic is that it conflates two distinct sources of variation when computing group advantages. In DanceGRPO, the group of images $\{x^i_0\}_{i=1}^G$ generated from the same prompt varies both because of the SDE perturbation at step T and the perturbation at step T-1, and so on. The final reward differences across the group are the accumulated effect of all these perturbations, making it impossible to determine which step's perturbation contributed positively or negatively to the outcome. The advantage computed for step T includes noise from steps T-1 through 0, and vice versa. This is not just inefficient—it actively misleads the optimization.

2. Incomplete Evaluation from Fixed-Granularity Denoising

The second limitation is more subtle and represents the paper's key conceptual insight. Even if we could perfectly isolate each step's contribution to the final reward (which Singular Stochastic Sampling achieves), we would still face a question: at what granularity should we evaluate the downstream trajectory?

The paper's Figure 3 provides the empirical evidence that motivates this concern. When the same SDE sampling direction (same perturbation at step k, leading to the same next state $x_{k-1}$) is followed by ODE denoising at different granularities—coarse (fewer steps, each covering a larger timestep interval) versus fine (more steps, each covering a smaller interval)—the resulting images can receive different reward scores from the same reward model. The paper shows specific examples where Sample 2 in the fine-grained group has a reward of 0.3534 (highest in its group) while the same direction in the coarse-grained context has a reward of 0.3440 (merely average). The group advantages flip based purely on the downstream denoising granularity.

This is not merely noise in the reward model. It reflects a genuine ambiguity: the "quality" of a sampling direction depends in part on how the remainder of the trajectory is resolved. A direction that sets up a good composition but requires fine detail work in later steps may look better under coarse-grained evaluation (where detail work is skipped and the overall structure dominates) than under fine-grained evaluation (where missing details become apparent). Conversely, a direction that captures initial detail well but has structural issues may score higher in the fine-grained setting.

Existing methods are locked to a single granularity: the denoising direction is always followed by the same fixed number of ODE steps. This means the evaluation of whether one direction is better than another is contingent on that specific granularity choice, which is essentially arbitrary. As the paper states in Section 3.3:

"the denoising trajectory generated by singular stochastic sampling is not robust when assessing the corresponding SDE denoising direction."

The consequence is that group advantages computed under a single granularity may be unreliable indicators of true sample quality. A direction that appears superior at one granularity may be inferior when evaluated more comprehensively, and vice versa. The policy gradient derived from these potentially misleading advantages will push the model in directions that may not generalize.

How This Paper Positions Itself

The paper frames its contribution not as a wholly new alignment paradigm but as a precision upgrade to existing flow-based GRPO methods. It accepts the established framework—ODE-to-SDE conversion for exploration, group-relative advantage computation, PPO-style clipped policy updates—and identifies the specific components where fidelity is lost.

The proposed solution is decomposed cleanly along the two identified problems:

• Singular Stochastic Sampling addresses the sparse reward problem by localizing exploration to individual timesteps. Rather than injecting SDE noise at all T steps, the method selects a single step k for stochastic exploration while keeping all other steps deterministic via ODE sampling. The reward signal arriving from the final image is then solely attributable to the perturbation at step k, creating a dense, step-specific advantage signal ($A^i_k$ rather than $A^i_0$ broadcast to all steps). This is a targeted intervention—it doesn't change the SDE formulation, the group size, or the optimization objective, but it fundamentally restructures the credit assignment chain.

• Multi-Granularity Advantage Integration addresses the incomplete evaluation problem by computing the advantage for each sampling direction multiple times—once under each of several denoising granularities—and then summing the per-granularity advantages to form a composite evaluation. The set $\Lambda = \{1, 2, 3\}$ specifies three granularity levels: $\lambda = 1$ (standard fine-grained, every step), $\lambda = 2$ (every other step), and $\lambda = 3$ (every third step). The composite advantage $A^{i,\text{mix}}_k = \sum_{j} A^{i,j}_k$ integrates assessments across these scales, so a sampling direction must perform well at multiple resolutions to receive a strong positive update.

The paper positions this approach as filling a gap between the theoretical desirability of fine-grained evaluation (more robust, less granularity-dependent) and the practical constraints of training (additional computation). The multi-granularity sampling adds approximately 45.7% more denoising steps during training (as detailed in the supplementary material, Section 11), but this cost is incurred only during the training phase—inference latency is completely unaffected since the model architecture and inference-time sampling procedure remain unchanged.

Relationship to Prior Work

The paper situates itself within a specific lineage. The original GRPO algorithm (Shao et al., 2024) was developed for language models and eliminated the need for a separate value function by using group-relative advantages. Flow-GRPO (Liu et al., 2025) first adapted this to flow models by converting the deterministic ODE into a stochastic SDE with the same marginal distribution, enabling GRPO-style group sampling and relative comparison. DanceGRPO (Xue et al., 2025) refined this approach with practical improvements like restricting training to the first half of timesteps and joint training with multiple reward models (HPS-v2.1 and CLIP Score) to improve robustness against reward hacking. MixGRPO (Li et al., 2025) further improved efficiency through hybrid ODE-SDE sampling while maintaining comparable performance.

G2RPO builds directly on DanceGRPO's architecture—it inherits the same SDE formulation (Eq. 3 with $\eta = 0.7$), the same group size ($G = 12$), the same total sampling steps ($T = 16$), the same advantage clipping ($\epsilon = 5$), the same KL penalty setting ($\beta = 0$, following DanceGRPO's finding that this stabilizes training), and the same joint reward training setup (HPS-v2.1 + CLIP Score). The paper's innovations are intra-step: rather than changing how multiple reward models are combined (which DanceGRPO already addressed), G2RPO changes how the advantage signal is computed for each individual step's sampling direction.

There's an important conceptual parallel acknowledged in the paper: the Multi-Granularity Advantage Integration sums advantages from different granularities, analogous to how DanceGRPO sums advantages from different reward models. The paper makes this connection explicit:

"Similar to the joint training with multiple reward models in DanceGRPO, where the advantages from different reward models are directly summed to provide a multi-dimensional comprehensive evaluation of a single denoising direction, we combine the advantages from different granularities."

This positions MGAI as a natural extension of the multi-reward philosophy from the reward-model axis to the denoising-granularity axis.

The Underlying Assumption and Its Justification

An important implicit assumption in this work is that a good sampling direction at step k should be robust to variations in downstream denoising granularity. As the paper states in the introduction, describing the motivation for Figure 3:

"an ideal high-quality direction intuitively is robust to the granularity of the remaining sampling trajectory. This means that although the absolute quality of the final results may vary with the number of steps, an excellent denoising direction still maintains a superior relative position within the group."

This assumption—that robustness across granularities is a signal of genuine quality rather than granularity-specific overfitting—is central to the MGAI design. The paper doesn't provide theoretical justification for this claim but supports it empirically: the fact that MGAI improves out-of-domain metrics (Unified Reward, Image Reward, UniGenBench++) more than in-domain metrics (HPS-v2.1) suggests that multi-granularity evaluation prevents the model from exploiting granularity-specific shortcuts that might improve HPS-v2.1 scores but degrade generalization. This is consistent with the paper's observation that single-granularity training with Singular Stochastic Sampling alone (G2RPO w/o MGAI) can sometimes degrade metrics like Style and Layout on UniGenBench++ (Supplementary Table 5) despite improving HPS-v2.1, and that MGAI helps recover and surpass baseline performance on these metrics.

What the Paper Does NOT Address

The paper carefully scopes its claims. It does not propose a new RL algorithm, a new reward model, or a new flow model architecture. It does not address the fundamental challenge of reward hacking—the tendency for models to exploit biases in reward models to achieve high scores without genuine quality improvements—beyond what DanceGRPO's multi-reward joint training already provides. The paper acknowledges that HPS-v2.1 alone is "prone to model hacking due to biases in the training set" (Section 4.1) and uses joint training as mitigation, but doesn't claim to solve this problem.

Additionally, the paper does not address the training time overhead systematically in the main text, deferring the quantification to the supplementary material. This is a notable gap in the main argument, since the 45.7% additional denoising steps represent a meaningful practical cost that users must weigh against the benefits. The paper also doesn't explore whether all three granularities ($\Lambda = \{1, 2, 3\}$) are necessary or whether a subset would achieve comparable performance at lower cost—the ablation in Table 2 shows that $\{1, 2\}$ and $\{1, 3\}$ both perform close to $\{1, 2, 3\}$, suggesting potential for optimization.

3. Technical Approach

3.1 Reader Orientation

This paper develops a training-time framework for online reinforcement learning that sits on top of an existing flow-based text-to-image model (Flux.1-dev) and an existing RL algorithm (GRPO), modifying how the model explores the space of possible denoising trajectories and how reward signals are computed from those trajectories, without changing the model architecture or the inference-time sampling procedure at all. The core problem it solves is that when existing flow-based GRPO methods inject noise across all denoising steps to explore, they create two compounding failures: (1) the final image reward is uniformly broadcast to every step, so no step can be individually credited or blamed for the outcome, and (2) each sampling direction is evaluated under a single, arbitrary downstream denoising resolution, making the assessment contingent on that resolution rather than on the direction's intrinsic quality.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major interacting components, all operating during training only:

1. Base Flow Model (Flux.1-dev): A pretrained rectified flow transformer that maps noise to images via a deterministic ODE. Serves as the policy $\pi_\theta$ being optimized. Its weights are what GRPO updates.

2. Singular Stochastic Sampling Module: A sampling strategy that replaces the full-trajectory SDE noise injection of prior methods with a focused procedure: for a chosen timestep $k$, inject Gaussian noise (via the ODE-to-SDE conversion from Flow-GRPO) at only that single step, generating $G = 12$ distinct denoising directions, while all other $T-1$ steps use deterministic ODE sampling. This creates a one-to-one causal link between a specific perturbation and the final reward.

3. Multi-Granularity Advantage Integration (MGAI) Module: A reward evaluation layer that, for each denoising direction produced by the SDE step, runs multiple parallel ODE denoising completions at different granularities (controlled by step-interval factors $\lambda \in \{1, 2, 3\}$), computes a separate group advantage under each granularity, and sums the per-granularity advantages into a single composite advantage. This ensures a direction must perform well at multiple resolutions to receive a strong positive update.

4. Reward Models (HPS-v2.1, CLIP Score, or both): Frozen scoring functions that map (image, text-prompt) pairs to scalar quality scores. These provide the terminal reward signals. The paper uses either HPS-v2.1 alone (in-domain evaluation) or both jointly (for robustness against reward hacking).

5. GRPO Optimization Engine: The policy-gradient update machinery adapted from DanceGRPO: compute group-relative advantages, form a PPO-style clipped surrogate objective with advantage weighting, add an optional KL penalty against a reference policy ($\beta = 0$ in practice, following DanceGRPO's finding that this stabilizes training), and update the flow model parameters via gradient ascent using the AdamW optimizer.

Information flows as follows: A text prompt enters the system → an initial noise sample $x_T \sim \mathcal{N}(0, I)$ is generated → for each timestep $k$ in the training set $M$ (first half of denoising steps), ODE sampling deterministically reaches $x_k$ → Singular Stochastic Sampling injects SDE noise at step $k$ only, producing $G$ candidate next states $\{x^i_{k-1}\}_{i=1}^G$ → the MGAI module branches each candidate into $J = 3$ parallel ODE completion paths at different granularities, producing $G \times J$ final images → each image is scored by the reward model(s) → per-granularity advantages $A^{i,j}_k$ are computed within each group → advantages are summed across granularities to form $A^{i,\text{mix}}_k$ → the GRPO objective uses these composite advantages and the log-probability ratios $r^i_k(\theta)$ at step $k$ to compute the policy gradient → gradient ascent updates $\theta$.

3.3 Roadmap for the Deep Dive

• First, the MDP formulation and SDE sampling (Section 3.1): I will establish the formal framework—how the denoising process is cast as a Markov decision process, what the states and actions are, and the ODE-to-SDE conversion that enables exploration. This grounds all subsequent components.
• Second, the baseline GRPO objective (Section 3.1 continued): I will walk through the standard GRPO training loop inherited from DanceGRPO—group sampling, advantage computation, the clipped surrogate loss, and the KL penalty—because G2RPO modifies what feeds into this objective, not the objective itself.
• Third, Singular Stochastic Sampling (Section 3.2): I will detail the core intervention: how stochasticity is confined to a single step, why the reward becomes step-specific and precise, and the practical efficiency benefit of sharing the starting state across the group.
• Fourth, Multi-Granularity Advantage Integration (Section 3.3): I will explain the second intervention: how multiple granularities are defined, how interval sampling works, how per-granularity advantages are computed and summed, and why this produces a more robust evaluation.
• Fifth, the complete G2RPO objective: I will show how the two modifications plug into the base GRPO objective, yielding the final training loss. This order mirrors the logical dependency chain: the MDP sets the stage → the SDE provides exploration → Singular Stochastic Sampling makes the reward precise → MGAI makes the evaluation comprehensive → the GRPO objective consumes the resulting advantages.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a training methodology paper whose core idea is that the quality of GRPO optimization for flow models is bottlenecked by the precision and comprehensiveness of the reward signal assigned to individual denoising steps, and that two targeted modifications—one to the sampling procedure, one to the reward evaluation procedure—can substantially raise the optimization ceiling.

---

The MDP Formulation and ODE-to-SDE Conversion

The Denoising Process as a Markov Decision Process

The paper inherits from DanceGRPO and Flow-GRPO a specific formulation of flow model denoising as a multi-step Markov decision process (MDP). Understanding this encoding is essential because it defines what the "states," "actions," and "rewards" are—the vocabulary in which both the problem and the solution are expressed.

Given a text prompt $c$, the agent—the flow model with parameters $\theta$—produces a reverse-time trajectory through the denoising process. This trajectory is formally defined as:

$\Gamma = (s_T, a_T, s_{T-1}, a_{T-1}, \ldots, s_0, a_0)$

where each state $s_t = (c, t, x_t)$ is a tuple of the prompt $c$, the current timestep index $t$, and the current noisy latent sample $x_t$. The timestep $t$ runs backward from $T$ (pure noise, $x_T \sim \mathcal{N}(0, I)$) to $0$ (the denoised image $x_0$). The action $a_t$ at timestep $t$ is the denoising operation that transforms $x_t$ into $x_{t-1}$, sampled from the policy $\pi_\theta$:

$x_{t-1} \sim \pi_\theta(x_{t-1} \mid x_t, c)$

What this encoding accomplishes: It reframes the continuous denoising trajectory as a sequence of discrete decision points, each a "choice" of which direction to move in latent space. The policy $\pi_\theta$ is the flow model itself, and the MDP structure enables reinforcement learning algorithms to treat each step's transition as an action that can be evaluated and optimized.

Why this specific encoding: The MDP framing matches how GRPO was originally applied to language models—each token generation is a discrete action—and the step indices $t$ provide natural state boundaries. Including $c$ in every state ensures the agent always conditions on the prompt, and including $t$ in the state ensures the policy is time-aware, which is critical because early denoising steps (determining global structure) and late steps (determining fine details) have qualitatively different roles.

Deterministic ODE Sampling

In standard flow matching, the denoising process from noise to image follows a deterministic ordinary differential equation (ODE):

$dx_t = v_\theta(x_t, t)\, dt$

where $v_\theta(x_t, t)$ is the vector field predicted by the flow model—the model's estimate of the instantaneous direction of change at state $x_t$ and time $t$. This ODE can be discretized into steps: starting from $x_T$, the model predicts $v_\theta(x_T, T)$, takes a step proportional to that direction and the timestep size, arriving at $x_{T-1}$, and repeats.

Why this is the default: Flow models are trained to predict this vector field via flow matching, and ODE sampling is the natural inference procedure. It is deterministic—given the same initial noise $x_T$ and the same prompt $c$, it always produces the same image $x_0$.

The SDE Conversion for Exploration

GRPO requires stochasticity: the algorithm needs a group of different outputs from the same prompt so it can compare them, compute relative advantages, and update the policy toward better outputs. The deterministic ODE provides no variation. The solution, introduced by Flow-GRPO and inherited by all subsequent methods, is to convert the ODE into an equivalent stochastic differential equation (SDE) that has the same marginal distribution—meaning that, in expectation over noise realizations, it produces the same distribution of images—but injects controlled Gaussian noise at each step to create variation:

$dx_t = \left(v_\theta(x_t, t) + \frac{\sigma_t^2}{2t}\left(x_t + (1 - t) v_\theta(x_t, t)\right)\right) dt + \sigma_t\, dw_t$

where $dw_t$ is a Wiener process increment (a standard Gaussian random variable scaled by $\sqrt{dt}$), and $\sigma_t$ controls the magnitude of stochasticity injected. The term $\frac{\sigma_t^2}{2t}(x_t + (1-t)v_\theta(x_t, t))$ is a correction factor that ensures the marginal distribution is preserved despite the added noise.

Why this specific form: The correction term is derived from the Fokker-Planck equation for the flow matching probability flow. It ensures that the SDE's steady-state distribution matches the ODE's—without it, added noise would shift the distribution and the model would generate out-of-distribution samples. This equivalence is critical because it means the SDE explores within the learned manifold rather than drifting off it.

Euler–Maruyama Discretization

For practical implementation, the continuous SDE is discretized using the Euler–Maruyama scheme, which approximates the differential equation over a finite step $\Delta t$:

$x_{t+\Delta t} = x_t + \left(v_\theta(x_t, t) + \frac{\sigma_t^2}{2t}\left(x_t + (1 - t) v_\theta(x_t, t)\right)\right) \Delta t + \sigma_t \sqrt{\Delta t}\, \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)$

What this computes: Starting from state $x_t$, the model: (1) predicts the vector field $v_\theta(x_t, t)$, (2) adds the distribution-preserving correction scaled by $\frac{\sigma_t^2}{2t}$, (3) multiplies by the step size $\Delta t$, (4) adds independent Gaussian noise with standard deviation $\sigma_t\sqrt{\Delta t}$. The result is the next state $x_{t+\Delta t}$, which is stochastic because of the $\epsilon$ term.

Noise schedule: The paper inherits the noise schedule from Flow-GRPO: $\sigma_t = \eta \sqrt{\frac{t}{1-t}}$, where $\eta = 0.7$ is a hyperparameter controlling the overall noise level. This schedule increases noise at intermediate timesteps (where $t$ is not too close to 0 or 1) and reduces it near the endpoints, reflecting that exploration is most useful in the middle of the denoising process where the model has flexibility.

Training Timestep Selection

Following DanceGRPO, the paper restricts GRPO training to the first half of the denoising timesteps: $M = \{T, T-1, \ldots, \lfloor T/2 \rfloor\}$. With $T = 16$, this means $M = \{16, 15, \ldots, 9\}$—eight training timesteps out of sixteen total.

Why this restriction: The paper states two reasons. First, "the exploration space of the SDE is larger and significantly determines the overall direction of the entire denoising chain" at early steps—perturbations early in the process cascade through the rest of the trajectory, affecting global composition, layout, and color palette. Second, "the improvements from GRPO in the later stages are relatively minor"—by the time the denoising process reaches the later steps, the image structure is largely determined and only fine details remain to be refined. Training only on the first half concentrates the optimization budget where it has the most impact, improving computational efficiency.

---

The Baseline GRPO Training Objective

The paper inherits the GRPO training procedure from DanceGRPO, and G2RPO modifies only how the advantages are computed—not the objective function itself. I will therefore describe the baseline objective in detail, then show exactly where G2RPO's modifications plug in.

Group Sampling

For a given prompt $c$, the flow model with the old (pre-update) policy $\pi_{\theta_{\text{old}}}$ samples a group of $G = 12$ complete denoising trajectories using the SDE formulation at every timestep. This produces $G$ distinct images $\{x^i_0\}_{i=1}^G$, all conditioned on the same prompt $c$. Each image differs from the others because of the independent Gaussian noise $\epsilon$ injected at each step.

Reward Assignment

Each generated image $x^i_0$ is evaluated by a frozen reward model $R$, which outputs a scalar score $R(x^i_0, c)$ reflecting the model's estimate of how well the image aligns with human preferences given the prompt. The paper uses either a single reward model (HPS-v2.1 for in-domain experiments) or two reward models jointly (HPS-v2.1 and CLIP Score for robust experiments), but the advantage computation structure is the same in both cases.

Group-Relative Advantage Computation

The key insight of GRPO is that the policy doesn't need absolute reward values—it only needs to know which outputs in a group are relatively better or worse than the group average. The advantage for image $i$ is computed as:

$A^i_0 = \frac{R(x^i_0, c) - \text{mean}\left(\{R(x^j_0, c)\}_{j=1}^G\right)}{\text{std}\left(\{R(x^j_0, c)\}_{j=1}^G\right)}$

where $\text{mean}(\cdot)$ and $\text{std}(\cdot)$ are computed over the group of $G$ samples for the same prompt.

What this computes: For each image in the group, subtract the group average reward (centering) and divide by the group standard deviation (scaling). An image with reward exactly at the group mean gets advantage 0. An image one standard deviation above the mean gets advantage +1. An image one standard deviation below gets advantage -1.

Why this form: The standardization serves two purposes. First, it makes the advantage scale invariant to the absolute magnitude of the reward model's scores, which can vary across prompts and reward models. Second, it creates a zero-mean, unit-variance signal within each group, which stabilizes policy gradient updates by preventing some groups from dominating the gradient due to large reward spreads. This is the core innovation of GRPO over methods that use absolute reward values or require a separately learned value function.

Uniform Advantage Broadcasting (The Problem)

In DanceGRPO and Flow-GRPO, the advantage computed from the final image is broadcast uniformly to every denoising step:

$A^i_t = A^i_0 \quad \text{for all } t = 0, 1, \ldots, T-1$

This means that step T (early structure) and step 1 (final detail) receive identical advantage signals, regardless of their actual contribution to the outcome.

Why this is problematic: Consider a scenario where the SDE perturbation at step T accidentally produces a good global composition, but the perturbation at step T-2 introduces a distracting artifact. The final image might still score decently because of the good composition, giving a moderate positive advantage—and that same moderate positive advantage would be applied to both the good step T perturbation and the bad step T-2 perturbation. The model has no way to distinguish which perturbation helped and which hurt, so it reinforces both equally.

Policy Ratio Computation

For each step $t$ in the training set $M$, and for each sample $i$ in the group, the probability ratio $r^i_t(\theta)$ compares the current policy's likelihood of the action to the old policy's likelihood:

$r^i_t(\theta) = \frac{p_\theta\left(x^i_{t-1} \mid x^i_t, c\right)}{p_{\theta_{\text{old}}}\left(x^i_{t-1} \mid x^i_t, c\right)}$

What this computes: Given the state $x^i_t$ (the noisy latent at step $t$ for sample $i$) and the observed action $x^i_{t-1}$ (the actual next state that was sampled during exploration), the ratio measures how much more or less likely the current policy $\theta$ is to produce that same action compared to the old policy $\theta_{\text{old}}$ that generated the data. A ratio > 1 means the current policy is more likely to repeat that action; a ratio < 1 means it's less likely.

Why this form: This is the standard importance sampling ratio from PPO. It enables off-policy learning—the data was generated by $\theta_{\text{old}}$, but we can evaluate (and optimize) a different policy $\theta$ by reweighting each action by this ratio. The ratio is computed in the probability space of the flow model's output distribution at each timestep, which is typically modeled as a Gaussian with mean predicted by $v_\theta$ and fixed variance.

Clipped Surrogate Objective

The GRPO loss for a single prompt $c$ and its group of samples is:

$J_{\text{Flow-GRPO}}(\theta) = \mathbb{E}_{c \sim \mathcal{C}, \{x^i\}_{i=1}^G \sim \pi_{\theta_{\text{old}}}(\cdot \mid c)} \left[ f(r, A, \theta, \varepsilon, \beta) \right]$

where the function $f$ is defined as:

$f(r, A, \theta, \varepsilon, \beta) = \frac{1}{G} \sum_{i=1}^G \frac{1}{T} \sum_{t=0}^{T-1} \left( \min\left(r^i_t(\theta) A^i_t,\; \text{clip}\left(r^i_t(\theta), 1 - \varepsilon, 1 + \varepsilon\right) A^i_t\right) - \beta D_{\text{KL}}(\pi_\theta \parallel \pi_{\text{ref}}) \right)$

What this computes, term by term:

• $\frac{1}{G} \sum_{i=1}^G$: Average over the group of $G$ samples.
• $\frac{1}{T} \sum_{t=0}^{T-1}$: Average over all $T$ denoising steps (in practice, restricted to $M$, the first half).
• $\min(r^i_t A^i_t,\; \text{clip}(r^i_t, 1-\varepsilon, 1+\varepsilon) A^i_t)$: The PPO clipped objective. If the advantage $A^i_t$ is positive (this action was good), the objective encourages increasing $r^i_t$ (making the current policy more likely to take this action), but the clip at $1+\varepsilon$ prevents $r^i_t$ from exceeding $1+\varepsilon$, which would mean the policy changed too much relative to the old policy. If the advantage is negative (this action was bad), the objective encourages decreasing $r^i_t$, but the clip at $1-\varepsilon$ prevents it from dropping below $1-\varepsilon$. The $\min$ operation selects the more conservative of the clipped and unclipped terms, ensuring the objective is a lower bound on the true improvement—this is the key PPO innovation that prevents destructively large policy updates.
• $-\beta D_{\text{KL}}(\pi_\theta \parallel \pi_{\text{ref}})$: An optional KL divergence penalty that keeps the current policy $\pi_\theta$ close to a reference policy $\pi_{\text{ref}}$ (typically the initial pretrained model). This prevents the policy from drifting too far from its starting distribution and losing general image quality.

Key parameters:

• $\varepsilon = 5$: The clipping threshold. This is unusually large compared to typical PPO implementations (which use $\varepsilon \approx 0.2$), suggesting that large policy changes are tolerated in this domain.
• $\beta = 0$: The KL penalty coefficient. Following DanceGRPO, the KL penalty is disabled because it was found to provide stable training without it for this specific model and reward setup.

Why the PPO-style objective for flow models: The clipping mechanism is essential because GRPO uses off-policy data—the samples were generated by $\theta_{\text{old}}$, but we're evaluating $\theta$, which has been updated. Without clipping, a single gradient step could make the policy radically different from the data-generating policy, invalidating the importance sampling assumption. The clipping ensures that updates are conservative enough that the importance sampling ratio remains a reliable estimate of the policy improvement.

The Optimization Goal

Maximizing $J_{\text{Flow-GRPO}}$ increases the probability of actions that received positive advantages (images better than the group average) and decreases the probability of actions that received negative advantages (images worse than the group average), while staying within a trust region around the old policy. Over many iterations, this shifts the policy toward producing higher-reward images.

---

Singular Stochastic Sampling

This is the first of G2RPO's two core innovations. It addresses the sparse reward problem by completely restructuring how stochasticity is introduced during the group sampling phase, confining SDE noise to a single timestep so that the final reward can be unambiguously attributed to that step's perturbation.

The Core Insight

The paper's key observation is that when SDE noise is injected at every step (DanceGRPO's approach), the final reward $R(x^i_0, c)$ reflects the accumulated effect of $T$ independent noise injections. The group advantage $A^i_0$ computed from this reward is then broadcast uniformly to all steps. This means that for any given step $t$, the advantage signal $A^i_t = A^i_0$ contains "contamination" from noise injected at steps other than $t$—the variation in group rewards is caused by perturbations at all steps, but the advantage is attributed to each step individually as if that step were solely responsible.

The solution is elegantly simple: only perturb one step at a time. By keeping all other steps deterministic, any variation in the final reward must be caused by the perturbation at the chosen step, creating a direct causal chain.

Step-by-Step Sampling Procedure

For each timestep $k$ in the training set $M$:

Step 1: Reach the common starting point via ODE. Starting from the initial noise $x_T \sim \mathcal{N}(0, I)$, apply $T - k$ steps of deterministic ODE sampling (Eq. 1) to reach the state $x_k$ at timestep $k$. Since ODE sampling is deterministic, given the same $x_T$ and prompt $c$, the state $x_k$ is identical for all group members:

$x_k = \text{ODE}_{T \to k}(x_T, c)$

Step 2: Apply SDE sampling at the single step $k$. At the chosen step $k$, apply the SDE formulation (Eq. 3) to generate $G = 12$ distinct next states $\{x^i_{k-1}\}_{i=1}^G$. For each group member $i$, independent Gaussian noise $\epsilon^i \sim \mathcal{N}(0, I)$ is sampled:

$x^i_{k-1} = x_k + \left(v_\theta(x_k, k) + \frac{\sigma^2_k}{2k}(x_k + (1 - k) v_\theta(x_k, k))\right) \Delta t + \sigma_k \sqrt{\Delta t}\, \epsilon^i$

The only difference between group members is the noise term $\epsilon^i$—all share the same $x_k$, the same $v_\theta(x_k, k)$, and the same drift and correction terms. The variation in $x^i_{k-1}$ is solely due to the injected noise.

Step 3: Complete the trajectory via ODE. From each $x^i_{k-1}$, apply $k - 1$ steps of deterministic ODE sampling to produce the final denoised image $x^i_{0 \leftarrow k}$. The notation $x^i_{0 \leftarrow k}$ emphasizes that this image is the result of perturbation at step $k$ specifically:

$x^i_{0 \leftarrow k} = \text{ODE}_{k-1 \to 0}(x^i_{k-1}, c)$

Since all subsequent steps are deterministic ODE, there is no additional source of variation—the entire difference between any two group members' final images traces back to the SDE noise at step $k$.

Step-Aware Precise Advantage Computation

With Singular Stochastic Sampling, the variance in the group of rewards $\{R(x^i_{0 \leftarrow k}, c)\}_{i=1}^G$ is entirely determined by the distinct denoising directions introduced at step $k$. This enables a step-specific, precise advantage:

$A^i_k = \frac{R(x^i_{0 \leftarrow k}, c) - \text{mean}\left(\{R(x^j_{0 \leftarrow k}, c)\}_{j=1}^G\right)}{\text{std}\left(\{R(x^j_{0 \leftarrow k}, c)\}_{j=1}^G\right)}$

What this computes: The same standardization as before, but now the advantage is indexed by $k$ (the step where stochasticity was injected) rather than by $0$ (the final image). The advantage $A^i_k$ tells us: "For the perturbation applied at step $k$ for sample $i$, how much better or worse was the resulting image compared to other perturbations applied at the same step $k$, starting from the same $x_k$?"

Why this is precise: Because all variation in the group rewards is caused by the perturbation at step $k$, the advantage $A^i_k$ is a clean causal signal. If sample $i$ has a high advantage, it means that specifically the direction taken from $x_k$ to $x^i_{k-1}$ was good, independent of what happened at other steps (since those were deterministic). There is no cross-step contamination.

Why this is dense: In DanceGRPO, the same advantage $A^i_0$ is used for all steps—the reward signal is "sparse" in the temporal sense because it arrives only after $T$ decisions. In G2RPO, each step $k$ in $M$ gets its own dedicated group sampling phase and its own set of advantages $A^i_k$, computed from images where the only source of variation was at step $k$. The reward signal is "dense" because every training step receives a customized, step-specific evaluation.

The Modified GRPO Objective with Singular Stochastic Sampling

The GRPO objective is updated to use the step-specific advantages $A^i_k$ and to only sum over the training timesteps $M$:

$f(r, A, \theta, \varepsilon, \beta) = \frac{1}{G} \sum_{i=1}^G \frac{1}{K} \sum_{k \in M} \left( \min\left(r^i_k(\theta) A^i_k,\; \text{clip}\left(r^i_k(\theta), 1 - \varepsilon, 1 + \varepsilon\right) A^i_k\right) \right)$

where $K = |M| = 8$ (the number of training timesteps) replaces the $T = 16$ from DanceGRPO, and the KL penalty term is omitted because $\beta = 0$.

What changed from DanceGRPO:

1. The outer average is now $\frac{1}{K}$ over $M$ rather than $\frac{1}{T}$ over all steps.
2. The advantages are $A^i_k$ (step-specific) rather than $A^i_0$ (uniformly broadcast).
3. The policy ratio $r^i_k(\theta)$ is computed at step $k$ using the probabilities at that specific step.

Computational Efficiency Benefit

The paper notes a practical advantage of Singular Stochastic Sampling: the common starting point $x_k$ enables computation reuse. Since all $G$ group members start from the same $x_k$, the flow model's prediction $v_\theta(x_k, k)$ is identical for all members. The model only needs to be evaluated once at $x_k$ to get this vector field, rather than $G$ separate times. The vector field is then reused in the SDE sampling for all $G$ members:

"each distinct sampling starting point $x^i_k$ needs to be fed into the flow model to compute the corresponding ODE denoising direction $v^i_k$. However, our sampling strategy shares a common starting point $x_k$, allowing a group of $G$ samples to reuse the same $v_k$, which in turn improves training efficiency."

What this means concretely: In DanceGRPO, at each step $t$, the model must compute $v_\theta$ for each of the $G$ samples separately because each has its own $x^i_t$ (they diverged due to previous SDE perturbations). In G2RPO, at the single SDE step $k$, the model computes $v_\theta(x_k, k)$ once and reuses it for all $G$ samples. This partially offsets the additional cost of running separate group sampling phases for each training timestep.

---

Multi-Granularity Advantage Integration (MGAI)

This is the second core innovation. Even with Singular Stochastic Sampling providing a precise, step-specific reward signal, the paper identifies a remaining problem: the evaluation of the SDE sampling direction depends on the granularity at which the subsequent ODE denoising is performed.

The Granularity Sensitivity Problem

The paper's Figure 3 provides the empirical motivation. Starting from the same $x_k$ and prompt $c$, applying the same SDE perturbation to reach $x^i_{k-1}$, the final image quality—and specifically the reward model's score—varies depending on how many ODE steps are used for the remainder of the trajectory.

The underlying mechanism: Flow models are discretized approximations of a continuous process. The number of discretization steps affects the accuracy of the ODE integration. Coarse-grained sampling (fewer steps, larger $\Delta t$) accumulates more discretization error, which can manifest as slightly different textures, edges, or color gradations compared to fine-grained sampling. A reward model like HPS-v2.1 might be sensitive to these differences—it might prefer the fine-grained version of one direction but the coarse-grained version of another, even though the underlying "goodness" of the SDE perturbation should ideally be granularity-invariant.

The consequence: If advantages are computed under a single granularity (say, standard $k-1$ step ODE), a direction that looks good at that granularity but poor at others might receive an inappropriately large positive advantage, pushing the policy in a direction that is brittle rather than robust.

The Multi-Granularity Solution

The MGAI module evaluates each SDE sampling direction multiple times, each time completing the denoising trajectory at a different granularity, and then aggregates the per-granularity advantages into a single composite evaluation.

Granularity Definition via Interval Sampling

The paper defines a set of integer scaling factors $\Lambda = \{\lambda_1, \lambda_2, \ldots, \lambda_J\}$ ($|\Lambda| = J$) where each $\lambda_j$ controls how many ODE steps are skipped between evaluations. For a given $\lambda_j$, the denoising timestep sequence $S_j$ is constructed by interval sampling from the $k-1$ available ODE steps:

$S_j = \{1, 1 + \lambda_j, 1 + 2\lambda_j, \ldots, \lfloor \frac{k-1}{\lambda_j} \rfloor \lambda_j\}$

What this computes: Starting from the step immediately after the SDE perturbation (step $k-1$ in the denoising sequence), sample every $\lambda_j$-th step. For $\lambda = 1$, every step is included: $S_1 = \{1, 2, 3, \ldots, k-1\}$ (full fine-grained denoising). For $\lambda = 2$, every other step is included: $S_2 = \{1, 3, 5, \ldots\}$. For $\lambda = 3$, every third step: $S_3 = \{1, 4, 7, \ldots\}$. The number of total denoising steps under granularity $\lambda_j$ is $N_j = \lceil \frac{k-1}{\lambda_j} \rceil$.

Why interval sampling rather than arbitrary subsets: Interval sampling ensures that the denoising process is performed at regular intervals, preserving the uniform temporal spacing of the original ODE discretization. This maintains the mathematical structure of the ODE integration—the step sizes are simply scaled by $\lambda_j$ rather than being irregular. Irregular step sizes could introduce additional integration errors unrelated to the SDE direction being evaluated.

The paper uses $\Lambda = \{1, 2, 3\}$, meaning $J = 3$ granularity levels:

• $\lambda_1 = 1$: Standard fine-grained denoising (every step).
• $\lambda_2 = 2$: Medium granularity (every other step, roughly half the steps).
• $\lambda_3 = 3$: Coarse granularity (every third step, roughly one-third the steps).

Per-Granularity Image Generation and Scoring

For each granularity $\lambda_j$, for each group member $i$, the ODE denoising proceeds from $x^i_{k-1}$ using the timestep sequence $S_j$, producing a denoised image $x^{i,j}_{0 \leftarrow k}$. The superscript $i,j$ indicates sample $i$ evaluated under granularity $j$. Each image is scored by the reward model:

$R(x^{i,j}_{0 \leftarrow k}, c)$

This yields, for each granularity $j$, a group of $G$ reward values $\{R(x^{i,j}_{0 \leftarrow k}, c)\}_{i=1}^G$.

Per-Granularity Advantage Computation

For each granularity $j$, the group advantage for sample $i$ is computed using the same standardization as before:

$A^{i,j}_k = \frac{R(x^{i,j}_{0 \leftarrow k}, c) - \text{mean}\left(\{R(x^{\ell,j}_{0 \leftarrow k}, c)\}_{\ell=1}^G\right)}{\text{std}\left(\{R(x^{\ell,j}_{0 \leftarrow k}, c)\}_{\ell=1}^G\right)}$

What this computes: $A^{i,j}_k$ is the advantage of sample $i$ at step $k$, evaluated under granularity $j$. It tells us: "Under this specific granularity, how did this sampling direction perform relative to the other directions in the group?"

Multi-Granularity Advantage Aggregation

The per-granularity advantages are summed across all $J$ granularities to form a composite advantage:

$A^{i,\text{mix}}_k = \sum_{j=1}^J A^{i,j}_k$

What this computes: The mixed advantage $A^{i,\text{mix}}_k$ is the sum of standardized advantages from all granularities. If a sampling direction consistently outperforms the group across all granularities, it will have positive $A^{i,j}_k$ for each $j$, and their sum will be large and positive. If a direction is good at one granularity but poor at another, the positive and negative contributions partially cancel, resulting in a moderate or zero mixed advantage.

Why summation rather than averaging: The paper explicitly parallels DanceGRPO's treatment of multiple reward models: "Similar to the joint training with multiple reward models in DanceGRPO, where the advantages from different reward models are directly summed to provide a multi-dimensional comprehensive evaluation of a single denoising direction, we combine the advantages from different granularities." Summation preserves the scale of each granularity's signal—a direction that is two standard deviations above the mean at both fine and coarse granularities gets a mixed advantage of approximately +4, creating a strong positive gradient. Averaging would dilute this to +2, reducing the gradient magnitude.

Why this produces a more robust evaluation: The composite advantage rewards directions that perform well across multiple resolutions. A direction that exploits a granularity-specific artifact—looking good only under coarse sampling because its structural errors are masked by integration error—will have a positive advantage at one granularity but neutral or negative at others, and the sum will be small. A direction that genuinely sets up a good image—good composition, good textures, good color balance—will tend to look good regardless of downstream resolution, and will accumulate a large positive mixed advantage.

Connection to robustness against reward hacking: The supplementary material provides evidence for this interpretation through the UniGenBench++ evaluation (Table 5). G2RPO without MGAI (single-granularity only) achieves high HPS-v2.1 but degrades on metrics like Style and Layout compared to the baseline. Adding MGAI recovers and surpasses baseline performance on these out-of-domain metrics. The interpretation: single-granularity training allows the model to overfit to reward model biases that manifest at a specific resolution, while multi-granularity training encourages learning features that are genuinely good (robust to resolution changes) and therefore generalize better.

---

The Complete G2RPO Objective

With both modifications incorporated, the final GRPO objective for G2RPO is:

$J_{\text{G2RPO}}(\theta) = \mathbb{E}_{c \sim \mathcal{C}, \{x^i\}_{i=1}^G \sim \pi_{\theta_{\text{old}}}(\cdot \mid c)} \left[ \frac{1}{G} \sum_{i=1}^G \frac{1}{K} \sum_{k \in M} \left( \min\left(r^i_k(\theta) A^{i,\text{mix}}_k,\; \text{clip}\left(r^i_k(\theta), 1 - \varepsilon, 1 + \varepsilon\right) A^{i,\text{mix}}_k\right) \right) \right]$

where:

• $r^i_k(\theta) = \frac{p_\theta(x^i_{k-1} \mid x_k, c)}{p_{\theta_{\text{old}}}(x^i_{k-1} \mid x_k, c)}$ is the policy ratio at step $k$ for sample $i$, computed using the common starting state $x_k$.
• $A^{i,\text{mix}}_k = \sum_{j=1}^J A^{i,j}_k$ is the mixed multi-granularity advantage.
• $K = |M| = 8$ is the number of training timesteps.
• $\varepsilon = 5.0$ is the clipping threshold.
• $\beta = 0$ (KL penalty disabled).

What changed from the DanceGRPO baseline, summarized:

1. Sparse → Dense rewards: $A^i_0$ (uniformly broadcast) → $A^i_k$ (step-specific). This is enabled by Singular Stochastic Sampling.
2. Single-granularity → Multi-granularity evaluation: $A^i_k$ (one granularity) → $A^{i,\text{mix}}_k = \sum_j A^{i,j}_k$ (aggregated across granularities). This is enabled by MGAI.
3. Full trajectory → Half trajectory training: $T$ steps → $K = |M|$ steps. Inherited from DanceGRPO.
4. Shared starting state efficiency: $v_\theta(x_k, k)$ computed once per group, reused for all $G$ members at step $k$.

What remains unchanged from DanceGRPO:

• The group size $G = 12$.
• The total sampling steps $T = 16$.
• The SDE formulation and noise schedule ($\eta = 0.7$, $\sigma_t = \eta\sqrt{t/(1-t)}$).
• The PPO clipped objective structure.
• The KL penalty setting ($\beta = 0$).
• The training hyperparameters: AdamW optimizer, learning rate $2 \times 10^{-6}$, weight decay $1 \times 10^{-4}$, batch size 1, mixed precision bfloat16, gradient clipping at max norm 1.0.

The Training Loop in Detail

The complete training procedure, as specified in Algorithm 1 of the paper, proceeds as follows for each training iteration:

1. Copy the current policy: $\pi_{\theta_{\text{old}}} \leftarrow \pi_\theta$. This freezes the data-generating policy for this iteration.
2. Sample a batch of prompts: $C_b \sim \mathcal{C}$ from the HPSv2 training set (103,700 prompts available, batch size = 1, so one prompt per iteration).
3. For each prompt $c \in C_b$:
   • Initialize: Sample shared initial noise $x_T \sim \mathcal{N}(0, I)$.
   • For each training timestep $k \in M$ (where $M = \{16, 15, \ldots, 9\}$):
     • ODE from $T$ to $k$: Apply $T - k$ deterministic ODE steps to reach $x_k$.
     • SDE at step $k$: Apply Singular Stochastic Sampling to generate $G = 12$ next states $\{x^i_{k-1}\}_{i=1}^G$ from the common $x_k$.
     • Multi-granularity ODE completions: For each granularity $\lambda_j \in \Lambda = \{1, 2, 3\}$, for each group member $i$, apply ODE sampling from $x^i_{k-1}$ to $x^{i,j}_0$ using interval sampling with step $\lambda_j$.
     • Compute rewards: For each granularity and each sample, evaluate $R(x^{i,j}_{0 \leftarrow k}, c)$. If using joint training, compute separate rewards from HPS-v2.1 and CLIP Score and sum the per-model advantages (as in DanceGRPO).
     • Compute per-granularity advantages: $A^{i,j}_k$ for each $j$, using group-wise standardization.
     • Compute mixed advantage: $A^{i,\text{mix}}_k = \sum_j A^{i,j}_k$.
   • Compute GRPO loss $J(\theta)$: Aggregate the clipped objective over all $i$ and all $k \in M$ using the mixed advantages.
4. Update policy: Gradient ascent on $J(\theta)$.

Training Scale: All experiments use $16 \times$ NVIDIA H200 GPUs with a batch size of 1 (one prompt per iteration, but each prompt generates $G = 12$ images across $K = 8$ timesteps and $J = 3$ granularities, yielding substantial parallelism).

Training Cost Overhead: The supplementary material (Section 11) quantifies the additional denoising steps introduced by multi-granularity sampling. For $M = \{16, 15, \ldots, 9\}$, single-granularity requires $S_1 = \sum_{m \in M} m = 16 + 15 + \cdots + 9 = 100$ ODE completion steps. The additional granularities add $S_2 = \sum_{m \in M} \lfloor m/2 \rfloor$ and $S_3 = \sum_{m \in M} \lfloor m/3 \rfloor$, with total $T_{\{1,2,3\}} = S_1 + S_2 + S_3 = 184$ and relative overhead $\Delta_{\{1,2,3\}} \approx 45.7\%$. This cost is incurred only during training; inference uses standard deterministic ODE sampling with no additional overhead.

4. Key Insights and Innovations

Innovation 1: Temporal Credit Assignment as the Bottleneck in Flow Model RL — Not Exploration Strategy

The paper's most fundamental contribution is a diagnostic reframing: it identifies that the central bottleneck in flow-based GRPO is not how much exploration occurs (the noise magnitude, controlled by η) or even where it occurs (early vs. late steps, addressed by DanceGRPO's training on M), but rather the inability to assign credit for exploration outcomes to specific decisions. This is a conceptual shift that moves the conversation from "how do we get good samples to compare?" (the question Flow-GRPO and DanceGRPO answered) to "once we have variation, how do we know which variation caused the good outcome?"

Before this work, the dominant assumption in flow-based GRPO — embedded in both Flow-GRPO and DanceGRPO — was that uniform advantage broadcasting across all timesteps was acceptable because the SDE noise at each step contributes to the final image and therefore each step "deserves" a share of the reward signal. This paper demonstrates that this assumption, while mathematically convenient, is information-theoretically impoverishing: the uniform broadcast means that for any given step t, the advantage signal A_t^i contains variance caused by perturbations at all other steps, making it impossible to determine whether the perturbation at step t actually helped or hurt. The consequence is gradient updates that are diluted, noisy, and potentially misdirected.

The diagnostic move is captured in the paper's Figure 1(b) and formalized in Section 3.2, but the intellectual contribution lies in recognizing this as a temporal credit assignment problem — a well-studied challenge in reinforcement learning — rather than as an unavoidable property of SDE-based exploration. The paper's language connects this explicitly to the broader RL literature: "mirroring the sparse-reward problem that plagues RLHF" (Section 1). By naming the problem precisely, the paper opens the door to solutions drawn from the credit assignment toolkit rather than from the noise-schedule-tuning toolkit that prior work had been exploring.

This is a fundamental reframing, not an incremental refinement. Prior work (DanceGRPO, MixGRPO) improved efficiency and robustness within the uniform-broadcasting paradigm — they made the existing approach work better without questioning whether the signal structure itself was sound. G2RPO argues that the signal structure is fundamentally compromised and that fixing it requires restructuring how rewards map to actions, not just tuning parameters within the existing structure. The evidence supporting this claim is the 6.52% relative improvement on in-domain HPS-v2.1 from Singular Stochastic Sampling alone (G2RPO w/o MGAI vs. DanceGRPO in Table 1, HPS-only training), which represents the gain achievable purely through precise credit assignment, before any multi-granularity enhancement.

---

Innovation 2: Robustness to Downstream Resolution as a Signal of Genuine Quality

The second conceptual contribution is the identification that a sampling direction's quality should be invariant to the granularity at which the downstream trajectory is evaluated, and that enforcing this invariance via multi-granularity advantage aggregation produces evaluation signals that generalize better to out-of-domain reward models. This is a diagnostic insight about what makes a "good" evaluation of an intermediate denoising step, and it represents a new axis of analysis that prior work had not considered.

Prior work evaluated each sampling direction under a single, fixed denoising schedule — the same number of ODE steps for every sample, every time. The implicit assumption was that the evaluation granularity was a neutral implementation detail that didn't affect the relative ranking of directions within a group. The paper's Figure 3 empirically disproves this assumption: the same SDE direction can be the best in its group under one granularity and merely average under another. The contribution is not the observation itself (which could be dismissed as reward model noise) but the interpretation that this granularity sensitivity indicates an incomplete evaluation — a direction that only looks good at one resolution has not been adequately assessed.

This connects to a broader principle in robust optimization: if a property (here, the relative quality of a sampling direction) is genuinely meaningful, it should be stable under reasonable variations in evaluation protocol. The Multi-Granularity Advantage Integration module operationalizes this principle by forcing each direction to "prove itself" at multiple resolutions before receiving a strong advantage signal. Directions that exploit granularity-specific artifacts get inconsistent evaluations and their aggregated advantage cancels out; directions that represent genuine quality improvements get consistently positive evaluations across granularities and accumulate large positive mixed advantages.

What makes this intellectually distinctive is that it treats evaluation granularity as a first-class variable in the RL training process, analogous to how DanceGRPO treated reward model choice as a first-class variable. The deliberate parallel the paper draws — summing advantages across granularities just as DanceGRPO sums advantages across reward models — signals that this is not an ad-hoc trick but a principled extension of the multi-dimensional evaluation philosophy from the reward-model axis to the denoising-resolution axis. The significance is supported by the UniGenBench++ results (Supplementary Table 5), where G2RPO without MGAI degrades on Style and Layout (suggesting granularity-specific overfitting) while full G2RPO surpasses baselines on these metrics, demonstrating that multi-granularity evaluation provides robustness that single-granularity evaluation cannot.

This is a fundamental conceptual contribution rather than an incremental improvement. It introduces a new evaluation dimension (denoising granularity) that had never been varied in prior work and establishes an empirical link between multi-granularity evaluation and out-of-domain generalization. The implication extends beyond this specific method: future work on flow model alignment should treat downstream evaluation resolution as a design choice that affects evaluation quality, not as a fixed constant.

---

Innovation 3: Isolating Exploration to Individual Steps as a General-Purpose Credit Assignment Strategy for Sequential Generative Models

While Singular Stochastic Sampling appears at first glance to be a simple mechanism — just turn off SDE noise at all but one step — the intellectual contribution lies in recognizing that this creates a general template for converting sparse-reward RL problems into dense-reward problems in any sequential generative model with deterministic intermediates. The paper doesn't claim this generality explicitly, but the structure of the solution implies it: if you have a generative process that can be decomposed into a sequence of transformations, and you can make any subset of those transformations deterministic while keeping others stochastic, then confining stochasticity to a single step yields a clean causal attribution between perturbation and outcome.

This matters because prior approaches to the credit assignment problem in diffusion/flow RL had explored fundamentally different directions. DDPO (Black et al., 2023) used per-step reward predictions from learned value functions. Step-aware preference optimization (Liang et al., 2024) attempted to assign per-step preferences. These approaches add complexity — additional models, additional training objectives, additional hyperparameters. Singular Stochastic Sampling achieves dense credit assignment with zero additional learned components: it restructures the sampling procedure so that the reward signal becomes step-specific by construction, not by learning. This is a conceptual (and practical) simplification relative to the value-function approach.

The specific advantage of this strategy for flow models is that it exploits a unique property: the deterministic ODE provides a principled, distribution-preserving way to reach intermediate states without variation, meaning the "common starting point" $x_k$ is genuinely the state the model would deterministically reach from the initial noise. This is not true in general for arbitrary sequential processes — it relies on the existence of a deterministic integrator that preserves the marginal distribution, which flow models happen to provide via their ODE formulation. This means the technique is both general in principle (apply to any process with a deterministic mode) and specific in its current instantiation (enabled by the ODE/SDE duality unique to diffusion and flow models).

The significance beyond raw performance: this innovation shows that careful restructuring of the data collection process can substitute for algorithmic complexity in RL. Rather than building a better credit assignment algorithm, G2RPO builds a better credit assignment situation — one where the causal structure of the data makes the assignment trivial. This is a design philosophy that could influence how RL is applied to other structured generative processes beyond image generation.

The evidence supporting this as more than an incremental change: the improvement from Singular Stochastic Sampling alone (without MGAI) is substantial across multiple metrics (Table 1, G2RPO w/o MGAI rows), demonstrating that precise credit assignment is independently valuable regardless of multi-granularity evaluation. The fact that it works with the same SDE formulation, same noise schedule, same group size, and same optimizer as DanceGRPO — purely by restructuring when noise is applied — underscores that this is an architectural insight about experimental design in RL, not a hyperparameter tuning achievement.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All experiments use the HPSv2 prompt dataset (Wu et al., 2023), which contains 103,700 text prompts for training and 400 diverse prompts for testing. The paper explicitly follows the same setting as DanceGRPO and MixGRPO in this choice.

• Base model. The text-to-image model is Flux.1-dev (Black Forest Labs, 2024), a leading rectified flow transformer in the community. The paper describes it as representative of state-of-the-art flow-based generation capability, and it serves as both the initial policy and the reference policy for KL regularization (though β = 0 in practice).

• Metrics. The paper evaluates using five reward models that assess alignment with human preferences from multiple dimensions: HPS-v2.1 (HPS), CLIP Score (CLIP), Pick Score (PS), Image Reward (IR), and Unified Reward (UR). HPS-v2.1 and CLIP Score serve as training reward models (in-domain when used for training, out-of-domain otherwise), while Pick Score, Image Reward, and Unified Reward are exclusively out-of-domain evaluation metrics. Additionally, the supplementary material evaluates on UniGenBench++, a benchmark with 10 fine-grained evaluation dimensions covering semantic evaluation, image quality, text alignment, and other aspects, scored by the UniGenBench-EvalModel-qwen-72b-v1 VLM.

• Baselines. Three baselines are compared: (1) Flux.1-dev — the base pretrained model without any RL fine-tuning, providing the lower bound; (2) DanceGRPO (Xue et al., 2025) — the primary flow-based GRPO baseline that injects SDE noise at all timesteps and trains with uniform advantage broadcasting; (3) MixGRPO (Li et al., 2025) — a more recent method that improves training efficiency through hybrid ODE-SDE sampling while maintaining comparable performance. Flow-GRPO (Liu et al., 2025) is mentioned but excluded from comparison because "it employs a similar ODE-to-SDE conversion across all timesteps as contemporary DanceGRPO, but [fits] a specific reward target" (Section 4.2). Additionally, the paper includes an internal ablation baseline: G2RPO w/o MGAI — Singular Stochastic Sampling without multi-granularity denoising, isolating the contribution of precise credit assignment.

• Generation budget / compute accounting. The training budget is measured in denoising steps, with all methods using total sampling steps T = 16 and group size G = 12. The paper quantifies the additional training cost of multi-granularity sampling in the supplementary material (Section 11): for the training timestep set M = {16, 15, ..., 9}, single-granularity requires S₁ = 100 ODE completion steps, while tri-granularity (Λ = {1, 2, 3}) requires T_{1,2,3} = 184 steps, representing approximately 45.7% additional training-time denoising steps. Inference latency is unchanged across all methods since they all use standard ODE sampling at inference time.

• Cross-validation / statistical protocol. No explicit cross-validation or statistical significance testing is reported. The paper evaluates on a fixed test set of 400 prompts from HPSv2 and reports aggregate scores across all test prompts for each metric. The supplementary evaluation on UniGenBench++ serves as an additional out-of-domain robustness check but does not constitute a formal statistical protocol. The main paper does not report confidence intervals or standard errors for any metric.

Main Quantitative Results

Single-Reward Training: In-Domain Performance Ceiling

The paper first evaluates the upper limit of in-domain performance by training exclusively with HPS-v2.1 as the reward model (Table 1, top block, "HPS" rows). This setting is acknowledged to be prone to reward hacking — the model can exploit biases in HPS-v2.1 to achieve high scores without genuine quality improvement — but it isolates the maximum achievable gain from precise credit assignment.

Headline numbers (HPS-only training, Table 1):

• DanceGRPO: HPS 0.353, CLIP 0.375, PS 0.228, IR 1.233, UR 3.548.
• MixGRPO: HPS 0.378, CLIP 0.358, PS 0.225, IR 1.266, UR 3.421.
• G2RPO w/o MGAI: HPS 0.376, CLIP 0.351, PS 0.228, IR 1.286, UR 3.469.
• G2RPO (full): HPS 0.385, CLIP 0.355, PS 0.229, IR 1.313, UR 3.487.

Key findings:

Singular Stochastic Sampling alone provides a 6.52% relative improvement on in-domain HPS-v2.1 over DanceGRPO (0.376 vs. 0.353). This is the gain attributable purely to precise per-step credit assignment, before any multi-granularity enhancement. However, consistent with the known HPS-v2.1 reward hacking problem, the CLIP Score degrades from 0.375 (DanceGRPO) to 0.351 (G2RPO w/o MGAI), suggesting that optimizing solely for HPS-v2.1 with precise signals causes the model to more efficiently exploit HPS-specific biases at the expense of other quality dimensions.

Adding MGAI further improves HPS-v2.1 to 0.385 — an additional 2.4% relative gain over G2RPO w/o MGAI, and a 9.1% total improvement over DanceGRPO. More importantly, MGAI partially mitigates the CLIP Score degradation (0.355 vs. 0.351 without MGAI), though it remains below DanceGRPO's 0.375, confirming that HPS-only training inherently trades off other quality dimensions regardless of evaluation strategy.

On out-of-domain metrics under HPS-only training: Image Reward increases from 1.233 (DanceGRPO) to 1.313 (G2RPO) — a 6.5% relative improvement, suggesting that precise credit assignment and multi-granularity evaluation provide some generalization even when trained on a single potentially-hackable reward model. Unified Reward shows mixed results: 3.548 (DanceGRPO), 3.469 (G2RPO w/o MGAI), 3.487 (G2RPO full). The G2RPO full value is slightly below DanceGRPO, indicating that HPS-only training's reward hacking problem is not fully resolved by the proposed methods when evaluated by the most comprehensive reward model.

MixGRPO achieves higher HPS-v2.1 (0.378) than G2RPO w/o MGAI (0.376) under HPS-only training, which is notable since MixGRPO was designed primarily for efficiency rather than precision. However, full G2RPO (0.385) surpasses MixGRPO, and MixGRPO's CLIP Score (0.358) is comparable to G2RPO's (0.355), suggesting the efficiency-focused hybrid ODE-SDE approach achieves a different tradeoff point that competes with precision-focused credit assignment on in-domain metrics but may lack the same generalization benefits.

Joint Multi-Reward Training: Robust Performance

The primary experimental setting — and the one the paper recommends for practical use — jointly trains with both HPS-v2.1 and CLIP Score as reward models to prevent reward hacking and achieve stable, robust improvements. This follows the established practice from DanceGRPO.

Headline numbers (HPS & CLIP joint training, Table 1, bottom block):

• DanceGRPO: HPS 0.331, CLIP 0.389, PS 0.227, IR 1.128, UR 3.569.
• MixGRPO: HPS 0.363, CLIP 0.399, PS 0.230, IR 1.436, UR 3.661.
• G2RPO w/o MGAI: HPS 0.372, CLIP 0.395, PS 0.234, IR 1.421, UR 3.688.
• G2RPO (full): HPS 0.376, CLIP 0.406, PS 0.235, IR 1.483, UR 3.783.

Key findings:

On in-domain metrics: G2RPO achieves the highest scores on both training reward models — HPS 0.376 (12.7% relative improvement over DanceGRPO's 0.331) and CLIP 0.406 (4.4% relative improvement over DanceGRPO's 0.389). This is notable because joint training requires balancing two potentially competing objectives, and the fact that G2RPO improves both simultaneously suggests that precise credit assignment and multi-granularity evaluation produce optimization signals that are genuinely better aligned with both reward models, rather than exploiting tradeoffs between them.

On out-of-domain metrics, the improvements are substantial and consistent:

• Image Reward: 1.483 (G2RPO) vs. 1.128 (DanceGRPO) — a 31.5% relative improvement. This is the largest relative gain across all metrics. Image Reward focuses on visual quality and aesthetic appeal, suggesting MGAI's multi-scale evaluation particularly benefits these dimensions.
• Unified Reward: 3.783 (G2RPO) vs. 3.569 (DanceGRPO) — a 6.0% relative improvement. Unified Reward leverages LLM-based visual understanding for comprehensive evaluation, and the gain here is particularly significant because it represents improvement on the most sophisticated out-of-domain metric, suggesting that the benefits of G2RPO are not merely exploiting superficial correlations that simpler reward models might share.
• Pick Score: 0.235 (G2RPO) vs. 0.227 (DanceGRPO) — a 3.5% relative improvement. The gain is smaller than for IR and UR, which may reflect Pick Score measuring dimensions that are less sensitive to the credit assignment and granularity issues that G2RPO addresses.

The contribution of MGAI under joint training is pronounced for out-of-domain metrics:

• Image Reward: 1.421 (w/o MGAI) → 1.483 (with MGAI), a 4.4% relative gain.
• Unified Reward: 3.688 (w/o MGAI) → 3.783 (with MGAI), a 2.6% relative gain.
• CLIP Score: 0.395 (w/o MGAI) → 0.406 (with MGAI), a 2.8% relative gain.

These gains from MGAI are substantially larger under joint training than under HPS-only training (where MGAI's contribution to IR was 1.286 → 1.313, or 2.1%). This suggests that multi-granularity evaluation is particularly valuable when the reward signal is already somewhat robust (joint training reduces hacking) — it provides fine-grained signal refinement that becomes visible only when the coarse-grained signal is not dominated by hacking.

Comparison with MixGRPO: G2RPO outperforms MixGRPO on all metrics under joint training. The gaps are largest on Image Reward (1.483 vs. 1.436, a 3.3% relative improvement) and CLIP Score (0.406 vs. 0.399, a 1.8% relative improvement). Since MixGRPO maintains comparable or better efficiency than DanceGRPO through hybrid sampling, this comparison demonstrates that G2RPO's gains are not achievable through efficiency improvements alone — the precision and comprehensiveness of the evaluation signal matter independently of computational cost.

The base Flux.1-dev model's scores (HPS 0.305, CLIP 0.388, PS 0.226, IR 1.040, UR 3.621) provide context for the magnitude of RL fine-tuning gains. DanceGRPO already provides substantial improvements over the base model (e.g., IR from 1.040 to 1.128, UR from 3.621 to 3.569 — interestingly, UR slightly decreases). G2RPO amplifies these gains: IR improves by 42.6% over the base model (1.040 → 1.483), while UR improves by 4.5% (3.621 → 3.783).

Qualitative Results

Figure 4 in the main paper and Figures 6–8 in the supplementary material provide visual comparisons across methods. The paper highlights several specific qualitative improvements:

• Expression and detail fidelity: In the chess piece example (Figure 4, second column), G2RPO "faithfully captures the specified expressions and even the nuances of the chess pieces as described in the prompt, delivering finer details and higher visual quality." The "proud" black king and "sad" white king — specific emotional attributes in the prompt — are rendered more distinctly than in baselines.

• Spatial and compositional adherence: In the poster example (Figure 4, last column), G2RPO "not only adheres to the spatial requirement of a clear left-right demarcation but also renders the reflections of the trees with remarkable clarity." This is notable because spatial reasoning (left vs. right, specific layout constraints) is a known weakness of text-to-image models, and the improvement suggests that precise credit assignment helps the model learn to respect compositional constraints that span multiple denoising steps.

• Style consistency: The poster example also shows that G2RPO's "overall style... is more consistent with the aesthetic demands of poster design," indicating that multi-granularity evaluation may help the model maintain global stylistic coherence rather than overfitting to local texture details that reward models might favor at specific granularities.

The supplementary material's Figure 5 provides a visual ablation of the MGAI module, comparing G2RPO with and without multi-granularity integration across eight prompt-seed pairs. The paper interprets these comparisons as showing that without MGAI, single-granularity denoising "is easily biased: the reward model over-attends to fine details while ignoring coarse structural coherence, so samples with higher reward frequently exhibit distorted textures or global misalignment." With MGAI, images show improved joint optimization of detail fidelity and global layout.

UniGenBench++ Comprehensive Evaluation

Table 5 in the supplementary material reports results on UniGenBench++, a benchmark with 10 fine-grained dimensions. This evaluation is significant because it uses a VLM-based evaluator (UniGenBench-EvalModel-qwen-72b-v1) that is completely independent of the training reward models, providing the strongest test of generalization.

Headline overall scores (Supplementary Table 5):

• Flux.1-dev: 61.59
• DanceGRPO: 66.06
• MixGRPO: 66.60
• G2RPO w/o MGAI: 66.32
• G2RPO (full): 69.21

Key findings from UniGenBench++:

G2RPO achieves the highest overall score (69.21), substantially ahead of DanceGRPO (66.06) and MixGRPO (66.60). The 3.15-point absolute improvement over MixGRPO and 3.15-point improvement over DanceGRPO represents roughly a 4.7% relative gain, which is substantial on a comprehensive benchmark.

The dimension-level breakdown reveals where G2RPO excels and where it doesn't:

• Strong gains: Attribute (79.91 vs. 73.72 DanceGRPO, +6.19 points), Relation (78.17 vs. 75.38, +2.79 points), Compound (70.36 vs. 64.95, +5.41 points), Text (46.55 vs. 34.77, +11.78 points), Logic Reasoning (42.66 vs. 38.76, +3.90 points), World Knowledge (89.08 vs. 87.82, +1.26 points).
• Modest gains or comparable: Action (71.10 vs. 68.82, +2.28 points), Grammar (58.82 vs. 59.63, −0.81 points, slightly below DanceGRPO).
• Degradation then recovery under MGAI: Style (DanceGRPO 76.00, G2RPO w/o MGAI 73.70, G2RPO 76.20) and Layout (DanceGRPO 80.78, G2RPO w/o MGAI 76.68, G2RPO 79.29) both show the pattern where Singular Stochastic Sampling alone degrades performance below DanceGRPO, but adding MGAI recovers to near or above DanceGRPO levels.

The pattern of "degradation then recovery" on Style and Layout is particularly informative. It supports the paper's interpretation that single-granularity precise credit assignment can cause the model to overfit to reward model biases, "collapsing into a narrower domain" (as the supplementary text describes), and that multi-granularity evaluation counteracts this collapse by requiring consistency across resolutions. The fact that G2RPO slightly underperforms DanceGRPO on Layout (79.29 vs. 80.78) even with MGAI suggests that the recovery is partial — some dimensions may be inherently in tension with the optimization objectives.

The Text dimension shows the largest absolute gain (+11.78 points over DanceGRPO). This is notable because text rendering in generated images is a known challenge for diffusion and flow models, requiring precise spatial coordination across the entire denoising trajectory. The large improvement suggests that G2RPO's per-step credit assignment particularly benefits tasks requiring coordinated decisions across multiple spatial locations, consistent with the credit assignment interpretation.

Robustness to Inference Step Count

Table 3 evaluates all methods at reduced inference timesteps (10 and 20 steps) under joint HPS-v2.1 + CLIP training. This experiment tests whether the benefits of G2RPO persist when the inference-time denoising schedule differs from the training-time schedule (which used 16 steps).

Headline numbers at 10 inference steps (Table 3, top block):

• Flux.1-dev: HPS 0.289, CLIP 0.388, PS 0.225, IR 0.939, UR 3.504
• DanceGRPO: HPS 0.325, CLIP 0.390, PS 0.227, IR 1.129, UR 3.576
• MixGRPO: HPS 0.358, CLIP 0.401, PS 0.230, IR 1.431, UR 3.641
• G2RPO: HPS 0.378, CLIP 0.408, PS 0.235, IR 1.519, UR 3.805

Headline numbers at 20 inference steps (Table 3, bottom block):

• Flux.1-dev: HPS 0.300, CLIP 0.389, PS 0.226, IR 1.034, UR 3.575
• DanceGRPO: HPS 0.329, CLIP 0.388, PS 0.228, IR 1.136, UR 3.586
• MixGRPO: HPS 0.363, CLIP 0.401, PS 0.230, IR 1.430, UR 3.651
• G2RPO: HPS 0.376, CLIP 0.407, PS 0.235, IR 1.511, UR 3.806

Key findings:

G2RPO maintains consistent performance across inference step counts. At 10 steps: HPS 0.378, IR 1.519, UR 3.805. At 20 steps: HPS 0.376, IR 1.511, UR 3.806. At 16 steps (from Table 1): HPS 0.376, IR 1.483, UR 3.783. The metrics are remarkably stable, with UR actually slightly higher at 10 and 20 steps than at 16 steps (3.805 and 3.806 vs. 3.783). This stability supports the paper's claim that the MGAI module "provides a more comprehensive assessment" and "exhibits stronger robustness to varying denoising step configurations."

The performance gap between G2RPO and baselines widens at lower step counts. At 10 steps, G2RPO's Image Reward (1.519) exceeds MixGRPO (1.431) by 6.2% and DanceGRPO (1.129) by 34.5%. At 20 steps, the gaps are 5.7% and 33.0% respectively. At 16 steps (Table 1), the gaps were 3.3% and 31.5%. This suggests that G2RPO's multi-granularity training — which exposed the model to coarse-grained (few-step) evaluation during training — provides particular benefits when inference uses fewer steps, since the model has learned to produce samples that look good even under coarse discretization.

The base model degrades significantly at 10 steps: IR drops from 1.040 (16 steps) to 0.939 (10 steps). DanceGRPO partially mitigates this (1.129 vs. 0.939 base), and G2RPO nearly fully compensates (1.519 vs. even the 16-step base model's 1.040), representing a 61.8% improvement over the 10-step base model on Image Reward.

Ablation Studies and Robustness Checks

Multi-granularity set composition (Table 2): The paper ablates different subsets of the granularity set Λ under joint HPS-v2.1 + CLIP training. With Λ = {1} (single granularity, equivalent to G2RPO w/o MGAI): HPS 0.372, CLIP 0.395, PS 0.234, IR 1.421, UR 3.688. Adding a second granularity: Λ = {1, 2} achieves HPS 0.375, CLIP 0.404, PS 0.234, IR 1.468, UR 3.759. Λ = {1, 3} achieves HPS 0.378, CLIP 0.404, PS 0.234, IR 1.465, UR 3.760. The full Λ = {1, 2, 3} achieves HPS 0.376, CLIP 0.406, PS 0.235, IR 1.483, UR 3.783. The finding is that Λ = {1, 2} and Λ = {1, 3} both achieve near-identical performance to Λ = {1, 2, 3} on most metrics — IR is 1.468 and 1.465 vs. 1.483, UR is 3.759 and 3.760 vs. 3.783 — while being computationally cheaper (fewer additional denoising steps). The full three-granularity set provides marginal gains on IR and UR at the cost of additional training overhead, but the two-granularity configurations capture most of the benefit.

G2RPO without MGAI (Tables 1, 2, and Supplementary Table 5): This ablation isolates the contribution of Singular Stochastic Sampling alone. Under HPS-only training (Table 1): HPS improves from 0.353 (DanceGRPO) to 0.376, but CLIP degrades from 0.375 to 0.351, UR degrades from 3.548 to 3.469. Under joint training (Table 1): all metrics improve over DanceGRPO — HPS 0.372 vs. 0.331, IR 1.421 vs. 1.128, UR 3.688 vs. 3.569. On UniGenBench++ (Supplementary Table 5): overall score improves from 66.06 (DanceGRPO) to 66.32, with gains on Attribute (+1.81), Relation (+1.90), and Text (+6.32) but degradations on Style (−2.30) and Layout (−4.10). The conclusion is that precise credit assignment alone provides meaningful improvements but introduces a tendency toward domain collapse (overfitting to reward model biases at the expense of stylistic and layout diversity), which MGAI partially counteracts.

Reward model hacking in HPS-only training (Table 1, top block): When training solely with HPS-v2.1, all methods show the characteristic pattern of reward hacking: HPS-v2.1 scores increase substantially over the base model (0.305 → 0.353–0.385) while other metrics degrade or show minimal improvement. DanceGRPO's CLIP drops from 0.388 (base) to 0.375. G2RPO's CLIP drops further to 0.355. Unified Reward drops from 3.621 (base) to 3.548 (DanceGRPO) and 3.487 (G2RPO). This confirms that HPS-v2.1 is "prone to model hacking due to biases in the training set" (Section 4.1), and that — critically — G2RPO does not solve the reward hacking problem; if anything, precise credit assignment can accelerate hacking by providing more efficient gradient signals toward reward model exploits. The mitigation is joint training with multiple reward models, not any mechanism within G2RPO itself.

Robustness to inference step count (Table 3): Already discussed in the Main Results section above. The key ablation finding is that G2RPO's performance is remarkably stable across 10, 16, and 20 inference steps, while baselines show more degradation at low step counts. At 10 steps, G2RPO's IR (1.519) is higher than at 16 steps (1.483), which is counterintuitive — typically, fewer steps reduce quality. This suggests the multi-granularity training has taught the model to produce samples robust to discretization error, potentially because coarse-grained evaluation during training (λ = 2, 3) exposed the model to exactly the types of artifacts that arise from fewer-step inference.

Comparison of MGAI on different evaluation types (Tables 1 and 5, Supplementary Table 5): The benefit of MGAI is larger on out-of-domain, VLM-based metrics (Unified Reward, UniGenBench++) than on training-domain metrics (HPS-v2.1, CLIP Score) or simpler out-of-domain metrics (Pick Score, Image Reward). Under joint training, MGAI improves UR by 2.6% (3.688 → 3.783) and UniGenBench++ overall by 4.4% (66.32 → 69.21), but improves Pick Score by only 0.4% (0.234 → 0.235) and HPS by 1.1% (0.372 → 0.376). This pattern supports the interpretation that MGAI provides robustness against granularity-specific overfitting, and this robustness is most detectable when evaluation captures dimensions (style, layout, logical consistency) that are vulnerable to such overfitting.

Critical Assessment

Does the paper demonstrate that G2RPO's gains come from the claimed mechanisms (precise credit assignment and comprehensive evaluation)?

The evidence is strong but circumstantial. The ablation comparing G2RPO with and without MGAI (Table 1, Supplementary Table 5) shows that adding multi-granularity integration consistently improves performance, and the qualitative interpretations (Figure 5, supplementary text about domain collapse) are plausible. However, the paper does not provide direct mechanistic evidence that the improvements are caused by the specific claimed mechanisms rather than, say, the additional compute from multi-granularity sampling (more denoising steps = more gradient signal per training iteration) or the regularization effect of evaluating at multiple resolutions (which might help regardless of whether it makes evaluation "comprehensive").

A direct test of the credit assignment claim would be: compare Singular Stochastic Sampling against an alternative that also provides dense per-step rewards but through a different mechanism (e.g., a learned value function or step-wise reward prediction). If both approaches achieve similar gains, the credit assignment interpretation is validated. If only Singular Stochastic Sampling works, the mechanism might be the specific structure of ODE-SDE-ODE trajectories rather than credit assignment per se. The paper does not include such a comparison.

A direct test of the multi-granularity claim would be: compare MGAI against an alternative that provides a similar amount of additional compute and regularization but without varying granularity (e.g., evaluating each direction multiple times at the same granularity with different ODE solver settings, or simply taking more samples per granularity). If MGAI outperforms such a baseline, the multi-resolution aspect is specifically valuable; if not, the benefit may come from the increased evaluation budget rather than the multi-granularity structure. The paper does not include such a comparison.

Does the paper demonstrate that G2RPO outperforms baselines in a statistically meaningful way?

No. The paper reports aggregate scores across the 400-prompt test set with no confidence intervals, standard deviations, or statistical tests. Given that the absolute differences between methods are often small (e.g., Pick Score: 0.235 vs. 0.234 vs. 0.230) and the test set is moderate in size, it is unclear whether some of the reported improvements are statistically significant or within the range of random variation. The paper would be strengthened by providing per-prompt score distributions, bootstrap confidence intervals for the mean differences, or any form of uncertainty quantification.

Does the paper's evaluation setting match the claimed generalization benefits?

Partially. The paper claims that MGAI provides "comprehensive evaluation" and robustness, and supports this with out-of-domain metrics (Image Reward, Unified Reward, UniGenBench++). The UniGenBench++ evaluation is particularly strong because it uses a completely independent VLM evaluator on diverse prompts. However, all training is done on the HPSv2 training set (103,700 prompts), and testing is on the HPSv2 test set (400 prompts) or UniGenBench++ prompts. The paper does not evaluate on prompts from a completely different distribution (e.g., DrawBench, PartiPrompts, or challenging compositional benchmarks like T2I-CompBench), which would more rigorously test whether G2RPO's improvements are robust to distribution shift in the prompt domain, not just the evaluator domain.

Are there missing experiments that would significantly strengthen the paper?

Several experiments are notably absent:

• Training curves and convergence analysis: Figure 1(a) shows reward curves for G2RPO vs. DanceGRPO during training, but this is presented only as a small inset illustration with no detailed discussion. The paper does not report how many training iterations were run, whether methods converge at different rates, or what the computational wall-clock time is. Given the 45.7% additional training overhead, understanding whether G2RPO requires fewer iterations to reach the same performance (and thus partially or fully offsets the per-iteration cost) is critical for practical adoption.

• Ablation on the noise level η: The paper inherits η = 0.7 from DanceGRPO without ablation. With Singular Stochastic Sampling, the optimal noise level might differ from the full-trajectory SDE setting, since the noise at a single step has a different effect on the final image than noise accumulated across all steps. Exploring η would strengthen the claim that G2RPO's gains are structural rather than tuning-dependent.

• Ablation on the training timestep set M: The paper inherits M = {16, ..., 9} from DanceGRPO. Since G2RPO fundamentally changes how steps receive reward signals, the optimal set of training timesteps might differ. Late steps might benefit more from precise credit assignment than they do from uniform broadcasting, potentially making M larger beneficial. The paper does not explore this.

• Comparison with non-GRPO alignment methods: The paper compares only against GRPO-based methods (DanceGRPO, MixGRPO). Comparisons against DPO-based methods (Diffusion-DPO) or PPO-based methods (DDPO) would contextualize whether GRPO + G2RPO is competitive with or superior to fundamentally different alignment paradigms.

• Human evaluation: All evaluation is through automated reward models and VLM judges. While Unified Reward and UniGenBench++ are sophisticated, there is no human preference study to validate that the reported metric improvements correspond to human-perceivable quality differences. This is a significant gap given that the claimed contribution is improved "alignment with human preferences."

Are the experimental conditions fair to baselines?

The hyperparameters (group size G = 12, total steps T = 16, noise level η = 0.7, AdamW settings) are held constant across all methods, which is appropriate for controlled comparison. However, DanceGRPO and MixGRPO were designed for the full-trajectory SDE setting, and it is possible that their performance could be improved with hyperparameter tuning specific to the G2RPO evaluation framework (e.g., different group sizes, different noise levels, different training step selections). The paper does not report whether baselines were re-tuned for the specific experimental conditions or used off-the-shelf configurations.

Does the paper's central claim — that G2RPO provides "fine-grained and comprehensive evaluation" — match what was actually demonstrated?

The paper demonstrates that G2RPO achieves higher reward scores across multiple metrics, which is consistent with the claimed mechanisms but does not directly validate them. A method could achieve higher scores through other means (more compute, better regularization, different gradient signal structure) without providing genuinely more "comprehensive" or "precise" evaluation. The paper would be strengthened by diagnostic experiments that specifically test the mechanistic claims: for example, showing that the per-step advantages A^i_k from Singular Stochastic Sampling have higher variance across steps than the uniform A^i_0 from DanceGRPO (indicating more differentiated per-step signals), or showing that the per-granularity advantages A^{i,j}_k exhibit partial disagreement (some directions look good at fine granularity but poor at coarse, and vice versa) that is resolved by summation (indicating that multi-granularity integration is doing meaningful reconciliation rather than just averaging correlated signals).

How do the results scale with compute and model size?

All experiments use a single model (Flux.1-dev) at a single scale. The paper does not investigate whether the benefits of G2RPO increase, decrease, or remain constant as model size increases. Given that larger models may have different exploration characteristics (potentially needing less SDE noise to explore meaningfully, or having different sensitivity to discretization error), the generalizability to other model scales is unknown. Similarly, the paper does not report how performance varies with total training iterations or whether compute-optimal early stopping would change the reported comparisons.

6. Limitations and Trade-offs

The Multi-Granularity Sampling Overhead Is Not Factored Into Headline Gains

The assumption or constraint: The MGAI module requires running additional ODE denoising completions at multiple granularities during training—specifically, the paper quantifies this in the supplementary material (Section 11) as approximately 45.7% more denoising steps relative to single-granularity training ($T_{\{1,2,3\}} = 184$ vs. $S_1 = 100$ for $M = \{16, 15, \ldots, 9\}$). The paper states:

"Although our MGAI module adds a moderate training-time overhead, this cost is incurred only once and leaves inference latency unchanged."

The consequence: The headline performance improvements in Table 1—G2RPO achieving Unified Reward 3.783 vs. DanceGRPO 3.569, Image Reward 1.483 vs. 1.128—are reported without accounting for the additional training compute. A practitioner evaluating whether to adopt G2RPO needs to know: am I getting these gains because the method is genuinely more sample-efficient (better optimization per gradient step), or because it uses more compute per training iteration and would be matched by simply training DanceGRPO for 45.7% more iterations? The paper provides no iteration-matched or FLOPs-matched comparison that would answer this. If DanceGRPO trained for 45.7% more iterations matched or exceeded G2RPO's performance, the claimed algorithmic advantage would be an artifact of unequal compute budgets.

What evidence exists in the paper: The supplementary material quantifies the overhead but does not integrate it into any experimental comparison. Figure 1(a) shows reward curves during training (G2RPO vs. DanceGRPO), but the x-axis is unreadable at the provided resolution and no training duration (iterations, wall-clock time, total FLOPs) is reported in the main text. The paper does not report whether G2RPO converges faster (fewer iterations to reach a given reward) such that the per-iteration overhead is partially or fully offset. The ablation in Table 2 shows that a two-granularity subset ($\Lambda = \{1, 2\}$ or $\{1, 3\}$) achieves near-identical performance (IR 1.468 and 1.465 vs. 1.483) with lower overhead, but the paper does not quantify the reduced overhead for these subsets or recommend them as a practical default.

Mitigation status: The paper acknowledges the overhead in the supplementary material but explicitly chooses not to account for it: "our experiments do not account for this cost largely for simplicity" (this language appears in the main text Section 3.2 regarding difficulty estimation, but the parallel treatment of sampling overhead is consistent). The limitation is flagged as future work only implicitly—the paper does not propose a mechanism for reducing the overhead or amortizing it across iterations. The recommendation of $\Lambda = \{1, 2, 3\}$ as the default configuration (Section 4.1) does not discuss the cost-benefit tradeoff relative to the cheaper two-granularity alternatives that Table 2 shows are nearly as effective.

---

Reward Hacking Is Amplified, Not Solved, Under Single-Reward Training

The assumption or constraint: The paper acknowledges that HPS-v2.1 alone is "prone to model hacking due to biases in the training set" (Section 4.1) and follows DanceGRPO's mitigation of joint training with CLIP Score. However, the paper's own mechanism—precise, step-specific credit assignment via Singular Stochastic Sampling—has an unintended consequence: it can make reward hacking more efficient by providing cleaner gradient signals toward reward model exploits.

The consequence: Under HPS-only training (Table 1, top block), G2RPO w/o MGAI achieves HPS 0.376 (vs. DanceGRPO 0.353) but degrades CLIP Score from 0.375 to 0.351 and Unified Reward from 3.548 to 3.469. Full G2RPO partially recovers CLIP (0.355) and UR (3.487) but both remain below DanceGRPO. This means that when the reward model is imperfect—which is the normal case in deployment, since no automated metric perfectly captures human preferences—G2RPO can accelerate overfitting to reward model biases rather than correcting them. The paper states this explicitly only for the general phenomenon of HPS-v2.1 hacking, not for G2RPO's specific amplification of it: "as demonstrated by DanceGRPO, optimizing solely with HPS-v2.1 can induce model hacking, which in turn compromises other out-of-domain evaluation metrics." The fact that G2RPO's hacking is worse than DanceGRPO's (lower CLIP, lower UR) is reported but not discussed as a direct consequence of the proposed mechanism.

What evidence exists in the paper: Table 1 (HPS-only rows) provides clear evidence. DanceGRPO: HPS 0.353, CLIP 0.375, UR 3.548. G2RPO: HPS 0.385, CLIP 0.355, UR 3.487. The HPS gain (+9.1%) comes at the cost of larger CLIP degradation (−5.3%) and UR degradation (−1.7%) relative to DanceGRPO. The pattern is consistent: precise credit assignment helps the model more efficiently find and exploit whatever shortcuts exist in the reward model. On UniGenBench++ (Supplementary Table 5), G2RPO w/o MGAI degrades on Style (73.70 vs. DanceGRPO 76.00) and Layout (76.68 vs. 80.78), confirming that the collapse generalizes beyond the specific metrics in Table 1.

Mitigation status: Joint training with multiple reward models (HPS-v2.1 + CLIP Score) is the paper's mitigation, inherited from DanceGRPO. Under joint training, the hacking reversal is substantial: G2RPO achieves the best scores on all metrics including out-of-domain ones. However, this shifts the burden entirely onto the reward model ensemble—the method itself has no internal mechanism for detecting or resisting reward hacking. If all available reward models share a systematic bias (e.g., all favoring high-contrast images or particular color palettes), joint training would not help, and G2RPO's precise credit assignment could again accelerate exploitation of that shared bias. The paper does not investigate this failure mode.

---

All Experiments Use a Single Model on a Single Dataset; Generalization Is Unverified

The assumption or constraint: Every experiment in the paper—quantitative results in Tables 1-3, qualitative results in Figures 4-8, UniGenBench++ evaluation in Supplementary Table 5—uses Flux.1-dev as the base model and the HPSv2 dataset (103,700 training prompts, 400 test prompts) as the training and primary evaluation distribution. The paper describes Flux.1-dev as "a leading flow model in the community" (Section 4.1) but draws no conclusions about whether the findings transfer to other architectures, model scales, or data distributions.

The consequence: The paper's central claims—that precise credit assignment and multi-granularity evaluation raise the GRPO optimization ceiling—may be specific to Flux.1-dev's architecture (a rectified flow transformer), its training data, its denoising behavior, or its particular sensitivity to discretization error. Several aspects of the findings could be model-specific:

• The granularity sensitivity that motivates MGAI (Figure 3) depends on how much the reward model's scores vary with ODE discretization accuracy. Different model architectures (e.g., U-Net-based diffusion models vs. transformer-based flow models) may have different discretization error characteristics, making granularity sensitivity more or less severe.
• The effectiveness of Singular Stochastic Sampling depends on how much the deterministic ODE trajectory captures the model's "typical" behavior. If a model's ODE integration is less stable or its marginal distribution is less well-preserved under the ODE-to-SDE conversion, the common starting point $x_k$ may not be representative.
• The training timestep set $M = \{16, 15, \ldots, 9\}$ was chosen by DanceGRPO for Flux.1-dev specifically. The optimal set of training timesteps—and therefore the effectiveness of confining stochasticity to individual steps—may differ for other models.

What evidence exists in the paper: None. The paper contains no experiments with any model other than Flux.1-dev (no Stable Diffusion, no SDXL, no other flow-based architectures) and no prompt dataset other than HPSv2 (no DrawBench, PartiPrompts, T2I-CompBench, or other standard text-to-image benchmarks). The supplementary evaluation on UniGenBench++ uses different prompts and a VLM evaluator, which tests robustness to evaluator distribution shift, but the base model and training data remain unchanged. The paper does not discuss model- or dataset-specific limitations.

Mitigation status: Not addressed. The paper does not claim generality beyond Flux.1-dev and HPSv2, but it also does not caution readers that findings may not transfer. The introduction frames G2RPO as a general framework for "flow-based generative models" and "flow models" broadly, which implies broader applicability than is empirically demonstrated.

---

No Human Evaluation Validates That Metric Gains Translate to Perceptible Quality Improvements

The assumption or constraint: All evaluation in the paper is conducted through automated reward models (HPS-v2.1, CLIP Score, Pick Score, Image Reward, Unified Reward) or a VLM judge (UniGenBench-EvalModel-qwen-72b-v1 for UniGenBench++). The paper's central motivation is aligning model behavior with "human preferences" (Abstract, Section 1), but no experiment directly measures whether the reported metric improvements correspond to human-perceivable quality differences.

The consequence: The paper's headline result—G2RPO achieves Unified Reward 3.783 vs. DanceGRPO 3.569, a 6.0% relative improvement—is only meaningful if UR improvements correlate with human preference. While Unified Reward is described as "the SOTA reward model that leverages the powerful visual understanding capabilities of LLM" (Section 4.1), it remains an automated proxy. The gap between proxy metrics and human judgment is well-documented in the text-to-image evaluation literature: reward models can be gamed (as the paper itself demonstrates with HPS-v2.1 hacking), can have systematic biases, and can disagree with human raters on specific image attributes. It is possible that G2RPO's improvements are concentrated in dimensions that automated metrics are sensitive to but humans are not (e.g., specific texture statistics, color distribution properties), or conversely that some quality dimensions humans care about (e.g., creative interpretation of prompts, emotional resonance) are not captured by any of the evaluated metrics and may have degraded.

What evidence exists in the paper: The qualitative comparisons in Figure 4 and supplementary Figures 6-8 are selected examples chosen by the authors, not a systematic human study. They demonstrate cases where G2RPO looks better, but there is no information about how representative these cases are—whether G2RPO also produces images that look worse than baselines, and if so, with what frequency. The paper's statement that G2RPO provides "superior performance in human preference alignment" (Figure 4 caption) and "superior visual fidelity and text-image alignment" (Supplementary Section 9) is based on automated metrics only.

Mitigation status: Not addressed. The paper does not mention human evaluation as a limitation or as future work. Given that the paper's title claims "Precise Preference Alignment" and its introduction frames the work as addressing "alignment with human preferences," the absence of human validation is a significant gap between the claimed contribution and the empirical evidence supporting it.

---

The Method's Gains Depend Critically on Multi-Reward Joint Training, Yet the Framework Provides No Internal Robustness Mechanism

The assumption or constraint: G2RPO's strong out-of-domain performance—the result that most clearly distinguishes it from baselines—is demonstrated exclusively under joint training with HPS-v2.1 and CLIP Score. Under single-reward HPS-only training, G2RPO's out-of-domain metrics are worse than DanceGRPO's (UR 3.487 vs. 3.548, CLIP 0.355 vs. 0.375). This means G2RPO's claimed benefit of "comprehensive evaluation" is conditional on having at least two reward models with complementary biases.

The consequence: In any deployment scenario where only a single reward model is available—which is common, since training multiple high-quality reward models is expensive and domain-specific reward models (e.g., for medical imaging, scientific visualization, or specialized design tasks) may not have natural complementary counterparts—G2RPO's advantages largely evaporate. G2RPO w/o MGAI under joint training (Table 1): HPS 0.372, IR 1.421, UR 3.688. DanceGRPO under joint training: HPS 0.331, IR 1.128, UR 3.569. The improvement is substantial. But G2RPO w/o MGAI under HPS-only training: IR 1.286, UR 3.469. DanceGRPO under HPS-only training: IR 1.233, UR 3.548. On Unified Reward—the most comprehensive metric—DanceGRPO actually outperforms G2RPO under single-reward training.

This reveals a fundamental dependency: G2RPO amplifies whatever signal it receives. If that signal comes from a well-constructed ensemble of complementary reward models, the amplification is beneficial. If it comes from a single, potentially biased reward model, the amplification accelerates reward hacking. The framework itself has no mechanism to distinguish between these scenarios or to internally regularize against hacking when only one reward model is available.

What evidence exists in the paper: The contrast between the top and bottom blocks of Table 1 provides direct evidence. Additionally, the UniGenBench++ results (Supplementary Table 5) show that G2RPO w/o MGAI degrades on Style and Layout—dimensions that HPS-v2.1 alone does not adequately capture—confirming that precise credit assignment without multi-reward diversity actively harms performance on dimensions not represented in the training signal. The paper's statement that "HPS-v2.1 is prone to model hacking due to biases in the training set" (Section 4.1) is the extent of the discussion; the paper does not analyze G2RPO's specific vulnerability to this problem.

Mitigation status: The paper's solution is to follow DanceGRPO's practice of joint training with HPS-v2.1 and CLIP Score, which is effective for the specific reward models and dataset studied. However, this is an external fix (choose better reward models) rather than an internal robustness mechanism. The paper does not propose or evaluate any G2RPO-specific mitigation for single-reward scenarios—no adaptive clipping, no uncertainty-based advantage weighting, no reward model ensemble distillation, no diversity regularization. The method as presented is only recommended for settings where multiple complementary reward models are available, which constrains its applicability relative to methods that include internal robustness mechanisms.

---

The Test Set Size Is Small and Statistical Significance Is Unreported

The assumption or constraint: All quantitative comparisons in Tables 1-3 and Supplementary Table 5 are reported as aggregate scores over a fixed test set of 400 prompts (from HPSv2) or an unspecified number of UniGenBench++ prompts, with no confidence intervals, standard deviations, standard errors, or statistical significance tests. The differences between methods on some metrics are very small in absolute terms—for example, Pick Score under joint training: G2RPO 0.235, G2RPO w/o MGAI 0.234, MixGRPO 0.230, DanceGRPO 0.227. The paper does not enable readers to assess whether these differences are statistically reliable or within the range of sampling variation.

The consequence: The paper makes strong ordinal claims—"G2RPO outperforms other baselines across both in-domain and out-of-domain rewards" (Section 4.2)—that may not be statistically justified. With 400 test prompts, a difference of 0.008 in Pick Score (0.235 vs. 0.227) corresponds to an average per-image difference that could easily be drowned out by per-image variance. The paper reports no per-image score distributions, no bootstrap confidence intervals, and no hypothesis tests. A practitioner reading the paper cannot determine:

• Whether the reported differences would replicate on a different test set of the same size.
• Whether G2RPO's advantage over MixGRPO on Image Reward (1.483 vs. 1.436, a 3.3% relative difference) is robust or fragile.
• Whether the ranking of methods would change with a different random seed (the paper reports using seed 42, but does not report variance across multiple seeds).
• Whether the granularity ablation in Table 2—where $\Lambda = \{1, 3\}$ achieves HPS 0.378 vs. $\Lambda = \{1, 2, 3\}$ at 0.376—reflects a genuine performance inversion or noise.

What evidence exists in the paper: None. The paper does not report any form of uncertainty quantification. The test set size (400 prompts) is mentioned in Section 4.1 but never discussed as a limitation. The results tables present single-point estimates without error bars, standard deviations, or significance indicators. The reward curves in Figure 1(a) show training dynamics but are too small to read specific values and cover only HPS-v2.1, not the out-of-domain metrics where the differences are most consequential.

Mitigation status: Not addressed. The paper does not acknowledge this as a limitation or suggest that future work should include statistical validation. This is particularly problematic because the paper's contribution is evaluated primarily through metric improvements—if those improvements are not statistically distinguishable from noise, the central empirical claims are unsupported regardless of the method's conceptual merits. The small test set is not inherently disqualifying—400 prompts is standard for HPSv2-based evaluations in this literature—but the absence of any uncertainty quantification means readers cannot assess the reliability of the reported comparisons.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a diagnostic reframing rather than a paradigm shift, but one with substantial practical consequences for how online RL is applied to flow-based generative models. The core contribution is identifying that the bottleneck in flow-based GRPO is not exploration strategy per se—how much noise, where, with what schedule—but rather the fidelity of the reward signal that translates exploration outcomes into gradient updates. This shifts attention from the sampling procedure (which DanceGRPO and MixGRPO had already refined) to the evaluation procedure: how advantages are computed, what causal structure they encode, and at what resolution they assess trajectory quality.

The magnitude of this shift is diagnostic rather than revolutionary. The paper does not propose a new RL algorithm, a new reward model architecture, or a new generative modeling paradigm. It accepts the established GRPO framework for flow models—ODE-to-SDE conversion for exploration, group-relative advantage computation, PPO-style clipped updates—and improves the information content of the signal flowing through that framework. This is an important class of contribution: it identifies where existing methods are losing information and shows that recovering that information yields measurable gains. The analogy is to improving a measurement instrument rather than redesigning the experiment. The instrument (GRPO) remains the same; the readings it takes (advantages) are now more precise and more comprehensive.

The most significant conceptual impact is establishing temporal credit assignment as a first-class design concern in flow model RL. Prior work treated the SDE noise schedule as the primary lever for controlling exploration quality—DanceGRPO's contribution of restricting training to early timesteps, and MixGRPO's hybrid ODE-SDE schedule, both operate on when and how much noise is injected. G2RPO argues, through its Singular Stochastic Sampling mechanism and the 6.52% relative improvement over DanceGRPO on HPS-only HPS-v2.1 (Table 1), that what you do with the resulting reward signal matters at least as much. This is likely to influence future work by making credit assignment an explicit design axis alongside exploration magnitude and schedule.

The paper also resolves a latent tension in the flow-based GRPO literature. DanceGRPO showed that training on early timesteps (M = {16, ..., 9}) is sufficient—even preferable—to training on all steps, implicitly acknowledging that not all steps contribute equally to final quality. But it retained uniform advantage broadcasting within those early steps, meaning that steps 16 and 9 still received identical advantage signals despite their qualitatively different roles (global composition vs. mid-level structure). G2RPO's step-specific advantages resolve this inconsistency: each step k gets its own advantage A^i_k computed from images where only step k was perturbed. The fact that training is still restricted to M (the first half) but each step within M now receives a distinct signal represents a more coherent design than DanceGRPO's hybrid of temporal restriction but uniform signal.

The paper's second contribution—Multi-Granularity Advantage Integration—introduces evaluation resolution as a new axis of robustness in RL for generative models. This extends DanceGRPO's insight that evaluating with multiple reward models improves robustness (to reward model bias) into the orthogonal dimension of evaluation granularity (robustness to discretization artifact). The explicit parallel the paper draws—summing advantages across granularities just as DanceGRPO sums advantages across reward models—frames this as a natural generalization rather than an ad-hoc trick. The consequence is that future work on alignment evaluation should consider not just which reward models to use, but at what resolution to evaluate trajectories, as a similarly consequential design choice.

The paper makes some research directions less attractive. The finding that lookahead search-like strategies (explored in the PRM search literature of the earlier example paper) are not needed here—Singular Stochastic Sampling achieves dense credit assignment with zero additional learned components, purely through sampling procedure restructuring—suggests that building learned value functions or per-step reward predictors for flow models may be unnecessary complexity when the ODE/SDE duality provides a simpler path to the same goal. Similarly, the paper's demonstration that multi-granularity evaluation improves out-of-domain generalization (Unified Reward +6.0%, UniGenBench++ +4.7% over DanceGRPO) without any explicit generalization-promoting regularization suggests that robustness can emerge from evaluation protocol design rather than from algorithmic interventions like gradient penalties or data augmentation.

However, the paper does not resolve the fundamental challenge of reward hacking in single-reward scenarios. Under HPS-only training, G2RPO amplifies hacking relative to DanceGRPO (CLIP drops from 0.375 to 0.355, UR drops from 3.548 to 3.487). This means the method's benefits are conditional on having a well-constructed ensemble of complementary reward models—the method itself provides no internal robustness mechanism. This is a boundary condition that the field needs to grapple with: credit assignment precision and evaluation comprehensiveness are force multipliers for whatever reward signal they receive, whether that signal is genuinely quality-correlated or hacking-prone. The implication is that progress on reward model quality and reward model diversity is a prerequisite for realizing the full benefits of methods like G2RPO, not an independent track.

---

Follow-Up Research This Work Enables

Compute-normalized comparison: does G2RPO's algorithmic advantage survive equal-FLOPs comparison with longer-trained baselines? The paper reports that multi-granularity sampling adds ~45.7% more denoising steps per training iteration (Supplementary Section 11), but all comparisons are iteration-matched—DanceGRPO and G2RPO are trained for the same number of iterations with different per-iteration costs. A critical follow-up would train DanceGRPO for 45.7% more iterations (or equivalently, G2RPO for proportionally fewer) and compare performance at equal total FLOPs. The key question: does the 3.15-point UniGenBench++ improvement (69.21 vs. 66.06) survive when the baseline is given equivalent compute? If DanceGRPO catches up with more iterations, G2RPO's contribution is the efficiency gain (faster convergence, not higher asymptote). If the gap persists, it confirms that the evaluation signal quality—not just quantity—raises the optimization ceiling. The experiment requires: fixed total GPU-hours, varying iteration counts across methods, reporting both convergence rates and final performance.

Does multi-granularity evaluation prevent granularity-specific reward hacking, or does it simply provide more evaluation budget per direction? The paper interprets MGAI's benefits (improved out-of-domain metrics, recovery of Style and Layout on UniGenBench++) as evidence that multi-resolution evaluation produces more robust assessments. An alternative explanation is that evaluating each direction three times (once per granularity) simply provides more signal per SDE perturbation—effectively tripling the reward model evaluation budget, which could improve performance regardless of whether the granularities differ. A direct test: compare MGAI (Λ = {1, 2, 3}) against a baseline where each direction is evaluated three times at the same granularity (Λ = {1, 1, 1} with different ODE solver random seeds, or simply three independent reward model evaluations of the same image). If MGAI matches or exceeds this same-granularity-multiple-evaluation baseline on out-of-domain metrics, the multi-resolution aspect is specifically valuable. If not, the benefit is increased evaluation budget, and practitioners could achieve similar gains by simply running the reward model multiple times on the same images at lower computational cost (since same-granularity ODE completions share more computation).

Can the granularity sensitivity that motivates MGAI be eliminated by training the reward model to be discretization-invariant? Figure 3 shows that the same SDE direction receives different HPS-v2.1 scores under coarse vs. fine ODE completion, which MGAI addresses by averaging across granularities. An alternative approach: fine-tune the reward model itself (HPS-v2.1) on pairs of images that differ only in ODE discretization error, with a loss that encourages identical scores for such pairs. If this reward-model-level fix eliminates the granularity sensitivity, MGAI becomes unnecessary and the 45.7% training overhead is avoided. If it doesn't—if the sensitivity reflects genuine ambiguity in image quality that no single-resolution reward model can resolve—then MGAI is addressing a fundamental limitation of single-resolution evaluation that cannot be engineered away at the reward model level. This experiment would clarify whether the granularity problem is a solvable artifact of current reward models or an inherent challenge in evaluating intermediate denoising states.

How does the optimal granularity set Λ scale with model size and training compute? The paper ablates Λ = {1}, {1,2}, {1,3}, and {1,2,3} on Flux.1-dev (Table 2) and finds that two granularities capture most of the benefit, with three providing marginal gains on IR (1.468 → 1.483) and UR (3.759 → 3.783). This may be specific to Flux.1-dev's scale and discretization error characteristics. Larger models (e.g., Flux Pro, or future scaled flow models) may have different ODE integration error profiles—potentially needing more granularities (if discretization error varies more across step counts) or fewer (if higher-capacity models produce more consistent trajectories). A scaling study: train G2RPO with varying Λ across at least two model sizes (e.g., Flux.1-dev and a hypothetical 2–4× larger variant, or equivalent Stable Diffusion model scales) and measure whether the optimal Λ changes. The hypothesis is that larger models benefit less from multi-granularity evaluation because their ODE integration is more accurate, making single-granularity evaluation already comprehensive. If confirmed, this would mean MGAI's value is largest at the model scales where alignment is most practically deployed (current-generation models, not future scaled ones), which has direct resource-allocation implications.

Does Singular Stochastic Sampling's credit assignment benefit extend beyond the top-half timestep restriction? The paper inherits M = {16, ..., 9} from DanceGRPO without ablation, training only on the first half of denoising steps. The rationale is that early steps determine overall structure and have larger exploration space. But with Singular Stochastic Sampling providing step-specific advantages, late steps might now benefit more from training than they did under uniform broadcasting—since the advantage signal isn't contaminated by early-step noise, a good late-step texture refinement could receive a clear positive advantage rather than being drowned out by the aggregate image reward. Experiment: expand M to include later steps (e.g., M = {16, ..., 5} or even all 16 steps) and measure whether performance improves over the M = {16, ..., 9} default. If later steps do benefit, this would mean G2RPO's credit assignment precision enables training across more of the trajectory, potentially raising the optimization ceiling further. If they don't, it confirms that early steps are genuinely where RL provides value regardless of credit assignment quality. Either outcome refines our understanding of where in the denoising process alignment is learned.

Stress-test: does G2RPO's robustness to inference step count (Table 3) generalize to extreme discretization regimes? Table 3 shows G2RPO maintaining performance at 10 and 20 inference steps, but these are relatively close to the 16-step training setting. A more aggressive test: evaluate at 4–6 inference steps (representing real-time generation scenarios) and at 50–100 steps (representing maximum-quality offline generation). The hypothesis: multi-granularity training with Λ = {1, 2, 3} should provide robustness at both extremes since the model has seen coarse-grained (λ = 3, ~2–3 effective steps from the SDE perturbation to completion) and fine-grained (λ = 1, ~8–9 effective steps) evaluation during training. If G2RPO maintains advantage at 4-step inference while baselines collapse, it becomes the method of choice for latency-constrained deployment. If it degrades at 100-step inference (a regime never seen during training), it reveals a limitation: multi-granularity robustness is bounded by the range of resolutions encountered during training, not a general property.

---

Practical Applications and Downstream Use Cases

Cost-efficient fine-tuning of deployed text-to-image models for domain-specific preference alignment. Organizations deploying text-to-image models for specialized domains—e-commerce product visualization, architectural rendering, medical illustration, game asset generation—often need to align outputs with domain-specific aesthetic or functional criteria that off-the-shelf models don't capture. The standard approach is to collect domain-specific preference data and fine-tune. G2RPO offers a concrete recipe: (1) define one or more domain-specific reward models (which could be automated metrics, user click-through rates, or learned from domain-expert annotations), (2) apply joint GRPO training with Singular Stochastic Sampling and MGAI using the same group size (G = 12), SDE noise level (η = 0.7), and granularity set (Λ = {1, 2, 3}) as the paper. The 31.5% relative improvement on Image Reward over DanceGRPO (Table 1, joint training) suggests substantial alignment gains are achievable purely through better credit assignment, without changing model architecture or inference cost. The training overhead (~45.7% more denoising steps) is a one-time cost amortized across all downstream inference, making this attractive for models that will be deployed at scale.

Improving prompt adherence in high-stakes visual generation. Applications requiring precise prompt following—legal document illustration, scientific figure generation, accessibility-focused image description—are limited by current models' tendency to drop or conflate prompt details. G2RPO's UniGenBench++ results show the largest improvements on dimensions directly related to prompt adherence: Text +11.78 points, Attribute +6.19 points, Relation +2.79 points over DanceGRPO. This suggests that precise per-step credit assignment particularly benefits tasks where specific prompt elements must survive through the entire denoising trajectory without being lost to SDE noise or reward signal dilution. A deployment scenario: a scientific publisher fine-tunes Flux.1-dev with G2RPO using reward models trained to detect figure-caption consistency, then uses the aligned model to generate draft figures from textual descriptions of experimental setups. The key claim: G2RPO's step-specific advantages mean that when the reward model detects a missing detail (e.g., "the error bars should be red"), the gradient signal more accurately identifies which denoising step failed to preserve that detail, enabling more targeted policy improvement than uniform broadcasting would allow.

Training data generation for self-improving visual models. The paper's framework integrates naturally into self-improvement pipelines where a model generates data that is then used to train the next iteration. In each self-improvement round, G2RPO's precise credit assignment and multi-granularity evaluation could be used to identify which intermediate denoising directions lead to high-quality outputs, and those trajectories could be distilled into the base model via supervised fine-tuning on (noise, image) pairs weighted by the mixed advantage. The practical benefit: cleaner training data for the next iteration, since the advantage-weighted sampling would upweight trajectories that are robustly good (high A^{i,mix}_k) and downweight trajectories that are granularity-specifically or spuriously good. The paper's finding that MGAI partially prevents domain collapse (Style and Layout recovery on UniGenBench++, Supplementary Table 5) is directly relevant here—without multi-granularity evaluation, self-improvement loops risk collapsing generation diversity, and MGAI provides a partial safeguard.

---

When to Prefer This Method

The paper itself positions G2RPO as an upgrade over DanceGRPO and MixGRPO within the flow-based GRPO paradigm, with the primary tradeoff being improved alignment ceiling vs. increased training-time computation. The decision boundaries can be extracted from the experimental results:

• Prefer G2RPO with joint multi-reward training when: (1) you are already using or planning to use flow-based GRPO for alignment, (2) you have access to at least two complementary reward models for joint training (to prevent the reward hacking amplification observed under HPS-only training), (3) the ~45.7% training-time overhead is acceptable (the model will be deployed at scale, making the one-time training cost worthwhile), and (4) your evaluation priorities include out-of-domain robustness on VLM-based metrics (Unified Reward, UniGenBench++) where G2RPO's gains over DanceGRPO are largest (+6.0% and +4.7%, respectively, under joint training).

• Prefer DanceGRPO or MixGRPO when: (1) only a single reward model is available and you cannot construct a complementary ensemble—G2RPO amplifies single-reward hacking (UR drops from 3.548 to 3.487 under HPS-only training), (2) training compute is severely constrained and the 45.7% per-iteration overhead cannot be absorbed—in this case, MixGRPO's efficiency-oriented hybrid sampling may provide better performance per GPU-hour, though the paper doesn't provide the FLOPs-matched comparison needed to confirm this, or (3) your primary evaluation metric is Pick Score, where G2RPO's advantage is marginal (0.235 vs. 0.227 for DanceGRPO under joint training, a 3.5% relative gain) and may not justify the overhead.

• Prefer G2RPO without MGAI (Singular Stochastic Sampling only) as a lightweight upgrade when: (1) you want the credit assignment benefits (6.52% HPS-v2.1 improvement over DanceGRPO under HPS-only training) without any additional ODE completion overhead—since S^3 confines stochasticity but doesn't add granularities, it has no per-iteration penalty beyond the pre-existing GRPO cost, and (2) you are operating in a domain where reward hacking is not a concern (e.g., the reward model is known to be well-calibrated) or you are using joint training to mitigate it. The ablation in Table 2 shows that a two-granularity Λ = {1, 2} or {1, 3} captures most of MGAI's benefit at lower overhead than the full three-granularity set, representing a practical middle ground for resource-constrained deployments.

Important caveat: These preferences are contingent on the assumption that G2RPO's gains survive a FLOPs-matched comparison, which the paper does not provide. If DanceGRPO trained for 45.7% more iterations matches G2RPO, then G2RPO's value proposition shifts from "higher alignment ceiling" to "faster convergence to the same ceiling"—still useful, but requiring different deployment calculus (time-to-deployment vs. total compute cost). The absence of this comparison means the above decision rules are provisional until validated by equal-FLOPs experiments.