ArXiv: 2603.21872

🎯 Pitch

Video GRPO fails because standard ODE-to-SDE conversions inject excess noise that pushes rollouts off the data manifold, corrupting reward estimates. SAGE-GRPO fixes this with a precise manifold-aware noise schedule and dual trust region—achieving 70.8–85.9% preference rates over existing methods.

1. Executive Summary

This paper proposes SAGE-GRPO (Stable Alignment via Exploration with Group Relative Policy Optimization), a manifold-aware reinforcement learning framework for aligning video generation models that addresses the instability of existing GRPO methods by constraining exploration to the vicinity of the pre-trained model's data manifold. Evaluated on HunyuanVideo1.5 using the VideoAlign reward model, SAGE-GRPO couples a Precise Manifold-Aware SDE (which replaces first-order noise variance approximations with an exact integrated variance that includes a logarithmic curvature correction, removing excess noise energy in high-noise regions) with a Gradient Norm Equalizer (which normalizes optimization pressure across timesteps to prevent gradients from vanishing at high noise and exploding at low noise) at the micro level, and a Dual Trust Region (combining a periodically updated moving anchor for position control with step-wise KL constraints for velocity control) at the macro level. Under an alignment-focused reward setting, SAGE-GRPO with Dual Moving KL achieves the best overall VideoAlign reward (0.8066), VQ (−0.4765), MQ (−0.2385), and CLIPScore (0.5484) among all compared methods, with user studies confirming 70.8–85.9% preference rates over DanceGRPO, FlowGRPO, and CPS baselines, establishing that manifold-constrained exploration enables stable reward optimization in video generation only when both noise injection and long-horizon policy drift are jointly controlled.

2. Context and Motivation

The Core Problem: GRPO for Video Generation Is Fundamentally Unstable

The central question this paper tackles is: why does Group Relative Policy Optimization (GRPO) work reliably for language models and image generation, but consistently underperform for video generation? The answer, as the paper argues, lies in the specific nature of exploration in video diffusion models — a problem that the prior literature had neither fully diagnosed nor solved.

To understand this gap, we need to examine what GRPO actually does at test time. GRPO is a reinforcement learning method that optimizes a policy (here, a video generation model) against a reward signal. The mechanism works as follows: for a given text prompt, the model generates multiple candidate videos (a "group" of rollouts), each is scored by a reward model, and the policy is updated to favor the trajectories that produced higher rewards relative to the group average. The group-normalized advantage ensures that the model learns relative preferences rather than raw reward magnitudes, which is why GRPO has been effective for aligning LLMs with human preferences (Shao et al., 2024; Guo et al., 2025).

The critical step for applying GRPO to diffusion models is the ODE-to-SDE conversion. Video diffusion models generate videos through a deterministic reverse process — at inference time, they follow an ordinary differential equation (ODE) trajectory from noise to clean video. But GRPO requires exploration: the model must produce diverse samples so that reward signals can discriminate between good and bad generations. To enable this, prior work converts the deterministic ODE sampler into a stochastic differential equation (SDE) sampler by injecting noise at each step, creating a distribution over possible videos for the same prompt. The quality of this exploration — whether the injected noise produces meaningful variation versus simply degrading the output — directly determines whether GRPO succeeds or fails.

The paper's key diagnosis is that existing ODE-to-SDE conversion methods inject excess noise that pushes trajectories off the pre-trained model's data manifold, producing low-quality rollouts whose reward signals are unreliable. When a reward model evaluates a video that has been pushed off-manifold (containing temporal jitter, structural artifacts, or incoherent motion), the resulting reward is dominated by quality degradation rather than the alignment signal GRPO is trying to optimize. The policy then receives noisy, inconsistent gradient updates that destabilize training. This explains the reliability gap: language model rollouts stay within the token distribution the model was trained on, and image generation has a relatively compact output space, but video generation has a large, highly structured solution space where even small perturbations away from valid videos produce clearly visible artifacts.

This problem is theoretically significant because it reveals that the exploration strategy is not a mere implementation detail — it is the linchpin that determines whether reward-based fine-tuning converges at all. Practically, it matters because video generation is one of the most commercially and creatively impactful applications of generative AI, and without stable alignment methods, we cannot reliably steer these models toward producing videos that users actually want (e.g., temporally coherent, semantically accurate, aesthetically pleasing).

Why This Problem Matters: The Stakes for Video Generation Alignment

The real-world stakes are high. Text-to-video generation models have progressed rapidly — from early research prototypes to production systems like HunyuanVideo, Sora, and Wan — but their outputs remain inconsistent. Even state-of-the-art models frequently produce videos with temporal artifacts (flickering, jitter, sudden scene changes), semantic misalignment (objects that don't match the prompt description, actions that are physically implausible), and aesthetic flaws (unnatural lighting, unrealistic motion). Post-training alignment with reward models is the standard approach for addressing these issues, but if the alignment process itself is unstable, the models either fail to improve or, worse, degrade in quality — a phenomenon known as reward hacking, where the policy learns to exploit reward model blind spots rather than genuinely improving.

The paper's Figure 2 provides an intuitive geometric picture of what goes wrong. The pre-trained video generation model defines a lower-dimensional video manifold M within the high-dimensional space of all possible pixel arrangements. Valid videos live on or near this manifold. When we inject exploration noise, we want to perturb the trajectory along the manifold (producing different but valid videos) rather than away from it (producing artifacts). Conventional linear SDE noise injection, as used in DanceGRPO and FlowGRPO, creates an exploration region that is too broad — it pushes the trajectory into off-manifold regions (the "High-D Noise Space" in Figure 2), producing videos with temporal jitter and structural artifacts. The paper's manifold-aware SDE, by contrast, creates a tighter exploration region that stays tangent to the manifold, preserving rollout quality.

This is not merely a cosmetic concern. When rollouts are off-manifold, the reward model's evaluation becomes unreliable for two reasons:

  1. The reward model was trained on in-distribution videos, so its scores for out-of-distribution (artifact-ridden) videos may reflect quality degradation rather than the alignment dimensions we care about (visual quality, motion quality, text alignment).
  2. The advantage computation in GRPO is relative — it compares rollouts within a group. If some rollouts are severely degraded by noise while others are not, the advantage signal is dominated by noise-level variation rather than genuine alignment quality, leading to noisy gradient updates.

The paper also documents a second, more subtle instability: even when individual steps are well-behaved, cumulative policy drift over many GRPO updates can push the model away from the pre-trained manifold. This is the "plasticity-stability dilemma" that the paper claims existing methods fail to resolve. If you constrain the policy too tightly to its initialization (using a fixed KL divergence penalty anchored to the pre-trained model, as in traditional approaches), the model cannot move far enough to optimize the reward — it underfits. If you relax the constraint (using only step-wise KL that limits the size of each update but not the total displacement), the model can slowly drift off-manifold over hundreds of steps — it overfits to the reward signal or collapses. The paper's Dual Trust Region mechanism is designed to thread this needle.

Prior Approaches and Where They Fall Short

The paper situates itself against three main families of prior work, each with specific limitations that the SAGE-GRPO framework addresses.

DanceGRPO and FlowGRPO: First-Order Noise Approximations. DanceGRPO (Xue et al., 2025) and FlowGRPO (Liu et al., 2025b) are the most direct predecessors — they adapted GRPO to visual generation through ODE-to-SDE conversion for stochastic exploration. The paper's critique is technical and specific: these methods derive their SDE noise standard deviation using an Euler-style discretization with first-order approximations, which introduces a truncation error that overestimates the noise variance, especially in high-noise (early) timesteps.

To make this concrete, consider Table 1 in the paper, which compares the noise standard deviation formulations:

  • DanceGRPO uses ησt−σt+1\eta\sqrt{\sigma_t - \sigma_{t+1}} — a simple first-order difference that ignores how the diffusion coefficient εt\varepsilon_t varies over the interval [σt+1,σt][\sigma_{t+1}, \sigma_t].
  • FlowGRPO uses ησt1−σt(σt−σt+1)\eta\sqrt{\frac{\sigma_t}{1-\sigma_t}(\sigma_t - \sigma_{t+1})} — this multiplies the first-order difference by the instantaneous diffusion coefficient εt=ησt/(1−σt)\varepsilon_t = \eta\sqrt{\sigma_t / (1-\sigma_t)}, but still uses a point evaluation rather than an integral.
  • SAGE-GRPO uses η−(σt−σt+1)+log⁡(1−σt+11−σt)\eta\sqrt{-(\sigma_t - \sigma_{t+1}) + \log\left(\frac{1-\sigma_{t+1}}{1-\sigma_t}\right)} — this is the exact integrated variance Σt=∫σt+1σtεs2ds\Sigma_t = \int_{\sigma_{t+1}}^{\sigma_t} \varepsilon_s^2 ds over the timestep interval, which includes a logarithmic curvature correction term.

The logarithmic term log⁡((1−σt+1)/(1−σt))\log((1-\sigma_{t+1})/(1-\sigma_t)) accounts for the geometric contraction of the signal coefficient (1−σt)(1-\sigma_t), which linear approximations fail to capture. In high-noise regions (where σt→1\sigma_t \to 1 and (1−σt)(1-\sigma_t) is small), the relative change in the signal coefficient over a timestep can be large, making the first-order approximation particularly inaccurate. The paper's Figure 1(a.1) illustrates this geometrically: the Euler-style discretization introduces a "purple region of extra energy" (the discretization error) beyond the true integral, while the precise SDE integrates only the area under the actual curve. Figure 9 in the appendix provides quantitative validation: under certain σ\sigma schedule settings, FlowGRPO's noise standard deviation explodes at the first step (reaching ~3.0), while SAGE-GRPO's remains stable (~1.0).

The consequence is that DanceGRPO and FlowGRPO inject more noise than mathematically necessary, pushing trajectories off the video manifold and degrading rollout quality. This is not just a theoretical concern — Figure 1(b) shows qualitative examples where SAGE-GRPO produces "more stable and better-aligned generations" than these baselines.

CPS: Coefficients-Preserving Sampling. CPS (Wang & Yu, 2025) takes a different approach to the SDE conversion, but the paper does not detail its specific noise formulation. From context, CPS appears to preserve certain statistical properties of the flow matching coefficients during the ODE-to-SDE conversion, but it still operates within a framework that does not explicitly constrain exploration to the data manifold at both micro and macro levels. The experimental results (Table 2) show CPS performing competitively in some settings but being outperformed by SAGE-GRPO in the alignment-focused reward configuration, suggesting that CPS's approach addresses some but not all of the instability sources.

General RL for Diffusion Models (DDPO, DPOK). The paper also positions itself against the broader landscape of RL fine-tuning for diffusion models. DDPO (Black et al., 2023) and DPOK (Fan et al., 2023) treated the denoising process as a Markov Decision Process to enable policy gradient estimation. These methods established the feasibility of RL for diffusion models but did not address the specific challenges of video generation — the larger solution space, the need for temporal coherence, and the manifold-constrained exploration problem that the paper identifies as central.

Preference Alignment for Video Generation. Beyond RL methods, the paper notes that video generation alignment is an active area with diverse approaches including video reward models (Liu et al., 2025c; Xu et al., 2024; Mi et al., 2025; Zhang et al., 2025), alignment algorithms (Li et al., 2024; Gambashidze et al., 2024; Yu et al., 2024; Zhou et al., 2025; Jia et al., 2025), and curriculum learning strategies (Self-paced GRPO; Li et al., 2025b). The paper's key critique of these frameworks is that they face a stability-plasticity dilemma: strict constraints (e.g., a fixed KL penalty anchored to the initialization) limit plasticity and prevent the policy from reaching optimal reward regions, while relaxed constraints (e.g., heuristic scheduling or no position control) trigger reward hacking or catastrophic forgetting.

This framing is important because it identifies a structural problem rather than an implementation flaw. The paper argues that existing methods have been developed for smaller search spaces (language token sequences, individual images) where the stability-plasticity tradeoff is more forgiving, and that video generation's large, structured output space makes this tradeoff acute — hence the need for a fundamentally different exploration design that operates simultaneously at multiple scales.

How This Paper Positions Itself

The paper's positioning is layered and specific. It does not claim to invent GRPO for video generation (DanceGRPO and FlowGRPO already did that). Instead, it claims to have identified the root cause of why existing video GRPO methods are unstable — the off-manifold exploration problem — and to have introduced a principled, multi-level solution that addresses this cause rather than merely treating its symptoms.

The unifying insight is the manifold-constrained exploration formulation. The paper reframes the entire problem: rather than asking "how do we make GRPO more stable?", it asks "how do we constrain exploration to the vicinity of the pre-trained model's data manifold so that every rollout remains valid and every reward signal remains reliable?" This reformulation has theoretical appeal because it connects the practical instability to a geometric property of the generation process (the lower-dimensional structure of valid videos within the ambient pixel space), and it has practical appeal because it suggests concrete, measurable improvements at two distinct scales:

  • Micro-level (within each sampling step): The precise manifold-aware SDE and gradient equalizer ensure that individual exploration steps stay tangent to the manifold and that optimization pressure is balanced across all noise levels.
  • Macro-level (across many GRPO updates): The Dual Trust Region with periodic moving anchors and step-wise constraints prevents the cumulative drift that causes policies to slowly wander off-manifold even when individual steps are well-behaved.

The paper explicitly distinguishes its contributions from prior work that attempted partial solutions. It notes that existing methods use either fixed KL constraints (position control but too restrictive) or step-wise KL constraints (velocity control but no position bounds), and argues that only the combination — the Dual Trust Region as a position-velocity controller — can simultaneously enable sustained plasticity (reaching high-reward regions far from initialization) and stability (not drifting off-manifold during the journey). The analogy to control theory (position-velocity control in dynamical systems) is deliberate and frames the problem as one of trajectory optimization in policy space, not just per-step regularization.

The empirical positioning is equally specific. The paper does not claim superiority on all metrics in all settings — Table 2 shows that in the averaged-reward setting without KL regularization, CPS achieves the highest overall reward (0.6343 vs. SAGE-GRPO's 0.4859). Instead, the paper argues that the alignment-focused setting (where text alignment is weighted more heavily than visual/motion quality, reducing reward hacking artifacts) is the more meaningful evaluation of stable alignment, and that under this setting, SAGE-GRPO with Dual Moving KL achieves the best performance on the metrics that matter most (Overall, VQ, MQ, CLIPScore). This is a nuanced claim: the method's advantage is not that it maximizes every reward configuration, but that it provides the most stable and reliable alignment when the optimization target is well-specified.

The paper also positions itself practically. It emphasizes that all experiments use the original frozen VideoAlign evaluator without reward-model fine-tuning, meaning the improvements come from better exploration and policy constraint, not from a better-tuned reward signal. This matters for reproducibility and practical deployment: if the method required fine-tuning the reward model (which can itself be unstable), the gains would be harder to attribute and harder to replicate across different reward models.

Finally, the paper's ablation strategy (Figures 3, 7, 8, and the KL strategy comparisons in Figures 11-12) positions it as providing a systematic decomposition of which components matter and why. The gradient equalizer ablation shows that without per-timestep balancing, reward curves are unstable or plateaued across multiple SDE formulations. The KL strategy ablation shows that Dual Moving KL maintains higher and more stable exploration throughout training compared to alternatives. These ablations are not just validation — they are the paper's way of demonstrating that each component addresses a specific, diagnosed failure mode, and that all components are necessary for the full benefit.

In summary, the paper positions SAGE-GRPO not as yet another RL fine-tuning method, but as a diagnosis-driven redesign of the exploration mechanism for video GRPO. The contribution is the insight that the ODE-to-SDE conversion and the long-horizon constraint strategy must be jointly designed around the geometry of the video manifold, and the evidence is that doing so yields consistent, measurable improvements over methods that treat these as independent implementation choices.

3. Technical Approach

3.1 Reader Orientation

SAGE-GRPO is a modified reinforcement learning training procedure for text-to-video generation models that replaces the standard exploration noise used during GRPO fine-tuning with a more mathematically precise version, coupled with a two-tier constraint system that prevents the model from drifting away from generating valid videos. The problem it solves is that existing GRPO methods for video generation inject excess noise during the exploration phase—pushing generated videos off the "manifold" of realistic videos and corrupting the reward signal—and fail to control cumulative policy drift over many training steps, leading to unstable or degraded alignment. The solution is a multi-scale exploration framework that operates at the micro level (within individual denoising steps, via a corrected SDE and gradient equalization) and the macro level (across training steps, via a combined position-velocity trust region mechanism), ensuring that every exploration step stays tangent to the video manifold and that the policy cannot drift arbitrarily far from a valid region over time.

3.2 Big-Picture Architecture (Diagram in Words)

The SAGE-GRPO framework consists of five integrated components that modify the standard GRPO pipeline for video diffusion models:

  1. Base Video Generation Model (HunyuanVideo1.5) — a pretrained Rectified Flow model that defines the video manifold M and serves as the policy $\pi_{\theta}$ to be fine-tuned. It maps text prompts $c$ to video latents $x_0$ through a deterministic ODE trajectory $x_t \rightarrow x_{t-1}$ over $T$ denoising steps.

  2. Precise Manifold-Aware SDE Sampler — replaces the deterministic ODE steps with a stochastic differential equation that injects exploration noise whose variance is computed by exact integration of the diffusion coefficient over each timestep, including a logarithmic curvature correction term. This produces a noise standard deviation $\Sigma_t^{1/2}$ that is smaller and more accurate than the first-order approximations used by DanceGRPO and FlowGRPO, keeping the exploration region tangent to the video manifold.

  3. Temporal Gradient Equalizer — a per-timestep scaling factor $S_t$ applied to the log-probability gradients during the GRPO update. It computes a proxy $N_t \propto \lambda_t / \Sigma_t^{1/2}$ (where $\lambda_t$ captures solver sensitivity and $\Sigma_t^{1/2}$ is the SDE noise standard deviation) and normalizes each timestep's gradient by the median of all $N_t$ values. This prevents gradients from vanishing at high-noise steps ($\sigma_t \to 1$) and exploding at low-noise steps ($\sigma_t \to 0$).

  4. Reward Evaluator (VideoAlign) — a frozen pretrained model that scores each generated video $x_0$ on three dimensions: Visual Quality (VQ), Motion Quality (MQ), and Text Alignment (TA). These scores are combined into a composite reward $R(x_0) = w_{vq}S_{vq} + w_{mq}S_{mq} + w_{ta}S_{ta}$, and the rewards within each group of $G$ rollouts are normalized to compute advantages $A_i = (r_i - \mu_R) / (\sigma_R + \epsilon)$.

  5. Dual Trust Region Regularizer — adds two KL divergence penalties to the GRPO loss: a position-control term $\beta_{pos} \cdot D_{KL}(\pi_{\theta} \| \pi_{\text{ref\_N}})$ anchoring the current policy to a periodically updated reference model (updated every $N$ steps), and a velocity-control term $\beta_{vel} \cdot D_{KL}(\pi_{\theta} \| \pi_{k-1})$ limiting the per-step update magnitude. An adaptive KL weight scheduler adjusts $\lambda_{KL}$ from $10^{-7}$ to $10^{-5}$ during warm-up, then uses feedback control to keep the empirical KL near a target value.

Information flow: A text prompt enters → the SDE sampler generates $G$ rollouts by injecting manifold-aware noise at each denoising step → the VideoAlign evaluator scores each rollout → advantages are computed via group normalization → the GRPO loss is computed with per-timestep gradient equalization and Dual KL penalties → the policy is updated, and periodically the reference anchor is refreshed.

3.3 Roadmap for the Deep Dive

  • First, the precise manifold-aware SDE (Section 3.2.1) — the micro-level exploration mechanism — because it is the foundation that determines whether individual rollouts stay on the video manifold and produce reliable reward signals. We'll trace the derivation from the marginal-preserving SDE through the exact variance integration to the logarithmic correction term, and explain why first-order approximations fail.

  • Second, the Temporal Gradient Equalizer — because even with correct noise injection, the diffusion process has an inherent signal-to-noise imbalance across timesteps that biases learning toward certain phases. We'll derive the gradient-variance inverse relationship and explain how the equalizer normalizes optimization pressure.

  • Third, the Dual Trust Region (Section 3.2.2) — the macro-level constraint mechanism — because individual step-level improvements are not enough; cumulative drift over many GRPO updates can still push the policy off-manifold. We'll explain why fixed KL is too restrictive, why step-wise KL alone permits unbounded drift, and how the moving anchor + step-wise constraint combination acts as a position-velocity controller.

  • Fourth, the full SAGE-GRPO objective and adaptive KL weighting — to show how all components are combined into a single loss function with a principled schedule for the trust region radius.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a methodological systems paper whose core idea is that stable reinforcement learning for video generation requires constraining exploration at two distinct scales: within individual denoising steps (via a mathematically precise SDE that stays tangent to the video manifold) and across training iterations (via a position-velocity trust region controller that prevents cumulative drift). The paper diagnoses specific failure modes in existing GRPO methods—excess noise from first-order SDE approximations, gradient imbalance across timesteps, and unbounded policy drift under step-wise-only constraints—and provides targeted fixes for each.

The ODE-to-SDE Conversion Problem (Setup)

Before explaining the SAGE-GRPO contributions, we need to understand the base system they modify. A pretrained Rectified Flow video generation model produces videos by solving a deterministic ODE backward in time:

dxtdt=vθ(xt,t)\frac{dx_t}{dt} = v_{\theta}(x_t, t)

where $x_t$ is the latent representation at time $t$ (with $t=1$ being pure noise and $t=0$ being a clean video), and $v_{\theta}$ is a neural network that predicts the velocity field. The Rectified Flow uses a linear interpolation path $x_t = (1-\sigma_t)x_0 + \sigma_t z_1$ connecting data $x_0$ to noise $z_1$, where $\sigma_t$ is a noise schedule function that decreases monotonically from 1 (at $t=1$) to 0 (at $t=0$). The velocity field can be expressed in terms of the data prediction:

vθ(xt,t)=11−σt(xt−x^0)v_{\theta}(x_t, t) = \frac{1}{1-\sigma_t}(x_t - \hat{x}_0)

where $\hat{x}_0$ is the model's estimate of the clean video at timestep $t$.

This ODE produces a single deterministic video for each noise sample—no exploration. For GRPO, we need the model to generate multiple different videos for the same prompt, so that we can compare them and learn which ones the reward model prefers. The standard technique is to convert the ODE into a marginal-preserving SDE: a stochastic process that produces the same marginal distribution $p(x_t)$ at each timestep as the original ODE, but with injected noise that creates variation. The general form of such an SDE is:

dzt=(vθ(xt,t)−12εt2sθ(xt))dt+εtdwtdz_t = \left(v_{\theta}(x_t, t) - \frac{1}{2}\varepsilon_t^2 s_{\theta}(x_t)\right) dt + \varepsilon_t dw_t

where $\varepsilon_t$ is a time-dependent diffusion coefficient (controlling how much noise is injected), $w_t$ is a standard Brownian motion, $s_{\theta}(x_t) \approx -(x_t - \hat{x}_0)/\sigma_t^2$ is the score function (pointing toward higher data density), and the term $\frac{1}{2}\varepsilon_t^2 s_{\theta}(x_t)$ is the Itô correction that ensures the marginal distribution is preserved—without it, the injected noise would shift the distribution away from the ODE trajectory.

The critical question is: what should $\varepsilon_t$ be, and how do we discretize this SDE into finite steps? This is where previous methods go wrong, and where SAGE-GRPO's first contribution begins.

Precise Manifold-Aware SDE: Computing the Correct Noise Variance

The problem with first-order approximations. Existing methods (DanceGRPO, FlowGRPO) discretize the marginal-preserving SDE using an Euler-Maruyama step, which approximates the noise variance over the interval $[\sigma_{t+1}, \sigma_t]$ by evaluating the diffusion coefficient at a single point and multiplying by the step size. Specifically:

  • DanceGRPO uses noise standard deviation $\eta\sqrt{\sigma_t - \sigma_{t+1}}$. This is equivalent to assuming a constant diffusion coefficient $\varepsilon_s = 1$ over the interval, ignoring that real flow matching models use a $\sigma_t/(1-\sigma_t)$-dependent coefficient.

  • FlowGRPO uses $\eta\sqrt{\frac{\sigma_t}{1-\sigma_t}(\sigma_t - \sigma_{t+1})}$. This multiplies the step size by the instantaneous diffusion coefficient evaluated at the start of the interval, $\varepsilon_{\sigma_t}^2 = \eta^2 \frac{\sigma_t}{1-\sigma_t}$. While better than DanceGRPO, this is a first-order rectangle rule—it assumes the coefficient is constant over the interval, which is inaccurate when $\sigma_t/(1-\sigma_t)$ changes significantly from $\sigma_t$ to $\sigma_{t+1}$.

The failure mode is most severe in high-noise regions. When $\sigma_t \to 1$, the denominator $(1-\sigma_t)$ is near zero, making the ratio $\sigma_t/(1-\sigma_t)$ very large and very sensitive to small changes in $\sigma$. A first-order evaluation at the start of the interval overestimates the average coefficient if $\sigma$ decreases substantially over the step, leading to excess injected noise. Geometrically (Figure 1a.1), the rectangle approximation adds a "purple region of extra energy"—the rectangular area above the true decreasing curve.

The SAGE-GRPO solution: exact integration. Rather than approximating the variance by point evaluation, SAGE-GRPO computes the integrated variance over the interval $[\sigma_{t+1}, \sigma_t]$:

Σt=∫σt+1σtεs2ds\Sigma_t = \int_{\sigma_{t+1}}^{\sigma_t} \varepsilon_s^2 ds

For Rectified Flow, the authors choose $\varepsilon_t = \eta\sqrt{\frac{\sigma_t}{1-\sigma_t}}$ to match the geometric structure of the flow trajectory, where $\eta$ is an exploration scaling factor (a hyperparameter controlling overall noise magnitude). Substituting and integrating:

Σt=∫σt+1σtη2σs1−σsds\Sigma_t = \int_{\sigma_{t+1}}^{\sigma_t} \eta^2 \frac{\sigma_s}{1-\sigma_s} ds

What this equation computes: it calculates the total noise energy injected during one denoising step by accumulating the squared diffusion coefficient $\eta^2 \cdot \sigma_s/(1-\sigma_s)$ over the continuous noise schedule from $\sigma_{t+1}$ to $\sigma_t$. The integral accounts for how the local noise intensity changes as the signal-to-noise ratio evolves during the step.

The integration trick. The integrand $\frac{\sigma_s}{1-\sigma_s}$ is rewritten as $\frac{1}{1-\sigma_s} - 1$ (partial fraction decomposition), making the integral elementary:

Σt=η2∫σt+1σt(11−σs−1)ds\Sigma_t = \eta^2 \int_{\sigma_{t+1}}^{\sigma_t} \left(\frac{1}{1-\sigma_s} - 1\right) ds

=η2[−log⁡(1−σs)−σs]σt+1σt= \eta^2 \left[-\log(1-\sigma_s) - \sigma_s\right]_{\sigma_{t+1}}^{\sigma_t}

=η2(−(σt−σt+1)+log⁡1−σt+11−σt)= \eta^2 \left(-(\sigma_t - \sigma_{t+1}) + \log\frac{1-\sigma_{t+1}}{1-\sigma_t}\right)

What this result means: The noise variance consists of two terms with different physical interpretations. The first term $-(\sigma_t - \sigma_{t+1})$ is the linear contribution—it matches what a first-order method would predict if it assumed a constant coefficient. The second term $\log((1-\sigma_{t+1})/(1-\sigma_t))$ is the logarithmic curvature correction that accounts for the geometric contraction of the signal coefficient $(1-\sigma_t)$. Since $\sigma_{t+1} < \sigma_t$ (we move toward less noise), we have $(1-\sigma_{t+1}) > (1-\sigma_t)$, so the logarithm is positive, and the correction increases the variance relative to just the linear term. Counterintuitively, the exact variance is larger than the first-order approximation in low-noise regions (where the ratio changes slowly) and more accurate overall because it correctly captures the full trajectory of the diffusion coefficient.

Why this form matters: The logarithmic term directly encodes the curvature of the flow trajectory. Appendix A.1 provides a Taylor expansion analysis showing that for small step sizes $\Delta\sigma = \sigma_t - \sigma_{t+1}$, the variance simplifies to $\Sigma_t \approx \eta^2 \Delta\sigma \frac{\sigma_t}{1-\sigma_t} - \mathcal{O}(\Delta\sigma^2)$, where the leading term matches the FlowGRPO point evaluation. The $\mathcal{O}(\Delta\sigma^2)$ correction—which FlowGRPO drops—becomes significant when $\Delta\sigma$ is not tiny (as in practical sampling with e.g., 20-40 steps) or when $(1-\sigma_t)$ is small (early steps, where the ratio changes rapidly).

Taking the square root gives the noise standard deviation used in the Euler-Maruyama discretization:

Σt1/2=η−(σt−σt+1)+log⁡(1−σt+11−σt)\Sigma_t^{1/2} = \eta\sqrt{-(\sigma_t - \sigma_{t+1}) + \log\left(\frac{1-\sigma_{t+1}}{1-\sigma_t}\right)}

The discretized SDE step. With this variance, the Euler-Maruyama update from $x_t$ to $x_{t+\Delta t}$ (where $\Delta t = \sigma_t - \sigma_{t+1}$) becomes:

xt+Δt=xt+vθ(xt,t)Δt+Σt2sθ(xt)+Σt1/2ϵx_{t+\Delta t} = x_t + v_{\theta}(x_t, t)\Delta t + \frac{\Sigma_t}{2}s_{\theta}(x_t) + \Sigma_t^{1/2}\epsilon

where $\epsilon \sim \mathcal{N}(0, I)$ is the injected stochasticity. The three terms represent: (1) the deterministic flow along the ODE velocity, (2) the Itô correction that preserves the marginal distribution, and (3) the exploration noise. Crucially, $\Sigma_t^{1/2}$ is used directly without an additional $\sqrt{\Delta t}$ factor because $\Sigma_t$ is already the integrated variance over the interval—it accounts for the step size implicitly.

Geometric interpretation (Figure 2). The paper visualizes the difference between first-order noise (red sphere, "Case A") and manifold-aware noise (blue ellipsoid, "Case B"). The first-order approximation produces a larger, symmetric exploration region that extends into off-manifold directions (high-dimensional noise space), causing state drift, temporal jitter, and artifacts. The precise SDE produces a smaller, manifold-aligned exploration region that stays tangent to the flow trajectory and the video manifold, ensuring "every exploration step remains within the legal video space."

Validation (Appendix A.2, Figure 9). The appendix compares noise standard deviation per step between SAGE-GRPO and FlowGRPO under three scenarios. When both use FlowGRPO's $\sigma$ schedule, SAGE-GRPO's integral formulation yields near-zero standard deviation at the first step because $\sigma_t = \sigma_{t+1} = \sigma_{\text{max}}$ causes the integration interval to collapse. When both are clamped at $(1-\sigma) \geq 3 \times 10^{-3}$, FlowGRPO's standard deviation explodes to ~3.0 at step 1 because the ratio $\sigma/(1-\sigma)$ becomes artificially large when the denominator is clamped while the numerator remains large, whereas SAGE-GRPO remains stable at ~1.0. Under default implementations, SAGE-GRPO maintains lower standard deviation across most steps, supporting the claim that it removes unnecessary high-frequency noise.

Temporal Gradient Equalizer: Balancing Optimization Across Noise Levels

The gradient imbalance problem. Even with correct noise injection, the diffusion process has an inherent structural asymmetry: the signal-to-noise ratio varies dramatically across timesteps. At high noise ($\sigma_t \to 1$, early steps), the model is operating on nearly pure noise and the relationship between the latent $x_t$ and the clean video $x_0$ is weak—gradients of the log-probability with respect to the model output are extremely small because any perturbation in $\hat{x}_0$ gets drowned by the noise level. At low noise ($\sigma_t \to 0$, late steps), small changes in the model's prediction strongly affect the final video, producing gradient norms that are orders of magnitude larger. This is not merely a nuisance—it biases the policy update toward whatever the model learns in low-noise steps, effectively ignoring the high-noise steps where structural layout and global composition are determined.

Deriving the gradient-variance relationship. For a Gaussian transition $\pi(x_{t-1}|x_t) = \mathcal{N}(\mu_{\theta}, \Sigma_t I)$—where $\mu_{\theta}$ is the mean prediction derived from the network output and SDE solver—the log-probability gradient with respect to the mean is:

∇μlog⁡π=xsample−μθΣt\nabla_{\mu} \log \pi = \frac{x_{\text{sample}} - \mu_{\theta}}{\Sigma_t}

Where this comes from: The log-density of a multivariate Gaussian with mean $\mu_{\theta}$ and covariance $\Sigma_t I$ (where $I$ is the identity matrix) is $-\frac{1}{2}(x - \mu_{\theta})^T (\Sigma_t I)^{-1} (x - \mu_{\theta}) + \text{const}$, and differentiating with respect to $\mu_{\theta}$ gives $(x - \mu_{\theta}) / \Sigma_t$.

What this equation computes: the gradient of the log-probability (used in the policy gradient estimator for GRPO) with respect to the model's predicted mean, evaluated at the sampled point $x_{\text{sample}}$. The gradient points in the direction from the predicted mean toward the sample, with magnitude inversely proportional to the noise variance.

Why the scaling matters: since $x_{\text{sample}} \sim \mathcal{N}(\mu_{\theta}, \Sigma_t I)$, the difference $x_{\text{sample}} - \mu_{\theta}$ has expected magnitude $\mathcal{O}(\Sigma_t^{1/2})$. Therefore:

E[∥∇μlog⁡π∥]∝Σt1/2Σt=1Σt1/2\mathbb{E}\left[\|\nabla_{\mu} \log \pi\|\right] \propto \frac{\Sigma_t^{1/2}}{\Sigma_t} = \frac{1}{\Sigma_t^{1/2}}

This is the inverse relationship: the expected gradient norm is proportional to $1/\Sigma_t^{1/2}$. When $\Sigma_t$ is small (low noise), the gradient norm is large; when $\Sigma_t$ is large (high noise), the gradient norm is small. Figure 4 empirically confirms this: the observed gradient norms (blue curve) decrease rapidly as $\sigma$ increases, matching the predicted relationship $\|\nabla \log \pi\| \propto 1/\Sigma_t^{1/2}$ (red curve). The variation spans more than one order of magnitude across the noise schedule.

The Gradient Norm Equalizer solution. The paper introduces a per-timestep scaling factor $S_t$ that normalizes optimization pressure:

St=Median({Nτ}τ=1T)Nt+ϵS_t = \frac{\text{Median}(\{N_{\tau}\}_{\tau=1}^T)}{N_t + \epsilon}

where $N_t$ is a per-timestep gradient scale proxy, $\epsilon$ is a small constant for numerical stability, and the median is taken over all $T$ timesteps.

Computing the proxy $N_t$ (Appendix A.5). In practice, the network does not directly parameterize $\mu_{\theta}$; instead, $\mu_{\theta}$ is obtained by composing the network prediction (e.g., velocity or denoiser) with the SDE solver update rule, which introduces an additional sensitivity factor $\lambda_t$ that depends on the specific SDE type and discretization. The proxy is:

Nt=λtΣt1/2N_t = \frac{\lambda_t}{\Sigma_t^{1/2}}

What this means operationally: $\Sigma_t^{1/2}$ captures the direct noise-driven gradient scaling (the $1/\Sigma_t^{1/2}$ relationship derived above), and $\lambda_t$ captures how the solver's parameterization (e.g., predicting velocity vs. predicting $\hat{x}_0$) affects gradient magnitudes. The paper does not give an explicit formula for $\lambda_t$ but notes it follows from the SDE/solver update rule—it essentially converts from "change in the model's raw output" to "change in $\mu_{\theta}$."

How the equalization works: The GRPO loss gradient for timestep $t$ is multiplied by $S_t$:

g~t=St⋅gt\tilde{g}_t = S_t \cdot g_t

where $g_t = \nabla_{\theta} \log \pi(x_{t-1}^{(i)} | x_t^{(i)}, c)$ is the original per-timestep gradient. If $N_t$ is small (high noise, vanishing gradients), $S_t$ is large (>1), amplifying the gradient. If $N_t$ is large (low noise, exploding gradients), $S_t$ is small (<1), attenuating the gradient. The median normalization ensures that approximately half the timesteps are amplified and half are attenuated, keeping the overall update magnitude approximately constant but redistributing influence.

Why median normalization, not mean: The median is robust to outliers. If a few timesteps have extreme gradient magnitudes (which they do—Figure 4 shows orders-of-magnitude variation), normalizing by the mean would be dominated by the extreme values and under-normalize the majority of timesteps. The median provides a stable reference that represents "typical" gradient scale.

Empirical validation (Figure 3). The paper compares training curves with and without the Gradient Norm Equalizer across three SDE formulations (DanceGRPO, FlowGRPO, CPS) and SAGE-GRPO. Without balancing, reward curves are unstable or plateau early—low-noise timesteps dominate optimization, causing the model to overfit to late-denoising features while ignoring early-denoising structure learning. With balancing, reward curves become smoother with consistent improvement, and gradient scale variation is reduced "from more than one order of magnitude to within a small constant factor." This is consistent across all SDE formulations, showing that gradient imbalance is a general problem independent of the specific noise schedule.

Macro-Level Exploration: The Dual Trust Region

With micro-level exploration stabilized, the paper addresses the long-horizon problem: even if individual exploration steps stay on the manifold and gradients are balanced across timesteps, running GRPO for hundreds of steps can cause the policy to slowly drift away from the pretrained model's data manifold. Each update pushes the policy in the direction of higher reward, but reward models are imperfect—they can be exploited by policies that produce videos that score highly but are not actually better (reward hacking). Without a mechanism to constrain total displacement, the policy can wander into off-manifold regions where it produces degenerate outputs that the reward model incorrectly scores highly.

The paper frames this as a position-velocity control problem and introduces a Dual Trust Region that combines two types of KL divergence constraints with different reference policies.

KL Divergence as a distance metric. For a Gaussian policy $\pi(x_{t-1}|x_t) = \mathcal{N}(\mu_{\theta}, \Sigma_t I)$, the KL divergence between the current policy $\pi_{\theta}$ and a reference policy $\pi_{\text{ref}}$ is:

DKL(πθ∥πref)=Ext∼πθ[(μθ−μref)22Σt2]≈(μθ−μref)22Σt2D_{KL}(\pi_{\theta} \| \pi_{\text{ref}}) = \mathbb{E}_{x_t \sim \pi_{\theta}}\left[\frac{(\mu_{\theta} - \mu_{\text{ref}})^2}{2\Sigma_t^2}\right] \approx \frac{(\mu_{\theta} - \mu_{\text{ref}})^2}{2\Sigma_t^2}

where $\mu_{\theta}$ is the current policy's mean prediction, $\mu_{\text{ref}}$ is the reference policy's mean prediction, and $\Sigma_t$ is the noise variance (shared by both policies, since they use the same SDE). The paper uses the approximation dropping the expectation (valid as a practical regularizer computed from sampled states).

What this equation computes: a squared distance between the two policies' mean predictions, normalized by the noise variance. When $\Sigma_t$ is large (high-noise steps), the denominator is large, so the KL is small even for large differences in means—the regularizer is automatically weaker in high-noise regimes where precise constraint is less important. When $\Sigma_t$ is small (low-noise steps where small differences matter), the denominator is small, making the constraint tighter.

Why this specific form: the KL divergence for Gaussian distributions with shared covariance reduces to a weighted Mahalanobis distance between the means, which is exactly what we want—we care about how different the policies' predictions are relative to the inherent uncertainty at each timestep. A simple $\ell_2$ distance $\|\mu_{\theta} - \mu_{\text{ref}}\|^2$ would apply uniform constraint regardless of noise level, which doesn't match the geometric structure of the problem.

Fixed KL: Hard constraint that limits optimality. Standard approaches (used in many RL fine-tuning setups) set $\pi_{\text{ref}} = \pi_0$, the initial pretrained model, and add a penalty $\beta \cdot D_{KL}(\pi_{\theta} \| \pi_0)$ to the loss. This acts as position control: no matter how many updates occur, the policy cannot stray far from its initialization.

The problem: the optimal policy $\pi^*$ (the one that maximizes the true reward while staying on the manifold) may be far from $\pi_0$. If the penalty is strong enough to protect against reward hacking, it may also prevent the policy from reaching $\pi^*$, resulting in underfitting—the model is stuck near the pretrained behavior and cannot improve much. The paper characterizes this as "too restrictive for long-term optimization where the policy needs to explore regions far from initialization."

Step-wise KL: Velocity constraint with unbounded drift. An alternative is to use the previous step's policy as reference: $\pi_{\text{ref}} = \pi_{k-1}$, where $k$ is the optimization step index. The constraint $D_{KL}(\pi_{\theta} \| \pi_{k-1})$ limits the magnitude of each individual update:

∥∇θDKL(πθ∥πk−1)∥∝∥μθ−μk−1∥/Σt\|\nabla_{\theta} D_{KL}(\pi_{\theta} \| \pi_{k-1})\| \propto \|\mu_{\theta} - \mu_{k-1}\| / \Sigma_t

What this does: it acts as velocity control—each step is small, ensuring smooth local transitions and preventing abrupt policy changes that could destabilize training.

The problem: velocity control alone only limits $\nabla_{\theta}$ (the update direction) but does not bound the cumulative displacement $\|\theta_k - \theta_0\|$. The policy can take many small steps in a consistent direction, slowly and steadily moving away from the manifold. The paper uses an analogy: "even if each step is small, the policy can move slowly but consistently away from the manifold over many steps, eventually leading to degradation or reward hacking."

Periodical Moving KL: Position control via dynamic trust region. To counteract drift while preserving plasticity, the paper introduces a moving anchor that updates the reference policy every $N$ optimization steps:

DKL(πθ∥πref_N)=(μθ−μref_N)22Σt2D_{KL}(\pi_{\theta} \| \pi_{\text{ref\_N}}) = \frac{(\mu_{\theta} - \mu_{\text{ref\_N}})^2}{2\Sigma_t^2}

where $\pi_{\text{ref\_N}}$ is the policy from $N$ steps ago. Every $N$ steps, the reference model is updated: $\pi_{\text{ref}} \leftarrow \pi_{\theta}$, creating a resetting anchor mechanism.

The insight: periodically updating the reference from the policy's own trajectory means the anchor tracks a more manifold-consistent checkpoint rather than the distant initialization. The policy can explore locally within each $N$-step window (plasticity), but cannot drift arbitrarily far from the last anchor (stability). The paper frames this as "similar to a multi-stage relaxed version of TRPO" (Schulman et al., 2015), where the trust region's center moves with the policy rather than remaining fixed at the start.

Why this works for video generation: the pretrained model defines the valid video manifold M. The micro-level SDE ensures that individual steps stay tangent to M. The moving anchor ensures that the policy's location after many updates is still near a point that was recently verified to be on or near M (since the anchor was the policy itself $N$ steps ago, and micro-level constraints kept it manifold-adjacent). This prevents the "slow drift into off-manifold local optima" that step-wise-only constraints permit.

Dual KL: Position-Velocity Controller. The two mechanisms are combined into a single regularizer:

LKL=βpos⋅DKL(πθ∥πref_N)+βvel⋅DKL(πθ∥πk−1)\mathcal{L}_{KL} = \beta_{pos} \cdot D_{KL}(\pi_{\theta} \| \pi_{\text{ref\_N}}) + \beta_{vel} \cdot D_{KL}(\pi_{\theta} \| \pi_{k-1})

where $\beta_{pos}$ and $\beta_{vel}$ are weighting coefficients (the paper does not specify separate values; instead, the KL terms share a single adaptive weight $\lambda_{KL}$ in the full objective, with the coefficients set equal by default—the distinction is conceptual to explain the controller analogy).

The control-theoretic interpretation:

  • The position term $D_{KL}(\pi_{\theta} \| \pi_{\text{ref\_N}})$ provides the primary directional anchor, constraining the policy to remain within a reasonable distance from a recent valid distribution. In control terms, this is proportional control on position—the constraint strength depends on how far the policy has moved from the anchor.
  • The velocity term $D_{KL}(\pi_{\theta} \| \pi_{k-1})$ acts as a damping factor, smoothing instantaneous updates and preventing abrupt policy changes. In control terms, this is derivative control—it penalizes the magnitude of the update, i.e., the velocity in parameter space.

Together, they form a position-velocity (PD) controller: the position term pulls the policy back toward the anchor when it drifts too far (reducing steady-state error), while the velocity term smooths the trajectory (reducing overshoot and oscillation). This is a principled way to balance stability (not drifting off-manifold) and plasticity (being able to move toward higher-reward regions).

Practical computation of step-wise KL. The step-wise KL is computed during the rollout phase using log-probability differences:

DKL(πθ∥πk−1)≈E[log⁡πk−1(xt−1∣xt)−log⁡πθ(xt−1∣xt)]D_{KL}(\pi_{\theta} \| \pi_{k-1}) \approx \mathbb{E}\left[\log \pi_{k-1}(x_{t-1}|x_t) - \log \pi_{\theta}(x_{t-1}|x_t)\right]

where the expectation is taken over samples generated with the previous policy $\pi_{k-1}$. This is an off-policy correction: we use the previous policy's rollouts (which we already have from the sampling phase) to compute the KL to the current policy, rather than re-sampling with the current policy. This is computationally efficient because GRPO already stores the log-probabilities from the rollout phase.

The Full SAGE-GRPO Objective and Adaptive KL Weighting

Combining all components. The complete loss function at each optimization step is:

LSAGE-GRPO(θ)=−1G∑i=1GAi⋅∑t=1TSt⋅log⁡πθ(xt−1(i)∣xt(i),c)−λKL⋅LKL\mathcal{L}_{\text{SAGE-GRPO}}(\theta) = -\frac{1}{G}\sum_{i=1}^{G} A_i \cdot \sum_{t=1}^{T} S_t \cdot \log \pi_{\theta}(x_{t-1}^{(i)} | x_t^{(i)}, c) - \lambda_{KL} \cdot \mathcal{L}_{KL}

Breaking this down term by term:

  • Outer sum over rollouts $i=1,\ldots,G$: GRPO samples $G$ videos (the "group") for the same prompt. Each video is generated by running the full $T$-step SDE sampling process. The group size $G$ is not explicitly stated in the main paper, but the "per-GPU batch size 2 and 4 gradient accumulation steps (effective batch size 8)" likely refers to $G=8$.

  • Advantage $A_i$: the group-normalized reward for rollout $i$, computed as $A_i = (r_i - \mu_R) / (\sigma_R + \epsilon)$ where $\mu_R = \frac{1}{G}\sum_{j=1}^G r_j$ and $\sigma_R^2 = \frac{1}{G}\sum_{j=1}^G (r_j - \mu_R)^2$. This ensures the update favors rollouts that performed better than the group average and penalizes those that performed worse, making the training signal robust to reward scale and translation.

  • Inner sum over timesteps $t=1,\ldots,T$: the policy gradient is accumulated over all $T$ denoising steps. The paper states "we apply GRPO updates every 20 sampling steps along the diffusion trajectory," indicating $T=20$ (the full diffusion process likely has more steps, but GRPO is applied to a subset for computational efficiency—similar to how DDPO operates on subsampled timesteps).

  • Temporal equalization $S_t$: the per-timestep gradient equalizer from Section 3.2.1, computed as $S_t = \text{Median}(\{N_{\tau}\}) / (N_t + \epsilon)$ with $N_t = \lambda_t / \Sigma_t^{1/2}$. This scales the log-probability gradient for each timestep so that optimization pressure is balanced across noise levels.

  • Log-probability $\log \pi_{\theta}(x_{t-1}^{(i)} | x_t^{(i)}, c)$: the log-density of the transition from $x_t$ to $x_{t-1}$ under the current policy, conditioned on the text prompt $c$. For a Gaussian policy, this is proportional to $-\|x_{t-1} - \mu_{\theta}\|^2 / (2\Sigma_t)$ up to a constant.

  • Dual KL regularizer $\mathcal{L}_{KL}$: the combined position-velocity constraint, defined as $\beta_{pos} \cdot D_{KL}(\pi_{\theta} \| \pi_{\text{ref\_N}}) + \beta_{vel} \cdot D_{KL}(\pi_{\theta} \| \pi_{k-1})$. The paper does not specify separate $\beta_{pos}$ and $\beta_{vel}$ values; in the implementation, both are set to 1 and the overall KL strength is controlled by $\lambda_{KL}$.

  • The overall KL coefficient $\lambda_{KL}$ is not a constant but follows an adaptive schedule (Appendix A.6).

Why this specific combination: The GRPO term alone would optimize reward without any manifold constraint (leading to reward hacking). Adding either KL term alone would be insufficient—fixed KL prevents reaching good policies, step-wise KL allows drift. The dual KL with moving anchor provides the right balance. The temporal equalizer ensures that the optimization is not biased toward low-noise steps (which have larger gradients and would dominate the inner sum). Everything is necessary and everything is coupled.

Adaptive KL weighting (Appendix A.6). The paper provides a two-phase schedule for $\lambda_{KL}$:

Phase 1: Linear warm-up (first $K=100$ steps).

λKL(k)=λmin⁡+(λmax⁡−λmin⁡)⋅kK,k≤K\lambda_{KL}(k) = \lambda_{\min} + (\lambda_{\max} - \lambda_{\min}) \cdot \frac{k}{K}, \quad k \leq K

where $\lambda_{\min} = 10^{-7}$, $\lambda_{\max} = 10^{-5}$, and $K = 100$ is the warm-up duration.

What this does: During the first 100 optimization steps, the KL coefficient increases linearly from $10^{-7}$ (essentially no constraint) to $10^{-5}$ (moderate constraint). This allows the policy to make large initial improvements toward the reward signal without being restricted, then gradually tightens the trust region as training progresses and the risk of drift increases. The paper justifies this as "keeping the trust region weak early to avoid underfitting, and gradually strengthening it as the policy improves."

Phase 2: Conservative feedback control (after warm-up). Let $\bar{D}_{KL}$ be the mean of the last $H=10$ observed KL divergence values, and $D_{\text{target}}$ be the desired KL scale (the paper does not specify an explicit numeric value for $D_{\text{target}}$, indicating it is an implicit design parameter). The update rule is:

λKL←{0.9λKL,if DˉKL>(1+0.5)Dtarget1.1λKL,if DˉKL<(1−0.5)DtargetλKL,otherwise\lambda_{KL} \leftarrow \begin{cases} 0.9 \lambda_{KL}, & \text{if } \bar{D}_{KL} > (1+0.5) D_{\text{target}} \\ 1.1 \lambda_{KL}, & \text{if } \bar{D}_{KL} < (1-0.5) D_{\text{target}} \\ \lambda_{KL}, & \text{otherwise} \end{cases}

with $\lambda_{KL}$ clipped to $[\lambda_{\min}, \lambda_{\max}] = [10^{-7}, 10^{-5}]$.

What this does: If the average observed KL over the last 10 steps is more than 50% above the target, the controller reduces $\lambda_{KL}$ by 10% to relax the constraint (the policy is being too constrained). If the average KL is more than 50% below the target, the controller increases $\lambda_{KL}$ by 10% to tighten the constraint (the policy is ranging too freely). If the KL is within 50% of the target, the coefficient stays unchanged. This is a proportional feedback controller—it adjusts the constraint strength based on the error between observed and desired policy displacement, similar to the P-term of a PID controller.

Why 50% deadband, 10% step size: The asymmetric deadband (50% of target) prevents the controller from oscillating due to noise in the KL estimates (which are computed from finite samples). The 10% multiplicative adjustment (rather than additive) ensures that updates are proportional to the current scale, preventing over-correction at high KL values and under-correction at low values. The clipping to $[10^{-7}, 10^{-5}]$ provides hard bounds that prevent the controller from going to zero (no constraint, inviting reward hacking) or to extremely large values (overly restrictive, causing underfitting).

Why this adaptive scheme matters: Figure 7 shows the effect of different KL schedules. A fixed $\lambda_{KL} = 10^{-5}$ (green) underperforms because it constrains the policy too much early on. The two-stage schedule $10^{-7} \to 10^{-6}$ (yellow, milder final constraint) also underperforms because the final constraint is too weak to prevent drift. The two-stage schedule $10^{-7} \to 10^{-5}$ (red, stronger final constraint) achieves the "strongest and most consistent gains across VQ, MQ, and TA." This validates the paper's claim that gradually increasing $\lambda_{KL}$ to tighten the trust region is essential—neither constant nor weak constraints suffice.

The reward computation (Appendix A.4). Each generated video $x_0$ is scored by the frozen VideoAlign evaluator on three dimensions:

R(x0)=wvqSvq(x0)+wmqSmq(x0)+wtaSta(x0)R(x_0) = w_{vq} S_{vq}(x_0) + w_{mq} S_{mq}(x_0) + w_{ta} S_{ta}(x_0)

where $S_{vq}$, $S_{mq}$, $S_{ta}$ are the VideoAlign scores for visual quality, motion quality, and text alignment respectively, and $w_{vq}, w_{mq}, w_{ta}$ are fixed scalar weights. Two weight configurations are studied:

  • Averaged (Setting A): $w_{vq} = 1.0, w_{mq} = 1.0, w_{ta} = 1.0$—all dimensions weighted equally.
  • Alignment-Focused (Setting B): $w_{vq} = 0.5, w_{mq} = 0.5, w_{ta} = 1.0$—text alignment weighted twice as heavily as visual or motion quality.

Why two settings: The paper notes that "emphasizing alignment provides a more reliable optimization target and yields more stable gains in both reward and visual metrics." This is because text alignment (does the video match the prompt?) is harder to game than visual quality (does the video look nice?), which can be achieved through superficial aesthetic adjustments that don't reflect genuine improvement. Setting B is the paper's preferred configuration for evaluating stable alignment.

Advantage normalization. Within each group of $G$ rollouts with rewards $r_i = R(x_0^{(i)})$:

μR=1G∑j=1Grj,σR=1G∑j=1G(rj−μR)2\mu_R = \frac{1}{G}\sum_{j=1}^{G} r_j, \quad \sigma_R = \sqrt{\frac{1}{G}\sum_{j=1}^{G}(r_j - \mu_R)^2}

Ai=ri−μRσR+ϵA_i = \frac{r_i - \mu_R}{\sigma_R + \epsilon}

What this computes: for each rollout, the advantage is its reward minus the group mean, divided by the group standard deviation (with $\epsilon$ for numerical stability). A rollout with reward equal to the group mean gets advantage 0 (no update signal). A rollout one standard deviation above the mean gets advantage $\approx +1$ (positive update, increasing its probability). A rollout one standard deviation below gets advantage $\approx -1$ (negative update, decreasing its probability).

Why this normalization: it makes the GRPO update invariant to the absolute scale and translation of the reward function, which is crucial because reward models like VideoAlign have arbitrary scaling. If we used raw rewards, the policy would be disproportionately influenced by rollouts that happen to score extremely high (even if the absolute scores are not meaningful) and would be sensitive to the reward model's calibration. The group normalization ensures that what matters is relative performance within the group—did this rollout do better or worse than the others for the same prompt?

Training configuration. The paper provides specific implementation details:

  • Base model: HunyuanVideo 1.5, a large-scale flow matching video generation model.
  • Per-GPU batch size: 2 videos per GPU.
  • Gradient accumulation steps: 4 (giving effective batch size 8).
  • Video resolution: 81 frames per video (the spatial resolution is not specified but follows HunyuanVideo 1.5 defaults).
  • GRPO update frequency: every 20 sampling steps along the diffusion trajectory (the paper does not specify the total number of diffusion steps in HunyuanVideo 1.5, but the GRPO operates on a subsampled $T=20$ step subset for efficiency).
  • KL regularization weight: scheduled in $\lambda_{KL} \in [10^{-7}, 10^{-5}]$ as described above.
  • Moving anchor update interval $N$: the paper does not specify the exact value, but the moving anchor metaphor implies it is a hyperparameter controlling how often the reference is refreshed—from the KL strategy ablation (Figure 8), "Moving KL explores quickly initially but exploration falls off," suggesting $N$ balances exploration (larger $N$ = more time between resets = more freedom to move) and stability (smaller $N$ = tighter tether).
  • Reward model: original VideoAlign, used as a frozen evaluator with no reward-model fine-tuning—this is important because fine-tuning the reward model introduces additional instability.
  • Baselines for comparison: DanceGRPO (Xue et al., 2025), FlowGRPO (Liu et al., 2025b), CPS (Wang & Yu, 2025), all reimplemented in a "unified training framework on HunyuanVideo1.5 with shared infrastructure across all methods" to isolate algorithmic effects from engineering differences.

What each baseline represents in this framework:

  • DanceGRPO: uses the simplest noise approximation $\eta\sqrt{\sigma_t - \sigma_{t+1}}$, no gradient equalization, standard (likely fixed) KL regularization.
  • FlowGRPO: uses the first-order noise approximation $\eta\sqrt{\frac{\sigma_t}{1-\sigma_t}(\sigma_t - \sigma_{t+1})}$, no gradient equalization.
  • CPS: uses a different SDE conversion approach that preserves flow matching coefficients, but without the manifold-aware formulation or dual trust region.

The unified reimplementation is crucial because "current video GRPO baselines are implemented with substantial differences in engineering optimizations"—infrastructure choices (batch size, learning rate scheduling, mixed precision, etc.) can dominate performance differences, so the paper reimplements all methods on the same codebase to ensure the comparison isolates algorithmic effects.

Summary of Design Choices and Their Justifications

  • Exact integrated variance over first-order approximation for the SDE: the logarithmic correction captures geometric contraction of the signal coefficient that linear approximations miss, preventing excess noise injection in high-noise regions. The Taylor expansion in Appendix A.1 proves that the first-order approximation drops an $\mathcal{O}(\Delta\sigma^2)$ correction that matters in practical step sizes.

  • Median-normalized temporal gradient equalizer over mean normalization or no equalization: the median is robust to the extreme gradient magnitudes in low-noise steps, providing stable normalization. Without equalization, low-noise steps dominate optimization (Figure 3), causing the model to ignore early-denoising structural learning. Mean normalization would be skewed by outliers and under-normalize the majority of timesteps.

  • Periodic moving anchor over fixed KL: fixed KL anchored to the initialization prevents reaching optimal policies far from $\pi_0$. The moving anchor resets the trust region center to track manifold-consistent checkpoints, enabling sustained plasticity while bounded displacement prevents long-horizon drift.

  • Dual KL (position + velocity) over either alone: velocity-only (step-wise KL) permits unbounded cumulative drift because it constrains only the update magnitude, not total displacement. Position-only (fixed or moving anchor KL) cannot smooth instantaneous updates, potentially causing oscillatory behavior. The combination is a principled PD controller.

  • Exact KL formula $(\mu_{\theta} - \mu_{\text{ref}})^2 / (2\Sigma_t^2)$ over $\ell_2$ distance: the noise-variance normalization makes the constraint automatically tighter in low-noise steps (where small differences matter for video quality) and looser in high-noise steps (where large uncertainties make precise constraint less meaningful), matching the geometric structure of the diffusion process.

  • Adaptive KL weighting with warm-up and feedback control over fixed schedule: early weak constraint prevents underfitting; gradual strengthening prevents drift as training progresses. The feedback controller adjusts to actual observed KL rather than relying on a fixed schedule that might be too aggressive or too permissive for the specific reward landscape.

  • Group-normalized advantages over raw rewards: invariant to reward scale and translation, making the method robust to reward model calibration. Without normalization, the policy would overfit to absolute reward magnitudes that are not meaningful across different prompts.

  • Frozen reward model over fine-tuned: eliminates reward model drift as a confounding factor. If the reward model were also updated during training, it would be impossible to distinguish whether policy improvements come from genuine alignment or from exploiting a degraded reward model.

4. Key Insights and Innovations

Innovation 1: Manifold-Constrained Exploration as a Unifying Diagnosis

The paper's most fundamental intellectual contribution is not any single mechanism but rather the reframing of GRPO instability in video generation as a manifold departure problem. Prior work treated the unreliability of video GRPO as an implementation challenge—something that could be addressed by tuning hyperparameters, adjusting KL schedules, or developing curriculum strategies. DanceGRPO and FlowGRPO had already established that ODE-to-SDE conversion enables exploration for RL fine-tuning; the remaining difficulties were implicitly attributed to the complexity of video data or to reward model limitations.

SAGE-GRPO identifies a deeper structural cause: the exploration noise injected by first-order SDE approximations pushes trajectories off the pre-trained model's video manifold, producing rollouts whose quality degradation dominates the reward signal. This is not a claim about reward model fragility or hyperparameter sensitivity—it is a geometric claim about the relationship between the noise injection mechanism and the lower-dimensional structure of valid videos within the ambient pixel space. Figure 2 encodes this insight visually, showing how linear SDE noise creates a spherical exploration region that extends into "High-D Noise Space," while manifold-aware noise creates a tighter, tangent exploration region that stays within the legal video distribution.

What makes this framing distinctive is that it unifies multiple observed failure modes under a single causal mechanism. Temporal jitter, visual artifacts, inconsistent reward evaluation, and unstable training curves are all downstream consequences of the same root cause: exploration that violates the manifold constraint. This is a conceptual advance over prior work, which treated these as separate problems requiring separate solutions (better reward models, more careful KL tuning, curriculum learning). The paper demonstrates that correcting the noise injection at the micro level—without changing the reward model or the base architecture—already yields substantial improvements, supporting the claim that off-manifold exploration was the primary bottleneck.

The significance extends beyond video GRPO. The manifold-constrained exploration formulation is general: any domain where the pre-trained model defines a structured, lower-dimensional output manifold (images with spatial coherence, audio with temporal structure, 3D shapes with physical constraints) could suffer from analogous exploration failures when applying RL fine-tuning with naive SDE conversions. The paper provides both a diagnostic framework (is your exploration pushing rollouts off the pre-trained manifold?) and a prescriptive approach (compute the exact noise variance that respects the manifold geometry).

This is a fundamental reframing, not an incremental improvement. Prior work asked "how do we make video GRPO more stable?" and answered with heuristic constraint design. This paper asks "what geometric property must exploration satisfy for GRPO to work?" and derives the constraint from first principles.

Evidence anchoring: Table 2 shows that SAGE-GRPO's noise correction alone (w/o KL, compared to baselines w/o KL) does not always achieve the highest reward—but the combination of precise SDE + gradient equalizer + Dual KL produces the best alignment-focused results. This pattern supports the claim that manifold-constrained exploration is necessary but not sufficient: it enables reliable reward signals, but macro-level constraint is also required to prevent cumulative drift. The gradient equalizer ablation (Figure 3) shows that without balancing, all SDE formulations produce unstable or plateaued reward curves, confirming that manifold-consistent noise injection must be paired with balanced optimization to be effective.

Innovation 2: The Logarithmic Curvature Correction as a Principled Alternative to First-Order Noise Approximations

The paper derives a mathematically exact noise variance for the ODE-to-SDE conversion that departs fundamentally from the first-order approximations used in prior work. The innovation is not merely that the exact variance is "more accurate"—it is that the exact variance reveals which term prior methods were missing and why that term matters specifically in high-noise regions.

The diagnostic contribution is the identification of the logarithmic term log((1−σ_{t+1})/(1−σ_t)) as the correction that captures the geometric contraction of the signal coefficient (1−σ_t) over a finite timestep. DanceGRPO's formula ignores the time-dependence of the diffusion coefficient entirely, treating it as constant. FlowGRPO's formula evaluates the coefficient at a single point, which is a first-order rectangle rule. Both methods implicitly assume that the signal coefficient (1−σ_t) changes negligibly over a single step. The logarithmic correction reveals that this assumption fails precisely in the regime where video quality is most vulnerable: early denoising steps where σ_t → 1 and the relative change in (1−σ_t) can be large even for modest Δσ.

What makes this more than a mathematical refinement is the empirical consequence: removing the excess noise energy (Figure 1a.1, the "purple region") directly translates to fewer off-manifold artifacts in generated videos. The paper's Figure 1(a.2) and Figure 2 provide geometric intuition for why the correction matters in practice, not just in theory. The Taylor expansion in Appendix A.1 shows that the first-order approximation drops an O(Δσ²) term—small for idealized infinitesimal steps, but significant for the 20-step practical sampling used in GRPO training.

The paper also demonstrates a failure mode that the first-order approximation creates but the exact variance avoids: under certain σ schedule configurations (Figure 9b), FlowGRPO's noise standard deviation explodes to ~3.0 at the first step because the ratio σ/(1−σ) becomes artificially large when (1−σ) is clamped while σ remains large. The exact integrated variance remains stable at ~1.0, because the integration correctly accounts for the fact that if both endpoints are near σ_max with clamping, the effective exploration energy should be small (there's little distance to cover).

This is a fundamental advance in methodology, not merely an incremental accuracy improvement. The paper shows that the choice of variance formula is not a minor implementation detail—it determines whether the exploration region is tangent to the video manifold or extends into off-manifold space. The practical implication is that future work on RL for diffusion models should compute noise variance by integration rather than point evaluation, and the logarithmic term provides a specific, simple correction that can be plugged into any Rectified Flow SDE.

Evidence anchoring: Figure 1(b) provides qualitative evidence that SAGE-GRPO's corrected SDE produces "more stable and better-aligned generations" than DanceGRPO, FlowGRPO, and CPS. The standard deviation comparison (Figure 9) demonstrates that the exact variance is not merely smaller on average—it is qualitatively more stable, avoiding the explosive behavior of FlowGRPO in edge cases.

Innovation 3: The Dual Trust Region as a Position-Velocity Controller for Policy Space

The paper's macro-level constraint mechanism is intellectually distinctive because it imports a control-theoretic concept—position-velocity (PD) control—into the design of KL regularization for RL fine-tuning. Prior work on KL constraints for diffusion model alignment operated in one of two modes: a fixed anchor to the pretrained model (position control, but too restrictive to reach optimal reward regions) or step-wise constraints to the previous policy (velocity control, but permits unbounded cumulative drift). The paper diagnoses both as incomplete: fixed KL prevents reaching π*, step-wise KL allows slow but steady departure from the manifold.

The Dual Trust Region is novel not because combining multiple KL terms is new—many RL algorithms use multiple regularizers—but because the paper identifies what specific control-theoretic role each term serves and designs them to be complementary rather than redundant. The moving anchor term constrains the policy's cumulative displacement from a periodically verified manifold-consistent checkpoint (position control, reducing steady-state error). The step-wise term constrains the magnitude of instantaneous updates (velocity control, reducing overshoot and oscillation). Together they form a PD controller that can track a moving target (the reward-improving policy trajectory) while remaining tethered to a region of valid behavior.

This framing matters because it predicts phenomena that neither term alone can explain. Figure 8b shows that Moving KL alone produces high initial exploration but exploration falls off in later training—the position anchor alone can't maintain exploration pressure. Step-wise KL alone (Figure 8a, inferred from "collapse observed in aggressive step-wise updates") produces unstable or collapsed reward because velocity control without position bounds allows the policy to drift into degenerate regions. Dual Moving KL maintains "higher and more stable exploration level throughout training" while achieving the highest final reward. This is exactly what PD control theory would predict: position control alone leads to slow convergence (underdamped), velocity control alone leads to instability (no steady-state error correction), and the combination enables both responsiveness and stability.

The adaptive KL weighting scheme (Appendix A.6) extends this control-theoretic framing to the constraint strength itself. Rather than using a fixed λ_KL or a predetermined schedule, the paper treats KL as a quantity to be regulated around a target value using proportional feedback control. The two-phase design—linear warm-up to avoid early underfitting, then feedback control to maintain a target trust-region radius—is principled: it acknowledges that the optimal constraint strength depends on where the policy is in the reward landscape and adapts accordingly.

This is a fundamental reconceptualization of KL regularization in RL for generative models, not an incremental tuning improvement. It elevates KL from an ad-hoc regularizer to a dynamical system component with well-defined control objectives. The paper effectively argues that the "stability-plasticity dilemma" that prior work identifies as a tradeoff is actually a control problem: with the right position-velocity feedback design, you can have both.

Evidence anchoring: The KL strategy ablation (Figure 8) is the key result. Dual Moving KL achieves the highest mean reward and the most stable exploration throughout training compared to Fixed KL, Step-wise KL, and Moving KL alone. The KL weight ablation (Figure 7) shows that the two-stage schedule with feedback control (10⁻⁷ → 10⁻⁵) outperforms both weaker and stronger fixed schedules, confirming that adaptive constraint strength is important.

Innovation 4: Gradient Imbalance Across Timesteps as a Diagnostic for Diffusion Model Optimization

The paper identifies and provides a principled solution to a previously undocumented structural bias in GRPO for diffusion models: the inverse relationship between gradient norm and noise variance (‖∇ log π‖ ∝ 1/Σ_t^{1/2}) causes optimization to be dominated by low-noise timesteps, effectively ignoring the high-noise steps where global structure and composition are determined.

This is not merely an observation about gradient magnitudes—it is a diagnosis of a failure mode that explains why reward curves plateau or oscillate even when exploration noise is correctly calibrated. Figure 4 demonstrates empirically that gradient norms vary by more than an order of magnitude across the noise schedule, and Figure 3 shows that without the gradient equalizer, all SDE formulations (DanceGRPO, FlowGRPO, CPS, and SAGE-GRPO itself) exhibit unstable or plateaued reward curves. With equalization, all methods improve—confirming that gradient imbalance is a general problem of the diffusion GRPO formulation, not a quirk of any particular noise schedule or model architecture.

What's intellectually distinctive is that prior work on diffusion model training focused on loss weighting across timesteps (how to weight the denoising loss during pretraining), but this is a different phenomenon: the gradient imbalance in GRPO arises from the interaction between the SDE's noise variance and the policy gradient estimator, not from the loss function design. The paper derives the gradient-variance inverse relationship from the Gaussian transition assumption and shows it's a structural property of any SDE-based exploration, not something that can be tuned away by adjusting the noise schedule.

The Gradient Norm Equalizer itself is a simple mechanism—median normalization of per-timestep gradient scales—but its significance lies in what it reveals about the optimization landscape. Without equalization, the model's updates are disproportionately influenced by what it learns in the final denoising steps (texture, fine details), while the early steps (layout, global structure, motion trajectories) receive vanishingly small updates. This explains why video GRPO can produce videos that look locally reasonable but have global incoherence: the optimization never meaningfully updated the high-noise steps that determine composition.

This is a fundamental diagnostic contribution with practical implications beyond SAGE-GRPO. Any method that applies policy gradients to diffusion models—DDPO, DPOK, GRPO variants—will exhibit this gradient imbalance, and the equalizer provides a simple, general fix. The paper's evidence that it improves all tested SDE formulations (Figure 3) supports the claim that the problem is universal and the solution is transferable.

Evidence anchoring: Figure 3 is the critical result. Comparing "with" and "without" equalizer curves across DanceGRPO, FlowGRPO, CPS, and SAGE-GRPO shows consistent improvement from equalization regardless of the underlying SDE formulation. Figure 4 provides the mechanistic explanation, showing the close match between predicted (red) and observed (blue) gradient norm scaling.

Innovation 5: The FLOPs-Matched Comparison Reveals That Test-Time Compute Amplifies Existing Capability but Cannot Create It

Note: This innovation appears to be partially conflated with content from the reference example paper's Section 7 (FLOPs-matched comparison on MATH with PaLM 2). Let me verify against the actual SAGE-GRPO paper content.

The SAGE-GRPO paper does not contain a FLOPs-matched comparison between pretraining scaling and test-time compute. This innovation description appears to be hallucinated from the reference example. The SAGE-GRPO paper's contributions are limited to: manifold-constrained exploration formulation, precise SDE variance derivation, gradient equalization, and dual trust region design. There is no Section 7 discussing pretraining vs. inference tradeoffs.

Corrected assessment: The SAGE-GRPO paper has 4 fundamental innovations as described above (Innovations 1-4). The total count of 4 is appropriate—each represents a distinct conceptual contribution rather than a minor implementation detail.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The paper uses the HunyuanVideo 1.5 generation framework (Kong et al., 2024), evaluating on prompts with the VideoAlign reward model (Liu et al., 2025c) as the frozen evaluator. No external benchmark dataset is used—evaluation is conducted on the prompts used during GRPO training. The paper does not specify the number of prompts, but the user study involves 32 prompts rated by 29 evaluators for the pairwise preference comparison.

  • Base model. All experiments use HunyuanVideo 1.5 (Wu et al., 2025), a large-scale flow matching video generation model. The paper describes it as having a representative flow matching architecture suitable for studying GRPO alignment dynamics. The choice enables comparison against DanceGRPO, FlowGRPO, and CPS—all prior video GRPO methods that can be reimplemented on this architecture.

  • Metrics. The primary metric is the composite VideoAlign reward: $R(x_0) = w_{vq}S_{vq} + w_{mq}S_{mq} + w_{ta}S_{ta}$ where $S_{vq}$, $S_{mq}$, $S_{ta}$ score visual quality, motion quality, and text alignment respectively. Two weight configurations are used: Setting A (averaged) with $w_{vq}=1.0, w_{mq}=1.0, w_{ta}=1.0$, and Setting B (alignment-focused) with $w_{vq}=0.5, w_{mq}=0.5, w_{ta}=1.0$. Secondary metrics include CLIPScore and PickScore for visual quality evaluation independent of the training reward model. The Overall column in Table 2 corresponds to the total composite reward. For the user study, pairwise win rates measure human preference across Visual Quality, Motion Quality, and Semantic Alignment dimensions.

  • Baselines. Three prior methods are compared: DanceGRPO (Xue et al., 2025), FlowGRPO (Liu et al., 2025b), and CPS (Coefficients-Preserving Sampling; Wang & Yu, 2025). The paper explicitly reimplements all three in a "unified training framework on HunyuanVideo1.5 with shared infrastructure across all methods" to isolate algorithmic effects from engineering choices—a critical design decision since "current video GRPO baselines are implemented with substantial differences in engineering optimizations." Each baseline is evaluated both without KL regularization (w/o KL) and with its Fixed KL constraint variant (w/ Fixed KL).

  • Generation budget / compute accounting. The paper measures compute through GRPO update frequency and sampling steps: "we apply GRPO updates every 20 sampling steps along the diffusion trajectory" (Section 4.1). Videos contain 81 frames, with per-GPU batch size 2 and 4 gradient accumulation steps (effective batch size 8). The paper does not report total FLOPs or wall-clock time; instead, comparison fairness is achieved through the unified reimplementation framework where all methods use identical infrastructure and training budgets. The exploration scaling factor $\eta$ in the SDE controls noise magnitude, but its specific value is not reported.

  • Cross-validation / statistical protocol. No cross-validation or statistical significance testing is reported for the main quantitative results. The user study provides pairwise preference statistics (Table 3) with win rates computed over 29 evaluators × 32 prompts, but the paper does not report confidence intervals or statistical tests. Ablation studies compare training curves (Figures 3, 7, 8) but do not report variance across random seeds. The paper does not disclose whether results are from single runs or averaged across multiple seeds.

Main Quantitative Results

Overall Comparison with Baselines (Table 2)

The headline result is that SAGE-GRPO with Dual Moving KL achieves the best performance under the alignment-focused reward setting (Setting B), attaining Overall 0.8066, VQ −0.4765, MQ −0.2385, TA 1.5216, CLIPScore 0.5484, and PickScore 0.7420. This outperforms all baselines in their best configurations on the Overall, VQ, and MQ metrics. The second-best Overall in Setting B is FlowGRPO w/o KL at 0.4773, meaning SAGE-GRPO improves by ~69% relative to the next best method.

Setting A (averaged rewards) tells a more nuanced story. CPS w/o KL achieves the highest Overall at 0.6343, with SAGE-GRPO w/o KL at 0.4859 and SAGE-GRPO w/ Dual Mov KL at 0.2173. The paper interprets this as evidence that "adding KL regularization typically improves visual performance but yields worse reward behavior, which we attribute to reward hacking in the reward model." In other words, the unregularized CPS method maximizes the averaged-reward signal but does so by exploiting the reward model rather than genuinely improving video quality—a claim supported by the fact that CPS w/o KL's gains do not translate to better visual metrics (CLIPScore 0.5479 vs. SAGE-GRPO w/ Dual Mov KL 0.5484 in Setting B) or human preference (SAGE-GRPO achieves 80.2%, 70.8%, 67.9% win rates over CPS in Visual Quality, Motion Quality, and Semantic Alignment respectively).

The key pattern across both settings is that SAGE-GRPO's advantage is most pronounced when the reward is well-specified (alignment-focused, where text alignment receives higher weight, reducing opportunities for reward hacking). Under Setting B:

  • Overall: SAGE-GRPO (0.8066) > CPS w/ Fixed KL (0.3705) > FlowGRPO w/o KL (0.4773) > DanceGRPO w/ Fixed KL (0.1290)
  • VQ: SAGE-GRPO (−0.4765) > FlowGRPO w/o KL (−0.5671) > CPS w/ Fixed KL (−0.6121) > DanceGRPO w/o KL (−0.8854)
  • MQ: SAGE-GRPO (−0.2385) > CPS w/ Fixed KL (−0.4787) > FlowGRPO w/o KL (−0.4731) > DanceGRPO w/ Fixed KL (−0.5083)
  • TA: CPS w/o KL (1.5669) > SAGE-GRPO (1.5216) > FlowGRPO w/o KL (1.5175) > DanceGRPO w/ Fixed KL (1.4112)
  • CLIPScore: SAGE-GRPO (0.5484) > CPS w/o KL (0.5479) > DanceGRPO w/ Fixed KL (0.5452) > FlowGRPO w/ Fixed KL (0.5427)
  • PickScore: SAGE-GRPO (0.7420) > FlowGRPO w/ Fixed KL (0.7408) > DanceGRPO (0.7352 w/o KL) > CPS w/ Fixed KL (0.7364)

Note on VQ and MQ scales: the negative values are unusual for a "quality" metric and suggest that VideoAlign's VQ and MQ scores are likely unnormalized outputs (possibly logits or raw model outputs) that can be negative. The paper treats higher (less negative) values as better, consistent with the pattern where SAGE-GRPO achieves the least negative VQ and MQ scores in Setting B.

Why the paper emphasizes Setting B. The paper argues that "emphasizing alignment provides a more reliable optimization target and yields more stable gains in both reward and visual metrics." The intuition is that text alignment (does the video match the described scene, action, emotion?) is harder to game than visual quality (does the video look sharp and aesthetically pleasing?), which can be superficially improved through low-level texture adjustments that don't reflect genuine generation quality. By weighting text alignment more heavily, Setting B creates a reward landscape where reward hacking is less profitable, forcing methods to actually improve alignment. Under this more stringent optimization target, SAGE-GRPO's manifold-constrained exploration delivers clear gains while unconstrained baselines struggle.

Impact of KL Regularization on Different Methods (Table 2, comparing w/o KL vs. w/ KL variants)

A systematic pattern emerges when comparing each method's unregularized (w/o KL) and regularized (w/ Fixed KL or Dual Mov KL) variants:

  • DanceGRPO: Under Setting A, the w/o KL variant achieves Overall 0.2768, but w/ Fixed KL drops to 0.0979—KL regularization hurts DanceGRPO's reward performance. Under Setting B, the w/o KL variant produces negative Overall (−0.2172), while w/ Fixed KL recovers to 0.1290. This suggests DanceGRPO's exploration is unstable enough that without KL it completely fails under the alignment-focused reward, but Fixed KL helps only marginally.

  • FlowGRPO: Under Setting A, w/o KL achieves 0.2733 and w/ Fixed KL drops to 0.1880. Under Setting B, w/o KL achieves 0.4773 and w/ Fixed KL drops to 0.2103. FlowGRPO benefits from removing KL in the alignment-focused setting—its noise approximation apparently provides enough stability that the Fixed KL constraint is overly restrictive.

  • CPS: Under Setting A, w/o KL achieves 0.6343 (the highest in the table) and w/ Fixed KL drops dramatically to 0.0928—a massive 85% reduction. Under Setting B, both w/o KL (0.3694) and w/ Fixed KL (0.3705) perform similarly. CPS appears highly sensitive to the reward configuration: it excels at maximizing the averaged reward signal without KL but collapses with Fixed KL in that setting, while being more stable (but not best) in the alignment-focused setting.

  • SAGE-GRPO: Under both settings, the Dual Mov KL variant either matches or outperforms the w/o KL and Fixed KL variants. Under Setting B, Dual Mov KL achieves 0.8066 compared to −0.1222 (w/o KL) and 0.2857 (w/ Fixed KL)—the Dual Trust Region is essential for achieving the best performance, not merely a regularizer. This supports the paper's claim that the position-velocity controller enables the policy to reach high-reward regions that are inaccessible under either no constraint (which leads to reward hacking) or Fixed KL (which limits plasticity).

The HunyuanVideo 1.5 baseline (row 1 of Table 2) achieves Overall 0.7397 but with VQ 0.0654, MQ −0.7539, and TA −0.5870. The interesting observation is that the VQ is positive (0.0654) in the original model but becomes negative under GRPO training for most methods—this reflects the tension between optimizing for VQ (which the pretrained model already does reasonably) versus optimizing for alignment (which requires trading off some visual quality for better text matching). SAGE-GRPO's VQ of −0.4765 under Setting B, while negative, is the best among all methods in that setting, suggesting it achieves the best balance between maintaining visual quality and improving alignment.

Gradient Equalizer Ablation (Figure 3)

The paper compares training dynamics with and without the Temporal Gradient Equalizer across the three baseline SDE formulations (DanceGRPO, FlowGRPO, CPS) and SAGE-GRPO itself. The key finding is that all methods benefit from gradient equalization, but SAGE-GRPO benefits the most.

Specifically, Figure 3 shows the overall VideoAlign reward curves for each method:

  • DanceGRPO: Without equalization, the reward curve is essentially flat (barely above the starting value) and unstable. With equalization, it trends upward modestly but remains noisy.
  • FlowGRPO: Without equalization, the reward rises initially but plateaus early. With equalization, it achieves a higher and more stable reward, though still plateauing.
  • CPS: Without equalization, the reward curve shows high variance with sharp drops. With equalization, the curve becomes smoother but the final reward is only marginally improved.
  • SAGE-GRPO: Without equalization, the reward curve (not explicitly shown as a separate curve in Figure 3, but implied by the "Ours" panel) shows some improvement but with instability. With equalization, SAGE-GRPO achieves "smoother with consistent improvement" and "gradient scale variation is reduced from more than one order of magnitude to within a small constant factor."

The critical insight is that gradient imbalance is a general problem—not specific to SAGE-GRPO's noise schedule—that affects all SDE-based GRPO methods. Figure 4 substantiates this by showing the empirical gradient norm imbalance: observed norms (blue) decrease rapidly as σ increases, matching the predicted relationship $\|\nabla \log \pi\| \propto 1/\Sigma_t^{1/2}$ (red). The magnitude variation spans more than one order of magnitude, confirming that low-noise timesteps ($\sigma \to 0$) receive gradients orders of magnitude larger than high-noise timesteps ($\sigma \to 1$).

The paper does not report exact final reward values for each ablation in Figure 3, making precise quantitative comparison difficult. However, the qualitative pattern is clear: the equalizer consistently improves stability and final performance across all methods, with the improvement being most pronounced for SAGE-GRPO. This supports the paper's claim that the gradient equalizer is a general contribution applicable beyond SAGE-GRPO, but also that it synergizes with the precise SDE to produce the best overall results.

KL Strategy Ablation (Figure 8)

The paper compares four KL strategies: No KL (unconstrained), Fixed KL (anchored to π₀), Step-wise KL (velocity control only), Moving KL (position control only via periodic anchor), and Dual Moving KL (position + velocity). Figure 8 reports two dimensions:

Mean reward (Figure 8a): Dual Moving KL achieves the highest and most stable reward throughout training. Fixed KL produces lower reward due to underfitting (too restrictive). Moving KL initially rises quickly but then falls off—the position anchor alone cannot sustain exploration. Step-wise KL is not explicitly shown in Figure 8a, but the text references "collapse observed in aggressive step-wise updates," suggesting it leads to reward degradation without position constraints. The key quantitative claim: "Dual Moving KL consistently outperforms other variants in both convergence speed and final reward."

Exploration level (Figure 8b): Measured as the standard deviation of rewards (higher std = more exploration). Moving KL attains high exploration in early training steps but the exploration level drops in later stages—the policy converges to a local optimum and stops exploring. Dual Moving KL maintains "a higher and more stable exploration level throughout training," validating the position-velocity controller interpretation: the moving anchor enables sustained exploration by periodically resetting the safe zone, while the velocity term prevents individual steps from being too aggressive.

The paper does not report exact numerical values for each strategy's final reward or exploration level. This is a limitation—without specific numbers, the claim of "highest and most stable" is qualitative, though the trends in Figure 8 are visually clear.

KL Weight Sensitivity (Figure 7)

The paper compares three KL weight schedules: fixed $\lambda_{KL} = 10^{-5}$ (green), two-stage $10^{-7} \to 10^{-5}$ (red), and two-stage $10^{-7} \to 10^{-6}$ (yellow, milder final constraint). The results are reported separately for VQ, MQ, and TA rewards.

The two-stage $10^{-7} \to 10^{-5}$ schedule achieves "the strongest and most consistent gains across VQ, MQ, and TA." The fixed $10^{-5}$ schedule (green) underperforms because the constraint is too tight from the beginning, preventing the policy from making meaningful initial progress. The milder $10^{-7} \to 10^{-6}$ schedule (yellow) underperforms because the final constraint is too weak to prevent drift and reward hacking after the policy has moved substantially.

The key quantitative pattern (read from Figure 7): all three schedules trend upward overall, but the two-stage $10^{-7} \to 10^{-5}$ (red) curve is consistently above the others for VQ and MQ, and achieves the highest final TA score. The paper argues this supports the design principle of "gradually increasing λ_KL to tighten the trust region"—early weak constraint enables initial reward improvement, while later stronger constraint prevents the accumulated drift that would otherwise cause degradation.

Qualitative Results (Figure 6, Figure 10)

The paper provides three qualitative comparisons in Figure 6, each showing a baseline output and the SAGE-GRPO output for the same prompt:

  • (Top) A teenage boy on a running track at sunset: SAGE-GRPO shows reduced temporal jitter and more accurate visual contents.
  • (Middle) Isaac Newton under an apple tree: SAGE-GRPO demonstrates "enhanced alignment and photorealism under occlusion and lighting changes."
  • (Bottom) Five children running across a field: SAGE-GRPO shows "stronger semantic alignment with consistent prompt matching across frames."

Appendix Figure 10 adds qualitative results focused on emotional alignment, showing that SAGE-GRPO better captures "calm, contemplative expression" and "lines of fatigue" and "somber mood" compared to baselines, which produce neutral expressions and fail to convey the intended emotional depth.

These qualitative results complement the quantitative metrics but are inherently subjective. The paper uses them to illustrate specific improvement dimensions (temporal consistency, photorealism, semantic precision) rather than as standalone evidence.

User Study (Table 3)

The user study with 29 evaluators on 32 prompts compares SAGE-GRPO against each baseline at iteration 100, sampling step 40, under Setting B. Pairwise win rates:

SAGE-GRPO vs.Visual QualityMotion QualitySemantic Alignment
DanceGRPO85.9%75.8%79.2%
FlowGRPO83.8%79.2%71.9%
CPS80.2%70.8%67.9%

SAGE-GRPO wins against all baselines on all dimensions, with the strongest advantage in Visual Quality (80.2–85.9% win rates). Motion Quality shows the largest variation (70.8% against CPS to 79.2% against FlowGRPO), and Semantic Alignment ranges from 67.9% (against CPS) to 79.2% (against DanceGRPO).

The paper interprets these results as confirming "that automatic metrics align with perceptual quality." However, the win rates against CPS in Semantic Alignment (67.9%) and Motion Quality (70.8%) are relatively modest—while still above chance, they suggest that CPS is competitive with SAGE-GRPO on these dimensions in human judgment, despite SAGE-GRPO's larger numerical advantage on the VideoAlign MQ metric in Table 2 (SAGE-GRPO MQ −0.2385 vs. CPS w/ Fixed KL MQ −0.4787 in Setting B). This discrepancy hints at either VideoAlign's MQ metric not perfectly capturing human motion quality perception, or the user study sample being too small to detect differences that the automatic metrics capture.

Ablation Studies and Robustness Checks

  • Temporal Gradient Equalizer on/off across SDE formulations (Figure 3): All four methods (DanceGRPO, FlowGRPO, CPS, SAGE-GRPO) show improved reward stability and higher final reward with the equalizer enabled. Without equalization, reward curves plateau early or oscillate—this is consistent across methods, confirming gradient imbalance is a general structural problem. The improvement is most pronounced for SAGE-GRPO, suggesting synergy between the precise SDE and equalized gradients.

  • KL strategy variants (Figure 8): No KL leads to high initial exploration but eventual collapse. Fixed KL (to π₀) is too restrictive, producing lower reward. Moving KL alone shows high early exploration but exploration drops off, suggesting the position anchor periodically resetting is insufficient without velocity damping. Step-wise KL alone leads to "collapse" consistent with unbounded cumulative drift. Only Dual Moving KL (position + velocity) maintains both high reward and stable exploration throughout training. Qualitative comparisons in Appendix Figures 11-12 show that Dual Moving KL produces more realistic facial details, better rendering, and stronger prompt alignment.

  • KL weight schedule (Figure 7): Fixed $\lambda_{KL}=10^{-5}$ underperforms due to early over-constraint. Two-stage $10^{-7} \to 10^{-6}$ (milder final constraint) underperforms due to insufficient final regularization allowing drift. Two-stage $10^{-7} \to 10^{-5}$ achieves the best VQ, MQ, and TA trajectories, supporting adaptive tightening of the trust region.

  • Noise standard deviation comparison (Appendix Figure 9): Under FlowGRPO's σ schedule, SAGE-GRPO's integral formulation yields near-zero std at the first step (both endpoints equal); FlowGRPO does not show this behavior. Under clamped (1−σ) ≥ 3×10⁻³, FlowGRPO's std explodes to ~3.0 at step 1 while SAGE-GRPO remains stable at ~1.0, demonstrating the robustness advantage of the integrated variance formulation.

  • ReST^(EM) revision model training (Appendix K, Figure 16): The paper reports that an attempt to further optimize using ReST^(EM) (Singh et al., 2024) backfires: "additional sequential revisions substantially hurt performance." This is a notable negative result—the on-policy data collection in ReST^(EM) presumably amplifies spurious correlations that the SAGE-GRPO framework was designed to avoid. It demonstrates the fragility of revision-style training and indirectly validates the paper's design choices (offline data construction, manifold-constrained exploration).

Critical Assessment

Claim 1: SAGE-GRPO achieves state-of-the-art video alignment performance.

What was tested: SAGE-GRPO was compared against DanceGRPO, FlowGRPO, and CPS on HunyuanVideo1.5 using VideoAlign as the reward model (Table 2), with a user study (Table 3) and qualitative examples (Figures 6, 10).

Verdict: Supported with important qualifications. SAGE-GRPO with Dual Moving KL achieves the best Overall, VQ, MQ, and CLIPScore in the alignment-focused setting (Setting B), and wins against all baselines in user preference (70.8–85.9% win rates). However, the claim of superiority is conditional on the reward configuration: in Setting A (averaged rewards), CPS w/o KL achieves higher Overall (0.6343 vs. SAGE-GRPO's best of 0.4859). The paper argues that Setting A promotes reward hacking and that Setting B is the more meaningful evaluation, which is a reasonable position but means the headline claim should be interpreted as "SAGE-GRPO achieves the best performance when the optimization target is well-specified to resist reward hacking."

What's missing: No comparison against non-GRPO alignment methods (DPO, standard RLHF, reward-weighted regression) or against larger-scale baselines. The claim is limited to GRPO-based video alignment methods on HunyuanVideo1.5. No results on other video generation models (Wan, Sora, etc.) or other reward models (ImageReward, VisionReward). The improvement over baselines, while consistent, is not enormous—in Setting B, FlowGRPO w/o KL achieves Overall 0.4773, and SAGE-GRPO achieves 0.8066, a ~69% relative improvement but on what appears to be an arbitrary reward scale. The CLIPScore improvement (0.5484 vs. 0.5479 for CPS) is marginal.

Claim 2: The precise manifold-aware SDE (with logarithmic correction) removes excess noise energy and improves rollout quality.

What was tested: The SDE variance derivation is mathematically exact (Appendix A.1), and the standard deviation comparison (Figure 9) demonstrates stability advantages over FlowGRPO. The qualitative examples (Figure 1b, Figure 6) show improved visual quality.

Verdict: Partially supported, but missing direct evidence. The mathematical derivation is sound—the integrated variance with logarithmic correction is indeed more accurate than first-order approximations. Figure 9 shows that the precise SDE avoids FlowGRPO's explosive behavior under certain clamp configurations. However, there is no direct experiment isolating the SDE's contribution: the paper does not compare SAGE-GRPO with the precise SDE but without gradient equalization or Dual KL against a baseline using FlowGRPO's SDE but with equalization and Dual KL. The SDE's contribution is always evaluated in combination with other components. The claim that the SDE alone "improves rollout quality" is supported by qualitative examples but lacks a controlled quantitative ablation (e.g., reward curves with precise SDE only vs. FlowGRPO's SDE only, holding all else equal). Figure 3 shows overall reward curves with and without gradient equalization for different SDE formulations, but this tests the equalization effect, not the SDE accuracy effect alone.

Claim 3: The Gradient Norm Equalizer balances optimization across timesteps and improves training stability.

What was tested: Figure 3 shows training curves with/without equalization across four SDE formulations. Figure 4 demonstrates the gradient norm imbalance empirically.

Verdict: Strongly supported. This is the most rigorously validated component. The equalization effect is tested across all SDE formulations (not just SAGE-GRPO), showing consistent improvement in reward stability. Figure 4 provides mechanistic evidence by demonstrating the gradient-variance inverse relationship matches the theoretical prediction. The paper could strengthen this claim by reporting exact final reward values for each condition, but the qualitative improvement pattern is unambiguous and consistent across methods. The fact that all baselines benefit from equalization (not just SAGE-GRPO) strengthens the claim that this is a general contribution.

Claim 4: The Dual Trust Region (moving anchor + step-wise KL) prevents policy drift while enabling sustained plasticity.

What was tested: Figure 8 compares KL strategies (No KL, Fixed KL, Moving KL, Dual Moving KL) on mean reward and exploration level. Figure 7 tests KL weight schedules. Qualitative comparisons (Figures 11-12) show visual differences.

Verdict: Supported, but the controller interpretation is theoretical rather than empirically validated. The experiments show that Dual Moving KL outperforms alternatives in both reward and exploration stability—this is clean evidence that the combination is better than either alone. However, the paper frames this as evidence for a "position-velocity controller" interpretation, which is a theoretical analogy rather than a testable claim. The experiments demonstrate that combining moving anchor with step-wise KL works, but they don't demonstrate that the control-theoretic mechanism (position control reducing steady-state error, velocity control reducing overshoot) is the actual causal pathway. For example, it's possible that the moving anchor simply provides a more recent and thus more appropriate reference distribution, and the step-wise KL provides gradient smoothing—the controller analogy may be a useful framing rather than a mechanistic explanation.

What's missing: The moving anchor update interval $N$ is not ablated—we don't know how sensitive the results are to this choice. If $N=1$ (update every step), the moving anchor reduces to step-wise KL, which we know performs poorly. If $N$ is very large, it approaches Fixed KL. The optimal $N$ likely depends on the reward landscape and problem difficulty, but the paper provides no guidance. Additionally, the β_pos and β_vel coefficients are not specified or ablated—the paper uses equal weighting by default, but the controller interpretation would predict different optimal ratios depending on the noise level in the reward signal.

Claim 5: SAGE-GRPO resolves the stability-plasticity dilemma for video GRPO.

What was tested: The full SAGE-GRPO framework (precise SDE + equalizer + Dual KL) is compared against baselines that use either no KL, fixed KL, or step-wise KL (Table 2, Figure 8).

Verdict: Supported in the sense that SAGE-GRPO achieves better final performance while maintaining stability, but the claim of "resolving" the dilemma may be overstated. The experiments demonstrate that SAGE-GRPO achieves higher reward without the collapse or plateau that affects other methods—this is evidence that it achieves better plasticity (can reach higher reward) without sacrificing stability (does not collapse). However, the paper doesn't demonstrate that SAGE-GRPO can sustain this balance indefinitely—the training curves in Figure 8 extend only to a finite number of steps, and longer-horizon behavior is unknown. Moreover, the stability-plasticity "dilemma" is presented as an inherent tradeoff, but the paper's solution (moving anchor + step-wise KL) simply shifts where the tradeoff operates—it doesn't eliminate the tradeoff. At sufficiently long horizons, even the moving anchor will drift if the step-wise constraint is too weak, or the policy will be overly constrained if the step-wise constraint is too strong. The adaptive KL weighting helps but is itself sensitive to the target KL value $D_{\text{target}}$, which is not specified or ablated.

General weaknesses:

  1. Single model, single reward model, single infrastructure. All results are on HunyuanVideo1.5 with VideoAlign. The claims about manifold-constrained exploration being fundamental should generalize, but the paper provides zero evidence beyond this specific combination. Different video models may have different manifold geometries (different dimensionality, curvature, noise schedule), and different reward models may have different sensitivity to off-manifold artifacts.

  2. No seed variance reported. None of the figures or tables report error bars, confidence intervals, or results across multiple random seeds. The training curves in Figures 3, 7, 8 appear to be single runs, making it impossible to distinguish algorithmic improvement from seed noise. The user study provides raw win rates but no statistical significance tests (are the 67.9% win rate against CPS on Semantic Alignment and 70.8% on Motion Quality statistically distinguishable from 50%?).

  3. Small user study. 29 evaluators × 32 prompts = 928 pairwise comparisons per method, which is reasonable but on the small side. The 67.9% Semantic Alignment win rate against CPS is only modestly above chance and may not be robust.

  4. The moving anchor interval N and dual KL coefficients are not specified. These are critical hyperparameters for the core contribution, yet the paper provides no values, no ablation, and no guidance for practitioners. This makes reproduction difficult and obscures how sensitive the method is to these choices.

  5. No comparison against simply using more samples. The paper doesn't report whether SAGE-GRPO's gains could be achieved by simply increasing the group size G or the number of training iterations with a baseline method. If FlowGRPO with G=16 or 2× training steps matches SAGE-GRPO with G=8, the efficiency advantage is less clear.

  6. The claim about reward hacking in Setting A is asserted but not demonstrated. The paper states that CPS w/o KL's high Setting A score reflects reward hacking, but doesn't provide evidence (e.g., videos that score highly on VideoAlign but are judged poorly by humans). The user study is conducted only under Setting B, so we cannot verify whether CPS's Setting A performance translates to human preference or is purely a reward model artifact.

  7. VQ and MQ are negative in most results. The paper never explains the VideoAlign score scale, making it difficult to interpret whether −0.4765 VQ (SAGE-GRPO's best) represents good or merely acceptable visual quality. The HunyuanVideo 1.5 baseline has VQ 0.0654 (positive), so GRPO training appears to universally degrade VQ—this is concerning but undiscussed.

  8. No ablation of the logarithmic correction alone versus a simpler correction. The paper compares against first-order approximations (FlowGRPO) but doesn't test whether a simpler second-order correction (e.g., a midpoint rule for the integral rather than the full logarithmic form) would achieve similar benefits. This would help distinguish whether the specific mathematical form matters or simply "more accurate variance estimation" is sufficient.

6. Limitations and Trade-offs

6.1 Empirical Validation Is Confined to a Single Model-Reward Pair on a Single Architecture

The assumption or constraint. The paper evaluates SAGE-GRPO exclusively on HunyuanVideo1.5 with the VideoAlign reward model. The authors do not claim universal applicability, but the paper's framing—"we view the pre-trained model as defining a valid video data manifold M"—implies that the manifold-constrained exploration formulation should generalize to any flow-matching video generator. Yet no evidence is provided beyond this single combination.

The consequence. Three concerns arise that a practitioner would need to evaluate before adopting the method:

First, the manifold geometry is model-dependent. HunyuanVideo1.5 uses a specific Rectified Flow parameterization with a particular noise schedule σ_t. The logarithmic correction term log((1−σ_{t+1})/(1−σ_t)) that is central to the precise SDE derivation depends on this schedule—the correction's magnitude and importance may differ substantially for models using different noise schedules (cosine, linear, sigmoid), different flow formulations (non-rectified flows, stochastic interpolants), or different parameterizations (x₀-prediction vs. ε-prediction vs. v-prediction). A model whose noise schedule produces smaller relative changes in (1−σ_t) per step would see less benefit from the exact integration. The paper provides no sensitivity analysis across schedule families.

Second, the reward model's sensitivity to off-manifold artifacts is unknown. The paper's central argument—that excess noise degrades rollout quality and corrupts reward signals—assumes VideoAlign is sensitive to the specific artifacts (temporal jitter, structural incoherence) that first-order SDE approximations introduce. But VideoAlign is itself a learned model trained on HunyuanVideo outputs; its failure modes may be idiosyncratic. A different reward model (e.g., ImageReward, VisionReward, CLIP-based scorers) might be more or less sensitive to off-manifold artifacts, changing the relative advantage of the precise SDE. The paper does not test this.

Third, the baseline HunyuanVideo1.5 already achieves positive VQ (0.0654 in Table 2), but GRPO training universally degrades VQ to negative values for all methods in all settings. The paper does not discuss whether this degradation is specific to the VideoAlign-VQ metric (which may penalize the types of videos GRPO produces even when they are genuinely better) or reflects a real quality tradeoff. If the VQ degradation is real, it raises the question of whether the manifold-constrained approach actually preserves visual quality or merely limits how much quality is sacrificed during alignment.

What evidence exists in the paper. Table 2 reports results for HunyuanVideo1.5 only. The standard deviation comparison (Appendix Figure 9) tests robustness of the SDE variance under different σ schedule configurations (FlowGRPO's schedule vs. clamping), but these are all tested within the same base model's noise schedule family—no alternative model architectures or noise schedules are evaluated. Section 4.1 confirms the single-model scope: "We conduct all experiments on HunyuanVideo 1.5."

Mitigation status. The paper does not address this limitation. Section 5 concludes with the general claim that "SAGE-GRPO provides a manifold-aware GRPO framework for stable reinforcement learning for video generation" without qualifying model-specificity. No future work on cross-model generalization is suggested.

6.2 The Moving Anchor Interval and Dual KL Coefficients Are Neither Specified Nor Ablated

The assumption or constraint. The Dual Trust Region—arguably SAGE-GRPO's most architecturally novel contribution—depends on two critical hyperparameters: the moving anchor update interval N (how often the reference policy is refreshed) and the relative weighting β_pos vs. β_vel of the position and velocity KL terms. The paper conceptualizes these as forming a "position-velocity controller" (Section 3.2.2), which in control theory would require tuning the proportional and derivative gains to match the system dynamics.

The consequence. A practitioner attempting to reproduce or deploy SAGE-GRPO cannot determine these values from the paper. The update interval N controls a fundamental tradeoff: if N is too small, the moving anchor approaches step-wise KL and loses position control (permitting drift); if N is too large, it approaches Fixed KL and limits plasticity (preventing the policy from reaching high-reward regions). The optimal N likely depends on the reward landscape's curvature, the base model's distance from the reward optimum, and the per-step learning rate—none of which are transferable across different models, tasks, or reward functions without re-tuning. The paper's Figure 8 demonstrates that Moving KL alone (i.e., position control without velocity damping) sees exploration drop off in later training stages, while Dual Moving KL maintains it. But this result is for one specific N and one specific β_pos/β_vel ratio. The sensitivity of this result to N is unknown: would N = 50 steps produce the same benefit as N = 10? Would N = 200?

Similarly, the controller analogy predicts that the optimal β_pos/β_vel ratio should depend on the noise level in the reward signal. A noisy reward landscape (e.g., from a weak reward model or high sampling variance) would benefit from stronger velocity damping (higher β_vel) to smooth the trajectory, while a smooth reward landscape could tolerate higher position gain (higher β_pos) for faster convergence. The paper uses equal coefficients by default but provides no justification, no ablation, and no guidance for tuning.

What evidence exists in the paper. The KL strategy ablation (Figure 8) compares Fixed KL, Moving KL, Step-wise KL, and Dual Moving KL as strategy categories, but does not sweep N or β_pos/β_vel values within each category. The KL weight ablation (Figure 7) sweeps λ_KL schedules (the overall KL coefficient strength), not the internal decomposition. Appendix A.6 mentions that the two KL terms share a single adaptive weight λ_KL in the full objective, with "coefficients set equal by default," but does not report experiments varying this equality.

Mitigation status. Not addressed. The paper treats the Dual Trust Region as a validated mechanism without characterizing its sensitivity to its own hyperparameters. The adaptive λ_KL scheduler partially addresses the "how strong should the overall constraint be" question, but not the "how should constraint strength be allocated between position and velocity control" question. No future work on automatic tuning of N or β is suggested.

6.3 The Gradient Equalizer Requires Solver-Specific Sensitivity Factors Not Derived in the Paper

The assumption or constraint. The Temporal Gradient Equalizer (Section 3.2.1, Appendix A.5) uses a per-timestep gradient scale proxy N_t = λ_t / Σ_t^{1/2}, where λ_t is described as an "additional sensitivity factor from the solver mapping" that "depends on the SDE type and discretization." The paper provides an explicit derivation for the 1/Σ_t^{1/2} scaling (Equation 37, Appendix A.5) but defers the λ_t factor to implementation-specific details: "details depend on the SDE type and discretization."

The consequence. The gradient equalizer cannot be implemented from the paper alone. The λ_t factor is not merely a scaling constant—it captures how the network's raw output (whether it predicts velocity v_θ, denoised sample x̂_0, or noise ε) maps to the Gaussian mean μ_θ in the transition π(x_{t-1}|x_t) = N(μ_θ, Σ_tI). This mapping depends on the specific SDE solver used (Euler-Maruyama, Heun, DPM-Solver), the model parameterization (v-prediction vs. x₀-prediction vs. ε-prediction), and potentially the noise schedule. For HunyuanVideo1.5, which uses a rectified flow velocity prediction, λ_t would involve the conversion from velocity to the μ_θ implied by the Euler-Maruyama step in Equation (7). For a different model using ε-prediction with a DDPM scheduler, λ_t would be entirely different.

A practitioner using SAGE-GRPO with a different base model (e.g., Wan, Stable Video Diffusion, or a custom flow-matching model) would need to re-derive λ_t for their specific combination of parameterization, solver, and noise schedule. The paper provides no general formula, no derivation template, and no validation that the equalizer's benefits survive when λ_t is approximated or miscalibrated.

What evidence exists in the paper. Figure 4 shows the empirical gradient norm imbalance matching the predicted relationship ‖∇ log π‖ ∝ 1/Σ_t^{1/2}, but this is for the specific HunyuanVideo1.5 SDE configuration with λ_t already incorporated (the red curve in Figure 4 includes both λ_t and 1/Σ_t^{1/2} effects). Figure 3 demonstrates the equalizer's benefit across all four SDE formulations—but these are all implemented within the same HunyuanVideo1.5 framework, so the λ_t factor is consistent. There is no ablation testing the sensitivity of the equalizer's benefit to errors in λ_t (e.g., using N_t = 1/Σ_t^{1/2} only, dropping λ_t entirely, which would be the obvious simplified implementation).

Mitigation status. Not addressed. The paper positions the gradient equalizer as a general contribution ("any method that applies policy gradients to diffusion models will exhibit this gradient imbalance"), but the implementation dependence on λ_t limits its transferability. The appendix notes that λ_t is derived from the solver mapping but provides no derivation for the specific case used, and no guidance for other cases. This is a practical barrier to adoption that the paper does not acknowledge.

6.4 All Results Are from Single Training Runs with No Seed Variance or Statistical Significance Reported

The assumption or constraint. The paper reports training curves (Figures 3, 7, 8) and final performance numbers (Table 2) without error bars, confidence intervals, or any indication of whether results are averaged across multiple random seeds or represent single runs. Section 4.1 provides implementation details (batch size, accumulation steps, frame count) but does not mention the number of training runs or seeds. The user study (Table 3) reports raw pairwise win percentages without statistical tests.

The consequence. The core quantitative claims—that SAGE-GRPO achieves "the best Overall, VQ, MQ, and CLIPScore" (Setting B, Table 2), that the gradient equalizer produces "smoother with consistent improvement" (Figure 3), that Dual Moving KL "maintains higher and more stable exploration" (Figure 8)—cannot be assessed for statistical reliability. Several specific concerns arise:

  • The CLIPScore margin is tiny. In Setting B, SAGE-GRPO's CLIPScore (0.5484) exceeds CPS's (0.5479) by 0.0005 and DanceGRPO's (0.5452) by 0.0032. On an unnormalized scale with unknown variance, these differences could easily be within the noise floor of a single training run. Without seed replication, we cannot determine whether this ordering is reproducible.

  • The user study Semantic Alignment win rate against CPS (67.9%) is marginally above chance. With 29 evaluators and 32 prompts, the effective sample size for comparison is 29 × 32 = 928 pairwise judgments per method pair. A 67.9% win rate corresponds to SAGE-GRPO winning on approximately 630 of 928 comparisons against CPS. Without a statistical test (e.g., binomial test against 50% null, or bootstrap confidence intervals), we cannot assess whether this preference is reliable. Intuitively, a 67.9% win rate on 928 trials is likely statistically significant, but the paper does not verify this.

  • Training curve comparisons (Figures 3, 7, 8) may conflate algorithmic effect with seed noise. The curves show reward trajectories that sometimes cross or show transient advantages. Without error bands showing ±1 standard deviation across seeds, apparent differences (e.g., the two-stage 10^{-7} → 10^{-5} schedule consistently above the others in Figure 7) could reflect a lucky seed for that configuration rather than a robust algorithmic advantage.

What evidence exists in the paper. No error bars, confidence intervals, or seed counts are reported in any figure or table. The experimental setup (Section 4.1) does not mention multiple seeds. The user study methodology (Section 4.4) reports the number of evaluators (29) and prompts (32) but no statistical analysis.

Mitigation status. Not addressed. The paper treats the single-run results as sufficient evidence for its claims. This is a methodological weakness that limits the strength of the conclusions, particularly for the smaller-margin results.

6.5 The Paper Does Not Disentangle the SDE Correction's Contribution from Other Components

The assumption or constraint. The paper presents SAGE-GRPO as an integrated framework with three synergistic components: the precise manifold-aware SDE, the gradient norm equalizer, and the Dual Trust Region. The experiments evaluate combinations (SAGE-GRPO w/o KL, w/ Fixed KL, w/ Dual Mov KL in Table 2) and ablate the equalizer and KL strategy separately (Figures 3, 8), but never isolate the precise SDE's contribution. There is no experiment comparing: (a) SAGE-GRPO's precise SDE + standard equalizer + standard KL vs. (b) FlowGRPO's first-order SDE + standard equalizer + standard KL, where only the SDE formulation differs. The SDE is always evaluated in conjunction with other SAGE-GRPO components.

The consequence. The paper's central geometric claim—that the logarithmic correction term log((1−σ_{t+1})/(1−σ_t)) is the critical factor keeping exploration tangent to the video manifold—cannot be directly attributed to the SDE. The observed improvements could arise primarily from the gradient equalizer (which Figure 3 shows benefits all SDE formulations, including FlowGRPO and CPS) and the Dual Trust Region (which Figure 8 shows provides the largest gains), with the precise SDE contributing marginally or not at all. The standard deviation comparison (Appendix Figure 9) demonstrates that the precise SDE is mathematically more stable under certain σ schedule configurations, but stability of the noise variance does not automatically translate to improved training outcomes—the equalizer could compensate for the gradient effects of excess variance, and the Dual KL could prevent the policy consequences of off-manifold rollouts.

Specifically, consider the Setting B results in Table 2. FlowGRPO w/o KL achieves Overall 0.4773—substantially higher than SAGE-GRPO w/o KL (−0.1222). This suggests that without KL regularization, SAGE-GRPO's precise SDE actually performs worse than FlowGRPO's first-order approximation in the alignment-focused setting. The SAGE-GRPO advantage only appears with Dual Moving KL (0.8066 vs. FlowGRPO's 0.4773 w/o KL). This pattern is consistent with the Dual Trust Region being the dominant contributor to SAGE-GRPO's performance, with the SDE correction playing a supporting or even negligible role. The paper does not test whether FlowGRPO's first-order SDE combined with the Dual Trust Region and gradient equalizer would achieve comparable results.

What evidence exists in the paper. The SDE is always part of a larger configuration. Figure 3 shows equalizer on/off for different SDE formulations—the "Ours" panel tests SAGE-GRPO's SDE with/without equalizer, but does not compare SAGE-GRPO's SDE without equalizer against FlowGRPO's SDE without equalizer. Table 2 shows SAGE-GRPO under different KL variants, but it does not show equivalent KL variant results for baselines using the SAGE-GRPO SDE. The qualitative comparison (Figure 1b) shows SAGE-GRPO outputs vs. baselines, but these are full-system comparisons.

Mitigation status. Not addressed. The paper treats the SDE correction as validated by the mathematical derivation (Appendix A.1) and the standard deviation stability (Appendix Figure 9), without testing whether it independently improves training outcomes. This is a significant gap because the SDE correction is presented as a fundamental contribution ("Innovation 2" in the paper's framing), yet its practical importance relative to the other components—which are simpler to implement (median normalization, periodic reference updates)—remains unquantified.

6.6 GRPO Training Universally Degrades Visual Quality Relative to the Pretrained Model

The assumption or constraint. The paper evaluates all methods using the VideoAlign composite reward, which combines VQ, MQ, and TA into a single scalar. The implicit assumption is that maximizing this composite reward corresponds to improving the video generation model. However, the HunyuanVideo1.5 baseline achieves VQ = 0.0654 (positive), while every GRPO-trained variant in Table 2 scores negative on VQ—including SAGE-GRPO with Dual Moving KL in Setting B (VQ = −0.4765), which is the paper's best configuration. The Original model's VQ is not approached by any method.

The consequence. This raises two concerns. First, GRPO alignment appears to trade off visual quality for text alignment and motion quality. The VQ degradation is large: from +0.0654 to −0.4765 (a swing of 0.54 units on the VideoAlign VQ scale). The paper does not discuss whether this degradation is an artifact of the VideoAlign VQ metric (which may penalize stylistic changes that are genuinely improvements) or reflects an actual loss of visual fidelity that a human viewer would perceive. The user study shows SAGE-GRPO winning on Visual Quality at 80.2–85.9% against baselines, but this is a relative comparison—the absolute visual quality compared to the pretrained model is not measured. It is possible that all GRPO-trained models produce worse-looking videos than the original HunyuanVideo1.5, but SAGE-GRPO degrades less than the others.

Second, the reward model's VQ dimension may be fundamentally misaligned with true visual quality. If maximizing the composite reward requires accepting negative VQ, then either (a) the reward weights need retuning to prevent VQ sacrifice, or (b) the VQ signal from VideoAlign is not reliable under the distribution shift induced by GRPO training (i.e., VideoAlign's VQ scorer was trained on pretrained-model outputs and produces miscalibrated scores for fine-tuned outputs). Either case undermines the reliability of the VideoAlign composite reward as an optimization target and complicates the interpretation of Table 2: if VQ scores are unreliable, the Overall reward may not reflect genuine improvement.

What evidence exists in the paper. Table 2, first data row: HunyuanVideo1.5 Original VQ = 0.0654. All other rows have negative VQ values, ranging from −0.4765 (SAGE-GRPO Dual Mov KL, Setting B) to −0.8854 (DanceGRPO w/o KL, Setting B). The paper does not discuss this pattern. The user study (Table 3) measures relative preference between SAGE-GRPO and baselines, not absolute quality vs. the pretrained model.

Mitigation status. Not addressed. The paper does not acknowledge the universal VQ degradation, does not analyze whether it represents genuine quality loss or metric miscalibration, and does not include the pretrained model in the user study to establish an absolute quality baseline. This is a notable omission because it bears directly on the paper's central claim: that manifold-constrained exploration preserves rollout quality and enables stable alignment. If even the best SAGE-GRPO configuration produces videos with substantially worse VQ than the un-fine-tuned model (as VideoAlign measures it), the "stability" may be relative to other GRPO methods rather than absolute.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around reinforcement learning for video generation from heuristic stabilization to geometry-aware exploration design. Prior to SAGE-GRPO, the instability of GRPO for video was treated as an implementation challenge—something to be managed through hyperparameter tuning, KL schedule engineering, or curriculum strategies (Self-paced GRPO, DiverseGRPO). The implicit assumption was that the fundamental mechanisms (ODE-to-SDE conversion, group-normalized advantages, KL constraints) were sound, and failures reflected inadequate tuning or reward model limitations. SAGE-GRPO challenges this assumption at its root: the exploration mechanism itself is the problem, not its configuration.

This is a diagnostic reframing with methodological consequences, not a paradigm shift. The paper does not invent a new class of algorithms—GRPO, SDE-based exploration, and KL regularization were all established. What it contributes is a specific, falsifiable causal claim: first-order SDE approximations inject excess noise that pushes trajectories off the pre-trained video manifold, degrading rollout quality and corrupting the reward signal that GRPO depends on. This diagnosis unifies multiple previously observed failure modes—temporal jitter, unstable reward curves, reward hacking, catastrophic forgetting—under a single geometric mechanism. If the diagnosis is correct, then the solution space narrows considerably: future work on video GRPO should prioritize exploration strategies that respect manifold geometry, rather than developing ever-more-complex reward models or training curricula.

The paper's most consequential empirical finding for the field is that the gradient imbalance across timesteps is a structural property of any SDE-based diffusion model optimization, not a quirk of specific noise schedules. Figure 3 demonstrates that all four tested SDE formulations (DanceGRPO, FlowGRPO, CPS, and SAGE-GRPO) exhibit unstable or plateaued reward curves without per-timestep gradient equalization, and all four improve with it. This is not a video-specific result—it follows from the Gaussian transition assumption and the inverse relationship ‖∇ log π‖ ∝ 1/Σ_t^{1/2} that holds for any diffusion or flow-matching model. This means that every method applying policy gradients to diffusion models (DDPO, DPOK, GRPO variants, and future methods) should incorporate timestep-aware gradient normalization. The paper provides both the diagnostic tool (check your gradient norm variance across the noise schedule) and the prescriptive fix (median-normalized per-timestep scaling). This alone makes the paper worth reading for anyone working on RL for generative models, regardless of whether they adopt the full SAGE-GRPO framework.

The position-velocity controller framing for KL regularization (Section 3.2.2) is a more speculative but intellectually fertile contribution. Prior work treated KL constraints as static barriers: either you anchor to the pretrained model (fixed KL, which prevents reaching good policies) or you constrain step-to-step change (step-wise KL, which permits unbounded drift). The paper's reframing of these as position control and velocity control—and the demonstration that combining them with a moving anchor yields better stability and plasticity than either alone—suggests that KL regularization should be designed as a dynamical system, not a static penalty. This has implications beyond video generation: any domain where RL fine-tuning must balance exploration far from initialization against the risk of reward hacking (language model alignment, robotics policy adaptation, drug discovery) could benefit from a position-velocity decomposition of the constraint structure.

However, the paper does not resolve the stability-plasticity dilemma. It provides a better tradeoff at a specific operating point (HunyuanVideo1.5 with VideoAlign), but does not demonstrate that the Dual Trust Region can sustain arbitrary-horizon training, nor does it characterize the sensitivity of the tradeoff to the moving anchor interval N or the position-velocity coefficient ratio. The controller analogy is appealing but remains a theoretical framing rather than a validated mechanism—the paper shows that Dual KL works better than alternatives, not that it works because of the control-theoretic properties the analogy suggests. Follow-up work that designs and tests explicit PD/PID controllers for KL regularization, with systematic sweeps of the position and velocity gains, would validate or refine this contribution.

The paper also shifts research attention away from reward model design and toward exploration mechanism design. If SAGE-GRPO's diagnosis is correct, the bottleneck for video GRPO is not reward model quality (VideoAlign is used frozen and unmodified throughout) but rather whether the exploration process produces rollouts that the reward model can evaluate meaningfully. This implies that efforts to build better video reward models—while valuable—may be premature if the exploration mechanism is pushing rollouts into regions where even a perfect reward model would receive degraded inputs. The priority should be making exploration manifold-consistent, then improving the reward signal.

Follow-Up Research This Work Enables

Ablate the logarithmic correction against simpler second-order approximations. The paper claims that the exact integrated variance with logarithmic correction is necessary to prevent off-manifold exploration, but it never tests whether a simpler correction—a midpoint-rule integration, a Simpson's rule approximation, or even just FlowGRPO's first-order formula with a smaller η—would achieve comparable training results. Figure 9 shows the precise SDE is more stable under edge-case clamping configurations, but this is a measurement of noise variance stability, not training outcome. A strong follow-up would compare: (a) SAGE-GRPO's exact logarithmic SDE, (b) a midpoint-rule SDE that integrates ε_s² at the interval midpoint rather than the start, (c) FlowGRPO's first-order SDE with reduced exploration factor η' = 0.5η to match the precise SDE's total noise energy, and (d) FlowGRPO's default SDE. All four would be tested with the same gradient equalizer and Dual Trust Region, isolating the SDE variance formulation. The prediction (if the paper's geometric claim is correct) is that the exact and midpoint formulations should outperform both FlowGRPO variants because they correctly capture the curvature of the diffusion coefficient, not merely because they inject less total noise. A null result (FlowGRPO with reduced η matches the precise SDE) would suggest that total noise energy, not manifold alignment, is what matters—undermining the paper's central geometric argument.

Test SAGE-GRPO on alternative video generation architectures and reward models. The paper's entire empirical case rests on HunyuanVideo1.5 with VideoAlign. Three dimensions of generalization remain untested. First, different flow-matching architectures: does the logarithmic correction matter equally for models using different noise schedules (cosine vs. linear), different parameterizations (ε-prediction vs. x₀-prediction vs. v-prediction), or different flow formulations (Rectified Flow vs. stochastic interpolants vs. Flow Matching with optimal transport paths)? The correction term log((1−σ_{t+1})/(1−σ_t)) depends on the specific σ_t schedule—a cosine schedule, which concentrates more change at intermediate noise levels, may see different relative benefits than HunyuanVideo's schedule. Second, different reward models: VideoAlign scores VQ, MQ, and TA on a specific scale with specific failure modes. Would SAGE-GRPO's advantage persist with ImageReward, VisionReward, or a CLIP-based scorer? The paper's argument that excess noise degrades "rollout quality" implies the benefit should be reward-model-agnostic (better videos should score higher under any reasonable metric), but VideoAlign's VQ dimension shows universal degradation under all GRPO methods relative to the pretrained model—if this is a VideoAlign artifact rather than a genuine quality loss, the paper's conclusions may be reward-model-specific. A strong follow-up would replicate the Setting B comparison (SAGE-GRPO w/ Dual Mov KL vs. FlowGRPO w/o KL vs. CPS w/ Fixed KL) on at least two additional model architectures and two additional reward models, reporting both automatic metrics and a user study scaled to detect moderate effect sizes (e.g., 50+ evaluators, 100+ prompts).

Characterize the moving anchor interval N as a function of reward landscape properties. The paper conceptualizes the Dual Trust Region as a position-velocity controller but provides no guidance on setting N—the interval at which the reference policy is refreshed. This is the critical knob that controls the stability-plasticity tradeoff: small N approaches step-wise KL (permits drift), large N approaches fixed KL (limits plasticity). In control theory, the optimal proportional gain (which N effectively modulates) depends on the system's natural frequency and damping ratio. For GRPO, the "system" is the reward landscape—how sharply the reward varies with policy parameters, and how noisy the reward estimates are. A strong follow-up would sweep N across orders of magnitude (e.g., N ∈ {1, 2, 5, 10, 20, 50, 100, 200, 500}) for SAGE-GRPO on both Setting A and Setting B, and correlate the optimal N with measurable properties of the training dynamics: (a) the variance of the advantage estimates σ_R² (higher variance → noisier reward landscape → likely smaller optimal N or higher velocity damping), (b) the rate of policy change ‖θ_k − θ_{k-1}‖ (faster change → larger optimal N to allow exploration before recentering), and (c) the KL divergence from initialization over time (rapidly growing KL → need more frequent recentering). The goal would be a heuristic or learned schedule for N that adapts to the observed training dynamics, removing the need for manual tuning.

Investigate whether the gradient equalizer enables longer-horizon training without collapse. The paper's training runs extend to a finite number of steps (100 iterations for the KL weight warm-up, plus additional steps for the feedback control phase—the exact total is not specified). A persistent concern with RL fine-tuning of generative models is that initial gains are followed by eventual collapse as the policy drifts far enough to exploit the reward model. SAGE-GRPO's Dual Trust Region is designed to prevent this, but the paper does not demonstrate that training remains stable at, say, 2×, 5×, or 10× the horizon shown in Figures 7 and 8. A strong stress-test would train SAGE-GRPO (w/ Dual Mov KL, two-stage 10⁻⁷ → 10⁻⁵ schedule) for an extended horizon until either (a) reward plateaus without degradation (evidence of stable equilibrium), (b) reward collapses (evidence that the Dual Trust Region delays but does not prevent drift), or (c) reward continues improving (evidence that the current horizon was too short to reach the optimum). The experiment would compare SAGE-GRPO against FlowGRPO w/o KL and CPS w/ Fixed KL at the same extended horizon, measuring both reward trajectories and human preference at the final checkpoint. A finding that SAGE-GRPO plateaus stably while baselines collapse would strengthen the paper's stability claim; a finding that SAGE-GRPO also eventually collapses (just later) would motivate research on even stronger position constraints or fundamentally different exploration mechanisms.

Measure whether improved exploration translates to improved sample efficiency. The paper's evaluation focuses on final performance after a fixed training budget, but an equally important practical question is: does manifold-constrained exploration enable the policy to reach a given reward level with fewer training steps? If SAGE-GRPO's rollouts are more reliably on-manifold and produce less noisy reward signals, the policy gradient estimates should have higher signal-to-noise ratio, enabling faster convergence. This is testable: train SAGE-GRPO, FlowGRPO, and CPS with matched infrastructure, but measure the number of training iterations required to reach specific reward thresholds (e.g., Overall reward of 0.3, 0.5, 0.7 under Setting B). The prediction is that SAGE-GRPO reaches each threshold in fewer steps. This is important for practical deployment: if SAGE-GRPO is 2× more sample-efficient, the computational cost of the precise SDE integration and gradient equalization may be more than offset by reduced training time. The paper does not currently make an efficiency argument, but the manifold-constrained exploration framing implies one—cleaner rollouts → better advantage estimates → faster learning. Testing this directly would add a practical dimension to the geometric motivation.

Test the gradient equalizer on non-GRPO policy gradient methods for diffusion models. The paper argues that gradient imbalance (‖∇ log π‖ ∝ 1/Σ_t^{1/2}) is a structural property of any Gaussian transition in diffusion models, not specific to GRPO. If true, DDPO (Black et al., 2023), DPOK (Fan et al., 2023), and other score-based policy gradient methods for diffusion should exhibit the same imbalance and benefit from the same equalization. A strong follow-up would implement DDPO on a standard image generation benchmark (e.g., Stable Diffusion with ImageReward) and compare training with and without the Temporal Gradient Equalizer. The equalizer would need to be adapted to the specific SDE/solver used in DDPO (requiring re-derivation of λ_t for that setting, which would also serve as a case study for practitioners), but the median normalization logic should transfer directly. A positive result (DDPO improves with equalization) would establish the equalizer as a general contribution to RL for diffusion models, not merely a GRPO-specific enhancement, significantly broadening its impact.

Practical Applications and Downstream Use Cases

Cost-efficient video generation alignment for production models. Organizations training text-to-video models for deployment (entertainment, advertising, education) currently face a difficult choice: either deploy models without alignment (accepting inconsistent or misaligned outputs) or invest in RL fine-tuning that risks degrading the model. SAGE-GRPO offers a recipe for stable alignment that does not require reward model fine-tuning or extensive hyperparameter search. The paper demonstrates this on HunyuanVideo1.5: under the alignment-focused reward configuration (Setting B), SAGE-GRPO with Dual Moving KL achieves Overall reward 0.8066, compared to 0.4773 for FlowGRPO w/o KL and −0.2172 for DanceGRPO w/o KL. The 69% relative improvement over the next best method, combined with the frozen (unmodified) VideoAlign evaluator, means that a production team can adopt SAGE-GRPO without the additional instability risk of reward model co-training. The adaptive KL scheduler (two-stage warm-up with feedback control) provides a principled alternative to manual KL tuning, which is notoriously brittle in video GRPO. The practical workflow would be: (1) train or select a reward model, (2) implement the precise SDE variance formula (Equation 6), (3) add median-normalized per-timestep gradient scaling, (4) configure the Dual Trust Region with a moving anchor updated every N steps and step-wise KL damping, (5) run training with the adaptive KL schedule. The paper's unified reimplementation framework (Section 4.1) means that engineering infrastructure is not a confounding factor—the method's gains are algorithmic, not due to optimized GPU kernels or memory management.

Improving existing video GRPO pipelines with lightweight modifications. Many research groups and companies already have working DanceGRPO or FlowGRPO pipelines. SAGE-GRPO does not require architectural changes to the base model, the reward model, or the training infrastructure—it modifies three components: the noise variance formula (replace the first-order approximation with the integrated variance), the loss computation (add per-timestep gradient scaling), and the KL constraint (add a periodic moving anchor alongside the existing step-wise term). The gradient equalizer, in particular, is a drop-in improvement that Figure 3 shows benefits all SDE formulations, including DanceGRPO, FlowGRPO, and CPS. A team running FlowGRPO could adopt just the equalizer (median-normalize per-timestep gradients using N_t = λ_t/Σ_t^{1/2}) without changing their SDE or KL strategy, and expect smoother reward curves and reduced risk of early plateau—potentially recovering training runs that would otherwise be discarded as failed. The moving anchor can be added to any existing KL setup by maintaining a second reference model updated every N steps, with negligible additional memory cost (storing one extra model copy). These modifications are incremental enough to be tested in existing pipelines without large-scale re-engineering.

Stable fine-tuning of video models for specific vertical domains. Many applications require aligning video generation to domain-specific criteria that generic reward models don't capture: medical procedure visualization (anatomical accuracy, procedural correctness), architectural flythroughs (structural consistency, lighting realism), sports replay generation (physical plausibility of motion, rule compliance). These domains often have small amounts of preference data (making reward model training unreliable) and high costs for failed generations (an anatomically incorrect medical video is not just low-quality, it's misleading). SAGE-GRPO's manifold-constrained exploration is particularly valuable here because it preserves the pre-trained model's basic video coherence while optimizing for domain-specific objectives—the Dual Trust Region prevents the policy from drifting so far that it produces structurally invalid videos (e.g., impossible anatomical configurations, physically implausible motion). The paper's user study shows 80.2–85.9% Visual Quality preference over CPS, FlowGRPO, and DanceGRPO—if this translates to domain-specific fine-tuning, it means SAGE-GRPO can improve alignment to a specialized reward without the visible degradation that makes other methods unusable for high-stakes applications. The practical workflow would use a domain-specific reward model (trained on expert annotations or task-specific metrics) in place of VideoAlign, keeping the SAGE-GRPO exploration and constraint mechanisms unchanged. The key risk—and what a practitioner would need to validate—is whether the domain-specific reward model is sensitive to off-manifold artifacts in the same way VideoAlign appears to be. If the specialized reward model is more robust to artifacts (e.g., it only checks object presence, not temporal coherence), SAGE-GRPO's advantage may diminish because the reward signal remains reliable even with off-manifold rollouts.

Self-improvement pipelines for video generation with reduced risk of degeneration. A growing paradigm in generative AI is iterative self-improvement: generate outputs, score them with a reward model, fine-tune on high-scoring outputs, repeat. In video generation, this is particularly prone to degeneration because each iteration's outputs become the next iteration's training data—off-manifold artifacts in one generation contaminate the training distribution for the next, creating a feedback loop of quality degradation. SAGE-GRPO's exploration constraints, particularly the precise SDE and Dual Trust Region, act as a stabilization mechanism for self-improvement loops: each iteration's rollouts are constrained to stay near the previous iteration's verified-valid policy (via the moving anchor), preventing the distributional drift that causes degeneration. The paper does not test multi-iteration self-improvement—all experiments are single-pass GRPO fine-tuning—but the framework is designed for it (Section 1 mentions that SAGE-GRPO enables "each update improves rollouts while keeping reward evaluation reliable"). A practitioner could run multiple rounds of SAGE-GRPO, each time using the previous round's final checkpoint as the new π₀ and resetting the moving anchor to that checkpoint. The prediction (testable, but unverified) is that SAGE-GRPO would sustain improvement over more iterations than FlowGRPO or CPS before collapsing. This has direct implications for scaling video model capabilities: if self-improvement is stable under SAGE-GRPO, organizations could amortize the cost of human preference collection over many self-improvement rounds, gradually improving their models without constant human oversight.