Video Generation Models Are Good Latent Reward Models

ArXiv: 2511.21541

🎯 Pitch

A video generation model can evaluate its own outputs directly from noisy latents, enabling dense reward supervision across the entire denoising chain without costly RGB decoding. This fully latent process reward feedback learning (PRFL) improves motion quality and human anatomy scores by large margins while cutting training time and memory usage.

---

1. Executive Summary

This paper introduces Process Reward Feedback Learning (PRFL), a framework that performs reward-based preference optimization for video generation entirely in latent space by repurposing the video generation model itself as a timestep-aware reward model. Using Wan2.1-14B as the base architecture on text-to-video and image-to-video generation tasks, PRFL trains a Process-Aware Video Reward Model (PAVRM) that evaluates video quality directly from noisy latent representations at arbitrary denoising timesteps, then optimizes the generator through single-step gradient backpropagation without VAE decoding. The method achieves substantial improvements in motion quality—including up to +56.00 in dynamic degree and +21.52 in human anatomy—while reducing memory consumption and accelerating training by at least 1.4× compared to pixel-space ReFL baselines, establishing that pre-trained video generation models are effective latent reward models whose process-level supervision can distribute learning signals across the full denoising trajectory.

2. Context and Motivation

The Core Problem: Video Generation Models Produce High-Fidelity Output But Fail on Motion Quality

The fundamental challenge this paper addresses is straightforward to state but difficult to solve: current video generation models can produce photorealistic frames, yet they consistently fail to generate videos that satisfy human preferences for motion quality, physical plausibility, and anatomical correctness. The paper frames this as an alignment problem—the models are trained to minimize a pixel-level reconstruction objective (the flow matching loss in Equation 2), but this objective does not capture what humans actually care about when they watch generated videos. A video with perfectly rendered individual frames might still contain unnatural motion trajectories, physically impossible object interactions, distorted human limbs, or temporal inconsistencies that break the viewer's sense of realism.

This gap is significant for several practical reasons the authors highlight throughout Section 1 and the experimental design:

• Real-world deployment of video generation: As video generation models move from research demonstrations to production applications (advertising, entertainment, education, synthetic data generation), their value depends not just on visual fidelity but on whether the outputs look believable to humans. A model that generates crisp but physically impossible motion is effectively unusable for most applications.
• The motion quality bottleneck: The paper's feasibility analysis (Section 3, Figure 3) reveals that even state-of-the-art video generation models (Wan2.1-14B) produce outputs where professional annotators flag significant portions as "unqualified" on physical plausibility and subject deformity dimensions. The annotation protocol described in Appendix A.2.3 defines three quality tiers—qualified, partially qualified, and unqualified—and the paper's dataset construction shows that many generated videos fall into lower tiers specifically because of motion artifacts, not visual quality issues.
• Resource allocation for alignment: Aligning video generation models with human preferences is computationally challenging in ways that image alignment is not. A single 81-frame video at 720P resolution requires orders of magnitude more memory and computation to process through pixel-space reward models than a single image, creating barriers to adoption of ReFL techniques that work well for images.

The Specific Gap: Reward Feedback Learning Cannot Scale to Video Generation

Reward Feedback Learning (ReFL) has been established as an effective approach for aligning image generation models with human preferences. The core idea is simple: use a differentiable reward model (typically a vision-language model trained on human preference data) to score generated images, then backpropagate the reward signal through the denoising process to update the generator's parameters. The paper cites ImageReward [42], AlignProp [32], and DRaFT [6] as canonical examples of this approach for images.

However, the paper identifies three interconnected barriers that prevent this approach from working effectively for video generation, and these barriers are not merely incremental scaling issues—they represent qualitative differences between image and video ReFL:

1. Evaluation Delay from Near-Complete Denoising. Pixel-space reward models (VLMs) require RGB images as input. For video generation models operating in latent space, this means the latent representation must be almost fully denoised and then decoded through the VAE decoder before the reward model can evaluate it. Figure 2 illustrates this clearly: in RGB ReFL, the generator must run nearly the full denoising chain (from $t=1$ to $t \approx 0$), then pass through the VAE decoder, before any reward signal is available. This creates a significant evaluation delay—the reward model sees only the final output, and gradient computation must wait for the full generation process to complete. Each training iteration is bottlenecked by this serial dependency: generate full video → decode to RGB → score with VLM → backpropagate.

2. GPU Memory Bottleneck from VAE Decoding. The memory problem is even more severe than the time problem. Backpropagating gradients from the reward model through the VAE decoder is required for end-to-end training, and for video—unlike images—the VAE must decode an entire sequence of frames. The paper quantifies this in Table 4: when attempting to process the full 81-frame video sequence, RGB ReFL encounters out-of-memory (OOM) errors on standard GPU hardware. The baseline that the paper compares against (ContentV [22]) works around this by decoding only the first frame—sacrificing holistic video quality assessment to stay within memory limits. This means existing approaches face a stark choice: either process the full video (impractical) or evaluate only a single frame (losing all temporal information).

3. Insufficient Supervision of Early Denoising Stages. This is the most subtle but arguably most important limitation. In diffusion-based and flow-based generation models, different denoising stages govern different aspects of the output. The paper makes a specific claim: early denoising stages establish fundamental structure and motion dynamics, while late stages refine visual details. RGB ReFL, by operating only on near-complete denoising outputs ($t \approx 0$), can only provide learning signals about final-frame visual quality. It cannot directly guide the early-stage decisions that determine whether the video's motion is physically plausible, whether the temporal structure is coherent, or whether human anatomy will remain consistent across frames.

The paper's sensitivity analysis (Table 3) empirically confirms this claim. When PRFL is restricted to sampling timesteps only from the late denoising stage, the model shows improvements in human anatomy (+7.3 points) but limited gains in dynamic degree (+22.00 → +44.00). When sampling from early and middle stages, dynamic degree improves more dramatically (+22.00 → +51.00), while human anatomy improvements are more modest. Full-stage sampling achieves the best of both worlds, producing the highest scores across all metrics. This directly validates that different quality aspects are optimized at different stages, and ReFL methods that only supervise the final output miss critical learning opportunities.

Why Existing Workarounds Fall Short

The paper identifies two recent approaches that attempt to address these limitations, but shows they are incomplete solutions:

ContentV [22] (first-frame-only optimization) bypasses the memory bottleneck by only decoding the first frame of the video and applying an image reward model (PickScore) to that single frame. While this makes training tractable, it fundamentally cannot capture motion quality, temporal consistency, or any video-level property. A video with perfect first frames but catastrophic temporal artifacts (melting, morphing, flickering) would receive a high reward under this approach. The paper's quantitative results (Tables 1 and 2) bear this out: RGB ReFL (which follows the ContentV approach) actually decreases subject consistency compared to the pretrained baseline (92.26 vs. 97.34 in the T2V 480P Inner Test Set), likely because optimizing for first-frame quality comes at the expense of temporal coherence.

DOLLAR [7] (VLM-based latent reward) trains a vision-language model to operate in latent space rather than pixel space, avoiding the VAE decoding step. However, this approach still functions as an outcome-based reward model—it evaluates only the final denoised latent, not intermediate states. It therefore addresses the memory and latency issues (barriers 1 and 2) but not the supervision issue (barrier 3). The paper characterizes this as "lacking timestep-wise optimization capability" and positions PAVRM as providing process-level supervision that DOLLAR cannot.

Gradient stopping and trajectory shortcuts [11, 39] attempt to distribute gradient updates across the denoising chain without full backpropagation through every step. These are image-focused methods that reduce memory by stopping gradients at intermediate points or using approximate shortcuts through the denoising trajectory. However, they still require fully denoised frames as input to the reward model (and thus VAE decoding), so they address memory but not the fundamental requirement of pixel-space rewards for the final evaluation.

Inference-time guidance with VGMs. The paper notes that VideoAlign [24] "briefly mentioned using VGMs as guidance during inference augmentation," meaning the video generation model's own features can steer sampling toward higher-quality outputs at inference time. But this is fundamentally different from training-time reward modeling—inference guidance doesn't change the model's parameters and can't produce the kind of persistent behavioral improvements that training does.

The Missing Piece: A Latent, Process-Level Reward Model for Video

The paper's central insight is that all existing approaches share a common deficiency: they cannot provide fine-grained, timestep-aware quality supervision in latent space. What the field needs is a reward model that can:

1. Evaluate video quality from noisy latent representations at any denoising timestep (not just $t \approx 0$)
2. Operate entirely in latent space without VAE decoding
3. Capture spatiotemporal quality signals (motion, structure, anatomy) rather than just per-frame visual quality
4. Be computationally efficient enough to backpropagate through during training

The paper's key hypothesis, stated in the introduction, is that pre-trained video generation models are naturally suited for this role. The reasoning is threefold:

• Inherent noise-aware feature extraction at arbitrary timesteps: Video generation models are explicitly trained to process noisy latent representations across the full range of $t \in [0, 1]$. Unlike VLMs, which are trained on clean RGB images and struggle with noisy inputs (Figure 3a shows VideoAlign's scores fluctuating dramatically across timesteps), VGMs have learned rich representations of video structure that are robust to noise levels.
• Sensitivity to generation artifacts: Because VGMs are trained on the same denoising task they're being asked to evaluate, their internal representations naturally encode information about what constitutes a "good" generation trajectory versus one that will produce artifacts. The paper demonstrates this empirically in Figure 3b: even a simple linear probe on frozen VGM features achieves 78.8% accuracy—matching the VideoAlign VLM baseline—without any fine-tuning.
• Native full-sequence processing without frame sampling: VGMs process video as spatiotemporal latent tensors, not frame-by-frame. This means they can evaluate motion quality and temporal consistency holistically rather than aggregating per-frame scores.

The paper explicitly connects this hypothesis to prior work in the image domain: LPO [46] "pioneered using diffusion models as noise-aware latent reward models for image generation." The current work extends this concept to video, where the challenges are substantially greater due to the spatiotemporal nature of video and the increased computational demands.

How This Paper Positions Itself

The paper frames its contribution not as proposing a fundamentally new type of reward model, but rather as recognizing and exploiting an underutilized capability of existing video generation architectures. The title itself—"Video Generation Models Are Good Latent Reward Models"—is a statement of discovery, not invention. The VGM already has the architecture and training needed for process-level quality assessment; the innovation lies in how to repurpose it effectively.

Specifically, the paper positions PRFL as addressing the full set of limitations that prior approaches face:

The paper's architecture decisions—query-based aggregation for variable-length videos, random timestep sampling during training, single-step gradient backpropagation—are all motivated by the practical requirements of making this repurposing work efficiently at scale.

The broader significance of the work, beyond the specific method, is that it establishes a new paradigm for video generation alignment: the generator and the evaluator share the same architectural backbone. This is efficient (no separate VLM to train and run), elegant (the reward model understands the generation process because it is the generation process), and empirically effective. The paper suggests this paradigm could extend to multi-aspect evaluation (aesthetics, semantics, not just motion) and to controllable generation, though these directions are left to future work.

3. Technical Approach

3.1 Reader Orientation

This paper develops a two-stage framework that first trains a video generation model to evaluate its own outputs at any stage of the denoising process, then uses that self-evaluation capability to optimize video quality through reinforcement learning—all without ever decoding videos to pixels during training. The system solves the problem that existing reward-based optimization methods for video generation crash due to GPU memory overflow when trying to process full videos through pixel-space reward models, while also providing a richer training signal by distributing quality feedback across the entire generation trajectory rather than just the final output.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components that operate in two sequential stages:

1. Video Generation Model (VGM) — the pretrained Wan2.1-14B model that serves both as the generator to be optimized and as the backbone whose frozen features provide the foundation for reward modeling. It operates on noisy latent representations $x_t \in \mathbb{R}^{F \times H \times W \times C}$ (a stack of $F$ video frames in latent space, each of spatial dimensions $H \times W$ with $C$ channels) conditioned on text prompt $p$ and timestep $t$.

2. Process-Aware Video Reward Model (PAVRM) — a modified version of the VGM that takes noisy latents $x_t$ at any timestep $t \in [0,1]$ (not just clean outputs), compresses the spatiotemporal features into a single quality-assessment token using learnable query attention, and outputs a scalar reward $r_\phi(x_t, t, p)$ indicating whether the video is on track to satisfy human preferences for motion quality. PAVRM is trained once on a preference-labeled dataset and then frozen.

3. Process Reward Feedback Learning (PRFL) Loop — the training procedure that optimizes the VGM using PAVRM's process-level rewards. It randomly samples a timestep $s$, runs the denoising chain without gradients from initial noise to timestep $s+\Delta t$, then takes one gradient-enabled denoising step to produce $x_s$, scores $x_s$ with PAVRM, and backpropagates through that single step to update the VGM parameters.

4. Supervised Fine-Tuning Regularizer — interleaved with PRFL updates, the VGM is also trained on a curated dataset of real high-quality videos using the standard flow matching loss. This prevents reward over-optimization (the generator learning to exploit PAVRM's blind spots rather than genuinely improving quality) and maintains generation diversity.

Information flows as follows: Stage 1 (PAVRM training) — real videos are encoded to latents $x_0 = E(V)$, noise is added at random timesteps to produce $x_t = (1-t)x_0 + t x_1$, PAVRM processes $x_t$ through frozen VGM DiT blocks followed by learnable query attention and MLP head to predict a binary quality label, and the prediction is optimized against human annotations via binary cross-entropy. Stage 2 (PRFL training) — text prompts are sampled from the preference dataset, initial noise is drawn, denoising proceeds without gradients up to timestep $s+\Delta t$, one gradient-enabled step produces $x_s$, PAVRM scores $x_s$ to produce reward $r_\phi(x_s, s, p)$, the VGM parameters are updated to maximize this reward, and this is alternated with SFT updates on real videos to maintain regularization.

3.3 Roadmap for the Deep Dive

• First: Flow Matching and ReFL foundations — the mathematical framework the VGM operates in (rectified flow) and the standard ReFL objective, since both are prerequisites for understanding what PRFL changes and why.
• Second: PAVRM architecture — how the video generation model is repurposed into a reward model, including the query-based aggregation mechanism and why it is needed to handle variable-length videos.
• Third: PAVRM training — the dataset construction, the random timestep sampling strategy, the binary cross-entropy objective, and why random timestep sampling is the critical design choice that enables process-level supervision.
• Fourth: PRFL optimization procedure — the single-step gradient backpropagation algorithm, how it avoids VAE decoding, and the alternation with SFT regularization to prevent over-optimization.
• Fifth: Design choices and their justifications — why specific architectural and algorithmic decisions (8 DiT blocks, query attention over pooling, BCE over BT loss, alternating training) were made over alternatives, supported by ablation evidence.

This order builds from mathematical foundations to reward model construction to optimization procedure, mirroring the two-stage training pipeline and ensuring each component's motivation is clear before its mechanics are explained.

3.4 Detailed, Sentence-Based Technical Breakdown

This is an architecture and training methodology paper whose core idea is that pre-trained video generation models, when augmented with query-based spatiotemporal aggregation and fine-tuned with timestep-aware binary preference labels, can serve as effective process-level reward models operating entirely in noisy latent space, enabling memory-efficient, full-trajectory optimization of video generators.

---

Rectified Flow Foundation

The video generation model operates in the rectified flow framework, which the paper adopts from Liu et al. (2023) and Lipman et al. (2023). Understanding this framework is essential because it defines what the model predicts, how timesteps relate to noise levels, and why random timestep sampling during reward model training is valid.

The transport path. Rectified flow establishes a continuous linear path between the data distribution and a noise distribution. Given a clean video $x_0$ (in latent space, after VAE encoding) sampled from the data distribution $q(x_0)$ and a noise tensor $x_1$ sampled from a standard Gaussian $p(x_1) = \mathcal{N}(0, I)$, the framework defines the intermediate representation at any timestep $t \in [0, 1]$ as:

$x_t = (1 - t) x_0 + t x_1$

where $x_t \in \mathbb{R}^{F \times H \times W \times C}$ is the noisy latent at timestep $t$, $x_0$ is the clean latent representation of the video (obtained by encoding the RGB video through the VAE encoder $E$: $x_0 = E(V)$), and $x_1 \sim \mathcal{N}(0, I)$ is pure Gaussian noise.

What it computes: This is a deterministic linear interpolation between clean data and pure noise. At $t = 0$, $x_0$ is the clean latent. At $t = 1$, $x_1$ is pure noise. At intermediate $t$, $x_t$ is a mixture where the signal-to-noise ratio decreases linearly with $t$. The key property is that given $x_0$ and $x_1$, every intermediate state is uniquely determined—there is no stochasticity in the forward process.

Why this form: The linear interpolation path is simpler than the stochastic forward process used in DDPM-style diffusion models. It enables a deterministic relationship between noise and data, which is crucial for the paper's reward model training: if the final output $x_0$ has a known quality label, then every intermediate state $x_t$ along the path to that output inherits the same label because the path is deterministic and linear. This is the mathematical justification for why PAVRM can be trained on noisy latents with labels derived from final output quality—the paper explicitly states this in Section 4.2: "since $x_t$ follows a deterministic linear interpolation from noise to data, preference labels of final outputs naturally propagate to intermediate states."

The velocity prediction objective. The neural network (composed of DiT blocks) parameterized by $\theta$ does not predict the clean data $x_0$ or the noise $x_1$ directly. Instead, it predicts the velocity field $v_\theta(x_t, t, c)$ that transports $x_t$ toward $x_0$, where $c$ represents conditioning information (text prompt $p$ and optionally an input image for I2V tasks). The training objective, which the paper calls SFT loss, is the flow matching objective:

$\mathcal{L}_{\text{FM}}(\theta) = \mathbb{E}_{t \sim U(0,1), x_0 \sim q(x_0), x_1 \sim p(x_1)}\left[\|v_\theta(x_t, t, c) - v\|^2\right]$

where $v = x_1 - x_0$ is the ground-truth transport direction from data to noise (sometimes called the "data-to-noise velocity"), $t \sim U(0,1)$ means the timestep is sampled uniformly from the unit interval, $x_0$ is the clean latent from the training dataset, $x_1$ is random Gaussian noise, and $c$ is the conditioning information (text prompt and optionally image).

What it computes: For a randomly sampled timestep $t$ and a randomly constructed noisy input $x_t$ (via the interpolation above), the model predicts a velocity vector $v_\theta$ and is penalized by the squared Euclidean distance between its prediction and the true velocity $v = x_1 - x_0$. The expectation is over the uniform distribution of timesteps and over the empirical data distribution. The output is a scalar loss value that, when minimized, teaches the model to push noisy latents toward clean data along straight-line paths.

Why this form: The flow matching formulation has two advantages over DDPM-style noise prediction. First, the uniform timestep sampling ($t \sim U(0,1)$) means the model is trained equally on all noise levels, which aligns naturally with the paper's goal of building a reward model that works at arbitrary timesteps. Second, the velocity prediction $v = x_1 - x_0$ directly captures the transport direction, making it straightforward to take denoising steps by moving in the negative velocity direction: $x_{t-\Delta t} = x_t - \Delta t \cdot v_\theta(x_t, t, c)$.

---

Standard ReFL Objective and Its Video-Specific Limitations

The paper builds on Reward Feedback Learning as introduced by Xu et al. (2023) for image generation. The standard ReFL objective is:

$\mathcal{L}_{\text{ReFL}} = -\lambda \mathbb{E}_{x_0 \sim \text{VGM}_\theta} \left[r_\phi(D(x_0))\right] + \mathcal{L}_{\text{FM}}(\theta)$

where $\lambda > 0$ is a hyperparameter controlling the strength of reward optimization relative to the SFT regularization, $\text{VGM}_\theta$ denotes the video generation model parameterized by $\theta$ (the generator being optimized), $r_\phi$ is a reward model parameterized by $\phi$ that scores outputs, $D$ is the VAE decoder that converts latents to RGB pixels, and $\mathcal{L}_{\text{FM}}$ is the flow matching loss from Equation 2.

What it computes: The first term $-\lambda \mathbb{E}[r_\phi(D(x_0))]$ is a reward maximization objective: it encourages the generator to produce outputs that the reward model rates highly. The negative sign converts maximization into minimization for gradient descent. The expectation is over videos generated by the current VGM ($x_0 \sim \text{VGM}_\theta$), meaning the reward model evaluates the generator's own outputs, not ground-truth data. The second term $\mathcal{L}_{\text{FM}}$ is the standard flow matching loss on real training data, which acts as a regularizer preventing the generator from drifting too far from its pretrained behavior (reward over-optimization). The total loss is a weighted combination where $\lambda$ trades off between following the reward signal and staying close to the pretrained distribution.

Why this form: The two-term structure addresses the exploration-exploitation tension in preference optimization. Pure reward maximization ($-\lambda \mathbb{E}[r_\phi]$ alone) would cause the model to collapse to outputs that maximally exploit the reward model's imperfections—generating videos that score highly under $r_\phi$ but are not actually high quality. The SFT regularization term $\mathcal{L}_{\text{FM}}$ anchors the model to the data distribution, ensuring that improvements come from genuine quality enhancements rather than reward hacking. The expectation over $\text{VGM}_\theta$ (rather than a fixed dataset) means the reward model evaluates the generator's current outputs, enabling online learning where the generator and reward model interact. This is distinct from offline approaches like RWR where rewards weight pre-generated samples.

The three failure modes for video. The paper identifies why this formulation, while effective for images, breaks down for video:

1. The decoding bottleneck: $D(x_0)$ for a video means running the VAE decoder on every frame. For an 81-frame video at 720P resolution, this requires decoding 81 separate images through the VAE, storing all intermediate activations for backpropagation. Table 4 quantifies this: RGB ReFL with full frames encounters out-of-memory errors. The paper's baseline (ContentV-style) workaround decodes only the first frame, but then $r_\phi(D(x_0))$ becomes a first-frame-only reward that ignores all video-level properties.

2. The late-timestep limitation: Even in image ReFL, the reward model operates on $x_0$ (or near-$x_0$, i.e., $t \approx 0$). This means the reward signal can only influence the final denoising steps where $x_0$ is being refined. The early denoising steps that establish global structure and motion patterns receive no direct reward signal because their contribution to the final output is mediated through many intermediate steps, and gradients flowing backward through these steps are noisy or memory-prohibitive.

3. Computational intractability of full backpropagation: Even if memory were unlimited, backpropagating through all 40 denoising steps (the standard inference budget) and the VAE decoder would make each training iteration prohibitively slow. The paper's alternative—sampling only a late timestep—reduces computation but at the cost of missing early-stage supervision.

---

PAVRM Architecture: Repurposing the VGM as a Reward Model

PAVRM is the paper's core architectural contribution. It transforms a video generation model into a quality assessment model that operates on noisy latents at arbitrary timesteps. The architecture consists of three components: a frozen-and-fine-tuned feature extractor (the first few DiT blocks of the VGM), a query-based spatiotemporal aggregation module, and a prediction head.

Feature extraction from noisy latents. PAVRM takes as input a noisy latent video $x_t \in \mathbb{R}^{F \times H \times W \times C}$ at any timestep $t \in [0, 1]$ and a text prompt $p$. The text is encoded through the T5 text encoder $T(\cdot)$ (frozen, inherited from the pretrained Wan2.1 model) to produce text embeddings. The noisy latent and text embeddings are processed through the first $K$ DiT (Diffusion Transformer) blocks of the VGM:

$h = \text{DiT}_\phi(x_t, t, T(p)) \in \mathbb{R}^{F \times H \times W \times D}$

where $h$ is the output feature tensor from the final selected DiT block, $F$ is the number of frames, $H \times W$ are the spatial dimensions of the latent representation at that layer, $D$ is the feature dimension (5120 for Wan2.1-14B), $\phi$ denotes the parameters of these DiT blocks, and $t$ is the timestep embedding injected into each block via the standard diffusion transformer conditioning mechanism.

What it computes: Given a noisy video representation and a text description, the DiT blocks perform self-attention and cross-attention operations that refine the representation based on the text conditioning and the noise level. The output $h$ is a spatiotemporal feature grid where each position contains a $D$-dimensional vector encoding both local visual information and, through the attention operations, global context about structure and motion. At high noise levels ($t \rightarrow 1$), these features capture coarse structural layout; at low noise levels ($t \rightarrow 0$), they capture fine details and textures.

Why the first 8 DiT blocks specifically: The paper's feasibility analysis (Figure 3b) shows that probing features from any DiT layer (L8 through L40) with a simple linear classifier yields uniform 78.8% accuracy on motion quality assessment—matching the VideoAlign VLM baseline. This demonstrates that motion-relevant information is distributed throughout the network, not concentrated in later layers. Using only the first 8 blocks achieves the same representational quality as using all 40 while being substantially more computationally efficient (fewer parameters to fine-tune, less memory). However, the ablation in Table 8 reveals a more nuanced picture: when DiT blocks are fine-tuned (not just probed), performance peaks at 16 blocks (85.51% average accuracy) and degrades with more blocks (full 40-block model: 83.27%). The paper hypothesizes that deeper layers overfit to model-specific generation details rather than learning generalizable quality assessment, making 8-16 blocks the sweet spot. The default configuration uses 8 blocks for efficiency.

Query-based spatiotemporal aggregation. The feature tensor $h$ has dimensions $F \times H \times W \times D$, representing a variable-length sequence of spatiotemporal tokens (the length varies with video resolution and frame count). To convert this variable-length representation into a fixed-size quality embedding, PAVRM employs learnable query attention—a mechanism inspired by the DETR object detection architecture and the Perceiver family of models:

1. Flattening: The spatiotemporal grid $h$ is reshaped into a sequence $\hat{h} \in \mathbb{R}^{N \times D}$ where $N = F \cdot H \cdot W$ is the total number of tokens.

2. Query attention computation: A single learnable query vector $q \in \mathbb{R}^{1 \times D}$ attends to all $N$ tokens:

$z_{\text{obs}} = \frac{\exp(q(\hat{h}W_K)^T / \sqrt{D})}{\sum \exp(q(\hat{h}W_K)^T / \sqrt{D})} (\hat{h}W_V) \in \mathbb{R}^{1 \times D}$

where $W_K \in \mathbb{R}^{D \times D}$ and $W_V \in \mathbb{R}^{D \times D}$ are learnable projection matrices for keys and values respectively, $q$ is the learnable query vector, $\hat{h}W_K$ produces key vectors for each of the $N$ tokens, $q(\hat{h}W_K)^T$ computes attention scores between the query and each token, the softmax normalizes these scores into a probability distribution over tokens, and the weighted sum $\text{softmax} \cdot (\hat{h}W_V)$ produces a single $D$-dimensional vector $z_{\text{obs}}$ that summarizes the observation.

What it computes: The query vector $q$ acts as a learned "question" that asks: "across all spatial positions and all frames, which features are most relevant for assessing video quality?" The attention mechanism computes how much each spatiotemporal token should contribute to the answer, then aggregates the value-transformed tokens into a single summary vector $z_{\text{obs}}$. Unlike mean pooling (which averages all tokens equally) or max pooling (which selects the most extreme feature per dimension), query attention learns to selectively attend to quality-relevant regions—for example, focusing on regions where motion artifacts or anatomical deformities are likely to appear.

Why query attention over pooling: Table 5 provides the empirical justification. Mean pooling (averaging all tokens) achieves 83.37% average accuracy. Max pooling (selecting the maximum activation per feature dimension) degrades severely in later timesteps (73.72% at $t \in (0.8, 1.0]$) and averages only 78.47%. Attention without a learnable query (using a fixed aggregation) achieves 83.42%. Attention with a learnable query achieves 84.18%—the best performance with the most consistent behavior across timestep ranges. The paper's explanation is that max pooling is sensitive to noise (a single spuriously high activation can dominate), mean pooling dilutes quality-relevant signals by averaging them with irrelevant background features, and query attention learns to filter out content-specific correlations in favor of quality-relevant patterns.

Quality-aware token construction. After attention aggregation, PAVRM constructs the final representation by combining the observation summary $z_{\text{obs}}$ with the learnable query $q$ itself:

$z = z_{\text{obs}} + q \in \mathbb{R}^{1 \times D}$

where $z_{\text{obs}}$ captures the content-dependent quality signals extracted from the video features, and $q$ serves as a content-agnostic quality prior.

What it computes: This residual connection means the final representation $z$ contains both what the model observed in the specific video ($z_{\text{obs}}$) and a learned bias that encodes general expectations about video quality independent of content ($q$). If the video features contain ambiguous quality signals (e.g., a video with high visual fidelity but subtle motion artifacts), the query prior can push the representation toward the quality-relevant subspace.

Why this form: The paper states this design "enables the model to reason about generation quality independent of content correlations." Without the skip connection, the model might learn to associate specific content patterns (e.g., "videos of guitars tend to be labeled as good in the training set") with quality labels, rather than learning to detect actual quality indicators. The query prior provides a stable baseline that the observation features modulate, making the model less susceptible to content-specific biases in the training data.

Prediction head. The $D$-dimensional representation $z$ is mapped to a scalar reward through a three-layer MLP:

$r_\phi(x_t, t, p) = \text{MLP}(z) \in \mathbb{R}$

where the MLP consists of three linear layers with non-linear activations (architecture details not specified beyond "three-layer MLP"), and the output is a scalar logit representing the model's confidence that the video satisfies motion quality preferences.

What it computes: The MLP projects the $D$-dimensional quality embedding onto a single real-valued score. This score is passed through a sigmoid function $\sigma(r_\phi)$ to produce a probability in $[0, 1]$ during training, interpreted as the predicted probability that the video is of good quality.

Why three layers: The paper does not explicitly justify the three-layer depth, but in the broader context of the architecture, this choice reflects standard practice for reward model heads: a shallow MLP (one layer) might not capture non-linear interactions between quality dimensions, while a deep MLP would add parameters that could overfit to the training set given the relatively small reward model dataset (24,000 samples). Three layers provide a reasonable trade-off.

---

PAVRM Training: Dataset, Objective, and Timestep Strategy

Dataset construction. PAVRM is trained on a binary preference dataset $\mathcal{D}_{\text{RM}} = \{(V_i, p_i, y_i)\}_{i=1}^N$ where $N \approx 23,500$ (24,000 total after filtering minus validation and test splits), $V_i$ is a generated video, $p_i$ is the text prompt used to generate it, and $y_i \in \{0, 1\}$ is a binary label where 1 indicates "good quality" and 0 indicates "bad quality."

The dataset creation process, detailed in Appendix A.2.3, is:

1. Source collection: 31,000 high-quality human portrait videos are collected from online sources.
2. Captioning: Text prompts are generated for each video using a video captioning model.
3. Generation: The first frame and text prompt of each real video are fed into Wan2.1-14B-I2V to generate corresponding synthetic videos. Each input condition produces one generated video.
4. Coarse filtering: Videos with obvious defects (black screens, visible watermarks) are removed.
5. Manual annotation: Professional annotators rate each generated video on two dimensions: Physical Plausibility and Subject Deformity. Each dimension uses a three-level scale: qualified, partially qualified, unqualified (detailed criteria in Table 6).
6. Label construction: Videos rated as "qualified" on both dimensions are labeled as good ($y = 1$). Videos rated as "unqualified" on both dimensions are labeled as bad ($y = 0$). Videos with mixed ratings or "partially qualified" on either dimension are excluded to "enhance distinctiveness and reduce annotation ambiguity."
7. Final dataset: 24,000 video pairs (real and generated), with generated videos used for PAVRM training and real videos used for SFT data in the PRFL stage. The dataset is split into ~23,500 training, 100 validation, and 400 test samples. Test set annotations use majority voting across at least three independent annotators for reliability.

Why binary labels over continuous scores: Binary classification simplifies the training objective and avoids the need for calibrated continuous quality scores, which are difficult to obtain consistently across annotators. The "partially qualified" videos are explicitly excluded to create a clearer separation between positive and negative examples, which the paper argues "enhances distinctiveness." The trade-off is that the model learns only a coarse good-versus-bad distinction rather than a fine-grained quality scale, but this is sufficient for the reinforcement learning objective that follows (the generator only needs a directional signal: "make videos more like the good ones").

Noisy latent construction during training. For each training sample $(V, p, y)$, PAVRM does not evaluate the clean video directly. Instead, it constructs noisy latents using the rectified flow interpolation:

1. Encode the clean video: $x_0 = E(V)$ where $E$ is the frozen VAE encoder.
2. Sample random noise: $x_1 \sim \mathcal{N}(0, I)$ (standard Gaussian, same shape as $x_0$).
3. Sample a random timestep: $t \sim U(0, 1)$ (uniform distribution on the unit interval).
4. Construct the noisy latent: $x_t = (1-t)x_0 + t x_1$.
5. Feed $(x_t, t, p)$ to PAVRM to produce $r_\phi(x_t, t, p)$.

Why random timestep sampling is the critical design decision: This is arguably the most important training detail in the paper. By sampling $t \sim U(0, 1)$ uniformly, PAVRM sees noisy latents at all noise levels—from nearly clean ($t \approx 0$, mostly $x_0$) to nearly pure noise ($t \approx 1$, mostly $x_1$)—and learns to associate quality labels with representations at every stage. This is what enables process-level supervision during PRFL: the reward model can evaluate a partially denoised latent at any intermediate timestep and provide a meaningful quality signal.

The alternative—training only on clean or near-clean latents ($t \approx 0$)—would produce an outcome-based reward model that, while operating in latent space, would only work near the end of denoising and would fail to provide early-stage guidance. This is exactly the limitation the paper identifies in DOLLAR [7]. The paper's feasibility analysis (Figure 3a) empirically demonstrates why this is necessary: a VLM-based reward model (VideoAlign-MQ) shows wildly fluctuating scores when evaluating videos decoded from intermediate timesteps, proving that pixel-space models cannot generalize to noisy inputs. VGMs, by contrast, are designed to process noisy inputs at arbitrary timesteps, making random timestep sampling not just possible but natural.

The mathematical justification that labels propagate to intermediate states is specific to rectified flow: because $x_t = (1-t)x_0 + t x_1$ is a deterministic linear function of $x_0$, $x_1$, and $t$, the quality of the final output $x_0$ (which determines the label $y$) is directly tied to the quality of every intermediate state. In a stochastic diffusion process (DDPM-style), the forward process injects random noise at each step, making the relationship between intermediate states and final quality noisier. The rectified flow formulation's determinism makes the label propagation assumption cleaner.

Training objective. PAVRM is trained with standard binary cross-entropy loss:

$\mathcal{L}_{\text{PAVRM}} = -\mathbb{E}_{t, (V,p,y)}\left[y \log \sigma(r_\phi(x_t, t, p)) + (1 - y) \log (1 - \sigma(r_\phi(x_t, t, p)))\right]$

where $t \sim U(0, 1)$ is the randomly sampled timestep, $\sigma(\cdot)$ is the sigmoid function that maps the raw logit $r_\phi$ to a probability in $(0, 1)$, $y \in \{0, 1\}$ is the binary quality label, and the expectation is over the training dataset and the uniform timestep distribution.

What it computes: For each training sample, the model produces a predicted probability $\hat{y} = \sigma(r_\phi(x_t, t, p))$ that the video is of good quality. The first term $y \log \hat{y}$ penalizes low predicted probabilities when the true label is 1 (the model is under-confident about good videos). The second term $(1-y) \log (1-\hat{y})$ penalizes high predicted probabilities when the true label is 0 (the model is over-confident about bad videos). The sum of these two terms, negated, forms the log-likelihood of the label under the model's Bernoulli prediction. Minimizing this loss maximizes the probability the model assigns to the correct label.

Why BCE over Bradley-Terry (BT) loss: The paper includes an ablation (Table 7) comparing BCE with pairwise BT loss (which models the probability that a good video is preferred over a bad video in a pairwise comparison). BCE achieves 80.05% average accuracy versus 79.85% for BT—essentially identical performance. However, BCE has two practical advantages: it "eliminates the computational overhead of pair construction" (BT requires forming explicit win-lose pairs from the binary labels) and it provides more direct per-sample learning signals (each video contributes one gradient update rather than pairs of videos sharing a single update). The paper also notes a timestep-dependent trade-off: BT performs better in high-noise regions ($t > 0.6$), where pairwise comparisons might be more robust because absolute quality is harder to assess, while BCE excels at intermediate timesteps ($t \leq 0.6$) where structure is forming and absolute quality signals are stronger. The overall tie with lower computational cost makes BCE the default.

Training configuration. The paper uses the AdamW optimizer with three different learning rates for different parameter groups:

• Query attention parameters and MLP head: learning rate $1 \times 10^{-5}$
• DiT blocks (feature extractor): learning rate $1 \times 10^{-6}$
• VAE encoder and text encoder: frozen (not trained)

The differential learning rates reflect that the DiT blocks are already well-pretrained and only need minor adaptation, while the attention and MLP components are randomly initialized and need more aggressive optimization. The VAE encoder is frozen because it is a generic compression module not specific to quality assessment. The paper does not specify batch size, number of training epochs, or other optimization hyperparameters for PAVRM training in the main text.

Stratified evaluation during testing. For evaluation only (not training), the paper samples timesteps from five fixed intervals to assess PAVRM's performance across the denoising trajectory: $[0, 0.2]$, $(0.2, 0.4]$, $(0.4, 0.6]$, $(0.6, 0.8]$, and $(0.8, 1.0]$. Each test sample is evaluated once per interval with a random timestep within that interval, and the final metric is the average accuracy across all intervals.

Why stratified evaluation: Random timestep sampling during training ensures the model is exposed to all noise levels, but evaluating with the same random strategy could mask performance variations across timesteps. The stratified approach ensures that PAVRM is tested at representative points across the full range and that the reported accuracy reflects consistent performance, not just good average performance driven by easy timestep ranges. Table 5 shows that PAVRM with query attention maintains 83-85% accuracy across all intervals, confirming that random timestep training produces robust representations.

Freezing during PRFL. After training, PAVRM's parameters $\phi$ are frozen. During PRFL optimization, PAVRM is used in inference mode (no gradient updates to $\phi$), and gradients flow through PAVRM to update the VGM parameters $\theta$. This is the standard actor-critic separation: the reward model provides a fixed evaluation criterion while the generator adapts to maximize scores under that criterion.

---

PRFL Optimization Procedure

PRFL is the training algorithm that uses PAVRM's process-level rewards to optimize the video generation model. The key innovation is that it operates entirely in latent space at intermediate timesteps, eliminating VAE decoding and distributing the learning signal across the denoising trajectory.

Algorithm structure. Algorithm 1 in the paper presents the full PRFL loop. Each training iteration consists of two alternating updates:

1. SFT update (lines 4-8 of Algorithm 1): A standard flow matching update on real high-quality videos from $\mathcal{D}_{\text{SFT}}$ to maintain regularization.
2. PRFL update (lines 9-16): A reward-maximization update using PAVRM's evaluation of a partially denoised latent.

The two updates alternate at each iteration, meaning the model sees both real data (maintaining generation quality and diversity) and reward-driven optimization (improving motion quality) in equal measure.

SFT regularization step. The SFT update follows the standard flow matching procedure:

1. Sample a real video and prompt $(V, p) \sim \mathcal{D}_{\text{SFT}}$ from the curated dataset of real high-quality videos.
2. Sample a timestep $t_{\text{sft}} \sim U(0, 1)$ uniformly.
3. Encode the video: $x_0 = E(V)$ using the frozen VAE encoder.
4. Sample noise: $x_1 \sim \mathcal{N}(0, I)$.
5. Construct the noisy latent: $x_{t_{\text{sft}}} = (1 - t_{\text{sft}}) x_0 + t_{\text{sft}} x_1$.
6. Compute the flow matching loss: $\mathcal{L}_{\text{FM}} = \|v_\theta(x_{t_{\text{sft}}}, t_{\text{sft}}, c) - (x_1 - x_0)\|^2$.
7. Update VGM parameters: $\theta \leftarrow \theta - \eta \nabla_\theta \mathcal{L}_{\text{FM}}$.

What it computes: This is identical to the pretraining objective—the model learns to predict the transport velocity from noisy to clean latents for real videos. By including this step, the VGM maintains its ability to generate videos that match the distribution of real high-quality content, preventing it from drifting toward PAVRM-pleasing but unnatural outputs.

Why alternate rather than sum the losses: The paper follows the established ReFL convention (Equation 3) of combining reward and SFT losses, but implements them as alternating updates rather than a single combined gradient step. Alternating updates provide independent control over the SFT and reward optimization dynamics—the optimizer state (Adam moments) evolves separately for each objective, preventing the reward signal from dominating the SFT signal or vice versa. This is particularly important because the reward and SFT losses operate on different data distributions (generated videos for reward, real videos for SFT) and have different variance characteristics (reward signal variance depends on PAVRM's calibration, while SFT loss variance depends on the data distribution).

PRFL reward maximization step. The core innovation of PRFL is how it computes the reward signal without full denoising or VAE decoding. The procedure is:

1. Sample a text prompt $p$ from $\mathcal{D}_{\text{PRFL}}$ (the same dataset used for PAVRM training, but using only the prompts for generation, not the pre-generated videos).
2. Sample initial noise: $x_T \sim \mathcal{N}(0, I)$, where $T$ is the maximum timestep (typically $T = 1$ in continuous time, or the nearest discrete step).
3. Sample a target timestep: $s \sim U(0, 1)$ uniformly.
4. Gradient-free denoising rollout (lines 11-13): Starting from $t = T$ down to $t = s + \Delta t$, take denoising steps without tracking gradients:

$x_{j-1} = x_j - \Delta t \cdot v_\theta(x_j, j, c) \quad \text{(with torch.no\_grad)}$

where $j$ iterates from $T$ down to $s + \Delta t$, $\Delta t = 1/N$ is the discrete step size with $N$ total denoising steps (the paper uses 40 inference steps during evaluation; the training step count is specified as 1000 training steps in Appendix A.2.1), and $c$ represents conditioning (text prompt, and optionally image for I2V).

5. Single gradient-enabled denoising step (line 14): Take one final denoising step with gradient tracking:

$x_s = x_{s+\Delta t} - \Delta t \cdot v_\theta(x_{s+\Delta t}, s + \Delta t, c) \quad \text{(with torch.enable\_grad)}$

6. Evaluate reward at intermediate timestep (line 15): Feed the partially denoised latent $x_s$ and the prompt $p$ to PAVRM:

$r_s = \text{PAVRM}(x_s, s, p) \in \mathbb{R}$

7. Compute reward loss: The objective is to maximize the reward:

$\mathcal{L}_{\text{PRFL}} = -\lambda \cdot r_s$

where $\lambda > 0$ controls the optimization strength. The negative sign converts reward maximization to loss minimization for gradient descent.

8. Update VGM parameters (line 16): $\theta \leftarrow \theta - \eta \nabla_\theta \mathcal{L}_{\text{PRFL}}$.

What this computes, operationally: The critical insight is that gradients flow through only one denoising step (step 5) and the PAVRM evaluation (step 6), but the input to that step $x_{s+\Delta t}$ was produced by a chain of gradient-free steps that simulate the full generation process up to timestep $s+\Delta t$. This means the gradient $\nabla_\theta \mathcal{L}_{\text{PRFL}}$ captures how changes to the VGM parameters affect the quality of the video at timestep $s$, given that the earlier part of the denoising trajectory was produced by the current VGM parameters without gradient. The single-step gradient backpropagation creates a learning signal at whatever timestep $s$ was sampled—if $s$ is early (say $s = 0.8$, meaning the latent is mostly noise), the model learns about early-stage structural decisions; if $s$ is late (say $s = 0.1$, meaning the latent is nearly clean), the model learns about detail refinement.

Why single-step backpropagation works: The paper's key technical contribution is recognizing that backpropagating through the entire denoising chain is unnecessary and memory-prohibitive, but backpropagating through only the final step (as in RGB ReFL) misses early-stage supervision. Single-step backpropagation at a random timestep provides a middle ground: it captures the local sensitivity of video quality to the denoising step at timestep $s+\Delta t \rightarrow s$, which is sufficient to provide a useful gradient signal. Over many training iterations with randomly sampled $s$, the model receives gradient information about all stages of the denoising process. The gradient-free rollout provides the necessary context so that the gradient-enabled step sees a realistic "what would the model currently produce" input.

Why no VAE decoding: Because PAVRM operates on noisy latents $x_s$ directly, there is no VAE decoder $D(\cdot)$ in the computation graph. This eliminates the memory bottleneck that Table 4 quantifies: PRFL with full 81-frame processing uses 66.81 GB of VRAM versus OOM for RGB ReFL with full frames. The VAE decoder for video requires storing activations for all 81 frames simultaneously during backpropagation, which exceeds GPU memory limits. PRFL sidesteps this by working entirely in latent space.

Timestep sampling for PRFL. The paper experiments with different timestep sampling strategies for PRFL (Table 3):

• Full-stage sampling ($s \sim U(0, 1)$): The default, achieves the best overall average score (89.58).
• Early-stage sampling only ($s$ in the first third of denoising): Strong dynamic degree (51.00) but weaker human anatomy (87.52).
• Middle-stage sampling only ($s$ in the middle third): Strong VBench dynamic degree (76.39) and good overall performance (87.02 avg).
• Late-stage sampling only ($s$ in the final third): Good human anatomy (91.54) but weaker dynamic degree (44.00).

Why full-stage sampling is optimal: Different quality dimensions are determined at different denoising stages. Early stages establish motion trajectories and global structure—supervising these stages improves dynamic degree. Late stages refine anatomical details—supervising these stages improves human anatomy scores. Middle stages handle the transition from coarse structure to fine detail. Sampling from all stages ensures the generator receives balanced feedback across all quality dimensions, whereas restricting to any single stage creates an imbalance where some aspects improve at the expense of others. This empirical finding is the strongest evidence for the paper's central claim that process-level supervision is necessary for video generation alignment.

Why not backpropagate through the full rollout: Backpropagating through all denoising steps from $t=T$ to $t=0$ would provide gradient information about every step's contribution to final quality, but would require storing all intermediate activations for all 40-1000 steps (depending on training step count), making memory consumption proportional to the number of steps. This is exactly the "depth-efficiency dilemma" the paper references from prior image ReFL work [6, 32, 42]. The single-step approximation assumes that the primary learning signal comes from understanding how the current denoising step at timestep $s$ affects PAVRM's quality assessment, while the earlier steps provide necessary context but not critical gradient paths. This assumption is validated by the strong empirical results—if full backpropagation were necessary, the single-step approximation would fail to produce improvements.

Training hyperparameters for PRFL (from Section 5.1):

• Optimizer: AdamW
• Learning rate: $5 \times 10^{-6}$ for the VGM parameters
• Sequence parallel size: 4 (distributed training across GPUs)
• Global batch size: 30 (batch size of 6 with gradient accumulation of 5)
• Training duration: one epoch over the text-video pairs
• Number of training denoising steps: 1000 (UniPCMultistepScheduler with 1000 training steps)
• Number of inference steps: 40 (standard evaluation setting)
• Resolution: 480P and 720P (separate experiments)

Why these specific hyperparameters: The learning rate $5 \times 10^{-6}$ is lower than the PAVRM learning rates because the VGM is a large pretrained model that needs careful fine-tuning to avoid catastrophic forgetting. The gradient accumulation (batch size 6 × 5 accumulations = effective batch size 30) allows training with limited GPU memory while maintaining statistical stability in the gradient estimates. The single-epoch training duration reflects that PRFL is a fine-tuning stage, not a from-scratch training—the model already generates plausible videos and only needs alignment refinement.

---

Design Choices and Their Justifications

Why 8 DiT blocks for PAVRM (not more, not fewer): The paper's analysis (Figure 3b, Table 8) shows that 8 blocks provide sufficient representational capacity while being computationally efficient. With 16 blocks, accuracy improves from 84.18% to 85.51% average, but at increased computational cost. With 24 blocks, accuracy is 85.25%—slightly worse than 16 blocks despite more parameters. With 40 blocks (full model), accuracy drops to 83.27% and computational cost is highest. This non-monotonic relationship suggests that deeper layers encode model-specific generation details (artifacts, texture patterns specific to Wan2.1) that interfere with general quality assessment. The early layers capture more universal structural and motion features. The paper defaults to 8 blocks as a sweet spot, trading a small accuracy loss (~1.3% vs. the 16-block peak) for substantially lower computational cost.

Why query attention over mean/max pooling: Mean pooling (83.37% avg) treats all spatiotemporal positions equally, diluting quality-relevant features (e.g., a distorted hand in one corner) with irrelevant background features. Max pooling (78.47%) is vulnerable to noise—a single spuriously high activation in an irrelevant region can dominate. Attention without learnable query (83.42%) allows adaptive weighting but lacks a content-independent quality prior. Attention with learnable query (84.18%) achieves the best performance by learning to attend to quality-relevant regions (the query vector learns "what should I look for to assess motion quality?") and by incorporating the query as a residual prior (the model can fall back on general quality heuristics when video-specific features are ambiguous).

Why BCE over BT loss: As shown in Table 7, BCE (80.05%) and BT (79.85%) achieve near-identical average accuracy. BCE is preferred for simplicity (no pair construction) and because it provides per-sample gradients rather than per-pair gradients, which may be more sample-efficient given the relatively small training set (23,500 samples). The paper notes BT performs slightly better at high noise levels ($t > 0.6$), possibly because pairwise comparisons are more robust when absolute quality assessment is difficult, but the overall tie makes BCE the pragmatic choice.

Why alternate SFT and PRFL updates rather than joint optimization: The paper follows the standard ReFL template of interleaving reward maximization with supervised fine-tuning. Joint optimization (summing the losses into a single objective) would require careful tuning of the $\lambda$ parameter to balance two objectives operating on different data distributions with different variance characteristics. Alternating updates decouple the two objectives, allowing the optimizer to take independent steps for each. This is particularly important because the SFT loss is bounded below by zero and has relatively stable gradients (it's a simple regression on real data), while the PRFL loss $-\lambda \cdot r_s$ is unbounded (the reward can theoretically increase indefinitely if the model exploits PAVRM) and has gradient variance that depends on PAVRM's evaluation surface.

Why the VAE encoder is frozen during PAVRM training: The VAE encoder is a generic compression module that maps RGB pixels to a lower-dimensional latent space. It is not specific to any particular generation model or quality assessment task. Freezing it during PAVRM training serves two purposes: (1) it reduces the number of trainable parameters, making training more efficient and reducing overfitting risk, and (2) it ensures that latent representations remain consistent with what the VGM expects during inference, preventing distribution shift between PAVRM's training and deployment.

Why only generated videos (not real videos) for PAVRM training: PAVRM is trained exclusively on videos generated by Wan2.1-14B, not on real videos. The rationale is that PAVRM needs to learn the specific artifact patterns, failure modes, and quality characteristics of the generator it will be evaluating. A reward model trained on real videos might penalize generation-specific textures or styles that are actually acceptable, or might fail to recognize generation-specific artifacts (e.g., characteristic DiT attention artifacts) that real videos never exhibit. This is an on-policy data collection strategy: the reward model's training distribution matches the distribution it will evaluate during PRFL. The cross-model generalization experiment in Table 9 (where Wan2.1-trained PAVRM is tested on HunyuanVideo and Veo3 outputs) shows this on-policy choice has consequences for transfer: the model generalizes well at high noise levels (where structural errors dominate) but poorly at low noise levels (where model-specific fingerprints dominate).

Why 40 inference steps despite 1000 training steps: The paper uses UniPCMultistepScheduler with 1000 training steps for gradient accuracy but evaluates with 40 inference steps for computational efficiency. This is standard practice in diffusion model research: more steps during training provide finer-grained velocity field approximations and more accurate gradient estimates, while fewer steps during inference reduce generation time. The velocity prediction $v_\theta$ learned with 1000-step supervision can be applied with coarser step sizes at inference time because the model learns a continuous velocity field, not step-specific predictions.

4. Key Insights and Innovations

Innovation 1: The Generator Architecture Is Already a Process-Level Reward Model — It Just Needs Query Attention and Timestep-Aware Fine-Tuning

The paper's most intellectually distinctive move is not building a new reward model, but recognizing that the pre-trained video generation model already contains the representations needed for quality assessment, and that the missing pieces are (1) a mechanism to extract a single quality judgment from the spatiotemporal feature grid and (2) training that teaches the model to interpret those representations as quality signals across all noise levels. This is fundamentally different from the dominant paradigm in visual reward modeling, where reward models are separate vision-language models trained on human preference data (VideoScore, VideoPhy, VideoAlign, PickScore, ImageReward). The standard assumption has been that quality assessment requires architectures pre-trained on natural images with language grounding; this paper demonstrates that the generation architecture itself—which was trained to predict transport velocities, not to assess quality—encodes quality-relevant information at least as effectively as purpose-built VLMs.

What makes this a genuine conceptual contribution rather than an engineering trick is the diagnostic evidence the paper provides in Figure 3. Figure 3b shows that a simple linear probe on frozen VGM features achieves 78.8% accuracy on motion quality assessment—matching the VideoAlign VLM baseline without any fine-tuning of the feature extractor. This is not obvious a priori: why would a model trained to denoise video latents develop internal representations that encode whether a video has physically plausible motion? The paper's implicit answer is that the denoising objective forces the model to learn the manifold of realistic video structure, and deviations from that manifold—artifacts, impossible motion, anatomical errors—are naturally detectable as anomalous patterns in the model's internal representations. The VGM has learned what "good video" looks like because that's what it was trained to produce; quality assessment is a natural byproduct.

The comparison to prior work sharpens this contribution. LPO [46] pioneered using diffusion models as noise-aware latent reward models, but only for images—and the extension to video is non-trivial because video introduces temporal dimensions the reward model must aggregate over. VideoAlign [24] briefly mentioned using VGMs for inference-time guidance, but did not develop this into a train-time reward model. DOLLAR [7] uses a VLM as a latent reward model, but the VLM operates only on clean or near-clean latents—it inherits the outcome-based limitation even though it avoids VAE decoding. PAVRM's key conceptual advance over all these is the marriage of architectural reuse (the generator is the evaluator) with process-level capability (the evaluator works at arbitrary timesteps). This is a fundamental shift: rather than building a separate quality-assessment pipeline, the paper shows the generator can be taught to evaluate its own outputs by adding a lightweight aggregation head and fine-tuning with timestep-aware binary labels.

The innovation's significance extends beyond performance gains. It establishes a new paradigm for video generation alignment where the generator and evaluator share the same backbone, which has practical implications (no separate VLM to train, deploy, and keep synchronized) and theoretical implications (the reward model's understanding of video quality is grounded in the same representational space as generation, potentially making the reward signal more relevant and less prone to distribution shift). The paper pushes this idea further with the query attention mechanism, which is not just an aggregation trick but a deliberate architectural choice to learn "what should I attend to for quality assessment" independently of "what content-specific patterns are in this video." The residual query connection ($z = z_{\text{obs}} + q$) encodes a content-agnostic quality prior that prevents the model from learning spurious content-quality correlations—a subtle but important design that shows the authors were thinking about what makes a reward model robust, not just accurate.

Evidence anchor: Figure 3b (frozen features match VLM baseline at 78.8%), Figure 3c (timestep-aware fine-tuning lifts accuracy to 85.46%, a ~6.6-point gain over the VLM baseline and any frozen-feature configuration), Table 5 (query attention outperforms mean pooling, max pooling, and attention without query, with consistent accuracy across all timestep ranges).

---

Innovation 2: Process-Level Supervision Distributes Learning Across the Denoising Trajectory — And Different Stages Optimize Different Quality Dimensions

The paper's second conceptual contribution is the empirical demonstration that different video quality dimensions are determined at different denoising stages, and that restricting reward supervision to late timesteps (as all prior ReFL methods do, whether pixel-space or latent-space) fundamentally limits what can be optimized. This is not a claim about memory efficiency or training speed—it is a claim about the structure of the optimization problem itself. The paper argues, and provides evidence for, a specific causal model: early denoising stages establish motion trajectories and global spatiotemporal structure, middle stages refine the transition from coarse structure to fine detail, and late stages polish visual appearance and anatomical correctness. Supervising only the final output can improve visual quality but cannot fix motion that was misplanned in the early stages, because the gradients from the final reward don't flow back through the full trajectory.

Prior work implicitly assumed—or at least acted as if—reward supervision at the final output is sufficient. Image ReFL methods (DRaFT [6], AlignProp [32], ImageReward [42]) optimize at $t \approx 0$ and achieve meaningful improvements, but images don't have temporal structure, so early-vs-late stage distinctions are less critical (though even for images, Clark et al. [6] note that final-step-only training "fails to optimize low-level objectives like symmetry"). Video methods that adopted the same approach (RGB ReFL, ContentV [22]) inherited this assumption without examining whether it holds for spatiotemporal quality dimensions. The paper's sensitivity analysis in Table 3 directly falsifies the assumption for video: training PRFL with only late-stage timestep sampling improves human anatomy (84.24 → 91.54) but produces only modest gains in dynamic degree (22.00 → 44.00). Training with only early/middle-stage sampling produces the opposite pattern—strong dynamic degree improvements but weaker human anatomy gains. Full-stage sampling achieves the best of both (DD: 68.00, HA: 94.73, best average score 89.58).

The significance of this finding is that it reframes ReFL optimization from a monolithic problem ("make the final output better") to a structured, stage-specific optimization problem ("make early planning better for motion, middle refinement better for coherence, and late polishing better for visual quality"). This has implications beyond the specific PRFL method: any future video alignment method that supervises only final outputs is leaving performance on the table, regardless of how sophisticated its reward model is. The finding also explains why the naive extension of image ReFL to video produces mediocre results—it's not that the reward model is bad, but that the optimization strategy ignores the temporal structure of the generation process.

This is a fundamental insight, not an incremental refinement. It restructures how someone building a video generation alignment system thinks about the problem: the unit of optimization is not the video, but the denoising stage, and the reward signal must be distributed across stages according to which quality dimensions each stage governs. The paper's random timestep sampling strategy for PRFL is the simplest possible implementation of this principle, but the principle itself is the contribution.

Evidence anchor: Table 3 (early, middle, late, and full-stage comparisons show differential effects on dynamic degree and human anatomy), Figure 3 (VLM-based reward models show erratic behavior across timesteps, demonstrating why pixel-space models can't provide process-level supervision even if decoded from intermediate latents).

---

Innovation 3: Verifier Over-Optimization Shows Up Even in Latent Space — But Single-Step Backpropagation With Full-Stage Sampling Contains It

While the paper does not frame this as a primary contribution, its training methodology embodies an important negative finding: reward over-optimization is a risk even when the reward model shares the generator's architecture, and the mitigation strategy—alternating SFT updates with single-step reward backpropagation at random timesteps—is a practical solution that the paper implicitly validates. This is significant because one might expect that a reward model built from the same backbone as the generator would be inherently more robust to exploitation (the reward model "understands" the generator's output space because it processes latents the same way). The paper's careful regularization (interleaved SFT, single-epoch training, the $\lambda$ trade-off parameter) and the ablation comparing full-stage to restricted-stage sampling suggest this is not the case—without balanced supervision across all timesteps, the generator can over-optimize for whatever aspect the reward model is focused on at the expense of others.

The comparison to prior work is instructive. Image ReFL methods (Xu et al. [42] and subsequent work) already identified the need for SFT regularization to prevent reward hacking, so the existence of the problem is not novel. What is novel is the stage-specific nature of the over-optimization risk. Table 3 shows that late-stage-only PRFL achieves the highest human anatomy scores (91.54 vs. 84.24 pretrained) but the lowest dynamic degree among restricted-stage variants (44.00), suggesting the model sacrifices motion quality to chase anatomical perfection that the late-stage reward signal emphasizes. Early-stage-only PRFL shows the opposite pattern. The full-stage strategy prevents either form of imbalance, but the paper's ablation demonstrates that imbalance is possible and that the training procedure must actively guard against it by distributing the reward signal uniformly across timesteps.

This is a diagnostic contribution rather than a theoretical one, but it has practical significance for anyone building on this work. The implication is that process-level reward models don't just enable richer supervision—they also require more careful regularization to prevent the generator from exploiting stage-specific weaknesses in the reward signal. The single-step backpropagation design (only one gradient-enabled denoising step per PRFL update) may also serve as an implicit regularizer: by only backpropagating through one step at a time, the model cannot coordinate multi-step exploitation strategies that would require joint optimization across several denoising stages. The paper doesn't ablate this directly (e.g., comparing single-step to multi-step backpropagation), but the success of the approach provides indirect evidence that the constraint is beneficial.

Evidence anchor: Table 3 (stage-restricted variants show imbalanced quality improvements, full-stage achieves the best balance), the SFT-PRFL alternating training design (Algorithm 1), the use of single-epoch training with a relatively small $\lambda$.

---

Innovation 4: Training Efficiency Is Not Just an Engineering Win — It Enables Optimization That Was Previously Impossible

PRFL's 1.4× training speedup and full-frame processing capability (Table 4) are easy to dismiss as "just" engineering improvements, but the paper demonstrates that these efficiency gains are conceptually enabling: without them, the optimization described in Innovation 2 (distributing supervision across denoising stages) cannot be executed. RGB ReFL with full frames hits out-of-memory errors. RGB ReFL with only the first frame (ContentV-style) stays within memory limits but cannot provide video-level quality signals—it literally cannot see motion, temporal consistency, or multi-frame anatomical coherence. The first-frame workaround isn't just a degraded version of full-video optimization; it's a qualitatively different optimization problem that optimizes for per-frame visual quality rather than video quality.

The paper's Table 4 presents a striking asymmetry: PRFL processes 81× more visual information than the first-frame RGB ReFL baseline (81 frames vs. 1 frame) while running 1.42× faster. This is not a marginal improvement—it's a categorical change in what the training procedure can observe. The field's prior assumption was that full-video ReFL was computationally infeasible (hence the first-frame workaround, the gradient-stopping tricks, and the trajectory shortcuts in prior work). PRFL changes this assumption: full-video process-level optimization is not just feasible, it's faster than the crippled single-frame version of the previous approach.

What makes this a genuine insight rather than an implementation detail is that efficiency unlocks a different optimization landscape. The paper's central claim—that different denoising stages govern different quality dimensions—could not have been discovered or exploited using methods that only process the first frame or only evaluate final outputs. The efficiency of PRFL is what makes the innovation in process-level supervision actionable. This is a recurring pattern in ML research where methodological breakthroughs (here, operating in latent space with the generator-as-evaluator) produce efficiency breakthroughs that in turn enable scientific discoveries (the stage-specific quality optimization findings). The paper doesn't make this meta-argument explicitly, but it's embedded in the structure of the contributions.

Evidence anchor: Table 4 (66.81 GB VRAM and 51.11s/step for full-frame PRFL vs. OOM for full-frame RGB ReFL; 1.42× speedup over first-frame-only RGB ReFL despite processing full 81-frame sequences; no VAE decoder in the PRFL computation graph as shown in Algorithm 1).

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper constructs an internal dataset of approximately 31,000 human portrait videos sourced from online collections, with text prompts generated by a video captioning model. The first frame and text prompt from each real video are fed into Wan2.1-14B-I2V to produce generated videos. Professional annotators rate each generated video on two dimensions—Physical Plausibility and Subject Deformity—using three-level scales (qualified, partially qualified, unqualified). Videos rated qualified on both dimensions are labeled as good; those rated unqualified on both are labeled as bad; mixed or partially-qualified videos are filtered out to enhance label distinctiveness. This yields 24,000 video pairs (real + generated). Generated videos serve as the reward model training set (approximately 23,500 training, 100 validation, 400 test, with test labels determined by majority voting across at least three annotators). Real videos serve as SFT data for video generation training. For video generation evaluation, 100 input conditions are randomly sampled from the reward model test set to form the Inner Test Set. The paper also incorporates the open-source VBench and VBench2 benchmarks: for T2V, the subject consistency subset from VBench (72 prompts) and the human anatomy subset from VBench2 (120 prompts, enhanced for Wan); for I2V, the I2V Subject subset from VBench-I2V in VBench++ (246 prompts).

• Base model(s). All experiments use Wan2.1-14B as the primary baseline, with both the I2V and T2V variants evaluated. The paper states this model is used because it represents a state-of-the-art open video generation model with publicly available weights. For the reward model analysis in Appendix A.3.3, HunyuanVideo and Veo3 are additionally used as out-of-domain test sources. Wan2.1-14B operates in the rectified flow framework with a DiT-based architecture at 40 DiT blocks. The feature extractor in PAVRM uses the first 8 DiT blocks of this model (frozen during initial reward model training, fine-tuned during PAVRM training). The paper trains at two resolutions—480P and 720P—with separate models for each.

• Metrics. The paper employs multiple evaluation dimensions adapted from the VBench framework. Motion Smoothness (MS) uses frame interpolation priors: odd-indexed frames are removed from the video sequence, reconstructed via interpolation, and the normalized MAE between reconstructed and original frames yields a score in [0, 1], with higher values indicating smoother motion. Dynamic Degree (DD) uses RAFT to estimate inter-frame optical flow; the mean of the top 5% flow magnitudes serves as a static/dynamic threshold, and the score represents the proportion of non-static videos generated (higher scores indicate more dynamic motion). Subject Consistency (SC) uses DINO-based features to measure subject identity preservation across frames via cosine similarity, jointly computing similarity with the first frame and between consecutive frames. Human Anatomy (HA) from VBench2 uses three ViT-based anomaly detectors (trained on human torso, hands, and faces) to compute the percentage of frames without detected anatomical anomalies. I2V Subject (IC) from VBench++ uses DINOv1 features to measure correspondence between the input image and generated video frames. PAVRM Score estimates the qualified sample ratio across the test set by randomly sampling a timestep $t \sim U(0, 1)$ for each test sample, feeding the partially denoised latent to PAVRM, and computing the binary prediction (0 = unqualified, 1 = qualified) across all samples. For PAVRM evaluation specifically, timesteps are sampled from five stratified intervals ([0, 0.2], (0.2, 0.4], (0.4, 0.6], (0.6, 0.8], (0.8, 1.0]), with one random sample per interval per test sample, and the final metric is average accuracy across all intervals. The paper also reports an "Avg" metric in Tables 1 and 3 that aggregates the available metrics for each setting. For the user study, participants are asked to select the video with higher overall quality considering text-video consistency, naturalness of motion, and absence of human deformities or physically implausible elements across 2,250 pairwise comparisons from 30 professional participants.

• Baselines. Four post-training methods are compared. Pretrain: the base Wan2.1 model without any post-training optimization, serving as the lower-bound reference. SFT (Supervised Fine-Tuning): standard flow matching training on the curated real video dataset using the loss in Equation 2; described as an offline, off-policy algorithm that is computationally efficient and widely adopted. RWR (Reward Weighted Regression): an offline, off-policy RL method where pre-generated videos are scored by VideoAlign-MQ, and training samples are weighted by $\exp(r_\phi(\text{video}, y))$ in the flow matching loss (Equation A.2.1); following the VideoAlign framework [24]. RGB ReFL: follows the ContentV [22] implementation, performing VAE decoding on only the first frame and using the image reward model PickScore [18] as the reward signal; optimized with Equation A.2.2. For the reward model evaluation, the baselines are VideoAlign-MQ [24] and VideoPhy-PC [1], two state-of-the-art VLM-based video reward models used in a zero-shot manner; accuracy is computed by establishing an optimal threshold on reward scores against ground-truth labels.

• Generation budget / compute accounting. For post-training, the paper uses approximately the same number of training samples across all methods with one epoch over text-video pairs, sequence parallel size of 4, learning rate of 5 × 10⁻⁶, and a global batch size of 30 (batch size 6 with gradient accumulation of 5). All methods are trained for equal epochs on comparable data volumes. For computational efficiency comparison (Table 4), the paper measures peak VRAM consumption and average time per training step, both with and without the SFT loss component, on the same hardware. Inference for evaluation follows the Wan2.1 standardized protocol: UniPCMultistepScheduler with 40 inference steps, classifier-free guidance weight of 5.5, and resolution-matched generation (480P or 720P). The early, middle, and late denoising stages for the sensitivity analysis correspond to steps 1-13, 14-26, and 27-40 respectively.

• Cross-validation / statistical protocol. The PAVRM test set (400 samples) uses majority voting across at least three independent annotators for label reliability. For video generation, the Inner Test Set is a fixed 100-sample subset drawn from the PAVRM test set. VBench and VBench2 evaluations use their standard fixed prompt sets. No repeated trials with error bars are reported for generation metrics. The user study involves 2,250 pairwise comparisons from 30 professional evaluators, with each comparison asking which of two videos has higher overall quality.

Main Quantitative Results

Reward Model Performance (PAVRM)

Table 5 presents the core reward model results across architectures, timestep ranges, tasks (T2V vs. I2V), and resolutions (480P vs. 720P). At 720P I2V resolution, PAVRM with query-based attention achieves 84.18% average accuracy (averaged across five stratified timestep intervals), compared to 78.83% for the VideoAlign VLM baseline and 77.04% for VideoPhy. The accuracy is remarkably stable across timestep ranges: 83.42% at [0, 0.2], 84.95% at (0.2, 0.4], 84.69% at (0.4, 0.6], 84.44% at (0.6, 0.8], and 83.42% at (0.8, 1.0]—a spread of only ~1.5 percentage points. This contrasts with the implicit behavior of VLM-based reward models, which Figure 3a shows as producing wildly fluctuating scores across timesteps when evaluated on intermediate denoised latents decoded to RGB.

The comparison across aggregation methods reveals a clear hierarchy. Mean pooling achieves 83.37% average but with more timestep-dependent variation (82.40% to 85.46%). Max pooling degrades substantially at later timesteps: 80.61% at [0, 0.2] dropping to 73.72% at (0.8, 1.0], averaging only 78.47%. Attention without a learnable query achieves 83.42%. Attention with learnable query achieves the best performance at 84.18%. The paper notes that max pooling's degradation at late timesteps is because "a single spuriously high activation can dominate" when noise levels are high.

Cross-task and cross-resolution generalization is demonstrated: attention with query at 720P T2V achieves 84.13% average, at 480P I2V achieves 83.42%, and at 480P T2V achieves 83.42%. All configurations maintain over 83% average accuracy.

Figure 3c shows the effect of timestep-aware fine-tuning. When training only a linear probe at fixed timesteps (t = 0.2, 0.4, 0.6, 0.8), accuracy hovers around the VideoAlign baseline of 78.83%. Random timestep training with MLP-only (no DiT fine-tuning) shows modest improvement but still fails to surpass the VLM baseline. Full fine-tuning with random timestep sampling yields a dramatic improvement to 85.46%, with peak performance at earlier timesteps (t = 0.8), confirming that fine-tuning the DiT blocks themselves is necessary to unlock the VGM's full potential as a timestep-robust reward model.

Text-to-Video Generation (Table 1)

At 480P resolution, PRFL achieves the best performance across almost all metrics compared to Pretrain, SFT, RWR, and RGB ReFL. On the Inner Test Set, PRFL attains DD of 68.00—a +46.00 absolute improvement over Pretrain's 22.00—while maintaining high MS (99.05 vs. 99.20, a negligible −0.15) and SC (96.34 vs. 97.34, a −1.00 decrease). HA improves from 84.24 (Pretrain) to 94.73 (+10.49). PAVRM score increases from 89.00 to 92.00 (+3.00). On VBench, PRFL achieves DD of 76.39 (vs. 68.06 for Pretrain, +8.33) and SC of 94.16 (vs. 92.74 for Pretrain, +1.42). MS shows 98.18 (vs. 98.00, +0.18). On VBench2, HA reaches 89.84 (vs. 74.38 for Pretrain, +15.46) and PAVRM score is 76.67 (vs. 69.17 for Pretrain, +7.50). The average across all metrics shows PRFL at 89.58 versus Pretrain at 81.03, an +8.55 absolute improvement.

The comparison to other post-training methods reveals that SFT and RWR improve DD substantially (44.00 and 60.00 respectively on Inner Test Set) but at the cost of degraded SC (96.61 and 95.93 versus Pretrain's 97.34). RGB ReFL (first-frame only) shows particularly poor SC (92.26, a −5.08 drop from Pretrain) and lower DD improvements (38.00) compared to PRFL. HA improvements from SFT (92.79) and RWR (91.85) are substantial but trail PRFL's 94.73. RWR achieves the highest PAVRM score on VBench2 (62.50) but this does not translate to better VBench2 HA (79.67 vs. PRFL's 89.84) or overall average (84.17 vs. PRFL's 89.58).

At 720P resolution, the pattern intensifies. PRFL achieves DD of 81.00 on Inner Test Set (+56.00 over Pretrain's 25.00), MS of 98.85 (vs. 99.09, −0.24), SC of 96.09 (vs. 96.69, −0.60), and HA of 90.89 (vs. 78.73, +12.16). On VBench, DD improves from 61.11 to 84.72 (+23.61) while MS increases from 97.70 to 98.06 (+0.36). On VBench2, HA reaches 90.40 (vs. 68.88 for Pretrain, +21.52) with a PAVRM score of 66.13 (vs. 62.10, +4.03). The average improves from 79.32 to 90.60 (+11.28). Only PRFL results at 720P are reported (no SFT, RWR, or RGB ReFL comparisons at this resolution).

Image-to-Video Generation (Table 2)

PRFL generalizes to I2V tasks. At 480P on the Inner Test Set, DD improves from 57.00 (Pretrain) to 87.00 (+30.00), MS from 98.66 to 98.88 (+0.22), SC from 91.73 to 93.18 (+1.45), IC from 96.86 to 97.31 (+0.45), and PAVRM from 87.00 to 93.00 (+6.00). On VBench-I2V, the improvements are even more dramatic: DD jumps from 40.65 to 81.30 (+40.65), MS from 97.86 to 98.04 (+0.18), SC from 93.86 to 94.57 (+0.71), IC from 97.21 to 97.79 (+0.58), and PAVRM from 92.28 to 92.68 (+0.40). The average across settings improves from 85.31 to 93.38 (+8.07).

At 720P I2V, PRFL improves DD on the Inner Test Set from 60.00 to 76.00 (+16.00), IC from 96.65 to 98.26 (+1.61), and PAVRM from 74.00 to 90.00 (+16.00). On VBench-I2V, DD jumps from 35.37 to 68.42 (+33.05), and PAVRM from 89.43 to 95.53 (+6.10). The average improves from 83.53 to 91.31 (+7.78). As with T2V 720P, only Pretrain and PRFL results are reported for this resolution.

Sensitivity to Sampling Timesteps (Table 3)

The paper ablates the effect of restricting PRFL timestep sampling to different denoising stages on T2V 480P. Full-stage PRFL achieves the best average of 89.58 across all metrics. Early-stage-only PRFL (first third of denoising) produces DD of 51.00 on Inner Test Set (+29.00 over Pretrain), MS of 99.00, SC of 96.63, and HA of 87.52. Middle-stage-only PRFL produces DD of 51.00 on Inner Test Set, MS of 99.11, SC of 96.69, HA of 89.38, and strong VBench DD of 76.39—matching the full-stage VBench DD. Late-stage-only PRFL achieves the highest HA (91.54 on Inner Test Set) but the lowest DD among restricted variants (44.00 on Inner Test Set). The key finding is that no single-stage strategy matches full-stage performance across all metrics: early and middle stages primarily govern DD improvements, middle stage drives VBench DD, and late stage drives HA improvements. Full-stage sampling achieves DD of 68.00 (Inner) and HA of 94.73, outperforming every restricted-stage variant on both dimensions simultaneously.

Computational Efficiency (Table 4)

PRFL with full 81-frame processing uses 66.81 GB of peak VRAM and takes 51.11 seconds per training step (43.69 seconds excluding SFT loss). RGB ReFL attempting to process full 81-frame sequences encounters out-of-memory errors and cannot run. RGB ReFL that decodes only the first frame (the ContentV baseline) uses 55.47 GB VRAM and takes 72.38 seconds per step (64.89 seconds without SFT). Despite processing 81× more visual information than the first-frame RGB ReFL baseline, PRFL achieves a 1.42× training speedup (1.49× when excluding SFT loss overhead). The paper does not report PRFL memory consumption when processing fewer frames for a more direct comparison, nor does it report the time per step for RGB ReFL if it were restricted to the same number of frames as PRFL processes in latent space.

User Study (Figure 5)

In human evaluation with 2,250 pairwise comparisons from 30 professional participants, PRFL wins 67.47% of comparisons against SFT, 63.20% against RWR, and 59.33% against RGB ReFL. The remaining comparisons are ties or competitor wins: against SFT, 18.80% ties and 13.73% SFT wins; against RWR, 16.27% ties and 20.53% RWR wins; against RGB ReFL, 18.53% ties and 22.13% RGB ReFL wins. PRFL consistently wins a majority of comparisons across all baselines, with the largest margin against SFT and the narrowest against RGB ReFL.

Qualitative Results (Figure 6)

Figure 6 shows representative frames from videos generated by all methods on two prompts. In the first prompt (a woman dancing in a flowing white dress with camera movement), Pretrain generates a distorted environment and figure; SFT produces facial distortions in the close-up frame; RWR shows body deformations; RGB ReFL generates a failed first frame. PRFL generates consistent, artifact-free frames with smooth motion transitions. In the second prompt (four people with musical instruments in a room with exposed brick walls), baseline methods exhibit anatomical errors (distorted hands and faces, highlighted in red boxes), while PRFL maintains visual quality and generates anatomically plausible figures across all frames. Additional cases are provided in the supplementary materials.

Ablation Studies and Robustness Checks

• PAVRM training loss (Table 7): Comparing binary cross-entropy (BCE) with pairwise Bradley-Terry (BT) loss shows nearly identical average accuracy (80.05% for BCE vs. 79.85% for BT). A timestep-dependent trade-off emerges: BT performs better in high-noise regions (87.75% at (0.8, 1.0] vs. 83.00% for BCE), while BCE excels at intermediate timesteps (79.75% at (0.4, 0.6] vs. 78.25% for BT). The paper adopts BCE as default for its simplicity and elimination of pair construction overhead. This 0.2% average gap is negligible, suggesting the reward model training is robust to the choice of binary classification loss.

• Number of trainable DiT blocks in PAVRM (Table 8): Performance follows a non-monotonic trend with increasing DiT block count. With 8 blocks: 84.18% average accuracy. With 16 blocks: 85.51% (+1.33), the peak performance. With 24 blocks: 85.25% (−0.26 from peak). With 32 blocks: 84.03% (−1.48 from peak). With 40 blocks (full model): 83.27% (−2.24 from peak, and −0.91 below the 8-block baseline). The paper hypothesizes that deeper layers overfit to model-specific generation details rather than learning generalizable quality assessment. The default configuration uses 8 blocks for efficiency, trading a ~1.3% accuracy gap from the 16-block optimum for lower computational cost.

• Cross-model generalization of PAVRM (Table 9): PAVRM trained only on Wan2.1-generated videos is tested on videos from HunyuanVideo and Veo3. When trained and tested on HunyuanVideo/Veo3 (in-domain for those models), accuracy is 85.60% average with strong performance across timesteps (77.00% at [0, 0.2], 86.00% at (0.2, 0.4], 89.00% at both (0.4, 0.6] and (0.6, 0.8], 87.00% at (0.8, 1.0]). When Wan2.1-trained PAVRM is evaluated on HunyuanVideo/Veo3 (cross-domain), average accuracy drops to 74.40%, with an inverted generalization pattern: performance is stronger at high noise levels (81.00% at (0.8, 1.0]) and weaker at low noise levels (70.00% at [0, 0.2]). The paper attributes this to model-specific "fingerprints"—unique texture patterns and artifact types learned during Wan2.1 training—that dominate low-noise latents and fail to transfer, while high-noise latents preserve more universal structural cues that the reward model can recognize across generators.

• VGM feature quality across DiT layers (Figure 3b): Probing features from DiT layers L8 through L40 with a simple MLP at fixed timestep t = 0.2 yields uniform accuracy of approximately 78.8%, matching the VideoAlign VLM baseline (78.83%). This demonstrates that motion-relevant information is distributed throughout the network rather than concentrated in specific layers, justifying the use of early layers for computational efficiency.

• Timestep-aware fine-tuning vs. frozen features (Figure 3c): Training only a linear probe at fixed timesteps yields accuracy around the VLM baseline (~78.8%). Training a linear probe with random timestep sampling (no DiT fine-tuning) shows minimal improvement. Full fine-tuning (DiT blocks + linear probe) with random timestep sampling achieves 85.46% accuracy, with performance peaking at early timesteps (t = 0.8). This confirms that the DiT blocks must be adapted—not just probed—to unlock timestep-robust quality assessment.

Critical Assessment

On the claim that PRFL substantially improves motion quality: The quantitative evidence in Table 1 is compelling for dynamic degree and human anatomy. PRFL improves DD by +46.00 at 480P T2V and +56.00 at 720P T2V on the Inner Test Set, with VBench DD improvements of +8.33 (480P) and +23.61 (720P). HA improvements range from +10.49 (480P T2V) to +21.52 (720P T2V). These are large, consistent effects that appear across tasks (T2V and I2V) and resolutions (480P and 720P). The user study (Figure 5) provides complementary evidence that humans perceive PRFL outputs as higher quality, with win rates of 59-67% against baselines.

However, the metrics reveal a tension that the paper acknowledges but does not fully explore. Subject consistency degrades slightly but consistently under PRFL: −1.00 at 480P T2V Inner Test Set, −0.60 at 720P T2V on the Inner Test Set. Motion smoothness also shows minor decreases in some configurations (−0.15 at 480P T2V, −0.24 at 720P T2V). While these drops are small relative to baseline values (SC of 97.34 dropping to 96.34, MS of 99.20 dropping to 99.05), they appear across multiple settings and suggest a potential quality trade-off: improving motion dynamics may come at a small cost to temporal coherence. The paper frames these as "marginal given their already high baseline values," which is reasonable but deserves more systematic investigation—ideally, an ablation on the $\lambda$ parameter in the PRFL loss to map the DD-SC trade-off curve explicitly.

On the claim that PRFL achieves 1.4× training speedup over RGB ReFL: Table 4 supports this claim quantitatively but with important caveats about what is being compared. PRFL (full 81 frames, 66.81 GB VRAM, 51.11s/step) is 1.42× faster than RGB ReFL (first frame only, 55.47 GB VRAM, 72.38s/step). This is a legitimate speedup, but it compares a method operating on 81 frames (PRFL) against a method operating on 1 frame (RGB ReFL). If RGB ReFL were restricted to process only the same number of latent frames that PRFL processes, the time comparison would be different—but the paper's point is exactly that RGB ReFL cannot process full frames due to OOM, so the practical comparison is the one reported. The missing comparison is: what would PRFL's speed be if processing only 1 frame? This would separate the algorithmic efficiency gain (latent vs. pixel space) from the information advantage (more frames processed). The 1.49× speedup when excluding SFT loss suggests the PRFL step itself is substantially faster, but the paper doesn't provide per-component timing breakdowns.

On the claim that process-level supervision distributes learning across the full denoising trajectory: Table 3 is the central evidence and it strongly supports this claim. The differential effects of early, middle, and late-stage sampling on DD versus HA demonstrate that different quality aspects are optimized at different stages. Full-stage sampling achieving the best overall average (89.58 vs. 87.02 for middle-stage, the next best) confirms that distributing supervision is beneficial. However, the experiment has an important confound: the restricted-stage variants see fewer total timesteps per training epoch (only roughly one-third of the timestep range), which means they receive fewer total PRFL updates if the number of iterations is held constant. If the paper instead held the number of PRFL updates constant across variants (by sampling more densely from the restricted range), the comparison would isolate the effect of which timesteps are sampled from the effect of how many timesteps are sampled. As reported, the full-stage advantage could partially reflect receiving a more diverse set of training examples rather than the specific benefit of full-range coverage. This does not undermine the core finding—differential effects across stages are clear—but means the magnitude of the full-stage advantage may be overstated relative to the per-stage optimal performance.

On the claim that PAVRM generalizes across tasks and resolutions: Table 5 shows consistent accuracy (83-84%) across I2V and T2V, 480P and 720P. This robustness is a strength. The cross-model generalization experiment (Table 9) provides the more stringent test: PAVRM trained on Wan2.1 drops from 84.18% (in-domain at 720P I2V, from Table 5) to 74.40% when tested on HunyuanVideo and Veo3 outputs. This is a substantial degradation (~10 points) that the paper attributes to model-specific fingerprints. However, the paper does not report what accuracy a VLM reward model (VideoAlign or VideoPhy) achieves on this same HunyuanVideo/Veo3 test set, making it impossible to assess whether PAVRM's cross-model degradation is better or worse than the baseline's zero-shot generalization. If VideoAlign also drops ~10 points on out-of-domain videos, PAVRM's cross-model performance would be competitive; if VideoAlign maintains its accuracy, then PAVRM's model-specificity is a genuine weakness relative to VLM-based reward models. This missing baseline limits the interpretation of the cross-model results.

On the I2V results and motion quality consistency: Table 2 shows impressive DD improvements for I2V (+30.00 at 480P Inner Test Set, +40.65 on VBench-I2V). However, the PAVRM score on VBench-I2V increases only modestly (+0.40 at 480P, +6.10 at 720P from a baseline of 89.43), which is substantially smaller than the T2V PAVRM improvements. This may indicate that I2V generation already produces temporally coherent outputs (since the first frame provides a strong anchor), and the main improvement is in making those outputs more dynamic rather than fixing structural problems. Alternatively, it could indicate that PAVRM—trained on I2V-generated videos—is better calibrated for I2V quality assessment and thus shows less room for improvement. The paper does not discuss this asymmetry.

On the limited baseline comparisons: The paper compares PRFL against SFT, RWR, and RGB ReFL. This covers the main post-training paradigms but has notable omissions. First, there is no comparison against simple majority voting or best-of-N sampling at inference time—a standard baseline that would establish whether PRFL's training-time improvements exceed what can be achieved by sampling multiple videos and selecting the best with a reward model. Second, the RWR and RGB ReFL baselines use different reward models (VideoAlign-MQ for RWR, PickScore for RGB ReFL) than PRFL uses (PAVRM). This confounds the training algorithm comparison with the reward model quality comparison—PRFL might outperform RGB ReFL partly because PAVRM is a better reward model for motion quality assessment than PickScore, not because PRFL's training algorithm is superior. A fairer comparison would use the same or comparable reward models across methods. Third, only Pretrain and PRFL results are reported at 720P resolution for T2V (Table 1) and I2V (Table 2). The absence of SFT, RWR, and RGB ReFL at 720P means the 720P comparisons are limited to showing PRFL improves over Pretrain, without establishing whether it outperforms alternative post-training methods at that resolution.

On the test set size and statistical reliability: The Inner Test Set contains only 100 input conditions. VBench contributes 72 prompts for T2V and 246 for I2V. VBench2 contributes 120 prompts for T2V. Metrics are reported as point estimates without confidence intervals or error bars. Given the small sample sizes (especially the 100-sample Inner Test Set), differences of a few points in metrics like MS (−0.15) or SC (−1.00) may not be statistically significant. The paper relies on consistency across metrics and settings (improvements appear in Inner Test Set, VBench, and VBench2) to establish reliability, which is reasonable but informal. The user study with 2,250 comparisons from 30 annotators provides the most statistically robust evidence, and it consistently favors PRFL, though the margin narrows against RGB ReFL (59.33% wins) compared to SFT (67.47% wins).

On the absence of video-level VLM evaluation: All automatic metrics come from VBench/VBench2 or PAVRM itself. The paper does not evaluate generated videos using an independent VLM-based video reward model (e.g., VideoScore or VideoPhy) as a metric. This means there is no external, architecture-independent assessment of whether PRFL's improvements in VBench metrics translate to improvements as judged by other automated quality assessment systems. Using PAVRM to evaluate PRFL-trained models introduces potential circularity, though the paper mitigates this somewhat by also reporting standard VBench metrics (which are model-agnostic) and conducting a human evaluation.

On the single-model-family limitation: All experiments use Wan2.1-14B as the base model. While the cross-model reward model generalization experiment (Table 9) provides some evidence about transferability, the generation experiments (Tables 1, 2, 3) are entirely within the Wan2.1 family. It remains unknown whether PRFL would produce similar improvements when applied to other video generation architectures (DiT-based models like HunyuanVideo, or 3D U-Net based models). The paper does not claim broader applicability, but the title's implication ("Video Generation Models Are Good Latent Reward Models") suggests generalizability that the experiments don't directly test.

Missing experiment that would have strengthened the paper: A direct PAVRM comparison against DOLLAR [7]'s VLM-based latent reward model. Since DOLLAR also operates in latent space but lacks timestep-aware capability, this comparison would isolate the contribution of process-level supervision from the contribution of operating in latent space. Similarly, an ablation comparing PRFL with PAVRM against PRFL with an outcome-based latent reward model (PAVRM evaluated only at t ≈ 0) would quantify the specific benefit of process-level over outcome-level supervision, independent of the latent-space advantage over RGB ReFL.

6. Limitations and Trade-offs

The Difficulty Estimation Bottleneck: PAVRM Requires a Curated Preference Dataset From the Same Generator

PAVRM is trained on a binary preference dataset $\mathcal{D}_{\text{RM}}$ constructed from videos generated specifically by Wan2.1-14B-I2V, scored by professional human annotators on two quality dimensions (Physical Plausibility and Subject Deformity), and filtered to exclude ambiguous cases. The paper acknowledges this implicitly in Section 4.2 ("PAVRM is trained on a binary preference dataset") and Section 5.1 (describing the 31,000-video collection, generation, and annotation pipeline). The consequence is that PAVRM is coupled to a specific generator's output distribution: it learns to recognize Wan2.1-specific artifacts, failure modes, and quality characteristics. This means deploying PRFL on a different base model (HunyuanVideo, CogVideoX, a future model) would require repeating the entire data pipeline—generating thousands of videos from that model, commissioning human annotations, and retraining PAVRM from scratch. The paper provides direct evidence for this coupling in Table 9: PAVRM trained on Wan2.1 drops from 84.18% average accuracy (in-domain, from Table 5 at 720P I2V) to 74.40% when evaluated on HunyuanVideo and Veo3 outputs. The degradation is most severe at low noise levels (70.00% at $t \in [0, 0.2]$), where the paper attributes the failure to model-specific "fingerprints"—texture patterns and artifact types unique to Wan2.1 that PAVRM learns to associate with quality labels but that do not transfer. This 9.8-point accuracy drop is substantial and means PAVRM cannot serve as a general-purpose video reward model in the way that VLM-based reward models (VideoAlign, VideoPhy) can, since VLMs are trained on diverse natural videos and generalize across generators.

Mitigation status: The paper does not attempt to mitigate this limitation. It does not train PAVRM on multi-model data, explore domain adaptation techniques, or investigate whether fine-tuning PAVRM on a small number of target-model samples could recover accuracy. Section 6 (Conclusion) suggests "multi-aspect evaluation" and "extensions to controllable generation" as future work but does not address the generator-specificity problem directly. A practitioner switching video generation backbones should expect to retrain PAVRM, with all the associated data collection and annotation costs.

---

Hard Problems Remain Unsolved: PRFL Cannot Improve Videos Where the Base Model's Pass Rate Is Near Zero

The paper's annotation protocol (Appendix A.2.3) and dataset construction reveal an implicit capability boundary: videos are filtered into good and bad categories based on human judgments, but the paper never reports what fraction of the base model's outputs fall into each category, how the difficulty of prompts correlates with quality, or whether PRFL improves performance uniformly across prompt difficulty levels. Unlike the companion example paper (which bins MATH problems by base model pass@1 rate and shows test-time compute fails on the hardest quintile), this paper provides no difficulty-stratified analysis of generation results. The consequence is that a practitioner cannot determine whether PRFL's improvements are concentrated on prompts where the base model already generates passable videos (easy cases needing refinement) or whether PRFL helps on prompts where the base model produces catastrophically bad outputs (hard cases needing fundamental restructuring). The qualitative examples in Figure 6 show PRFL fixing specific artifacts (facial distortions, body deformations, failed first frames) on prompts where the base model produces recognizable but flawed content—these appear to be moderate-difficulty cases. No examples are shown where the base model's output is completely incoherent, and there is no quantitative evidence that PRFL can rescue such cases.

This matters because the claim that PRFL "substantially improves motion quality" (Abstract) could mean either (a) PRFL elevates mediocre videos to good ones across the board, or (b) PRFL polishes already-decent videos while leaving fundamentally broken generations unchanged. If (b) is true, the practical value of PRFL depends heavily on the quality of the base model and the difficulty distribution of the deployment prompts. The VBench and VBench2 metrics aggregate over fixed prompt sets, masking any difficulty-dependent variation.

Mitigation status: The paper does not address this. It does not report per-prompt or per-difficulty-bin metrics, does not analyze correlation between base model quality and PRFL improvement magnitude, and does not discuss failure cases where PRFL produces outputs no better than the pretrained model. The annotation categories (qualified/partially qualified/unqualified) could have been used to stratify results by base model quality, but this analysis is absent. This is a significant gap for practitioners evaluating whether PRFL is worth the training cost for their specific use case.

---

The Sequential-to-Parallel Compute Trade-Off Is Not Analyzed for Latency-Sensitive Deployment

PRFL's efficiency gains are reported exclusively in terms of training throughput: 1.42× faster per training step and 66.81 GB VRAM for full 81-frame processing (Table 4). The paper does not measure or discuss inference-time latency. This matters because in production video generation systems, wall-clock time per video is often the binding constraint—a user waiting for a 10-second video clip cares about end-to-end generation time, not the number of FLOPs or the memory efficiency of training. The paper's evaluation protocol uses 40 inference steps for all methods (Section 5.1, Appendix A.2.5), so PRFL does not reduce inference-time compute relative to the pretrained model. However, the paper also does not investigate whether PRFL's improvements in generation quality could be traded off against fewer inference steps—for example, whether a PRFL-trained model at 20 inference steps matches or exceeds the pretrained model at 40 steps. This is a standard efficiency-quality trade-off analysis in diffusion model literature that is absent here.

Furthermore, the paper's training algorithm (Algorithm 1) introduces an asymmetry: gradient-free rollouts cover most of the denoising chain, but the single gradient-enabled step at a random timestep means that during inference, the model has been optimized to perform well when denoising from that particular distribution of partially denoised states. If inference-time compute budgets change (e.g., using 20 steps instead of 40), the distribution of intermediate latents shifts, and the PRFL-trained model's optimization may not transfer. The paper does not test robustness to inference step count.

Mitigation status: Not addressed. Section 6 (Conclusion) does not mention inference latency or step-count robustness as future work. The paper frames efficiency purely in terms of training cost, which is valuable for researchers and model developers but leaves deployment engineers without guidance on whether PRFL-trained models maintain their advantages under reduced inference budgets.

---

The SFT Regularization Dataset Uses Real Videos, Creating a Domain Mismatch in the Training Objective

PRFL alternates between two training objectives: reward maximization on generated videos (via PAVRM) and supervised fine-tuning on real videos (via the standard flow matching loss, Equation 10). The real videos serve as a regularizer to prevent reward over-optimization—they anchor the model to the distribution of high-quality human-captured content. However, the paper's dataset description (Section 5.1, Appendix A.2.3) reveals a subtle mismatch: the reward model (PAVRM) is trained exclusively on generated videos to learn generator-specific quality characteristics, but the SFT regularization uses real videos. This means the model is simultaneously pulled in two directions: the PRFL objective pushes it toward PAVRM-pleasing generated outputs, while the SFT objective pulls it toward the distribution of real videos.

The consequence is that the regularization signal may not be precisely aligned with the reward signal. Real videos have different statistical properties than generated videos—different texture distributions, different artifact profiles, potentially different motion characteristics—so optimizing toward real video latents does not necessarily help the model avoid PAVRM-specific over-optimization. In the worst case, the SFT objective could fight the PRFL objective rather than complementing it, reducing the effective learning signal. The paper's reported improvements (Tables 1, 2) suggest this is not catastrophic, but the tension is unexamined: there is no ablation comparing SFT on real vs. generated videos (e.g., using the high-quality subset of generated videos that passed human annotation), and there is no analysis of how the $\lambda$ parameter interacts with the real-vs-generated domain gap.

The paper's SFT data also introduces a content bias: the dataset consists of "approximately 31,000 portrait videos from online sources" with a focus on human subjects. PRFL's strong improvements in human anatomy (HA: +10.49 to +21.52 across settings) may partially reflect the SFT data's emphasis on human-centric content, rather than a general improvement in video quality. The paper does not report results on non-human prompts (the VBench subject consistency subset includes diverse object categories, but results are aggregated).

Mitigation status: Not addressed. The paper does not discuss the real-vs-generated domain gap in the training objectives, does not ablate SFT data sources, and does not analyze content-specific vs. general quality improvements. A practitioner training on a different content domain (e.g., landscape videos, abstract animations) would need to collect domain-matched SFT data, but the paper provides no guidance on how SFT data composition affects PRFL's effectiveness.

---

The Binary Preference Labels Discard Quality Granularity and May Amplify Annotation Noise

PAVRM is trained with binary cross-entropy loss on hard good/bad labels, with all "partially qualified" videos explicitly filtered out to "enhance distinctiveness and reduce annotation ambiguity" (Appendix A.2.3). This design choice has two consequences. First, the model learns a coarse quality boundary rather than a fine-grained quality scale. During PRFL optimization, the reward signal $r_\phi(x_s, s, p)$ provides only a directional push ("make this more good-like") without distinguishing between marginally bad and catastrophically bad outputs. This limits the granularity of the learning signal: a video that is slightly unqualified but close to the decision boundary receives the same negative label as a completely incoherent mess, even though the former provides a much more informative gradient for improving the generator.

Second, the binary filtering may amplify the effect of annotation errors near the decision boundary. The annotation protocol in Table 6 defines qualified, partially qualified, and unqualified tiers with subjective criteria like "Motion appears smooth and natural, following real-world physics with realistic acceleration, deceleration, and interactions." Videos near the boundary between qualified and partially qualified are excluded from training, but videos near the qualified/unqualified boundary (which are retained on opposite sides of the binary split) may have genuinely ambiguous quality. If annotators disagree on these boundary cases, the binary labels become noisy in precisely the region where the model needs the most precise signal—the boundary between acceptable and unacceptable quality. The paper uses majority voting across three annotators for the test set (Appendix A.2.3), which helps but does not eliminate this issue, and training set labels use single-annotator judgments (the paper describes the annotation process but does not specify whether training labels are single-annotator or majority-voted).

The paper's ablation in Table 7 suggests this is manageable: BCE (80.05%) and BT loss (79.85%) achieve similar accuracy, indicating the binary signal is strong enough that loss function choice does not dominate. But the ~20% error rate (100% − 80.05% = 19.95% misclassification) means PAVRM makes non-trivial mistakes, and the paper does not analyze which types of videos are misclassified or whether misclassifications cluster near the quality boundary. During PRFL, these reward model errors could push the generator in wrong directions, and the SFT regularizer is the only defense.

Mitigation status: The paper explicitly acknowledges the binary labeling choice in Appendix A.2.3, framing it as a way to "enhance distinctiveness" and "ensure clear decision boundaries." The use of majority voting for the test set and the filtering of partially qualified samples shows awareness of annotation noise. However, the paper does not explore alternatives: continuous quality scores, multi-class labeling (qualified/partially qualified/unqualified with ordinal regression), or training PAVRM with label smoothing to account for boundary uncertainty. The loss function ablation (Table 7) partially addresses the robustness concern by showing BCE and BT perform similarly, but this does not speak to whether a richer label space would improve PAVRM's quality as a reward signal.

---

Single-Model, Single-Benchmark, and Single-Aspect Evaluation Limits Generality

All generation experiments in Tables 1, 2, and 3 use Wan2.1-14B as the base model. The reward model is trained on Wan2.1-generated videos. The evaluation uses five metrics derived from VBench/VBench2 plus PAVRM's own score. The domain is human-centric video (portrait dataset for training, VBench prompts for evaluation). The optimization target is motion quality (dynamic degree, human anatomy), with aesthetics and semantic alignment explicitly deferred to future work in Section 6 ("may benefit from multi-aspect evaluation covering aesthetics and semantics").

The consequence is that the paper demonstrates PRFL works for one architecture, on one quality dimension, as measured by one benchmark suite. It provides no evidence that PRFL would improve video quality under different evaluation rubrics (e.g., text-video alignment as measured by CLIP scores, aesthetic quality as measured by AVA-style predictors, temporal consistency as measured by frame-by-frame VLM assessment) or that the approach transfers to different generator architectures (3D U-Net based models, models with different latent space dimensionalities, autoregressive video models). The cross-model reward model experiment (Table 9) provides some evidence about PAVRM transfer, but with a 10-point accuracy drop that suggests substantial degradation, and no corresponding generation experiment tests whether a PAVRM trained on Wan2.1 could effectively optimize a different generator.

The single-aspect focus is a deliberate scope limitation—the paper is about motion quality alignment—but it means PRFL's value proposition is narrower than the title implies. "Video Generation Models Are Good Latent Reward Models" suggests a general capability, but the experiments support a specific claim: VGMs are good latent reward models for motion quality assessment on Wan2.1-generated outputs. Whether VGMs would be equally effective for assessing text-video alignment, aesthetic quality, or temporal coherence is untested.

Mitigation status: The paper explicitly acknowledges the scope limitation in Section 6: "Our method focuses on motion quality and may benefit from multi-aspect evaluation covering aesthetics and semantics. Future work could explore hybrid reward frameworks, richer preference signals, and extensions to controllable generation and video editing." This is a clear and honest statement of scope. However, the paper provides no preliminary evidence that the approach could extend to other aspects—no experiment training PAVRM on aesthetic preference data, no analysis of whether VGM features encode aesthetic or semantic quality signals. For a practitioner interested in multi-aspect alignment, the paper demonstrates feasibility for motion but leaves the broader case entirely to future work, with no guidance on how much additional annotation data or architectural modification would be needed.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a paradigm shift in the architecture of visual reward modeling—but not the kind that announces itself with a new model architecture or training objective. The shift is conceptual: the generator and the evaluator can share the same backbone, and when they do, the evaluator inherits the generator's most valuable property for alignment—the ability to process noisy latents at arbitrary denoising timesteps. This is not an incremental improvement over prior video reward models; it is a redefinition of what a reward model for generative video alignment should be.

The scale of the shift. Prior to this work, the dominant approach to video reward modeling was to train or adapt vision-language models (VideoScore, VideoAlign, VideoPhy, PickScore) on human preference data in pixel space. These models operate on rendered RGB frames—a natural choice if you think of quality assessment as a perceptual task, but a catastrophic bottleneck if you think of quality assessment as a component in a gradient-based optimization loop. The paper demonstrates that this bottleneck is not merely inconvenient but architecture-limiting: RGB reward models force ReFL to operate at the final denoising step (since intermediate latents don't decode cleanly), which means they can only supervise visual polish, not the motion planning and structural decisions made in early denoising stages. The innovation is not that PRFL solves this problem—it's that the paper identifies the VGM's own backbone as the solution and provides the first complete recipe for repurposing it.

This is a paradigm shift because it changes what researchers should think of as a reward model. Before, the reward model was a separate system—often much larger or more complex than the generator (VideoAlign and VideoPhy are multimodal LLMs). Now, it can be a lightweight head bolted onto the generator's own early layers. The implications cascade: reward model training becomes faster and more data-efficient (24,000 generated videos rather than millions of diverse real videos), reward evaluation avoids VAE decoding entirely, and the reward signal becomes natively timestep-aware. These are not separate improvements—they are consequences of a single architectural insight.

What this work resolves. The paper reconciles a tension that has been implicit in the video generation alignment literature: the recognition that motion quality matters (motivating benchmarks like VBench and reward models like VideoAlign-MQ) versus the practical impossibility of optimizing motion quality via pixel-space ReFL (because full-frame VAE decoding crashes GPUs and single-frame workarounds can't see motion). Prior solutions accepted this tension as a constraint and worked around it—ContentV optimizes only the first frame, DOLLAR stays in latent space but loses timestep awareness. PRFL dissolves the tension by showing that it was an artifact of the reward model architecture, not an inherent limitation of ReFL. The result is not just better motion quality metrics (+56.00 DD at 720P) but a proof that motion-aware optimization was always possible—the field just needed to look at the generator differently.

Which directions become more attractive. The paper makes latent-space, process-level reward modeling the default paradigm for video alignment. Every method that requires pixel-space rewards (gradient stopping, trajectory shortcuts, first-frame workarounds) becomes a second-best solution—useful when a generator backbone isn't available for reward modeling, but clearly suboptimal when it is. The paper also elevates the query-based aggregation pattern (using learnable queries to compress spatiotemporal features into a quality token) as a general technique for extracting scalar judgments from video representations. This pattern could transfer to other spatiotemporal understanding tasks beyond reward modeling.

Which directions become less attractive. The paper implicitly argues against continued investment in VLM-based video reward models for ReFL training. While VLMs remain valuable for zero-shot evaluation and benchmark construction (the paper still uses VBench metrics for assessment), their role as training-time reward models for video generation is now challenged by an approach that is faster, more memory-efficient, and provides richer supervision. A practitioner choosing between training a VLM-based reward model on millions of preference pairs versus fine-tuning their generator's own early layers on 24,000 annotated samples faces a clear efficiency argument. VLM-based reward models may retain advantages for cross-model generalization (the paper's Table 9 shows PAVRM's accuracy drops on out-of-domain generators) and for multi-aspect evaluation beyond motion, but for the specific task of optimizing motion quality within a single model family, the VGM-based approach is now the baseline to beat.

The paper also makes first-frame-only ReFL a clearly deprecated approach. The quantitative results show RGB ReFL degrading subject consistency (92.26 vs. 97.34 at 480P T2V) while producing only modest DD improvements (38.00 vs. PRFL's 68.00). This validates the intuition that optimizing for first-frame quality without seeing temporal dynamics can actively harm video-level coherence. Future work that processes only subsets of frames for efficiency will need to justify why the temporal information loss is acceptable—a bar that Table 4 (PRFL processes 81 frames faster than RGB ReFL processes 1) makes very difficult to clear.

Follow-Up Research This Work Enables

Combining PAVRM with multi-aspect reward models for holistic video alignment. The paper explicitly scopes its contribution to motion quality and defers aesthetics and semantics to future work. A natural extension is to train multiple PAVRMs—or a single PAVRM with multiple query heads—on different quality dimensions (motion smoothness, text-video alignment, aesthetic appeal, temporal consistency) and combine their rewards during PRFL. The architecture already supports this: instead of one learnable query vector $q$, use $K$ query vectors $\{q_1, ..., q_K\}$, each attending to the same spatiotemporal features but learning to extract different quality signals. Each query's aggregated representation feeds a separate MLP head predicting a dimension-specific score. During PRFL, the total reward would be a weighted sum $r = \sum_k w_k r_k$, with weights $w_k$ controlled by the practitioner to emphasize different quality aspects. The key question for this extension is whether different quality dimensions require different timestep sampling strategies—the paper shows that dynamic degree is optimized at early/middle stages while human anatomy benefits from late-stage supervision (Table 3). Multi-aspect PRFL might need aspect-specific timestep sampling schedules, not the uniform $U(0,1)$ default. A strong experiment would train three PAVRMs (motion, aesthetics, alignment) on separate annotation dimensions, then compare uniform timestep sampling against dimension-aware schedules where the timestep distribution shifts based on which aspect is being optimized.

Stress-testing PRFL on non-human, non-portrait video domains. The paper's dataset consists of "approximately 31,000 portrait videos from online sources" with annotations focused on physical plausibility and subject deformity of human figures. The SFT regularization data is drawn from the same distribution. This raises the question: do the reported improvements in human anatomy (+21.52 HA at 720P T2V) and dynamic degree (+56.00 DD) generalize to non-human content, or has PRFL specialized to human-centric motion patterns? A direct test would replicate the entire pipeline on a different domain—for example, physics simulation videos (falling objects, fluid dynamics, rigid body interactions) where "motion quality" has a ground-truth referent (comparing generated trajectories against simulation outputs) rather than relying on human judgment. If PRFL trained on physics videos improves trajectory accuracy as measured by physical simulation metrics, the approach generalizes across content domains. If performance degrades, it suggests PAVRM's quality representations are domain-specific, and practitioners would need domain-matched reward model training data—a significant practical limitation. This experiment would also test whether the SFT regularization's dependence on real video data creates a content bias: does PRFL trained with human-portrait SFT data lose quality on non-human prompts, and can this be fixed by using domain-matched SFT data?

Direct comparison of PAVRM against VLM-based latent reward models (DOLLAR-style) to isolate process-level from latent-space benefits. The paper claims two advantages over RGB ReFL: operating in latent space (eliminating VAE decoding) and providing process-level supervision (evaluating at arbitrary timesteps). These advantages are confounded in the main comparisons, where the baselines (RGB ReFL, ContentV) lack both. To isolate the process-level contribution, a controlled experiment would compare PRFL with PAVRM (process-level, latent-space) against PRFL with an outcome-based latent reward model—for example, a version of PAVRM trained and evaluated only at $t \approx 0$ (near-clean latents). If the process-level variant substantially outperforms the outcome-based variant, the timestep-aware supervision is the active ingredient. If they perform similarly, the latent-space operation is the primary driver, and the paper's emphasis on process-level supervision would need tempering. A second controlled experiment would use DOLLAR's VLM-based latent reward model as the evaluator within the PRFL framework (single-step backpropagation at random timesteps, no VAE decoding). This tests whether a VLM adapted to latent space can provide process-level supervision comparable to a VGM-based reward model, or whether the VGM's architectural alignment with the generation process provides benefits beyond what a general-purpose model can achieve. These experiments would sharpen the paper's central claim by disentangling the effects of latent-space operation, architectural reuse, and timestep-aware training.

Testing whether PRFL-trained models maintain quality advantages under reduced inference step counts. The paper evaluates at 40 inference steps, the Wan2.1 default. A standard quality-efficiency trade-off in diffusion model research asks: can a model trained with sophisticated objectives produce better outputs at fewer inference steps than a pretrained model? Since PRFL optimizes the VGM to produce higher-quality latents at intermediate timesteps (the reward signal is applied at randomly sampled $s$), it may produce latents that require less refinement in the final denoising stages. A direct experiment would evaluate PRFL-trained and pretrained models at step counts $\{5, 10, 20, 40\}$, measuring VBench metrics at each budget. If PRFL at 20 steps matches or exceeds the pretrained model at 40 steps, this would demonstrate a practical inference-time benefit beyond the training-time efficiency already shown. The experiment would also test whether the random timestep sampling strategy during training (which optimizes the model to handle diverse noise levels) produces inference-time robustness to step count variation. If PRFL degrades more rapidly than the pretrained model at low step counts, it would suggest the training procedure over-specializes to intermediate states that only appear at specific step budgets—an important limitation for deployment flexibility.

Dynamic difficulty estimation for compute allocation during PRFL training. The paper samples timesteps uniformly during PRFL ($s \sim U(0,1)$), treating all prompts identically. However, the pretrained model's quality varies across prompts—some prompts produce consistently good videos needing only minor refinement, while others produce consistently bad videos requiring fundamental restructuring. Drawing an analogy to the test-time compute scaling literature (where compute allocation adapts to problem difficulty), a natural extension would be to estimate prompt difficulty from early PAVRM evaluations and allocate more PRFL updates (or different timestep distributions) to hard prompts. Concretely: during the gradient-free rollout (Algorithm 1, lines 11-13), evaluate PAVRM at several intermediate timesteps to estimate the trajectory's quality. If the trajectory looks promising (high PAVRM scores at middle timesteps), sample $s$ from the late stage to refine details. If the trajectory looks poor, sample $s$ from the early stage to restructure motion planning. This is an adaptive timestep sampling strategy that replaces the uniform distribution with a learned or heuristic policy. A strong experiment would compare uniform sampling against this adaptive strategy, measuring whether it improves sample efficiency (better quality at the same number of PRFL steps) or final quality (better quality after a fixed number of training epochs). The paper's existing sensitivity analysis (Table 3) provides the necessary evidence that different stages optimize different aspects—the missing piece is a policy for deciding which stage each prompt needs.

Open-vocabulary video quality assessment using PAVRM's query attention as a general-purpose video understanding head. The paper trains PAVRM's query attention mechanism to extract a single binary quality judgment. But the same architecture—frozen VGM features + learnable query attention + MLP head—could be trained on any spatiotemporal understanding task that benefits from process-level representations: action recognition in noisy videos, temporal action localization, video anomaly detection, or physical plausibility prediction. The key question is whether VGM features are generally useful for video understanding (in the way that ImageNet-pretrained features became general-purpose image representations) or specifically useful for generation-quality assessment (because the features encode the manifold of plausible generations). A systematic study would train PAVRM-style heads on standard video understanding benchmarks (Kinetics-400 for action recognition, UCF-Crime for anomaly detection, Something-Something-V2 for temporal reasoning) and compare against frozen CLIP features, VideoMAE features, and task-specific architectures. If VGM features are competitive, the paper's approach becomes a general technique for building lightweight, timestep-aware video classifiers from any pretrained video generator—a contribution that extends well beyond alignment.

Practical Applications and Downstream Use Cases

Efficient fine-tuning for production video generation APIs. A company deploying a video generation model as a cloud API (e.g., a Wan2.1-based service) can use PRFL to continuously improve motion quality from user feedback with minimal computational overhead. The scenario: as users rate generated videos (thumbs up/down on motion quality), these ratings accumulate into a preference dataset. PRFL enables the company to fine-tune the deployed model on this feedback without decoding videos to pixels during training—the 1.42× speedup and full-frame processing capability (Table 4) mean the fine-tuning loop is fast enough to run weekly or even daily. The memory efficiency (66.81 GB VRAM, fitting on a single 80GB A100 or H100) means this fine-tuning doesn't require multi-node distributed training. The SFT regularization dataset can be continually updated with high-quality user-approved outputs, creating a flywheel where better generation quality → more positive feedback → better SFT data → better quality. The key deployment metric: a 1.42× training speedup translates directly to faster model update cycles, and the +56.00 DD improvement (720P T2V, Table 1) represents the motion quality gain users would perceive after each cycle.

Content moderation and quality filtering for user-generated video platforms. PAVRM can be deployed as a standalone quality filter that operates in latent space. The scenario: a platform allows users to generate videos via an API (text-to-video or image-to-video). Before returning the video to the user, the platform wants to flag or reject outputs with severe motion artifacts (distorted human anatomy, physically impossible motion) to maintain content quality standards. PAVRM evaluates the generated video in latent space by: (1) encoding the video through the VAE (one forward pass, no gradient), (2) running the first 8 DiT blocks of the VGM with a randomly sampled timestep (or several timesteps for robustness), and (3) classifying good vs. bad via the query attention + MLP head. This is orders of magnitude faster than decoding to pixels and running a VLM-based quality classifier—the video never leaves latent space except for final delivery to the user. PAVRM's 84.18% accuracy (Table 5, 720P I2V) provides a baseline quality gate; the ~16% error rate (mostly false positives/negatives near the decision boundary) could be managed by using PAVRM as a first-pass filter with ambiguous cases escalated to human review or a more expensive VLM. The cross-model generalization experiment (Table 9) suggests this filter would need to be retrained if the underlying generation model changes, but within a single deployed model, retraining is straightforward.

Data curation for self-improving video generation pipelines. The paper's finding that generated videos can be reliably classified as good or bad by a lightweight head on the generator's own backbone enables automated data curation for self-improvement loops. The scenario: a research team wants to iteratively improve their video generation model using a ReST-style pipeline (generate many videos, filter for quality, fine-tune on the good ones, repeat). The standard approach requires either (a) human evaluation at each iteration (slow, expensive) or (b) a separate reward model trained on human preferences (requires maintaining and updating a separate model). PRFL's PAVRM provides a third option: after each generation round, score all outputs with PAVRM (fast, no VAE decoding needed), select videos above a confidence threshold for the next SFT round, and optionally update PAVRM on the new generated-vs-real distribution to prevent staleness. The key efficiency: PAVRM's 84% accuracy on motion quality (Table 5) means automated filtering is imperfect but better than random, and the SFT + PRFL training loop can run entirely in latent space without human annotation cycles between iterations. The +30.00 to +56.00 DD improvements in Tables 1 and 2 represent the per-iteration quality gain that such a self-improvement loop could sustain.

When to Prefer This Method

This paper does not articulate a structured decision framework with explicit conditions for choosing PRFL over named alternatives. It compares against specific baselines (SFT, RWR, RGB ReFL) and demonstrates superior performance across the evaluated metrics, but does not characterize the failure modes or boundary conditions where a practitioner would prefer a different approach. The limitations section in the Conclusion acknowledges that PRFL "focuses on motion quality and may benefit from multi-aspect evaluation," but this is framed as an avenue for extending PRFL rather than a condition for choosing alternatives. Similarly, the cross-model generalization experiment (Table 9) identifies a weakness but does not position it as a decision criterion—e.g., "prefer VLM-based reward models when you need to optimize across multiple generator architectures without retraining." A forced "prefer X when Y" matrix would fabricate trade-offs the paper does not discuss.