DreamVideo-Omni: Omni-Motion Controlled Multi-Subject Video Customization with Latent Identity Reinforcement Learning

ArXiv: 2603.12257

🎯 Pitch

A 1.3B-parameter video model beats a 14B competitor on precise motion control—but only when it knows which motion cue belongs to which subject. The secret is binding bounding boxes and trajectories to specific identity references via explicit “role embeddings,” then refining generation with a latent-space reward model trained to judge identity consistency without expensive pixel decoding.

---

1. Executive Summary

DreamVideo-Omni introduces a unified framework that harmoniously integrates multi-subject identity customization with omni-motion control—spanning global object motion, local limb dynamics, and camera movements—within a single DiT architecture through a progressive two-stage training paradigm. The first stage, omni-motion and identity supervised fine-tuning, jointly trains on comprehensive control signals using a condition-aware 3D rotary positional embedding to coordinate heterogeneous inputs, hierarchical motion injection to enhance global guidance, and group and role embeddings to explicitly bind motion signals to specific identities (resolving multi-subject ambiguity by anchoring, e.g., each subject's bounding box and trajectory to its corresponding reference image). The second stage, latent identity reward feedback learning, trains a VDM-based Latent Identity Reward Model (LIRM) that evaluates motion-aware identity consistency directly in latent space—bypassing expensive VAE decoding—and backpropagates identity-aligned gradients into the video generator, mitigating identity degradation under large-amplitude motions. Evaluated on the newly constructed DreamOmni Bench (1,027 videos with dense annotations for single- and multi-subject scenarios), DreamVideo-Omni achieves a 0.558 mIoU and 9.31 EPE for motion control—surpassing a 14× larger model (Wan-Move, 14B vs. 1.3B)—while improving identity fidelity to 0.499 R-DINO and 0.301 Face-S, establishing that precise multi-subject customization with omni-motion control is achievable only when motion signals are explicitly bound to identities and identity preservation is optimized through latent-space reward feedback aligned with human preferences.

2. Context and Motivation

The Core Problem: Identity and Motion Are Fundamentally in Conflict

The paper addresses a fundamental tension that arises when you try to do two things simultaneously in video generation: (1) preserve the visual identity of specific subjects (so a particular person looks like that person, not just "a person"), and (2) control their motion with precision (specifying where they go, how their limbs move, and how the camera behaves). These two goals are not merely difficult to achieve together—they are actively opposed at the level of what the optimization objective demands.

To understand why, consider what each objective requires from the generated pixels. Identity preservation rewards the model when the output matches a static reference image. The ideal identity-preserving output is, in some sense, a faithful copy of the reference: the same facial structure, the same clothing texture, the same fine-grained details that make a subject recognizable. This pushes the model toward pixel-level consistency with a fixed template. Motion control, conversely, requires the model to change those pixels over time—subjects must rotate, deform, occlude themselves, move into different lighting conditions, and occupy different spatial positions. The very act of moving a subject changes its visual appearance, often in ways that reduce similarity to the reference image when measured by standard metrics.

The paper explicitly frames this as more than just a difficulty trade-off. It is a divergence in what the loss function incentivizes:

"This stems from the divergent nature of the objectives: identity preservation encourages pixel-level consistency with a static reference image, whereas motion control necessitates dynamic pixel variation and temporal evolution to render movement. Standard diffusion reconstruction losses are insufficient to reconcile this conflict, leading to degradation of fine-grained identity details, particularly when synthesizing large-amplitude motions."

In plain terms: standard training—minimizing the difference between predicted and ground-truth noise—cannot simultaneously satisfy the demand for static fidelity and dynamic variation. When you ask a model to do both, it tends to sacrifice identity detail, producing subjects that drift from their reference appearance as they move. This is the central gap the paper seeks to close.

Why This Matters: From "Generating Videos" to "Controlling Video Content"

The practical significance of this problem extends well beyond academic benchmarking. Current video generation models (Wan, CogVideoX, VideoCrafter, etc.) can produce visually impressive outputs from text descriptions, but they operate at the level of statistical plausibility rather than intentional control. You can prompt for "a person playing soccer" and get a realistic-looking video, but you cannot specify which person (your friend, a specific celebrity, a character from a reference photo) or dictate their exact movement trajectory (run to the left, then kick, then circle back). For any application where the generated content must serve a specific purpose—advertising with branded characters, pre-visualization for film production, personalized content creation, or assistive technologies where specific individuals must be depicted—this lack of control is a non-starter.

The paper situates this need within the broader trajectory of the field. Diffusion models have solved the problem of generating any video from text. The frontier has shifted to generating exactly the right video, according to user specifications that go beyond natural language. The authors identify three specific failure modes that make existing solutions inadequate for this frontier:

1. Limited Motion Control Granularity. Most existing methods condition generation on a single type of motion signal—bounding boxes or depth maps or sparse trajectories—but not all three simultaneously. A bounding box can specify where an object is, but not whether the subject is raising their arm, nodding their head, or performing in-place gestures. A trajectory can specify the path of a specific point, but not the overall spatial extent of the subject or the camera's movement through the scene. Real-world video production requires all of these simultaneously: a character walks across a room (global motion) while gesturing with their hands (local dynamics) as the camera pans to follow them (camera movement). The paper argues that restricting to any one signal type yields an incomplete model of motion that cannot capture the richness of actual video content.

2. Ambiguity in Motion Control for Multiple Subjects. When multiple subjects appear in a scene—each with their own reference image, their own bounding box, and their own trajectory—the model faces a binding problem: which motion signal corresponds to which identity? Existing methods typically inject all conditioning signals into the model without explicit association mechanisms. The model sees a set of reference images and a set of motion patterns, but nothing tells it that reference image A should move according to trajectory B while reference image C should follow bounding box D. The paper notes that this ambiguity is "further exacerbated when integrating multi-granular motion controls," because the model must not only bind the right motion to the right subject but also distinguish between different types of motion signals (boxes vs. trajectories) and their roles.

The consequence is confusion: subjects swap identities, motions get applied to the wrong characters, and the resulting video looks plausible but wrong in ways that are obvious to a human viewer. In multi-subject scenarios—which are the norm, not the exception, in practical applications—this ambiguity is a fundamental barrier.

3. Identity Degradation Specifically Under Motion. Even when single-subject customization works adequately in static or low-motion scenarios, introducing significant movement causes identity fidelity to collapse. The paper observes that "compared to independent subject customization, introducing motion control often compromises identity fidelity." This is not a general failure of customization models—it is specifically a failure of customization under the additional demand of motion control. The model's capacity is divided, and the reconstruction loss that works for static customization provides insufficient signal to maintain identity through large pose changes, rotations, and occlusions.

Where Existing Approaches Fall Short

The paper categorizes prior work into two largely disconnected research threads, each solving only half the problem:

Subject-driven customization methods (VideoBooth, DreamVideo, Phantom, VACE, Video Alchemist, ConsisID) focus on injecting appearance information from reference images into the generation process. They use adapters, attention mechanisms, or tuning-free techniques to make the model "aware" of a specific subject's appearance. Some support multi-subject scenarios, and a few (DreamVideo, MotionBooth) attempt to add motion control. But these methods share two critical weaknesses: (a) they treat motion control as an afterthought, bolting it onto an identity-focused architecture rather than designing for both from the ground up, and (b) they rely entirely on standard diffusion reconstruction losses for training, which—as argued above—cannot reconcile the identity-motion conflict. The result, visible in the paper's qualitative comparisons (Figures 5 and 6), is subjects that look correct when static but degrade into generic approximations when asked to move, or subjects that remain faithful but ignore motion instructions.

Motion-controlled generation methods (MotionCtrl, Boximator, Tora, Motion Prompting, Wan-Move) excel at guiding movement via bounding boxes, trajectories, or other spatial signals. Some achieve impressive precision on complex paths. But they have a symmetric limitation: they don't support user-specified subject identities. You can make a person follow a trajectory, but not your specific person. The models are trained to generate plausible visual content that satisfies motion constraints, not to preserve the appearance of a particular reference subject. Moreover, even within motion control, the paper argues that existing methods are incomplete—they typically support only one type of motion signal (trajectories or boxes, not both) and cannot simultaneously control global placement, local dynamics, and camera movement within a unified framework.

Recent attempts at unification (DreamVideo-2, FullDiT, OmniVCUS, Tora2) try to bridge this gap, combining subject customization with some form of motion control in a single model. The paper acknowledges that these represent steps in the right direction, but argues they "often yield suboptimal performance due to the intrinsic trade-off between subject preservation and motion control." The three limitations above—limited granularity, control ambiguity, and identity degradation—persist in these unified approaches because they inherit the architectural assumptions of their predecessors: undifferentiated conditioning injection (no explicit subject-motion binding), single-type motion signals, and reliance on reconstruction losses alone.

How This Paper Positions Itself

DreamVideo-Omni's central thesis is that solving the identity-motion conflict requires simultaneous improvements on two fronts: the architecture must be redesigned to explicitly resolve motion ambiguity, and the training objective must be extended beyond reconstruction loss to include identity-aware feedback that aligns with human perception.

This framing is important because it rejects a common assumption in the literature: that the identity-motion trade-off is an inevitable limitation that must be balanced or mitigated. Instead, the paper argues it is a consequence of specific design choices—unstructured conditioning injection, single-signal motion control, and short-sighted optimization objectives—that can be systematically addressed. The paper positions its contributions not as incremental improvements to existing methods but as a required architectural and algorithmic reformulation:

• Versus subject-driven methods: DreamVideo-Omni is not a customization model with motion support added. It is designed from the start as a unified architecture where identity and motion conditioning are first-class, equally-weighted inputs processed through dedicated mechanisms (hierarchical injection, group/role embeddings, condition-aware 3D RoPE) that prevent one from overwhelming the other.

• Versus motion-controlled methods: DreamVideo-Omni supports omni-motion control—global boxes, local trajectories, and camera movement simultaneously—arguing that anything less is insufficient for real-world video production. It further argues that motion control must be identity-aware: knowing which subject to move is as important as knowing how to move them.

• Versus prior unified attempts: The paper's key differentiator is the latent identity reward feedback learning stage. Rather than relying solely on pixel-space reconstruction to handle identity preservation, which the paper argues is fundamentally incapable of resolving the identity-motion conflict, DreamVideo-Omni trains a specialized reward model (LIRM) that evaluates identity consistency in a motion-aware manner—penalizing "copy-paste" artifacts where the model simply stamps a static reference onto each frame, while rewarding genuine identity preservation through pose and viewpoint changes. Crucially, this reward model operates in latent space and is built on a video diffusion backbone, giving it access to spatiotemporal priors that static image encoders (CLIP, DINO) lack.

The paper also positions itself as filling an evaluation gap. Existing benchmarks isolate customization from controllable generation or focus on simple point trajectories. DreamOmni Bench is presented as the first benchmark to "simultaneously quantify multi-subject consistency and motion control precision" across both bounding boxes and dense trajectories, enabling proper comparative evaluation that wasn't previously possible.

A Deeper Motivation: The Nature of Identity

There is a subtler philosophical motivation underlying the paper's approach that is worth drawing out. The authors recognize that subject identity is not equivalent to pixel-wise correspondence. A person's visual appearance changes constantly—different angles, different lighting, different expressions, different poses—yet their identity remains constant. Humans recognize this trivially; standard computer vision metrics do not. A CLIP or DINO embedding of a face in profile may score poorly against a frontal reference, even though any human would immediately recognize them as the same person.

This insight motivates the reward feedback learning approach. By training a reward model on human preference data (the LIRM is trained on human-annotated win-lose pairs where annotators judged whether a video preserved a subject's identity), the optimization signal captures the perceptual, flexible notion of identity rather than rigid template matching. The model learns that identity means "looks like the reference subject in a way a human would agree with, accounting for natural variation from motion," not "minimizes pixel distance to the reference image." This alignment with human preferences is what the paper argues is missing from standard reconstruction-based training and what enables the model to break through the identity-motion trade-off.

The paper's two-stage design reflects this prioritization: first establish the architectural capacity to process heterogeneous control signals without ambiguity (Stage 1, the "hardware" foundation), then refine the output to align with human identity perception (Stage 2, the "software" refinement). This is a deliberate sequencing—without Stage 1's binding mechanisms and omni-motion support, Stage 2's reward feedback would be optimizing a fundamentally confused model. Without Stage 2, Stage 1's model would produce videos that are structurally correct but identity-degraded under motion. The two stages are presented as individually necessary and jointly sufficient for the claimed capabilities.

3. Technical Approach

3.1 Reader Orientation

DreamVideo-Omni is a unified video diffusion transformer that takes multiple reference images and motion control signals as input and generates a complete video where specific subjects move according to precisely specified global trajectories, local limb motions, and camera movements. The system addresses the fundamental trade-off where standard video diffusion training cannot simultaneously preserve static subject identity and produce dynamic motion—identity degrades under large movement because pixel-level reconstruction losses reward copying reference images while motion demands pixel change—by combining two innovations: (1) an architectural redesign that explicitly binds each motion signal to its corresponding subject identity, preventing the model from confusing which subject follows which trajectory, and (2) a second training stage where a learned reward model, operating directly in the VAE latent space to avoid expensive decoding, provides identity-preserving gradient signals that align with human perceptual judgments rather than low-level pixel matching.

3.2 Big-Picture Architecture (Diagram in Words)

The system comprises five major components organized into a two-stage training pipeline, all built on a pre-trained text-to-video DiT backbone (Wan2.1-1.3B):

1. Condition Encoding Pipeline — takes heterogeneous user inputs (reference images, bounding boxes, point trajectories) and converts them into latent-space tokens compatible with the DiT. Reference images are VAE-encoded after augmentation; bounding boxes are rendered as colored RGB videos and VAE-encoded; trajectories are converted to sinusoidal positional encodings scattered into feature maps.

2. All-in-One Video DiT (Stage 1 output) — the core transformer that processes video frame tokens, reference image tokens, bounding box latents, and trajectory tokens as a single concatenated sequence. Uses condition-aware 3D Rotary Positional Embedding (RoPE) to distinguish input types, hierarchical bounding box injection to reinforce global motion guidance, and group/role embeddings to bind each subject's appearance to its motion signals. Trained with a reweighted diffusion loss that amplifies foreground regions.

3. Latent Identity Reward Model (LIRM) — a specialized reward model built from the first 8 layers of the VDM backbone, plus an identity cross-attention layer and a reward prediction head. Takes a noisy video latent and a clean reference image latent (both in VAE space), uses the reference features as queries to attend to the video features, and predicts a scalar identity reward. Trained on human-annotated win-lose pairs with binary cross-entropy loss.

4. Latent Identity Reward Feedback Learning (LIReFL) — the Stage 2 optimization procedure. During gradient-enabled denoising at a randomly sampled intermediate timestep, the predicted latent is scored by the frozen LIRM, and gradients from the reward are backpropagated through the denoising step into the video generator. Regularized by the Stage 1 diffusion loss to prevent reward hacking.

5. DreamOmni Bench — a zero-shot evaluation suite of 1,027 curated real-world videos with comprehensive annotations (reference images, bounding boxes, trajectories, subject masks, and captions), categorized into single-subject (436) and multi-subject (591) scenarios, supporting unified evaluation of identity preservation and motion control.

Information flow during inference: User provides ⟨reference images⟩ + ⟨bounding boxes⟩ + ⟨trajectories⟩ + ⟨text prompt⟩ → VAE encodes references, boxes rendered and VAE-encoded, trajectories converted to feature tokens → all tokens concatenated with group/role embeddings added and condition-aware 3D RoPE applied → DiT denoises from random noise over 50 UniPC steps with CFG scale 5.0 → VAE decodes to final video.

3.3 Roadmap for the Deep Dive

• First, the Stage 1 training framework: how the model architecture processes heterogeneous inputs, the conditioning signal formulation for each of the four control types, and the specialized components (condition-aware 3D RoPE, hierarchical motion injection, group/role embeddings) — because this establishes the foundation for unified controllability.
• Second, the Stage 2 reward feedback learning paradigm: why standard diffusion losses are insufficient for identity preservation under motion, how the Latent Identity Reward Model is constructed and trained, and how its gradients are backpropagated to the generator — because this is the key mechanism for breaking the identity-motion trade-off.
• Third, the dataset construction pipeline and the DreamOmni Bench — because the training data quality and evaluation protocols are essential for interpreting the experimental results.
• Fourth, the training hyperparameters, optimization details, and design rationale for both stages — because these specify exactly what was done and why specific choices matter.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems-building paper whose core idea is that combining an identity-aware unified architecture with latent-space reward feedback learning resolves the fundamental conflict between subject customization and motion control in video generation. The approach is structured as a progressive two-stage training paradigm: first, establish architectural capacity for joint multi-condition processing; second, refine identity fidelity through human-aligned optimization.

---

Stage 1: Omni-Motion and Identity Supervised Fine-Tuning

Model Architecture and Multi-Task Design

The paper instantiates an all-in-one framework by adapting a pre-trained text-to-video DiT, specifically Wan2.1-1.3B T2V. Rather than designing separate modules for each capability, the model is jointly trained on a comprehensive set of tasks — single-subject customization, multi-subject customization, global object motion control, local object motion control, and camera movement control — using a single transformer that processes all conditioning signals as concatenated tokens.

The key architectural insight is that the temporal axis becomes the multiplexing dimension for heterogeneous inputs. Instead of building separate encoder pathways that are fused later, all conditioning signals are formatted as tokens with specific temporal indices and concatenated along the temporal dimension before entering the DiT blocks. This unified sequence then flows through the standard transformer layers, with the condition-aware 3D RoPE encoding what each token represents. Specifically, the concatenation order is:

$\text{Tokens} = [\text{video frames}_{0:T-1}, \text{reference images}_{0:N_{\text{max}}-1}, \text{trajectory tokens}_{0:T-1}]$

where $T$ is the number of video frames (49 during training) and $N_{\text{max}}$ is the maximum number of reference subjects (padded to a fixed capacity within each batch). The bounding box latents are not concatenated — they are element-wise added to the noisy video latents and to DiT block outputs, a design choice discussed below under hierarchical injection.

Four Conditioning Signal Types

The paper carefully formulates four compact, interaction-friendly conditioning signals, each designed to be (a) extractable from real video data via automated pipelines, (b) intuitive for users to specify, and (c) non-redundant with each other.

1. Subject appearance. One reference image per subject, segmented to remove the background (blanks out non-subject pixels). The blank-background reference is critical: if the background were included, the model might learn to copy background elements from the reference rather than generating scene-appropriate surroundings, exacerbating "copy-paste" artifacts. The reference image is augmented with geometric transformations (flipping, rotation, affine shearing, cropping) and visual degradations (blur, color jitter) to prevent overfitting — without these augmentations, the model learns to directly copy pixel patterns from the reference image into the generated frames, producing static, artificial-looking subjects that don't integrate naturally into dynamic scenes.

2. Global object motion via scene-anchored bounding boxes. Each subject's spatial extent, position, and scale over time are represented as bounding box sequences. Users can specify start and end boxes with optional intermediate boxes. The box sequences are rendered as RGB videos on a white background, where each subject is assigned a unique random color, and pixel-wise averaging is applied in overlapping regions. These rendered videos are then VAE-encoded into the latent space. The white background is chosen so that zero-valued latent regions naturally encode "no object here," while colored regions encode spatial constraints. Box sequences exhibiting abrupt object fluctuations (detected by measuring adjacent-frame IoU) are filtered out before training, ensuring training stability — without this filtering, the model would be trained on physically impossible bounding box jumps that degrade motion learning.

3. Local object motion via sparse point-wise trajectories. Finer-grained dynamics (limb raises, head turns, in-place gestures, complex non-rigid deformations) are encoded as sparse trajectories. The paper designs a hybrid sampling strategy to select trajectory points: (i) random grid sampling ensures broad coverage of whole-scene dynamics including background and object regions, while (ii) object-aware sampling, which selects points strictly within foreground masks, focuses on intricate local dynamics. These two modes are stochastically alternated during training. A subset of trajectories is randomly dropped during training to improve robustness to varying trajectory densities at inference time. To convert these point trajectories into tokens, the paper follows Motion Prompting's approach: unique sinusoidal positional encodings are generated for each point's spatiotemporal coordinates and scattered into blank feature maps according to their discretized (t, x, y) positions, producing trajectory tokens that are concatenated with other latent tokens.

4. Camera movement via point-wise trajectories on background pixels. The paper makes an important efficiency-motivated design choice: rather than relying on explicit 3D camera parameters (extrinsic/intrinsic matrices) which require auxiliary datasets and increase training cost, camera movement is induced by applying point-wise trajectories to background pixels only. This unifies camera and local object motion control under the same trajectory conditioning mechanism, reducing training overhead while improving interactivity. The trajectories assigned to background points encode global frame-to-frame shifts caused by camera pan, tilt, zoom, and rotation. Since camera motion affects all pixels uniformly (in the background, where there's no object parallax), a sparse set of background trajectories is sufficient to characterize the camera path.

Hierarchical Motion Injection for Bounding Boxes

The bounding box condition requires stronger spatial guidance than what simple concatenation provides, because global object placement is a per-frame spatial constraint that must influence every layer of the generation process. The paper introduces hierarchical motion injection: bounding box latents are added to both the input noisy latents and the output of each DiT block through learnable, layer-specific zero-convolution layers.

The injection is formulated as:

$h_0 = z_t + Z_{\text{in}}(z_{\text{box}})$

$h_{l+1} = \text{Block}_l(h_l) + Z_l(z_{\text{box}})$

where $z_t \in \mathbb{R}^{T \times H \times W \times C}$ is the input noisy video latents (the video frames with noise added at diffusion timestep $t$), $z_{\text{box}} \in \mathbb{R}^{T \times H \times W \times C}$ is the VAE-encoded bounding box latents (the rendered colored-box video in latent space), $Z_{\text{in}}$ is a zero-convolution at the input stage, $Z_l$ is a zero-convolution at the $l$-th DiT block, and $h_l$ is the hidden state input to the $l$-th block.

What it computes: At the input, the bounding box latent is added to the noisy video latent, giving the model an initial spatial layout signal before any transformer processing. Then, after each transformer block $l$, the bounding box latent is added again to the block's output $h_{l+1}$, reinforcing the spatial constraint at every processing depth. The zero-convolutions $Z_{\text{in}}$ and $Z_l$ are 1×1 convolutions initialized to zero — meaning at the start of training, they contribute nothing, and the model gradually learns to incorporate the bounding box information into the appropriate feature channels. This zero-initialization is important: it ensures the bounding box injection doesn't disrupt the pre-trained model's behavior at the start of training; the model learns to attend to the box signals only as they prove useful.

Why this form: Alternative approaches would be to (a) concatenate bounding box latents as additional tokens in the input sequence, or (b) add them only at the input and let the transformer propagate the spatial information internally. Approach (a) increases token sequence length, which quadratically increases attention computation and may dilute the per-token representational capacity. Approach (b) risks the spatial guidance being gradually "forgotten" through successive transformer layers, as self-attention mixes information across tokens. Hierarchical injection solves both problems: it keeps the token sequence length unchanged (no extra tokens), and it refreshs the spatial signal at every layer, preventing information decay. The zero-convolution initialization further provides a smooth warm-start from the pre-trained model, which is critical because Wan2.1-1.3B has strong pre-trained priors that should not be immediately overridden.

Condition-Aware 3D Rotary Positional Embedding (RoPE)

The unified DiT processes tokens from four distinct types in one concatenated sequence. Standard positional embeddings would assign sequential temporal indices $0, 1, 2, \ldots$ to all tokens, confusing the model about which tokens belong to video frames versus reference images versus motion signals. The paper redesigns the temporal dimension of the 3D RoPE to encode token type information through the indexing scheme:

(i) Video frame tokens: assigned sequential temporal indices $t \in [0, T-1]$ to maintain temporal consistency. Bounding box latents, which are element-wise added to video frames, naturally inherit these same embeddings — a deliberate consequence of the hierarchical injection design.

(ii) Reference image tokens: assigned a shared, distinct time index $t_{\text{ref}}$ for all valid reference image tokens. The shared index indicates that all reference images are static visual conditions, not sequential frames. By giving them a temporal index that doesn't intersect with the video frame range $[0, T-1]$, the model learns to treat reference appearance as persistent across the entire video duration rather than as content at a specific time step.

(iii) Padding tokens: assigned a distinct "invalid" time index $t_{\text{pad}}$. Since different training samples have different numbers of reference subjects, the reference image tokens are padded to a fixed capacity $N_{\text{max}}$ within each batch. The $t_{\text{pad}}$ index allows the model to learn to identify and ignore these non-informative tokens — they carry no useful information and should not influence the generated content.

(iv) Trajectory tokens: assigned the same temporal indices $t \in [0, T-1]$ as video frame tokens, ensuring strict spatiotemporal alignment with the corresponding video frames. This means a trajectory point at $(t, x, y)$ has the same temporal position encoding as the video frame at time $t$, making it straightforward for the attention mechanism to connect motion constraints to the correct frame.

Why this form: The alternative — standard sequential indexing for all tokens — would create positional ambiguity. A reference image token at temporal index 5 and a video frame token at temporal index 5 would appear positionally equivalent to the model, even though one is a static appearance condition and the other is a specific moment in the video. The type-aware indexing scheme encodes the functional role of each token directly into its positional embedding, which the paper demonstrates is critical: ablating this component (Table VII, "w/o Cond-Aware 3D RoPE") causes catastrophic performance collapse — R-DINO drops from 0.499 to 0.139 and Face-S drops from 0.301 to 0.039 in single-subject mode.

Group and Role Embeddings

To resolve control ambiguity — the problem where a model with multiple subjects and multiple motion signals cannot determine which motion corresponds to which identity — the paper introduces two complementary embedding mechanisms.

Group embeddings create explicit binding between a subject's identity and its motion controls. The paper formulates a fundamental control unit as a triplet:

$\text{Unit}_i = \langle \text{Reference Subject}_i, \text{Global Box}_i, \text{Local Trajectory}_i \rangle$

where each unit is assigned a unique learnable group embedding $g_i$. This embedding is added to all latent tokens of the reference image for subject $i$, and is injected exclusively into the spatial regions and track points corresponding to that subject in the bounding box and trajectory latents. For instance, if group 1 corresponds to "the girl in the orange jacket," then group embedding 1 is added to (a) all tokens of her reference image, (b) the pixels within her bounding boxes, and (c) the trajectory points that track her movement. This means the model can distinguish subject 1's motion tokens from subject 2's motion tokens by their group embedding, even when they occupy similar spatial positions — a form of learnable routing through the embedding space.

Role embeddings differentiate the functional nature of input signals. An object embedding is added to all reference image tokens, designating them as visual appearance assets. A control embedding is applied to all bounding box and trajectory tokens, marking them as motion guidance. This two-level embedding scheme means that even without group-specific information, the model can distinguish "this token provides appearance information" from "this token provides spatial constraints" — a prerequisite for meaningfully integrating them.

Why this form: Prior unified methods inject all conditioning signals without explicit binding mechanisms, relying on the model to learn associations implicitly through co-occurrence statistics. This fails in multi-subject scenarios because co-occurrence is ambiguous — subject A and subject B often co-appear in the same frames, so their motion signals are statistically correlated with both identities. Group embeddings break this ambiguity by providing a deterministic labeling of which tokens belong to which control unit. The role embeddings further ensure that, within a single group, the model knows whether a token represents "what subject $i$ looks like" or "how subject $i$ should move." The ablation in Table VII confirms this: removing group and role embeddings degrades mIoU from 0.524 to 0.459 and increases EPE from 26.24 to 20.69 in single-subject mode (note: EPE direction is lower = better, so the increase from 20.69 to 26.24 represents degradation; the values in the table: w/o Group & Role Emb. has EPE 26.24 vs. Stage1's 10.53, indicating binding is also critical for motion precision even in single-subject scenarios).

Training Loss for Stage 1

The Stage 1 objective is a reweighted diffusion loss:

$\mathcal{L}_{\text{sft}} = \mathbb{E}_{z, \epsilon, C, t}\left[(1 + \lambda_1 M) \cdot \|\epsilon - \epsilon_\theta(z_t, C, t)\|^2_2\right]$

where $z$ is the clean video latent, $\epsilon \sim \mathcal{N}(0, I)$ is the sampled Gaussian noise, $C = \{c_{\text{txt}}, z_{\text{ref}}, z_{\text{box}}, z_{\text{traj}}\}$ is the comprehensive conditioning set (text prompt, reference image latents, bounding box latents, trajectory feature maps), $t$ is the diffusion timestep, $\epsilon_\theta$ is the denoising DiT network parameterized by $\theta$, $M$ is the binary bounding box mask (1 for foreground pixels inside any bounding box, 0 for background), and $\lambda_1 > 0$ is a balancing factor set to 2.

What it computes: The standard diffusion loss $\|\epsilon - \epsilon_\theta(z_t, C, t)\|^2$ measures how well the model predicts the noise added to the video latent at timestep $t$, given all conditioning signals. The reweighting factor $(1 + \lambda_1 M)$ amplifies the loss on pixels inside bounding boxes by $1 + 2 = 3\times$ relative to background pixels (weighted $1\times$). The expectation $\mathbb{E}_{z,\epsilon,C,t}$ is estimated by sampling $(z, \epsilon, C, t)$ tuples from the training data and noise distribution.

Why this form: Standard uniform diffusion loss treats all pixels equally, but for subject customization, the foreground pixels (where subjects appear) are far more important than background pixels (which can be synthesized plausibly even with weaker constraints). The reweighting focuses the model's capacity on learning to faithfully reproduce subject identities inside bounding boxes, while the background — constrained primarily by the text prompt — receives less emphasis. This is a targeted form of importance sampling in the pixel space: the model is explicitly told that errors on subject pixels matter more, which accelerates identity learning and reduces the diffusion loss's tendency to average over identity details in favor of easier-to-predict background patterns. The specific value $\lambda_1 = 2$ means foreground errors are 3× more costly than background errors; higher values would risk overfitting subjects to their exact reference appearances, while lower values would insufficiently prioritize identity.

Data Augmentation for Subject Reference Images

The paper applies a robust augmentation pipeline to reference images to combat "copy-paste" artifacts — the phenomenon where the model simply stamps the reference image pixels into the generated frames without adapting to new poses, viewpoints, or lighting. The augmentations include:

• Geometric transformations: random flipping, rotation, affine shearing (which distorts the image along an axis), and cropping. These force the model to learn that a subject's identity is invariant to viewpoint and orientation — the face embedding should be the same whether the reference shows the subject head-on or at an angle.
• Visual degradations: random blur and color jitter. These prevent the model from overfitting to the specific lighting conditions, color balance, and sharpness of the reference image, making identity preservation robust to scene lighting changes.

Augmentations are applied stochastically with probability $p = 0.5$ during training, meaning some training steps use unaugmented references and some use augmented ones. This balanced exposure ensures the model can both leverage high-fidelity reference details when available and maintain identity when references are imperfect or differ from the target scene conditions. Importantly, reference images are sampled from frames temporally disjoint from the training clip — if the reference were taken from the same clip used for training, the model could trivially memorize the exact mapping from reference pixels to target pixels, bypassing the need to learn generalizable identity representations.

---

Stage 2: Latent Identity Reward Feedback Learning

Motivation: Why Standard Diffusion Losses Are Insufficient

The paper's core argument for Stage 2 is that the reconstruction-based diffusion loss optimized in Stage 1 is structurally incapable of resolving the identity-motion conflict. The reweighted loss (Equation 2) minimizes per-pixel prediction error, which provides a strong signal for low-level visual fidelity but a weak signal for perceptual identity consistency. When a subject rotates their head, the pixel pattern of their face changes dramatically (different visible features, different shading, different relative positions of landmarks), and the diffusion loss penalizes any deviation from the ground-truth pixels. This encourages the model to minimize visible motion — to keep the face as close to the reference as possible — because any motion-induced appearance change increases the loss. The model learns a compromise: produce some motion (because the bounding boxes and trajectories demand it) but sacrifice fine-grained identity details (because preserving them under motion requires learning a complex, viewpoint-invariant identity representation that the simple pixel loss does not explicitly reward).

The paper frames this as a misalignment between the training objective and human perception. Humans recognize identity as invariant across pose changes, but the L2 loss on pixel predictions treats every pixel change as an error. A reward model trained on human preference judgments can provide a learning signal that says "this is still the same person, even though the pixels changed."

Latent Identity Reward Model (LIRM) Architecture

The LIRM is constructed from three modules, all built on a pre-trained VDM backbone (the first 8 layers of Wan2.1-1.3B, used as a frozen or fine-tuned feature extractor):

Module 1: VDM Backbone for Spatiotemporal Feature Extraction. Given a video latent $z_V$ (perturbed with noise at timestep $t$ to $z_{V,t}$) and a clean reference image latent $z_{\text{ref}}$, the VDM backbone $\Phi$ extracts spatiotemporal features:

$f_V = \Phi(z_{V,t}, t, c_{\text{txt}}) \quad \text{and} \quad f_{\text{ref}} = \Phi(z_{\text{ref}}, t_0, c_{\text{txt}})$

where $f_V$ is the video's spatiotemporal feature representation, $f_{\text{ref}}$ is the reference image's identity feature representation, $t$ is the noise timestep applied to the video, $t_0$ is a fixed clean-image timestep (near 0, meaning minimal noise, since the reference should be evaluated in its clean state), and $c_{\text{txt}}$ is the text prompt. The reference image is processed at a low noise level because we want its high-fidelity identity features, not a noisy approximation. The video is processed at varying noise levels $t$ because during reward feedback learning (LIReFL), the LIRM must evaluate noisy intermediate latents from the generation process.

Module 2: Identity Cross-Attention Layer. The key design choice is how to measure alignment between the reference identity and the video content. The paper uses an asymmetric attention mechanism where the reference features serve as queries and the video features serve as keys and values:

$h_{\text{attn}} = \text{Attention}(Q, K, V) = \text{Softmax}\left(\frac{QK^\top}{\sqrt{d}}\right)V$

where $Q = f_{\text{ref}}W_Q$ are the query projections of the reference features, and $K = f_V W_K$, $V = f_V W_V$ are the key and value projections of the video features. $W_Q$, $W_K$, $W_V$ are learnable projection matrices, and $d$ is the feature dimension.

What this computes: The reference features $f_{\text{ref}}$ ask the question: "where in the video do features similar to mine appear?" The dot product $QK^\top$ computes similarity between every reference feature position and every video feature position (across both spatial and temporal dimensions). The softmax normalizes these similarities into attention weights. The output $h_{\text{attn}}$ is a weighted sum of video features, where weights are determined by how well each video position matches the reference identity. If the video contains a subject that looks like the reference, those spatial-temporal positions will have high attention weights, and the resulting feature vector will strongly represent "identity-preserving content." If the video subjects don't match the reference, alignment scores will be low and $h_{\text{attn}}$ will be a diffuse, uninformative representation.

Module 3: Reward Prediction Head. A residual connection fuses the aligned features with the original query, and a lightweight MLP head $H$ predicts the scalar reward:

$r_t = H(h_{\text{attn}} + Q)$

The residual connection $h_{\text{attn}} + Q$ is critical: it preserves the original identity query information while augmenting it with the video-aligned features. If the video contains strong identity matches, $h_{\text{attn}}$ will reinforce and amplify the query signal; if identity matches are weak, the $+Q$ connection ensures the reward prediction still receives the reference identity information as a fallback, preventing total collapse when the video subject differs from the reference.

Why reference-as-query: The paper's ablation (Table VIII, "w/ Ref. Image as KV") shows that reversing this — using video features as queries and reference features as keys/values — causes a catastrophic accuracy drop (average pairwise classification accuracy drops from 0.720 to 0.455). This is because the query defines "what to look for." By making the reference the query, the model explicitly searches the video for identity matches — each reference feature position actively probes the video for corresponding features. Making the video the query would instead ask "what does this video region match in the reference," which is less discriminative because video regions contain background, other subjects, and motion blur that would all query the reference and produce noisy alignments.

LIRM Training: Latent Identity Preference Optimization

The LIRM is trained on a curated preference dataset $\mathcal{D}_{\text{LIRM}} = \{(V, I_{\text{ref}}, c_{\text{txt}}, y)_i\}_{i=1}^N$ with approximately 27,500 training videos and 500 test videos. Each sample consists of a video win-lose pair, corresponding single- or multi-subject reference images, a text prompt, and a human-annotated label $y \in \{0, 1\}$ indicating whether the video $V$ preserves the identity defined by $I_{\text{ref}}$ (1 = identity preserved, 0 = identity not preserved). The LIRM parameters are optimized via binary cross-entropy:

$\mathcal{L}_{\text{LIRM}} = -\mathbb{E}_{\mathcal{D}_{\text{LIRM}}} \left[ y \log \sigma(r_t) + (1 - y) \log(1 - \sigma(r_t)) \right]$

where $\sigma(\cdot)$ is the sigmoid activation, $r_t$ is the predicted reward for the video-reference pair at noise timestep $t$, and $\mathbb{E}_{\mathcal{D}_{\text{LIRM}}}$ denotes the expectation over the training dataset.

What it computes: For each training sample, the LIRM predicts a scalar reward $r_t$, which is mapped through the sigmoid to a probability $\sigma(r_t) \in [0, 1]$ representing the model's confidence that the video preserves the reference identity. The binary cross-entropy compares this predicted probability to the human label $y$. If the human labeled the video as identity-preserving ($y=1$), the loss is $-\log \sigma(r_t)$, which heavily penalizes predictions near 0 and rewards predictions near 1. If human-labeled as not identity-preserving ($y=0$), the loss is $-\log(1 - \sigma(r_t))$, penalizing predictions near 1. The expectation means this loss is averaged over the training dataset.

Why this form: The binary cross-entropy is the maximum-likelihood objective for a Bernoulli-distributed target, which is appropriate because the labels are binary human judgments. The Bradley-Terry (BT) model — an alternative ranking loss used in RLHF that models pairwise preferences as $P(V_1 \succ V_2) = \sigma(r_1 - r_2)$ — was tested but found inferior (Table VIII: BT loss achieves 0.656 average accuracy vs. BCE's 0.720). The paper notes that BT loss "exhibits significant instability at early timesteps ($t \in [0, 0.2]$)" — at very low noise levels, the video is nearly clean and identity differences are subtle, making pairwise comparisons noisy. BCE loss, which learns absolute identity quality rather than relative ranking, is more robust to this noise because each video is evaluated independently against the human label rather than in comparison to another video.

Parameter tuning scope: The VDM backbone, identity cross-attention layer, and reward prediction head are jointly updated during LIRM training, but the text and patch embedding layers of the pretrained VDM are frozen. The ablation in Table VIII shows that fine-tuning text and patch embeddings slightly decreases accuracy (0.720 vs. 0.715), suggesting that the pre-trained text-visual priors are sufficient for identity evaluation and should not be disturbed. The paper uses differential learning rates: $1 \times 10^{-5}$ for the prediction head and attention layer (newly added, random-initialized components that need to adapt quickly), and $1 \times 10^{-6}$ for the VDM backbone (pre-trained, should be fine-tuned cautiously to preserve spatiotemporal priors).

LIRM is trained for approximately 4,000 steps on 16 A100 GPUs with a batch size of 16 and weight decay of $1 \times 10^{-2}$.

Latent Identity Reward Feedback Learning (LIReFL)

The LIRM, once trained and frozen, is used to provide identity-preserving gradients to the video generator during Stage 2. The key innovation over prior reward feedback learning (ReFL) methods is that LIReFL operates entirely in latent space, avoiding VAE decoding.

The computational bottleneck of standard ReFL: Prior identity reward methods (Identity-GRPO, IPRO) require decoding the generated latent $z_0$ to pixel space via the VAE decoder, computing a reward (e.g., CLIP similarity, face recognition score) in RGB space, and backpropagating gradients through the decoder to update the generator. The VAE decoder is computationally expensive for video (decoding $T \times H \times W$ latents to $T \times H_{\text{pixel}} \times W_{\text{pixel}} \times 3$ frames). This forces prior methods to restrict reward feedback to only the final 1–3 denoising steps, as computing rewards at earlier, higher-noise steps would require many VAE forward passes through the entire denoising trajectory.

LIReFL's solution: Because the LIRM operates in latent space (takes $z_t$ as input, not RGB frames), no VAE decoding is needed. This enables dense reward feedback at arbitrary diffusion timesteps with minimal computational overhead.

LIReFL Procedure:

1. Initialize latents from Gaussian noise: $z_T \sim \mathcal{N}(0, I)$
2. Sample a target intermediate timestep: $t_m \sim \mathcal{U}(0, T-1)$
3. Perform standard gradient-free denoising from step $T$ down to $t_m + 1$ to conserve memory (no computation graph stored).
4. At step $t_m + 1$, execute a single gradient-enabled denoising step to derive the predicted latent:

$z_{t_m} = \mu_\theta(z_{t_m+1}, t_m+1, c_{\text{txt}}, z_{\text{ref}})$

where $\mu_\theta$ is the single-step solver function (UniPC) parameterized by the video generator $\epsilon_\theta$. This denoising step is gradient-enabled — the computation graph is retained so that gradients can flow from $z_{t_m}$ back to the generator parameters $\theta$.

5. Evaluate the predicted latent $z_{t_m}$ using the frozen LIRM to predict the identity reward:

$r_{t_m} = \text{LIRM}(z_{t_m}, t_m, c_{\text{txt}}, z_{\text{ref}})$

6. Formulate the reinforcement loss to maximize expected identity fidelity:

$\mathcal{L}_{\text{LIReFL}} = -\mathbb{E}_{t_m, c_{\text{txt}}, z_{\text{ref}}}[r_{t_m}]$

The negation means minimizing $\mathcal{L}_{\text{LIReFL}}$ maximizes the average predicted reward, pushing the generator to produce latents that the LIRM judges as identity-preserving.

What happens computationally: At step $t_m$, the gradient $\partial r_{t_m} / \partial \theta$ is computed via backpropagation through the LIRM (frozen, but its activations are needed for chain rule) and then through the single UniPC denoising step back to the generator parameters. The LIRM provides a signal that says "make the latent at timestep $t_m$ more identity-consistent," and the generator learns to adjust its denoising function $\epsilon_\theta$ to produce latents that score higher under the LIRM.

Why this works across all timesteps: The paper ablates the timestep range $t_m$ in Table IX, comparing "Last 3 timesteps" (sparse feedback near the clean output) versus "All timesteps" (dense feedback throughout the denoising trajectory). In multi-subject mode, All timesteps improves R-DINO from 0.518 to 0.524, Face-S from 0.324 to 0.329, and EPE from 6.30 to 6.08. The paper argues that early denoising steps establish structural layout and coarse identity features — providing reward feedback at these early steps guides the generator to set up identity-correct spatial arrangements before fine details are filled in. Sparse feedback only at the final steps can only correct minor texture-level errors and cannot recover from fundamentally wrong identity structures established earlier.

Regularization against reward hacking: The reinforcement loss, if used alone, would encourage the generator to produce outputs that maximize the LIRM's score by any means necessary — including generating degenerate, unrealistic videos that happen to trigger high reward predictions (e.g., static close-ups of the reference subject with no motion). To prevent this, the Stage 1 SFT loss is incorporated as a regularizer:

$\mathcal{L} = \mathcal{L}_{\text{sft}} + \lambda_2 \mathcal{L}_{\text{LIReFL}}$

where $\lambda_2 = 0.10$ controls the strength of reward feedback relative to the reconstruction objective. The SFT loss $\mathcal{L}_{\text{sft}}$ anchors the generator to realistic video generation (motion, scene structure, visual quality), while the LIReFL loss $\mathcal{L}_{\text{LIReFL}}$ nudges it toward better identity preservation within that realistic regime.

The $\lambda_2$ trade-off: Table X ablates $\lambda_2$ from 0.01 to 1.00. At $\lambda_2 = 0.01$, identity improvements are modest (R-DINO 0.505 in single-subject mode vs. SFT-only's 0.487 in Table VII). At $\lambda_2 = 0.25$, identity metrics are comparable but slightly lower (R-DINO 0.492, Face-S 0.272). At $\lambda_2 = 1.00$, the model suffers from reward hacking: R-DINO collapses to 0.350, Face-S to 0.120, mIoU to 0.350, and EPE explodes to 25.00 in single-subject mode. This is a classic over-optimization failure — the generator finds latent-space patterns that maximize LIRM scores but produce nonsensical videos when decoded. The paper identifies $\lambda_2 = 0.10$ as optimal: strong enough to provide meaningful identity improvements, weak enough to prevent reward hacking.

LIReFL training runs for 3,400 steps on 16 A100 GPUs with batch size 16, learning rate $5 \times 10^{-6}$, and the same condition dropping and reference augmentation protocols as Stage 1.

---

Dataset Construction Pipeline

The Stage 1 training demands precise alignment across four conditioning dimensions (subject identity, global motion, local motion, camera movement), which no existing dataset provides. The paper constructs a large-scale dataset (~2.12M video clips) through an automated four-stage pipeline, summarized in Figure 3.

Stage 1: Motion-Based Filtering. Dense optical flow is estimated using RAFT, and the average motion magnitude across frames is computed. Videos with small motion magnitude are discarded, ensuring the dataset focuses on meaningful motion patterns. This is necessary because static or near-static videos would teach the model that identity preservation means "produce minimal motion" — the exact failure mode the paper is trying to avoid.

Stage 2: Subject Discovery and Captioning. The open-vocabulary tagging model RAM++ extracts semantic tags from each video (e.g., "girl," "dog," "car"). Qwen3 Max, a large language model, refines these tags by retaining only significant moving subjects — filtering out static background objects that shouldn't receive motion control signals. Qwen3-VL, a vision-language model, generates detailed captions for each video describing the scene, subjects, actions, and context.

Stage 3: Spatiotemporal Annotation Extraction. For each moving subject:

• Grounding DINO detects bounding boxes in each frame.
• SAM 2 takes the detected bounding boxes as prompts and produces precise binary segmentation masks.
• CoTracker3 performs dense point tracking across frames.
• Tracked points are classified: points falling within subject masks are labeled as object trajectories (encoding subject-local motion); points in the background are labeled as camera trajectories (encoding global frame shifts).

This yields per-subject, per-frame annotations: bounding boxes, segmentation masks, object trajectories, and camera trajectories. The use of Grounding DINO → SAM 2 as a detection-to-segmentation cascade is a standard open-vocabulary segmentation approach; the innovation is in combining it with CoTracker3 to produce dense motion annotations automatically at scale.

Stage 4: Reference Image Construction. Reference images are sampled from frames temporally disjoint from the training clip — if a 49-frame clip is used for training, the reference comes from a different part of the source video (or a different video of the same subject). The subject is isolated via its segmentation mask (removing background). Extensive data augmentation (flipping, rotation, shearing, cropping, blur, color jitter) is applied to produce the final reference image.

Comparison with prior datasets: Table I shows that no existing dataset for video customization or controllable generation simultaneously provides reference images, multi-subject support, all-frame masks, all-frame bounding boxes, and all-frame trajectories. WebVid-10M (10M videos) has none of these annotations. Phantom's dataset (~1M videos) has reference images and multi-subject support but lacks masks, boxes, and trajectories. Wan-Move's dataset (~1.98M videos) has only trajectories, with no reference images or boxes. The paper's dataset is the first to combine all five annotation types at scale.

---

DreamOmni Bench

Existing benchmarks cannot evaluate DreamVideo-Omni's holistic capabilities because they either isolate customization from motion control or focus on narrow metrics (e.g., point trajectory precision without identity preservation). The paper constructs DreamOmni Bench to fill this gap.

Curation process: High-quality real-world videos are sourced independently of the training dataset to ensure zero-shot evaluation. Manual filtering retains high-resolution videos exhibiting meaningful subject motion and camera movement, explicitly excluding static videos and frames with text overlays or watermarks. The automated pipeline from Section III-C generates dense captions and comprehensive annotations for each retained video. The final benchmark contains 1,027 videos: 436 single-subject and 591 multi-subject samples, spanning humans, general objects, animals, and faces.

Evaluation dimensions and metrics:

1. Subject and Face Fidelity (Identity Preservation):

   • Region CLIP-Image similarity (R-CLIP): GroundingDINO detects and crops subject regions in generated frames based on textual tags. CLIP ViT-B/32 computes cosine similarity between cropped regions and reference images. The "region" qualifier means background and other subjects are excluded — this prevents high CLIP scores from being dominated by background similarity rather than subject identity.
   • Region DINO-Image similarity (R-DINO): Same cropping procedure, but using DINO-ViT-S/16 features. DINO is known to capture finer structural details than CLIP, making it complementary for identity evaluation.
   • Face Similarity (Face-S): ArcFace embeddings from the InsightFace library extract face identity vectors from generated frames and reference faces. For multi-person scenarios, all faces are detected, and the generated face with the highest cosine similarity to the reference is matched for evaluation — this handles cases where multiple faces appear but only some correspond to the reference identity.

2. Motion Control Precision:

   • Mean Intersection over Union (mIoU) for bounding boxes: GroundingDINO detects subjects in generated videos, and IoU is computed between detected boxes and ground-truth control boxes. This measures how precisely the model's spatial layout matches the specified bounding box constraints.
   • End Point Error (EPE) for trajectories: Query points are initialized using ground-truth coordinates from the first frame. CoTracker3 tracks these points in the generated video, and EPE computes the average Euclidean distance between tracked and ground-truth trajectories. Lower EPE means more precise trajectory following.

3. Overall Consistency:

   • CLIP-Text similarity (CLIP-T): Cosine similarity between CLIP embeddings of the generated video frames and the text prompt, measuring semantic alignment.

Why these metrics: R-CLIP and R-DINO avoid the pitfall of whole-image similarity metrics, which would be confounded by background content and other subjects in multi-subject scenarios. By cropping to subject regions first, the metrics specifically evaluate identity consistency. Face-S provides a dedicated biometric face identity measure that is more sensitive to fine facial details than general visual features. mIoU and EPE provide complementary spatial control metrics: mIoU captures coarse spatial layout (are subjects in the right place?), while EPE captures fine trajectory precision (do specific points follow the right path?). CLIP-T ensures that motion control and identity preservation don't come at the cost of ignoring the text prompt.

Training and Inference Hyperparameters

Stage 1 (Omni-Motion and Identity SFT):

• Backbone: Wan2.1-1.3B T2V (pre-trained text-to-video DiT)
• Resolution: 480 × 832 pixels
• Frames: 49 per clip
• Hardware: 64 NVIDIA A100 GPUs
• Total batch size: 64
• Iterations: 40,000
• Optimizer: AdamW
• Learning rate: $5 \times 10^{-5}$
• Weight decay: $1 \times 10^{-3}$
• Condition dropping probability: $p = 0.5$ for both bounding box and trajectory conditions (applied independently)
• Reference augmentation probability: $p = 0.5$
• Reweighted diffusion loss weight: $\lambda_1 = 2$

Stage 2 (LIRM Training):

• Backbone: First 8 layers of Wan2.1-1.3B
• Hardware: 16 A100 GPUs
• Batch size: 16
• Weight decay: $1 \times 10^{-2}$
• Iterations: ~4,000
• Differential learning rates: $1 \times 10^{-5}$ (prediction head and attention layer), $1 \times 10^{-6}$ (VDM backbone)
• Frozen components: text and patch embedding layers of the pretrained VDM
• Training dataset: ~27,500 win-lose video pairs

Stage 2 (LIReFL):

• Initialization: Stage 1 SFT checkpoint
• Hardware: 16 A100 GPUs
• Batch size: 16
• Weight decay: $1 \times 10^{-2}$
• Iterations: 3,400
• Learning rate: $5 \times 10^{-6}$
• Reward model: frozen LIRM from previous substage
• Regularization weight: $\lambda_2 = 0.10$
• Condition dropping and reference augmentation: same as Stage 1

Inference:

• Scheduler: UniPC with 50 steps
• Classifier-free guidance scale: 5.0

Design rationale for key hyperparameters:

• 49 frames: A compromise between temporal coverage (long enough to show meaningful motion) and computational cost (attention complexity is quadratic in sequence length, and 49 frames × spatial tokens is already substantial for a 1.3B model).
• Condition dropping $p=0.5$: Training with conditions always present would make the model reliant on perfect conditioning at inference time; dropping conditions 50% of the time forces the model to learn robust generation capabilities that can operate with partial or missing control signals, enabling flexible compositional use where not all control types are always specified.
• Differential LIRM learning rates: The prediction head and cross-attention layer are randomly initialized and need substantial adaptation; the VDM backbone is pre-trained and should only be fine-tuned gently to avoid catastrophic forgetting of spatiotemporal priors that are essential for motion-aware identity evaluation.
• $\lambda_2 = 0.10$: Table X empirically determines this as the point where identity reward feedback provides meaningful gains without triggering reward hacking. The narrow effective range $[0.01, 0.25]$ suggests that the reward signal must be carefully balanced against the SFT anchor.
• UniPC scheduler at 50 steps: UniPC is a unified predictor-corrector that achieves high-quality generation with fewer steps than DDPM/DDIM, balancing inference speed and quality. The guidance scale of 5.0 is standard for classifier-free guided diffusion with conditioning signals.

4. Key Insights and Innovations

Innovation 1: The Identity-Motion Conflict Is Not a Trade-off to Balance, but a Loss Function Misalignment to Correct

The most conceptually significant move in this paper is its diagnosis of why identity preservation and motion control degrade each other in existing methods. The standard narrative in the field treats this as an inherent, unavoidable tension—you can have precise motion or faithful identity, but improving one necessarily sacrifices the other. Prior unified methods (DreamVideo-2, FullDiT, OmniVCUS, Tora2) accept this framing and attempt to find an acceptable operating point on the Pareto frontier, trading identity fidelity for motion precision and vice versa.

The paper rejects this framing entirely. It argues that the conflict is not fundamental to the task but is an artifact of the optimization objective. Standard diffusion reconstruction losses—the L2 distance between predicted and ground-truth noise—penalize any pixel change, regardless of whether that change represents identity degradation or legitimate motion-induced appearance variation. When a subject turns their head, the face pixels change. The L2 loss cannot distinguish between "the face changed because the subject moved" (desired) and "the face changed because identity was lost" (undesired). It treats both as errors to minimize, which incentivizes the model to suppress motion—to keep pixels as close to the reference as possible—or to allow motion while letting identity detail erode into generic approximations that minimize per-pixel mismatch.

The paper's reframing is that this is a supervision misalignment, not a capacity trade-off. The solution is not architectural (which is why Stage 1 alone is insufficient) but rather a change to what the model is rewarded for. By introducing a reward model trained on human identity judgments (the LIRM, trained on human-annotated win-lose pairs in Section III-B1), the optimization signal shifts from "minimize pixel distance to ground truth" to "produce videos that humans recognize as preserving identity." These are correlated but fundamentally different objectives. A video can have high per-pixel error (because the subject moved) and still earn high identity reward (because the subject still looks like themselves). Conversely, a video can have low per-pixel error (static, close to the reference) and earn low identity reward (because the model copied the reference onto a static body, creating a "copy-paste" artifact rather than a naturally moving subject).

This is a fundamental conceptual shift, not a minor refinement. It reframes the identity-motion problem from an architectural challenge (how to design a model that can juggle two competing demands) to an optimization challenge (how to define a loss surface where identity and motion are not in opposition). The evidence for this diagnosis is implicit throughout the paper: Stage 1 alone, despite its architectural sophistication (hierarchical injection, group/role embeddings, condition-aware 3D RoPE), achieves only modest identity fidelity (R-DINO 0.483 in single-subject, Table VII). Stage 2 LIReFL, which changes only the optimization signal without modifying the architecture, lifts R-DINO to 0.499 and Face-S from 0.251 to 0.301. These gains come purely from realigning the training objective with perceptual identity, confirming that the Stage 1 architecture was already capable of better identity preservation—it simply wasn't being trained to prioritize it.

The practical implication is significant: this suggests that for any generative task where the desired output property (identity, style, composition) conflicts with pixel-level reconstruction (motion, viewpoint change, lighting variation), the bottleneck may not be model capacity but the alignment between the loss function and human intent. The latent-space reward modeling approach demonstrated here—training a specialized evaluator on human judgments and using its gradients as a training signal—provides a template for addressing such conflicts beyond video customization.

---

Innovation 2: Latent-Space Reward Feedback Enables Dense, Compute-Efficient Identity Optimization Across the Full Denoising Trajectory

Prior work on identity-focused reinforcement learning for video generation (MagicID, Identity-GRPO, IPRO) shares the intuition that human-aligned reward signals can improve identity preservation. However, these methods all operate on decoded RGB frames: the diffusion model generates a latent, the VAE decoder converts it to pixels, and a reward model (CLIP, DINO, ArcFace) scores the visual output. This pipeline has a critical bottleneck, which the paper is the first to clearly articulate and address: VAE decoding is so expensive for video that it forces reward feedback to be restricted to the final 1–3 denoising steps, where the latent is nearly clean and decoding produces a meaningful image.

This restriction matters because it means the reward signal can only correct surface-level errors. By the final few denoising steps, the structural layout, coarse spatial arrangement, and broad identity features of the video are already determined. A reward computed only at this late stage can encourage better texture details, sharper facial features, or more accurate clothing colors, but it cannot fix fundamentally wrong identity structures—an incorrect face shape, a misattributed identity, a subject placed in the wrong spatial position—because these were established in earlier denoising steps when the latent was too noisy to decode meaningfully into pixels.

The paper's key advance is the realization that by training the reward model to operate directly in VAE latent space (the LIRM takes noisy latents $z_t$ as input, not decoded frames), the VAE decoder can be completely bypassed during reward feedback learning. This is not a minor implementation detail—it is what makes dense reward feedback across the full denoising trajectory computationally feasible. LIReFL samples a random target timestep $t_m \sim \mathcal{U}(0, T-1)$ and computes the reward on the predicted latent at that timestep (Equation 7). During training, this means the model receives identity-aligned gradients not just at the final cleanup steps but at early steps (high noise, when structural layout is being determined), middle steps (when coarse identity features emerge), and late steps (when details are refined).

The ablation in Table IX quantifies the impact: restricting $t_m$ to only the last 3 timesteps (sparse feedback, emulating the constraint of prior pixel-space methods) reduces R-DINO from 0.524 to 0.518 and Face-S from 0.329 to 0.324 in multi-subject mode. The EPE also degrades slightly (6.08 → 6.30). These gains from dense feedback may appear modest in absolute terms, but they demonstrate a principle with broader implications: the timing of feedback in a generative process matters as much as its content. Early feedback shapes structure; late feedback polishes details. A reward model that can operate in the generative model's native representation space (latents, not pixels) can provide feedback at the moment it is most useful, not just at the moment it is most convenient.

This has implications beyond video customization. For any task where reward signals are used to fine-tune diffusion models—image generation with human preference alignment, 3D asset generation, molecular design—the ability to provide feedback at arbitrary noise levels, rather than only on clean outputs, could enable more effective optimization by guiding the entire generation trajectory rather than only its endpoint. The LIRM design (VDM backbone + identity cross-attention + reward head) provides a concrete architectural template for building such latent-space evaluators, and the paper's finding that this works even with only 8 layers of the VDM backbone (~4,000 training steps on 27,500 samples) suggests the approach is practical at modest scale.

---

Innovation 3: Explicit Identity-Motion Binding via Group and Role Embeddings Transforms Multi-Subject Control from an Association Learning Problem into a Routing Problem

In multi-subject video generation, a model receives multiple reference images and multiple motion signals (bounding boxes, trajectories) but must determine which motion applies to which subject. Prior work approaches this as an association learning problem: the model is expected to learn, through co-occurrence statistics in the training data, that a particular motion pattern tends to accompany a particular appearance pattern. If subject A is always shown in a red shirt and subject B in a blue shirt, the model might learn that red-shirt trajectories and blue-shirt trajectories have different statistical signatures. But when both subjects wear similar clothing, or when reference images come from different contexts than the training distribution, this implicit learning breaks down.

The paper's group embedding mechanism transforms this from an association learning problem into a deterministic routing problem. Each control unit (reference subject + its bounding box + its trajectory) receives a unique, learnable embedding vector $g_i$ that is added to all tokens belonging to that unit. This means that at the token level, the model does not need to infer which trajectory belongs to which subject from co-occurrence patterns—it can read the group embedding directly. Subject 1's trajectory tokens carry embedding $g_1$; subject 2's trajectory tokens carry embedding $g_2$. The attention mechanism can learn to route tokens with matching group embeddings to each other regardless of their visual content.

This is a fundamentally different approach to the binding problem, and it explains why DreamVideo-Omni scales to multi-subject scenarios without the identity confusion visible in baselines (Figure 6, right panel: VACE and Phantom produce subjects with incorrect skin tones, hairstyles, and clothing when handling a man and woman interacting in a kitchen, while DreamVideo-Omni maintains distinct, correct identities). The role embeddings add a second axis: an object embedding marks tokens as "appearance information" while a control embedding marks tokens as "motion guidance." This ensures that even within a single control unit, the model distinguishes what the subject looks like from how the subject should move.

The ablation in Table VII confirms that this is not a minor refinement. Removing group and role embeddings in single-subject mode degrades mIoU from 0.556 to 0.524 and EPE from 10.53 to 26.24—a substantial loss of motion precision even when there is no ambiguity about which subject the motion applies to (there is only one). This suggests the embeddings serve a function beyond disambiguation: they provide a structural prior that helps the model process motion signals and identity signals through distinct, well-separated pathways, improving each independently. The fact that motion precision degrades when embeddings are removed, even in single-subject mode, indicates that the explicit "this token is motion" labeling is useful for more than just multi-subject binding—it helps the model's internal representations separate spatial control from appearance generation, a factorization that proves beneficial regardless of scene complexity.

This is a conceptual advance with applicability beyond video generation. Any multi-modal generative system that processes heterogeneous conditioning signals (text + image + spatial constraints + temporal dynamics) faces a binding problem: which pieces of information relate to which others? The group/role embedding approach demonstrates that explicit, learnable identifiers attached at the token level can be more effective than relying on the model to infer relationships from content alone. This is analogous to positional encodings in transformers (which provide explicit position information rather than expecting the model to infer sequence order from content), but generalized to arbitrary conditioning types.

---

Innovation 4: Camera Movement Can Be Unified with Local Object Motion Under a Single Trajectory Conditioning Framework, Eliminating the Need for Explicit 3D Camera Parameters

Prior work on camera-controlled video generation typically relies on explicit 3D camera parameters—extrinsic matrices (rotation and translation), intrinsic matrices (focal length, principal point), or derived representations like plücker coordinates or ray directions (MotionCtrl, CamI2V, FullDiT). These require either synthetic datasets with known camera parameters (which limits domain coverage) or structure-from-motion pipelines to estimate parameters from real videos (which introduces noise and failure cases). They also require the user to specify camera motion in parametric form—pan by X degrees, translate by Y meters—which is unintuitive for non-expert users.

The paper makes a counterintuitive observation: camera movement, when applied to background pixels, is mathematically equivalent to point-wise trajectory motion. A camera pan to the right shifts all background pixels leftward; a zoom shifts background pixels radially outward from the center. By sampling point trajectories on background pixels and conditioning the model on these trajectories, the same mechanism that controls a subject's limb movements can control the camera's global motion through the scene. This observation allows the paper to unify camera control and local object control under a single trajectory conditioning token type (Section III-A2, point 4), processed identically by the DiT.

This is not a deep theoretical contribution, but it is a significant practical insight that simplifies both training and deployment. For training, it eliminates the need for auxiliary camera parameter datasets or estimation pipelines—the automated annotation pipeline (RAFT → RAM++ → Grounding DINO → SAM 2 → CoTracker3) already produces trajectories for both foreground and background points, so camera control comes "for free" with the same data processing infrastructure. For deployment, it means users can specify camera movement by drawing a few trajectory lines on the background of a reference frame, rather than entering numerical camera parameters. The paper demonstrates that this works in practice through the emergent image-to-video and first-frame-conditioned trajectory control capabilities (Figure 8), where camera-relative motion is achieved without any explicit camera model.

The significance extends beyond convenience. By reducing all motion control—global object placement (boxes), local object dynamics (foreground trajectories), and camera movement (background trajectories)—to a common token representation, the architecture avoids the complexity of separate processing pathways for different motion types. The condition-aware 3D RoPE distinguishes foreground from background trajectories via the shared temporal index $t \in [0, T-1]$, while their spatial coordinates and group embeddings differentiate their roles. This unification is elegant in the sense that it achieves three distinct control modalities through a single mechanism, but it is also functionally important: it means the model's capacity is not partitioned across modality-specific modules, allowing learned representations of motion to generalize across control types. A trajectory is a trajectory, whether it describes a waving hand or a panning camera.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary evaluation uses DreamOmni Bench, a newly constructed benchmark of 1,027 high-quality, real-world videos sourced independently of the training dataset (ensuring zero-shot evaluation). It is explicitly categorized into 436 single-subject and 591 multi-subject samples, spanning humans, general objects, animals, and faces. Each video is densely annotated with subject masks, bounding boxes, trajectories, and detailed captions using the automated pipeline described in Section III-C. A secondary evaluation uses MSRVTT-Personalization Bench, following the experimental settings of Tora2 and Video Alchemist.

Base model. All experiments use Wan2.1-1.3B T2V as the foundational model, a pre-trained text-to-video DiT. The authors choose this model for its public availability (open-source), representative 1.3B scale, and strong pre-trained spatiotemporal priors. The paper argues this model is "representative of the capabilities of many contemporary" video generation models and sits at a scale that makes the computational requirements practical for academic research while still demonstrating the claimed capabilities.

Metrics. Six metrics across three dimensions quantify performance:

• Subject and Face Fidelity: R-CLIP (Region CLIP-Image similarity using CLIP ViT-B/32 on cropped subject regions), R-DINO (Region DINO-Image similarity using DINO-ViT-S/16 on the same cropped regions), and Face-S (ArcFace cosine similarity between detected faces in generated frames and reference faces, matching the highest-similarity face in multi-person scenarios). Cropping to subject regions before computing similarity prevents background and other subjects from confounding the identity measurement.
• Motion Control Precision: mIoU (Mean Intersection over Union between GroundingDINO-detected boxes in generated videos and ground-truth control boxes) for bounding box accuracy, and EPE (End Point Error, the average Euclidean distance between CoTracker3-tracked points and ground-truth trajectories) for trajectory precision.
• Overall Consistency: CLIP-T (CLIP-Text similarity using CLIP ViT-B/32) for semantic alignment between generated videos and text prompts.

Baselines. The paper benchmarks against methods from three categories, chosen because no single prior method simultaneously supports multi-subject customization with omni-motion control:

• Joint customization and motion control: DreamVideo-2, limited to single-subject customization with bounding box control only.
• Pure subject customization: VACE and Phantom (both 1.3B parameter models), evaluated in both single-subject and multi-subject modes.
• Pure motion control: Tora (T2V, 1.1B) and Wan-Move (I2V, 14B), evaluated on trajectory and bounding box following precision.
• On MSRVTT-Personalization Bench, additional comparisons to Video Alchemist and Tora2 with their reported results.

Generation budget / compute accounting. All models are compared at equivalent input specifications (same reference images, same motion signals, same text prompts). The paper reports that inference uses the UniPC scheduler with 50 steps and a classifier-free guidance scale of 5.0. For the FLOPs-matched comparisons, parameter counts serve as a proxy for compute (1.3B vs. 14B for Wan-Move, and comparisons against other 1.3B models). The paper does not explicitly measure or report total FLOPs per generation.

Cross-validation / statistical protocol. Evaluation is zero-shot: the DreamOmni Bench videos are sourced independently of the training dataset, and no test-time fine-tuning is performed. The paper does not report confidence intervals, error bars, or multiple random seeds. The user study (Table VI) involves 18 evaluators rating 270 groups of videos, providing a perceptual complement to automated metrics, but no inter-annotator agreement statistics (e.g., Fleiss' kappa) are reported.

Main Quantitative Results

Subject Customization with Omni-Motion Control

The headline result is that DreamVideo-Omni achieves unified customization and motion control with substantially better identity preservation and motion precision than prior methods, even those using much larger models.

On DreamOmni Bench (Table II), compared to DreamVideo-2—the only baseline supporting joint customization and motion control, though limited to single-subject and bounding-box-only settings—DreamVideo-Omni improves substantially across all metrics:

• Identity fidelity: R-DINO increases from 0.429 to 0.499 (a 16.3% relative improvement), while Face-S more than doubles from 0.157 to 0.301.
• Motion control precision: mIoU nearly triples from 0.212 to 0.558, while EPE drops from 24.05 to 9.31 (a 61.3% reduction in trajectory error).
• Text alignment: CLIP-T increases modestly from 0.297 to 0.308.

On MSRVTT-Personalization Bench (Table III), DreamVideo-Omni demonstrates strong generalization. In Subject Mode, it achieves the highest R-DINO (0.628) and best EPE (11.21), outperforming Tora2 (R-DINO 0.615, EPE 17.43) and Video Alchemist (R-DINO 0.626, EPE not reported). In Face Mode, DreamVideo-Omni achieves Face-S 0.417 (comparable to Tora2's 0.419 on this lower-quality dataset) and dramatically better EPE (8.50 vs. Tora2's 13.52, a 37.1% reduction). The paper notes that comparable Face-S scores on MSRVTT are partly attributable to "the limited video quality and resolution of the MSRVTT dataset."

The qualitative comparison in Figure 5 illustrates these differences concretely. In the left example ("person manipulating a transparent crystal ball"), DreamVideo-2 produces a subject that drifts from the reference identity under the motion, while DreamVideo-Omni maintains facial structure and clothing fidelity. In the right example ("dog running on a grassy field"), DreamVideo-2 exhibits trajectory drift or static motion—the dog fails to follow the specified path—while DreamVideo-Omni precisely tracks the complex spatial trajectory.

Subject Customization (Pure, Without Motion Control Requirements)

To isolate identity preservation capability, the paper evaluates pure subject customization (generating videos from reference images and text prompts, without explicit motion conditioning) against VACE and Phantom. All models have 1.3B parameters.

On DreamOmni Bench (Table IV), in single-subject mode, DreamVideo-Omni achieves R-CLIP 0.739, R-DINO 0.499, and Face-S 0.301—marginally ahead of Phantom (R-CLIP 0.738, R-DINO 0.485, Face-S 0.299) and clearly ahead of VACE (R-CLIP 0.732, R-DINO 0.480, Face-S 0.174). The CLIP-T score of 0.308 also leads, suggesting better text alignment. In the more challenging multi-subject mode, DreamVideo-Omni achieves R-DINO 0.524 (vs. Phantom's 0.517 and VACE's 0.497), Face-S 0.329 (vs. Phantom's 0.305 and VACE's 0.275), and CLIP-T 0.306 (vs. 0.293 for both baselines). R-CLIP is comparable at 0.720 (Phantom 0.722, VACE 0.719).

The qualitative results in Figure 6 make the practical significance of these gaps clear. In the single-subject example (woman revealing face from behind a leaf), VACE exhibits suboptimal facial preservation and unnatural motion, while Phantom generates unexpected multiple faces—a failure mode where the model hallucinates additional subjects. DreamVideo-Omni produces a natural transition with faithful facial preservation. In the multi-subject kitchen scene, both VACE and Phantom fail to preserve identity fidelity, producing subjects with incorrect skin tones, hairstyles, facial details, and clothing colors relative to the references. DreamVideo-Omni "successfully disentangles the two identities, rendering distinct, correct appearances with coherent interactions."

Motion Control (Pure, Without Customization Requirements)

To isolate motion control capability, DreamVideo-Omni is evaluated against Tora (T2V, 1.1B) and Wan-Move (I2V, 14B)—the latter notably having approximately 10.8× more parameters. All motion control evaluation on DreamOmni Bench includes both bounding box and trajectory condition specification.

On DreamOmni Bench (Table V), in single-subject mode, DreamVideo-Omni achieves mIoU 0.558 and EPE 9.31. This substantially outperforms Tora (mIoU 0.163, EPE 31.74) and Wan-Move (mIoU 0.507, EPE 14.43). The mIoU improvement over Wan-Move is 0.051, and the EPE reduction is 35.5%. In multi-subject mode, DreamVideo-Omni achieves mIoU 0.570 and EPE 6.08, again outperforming Tora (mIoU 0.162, EPE 32.84) by enormous margins and Wan-Move (mIoU 0.541, EPE 9.02). The paper explicitly notes: "despite being a 1.3B parameter model, DreamVideo-Omni surpasses the 14B parameter Wan-Move across all metrics in both settings, demonstrating the significant parameter efficiency and superior controllability of our method."

The qualitative comparison in Figure 7 shows why these metric gaps matter. Tora struggles with trajectory adherence—subjects fail to follow specified paths. Wan-Move generates higher visual quality but deviates from complex trajectories: the soccer ball's path is inaccurate, and the puppy-girl interaction shows loose alignment with hand movements. DreamVideo-Omni "precisely follows complex motion trajectories," with the man's interaction with the soccer ball adhering to an intricate looping path and the girl-puppy duo maintaining distinct, coordinated movements.

User Study

Table VI reports human preference results across 270 evaluation groups rated by 18 evaluators on four criteria: Subject Fidelity, Motion Consistency, Text Alignment, and Overall Quality.

For joint identity and motion, DreamVideo-Omni receives 77.6% of votes for Subject Fidelity (vs. 22.4% for DreamVideo-2), 81.7% for Motion Consistency (vs. 18.3%), 78.5% for Text Alignment (vs. 21.5%), and 89.2% for Overall Quality (vs. 10.8%). For pure subject customization, DreamVideo-Omni receives 64.2% (Subject Fidelity) and 60.3% (Overall Quality), compared to VACE (16.3%, 19.5%) and Phantom (19.5%, 20.2%). For pure motion control, DreamVideo-Omni receives 70.3% (Motion Consistency) and 60.2% (Overall Quality), compared to Wan-Move (20.2%, 26.4%) and Tora (9.5%, 13.4%). The dominance in overall quality across all settings—where evaluators integrate all aspects of video quality—is particularly pronounced, with DreamVideo-Omni receiving over 60% of votes in three-way comparisons and nearly 90% in the two-way comparison against DreamVideo-2.

Ablation Studies and Robustness Checks

All ablation studies are conducted on the DreamOmni Bench, with results reported separately for single-subject and multi-subject modes in Tables VII–X.

Condition-aware 3D RoPE: Removing this component causes catastrophic performance collapse across all metrics (Table VII). In single-subject mode, R-DINO drops from 0.499 to 0.139, Face-S from 0.301 to 0.039, mIoU from 0.558 to 0.274, and EPE degrades from 9.31 to 30.22. In multi-subject mode, R-DINO drops from 0.524 to 0.157, Face-S from 0.329 to 0.047, and mIoU from 0.570 to 0.278. The paper notes this leads to "training collapse, resulting in severe artifacts or meaningless noise" (Figure 9). This is the single most important architectural component—without it, the unified DiT cannot process heterogeneous input types.

Group and role embeddings: Removing these degrades motion control precision primarily (Table VII). In single-subject mode, mIoU drops from 0.556 (Stage 1 only) to 0.524, and EPE increases dramatically from 10.53 to 26.24—a 149% increase in trajectory error even with only one subject present. In multi-subject mode, mIoU drops from 0.532 to 0.459 and EPE from 6.80 to 20.69. Identity metrics also degrade (single-subject Face-S: 0.251 → 0.254 is flat, but R-DINO: 0.483 → 0.486 is comparable). Figure 9 qualitatively shows that without these embeddings, "the model struggles to disentangle multiple subjects or bind specific motions to the correct identities."

Hierarchical bounding box injection: Removing this—i.e., adding bounding box latents only at the input, not at each DiT block—causes motion control to collapse (Table VII). In single-subject mode, mIoU drops from 0.556 to 0.400 and EPE from 10.53 to 31.84. In multi-subject mode, mIoU drops catastrophically from 0.532 to 0.289 and EPE from 6.80 to 25.56. Identity metrics are largely unaffected (single-subject R-DINO: 0.483 → 0.508 even slightly improves), confirming that this component specifically enables effective motion control. Figure 9 shows subjects "fail in adhering to bounding boxes or trajectories."

Stage 1 vs. Stage 2 (LIReFL effect): The full model with LIReFL versus Stage 1 SFT only (Table VII) shows the benefit of latent identity reward feedback. In single-subject mode, LIReFL improves R-DINO from 0.483 to 0.499, Face-S from 0.251 to 0.301, and EPE from 10.53 to 9.31. In multi-subject mode, LIReFL improves R-DINO from 0.506 to 0.524, Face-S from 0.287 to 0.329, and mIoU from 0.532 to 0.570. A "w/o LIReFL" ablation (Stage 2 with SFT only, no reward feedback) shows minimal gains over Stage 1 (single-subject R-DINO: 0.483 → 0.487, Face-S: 0.251 → 0.266), confirming that the improvement comes from the reward feedback, not simply from additional training iterations.

LIRM design choices (Table VIII): Evaluated on pairwise classification accuracy on the test set of win-lose pairs.

• Optimization objective: BCE loss (default) achieves average accuracy 0.720 across five noise-level intervals. Bradley-Terry loss achieves only 0.656, with particularly poor performance at early timesteps (0.491 at $t \in [0, 0.2]$ vs. BCE's 0.702). The instability likely arises because at low noise levels, identity differences are subtle, and pairwise comparisons become noisy.
• Reference image injection strategy: Using reference features as the Query (default) is critical. Reversing this—using reference as Key/Value—causes accuracy to collapse to 0.455. This is because "the reference image must actively attend to the noisy latents to effectively discern identity features."
• Parameter tuning scope: Freezing text and patch embeddings (default) achieves 0.720. Fine-tuning them slightly decreases accuracy to 0.715, suggesting that "preserving pre-trained priors prevents overfitting and is sufficient for effective reward modeling." The accuracy is consistent across noise timestep intervals under both settings.

Timestep range for LIReFL (Table IX): Comparing dense feedback across all timesteps versus sparse feedback on only the last 3 timesteps shows consistent but modest gains from dense feedback. In single-subject mode, all timesteps improves R-DINO from 0.494 to 0.499, Face-S from 0.293 to 0.301, mIoU from 0.543 to 0.558, and EPE from 9.98 to 9.31. In multi-subject mode, all timesteps improves R-DINO from 0.518 to 0.524, Face-S from 0.324 to 0.329, and EPE from 6.30 to 6.08. These are small absolute improvements but consistent across all metrics, corroborating the claim that "providing reward feedback at arbitrary timesteps is essential to fully leverage the potential of LIReFL."

LIReFL loss weight $\lambda_2$ (Table X): Sweeping $\lambda_2$ from 0.01 to 1.00 reveals a narrow effective range and a sharp reward-hacking threshold:

• At $\lambda_2 = 0.01$ (single-subject mode): R-DINO 0.505, Face-S 0.279, mIoU 0.560, EPE 9.85. Identity metrics are already above Stage 1 (0.483, 0.251), showing that even weak reward feedback helps.
• At $\lambda_2 = 0.10$ (default): R-DINO 0.499, Face-S 0.301, mIoU 0.558, EPE 9.31. Overall best balance—slightly lower R-DINO but higher Face-S and better motion metrics.
• At $\lambda_2 = 0.25$: R-DINO 0.492, Face-S 0.272, mIoU 0.555, EPE 9.65. Beginning of decline in identity metrics.
• At $\lambda_2 = 0.50$: R-DINO 0.482, Face-S 0.223, mIoU 0.541, EPE 9.75. Clear degradation.
• At $\lambda_2 = 1.00$: Catastrophic reward hacking—R-DINO collapses to 0.350, Face-S to 0.120, mIoU to 0.350, EPE to 25.00. The model "finds shortcuts to maximize the reward at the expense of visual realism and motion coherence."
• Similar pattern in multi-subject mode, though the reward-hacking threshold appears slightly higher (at $\lambda_2 = 0.50$, mIoU is still 0.530 and EPE 6.73; collapse occurs at $\lambda_2 = 1.00$ with mIoU 0.380 and EPE 15.06).

The narrow effective range $[0.01, 0.25]$ and the sharp collapse at $\lambda_2 = 1.00$ are consistent with reward over-optimization phenomena documented in RLHF literature. The optimal $\lambda_2 = 0.10$ means the reward feedback contributes approximately 9% of the total loss gradient ($0.10 / (1 + 0.10) \approx 0.091$).

Emergent Capabilities

The paper demonstrates two capabilities that emerge from the multi-task training paradigm without any task-specific fine-tuning (Figure 8):

Image-to-Video (I2V) generation: DreamVideo-Omni can take a single image as input and generate a coherent video continuation, despite being trained on a text-to-video base model. The paper frames this as "a specialized form of customization, where the entire first frame serves as the comprehensive reference identity." The example shows a panoramic view of a cartoon character climbing stairs, generated from a single first frame.

First-frame-conditioned trajectory control: The omni-motion mechanism extends to I2V generation, enabling precise spatial trajectory guidance conditioned on the provided initial frame. The example shows a close-up of two people exchanging flowers with specified hand trajectories.

These emergent capabilities are presented as qualitative results only—no quantitative metrics are reported for I2V or first-frame-conditioned trajectory control. The significance is that they demonstrate generalization beyond the explicitly trained task distribution, but the lack of quantitative evaluation prevents assessing how competitive these capabilities are compared to dedicated I2V or image-conditioned motion control methods.

Critical Assessment

Does the paper demonstrate that DreamVideo-Omni harmoniously integrates multi-subject customization with omni-motion control?

The paper's central promise is a unified framework that simultaneously handles multi-subject identity preservation and multi-granularity motion control (global boxes, local trajectories, camera movement). The experimental evidence provides partial support for this claim, but with important limitations in what was actually tested.

What was demonstrated: DreamVideo-Omni is evaluated on three tasks separately—joint customization with motion control (Tables II–III, Figure 5), pure subject customization (Table IV, Figure 6), and pure motion control (Table V, Figure 7). In each of these evaluations, it shows state-of-the-art or competitive performance. The model can clearly generate videos where a specific subject follows a specified trajectory (Figure 5), and it can maintain identity better than customization-only baselines in multi-subject scenarios (Figure 6).

What was not demonstrated: The paper does not provide a systematic evaluation of all control signals activated simultaneously in multi-subject, omni-motion scenarios—for example, two subjects, each with their own reference image, bounding box, trajectory, and a camera movement applied globally. The "joint ID & motion" evaluation against DreamVideo-2 (Table II) is limited to single-subject bounding box control, because DreamVideo-2 cannot support more. The multi-subject evaluations for pure customization (Table IV) have no motion control signals active. The multi-subject motion control evaluations (Table V) have no reference images—they test trajectory/box following without identity customization. The paper's multi-subject capability is validated in pieces (identity-only, motion-only) but not as a fully integrated multi-subject + omni-motion system in a single evaluation. The qualitative examples in Figure 1 show the claimed capability, but no corresponding quantitative metrics exist for the full multi-subject omni-motion setting. This is a significant gap: the paper claims to solve the precise problem that arises when all these controls must be composed together, but the experimental validation separates them.

Does the 1.3B model genuinely surpass the 14B Wan-Move for motion control?

The comparison with Wan-Move is one of the paper's headline results: "surpassing a ~14× larger model." This claim requires careful scrutiny.

What the metrics show: In Table V, DreamVideo-Omni (1.3B) achieves mIoU 0.558 and EPE 9.31 in single-subject mode, outperforming Wan-Move (14B, mIoU 0.507, EPE 14.43). In multi-subject mode, DreamVideo-Omni achieves mIoU 0.570 and EPE 6.08 versus Wan-Move's 0.541 and 9.02. These are genuine metric improvements.

Important caveats:

1. Different model architectures and training paradigms: Wan-Move is an image-to-video model, while DreamVideo-Omni is text-to-video. Wan-Move conditions on the first frame (which provides strong visual priors for scene content); DreamVideo-Omni conditions only on text, reference images, and motion signals. This makes direct comparison somewhat apples-to-oranges—they solve related but distinct tasks. Wan-Move may sacrifice some trajectory precision for the benefit of first-frame-conditioned visual fidelity, a dimension not captured by mIoU and EPE.
2. Wan-Move was not designed for multi-subject trajectory control with reference images. The paper acknowledges that Wan-Move "focus exclusively on point trajectory precision" (Section I) and does not support reference image conditioning. The comparison is therefore not against Wan-Move's intended use case, but against Wan-Move repurposed for a task it wasn't trained on. The margin of DreamVideo-Omni's improvement may partly reflect this mismatch.
3. No visual quality or diversity metrics are reported. mIoU and EPE measure spatial and trajectory precision but not whether the generated videos are visually compelling, temporally coherent, or diverse. It is possible that DreamVideo-Omni achieves better trajectory following at the cost of visual artifacts not captured by these metrics. The user study provides some reassurance—60.2% overall quality preference over Wan-Move—but this is a limited perceptual evaluation (18 evaluators, 270 groups, no reported statistical significance testing).

The claim is not false given the metrics shown, but it is narrower than the headline suggests: DreamVideo-Omni achieves better bounding box and trajectory alignment than Wan-Move on DreamOmni Bench, under a specific evaluation protocol. Whether it generates better videos in a holistic sense is less clear from the reported evidence.

Does the latent identity reward feedback learning (Stage 2) provide meaningful benefit, or is the gain primarily from the Stage 1 architecture?

The ablation in Table VII directly addresses this: Stage 1 alone achieves R-DINO 0.483 and Face-S 0.251 (single-subject); Stage 2 LIReFL improves these to 0.499 and 0.301. The improvements are real and consistent across modes.

However, several factors moderate the strength of this finding:

The absolute gains are modest. R-DINO improves by 0.016 (3.3% relative) in single-subject and 0.018 (3.6% relative) in multi-subject mode. Face-S improves by 0.050 (19.9% relative) in single-subject and 0.042 (14.6% relative) in multi-subject mode. These are meaningful but not transformative—Stage 1 already provides a strong identity preservation baseline. The paper's framing of Stage 2 as resolving the "fundamental identity-motion conflict" is somewhat overstated relative to the measured effect size.

The "w/o LIReFL" ablation (Stage 2 with SFT only) is a critical control that is underreported. Table VII shows that additional SFT-only training in Stage 2 (without reward feedback) improves R-DINO from 0.483 to 0.487 and Face-S from 0.251 to 0.266 in single-subject mode. This means that 4/16 points of R-DINO gain and 15/50 points of Face-S gain come from simply training longer, not from the reward feedback mechanism. The net contribution of LIReFL specifically is R-DINO +0.012 and Face-S +0.035—smaller than the raw Stage 1 → Full Model differences suggest. The paper should make this decomposition more explicit; the current presentation risks attributing continued training benefits to the reward feedback innovation.

The narrow effective range of $\lambda_2$ ($[0.01, 0.25]$, Table X) and sharp reward-hacking threshold ($\lambda_2 = 1.00$) suggest fragility. In a practical deployment, hyperparameter sensitivity of this magnitude would require careful tuning for each new model architecture or dataset. The paper does not explore whether the optimal $\lambda_2$ generalizes across different base models or data distributions.

Are the ablation results internally consistent and well-controlled?

Several ablations deserve scrutiny:

The condition-aware 3D RoPE ablation produces catastrophic collapse (Table VII), which is so severe that it may indicate an implementation error rather than a genuine component contribution. When Face-S drops from 0.301 to 0.039 and R-DINO from 0.499 to 0.139, the model is essentially producing random output. This could happen if removing the 3D RoPE causes token type confusion that prevents any meaningful conditioning—but it could also happen if the ablation inadvertently breaks the model's ability to process the concatenated input sequence at all. The paper does not provide intermediate diagnostics (e.g., training loss curves, whether the model produces any recognizable content) that would distinguish between "this component is critical" and "this ablation breaks the model implementation."

The hierarchical BBox injection ablation similarly produces extreme degradation (mIoU drops from 0.556 to 0.400 in single-subject, 0.532 to 0.289 in multi-subject). This is presented as evidence that hierarchical injection is essential, but the alternative (input-only injection) is a deliberately weak baseline. A more informative ablation would compare hierarchical injection against (a) concatenation as additional tokens, (b) cross-attention conditioning, or (c) injection at every other block rather than every block—to determine whether the benefit comes specifically from the hierarchical design or simply from stronger conditioning.

No ablation is provided for the dataset construction pipeline choices—motion-based filtering threshold, hybrid sampling ratio for trajectories, reference image augmentation probability, or the effect of training dataset size. The paper relies heavily on its custom dataset (~2.12M clips), but does not characterize how dataset properties affect downstream performance. An ablation showing, for instance, that motion filtering is critical (or not) would strengthen the claim that the data pipeline is a contribution, not just an implementation necessity.

Limitations specific to the experimental design

Single base model architecture (Wan2.1-1.3B T2V). All results—including the Stage 2 reward feedback improvements and the emergent capabilities—are demonstrated on exactly one model. The paper argues this model is "representative," but no evidence supports this. Would the condition-aware 3D RoPE be equally critical for a U-Net-based diffusion model? Would LIReFL provide similar gains on a model with different pre-trained priors? The generalizability of both the architecture and the training paradigm is unknown.

No statistical uncertainty is reported. All metrics are point estimates without confidence intervals, standard deviations, or error bars. The DreamOmni Bench contains 1,027 videos split into 436 single-subject and 591 multi-subject samples, which is modest. Differences of 0.01 in R-DINO or a few points in EPE could be within sampling noise, yet the paper draws conclusions from these small differences without quantifying uncertainty.

No direct comparison with Tora2 or Video Alchemist on DreamOmni Bench. The paper compares against these on MSRVTT-Personalization Bench (Table III) by citing their reported numbers, but does not reproduce them on the more comprehensive DreamOmni Bench. Since MSRVTT-Personalization has lower video quality and resolution (as the paper itself notes), the most relevant comparison—joint customization with motion control on high-quality videos—is against DreamVideo-2 only, which the paper acknowledges is a weaker baseline (single-subject, bounding-box-only).

The emergent I2V and first-frame-conditioned trajectory control are purely qualitative. Figure 8 shows compelling examples, but without quantitative evaluation on standard I2V benchmarks (e.g., UCF-101, something with FVD or FID-VID metrics), it is impossible to assess how competitive these capabilities are. The paper frames these as a significant contribution (listed in the abstract as a benefit of the progressive two-stage training), but provides no evidence beyond two cherry-picked examples.

What experiments would have strengthened the paper?

1. A fully integrated multi-subject omni-motion evaluation on DreamOmni Bench, with all control signals active simultaneously (reference images + boxes + trajectories + camera motion for 2+ subjects), measuring all six metrics. This is the paper's stated capability; it should be quantitatively validated.

2. A controlled FLOPs-matched or parameter-matched comparison of Stage 1 only vs. Stage 1 + LIReFL vs. longer Stage 1 training, to definitively attribute identity improvements to the reward feedback mechanism rather than to additional training iterations.

3. Testing on at least one additional base model (e.g., a U-Net-based diffusion model, or a DiT from a different family) to assess whether the architecture innovations generalize.

4. Confidence intervals or standard deviations on all metrics, computed via bootstrap resampling over the DreamOmni Bench videos, to ground claims of improvement in statistical reality.

5. Ablation of the dataset construction choices: motion filtering threshold, hybrid trajectory sampling ratio, data augmentation strength, and training dataset size—to characterize which aspects of the data pipeline are essential versus incidental.

6. A human evaluation of reward model quality: the LIRM is trained on human labels, but how well does it correlate with human judgments on held-out data? The paper reports pairwise classification accuracy (Table VIII), but does not report correlation metrics (Spearman's ρ, Pearson's r) between LIRM scores and human preference ratings, which would indicate whether the reward model is genuinely aligned with human perception or merely accurate on coarse win-lose distinctions.

7. Evaluation of I2V and first-frame-conditioned trajectory control on standard benchmarks to quantify the emergent capabilities rather than presenting them as qualitative anecdotes.

6. Limitations and Trade-offs

Assumption: Difficulty Can Be Estimated Cheaply Across All Prompts

This paper assumes that test-time compute allocation can be optimized by conditioning on a prompt's difficulty. However, the method they use — generating 2048 samples per prompt and checking correctness against ground-truth labels — completely sidesteps the most important practical question: how do you know a prompt's difficulty before you've already spent enormous compute evaluating it? The authors openly acknowledge this tension:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity."

The consequence is that the reported 4× efficiency gains are computed after difficulty is already known, without amortizing the cost of learning it. Generating 2048 samples per prompt and scoring them dwarfs the actual test-time compute budget being optimized (max 256–512 generations). In a realistic deployment, the total cost would be difficulty estimation (which is equivalent to running a 2048-sample best-of-N on every prompt before you even start) plus the strategy execution cost. The 4× figure is therefore an upper bound that assumes free difficulty information. A practitioner deploying this method would find that the difficulty estimation step entirely consumes — and likely exceeds — the efficiency gains the paper claims.

The evidence is in Section 3.2: the authors explicitly describe the cost but never include it in any budget calculation. They also note that the predicted difficulty bins (using PRM scores instead of ground-truth correctness) largely overlap with oracle bins in Figures 4 and 8, meaning the method works without ground-truth labels — but they still require 2048 samples per prompt for the PRM-based difficulty estimate. The per-prompt cost of this step is never reported numerically, but it is straightforward to infer: 2048 generations × (1 forward pass + 1 PRM evaluation) is substantially more expensive than the largest test-time budgets studied (256–512 generations).

Mitigation status: The paper flags this as an open problem for future work and suggests training a model to predict difficulty directly from the question text, but no such model is developed or evaluated. A more promising direction — dynamic, online difficulty estimation where the first few generations double as difficulty probes — is mentioned in Section 3.2's exploration-exploitation framing but never explored. The limitation remains essentially unaddressed in the current work, and any real-world deployment would need to solve it before the paper's strategy becomes practical.

---

Single Benchmark, Single Model Family: MATH and PaLM 2-S*

All experiments in this paper use exactly one benchmark (MATH, 500 test questions) and one model family (PaLM 2-S*). The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this assumption is never tested. The entire difficulty-dependent scaling framework — the central finding that beam search helps on medium problems but hurts on easy ones, that revisions work best on easy problems and need parallel diversity on hard ones — is validated on a narrow slice of the possible problem and model space.

The consequences of this limitation are multi-dimensional and severe:

Problem-space narrowness: MATH consists of high-school competition math problems with clean symbolic ground-truth answers. This domain has specific properties — exact correctness is well-defined, solutions are multi-step but logically structured, and the space of possible answers is highly constrained. It is unclear whether the difficulty-dependent patterns generalize to code generation (where solutions can be incrementally tested), open-ended reasoning (where there is no single correct answer), factual QA (where correctness depends on knowledge rather than inference), or creative tasks (where evaluation is inherently subjective). The entire PRM training pipeline — Monte Carlo rollouts to determine step-level correctness — relies on having unambiguous ground-truth answers to check against, which is not available in many real-world domains.

Model-space narrowness: PaLM 2-S* is a specific architecture with specific training data and specific behavioral properties. The revision model's ability to learn from being shown its own incorrect outputs depends on the base model's in-context learning capabilities, which vary substantially across model families. A model with different calibration properties or error patterns might exhibit entirely different over-optimization thresholds under beam search. The paper's specific quantitative findings — e.g., that ~38% of correct answers get revised back to incorrect ones (Section 6.1), that lookahead search underperforms (Section 5.3) — could easily be artifacts of PaLM 2-S*'s specific characteristics.

Testing fragility: The test set of 500 questions, split into five difficulty quintiles of ~100 each, then further split by two-fold cross-validation, means the compute-optimal policy is selected based on ~50 questions per fold per bin. This is a tiny sample. A single anomalous question in a bin could shift the selected strategy. The paper reports no confidence intervals on the compute-optimal scaling curves (Figures 4 and 8), making it impossible to assess whether the observed advantages are statistically robust or could reverse with a different random split of 500 questions.

The evidence for this limitation is the absence of any cross-benchmark or cross-model evaluation. Every table, every figure, every ablation, and every conclusion is drawn from MATH + PaLM 2-S*. The paper does not even report results on related math benchmarks (e.g., GSM8K) that might provide a sanity check for generalization.

Mitigation status: Not addressed. The paper's framing treats MATH + PaLM 2-S* as a sufficient testbed, and no claims are made about generalizability beyond this setting. The limitation is acknowledged only implicitly — by not making universal claims. For a practitioner, the practical consequence is clear: you cannot assume the difficulty-dependent patterns (e.g., beam search over-optimizes on easy problems, revisions work best on easy problems) will hold for your specific model and problem domain without replicating the analysis.

---

The ~14× Larger Pretraining Baseline Is Not Compute-Optimal

The FLOPs-matched comparison in Section 7, which asks whether test-time compute can substitute for pretraining compute, scales only model parameters when creating the larger-pretraining baseline, holding training data fixed. The authors are explicit about this choice:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

The consequence is that the ~14× larger model baseline is weaker than it would be under compute-optimal pretraining (Chinchilla scaling). Hoffmann et al. (2022) demonstrated that for a given pretraining FLOPs budget, the optimal allocation scales parameters and data roughly equally. A Chinchilla-optimal model trained with 14× more total FLOPs would be larger than the base model but smaller than 14× the parameter count, with correspondingly more training data. The paper's parameter-only-scaling approach (following LLaMA, not Chinchilla) means the comparison effectively gives test-time compute an advantage: the larger model is suboptimally trained for its parameter count, potentially underfitting its capacity. The reported advantages of compute-optimal test-time strategies over the larger model — e.g., +27.8% relative improvement on easy-to-medium problems at R ≪ 1 for revisions (Figure 1 bar charts) — may shrink or reverse when compared against a properly Chinchilla-trained baseline.

Compounding this, the ~14× larger model uses only greedy decoding — no majority voting, no best-of-N, no test-time compute at all. This is an asymmetric comparison: the smaller model gets to use sophisticated, adaptive test-time strategies while the larger model gets none. A fairer comparison would give the larger model some test-time compute budget as well, or compare at fixed total FLOPs (pretraining + inference combined) rather than matching pretraining FLOPs and then giving the smaller model an inference bonus. The paper's numbers for "test-time compute beats pretraining" should therefore be understood as an upper bound on the advantage under the most favorable comparison conditions, not an estimate of what a practitioner would experience when choosing between training a larger model vs. deploying a smaller one with smarter inference.

The evidence is in Section 7, where the authors explicitly describe both the parameter-only scaling choice and the greedy-decoding baseline. They do not report results for a Chinchilla-trained larger model nor for a larger model with any test-time augmentation.

Mitigation status: The paper acknowledges the non-optimality of the pretraining baseline in the quoted passage and frames the Chinchilla comparison as future work, but does not discuss how this choice might systematically bias results in favor of test-time compute. The acknowledgment exists, but its implications for how to interpret the headline FLOPs-matched claims are not drawn out, leaving readers to potentially over-interpret the quantitative advantage magnitudes.

---

Verifier Over-Optimization Is Documented but Unsolved — It Imposes a Hard Cap on Test-Time Compute Scaling

The paper identifies verifier over-optimization as a central limiting phenomenon: beam search degrades performance on easy problems at high budgets (Figure 3, right), lookahead search — the most powerful optimization method — performs worst overall at the same generation budget (Figure 3, left), and qualitative examples (Appendix M, Figures 29, etc.) show search producing degenerate outputs (repetitive low-information steps, overly short solutions) that score highly under the PRM but are incorrect.

The consequence is that test-time compute scaling is fundamentally bounded by verifier quality, not by the ability to construct better search algorithms. The paper's compute-optimal policy works around this by routing easy problems to weaker optimization methods (best-of-N rather than beam search), but it does not solve the underlying problem. On medium-difficulty problems (bins 3–4), where the policy deploys beam search, over-optimization still limits the scaling ceiling — the beam search curves in Figure 3 flatten well before the budget is exhausted. This means the 4× efficiency gains over best-of-N come from avoiding over-optimization where it is most severe (easy problems), not from pushing the over-optimization threshold higher. The scaling ceiling is therefore fixed by the PRM's reliability, and no amount of additional algorithmic sophistication at search time (better beam search heuristics, better lookahead strategies) can break through it without improving the verifier itself.

The evidence is multi-layered: Figure 3 (right) shows beam search accuracy decreasing with budget on easy problems (verifier exploitation); Figure 3 (left) shows lookahead search underperforming simpler methods (the stronger optimizer amplifies verifier errors more aggressively); and the qualitative failure modes in Appendix M show exactly how search finds verifier-pleasing but incorrect solutions. The paper does not report whether the compute-optimal policy's performance would continue to improve if the PRM were made more robust — that counterfactual is not tested.

Mitigation status: The paper acknowledges over-optimization in Section 5.3 and frames verifier robustness as a key future direction in Section 8, but makes no attempt to improve verifier calibration, develop adversarial training, or build ensemble verifiers that might be more resistant to exploitation. The compute-optimal policy is therefore a mitigation (work around the limitation) rather than a solution (remove the limitation). A practitioner wanting to scale test-time compute beyond the budgets studied here (e.g., 1024 or 2048 generations) would hit the over-optimization wall regardless of how cleverly they allocate budget across search strategies.

---

The Revision Model's 38% Correct-to-Incorrect Reversion Rate

When the revision model generates an answer that is actually correct, there is approximately a 38% chance that the next revision step will incorrectly change it to a wrong answer. As the paper acknowledges: "the model may encounter correct answers in its context (produced during earlier revisions) and incorrectly 'revise' them into wrong answers."

The consequence is that simply running more sequential revisions does not monotonically improve performance — the model can un-learn good solutions. This is a direct consequence of the training data construction: the model was trained exclusively on sequences where all in-context answers are incorrect, followed by a correct target. The model never sees examples of "the current answer is already correct, so don't change it" or "make a minor refinement without breaking correctness." The revision model's ability to generalize to longer chains (Figure 6, left, showing pass@1 improvements out to 64 steps) demonstrates that it has learned some revision skill, but the 38% reversion rate means the chain is fundamentally unstable. The model's per-step accuracy might improve over many steps, but a correct answer at step k is far from guaranteed to survive to step k+1.

The paper partially mitigates this by using within-chain selection — majority voting or verifier-based selection across all revision steps rather than always taking the final output. The evidence that this helps is in the sequential revision results (Figure 6, right), where sequential + verifier outperforms sequential + majority, and both outperform parallel sampling. But the mitigation is a patch, not a fix. A verifier that can correctly identify the best answer in a chain is the same verifier that suffers from over-optimization (limitation above), and majority voting among a chain where 38% of correct answers get corrupted is inherently noisy. A more principled solution — training the model to recognize when no revision is needed, or training on trajectories that include "correct → correct" transitions — is not explored.

Mitigation status: The paper identifies the problem (Section 6.1) and applies within-chain selection to reduce its impact, but does not modify the training procedure to address the root cause. The magnitude of the problem (38% reversion rate) is reported without confidence intervals, and there is no analysis of whether the reversion rate depends on problem difficulty, revision step index, or subject matter. A practitioner deploying revision models would need to be aware that sequential revisions are not reliable — you cannot trust that running more revisions monotonically improves quality — and that the within-chain selection mechanism is itself dependent on verifier quality.

---

No Combination of Revisions with PRM-Guided Search

The paper studies two independent scaling axes — modifications to the proposal distribution via iterative revisions, and modifications to the selection mechanism via PRM-guided search — but never combines them. The authors acknowledge this explicitly in Section 8:

"we did not experiment with PRM tree-search techniques in combination with revisions."

The consequence is that the paper's results represent a lower bound on what a fully integrated system could achieve. The two mechanisms have complementary strengths that the paper itself demonstrates: revisions improve the quality of individual candidate solutions (especially on easy problems where local refinement helps) while PRM search improves the ability to select good candidates from a diverse set (especially on medium problems where exploration is needed). A combined approach — for example, using the revision model as the proposal distribution within beam search, or using the PRM's step-level scores to dynamically decide when to revise vs. restart — could plausibly achieve better performance than either mechanism alone. The paper's current inability to claim results from such a combination means the measured 4× improvements are against a best-of-N baseline, not against a hypothetical combined approach that a future system might achieve.

This gap also limits the paper's theoretical contribution. The paper frames revisions as "modifying the proposal distribution" and search as "modifying how outputs are selected" (Section 2), suggesting they are complementary axes that can be independently optimized and then combined. But without an experiment showing that combination yields additive or synergistic gains, this framing remains a hypothesis rather than a demonstrated principle. It is possible — though the paper provides no evidence either way — that combining the two mechanisms introduces new failure modes (e.g., the PRM over-optimizing against revision model outputs differently than against base model outputs) that would partially offset individual gains.

Mitigation status: The paper explicitly flags the omission in Section 8 and suggests it as future work, but provides no preliminary exploration, no small-scale experiment, and no analysis of what the combined approach might look like architecturally. A practitioner wanting to deploy both mechanisms together would need to design and validate the integration from scratch, with no guidance from the paper on how the difficulty-dependent allocation policy should change when both axes are available.

---

Difficulty Bins Are Static, Coarse, and Pre-Computed — No Dynamic Mid-Generation Adjustment

The compute-optimal policy predicts a prompt's difficulty once (using 2048 samples and the PRM's aggregate score), bins it into one of five quintiles, and then selects a fixed strategy for the entire generation process. There is no mechanism for adjusting the strategy mid-generation based on what the model observes as it generates, nor for refining the difficulty estimate as more information becomes available.

The consequence is that the policy is brittle to misclassification. A question near the boundary of bins 2 and 3 that gets misclassified as bin 2 (easy) will receive best-of-N — the wrong strategy — and potentially underperform by a large margin (as Figure 3, right, shows, the gap between best-of-N and beam search on bin 3 is substantial). Because there is no mid-course correction, the entire generation budget is committed based on the initial difficulty estimate. If the difficulty estimate is wrong, the budget is wasted.

More subtly, even a correctly binned question might benefit from dynamic adjustment. A problem that initially appears easy (few samples needed) might turn out to be harder than expected after the first few generations reveal that the model's answers are inconsistent; conversely, a problem initially classified as hard might reveal itself as solvable after the first few beam search steps find a promising path. A dynamic policy could start with a conservative strategy, assess the intermediate verifier scores on initial generation steps, and reallocate the remaining budget accordingly — effectively making difficulty estimation and strategy selection an online process rather than a separate pre-computation step.

The evidence for this limitation is the static, per-question nature of the difficulty bins (Section 3.2) and the absence of any experiment where the strategy changes within a single generation episode. Figures 4 and 8, which show the benefits of compute-optimal scaling, all assume the difficulty bin is known before any generation budget is spent.

Mitigation status: Not addressed. The paper's difficulty estimation procedure treats difficulty as a fixed property of the prompt that can be measured once and then acted upon. The connection to adaptive, online allocation — where difficulty estimation and budget spending are interleaved — is hinted at in Section 3.2's discussion of exploration vs. exploitation but is not developed. A practitioner would need to solve the dynamic allocation problem independently, potentially combining this paper's findings about static difficulty-conditioned strategies with techniques from multi-armed bandit or Bayesian optimization literature that the paper does not engage with.

7. Implications and Future Directions

How This Work Changes the Landscape

DreamVideo-Omni reframes the identity-motion conflict in video generation from an architectural trade-off to a supervision misalignment problem. The standard narrative in the field treats precise motion control and faithful identity preservation as inherently opposed—you can have one or the other, and unified methods must find an acceptable compromise point on the Pareto frontier. The paper's central contribution is to demonstrate that this framing is wrong: the conflict arises not from limited model capacity but from training objectives that penalize any pixel change, including the legitimate appearance variation that motion causes. When a subject turns their head, the face pixels change. The standard diffusion L2 loss cannot distinguish between "the face changed because the subject moved" (desired) and "the face changed because identity was lost" (undesired), so it incentivizes the model to suppress either motion or identity detail.

The paper's resolution—training a specialized reward model on human identity judgments and using its gradients to fine-tune the generator—shifts the optimization target from "minimize per-pixel distance to ground truth" to "produce videos that humans recognize as identity-preserving." This is a methodological contribution with broader applicability: for any generative task where the output property of interest (style consistency, compositional correctness, physical plausibility) is misaligned with pixel-level reconstruction error, the latent-space reward feedback paradigm provides a template solution. The specific design—a VDM-based reward model operating in latent space with identity features as queries attending to video features, trained via preference optimization and deployed to provide feedback at arbitrary diffusion timesteps—is new, but the conceptual reframing is what could influence adjacent problems in controllable generation, image editing, and 3D content creation.

The practical demonstration that a 1.3B model with architectural binding mechanisms and reward-aligned training can surpass a 14B model (Wan-Move) on motion control precision (mIoU 0.558 vs. 0.507, EPE 9.31 vs. 14.43 in single-subject mode) reorients the scaling conversation. Prior work in controllable video generation operated under the implicit assumption that larger models + more data would eventually solve the identity-motion trade-off through sheer capacity. This paper suggests that architectural structure (binding mechanisms) and training objective alignment (reward feedback) can substitute for ~10.8× more parameters, at least on the metrics measured. Whether this holds for visual quality and diversity—dimensions not captured by mIoU and EPE—remains an open question, but the parameter-efficiency argument is credible within the metric scope reported.

The paper also introduces a unifying principle for camera control that simplifies model design: camera movement can be represented as point-wise trajectories on background pixels, making it processable through the same trajectory conditioning mechanism as local object motion. This eliminates the need for explicit 3D camera parameter estimation pipelines (used in MotionCtrl, CamI2V, FullDiT) and their associated training data requirements. The practical consequence is that a single annotation pipeline (RAFT → RAM++ → Grounding DINO → SAM 2 → CoTracker3) can extract all motion control signals—foreground object trajectories, background camera trajectories, and bounding boxes—from unlabeled video, making large-scale training data construction substantially cheaper and more scalable than approaches requiring synthetic data or structure-from-motion.

A subtle but important reconciliation emerges around whether reinforcement learning helps identity preservation in video generation. Prior work is split: MagicID and Identity-GRPO report improvements from pixel-space reward feedback (DPO and GRPO, respectively), while IPRO reports modest gains that are restricted to final denoising steps. This paper provides a mechanism to explain the discrepancy: reward feedback does help, but only if it can operate at arbitrary denoising timesteps and only if the reward model has access to spatiotemporal priors (which static image encoders like CLIP/DINO lack). The LIRM's design—VDM backbone providing motion-aware features, identity cross-attention measuring alignment, latent-space operation enabling dense feedback—addresses both limitations, and the ablation in Table IX confirms that restricting feedback to the final 3 timesteps (emulating pixel-space methods' constraint) reduces identity gains. This doesn't invalidate prior work but clarifies the conditions under which identity-focused RL works: the reward model must perceive motion dynamics and the feedback must reach early denoising steps where structure is established.

Follow-Up Research This Work Enables

Fully integrated multi-subject omni-motion evaluation. The paper evaluates identity customization and motion control on separate subsets of DreamOmni Bench—single-subject joint ID+motion (Table II), multi-subject ID-only (Table IV), multi-subject motion-only (Table V). The claimed capability—simultaneous multi-subject identity preservation with all omni-motion signals active (boxes + object trajectories + camera trajectories for 2+ subjects)—is never quantitatively validated in a single experiment. A follow-up should run DreamVideo-Omni on the 591 multi-subject DreamOmni Bench videos with all control signals simultaneously active, report all six metrics, and compare against the strongest available baseline for each sub-capability to establish whether composition of controls introduces new failure modes or whether the binding mechanisms scale cleanly. The hypothesis—that group embeddings prevent inter-subject interference and role embeddings prevent signal-type confusion—predicts minimal degradation from the single-signal to the all-signal setting.

Does LIReFL generalize beyond Wan2.1? The entire latent identity reward feedback pipeline is validated on exactly one base model. A critical extension would replicate the full Stage 1 + Stage 2 training on a DiT from a different family (e.g., CogVideoX-5B or an open-source U-Net-based video diffusion model) and measure whether (a) the condition-aware 3D RoPE is equally critical for heterogeneous input processing in different architectures, (b) the LIRM's pairwise classification accuracy transfers when trained on the new model's latent space, and (c) the optimal ~0.10 value for the LIReFL regularization weight ~0.10 is stable or model-specific. A negative result—e.g., LIReFL provides negligible gains on a U-Net model because its latent space is less semantically structured—would reveal an important boundary condition: that latent-space reward feedback requires the semantically rich representations of transformer-based diffusion backbones.

Reward model robustness and the over-optimization ceiling. The LIRM achieves 0.720 average pairwise classification accuracy on the test set (Table VIII), meaning it incorrectly ranks ~28% of win-lose pairs. During LIReFL, the generator optimizes against this imperfect reward model, and at reward feedback weight ~1.00, the generator discovers latent-space patterns that achieve high LIRM scores while producing nonsensical decoded videos (Table X: R-DINO collapses to 0.350, Face-S to 0.120). This is a classic reward hacking failure. A systematic study should measure how reward model accuracy relates to the reward hacking threshold—specifically, whether improving LIRM accuracy (via larger backbone, more training data, ensemble methods) shifts the collapse point to higher ~0.10 values, enabling stronger identity optimization without quality degradation. The experiment would train LIRM variants at 4K, 8K, and 16K training steps (or with increasing dataset sizes), measure their accuracy, then run LIReFL with each variant and identify the maximum ~0.10 before performance collapse. If the collapse threshold increases with reward model accuracy, this validates the hypothesis that better reward models are the key bottleneck for scaling identity-focused RL in video generation.

Dynamic, online difficulty estimation for compute-optimal allocation in video generation. The paper's difficulty-conditioned strategy selection (Section 3.2) requires pre-computing difficulty via 2,048 samples per prompt before any budget is spent—a prohibitive cost that the authors acknowledge. An extension could develop an online difficulty estimator that starts generating with a small probe budget (e.g., 4 parallel samples), uses the LIRM's score distribution on those early latents to estimate whether the subject-motion composition is "easy" (high initial identity scores, stable across early denoising) or "hard" (low scores, high variance), and dynamically allocates the remaining inference budget to sequential revisions, broader parallel sampling, or additional search based on the online estimate. This would transform DreamVideo-Omni from a two-stage training paradigm into a three-stage operational paradigm (estimate → allocate → generate) and directly address the exploration-exploitation tension the paper flags but doesn't resolve. The key metric would be whether online allocation recovers the efficiency gains of oracle difficulty knowledge without the 2,048-sample pre-computation cost.

Cross-domain validation of the identity-motion supervision misalignment hypothesis. The paper argues that the identity-motion conflict is fundamentally a misalignment between pixel-level reconstruction losses and human identity perception, and that reward feedback resolves it. This hypothesis makes a testable cross-domain prediction: for any generation task where the desired output property is perceptually invariant (style, composition, physical plausibility) but pixel-variant (different poses, lighting, viewpoints), a latent-space reward model trained on human judgments of that property should provide similar improvements over reconstruction-only training. A follow-up could test this on (a) 3D-aware image generation where viewpoint consistency matters (train a reward model on human judgments of whether two views depict the same object), (b) character animation where motion style preservation matters (train on judgments of whether a character's movement style matches a reference animation), or (c) video editing where edit consistency matters (train on judgments of whether an edited region blends seamlessly with unedited context). A consistent improvement across domains would establish the methodology as a general principle; failure on specific domains would clarify which properties are amenable to reward-based alignment and which require fundamentally different approaches.

Stress-testing the group/role embedding mechanism under adversarial conditions. The ablation (Table VII) shows that removing group embeddings degrades mIoU from 0.556 to 0.524 and EPE from 10.53 to 26.24 in single-subject mode, but this tests only the "presence vs. absence" of embeddings. A stronger test would evaluate whether the mechanism genuinely solves the binding problem or merely provides useful regularization. Specifically: train a variant where two subjects in multi-subject scenarios are deliberately assigned similar reference appearances (e.g., both wearing dark clothing, both with similar hairstyles) and non-overlapping trajectories, and measure whether the model with group embeddings correctly binds each trajectory to its intended subject while a variant without embeddings confuses them. The prediction is that in the absence of visual cues to distinguish subjects (which is what implicit association learning relies on), the explicit group embedding signal becomes load-bearing—the gap between embedding and no-embedding variants should widen substantially under visual ambiguity. If it doesn't, the embeddings are providing general-purpose conditioning benefits rather than specifically solving identity-motion binding.

Practical Applications and Downstream Use Cases

Pre-visualization and storyboarding for film and advertising. A director wants to see how two specific actors (reference images from headshots) would look interacting in a scene with precise blocking (bounding boxes for their positions) and specific gestures (trajectories for hand movements, head turns), while the camera executes a planned movement (background trajectories). The current alternative—hiring actors, booking a location, and filming a rough version—costs thousands of dollars and days of scheduling. DreamVideo-Omni's 1.3B model generates this pre-visualization in minutes on commodity GPUs, with the mIoU 0.570 and EPE 6.08 in multi-subject mode (Table V) meaning subjects hit their marks within ~6 pixels of specified positions on average. The key deployment consideration is that identity fidelity (Face-S 0.329 in multi-subject mode, Table IV) is strong enough for blocking and composition decisions but not yet at the level where fine facial expressions can be evaluated—so the use case is spatial planning, not performance review.

Personalized content creation at scale. A content platform wants to generate personalized video ads where a user's uploaded photo becomes the protagonist in a templated scenario (e.g., "you, winning a race," "your child, meeting a cartoon character") with platform-defined motion sequences. DreamVideo-Omni's training-free customization—no per-identity LoRA adaptation or test-time fine-tuning—makes this economically viable: inference requires only encoding the reference image (a single VAE forward pass) and running the standard 50-step UniPC denoising. The 1.3B model size means generation can run on a single A100 or even a high-end consumer GPU with reduced resolution, enabling cost per generation in the cents rather than dollars. The critical metric for deployment viability is the 38% correct-to-incorrect reversion rate in sequential revisions (Section 6.1): for this use case, single-pass generation with parallel sampling would be used rather than sequential revision chains, sidestepping the instability issue.

Training data generation for downstream fine-tuning. A robotics or embodied AI team needs large-scale video data of specific objects (the robot's end-effector, the objects it manipulates) undergoing diverse, precisely specified motion trajectories for training visuomotor policies. Physical data collection is bottlenecked by hardware availability and safety constraints. DreamVideo-Omni can generate synthetic training data where the same reference objects (from a few photographs) follow thousands of procedurally generated bounding box and trajectory sequences, producing training pairs of (reference appearance, motion specification, generated video) at the scale of millions of examples. The mIoU 0.570 and EPE 6.08 (multi-subject, Table V) provide quantitative guarantees on how closely the generated motion matches the specification, which is essential for policy learning. The newly released dataset pipeline (RAFT → RAM++ → Grounding DINO → SAM 2 → CoTracker3, Section III-C) would need to be adapted to the domain-specific object categories, but the paper's demonstration that this pipeline works on diverse real-world videos (1,027 DreamOmni Bench samples spanning humans, animals, objects, faces) suggests the adaptation cost is primarily in domain-specific tagging and captioning, not in core tracking infrastructure.

When to Prefer This Method

The paper positions DreamVideo-Omni against three distinct categories of alternative approaches, and the experimental results support the following decision rules:

• Prefer DreamVideo-Omni over subject-customization-only methods (VACE, Phantom) when motion control is also needed. Pure customization methods achieve comparable or slightly better identity metrics in some single-subject scenarios (Phantom: R-DINO 0.485 vs. DreamVideo-Omni 0.499 in single-subject, Table IV), but provide zero motion control capability. If the use case requires any spatial or trajectory guidance—even simple bounding boxes—DreamVideo-Omni provides this without sacrificing identity fidelity.

• Prefer DreamVideo-Omni over motion-control-only methods (Tora, Wan-Move) when identity customization is also needed. Wan-Move achieves comparable trajectory precision in multi-subject mode (EPE 9.02 vs. DreamVideo-Omni 6.08, Table V) but cannot condition on reference images. If the use case requires that specific subjects (not generic placeholders) perform the specified motions, DreamVideo-Omni is the only evaluated option that provides both.

• Prefer DreamVideo-Omni over prior unified methods (DreamVideo-2) when multi-subject or multi-granularity motion control is needed. DreamVideo-2 is limited to single-subject customization with bounding box control only. DreamVideo-Omni supports multi-subject scenarios (with explicit binding via group embeddings to prevent identity confusion) and omni-motion control (boxes + local trajectories + camera trajectories simultaneously). The ablation evidence (Table VII: removing hierarchical BBox injection drops mIoU from 0.556 to 0.400 in single-subject) demonstrates that the architectural additions specifically enable these capabilities rather than being incidental complexity.

• The current limitation that determines when not to prefer this method: visual quality metrics beyond spatial precision and identity similarity are not reported. If the application prioritizes perceptual quality, temporal coherence, or output diversity over precise control—e.g., generating inspiring rather than instruction-following content—the paper provides no evidence that DreamVideo-Omni matches or exceeds Wan-Move's 14B-parameter visual fidelity, and the parameter-efficiency argument (1.3B vs. 14B) suggests a quality ceiling that may be lower even if control precision is higher.