Video-As-Prompt: Unified Semantic Control for Video Generation

ArXiv: 2510.20888

🎯 Pitch

A single frozen video diffusion model, given just a reference video showing a desired style or motion, can control generation without any pixel-aligned guidance, rivaling condition-specific commercial systems. It replaces per-task architectures and finetuning with an in-context expert that retrieves semantic signals from the prompt video while deliberately blocking spurious spatial mappings.

---

1. Executive Summary

This paper introduces Video-As-Prompt (VAP), a unified semantic-controlled video generation framework that reframes heterogeneous condition types—concept, style, motion, and camera—as in-context generation by treating a reference video as a direct semantic prompt. Built on a frozen Video Diffusion Transformer augmented with a plug-and-play Mixture-of-Transformers (MoT) expert and a temporally biased Rotary Position Embedding that eliminates spurious pixel-mapping priors, VAP is trained on VAP-Data, the largest semantic-controlled video generation dataset with over 100K paired videos across 100 semantic conditions. As a single unified model, VAP achieves a 38.7% user preference rate that rivals leading condition-specific commercial models and demonstrates strong zero-shot generalization to unseen semantics, establishing that a single in-context paradigm can replace per-condition finetuning and per-task architectural designs only when the reference video signals can be retrieved without imposing nonexistent pixel-wise correspondences.

2. Context and Motivation

What Specific Problem Does This Paper Address?

The paper addresses a fundamental fragmentation in controllable video generation: semantic-controlled generation—where the condition shares higher-level semantics (concept, style, motion, camera movement) with the target video but lacks pixel-wise spatial correspondence—has no unified, generalizable framework. While structure-controlled video generation (using pixel-aligned conditions like depth, pose, optical flow, or segmentation masks) has matured into unified "all-in-one" models with well-understood injection mechanisms, semantic-controlled generation remains a collection of disjoint, task-specific solutions. Each semantic category (concept transformation, stylization, motion transfer, camera control) has its own model architecture, training recipe, or inference protocol, and there is no single model that can handle them all—let alone generalize to unseen semantic conditions.

The paper articulates this problem in Section 1 with unusual clarity. Structure-controlled methods like VACE [34] thrive because the condition (e.g., a depth map) and the target video share exact spatial correspondence: the depth value at position (i, j) describes the geometry at position (i, j) in the output. This pixel-aligned prior makes control straightforward—you can inject the condition via residual addition, a ControlNet-style branch, or concatenation, and the model naturally learns to exploit the spatial mapping. But semantic conditions break this assumption. A reference video showing "Ghibli style" or "expand like a balloon" has completely different content, layout, and subjects from the target video. The shared property is abstract: the manner of stylization, the pattern of motion, the trajectory of camera movement. There is no spatial correspondence to anchor the control signal.

The paper identifies that prior work falls into two inadequate paradigms for handling this mismatch:

Condition-Specific Overfit (Figure 2b). Methods in this category fine-tune the entire video diffusion model backbone or attach LoRA adapters [27] for each individual semantic condition—e.g., one LoRA for "Ghibli style," another for "Hitchcock zoom," another for "turn into Ladudu." Each condition requires its own training run, its own stored weights, and produces a model that works only for that specific semantic. The cost scales linearly with the number of conditions, making this approach economically prohibitive for any platform wanting to offer hundreds or thousands of semantic effects. Moreover, these models cannot generalize: a LoRA trained on "Minecraft style" cannot generate "Simpsons style" even though both are stylization tasks.

Task-Specific Design (Figure 2c). These methods craft specialized architectural modules or inference pipelines for a category of semantics (e.g., a dedicated style transfer branch, a camera control adapter, a motion feature extractor). StyleMaster [78] builds a style-specific encoder and injection mechanism. ReCamMaster [2] and SynCamMaster [3] design camera-specific trajectory modeling. MotionDirector [84] and FlexIAct [82] create motion-specific feature representations and transfer pipelines. While more efficient than per-condition overfitting (one design covers multiple conditions within a category), each category still requires its own architecture and training procedure. These approaches cannot handle a new semantic category without engineering a new module, and they provide no zero-shot bridge between categories.

The paper quantifies this fragmentation in Section 3.2: if we have $n$ condition types with conditions $c \in C_i$ (total $m$ conditions), prior methods require up to $m$ separate models (condition-specific overfit) or $n$ separate architectures (task-specific design). The core gap is the absence of a single model $u_\Theta$ that jointly learns $p(x \mid c)$ for any semantic condition $c \in C = \bigcup_{i=1}^n C_i$ without per-condition retraining or per-task modules.

A concurrent work, Omni-Effects [49], attempts unification via a LoRA mixture-of-experts—each LoRA expert handles one semantic condition, and a router selects which expert to activate. But this still fundamentally overfits subsets of parameters to each condition and, crucially, fails to generalize to unseen semantics. The paper positions Omni-Effects as evidence that simply composing condition-specific modules under a routing mechanism does not solve the unification problem—you need a paradigm that treats semantic control as a general capability rather than a collection of specialized skills.

Why Is This Problem Important?

Practical Impact: The Economics of Controllable Video Generation. The paper makes an implicit but powerful economic argument. Creative tools for visual effects, video stylization, motion imitation, and camera control are in high demand across film production, social media content creation, advertising, and gaming. Current commercial platforms (Kling [40], Vidu [68], Pika [55], PixVerse [56]) provide these capabilities through dedicated interfaces—each effect is a separate feature with its own backend model or finetuning pipeline. This architecture does not scale. Adding a new semantic effect requires collecting paired data, training a new model or LoRA, and deploying it—a process measured in weeks or months and significant GPU-hours. VAP promises a single deployed model that can be prompted with any reference video to produce any semantic effect, including ones not seen during training. This shifts the cost structure from per-condition deployment to per-query inference, which is transformative for platforms wanting to offer rapid iteration on creative tools.

Scientific Significance: Understanding What Makes In-Context Control Work. Beyond the practical motivation, this paper addresses a deeper scientific question: under what conditions can in-context learning transfer from language and images to video, and what architectural priors need to be removed to make it work? The observation that structure-controlled video generation works well with residual addition but semantic-controlled generation fails with the same mechanism is not just an engineering inconvenience—it reveals something fundamental about how Diffusion Transformers process conditional information. The paper's analysis in Section 3.3-3.4 shows that two factors are critical: (1) the architectural mechanism for fusing condition and target representations (residual addition assumes pixel-wise correspondence; full attention with independent normalization supports abstract semantic transfer), and (2) the positional encoding scheme (shared RoPE imposes a spurious spatial mapping prior; temporally biased RoPE removes it). These findings contribute to our understanding of what DiTs learn from position encodings and how attention-based conditioning differs from additive conditioning—insights that generalize beyond video generation to any multimodal diffusion model.

Catalyzing Data-Driven Research. The paper constructs and releases VAP-Data, the largest semantic-controlled video generation dataset (100K+ paired videos across 100 conditions). Prior to this, researchers working on semantic video control had to collect their own small datasets for each condition type, making systematic comparison impossible and limiting progress to small-scale proof-of-concepts. VAP-Data provides a standardized benchmark that can catalyze follow-up work, much as how the structure-controlled community benefited from standardized benchmarks and datasets [34, 37].

Where Do Prior Approaches Fall Short?

The paper provides a detailed critique organized by paradigm. Let me walk through each one systematically, since understanding these failure modes is essential to appreciating why VAP's design choices are necessary.

Structure-Controlled Methods Applied to Semantic Control (Figure 5a)

The most straightforward baseline is to take a state-of-the-art unified structure-controlled model like VACE [34], feed the semantic reference video as the condition, and let the model's existing conditioning mechanism handle it. This fails catastrophically, and understanding why it fails is the paper's first critical insight.

Structure-controlled models assume pixel-aligned correspondence between condition and target. Their conditioning mechanisms—typically an extra encoder branch whose features are injected via residual addition at each DiT layer—are designed to transfer spatial information: "the depth here means the surface should be this far; the pose keypoint here means the limb should be in this position." When you feed a semantically-related but pixel-misaligned video through this same mechanism, the model tries to enforce a spatial mapping that doesn't exist.

The result, shown in Figure 5a and quantified in Table 1, is copy-and-paste artifacts: the model reproduces appearance, layout, and structure from the reference video rather than extracting only the abstract semantics. The quantitative evidence in Table 1 is stark: VACE with the original reference video as condition achieves a semantic alignment score of only 35.38 (on a 100-point scale) and receives a 0.6% user preference rate—essentially, humans almost never prefer its outputs. When the reference is abstracted to depth or optical flow (removing appearance information), scores improve (43.35, 46.71 respectively) because the pixel-wise prior weakens—the model can no longer copy appearance, so it is forced to extract what remains (mostly motion and structure). But even the best-performing structure-based variant (VACE with optical flow, 46.71 alignment score) is substantially below VAP (70.44) and condition-specific methods (68.60).

The paper's interpretation (Section 4.4) is important: "As control moves from raw video, depth to optical flow, appearance detail decreases, and metrics improve, confirming that the pixel-wise prior is ill-suited for semantic-controlled generation." This is not just empirical observation—it is a mechanism diagnosis. The monotonic improvement from video → depth → flow demonstrates that the failure is specifically caused by appearance-level spatial correspondence, not by an inherent inability of the model to process video conditions.

Condition-Specific Overfit: LoRA Per Condition (Figure 5b analog)

The dominant community practice (documented on platforms like Civitai [11] and in works like VFX Creator [47]) is to fine-tune a LoRA adapter for each semantic condition and swap adapters at inference time. The paper treats this as a strong baseline by training a separate LoRA for each of the 24 benchmark conditions and reporting the average performance.

The results in Table 1 reveal a characteristic tradeoff: CogVideoX-I2V with per-condition LoRA achieves a high semantic alignment score (68.60, second only to VAP and commercial models), but its video quality metrics degrade compared to the base model. Motion smoothness drops from 98.48 to 98.34, aesthetic quality from 56.75 to 54.23. This is classic overfitting behavior: the LoRA learns to produce the target semantic but at the cost of distorting the base model's generation quality, likely because the limited LoRA capacity forces it to learn spurious shortcuts rather than genuine semantic transfer.

More critically, this approach has three structural failures that VAP explicitly addresses:

1. No unification: Each condition requires a separate model. The benchmark alone would require 24 stored LoRA weights; a production system with 100+ effects would require managing and loading 100+ adapters.

2. No generalization: A LoRA trained on "Ghibli style" cannot generate "watercolor style" or any unseen stylization. The knowledge of "what stylization means as a general operation" is not learned—only the specific mapping from any input to "Ghibli output."

3. No cross-task transfer: Training on concept transformation provides no benefit for motion transfer. Each LoRA learns in isolation.

The paper positions VAP as solving all three: one model handles all conditions, generalizes to unseen ones, and benefits from cross-task training data.

Task-Specific Architectural Designs

StyleMaster [78], ReCamMaster [2], MotionDirector [84], FlexIAct [82] and others each design specialized mechanisms for their respective semantic categories. The paper acknowledges these are effective within their domains—StyleMaster produces high-quality stylization, ReCamMaster achieves precise camera control—but argues they are fundamentally non-unifiable because each relies on category-specific inductive biases.

For example, StyleMaster encodes reference style into a specialized style code and injects it through adaptive instance normalization layers. This works for style because style is spatially uniform—every pixel should receive the same stylistic treatment. But motion transfer requires spatially-varying, temporally-coherent feature displacement, which adaptive normalization cannot represent. Similarly, ReCamMaster models camera trajectory as an explicit geometric transformation (rotation + translation), which is appropriate for camera control but meaningless for concept transformation like "turn into a balloon animal." Each task-specific design makes assumptions that are valid for its category but invalid for others.

The consequence is that building a system supporting all four categories requires implementing and maintaining four separate architectures, four training pipelines, and four inference stacks. Adding a new category (say, "lighting control" or "physics-based effects") requires starting from scratch.

The Naive In-Context Baseline: Concatenate and Fine-tune (Figure 5b)

Given that structure-controlled video generation has been unified through in-context approaches (FullDiT [37], long-context tuning [23]), a natural baseline is to simply concatenate the reference and target video tokens along the sequence dimension and fine-tune the DiT on the combined sequence. This is what the paper calls "Single-Branch Finetuning" ($u^s_\Theta$) in the ablation study (Table 2, 7).

This approach suffers from catastrophic forgetting: the DiT was pre-trained only for generation (denoising), not for in-context conditioning. When fine-tuned on the relatively small VAP-Data (100K pairs), the model loses its generation capability because (1) the training signal is dominated by the conditioning task, which is fundamentally different from the pre-training objective, and (2) semantic in-context generation without pixel-aligned priors is much harder than structure-controlled in-context generation, requiring the model to discover the abstract semantic mapping from scratch.

The quantitative evidence: Single-Branch Finetuning achieves a CLIP score of 23.03 vs. VAP's 24.13 and a semantic alignment score of 68.74 vs. VAP's 70.44 (Table 2). The gaps are consistent but not enormous—the more concerning issue is that this approach cannot be plugged into arbitrary pre-trained DiTs without destroying their base capabilities. VAP's MoT design solves this by freezing the backbone and routing control through a separate expert, preserving generation quality.

Text-Prompt Only Baselines

The paper also evaluates the base CogVideoX-I2V model driven only by text prompts that describe the desired semantics (e.g., "a young woman in Ghibli style"). This achieves reasonable video quality (98.48 motion smoothness, 56.75 aesthetic quality) but catastrophically poor semantic alignment (26.04 alignment score—the lowest of all methods). This is because many semantic concepts are extremely difficult to describe precisely in text. How do you textually specify the exact motion pattern of "shake it dance" or the precise visual characteristics of "bloom magic style"? The reference video provides a dense signal that text fundamentally cannot capture, motivating the entire video-as-prompt paradigm.

How Does This Paper Position Itself Relative to Existing Work?

The paper's positioning operates on multiple levels, each building on the previous:

Level 1: Paradigm Shift from Condition-Specific to In-Context. The paper's title—"Video-As-Prompt"—encapsulates its core thesis: semantic control should be reframed as a prompting problem rather than a conditioning problem. This draws a direct analogy to how large language models use in-context examples as prompts rather than requiring fine-tuning for each task. Just as you can show GPT-4 a few examples of a new task format and it generalizes, VAP can be shown a reference video demonstrating a semantic effect and it transfers that effect to a new subject. This framing is more than rhetorical—it has concrete architectural implications:

• Conditioning assumes a fixed set of known condition types with dedicated processing pathways. Prompting assumes an open-ended set of possible conditions processed through a general mechanism.
• Conditioning typically uses asymmetric architectures (condition encoder → generator decoder). Prompting uses symmetric architectures (reference and target processed through the same transformer with full bidirectional attention).
• Conditioning is evaluated on in-distribution conditions seen during training. Prompting is evaluated on both in-distribution and zero-shot conditions, since the mechanism is general.

The paper explicitly connects to in-context learning in diffusion models, citing OminiControl [66] for images and FullDiT [37] for structure-controlled video. VAP extends this lineage to semantic video control, which the paper argues is qualitatively harder because the absence of pixel-aligned priors forces the model to learn a more abstract mapping.

Level 2: Architectural Innovation for Semantic Transfer. Within the in-context paradigm, VAP makes two architectural contributions that distinguish it from prior in-context video generation work:

Mixture-of-Transformers (MoT) for plug-and-play control. Prior in-context video work (FullDiT [37]) fine-tuned the full DiT backbone on concatenated sequences. This works for structure-controlled generation because the pixel-aligned prior provides strong training signal—the model quickly learns to exploit spatial correspondence. For semantic control, the paper shows this leads to catastrophic forgetting (Section 3.3). The MoT design—freezing the backbone and adding a trainable parallel expert with independent layer-norm and feed-forward paths—prevents this by isolating the new capability (in-context semantic interpretation) from the existing capability (video generation). The expert and backbone communicate only through full attention, which the paper argues is sufficient for semantic transfer while protecting generation quality.

This borrows from Mixture-of-Transformers [44] in multimodal LLMs but adapts it to a fundamentally different setting: in multimodal LLMs, MoT typically separates modalities (text expert, image expert); in VAP, MoT separates roles (reference interpreter vs. target generator) within the same modality.

Temporally biased RoPE. This is perhaps the paper's most subtle contribution. RoPE [65] encodes position by rotating query and key vectors based on their spatial and temporal coordinates. In standard video DiTs, all tokens in a sequence share the same RoPE basis—a token at spatial position (i,j) and temporal position t gets the same positional encoding regardless of whether it's a reference or target token. The paper identifies that this imposes a spurious correspondence: a reference token at (i,j,t) and a target token at (i,j,t) receive identical positional encodings, causing the attention mechanism to treat them as spatially corresponding—exactly the inappropriate pixel-wise prior that breaks semantic control.

The solution is elegant: shift the reference video's temporal indices backward by a fixed offset $\Delta$, placing all reference tokens before all target tokens along the time axis while keeping spatial indices unchanged. This removes the spatial correspondence prior (reference and target tokens at the same spatial position no longer share position encodings because their temporal coordinates differ by $\Delta$) while preserving the temporal ordering that in-context generation expects (reference comes before target, matching the natural left-to-right processing of video sequences).

The paper validates this choice through ablation (Table 2): identical RoPE ($u^i_\Theta$) achieves 68.98 semantic alignment vs. VAP's 70.44; adding a width shift ($u^n_\Theta$, placing reference to the left of target spatially) actually degrades performance to 69.05 because it "increases the difficulty of spatial referencing"—the model needs to attend to arbitrary spatial positions in the reference regardless of where the target token is spatially, so biasing attention toward the left hurts.

Level 3: Dataset as Contribution. The paper explicitly positions VAP-Data as more than just training data—it is an argument that unified semantic control is a data problem, not an architecture problem. The paper's thesis (articulated in Appendix D.1) is: "We show that we can train a single generalist model to learn the unified underlying principle of semantic control by showing it various examples from disparate specialist models." In other words, if you have enough diverse examples of semantic transfer—even if those examples are generated by condition-specific commercial models and LoRAs—a sufficiently expressive architecture can extract the common principle and generalize beyond its training sources.

This is a bold claim. VAP-Data is entirely synthetic, generated by commercial APIs (Kling, Vidu) and community LoRAs. The paper acknowledges this limitation explicitly (Appendix D.2) but argues that the zero-shot results validate the approach: VAP trained on synthetic data from known condition types generalizes to unseen condition types from VFX Creator [47] (crumble, dissolve, levitate, melt—see Figure 7). This suggests that the "unified principle of semantic control" the model learns is genuine and not merely an artifact of memorizing training templates.

The dataset's scale is also part of the positioning: 100K pairs across 100 conditions makes it the largest semantic-controlled video dataset by a wide margin. Prior datasets were typically hundreds of videos for a single condition type. This scale is what enables the in-context approach—in-context learning is fundamentally data-hungry, and without VAP-Data, training a generalist semantic control model would be infeasible.

Level 4: Unified vs. Commercial. The paper positions VAP against commercial models (Kling, Vidu) not as a superior system but as a comparable unified alternative. The user study (Table 1) shows VAP at 38.7% preference rate vs. 38.2% for Kling/Vidu—statistically indistinguishable. But Kling and Vidu achieve this through condition-specific implementations (dedicated interfaces and backend models for each effect), while VAP does it with a single unified model. The paper frames this as proof that unification doesn't require sacrificing quality—you can have generality without compromising on output fidelity.

Level 5: Zero-Shot as the Ultimate Test. The paper's most ambitious positioning is around zero-shot generalization (Section 4.4, Figure 7). VAP is evaluated on semantic conditions that do not appear in VAP-Data—crumble, dissolve, levitate, melt—taken from VFX Creator [47]. The model succeeds in transferring these abstract semantic patterns to new subjects, demonstrating that it has learned the general operation of semantic transfer rather than a collection of condition-specific mappings. This is positioned as the key differentiator from all prior work: condition-specific methods cannot do zero-shot by definition (they're trained for one condition), task-specific methods cannot do cross-task zero-shot (they're designed for one category), and even the concurrent Omni-Effects [49] mixture-of-experts cannot generalize because each expert only handles its trained condition.

The success of zero-shot transfer is attributed to three design choices working in concert: (1) the video-as-prompt formulation treats all semantics uniformly, so the model doesn't need to know the condition type in advance; (2) the MoT architecture preserves the backbone's generation ability, so the model can apply its general video generation knowledge to novel semantic contexts; (3) the temporally biased RoPE removes spurious priors that would otherwise cause the model to default to pixel-wise copying when confused by an unfamiliar semantic.

In summary, the paper positions itself at the intersection of several research trajectories—in-context learning in diffusion models, unified controllable generation, and video diffusion architectures—and argues that the key insight (treating reference videos as prompts with appropriate architectural support for abstract semantic transfer) is what enables the jump from fragmented task-specific solutions to a truly unified and generalizable framework.

3. Technical Approach

3.1 Reader Orientation

VAP is a unified video generation system that takes a reference video showing a desired semantic effect (e.g., Ghibli style, expand-like-balloon motion, Hitchcock zoom) and an image of a new subject, and produces a video where that subject undergoes the same semantic transformation—all using a single model rather than separate models for each effect type. The system solves the semantic transfer problem: how to extract an abstract, non-spatial property (style, motion pattern, camera trajectory, concept transformation) from one video and apply it to completely different visual content, without the model mistakenly copying pixels, layout, or appearance from the reference.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components connected in a pipeline:

1. Video VAE Encoder/Decoder: Compresses raw video frames into a compact latent representation (and decompresses generated latents back to pixels), reducing the computational cost of processing video inside the diffusion transformer.

2. Frozen Pre-trained Video Diffusion Transformer (DiT): A large text-to-video or image-to-video generation model (CogVideoX-I2V-5B or Wan2.1-I2V-14B) whose weights are completely frozen. It processes the noisy target video latents, target caption text tokens, and target image tokens to perform the denoising that generates the output video. Freezing it preserves all pre-trained generation capabilities (visual quality, temporal coherence, text alignment).

3. In-Context DiT Expert (Trainable): A second transformer initialized as a copy of (or subset of layers from) the frozen DiT, running in parallel. It processes the reference video latents and reference caption text tokens. It has its own independent query/key/value projections, feed-forward networks, and layer normalization—meaning it learns to interpret reference semantics without interfering with the frozen DiT's generation parameters.

4. Mixture-of-Transformers (MoT) Full Attention: At every transformer layer, the reference expert's representations and the frozen DiT's target representations are concatenated and processed through a single bidirectional attention operation. This is the only communication channel between the two branches. It allows the expert to condition the target generation on reference semantics synchronously at every layer, while the independent feed-forward paths ensure the two branches maintain separate "skills."

5. Temporally Biased Rotary Position Embedding (RoPE): Before tokens enter the transformer, they are assigned position encodings. The reference video tokens receive temporal indices shifted backward by a fixed offset $\Delta$, placing them before all target video tokens in the time dimension, while spatial indices remain unchanged. This removes a harmful prior that would otherwise cause the model to assume pixel-wise correspondence between reference and target positions.

Information flow (test-time inference): A reference video and caption + a target image and caption enter the system → the VAE encodes both videos into latents → temporally biased RoPE assigns position encodings → the reference latents + caption flow through the trainable DiT Expert, while noisy target latents + caption flow through the frozen DiT → at each of N transformer layers, full attention lets the target tokens attend to reference tokens bidirectionally → after all layers, the frozen DiT outputs a denoised latent → the VAE decoder produces the final video.

3.3 Roadmap for the Deep Dive

• First, the mathematical formulation of semantic-controlled video generation as a conditional distribution $p(x \mid c)$ with a unified conditioning space $C$, which defines what "unified" means operationally and shows the scaling problem with prior approaches.
• Second, the reference-video-as-prompt paradigm and the specific semantic categories studied (concept, style, motion, camera), since these define the training data taxonomy and the scope of unification.
• Third, the Mixture-of-Transformers architecture in full detail—how the frozen backbone and trainable expert are connected, why this prevents catastrophic forgetting, and how the ablation study validates each design choice.
• Fourth, the temporally biased RoPE scheme—what problem shared RoPE causes, the precise mechanics of the temporal shift, and why spatial indices are deliberately left unchanged.
• Fifth, the training procedure on VAP-Data: how the paired dataset was constructed, the training objective, hyperparameters, and the scalability properties observed.
• Sixth, inference-time mechanics: how the model is invoked with a reference video + target image + captions, the denoising process, classifier-free guidance, and the computational cost.

3.4 Detailed, Sentence-Based Technical Breakdown

This is an architectural and systems paper whose core idea is that semantic-controlled video generation can be unified by: (1) reframing all semantic conditions as in-context "video prompts" processed through a symmetric transformer architecture with full bidirectional attention, rather than through task-specific conditioning branches; (2) isolating the new in-context capability in a plug-and-play expert that communicates with a frozen backbone only via attention, preventing catastrophic forgetting; and (3) removing the harmful pixel-wise correspondence prior that shared position encodings would otherwise impose, using a simple temporal offset.

---

The Unified Conditional Distribution Formulation

The paper formally defines the unification problem in Section 3.2. Let $C$ be the set of all semantic condition types, partitioned into $n$ categories $C_1, C_2, \ldots, C_n$ (concept, style, motion, camera), where each category $C_i$ contains multiple specific conditions $c$ (e.g., "Ghibli style," "Minecraft style," "balloon expansion," "Hitchcock zoom"). The total number of conditions is $m$.

Prior approaches model the conditional distribution $p(x \mid c)$ for each condition separately—either by training $m$ distinct models (condition-specific overfit, one per LoRA) or $n$ distinct architectures (task-specific design, one per category). The paper's goal is a single model $u_\Theta$ that jointly learns:

$p(x \mid c) \quad \text{for any } c \in C = \bigcup_{i=1}^{n} C_i$

where $x$ is the target video, $c$ is any semantic condition from any category, and $\Theta$ are the model parameters.

What this formulation implies: The model must learn a general mechanism for semantic transfer that works across heterogeneous condition types, rather than memorizing condition-specific mappings. The same forward pass—same parameters, same architecture, same inference procedure—must handle a concept transformation ("turn into a balloon animal") and a camera movement ("dolly zoom") and a style transfer ("watercolor painting"), discriminating between them based solely on the content of the reference video and captions.

Why this form matters: This definition makes explicit why prior approaches fail the unification criterion. If you need to know which category a condition belongs to before selecting the model or module, you haven't achieved unified control. The conditional distribution $p(x \mid c)$ does not condition on the type of $c$—it conditions only on $c$ itself. This forces the architecture to treat all conditions through the same pathway, which is what enables zero-shot generalization: an unseen condition is just another $c'$ fed through the same $u_\Theta$, with no mechanism for the model to "know" it hasn't seen this specific condition before.

The paper also conditions on captions $P_{ref}$ (describing the reference video) and $P_{tar}$ (describing the target video), so the full learned distribution is:

$p(x \mid C_{co}, C_s, C_m, C_{ca}, P_{ref}, P_{tar})$

where $C_{co}$, $C_s$, $C_m$, and $C_{ca}$ represent the concept, style, motion, and camera conditions respectively—but crucially, the model sees these as undifferentiated "reference video content," not as labeled categories. The captions help the model identify which semantic property to transfer by explicitly mentioning the shared attribute (e.g., both reference and target captions might contain "liquid metal covers the subject").

---

Reference Videos as Task-Agnostic Prompts

Section 3.2 defines the four semantic categories that VAP unifies. These are not chosen arbitrarily—each represents a qualitatively different type of semantic control that prior work handled with entirely separate approaches:

Concept-Guided Generation ($C_{co}$). Videos sharing a conceptual transformation, further divided into two subtypes:

• Entity Transformation (24 conditions in VAP-Data): The subject changes its fundamental nature—a person becomes a Ladudu doll, a Minecraft character, a cartoon figure, or undergoes gender swap, muscle growth, hair color change, etc. The transformation is identity-changing: the output subject belongs to a different semantic class than the input subject.
• Entity Interaction (21 conditions): An external entity or effect interacts with the subject—aliens arrive, a couple drops in, liquid metal covers the body, a watermelon is thrown, the soul departs. The subject's identity is preserved, but the scene changes dramatically through the introduced interaction.

Style-Guided Generation ($C_s$). Videos rendering the same scene in a reference artistic style (11 conditions): Ghibli animation, American comic, BJD doll aesthetic, claymation, Simpsons style, JoJo style, irasutoya (Japanese illustration), painting, bloom magic, sakura season. Style transfer is identity-preserving: the subject remains recognizable, but the entire visual appearance (color palette, texture, edge treatment, rendering technique) transforms to match the reference aesthetic.

Motion-Guided Generation ($C_m$). Videos where the subject moves according to a reference motion pattern, despite having completely different appearance, proportions, and skeleton (41 conditions total):

• Human Motion Transfer (16 conditions): Dancing (shake it dance, hip twist), emotional expressions (crying, laughing), actions (flying, walking forward, break glass). The reference might show a specific human performing a dance; the target is a different subject (possibly non-human, like a pet) performing the same motion.
• Non-human Motion Transfer (16 conditions): Physical transformations—object expands like a balloon, crushes flat, dissolves, explodes, rotates, squishes, grows wings. These are often physics-defying effects applied to arbitrary subjects.

Camera-Guided Generation ($C_{ca}$). Videos following a reference camera trajectory (12 conditions): basic translations (move up/down/left/right), zoom in/out, orbit, dolly zoom, and the complex Hitchcock zoom (simultaneous dolly + zoom that keeps the subject size constant while the background perspective changes dramatically).

What makes these "task-agnostic prompts": The paper's key design choice is that all these conditions enter the model through the same interface: a reference video and its caption. There is no condition-type classifier, no routing mechanism, no category-specific preprocessing. The model sees raw video pixels (encoded as latents) and text tokens. It must infer from the content what kind of semantic to extract—is the reference showing a style that should be applied uniformly? A motion that should be retargeted to the new subject's proportions? A camera movement that should be applied to the 3D scene understanding? This inference happens implicitly through the transformer's attention mechanism, which is exactly what makes it "in-context": the model learns to attend to different aspects of the reference depending on what the captions indicate and what the target content requires.

The caption design choice: The paper uses "standard video descriptions" as captions (e.g., "A static Grogu is centered… a viscous, reflective gold liquid appears on the forehead…") rather than instruction-style captions (e.g., "apply the liquid metal effect to the subject"). This is explained in Section 3.2 and Appendix E.1 as a deliberate strategy to "stay close to the original DiT distribution"—the pre-trained DiT was trained on descriptive captions, so using the same caption style prevents distribution shift that could degrade generation quality. The shared semantics between reference and target captions (both mentioning "liquid metal" or "gold liquid") are what the model uses to identify which aspect of the reference to transfer. The paper acknowledges (Section 5, Limitations) that instruction-style captions might more effectively capture intended semantics and is left to future work.

---

Mixture-of-Transformers (MoT) for Plug-and-Play In-Context Control

This is the architectural core of VAP (Section 3.3, Figure 4). The design addresses a specific failure mode: fine-tuning a pre-trained DiT directly on concatenated reference+target sequences causes catastrophic forgetting because the model was pre-trained only for generation (denoising video tokens given text), not for in-context conditioning (interpreting a reference video and transferring its semantics). The MoT design solves this by physically separating the new capability from the pre-trained capability while allowing them to communicate.

Tokenization and input construction. Before entering the transformer, all inputs are tokenized:

1. Reference video: Encoded by the VAE encoder into latents $\hat{c} \in \mathbb{R}^{n' \times h' \times w' \times d}$, where $n'$ is the number of latent frames, $h' \times w'$ is the latent spatial resolution, and $d$ is the latent channel dimension. For CogVideoX-I2V-5B, videos are resized to $480 \times 720$ (or 832) and sampled at 49 frames at 16 fps.

2. Target video (noisy during training, pure noise during inference): Encoded to $x \in \mathbb{R}^{n' \times h' \times w' \times d}$ with the same VAE. During training, this is the ground-truth target video corrupted with noise according to the flow matching schedule; during inference, it is initialized as Gaussian noise $x_0 \sim \mathcal{N}(0, 1)$.

3. Reference image: The first frame of the reference video is also encoded, providing the initial appearance and subject that the target generation should start from. This leverages the Image-to-Video backbone's pre-trained ability to condition on a first frame.

4. Captions: Reference caption $P_{ref}$ and target caption $P_{tar}$ are tokenized into $\hat{t}_c, t_x \in \mathbb{R}^{n_t \times d}$ text embeddings, where $n_t$ is the number of text tokens.

These are concatenated into a single sequence: $[\hat{t}_c, \hat{c}, t_x, x]$ plus the encoded reference and target first frames (concatenated with their respective video tokens). The "in-context" nature comes from this concatenation: the reference tokens serve as a prefix that the model can attend to when processing the target tokens, analogous to how in-context examples are prepended to a query in language models.

The MoT mechanism (parallel expert with independent paths). The key architectural decision is not to process this entire concatenated sequence through a single transformer, but to split processing between two parallel transformers at each layer:

• Frozen DiT (processing $[t_x, x]$ and target image tokens): This is the original pre-trained video diffusion transformer with all weights frozen. It has its own query projection $Q_{frozen}$, key projection $K_{frozen}$, value projection $V_{frozen}$, feed-forward network $\text{FFN}_{frozen}$, and layer normalization $\text{Norm}_{frozen}$. Freezing means no gradients flow through these parameters during training—they are read-only.

• In-Context DiT Expert (processing $[\hat{t}_c, \hat{c}]$ and reference image tokens): This is a second transformer initialized as a copy of the frozen DiT's weights. It has its own independent $Q_{expert}$, $K_{expert}$, $V_{expert}$, $\text{FFN}_{expert}$, and $\text{Norm}_{expert}$—all trainable. The expert is trained from scratch to interpret reference videos and extract transferable semantic information.

At each transformer layer, after both branches compute their independent queries, keys, and values, full attention is performed over the concatenated sequence:

$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$

where:

• $Q = [Q_{expert}; Q_{frozen}]$ is the concatenation of queries from the expert (for reference tokens) and the frozen DiT (for target tokens),
• $K = [K_{expert}; K_{frozen}]$ is the concatenation of keys from both branches,
• $V = [V_{expert}; V_{frozen}]$ is the concatenation of values,
• $d_k$ is the attention head dimension, and the $\sqrt{d_k}$ factor stabilizes training by keeping the dot products from growing too large.

What "full attention" means operationally: Every target token can attend to every other target token (standard self-attention for generation), every reference token can attend to every other reference token (standard self-attention for reference understanding), and—critically—every target token can attend to every reference token and every reference token can attend to every target token (bidirectional cross-attention). This bidirectional communication is what the paper identifies as essential: the reference expert can query the target's current generation state to determine what semantic guidance is needed, and the frozen DiT can query the reference to retrieve relevant semantic features. The ablation study (Table 2) validates this by showing that unidirectional cross-attention ($u^{uc}_\Theta$, where reference features are injected into the frozen DiT but the DiT cannot influence the reference representation) achieves only 67.16 semantic alignment vs. 70.44 for bidirectional full attention.

Independent layer norm and FFN: the critical isolation. After attention, each branch processes the attention output through its own independent layer normalization and feed-forward network:

• Expert branch: $\text{FFN}_{expert}(\text{Norm}_{expert}(\text{AttentionOutput}_{expert}))$
• Frozen branch: $\text{FFN}_{frozen}(\text{Norm}_{frozen}(\text{AttentionOutput}_{frozen}))$

This independent post-attention processing is what enables the plug-and-play property: the frozen DiT's layer norm and FFN are pre-trained for generation and remain unchanged, while the expert's layer norm and FFN learn the new skill of interpreting reference semantics. If the two branches shared layer norm and FFN, the training signal for reference interpretation would distort the generation parameters—exactly the catastrophic forgetting that the MoT design prevents.

Why MoT over alternatives (ablation study, Table 2):

• Single-Branch Finetuning ($u^s_\Theta$): Expanding the pre-trained DiT's input sequence to include reference tokens and fine-tuning the entire model achieves semantic alignment of 68.74 but with degraded video quality (motion smoothness 97.97 vs. VAP's 98.59). The single branch mixes generation and reference-interpretation gradients, causing forgetting of pre-trained generation skills. The paper explains this as: "DiTs are pre-trained only for generation, not in-context conditioning, and our reference/target pairs lack pixel-aligned priors, making semantic in-context generation much harder."

• Single-Branch LoRA Finetuning ($u^{sl}_\Theta$): Freezing the backbone and training only LoRA layers preserves generation quality better (motion smoothness 98.25) but the limited LoRA capacity "struggles with complex in-context generation" (semantic alignment 69.08). LoRA's low-rank updates cannot represent the rich semantic transfer mapping that full-rank expert training provides.

• Unidirectional Cross-Attention ($u^{uc}_\Theta$): Freezing the DiT, adding a new branch with the same weights, and injecting its features via cross-attention into the frozen DiT (but not vice versa) achieves only 67.16 semantic alignment. The paper's interpretation: "Layer-wise bidirectional information exchange in MoT lets the reference video-prompt representation adapt synchronously to the target tokens, improving semantic alignment." Without bidirectional flow, the reference representation is computed independently of what the target generation currently needs.

• Unidirectional Residual Addition ($u^{ua}_\Theta$): Same as cross-attention but injecting features via residual addition (the standard structure-controlled method). This performs worst of all variants (55.99 semantic alignment), confirming that "residual-addition methods rely on rigid pixel-to-pixel mapping, mismatching semantic-controlled generation." Addition assumes spatial correspondence because it adds reference features directly to target features at corresponding positions—exactly the inappropriate prior that VAP's design eliminates.

Expert layer distribution. The paper experiments with where to place the in-context expert layers within the DiT's depth (Table 7). The CogVideoX-I2V-5B backbone has 42 transformer layers. By default, VAP uses all 42 layers for the expert ($L_{odd} = \{1, 3, \ldots, 41\}$ in the table, but the default VAP actually uses all layers). Reducing expert coverage—using only odd layers, only the first half, or only first and last layers—degrades performance: "balanced feature interaction improves generation quality." The best non-full variant is odd layers ($u_\Theta(L_{odd})$) at 70.22 alignment, slightly below the full model's 70.44. The Wan2.1-I2V-14B variant, which has 40 transformer layers, uses only 10 expert layers ($L = \{0, 4, 8, 12, 16, 20, 24, 28, 32, 36\}$—evenly distributed across the depth) to match the parameter count of the CogVideoX-based version (≈5B parameters). This reduced expert coverage is noted as causing "slightly worse reference alignment."

What the MoT design achieves. The combination of (1) frozen backbone, (2) independent QKV/FFN/Norm per branch, and (3) bidirectional full attention means VAP can be plugged into any pre-trained Video DiT without modifying its weights. The generation quality is preserved because the frozen DiT continues to perform exactly the operations it was pre-trained for. The new in-context control capability is isolated in the expert, which can be trained on diverse semantic data without risk of degrading the backbone's fundamental video generation skills. The bidirectional attention is the only interface, and the paper demonstrates it is sufficient for rich semantic transfer.

---

Temporally Biased Rotary Position Embedding (RoPE)

Section 3.4 addresses a subtle but critical problem: how to encode the spatial and temporal positions of reference and target tokens so that the model can attend between them without being misled by spurious position-based correspondences.

Background: standard RoPE in video DiTs. Rotary Position Embedding (RoPE) [65] encodes position information by rotating the query and key vectors in the attention computation. For a token at 3D position $(t, h, w)$ (time, height, width), RoPE applies a rotation to each pair of feature dimensions, where the rotation angle depends on $t$, $h$, and $w$. The crucial property: the attention score between two tokens depends on their relative positions. If token A is at $(t_A, h_A, w_A)$ and token B is at $(t_B, h_B, w_B)$, the dot product $q_A^T k_B$ encodes the offset $(t_A - t_B, h_A - h_B, w_A - w_B)$.

In standard video DiT usage, all tokens in a sequence (whether they come from the reference or target video) share the same RoPE basis. A reference token at position $(t, h, w)$ and a target token at the same position $(t, h, w)$ receive identical positional encodings. This means the attention mechanism will treat these two tokens as being at the same spatiotemporal location.

The problem: shared RoPE imposes a spurious pixel-aligned prior. For structure-controlled generation (depth, pose, optical flow), this shared RoPE is actually helpful: the depth value at position $(t, h, w)$ should correspond to the generated pixel at the same position, so having identical positional encodings reinforces the correct spatial mapping.

For semantic-controlled generation, this is actively harmful. The reference video and target video have no pixel-wise correspondence. The reference might show a dog dancing; the target is a human dancing. The model should learn that the semantic information (dance motion pattern) is independent of spatial position—a hand movement in the reference at position $(t, h_1, w_1)$ might correspond to a hand movement in the target at a completely different position $(t, h_2, w_2)$. But shared RoPE tells the attention mechanism that tokens at the same spatiotemporal coordinates are "the same location," creating an inductive bias toward pixel-wise copying.

The result (visualized in Figure 5c) is that the model copies appearance, layout, and structure from the reference rather than extracting only the abstract semantics. This is the same failure mode as VACE's residual addition but caused by a different mechanism: here it's the position encoding that creates the spurious prior, not the feature injection method.

The solution: temporal bias $\Delta$, spatial unchanged. The paper introduces a simple modification: shift the reference video's temporal indices backward by a fixed offset $\Delta$, while keeping spatial indices $(h, w)$ identical for reference and target:

• Reference token positions: $(t - \Delta, h, w)$
• Target token positions: $(t, h, w)$

For implementation, $\Delta$ is chosen to be larger than the total number of temporal positions in the target sequence, ensuring that all reference frames come before all target frames in the temporal ordering. The paper does not specify the exact numeric value of $\Delta$, but the principle is: reference tokens occupy temporal positions $[-\Delta, -\Delta + n' - 1]$ and target tokens occupy $[0, n' - 1]$.

What this achieves operationally:

• Spatial correspondence prior is eliminated. A reference token at $(t - \Delta, h, w)$ and a target token at $(t, h, w)$ now have different temporal coordinates (difference of $\Delta$), so their positional encodings are different. The attention mechanism no longer sees them as being at "the same location." The model is forced to learn content-based attention patterns (attending to semantically related tokens regardless of position) rather than position-based shortcuts.

• Temporal ordering matches in-context generation expectations. Placing the reference before the target in the temporal dimension matches the natural left-to-right processing of video sequences: the reference is "the past" that the target generation should condition on. This is analogous to how causal language models attend to previous tokens but not future ones—except VAP uses full bidirectional attention (the target can also influence the reference representation), so the ordering is a soft inductive bias rather than a hard causal mask.

• Spatial indices remain unchanged. This is a deliberate choice: the spatial coordinates $(h, w)$ for reference and target tokens at the same grid position are identical. Why keep this spatial alignment? The paper's ablation ($u^n_\Theta$ in Table 2, which additionally shifts the reference's width indices to place them left of the target) shows that removing spatial alignment degrades performance (semantic alignment drops from 70.44 to 69.05). The reason: the model needs to be able to attend to any spatial position in the reference regardless of the current target token's position, so spatially biasing attention toward one side (left) makes it harder to attend to reference tokens on the right. Keeping spatial indices aligned means the attention mechanism starts from a "neutral" spatial prior—all spatial offsets are equally easy to attend to—and learns content-based spatial attention patterns from data.

Ablation validation (Table 2):

• Identical PE ($u^i_\Theta$, shared RoPE, no shift): Semantic alignment 68.98. The pixel-wise correspondence prior causes the model to default to spatial copying, reducing semantic transfer quality.
• Temporally biased RoPE (VAP, $u_\Theta$): Semantic alignment 70.44. Removing the pixel-wise prior allows the model to learn genuine semantic correspondence.
• Negative temporal + width shift ($u^n_\Theta$): Semantic alignment 69.05. Adding a width shift makes spatial referencing harder because the model's attention must overcome the spatial bias, partially undoing the benefit of the temporal shift.

Why this simple solution works when alternatives would fail. The paper could have used completely separate position encodings for reference and target (e.g., two independent RoPE bases), but this would prevent the model from learning any spatial correspondence between reference and target, which is sometimes needed (e.g., for camera control, where the direction of camera movement in the reference should map to the same direction in the target). The temporal-only shift is a minimal intervention: it removes the false prior (pixel-wise spatial correspondence) while preserving the true prior (temporal ordering) and allowing the model to discover whatever spatial relationships are appropriate for each semantic category.

The paper draws an explicit connection to findings in in-context image generation (OminiControl [66], cited in Section 3.4): "Similar to observations on Rotary Position Embedding in in-context image generation, we find that sharing position embedding between the reference condition and the target video is suboptimal." The temporal-bias solution for video is the natural extension of the spatial-bias solution for images, adapted to the additional temporal dimension of video.

---

Training Procedure and Flow Matching Objective

Flow matching formulation (Section 3.1). VAP uses the flow matching framework [46] rather than the more common DDPM (denoising diffusion probabilistic model) formulation. In flow matching:

• A clean video latent $x_1$ (the ground-truth target) is corrupted by mixing with Gaussian noise $x_0 \sim \mathcal{N}(0, 1)$ along a linear interpolation path:

$x_t = t \cdot x_1 + (1 - (1 - \sigma_{min})t) \cdot x_0$

where $t \in [0, 1]$ is the time step, $\sigma_{min} = 10^{-5}$ is a small constant to avoid division-by-zero issues at $t=0$, $x_0$ is pure Gaussian noise, and $x_1$ is the clean video latent.

• At time $t$, the intermediate state $x_t$ is a blend: when $t=0$, $x_0$ is pure noise; when $t=1$, $x_1$ is the clean video (minus the small $\sigma_{min}$ correction). For intermediate $t$, $x_t$ contains a fraction $t$ of the clean signal and $(1-(1-\sigma_{min})t)$ of the noise.

• The velocity field—the instantaneous direction of change—is:

$V_t = \frac{dx_t}{dt} = x_1 - (1 - \sigma_{min}) x_0$

What the velocity represents: $V_t$ is the vector that points from the current noisy state toward the clean state, scaled by the noise schedule. If you integrate this velocity field from $t=0$ to $t=1$, you recover the clean video from noise. The model's job is to predict this velocity given the noisy state, the time step, and the conditioning information.

• The model $u_\Theta$ is trained to minimize the mean squared error between its predicted velocity and the ground-truth velocity:

$\mathcal{L} = \mathbb{E}_{t, x_0, x_1, C} \left\| u_\Theta(x_t, t, C) - (x_1 - (1 - \sigma_{min}) x_0) \right\|^2$

where $C$ represents all conditioning information: the reference video latents, reference caption tokens, target caption tokens, and reference/target first frame tokens.

Operational meaning of the loss: For each training sample, the model takes the noisy target video $x_t$, the time step $t$, and all conditioning $C$, and predicts a velocity vector field of the same shape as $x_t$. The loss penalizes the squared Euclidean distance between this prediction and the true velocity (which is known because we know $x_1$, the clean target, and $x_0$, the noise we added). Minimizing this loss teaches the model to predict the optimal direction to move at each denoising step to recover the clean video.

Why flow matching instead of DDPM: Flow matching provides a simpler mathematical framework (linear interpolation path, direct velocity prediction) and is used by the base models (CogVideoX-I2V and Wan2.1-I2V). The paper inherits this choice from the pre-trained backbones. The prediction type is specified as "Velocity" for CogVideoX and "Flow Matching" for Wan2.1 in Table 3.

Training hyperparameters (Table 3):

Key design choices in training:

• Constant learning rate with warmup. The learning rate stays at $1 \times 10^{-5}$ after 1,000 warmup steps, with no decay. This is unusual—most diffusion training uses cosine or linear decay. The paper does not justify this choice explicitly, but it may be motivated by the plug-and-play nature of MoT: the frozen backbone doesn't need learning rate decay since it's not being trained, and the expert benefits from sustained learning rate to absorb diverse semantic conditions.

• No gradient accumulation. Batch size of 1 per GPU, effective batch size of 48 (one sample per GPU across 48 GPUs). This is the maximum feasible given the memory requirements of processing two full video sequences (reference + target) through a 5B-parameter DiT.

• 20,000 training steps on 100K samples. With 48 samples per step, the model sees approximately 960,000 samples over training—roughly 9.6 epochs over the 100K dataset. This is relatively modest training duration for a model of this scale, suggesting the in-context capability emerges quickly from the strong initialization (copy of pre-trained weights).

• Wan2.1 variant uses only 10 MoT layers. To keep the trainable parameter count comparable to the CogVideoX version (≈5B), the Wan2.1 variant distributes expert layers evenly across the 40-layer backbone at indices 0, 4, 8, ..., 36. The paper notes this causes "slightly worse reference alignment" (70.23 vs. 70.44) but benefits from Wan2.1's stronger base generation (dynamic degree 79.17 vs. 77.08, aesthetic quality 58.09 vs. 57.71).

Scalability (Table 2, 7): The paper trains VAP variants with 1K, 10K, 50K, and 100K training pairs to demonstrate that performance improves with data scale:

Every metric improves monotonically with data scale. The paper attributes this scalability to "our unified design that treats reference videos as prompts without task-specific modifications, together with the MoT framework, which preserves the backbone's generative capacity." In other words, because the architecture doesn't impose per-task bottlenecks, it can absorb more diverse training data without saturating.

---

VAP-Data Construction

While the dataset was discussed in the context of motivation and positioning in prior sections, the construction methodology (Section 4.3 and Appendix D) is a technical contribution that needs detailed explanation.

The fundamental challenge: Semantic-controlled video generation requires paired data: a reference video demonstrating a semantic effect and a target video showing the same effect applied to different content. Such pairs do not exist naturally. You cannot record a real video of "a dog turning into a balloon animal" and find a matching pair with "a cat turning into a balloon animal" that shares the exact same transformation pattern. Unlike structure-controlled data (where you can compute depth maps, optical flow, or pose from any video using off-the-shelf models), semantic pairs require either expensive manual creation or synthetic generation.

The bootstrapping strategy (Appendix D.1): The paper's solution is to treat existing specialist models as a data generation engine. The process:

1. Curate 2,000 high-quality reference images from the Internet, spanning diverse subjects: men, women, children, animals, objects, landscapes, and multi-subject cases. These images serve as the "starting point" for target videos—they represent the diverse content that the model should be able to apply semantic effects to.

2. Apply commercial and community specialist models to each reference image across all compatible semantic templates:

   • Commercial APIs: VIDU [68] templates and Kling [40] image-to-video special effects. These are closed-source, condition-specific models that can generate videos applying a specific semantic effect (e.g., "turn into Ladudu," "Ghibli style") to an input image.
   • Community LoRAs: Civitai [11] and other community-trained LoRA adapters for CogVideoX [76] and Wan2.1 [70]. These are open-source, condition-specific adapters that fine-tune open-source DiTs for individual effects.

3. Match each reference image to all compatible templates. Some templates restrict subject categories (e.g., a "gender swap" template may only work on human subjects; a "pet to human" template requires animal subjects). The paper matches each image to all templates whose subject restrictions are satisfied.

4. The specialist model's output becomes the target video; the template's "example" video becomes the reference video. For each (image, template) pair, the commercial API or LoRA generates a target video applying the effect to the reference image. The template's canonical example video—the one that demonstrates what the effect looks like—becomes the reference video for that pair. This creates a paired dataset: (reference video, target video) where both share the same semantic effect but have different content.

Dataset scale and composition (Table 6): The result is VAP-Data, with over 100,000 paired samples across 100 semantic conditions organized into four primary categories:

Test set construction: From the 100 conditions, the paper evenly samples 24 conditions across the four categories (concept, style, motion, camera), with 2 test samples per condition, totaling 48 test samples. This ensures balanced evaluation across all semantic types.

Captions for training: For each pair, captions are generated as standard video descriptions (not instructions). The paper uses descriptive captions like "A static Grogu is centered… a viscous, reflective gold liquid appears on the forehead…" matching the pre-training data distribution of the base DiT models. The reference and target captions are written to share the mention of the key semantic attribute (e.g., both mention "liquid metal" or "gold liquid"), which helps the model identify which aspect of the reference to transfer. The paper notes (Appendix E.1) that this caption style choice means the model relies on caption overlap to locate the shared semantics, and performance degrades when captions use different terminology for the same effect or when subject mismatch is extreme.

Dataset limitations (Appendix D.2): The paper is explicit that VAP-Data is entirely synthetic, derived from other generative models. This has several consequences:

• VAP may inherit stylistic biases, artifacts, and conceptual limitations of the source models (e.g., if the source models generate poor hands, VAP's training data will contain poor hands).
• The semantic conditions are limited to effects available in commercial APIs and community LoRAs, which skews toward visual effects templates popular in creative tools.
• No real-world semantic pairs exist in the training data, so the model learns only from artifacts generated by other models.

However, the paper argues this is not just an acceptable compromise but a proof of concept: "We show that we can train a single generalist model to learn the unified underlying principle of semantic control by showing it various examples from disparate specialist models." The zero-shot results on unseen conditions from VFX Creator [47] (crumble, dissolve, levitate, melt) validate this claim: the model extracts a general semantic transfer capability, not just a collection of memorized templates.

---

Inference Procedure

Inputs at test time: A user provides:

1. A reference video (demonstrating the wanted semantic effect)
2. A reference image (the first frame of the reference video, providing the initial appearance)
3. A reference caption (describing the reference video's content)
4. A target image (the new subject to apply the effect to)
5. A target caption (describing the desired output, sharing semantic keywords with the reference caption)

Denoising process:

1. The VAE encoder compresses the reference video, reference image, and target image into latents.
2. The target video latent is initialized as pure Gaussian noise $x_0 \sim \mathcal{N}(0, 1)$.
3. All inputs are tokenized, assigned temporally biased RoPE positions, and assembled into the in-context sequence $[\hat{t}_c, \hat{c}, t_x, x]$ plus image tokens.
4. The model runs $N$ denoising steps using an ODE solver (CogVideoX DDIM for CogVideoX-based VAP, Flow Euler for Wan2.1-based VAP):
   • Sample steps (Table 3): 50 steps for CogVideoX-based VAP, 30 steps for Wan2.1-based VAP.
   • At each step, the model predicts the velocity field $u_\Theta(x_t, t, C)$ and the solver updates $x_t$ toward $x_{t + \Delta t}$.
5. After all denoising steps, the final latent $x_1$ is decoded by the VAE decoder to produce the output video frames.

Classifier-free guidance (CFG): The paper uses CFG with scale 6.0 for CogVideoX-based VAP and 5.0 for Wan2.1-based VAP (Table 3). CFG works by training the model to handle both conditional and unconditional generation, then at inference time interpolating between the conditional and unconditional predictions:

$u_{guided} = u_{uncond} + s \cdot (u_{cond} - u_{uncond})$

where $s$ is the guidance scale (6.0 or 5.0). A scale > 1 amplifies the conditioning signal, improving adherence to the reference semantics at the cost of potentially reducing diversity. The paper does not specify how unconditional generation is implemented—likely by randomly dropping the reference and captions during training with some probability.

Generation speed (Table 3): The paper reports inference time:

• CogVideoX-based VAP: ~540 seconds (9 minutes) on a single A100
• Wan2.1-based VAP: ~420 seconds (7 minutes) on a single A100

The Wan2.1 version is faster despite having more parameters because it uses fewer denoising steps (30 vs 50) and a simpler ODE solver (Flow Euler vs. DDIM). The paper notes that the added MoT expert and in-context full attention roughly doubles inference time compared to the base DiT running without VAP. This overhead is characterized as "acceptable" given the unified semantic control capability and the fact that it avoids retraining the backbone.

Multiple reference videos (Appendix E.2): The model supports 1–3 reference videos as input. With multiple references, "results are similar to using a single reference" overall, but the model "may blend unwanted visual details across videos" when the references differ in structure. For example, if three references for "AI Lover Drop" show a human, a spider, and a flatfish, the model mixes the spider's legs and fish's shape onto the target dog. The paper hypothesizes this leakage happens because general-purpose captions lack an explicit referent specifying which reference to follow, and suggests that instruction-style captions or a tailored multi-reference RoPE scheme could address this.

Noisy reference video prompts (Appendix F, Table 7): Inspired by Diffusion Forcing [9, 64], the paper experiments with adding noise to the reference video before feeding it to the expert ($u^{n\_ref}_\Theta$). This attempt backfires: it leads to "severe artifacts" and degrades semantic alignment (70.18 vs. 70.44). The explanation: in Diffusion Forcing, noise is added to long video generation to prevent copy-paste or static outputs. But in VAP, the reference and target videos already differ significantly in appearance and layout, so the model has no tendency to copy. Adding noise only "corrupts the contextual information and degrades generation quality."

Application modes (Appendix B): VAP supports several downstream applications that exercise different aspects of its unified capability:

1. Different semantics, same image (Figure 10): Given a fixed reference image and different reference videos (Ladudu transformation, Ghibli style, Mecha X, paper man, gender swap, cloning, zoom out, Hitchcock zoom), VAP consistently generates a new video for each semantic. This demonstrates multi-condition control from a single model.

2. Same semantics, different references, same image (Figure 11): Given different reference videos showing the same semantic (e.g., two different "expand like a balloon" examples, two different "Ghibli style" examples) and the same target image, VAP consistently generates aligned videos. This shows robustness to reference variation within a semantic category.

3. Same semantic, different images (Figure 12): Given one reference video and multiple different target images (different subjects), VAP transfers the same semantics to each image. This demonstrates subject-agnostic semantic transfer.

4. Fine-grained text-based editing (Figure 13): With a fixed reference video and image, modifying a single word in the target caption (e.g., "black" → "green" → "purple" → "red" → "golden" → "white" Ladudu) produces corresponding attribute changes while preserving identity and motion. This shows that VAP retains the text-following capability of the base DiT and can combine reference-video semantics with text-controlled refinements.

---

Summary of Design Choices and Their Justifications

• Video-as-prompt paradigm over condition-specific or task-specific approaches: Treating reference videos as in-context prompts enables a single model to handle any semantic condition through the same interface, enabling both unification (no per-condition training) and zero-shot generalization (the model learns how to transfer semantics rather than how to produce specific effects).

• Mixture-of-Transformers over single-branch fine-tuning: Freezing the backbone and isolating the new in-context capability in a parallel expert prevents catastrophic forgetting of pre-trained generation skills, which single-branch fine-tuning causes because the semantic transfer gradient conflicts with the denoising gradient.

• Bidirectional full attention over unidirectional cross-attention or residual addition: The reference representation needs to adapt to the target's current generation state (and vice versa) for optimal semantic transfer. Bidirectional attention enables this synchronous adaptation. Residual addition fails completely because it assumes pixel-wise correspondence.

• Temporally biased RoPE over shared RoPE or full spatial-temporal separation: Shifting only the temporal indices removes the spurious pixel-wise correspondence prior (caused by shared spatial positions having identical encodings) while preserving the ability to learn content-based spatial attention (by keeping spatial indices aligned) and the temporal ordering expected by in-context generation.

• Synthetic VAP-Data over real pairs: Real semantic-controlled video pairs at scale are infeasible to collect. Using commercial and community specialist models as a data engine bootstraps a large, diverse dataset that—despite being synthetic—provides sufficient training signal for a generalist model to extract the underlying principle of semantic transfer.

• Standard descriptive captions over instruction-style captions: Matches the pre-training distribution of the base DiT, preventing text-encoder distribution shift that could degrade generation quality, even though instruction-style captions might more precisely specify the intended semantic transfer.

• 50 denoising steps with CFG scale 6 (CogVideoX): Standard inference hyperparameters inherited from the base DiT, adjusted slightly for the in-context setting. The paper does not ablate these, treating them as fixed.

4. Key Insights and Innovations

Innovation 1: The Diagnostic Separation of Priors — Why Structure-Controlled Methods Fail on Semantic Control

The paper's deepest conceptual contribution is not a new architecture but a diagnostic insight that explains a previously mysterious failure mode. The field has known empirically that structure-controlled video generation methods (ControlNet-style branches, residual addition) produce artifacts when applied to semantic control tasks. But why they fail—whether the problem is insufficient model capacity, wrong training data, or something more fundamental—was not understood. The paper isolates two distinct mechanisms that independently impose inappropriate pixel-wise priors, and shows that removing both is necessary and (apparently) sufficient.

The two priors and why they're distinct. The paper identifies that structure-controlled methods embed a pixel-aligned correspondence assumption through two separate channels: (1) the conditioning mechanism itself—residual addition assumes that adding reference features to target features at the same spatial position is meaningful, which only holds when reference and target are pixel-aligned; and (2) the position encoding scheme—shared RoPE gives identical positional encodings to reference and target tokens at the same spatiotemporal coordinates, creating an attention bias toward position-based correspondence.

What makes this diagnostic powerful is that these two mechanisms are architecturally independent and can be ablated separately. The paper demonstrates this through a controlled series of experiments (Table 2, Figure 5):

• Residual addition alone (with MoT bidirectional attention but using addition instead of full attention for feature injection, variant $u^{ua}_\Theta$) crashes to 55.99 semantic alignment—the worst of all variants. This isolates the harm from the conditioning mechanism's spatial bias.
• Shared RoPE alone (with MoT full attention but identical positional encodings, variant $u^i_\Theta$) degrades to 68.98 alignment. This isolates the harm from the positional encoding's spatial bias, and it's a smaller but still meaningful degradation.
• Having both problems (implicit in the VACE baseline and the naive concatenate-and-finetune approach) produces the catastrophic failures seen in Figure 5a-5c: copy-paste artifacts, layout reproduction, appearance leakage.

The field's prior assumption—implicit in works like VACE [34], ControlVideo [83], and most structure-controlled frameworks—was that conditioning mechanisms are generic: feed any condition video through a ControlNet-style branch, and the model will learn to use whatever information is present. This paper shows that assumption is false. The mechanism matters because it encodes assumptions about the relationship between condition and target. Residual addition encodes "same position = same meaning." Shared RoPE encodes "same coordinates = corresponding locations." For semantic control, both assumptions are actively harmful, and you need to remove them at the architectural level—not just hope the model learns to overcome them through data.

Why this is a fundamental contribution, not an incremental fix. Prior work had two responses to the failure of structure-controlled methods on semantic tasks: either build task-specific architectures that don't use residual addition (StyleMaster [78], ReCamMaster [2], FlexIAct [82]), or overfit per-condition models that memorize the mapping (LoRA per effect [27, 47]). Neither approach engaged with why the generic mechanism fails. VAP's diagnosis says: the generic mechanism can work, but only if you strip out the two sources of spurious pixel-wise prior. This is a conceptual reframing of the problem from "semantic control requires specialized architectures" to "semantic control requires a generic architecture that doesn't smuggle in false assumptions about spatial correspondence."

The strength of this diagnostic is also visible in what the paper doesn't have to change. The VAE, the diffusion objective, the transformer backbone architecture, the text conditioning, the first-frame conditioning—all are inherited unchanged from the base DiT. Only the inter-stream communication mechanism (full attention vs. residual addition) and the position encoding (temporally biased vs. shared RoPE) are modified. The fact that changing only these two things enables unified semantic control across four heterogeneous categories supports the paper's claim that the priors, not the overall architecture, were the bottleneck.

Evidence anchor: The monotonic improvement in Table 1 as VACE's condition is abstracted from raw video (alignment 35.38, copying appearance) → depth (43.35, copying geometry but not texture) → optical flow (46.71, copying only motion)—combined with the architectural ablation in Table 2 showing residual addition (55.99) and shared RoPE (68.98) independently degrade VAP—provides converging evidence that the pixel-wise prior, not any other factor, is the root cause of failure.

---

Innovation 2: Semantic Control as In-Context Generation — A Unifying Paradigm That Subsumes Task-Specific Architectures

The paper makes a paradigmatic claim: semantic-controlled video generation across heterogeneous categories (concept transformation, stylization, motion transfer, camera control) is a single problem—in-context semantic transfer—rather than a collection of distinct tasks. This claim is more audacious than it might initially appear, and it's what distinguishes VAP from incremental unification attempts like Omni-Effects [49] (which unifies through a mixture-of-experts but still treats each condition as a separate skill).

What "in-context" means in this setting. In language models, in-context learning means the model conditions on a few demonstration examples prepended to the input, inferring the task from the pattern in those examples and applying it to the query—all without weight updates. VAP's analogy is direct: the reference video is the demonstration; the model must infer what kind of semantic transfer is being requested from the reference's content and the captions, then apply that transfer to the target image. Crucially, the model doesn't need to know whether this is a "style transfer" or a "motion transfer" or a "concept transformation" task. It just needs to extract whatever semantic property the reference and target captions indicate they share, and map it onto the new subject.

Why this is a genuine unification, not just a multi-task model. Prior work that handles multiple semantic categories (Omni-Effects [49]) uses a router or classifier to select task-specific modules. The knowledge of which task is being performed is explicit in the architecture. VAP eliminates this: there is no task classifier, no routing, no category-specific processing pathway. The same forward pass handles Ghibli style, balloon expansion, and Hitchcock zoom. The model discriminates between these based purely on the content of the reference video and captions—exactly as a language model discriminates between translation and summarization based on the in-context examples, without being told which task it's performing.

This matters because it enables transfer across semantic categories. A LoRA trained on "Ghibli style" learns nothing about "shake it dance." But VAP, trained on all four categories simultaneously through the same pathway, can learn that certain attention patterns are useful across categories (e.g., attending to moving regions in the reference when the captions mention motion, regardless of whether that motion is a dance or a balloon expansion). The paper doesn't directly ablate cross-category transfer (e.g., training on 3 categories and testing on the 4th), but the zero-shot results on unseen conditions (Figure 7) provide indirect evidence: conditions unseen in training (crumble, dissolve, levitate, melt) are handled successfully, suggesting the model has learned a general semantic transfer operation, not just a union of category-specific skills.

The enabling insight: captions as semantic pointers. A subtle but crucial design choice makes this in-context paradigm work: the use of shared semantic keywords in reference and target captions. The captions serve as pointers: they tell the model which aspect of the reference video to extract and transfer. Without captions, the model would have to guess what property is shared—is it the style? the motion? the camera movement? With captions that both mention "liquid metal covers the subject," the model can focus attention on the parts of the reference that demonstrate the liquid metal effect, and apply that to the target.

This is a clever instance of weak supervision through natural language. The captions don't need to explicitly say "transfer the liquid metal effect" (instruction-style). They just need to describe what's happening in both videos, and the fact that both descriptions share key terms is sufficient for the model to identify the shared semantics. It's a minimal-intervention approach that preserves the base DiT's text distribution while providing the necessary signal for in-context control.

Comparing to prior paradigms. The paper's Figure 2 visualizes the shift: Panel (a) shows the structure-controlled paradigm (pixel-aligned condition injected via residual addition); panels (b) and (c) show the fragmentation of semantic control into per-condition overfitting and per-task design; panel (d) shows VAP's unified paradigm (reference video as prompt, MoT-based in-context control). This is not just a diagram—it's a taxonomy of approaches organized by their underlying assumptions about the condition-target relationship. Structure-controlled assumes spatial correspondence. Condition-specific overfitting assumes no generalization between conditions. Task-specific design assumes categories are fundamentally different. VAP assumes that, given the right architectural support for abstract semantic transfer without spurious spatial priors, all semantic control is in-context generation.

Evidence of unification quality: The user study (Table 1) shows that VAP as a single unified model achieves 38.7% preference rate—statistically tied with Kling/Vidu's 38.2%, which are condition-specific commercial systems. This is the paper's strongest evidence that unification doesn't require sacrificing per-condition quality. If VAP were merely a "jack of all trades, master of none," it would lose to condition-specific models on user preference. That it doesn't—and that it additionally offers zero-shot generalization that condition-specific models cannot—makes the unified paradigm compelling.

---

Innovation 3: Catastrophic Forgetting as the Bottleneck for In-Context Video Diffusion — and MoT as a Surgical Solution

The paper identifies a previously underexplored failure mode in video diffusion models: catastrophic forgetting when fine-tuning pre-trained DiTs for in-context conditioning, and provides a solution (MoT with independent normalization and feed-forward paths per branch) that preserves generation quality while enabling new conditioning capabilities. This is not just an engineering trick—it's a diagnostic contribution that reveals something about how DiTs encode their capabilities.

The nature of the forgetting. The paper shows (Table 2, variant $u^s_\Theta$) that fine-tuning the full DiT backbone on concatenated reference+target sequences degrades video quality compared to VAP's MoT approach. Motion smoothness drops from 98.59 (MoT) to 97.97 (full fine-tuning). But more importantly, the paper's ablation of what gets preserved vs. degraded reveals where the forgetting occurs. The frozen backbone in VAP preserves motion smoothness, dynamic range, and aesthetic quality because these capabilities are encoded in the DiT's layer normalization and feed-forward parameters—exactly the components that MoT keeps independent per branch.

Why this is a diagnostic contribution, not just an architecture choice. The standard approach to adding capabilities to pre-trained models is either full fine-tuning (which risks forgetting) or adapter-based methods like LoRA (which preserve the backbone but have limited capacity). The paper's ablation shows that both extremes fail for in-context semantic video generation, but for different reasons:

• Full fine-tuning forgets generation quality because the in-context training signal (learning to extract semantics from a reference) conflicts with the pre-training objective (denoising given text). The model's normalization statistics and feed-forward representations, tuned for pure generation, are distorted by the new task.

• LoRA has insufficient capacity for the complexity of in-context semantic transfer. The paper doesn't specify the LoRA rank used, but the result (69.08 semantic alignment vs. 70.44 for full-expert MoT, with better quality preservation at 98.25 motion smoothness) shows that low-rank adaptation can't represent the rich cross-attention patterns needed for abstract semantic mapping. LoRA is effective for concept-specific overfitting (Table 1 shows per-condition LoRA achieves 68.60 alignment) but not for learning a general in-context capability.

The MoT insight: separation of role, not modality. Mixture-of-Transformers [44] was originally designed for multimodal models, separating processing by modality (text expert, image expert, etc.). VAP adapts it to separate by role within the same modality: the frozen DiT is the "generation specialist" and the trainable expert is the "reference interpretation specialist," both working on video tokens. This repurposing is non-obvious. The key insight is that independent layer norm and feed-forward paths per role are what prevent interference—not just independent attention parameters. The ablation of unidirectional cross-attention ($u^{uc}_\Theta$, 67.16 alignment) vs. bidirectional full attention (VAP, 70.44) shows that attention-based communication is necessary, but the independent post-attention processing is what preserves the backbone's generation quality.

Scalability as validation of the MoT design. The paper's data scaling experiment (1K → 10K → 50K → 100K pairs, Table 2) shows monotonic improvement across all metrics with no sign of saturation. This would not be possible if the expert's capacity were being consumed by having to re-learn generation fundamentals—the expert can focus entirely on semantic interpretation because the frozen backbone handles generation. This division of labor makes the model data-efficient for the new capability while preserving the data-efficiency of the pre-trained backbone for generation. It's a clean separation of concerns that the scaling results validate.

Evidence anchor: Table 2, rows comparing $u^s_\Theta$, $u^{sl}_\Theta$, $u^{uc}_\Theta$, $u^{ua}_\Theta$, and VAP across all metrics, plus the scalability rows showing monotonic improvement with data scale under the MoT design. The fact that full fine-tuning ($u^s_\Theta$) achieves only 97.97 motion smoothness vs. VAP's 98.59, while LoRA ($u^{sl}_\Theta$) achieves 98.25 but only 69.08 alignment vs. VAP's 70.44, shows that MoT occupies a Pareto-optimal point that neither alternative reaches.

---

Innovation 4: Synthetic Bootstrapping as a Research Enabler — VAP-Data and the Specialist-to-Generalist Pipeline

The paper's dataset contribution goes beyond "we collected a large dataset." The methodology for constructing VAP-Data is itself an innovation: using specialist models as a data generation engine to train a generalist model that surpasses its teachers in generality (if not in per-task quality). This is a concrete instantiation of a broader research strategy that has implications beyond video generation.

The specialist-to-generalist pipeline. The process is: (1) identify a target capability (semantic-controlled video generation) for which paired training data doesn't naturally exist; (2) collect a diverse set of specialist models that each handle one narrow slice of the capability (commercial effect templates, community LoRAs); (3) use these specialists to generate a large, diverse, paired dataset by applying them to a curated set of reference images; (4) train a single generalist model on this synthetic data using an architecture designed for in-context transfer; (5) demonstrate that the generalist exceeds the specialists in zero-shot generalization, even though every training example was generated by a specialist.

Why this is non-obvious and important. The conventional wisdom is that models trained on synthetic data inherit the limitations of their data sources—garbage in, garbage out. VAP challenges this by showing that if the architecture is designed to extract the common principle underlying the specialist outputs (semantic transfer) rather than to memorize individual mappings, it can transcend the limitations of any individual specialist. The specialist models are like teachers who each know one subject; the generalist is a student who learns how to learn subjects from watching many teachers, and can therefore handle subjects none of the teachers knew.

What makes this work: diversity of specialists + unified architecture. The paper implicitly argues that two conditions are necessary for the specialist-to-generalist pipeline to succeed: (1) the specialists must be diverse enough that their commonality (the general principle) is learnable, and their idiosyncrasies (the artifacts, biases, and limitations of each source) average out; (2) the architecture must be capable of extracting this commonality—it must have an inductive bias toward learning the general operation rather than memorizing per-source patterns.

VAP-Data satisfies the first condition by aggregating outputs from multiple commercial APIs (Kling, Vidu) and community LoRAs (Civitai) across 100 conditions. No single source dominates; the model sees each semantic effect rendered through multiple "teachers" with different stylistic signatures. VAP's architecture satisfies the second condition through the video-as-prompt formulation: because all conditions enter through the same interface, the model cannot learn per-condition shortcuts (there are no condition-specific parameters); it must learn a unified transfer mechanism.

The zero-shot results as validation. The paper's strongest evidence for the specialist-to-generalist claim is Figure 7: VAP successfully transfers unseen semantic conditions (crumble, dissolve, levitate, melt) from VFX Creator [47] to new subjects. These conditions were not in VAP-Data—no specialist model was available to generate training examples for them. Yet VAP handles them because it learned the operation of "extract abstract visual transformation pattern from reference and apply to target" rather than memorizing a catalog of known transformations. This is the acid test for generalization, and VAP passes it.

Limits and caveats. The paper is honest that VAP-Data is synthetic and may encode biases from its source models (Appendix D.2). The zero-shot conditions tested are still visual effects (crumble, dissolve, levitate, melt), not radically different semantic categories (e.g., narrative structure transfer, emotional tone transfer). The specialist-to-generalist pipeline may not extend to capabilities where the common principle is harder to extract or where specialist diversity is insufficient.

Evidence anchor: Section 4.3 and Appendix D describe the dataset construction pipeline. Figure 7 shows zero-shot results on unseen conditions. Table 6 quantifies the dataset's scale and diversity. The scalability experiment (Table 2, 1K → 100K pairs) shows that the generalist's performance improves with more specialist-generated data, supporting the claim that diversity, not just volume, drives the learning of the general principle.

Innovation 5: Verifier-Free Semantic Evaluation via VLM-as-Judge

While this is a methodological contribution rather than a core algorithmic one, the paper introduces and validates a semantic alignment metric based on VLM scoring (Gemini-2.5-Pro and GPT-5) that addresses a genuine measurement gap in controllable video generation. Standard video quality metrics (CLIP score, FVD, motion smoothness, aesthetic quality) measure generic properties but cannot assess whether a generated video faithfully reproduces a specific semantic condition from a reference. The paper's solution—providing the VLM with a structured evaluation template, reference video, and generated video—is validated against human preference (Table 5 shows strong correlation between VLM scores and user study rankings) and cross-validated across two different VLMs (Gemini and GPT-5).

Why this matters beyond this paper. The controllable generation field lacks standardized evaluation protocols for semantic alignment. Structure-controlled methods can use pixel-level metrics (MSE against ground-truth depth maps, pose keypoint accuracy) because the condition and target are aligned. Semantic-controlled methods have no such luxury—the reference and target share no pixels, so pixel-level metrics are meaningless. The paper's VLM-as-judge approach is a practical, scalable alternative to expensive user studies, and the cross-VLM validation (Gemini and GPT-5 producing closely matching scores, Table 5) provides some evidence of reliability. This is an infrastructure contribution that could accelerate research by providing a standardized, reproducible semantic alignment metric.

Caveats. VLM judges are known to have biases (position bias, verbosity bias, sensitivity to prompt wording). The paper's template is carefully designed with regime-specific criteria (ID-TRANSFORM vs. NON-ID-TRANSFORM), hard-fail conditions, and detailed rubrics, which mitigates but doesn't eliminate these concerns. The metric is validated on only 48 test samples from 24 conditions, a relatively small sample for establishing metric reliability. Nevertheless, the strong correlation with the more expensive user study (38.7% preference for VAP vs. 38.2% for Kling/Vidu in both the human study and the VLM scores) is encouraging.

Evidence anchor: Table 5, comparing semantic alignment scores from Gemini-2.5-Pro, GPT-5, and human preference rates. Table 4, showing the detailed evaluation template.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All experiments are conducted on VAP-Data, the paper's newly constructed semantic-controlled video generation dataset comprising over 100K paired videos across 100 semantic conditions organized into four primary categories: concept (entity transformation and interaction), style (stylization), motion (human and non-human motion transfer), and camera (camera movement control). The dataset is entirely synthetic, generated by applying commercial APIs (VIDU, Kling) and community LoRAs to a curated set of 2,000 reference images spanning diverse subjects. For evaluation, the paper evenly samples 24 semantic conditions from the four categories, with 2 test samples per condition, totaling 48 test samples. This is a deliberately small but balanced test set designed to cover all four semantic categories proportionally.

• Base model(s). The primary experiments use CogVideoX-I2V-5B as the frozen Video Diffusion Transformer backbone. Transferability is validated on Wan2.1-I2V-14B. CogVideoX-I2V-5B is an image-to-video diffusion transformer with 5 billion parameters, 42 transformer layers, operating at 480p resolution with 49 frames at 16 fps. The choice is motivated by the model being "representative of the capabilities of many contemporary" video DiTs and sitting in a useful capability regime: strong base generation quality (evidenced by high motion smoothness and aesthetic scores for text-only prompts in Table 1) but no built-in semantic control, leaving room for VAP's in-context conditioning to demonstrate clear improvements.

• Metrics. The paper evaluates across three axes with five metrics. Text alignment: CLIP similarity score measuring consistency between the generated video and the target caption. Overall video quality: Motion Smoothness (temporal coherence), Dynamic Degree (motion magnitude, using optical flow), and Aesthetic Quality (learned aesthetic predictor from LAION). Semantic alignment: A newly introduced metric where each (reference, generation) pair is submitted to Gemini-2.5-Pro with a detailed evaluation template (Table 4) that includes regime-specific criteria (ID-TRANSFORM for identity-changing effects vs. NON-ID-TRANSFORM for identity-preserving effects), hard-fail conditions (e.g., "REFERENCE shows an ID-TRANSFORM, but TEST lacks the transformation"), and a structured 0–100 scoring rubric. The metric is cross-validated against GPT-5 (Table 5) showing close agreement (e.g., VAP scores 70.44 from Gemini vs. 70.26 from GPT-5) and against a 20-person user study showing consistent preference rankings. User study: 20 video-generation researchers compare outputs from different methods on both semantic alignment and overall quality, with results reported as normalized preference rates summing to 100%.

• Baselines. The paper compares against five categories of baselines. (1) Structure-controlled method applied to semantic control: VACE (Jiang et al., 2025), the state-of-the-art unified structure-controlled video generation model, evaluated with three condition types—original reference video, depth map extracted from the reference, and optical flow extracted from the reference—to test whether abstracting away appearance information improves semantic control under a pixel-aligned conditioning mechanism. (2) Unconditional DiT backbone: CogVideoX-I2V-5B driven only by text captions carrying semantic cues, without any reference video conditioning. (3) Condition-specific overfit: Per-condition LoRA fine-tuning (Hu et al., 2022) on CogVideoX-I2V-5B, training a separate LoRA for each of the 24 benchmark conditions and reporting averaged performance. (4) Closed-source commercial models: Kling (Kling, 2025) and Vidu (Vidu, 2025), which provide dedicated interfaces for each semantic condition and are treated as condition-specific. (5) Ablation variants of VAP itself: Single-branch fine-tuning, single-branch LoRA fine-tuning, unidirectional cross-attention injection, unidirectional residual addition, and various RoPE configurations (detailed in Section 4.5).

• Generation budget / compute accounting. The paper does not standardize on a single computational budget metric across all comparisons, since different methods have fundamentally different computational profiles. For VAP and its ablation variants, compute is implicit in model scale (≈5B trainable parameters for the in-context expert), training data volume (100K pairs), and inference steps (50 denoising steps for CogVideoX-based, 30 for Wan2.1-based). For LoRA baselines, each condition requires a separate training run. For commercial models, compute is opaque. Inference time is reported for VAP: ~540 seconds on a single A100 for CogVideoX-based, ~420 seconds for Wan2.1-based. The paper notes this roughly doubles the base DiT's inference time due to the added MoT expert and in-context full attention, characterizing this overhead as "acceptable" but not including it in any formal efficiency metric. There is no FLOPs-matched comparison between VAP and alternatives—a notable absence given that VAP processes two full video sequences (reference + target) through parallel transformers with full attention, while LoRA-based approaches process only the target video with a lightweight adapter.

• Cross-validation / statistical protocol. The paper does not employ cross-validation for model selection or hyperparameter tuning. The test set of 48 samples (24 conditions × 2 samples each) is fixed. The semantic alignment metric is validated through two mechanisms: (1) cross-VLM validation comparing Gemini-2.5-Pro and GPT-5 scores (Table 5), showing close numerical agreement and identical model rankings, and (2) correlation with the 20-person user study (Table 1), showing that VLM scores and human preference rates produce consistent orderings (e.g., VAP and Kling/Vidu are top-ranked in both). No confidence intervals, standard deviations, or statistical significance tests are reported for any metric, which is a meaningful omission for a test set of only 48 samples.

Main Quantitative Results

Comparison Against Structure-Controlled, Condition-Specific, and Commercial Methods

The headline result appears in Table 1: VAP, as a single unified model, achieves a 38.7% user preference rate—statistically indistinguishable from the 38.2% achieved by the commercial, condition-specific models Kling and Vidu—while substantially outperforming all open-source baselines. This is the paper's central empirical claim: unification does not require sacrificing per-condition quality.

VACE baselines: the pixel-wise prior degrades monotonically with appearance information. The structure-controlled method VACE, when fed the original reference video as condition, achieves a semantic alignment score of only 35.38 and a user preference rate of 0.6%—essentially never preferred by human raters. The qualitative analysis in Figure 6 explains why: VACE copies appearance and layout from the reference (e.g., "the frog stands like the dog; the Statue of Liberty imitates a sheep"). When the reference is replaced by its depth map (removing texture and color but preserving geometry), semantic alignment improves to 43.35 with 0.7% preference. When further abstracted to optical flow (removing geometry, preserving only motion), alignment reaches 46.71 with 1.8% preference. The monotonic improvement as appearance information is stripped away confirms the paper's diagnostic: the failure is specifically caused by appearance-level pixel-wise correspondence bias in the conditioning mechanism, not by an inherent inability to process video conditions. Even the best VACE variant (optical flow, 46.71) remains substantially below VAP's 70.44, demonstrating that merely abstracting the condition is insufficient—the architectural priors must be removed at the mechanism level.

CogVideoX-I2V text-only: good video quality, catastrophic semantic alignment. The base DiT driven by text captions alone (without reference video) achieves the second-highest video quality metrics among all methods—98.48 motion smoothness and 56.75 aesthetic quality—but the lowest semantic alignment score at 26.04. The gap between this and VAP's 70.44 (a 44.4-point difference) quantifies the value of the reference video: text captions fundamentally cannot capture the dense spatiotemporal information needed for precise semantic control. Notably, text-only generation achieves a reasonable CLIP score of 22.82 (second only to commercial models' 24.05), suggesting that text-image similarity metrics are insufficient for evaluating semantic control—the generated video matches the text description in a generic sense but fails to reproduce the specific semantic pattern from the reference.

Condition-specific LoRA: quality-semantics tradeoff. Per-condition LoRA fine-tuning achieves strong semantic alignment (68.60, only marginally below VAP's 70.44) but at the cost of degraded video quality: motion smoothness drops from the base model's 98.48 to 98.34, and aesthetic quality drops from 56.75 to 54.23. This pattern—better semantic adherence at the expense of generation fidelity—is classic overfitting. Each LoRA learns to force the target semantic onto any input, but the limited low-rank capacity causes it to learn spurious shortcuts that distort the base model's generation process. The paper does not report per-condition variance, so we cannot assess whether some conditions overfit more than others. The user preference rate of 13.1% is substantially below both VAP (38.7%) and commercial models (38.2%), suggesting that human raters penalize the quality degradation more heavily than the automatic metrics capture.

Commercial models (Kling/Vidu): the ceiling for condition-specific approaches. The commercial models achieve the highest semantic alignment (74.02 vs. VAP's 70.44), the highest text alignment (24.05 vs. 24.13 for VAP—essentially tied), and the highest aesthetic quality (59.16 vs. 57.71). Their user preference rate of 38.2% is statistically tied with VAP's 38.7%. This is the paper's most important comparative result: a single unified open-source model can match—within measurement error—the user preference of dedicated commercial systems that use per-condition implementations. The remaining gaps (semantic alignment +3.58 for commercial, aesthetic quality +1.45) represent the frontier that further improvements in unified architectures might close.

Quantitative interpretation of the user study. The user study reports preference rates as "the normalized share of selections across all comparisons, totaling 100%." The distribution—VAP 38.7%, Kling/Vidu 38.2%, CogVideoX-LoRA 13.1%, CogVideoX-I2V 6.9%, VACE variants collectively ~3.1%—shows a clear bimodal split: VAP and commercial models occupy the top tier, with a large gap to the next best method. This suggests that human raters perceive a qualitative jump between "generic or overfit generation" and "genuinely semantically controlled generation," and that VAP crosses this threshold. However, the paper does not report inter-rater agreement (e.g., Fleiss' kappa), making it impossible to assess whether the 0.5 percentage point difference between VAP and Kling/Vidu is meaningful or noise.

Zero-Shot Generalization to Unseen Semantics

Figure 7 demonstrates VAP's zero-shot capability on four semantic conditions not present in VAP-Data: crumble, dissolve, levitate, and melt (from VFX Creator). The paper does not report quantitative metrics for these zero-shot results, relying instead on qualitative visualization. The generated videos show the reference image subjects undergoing the target transformations with apparent semantic fidelity—a dog crumbling into pieces, a woman dissolving into particles, a figure levitating, a subject melting—matching the abstract pattern shown in the reference videos.

What this demonstrates and what it doesn't. The zero-shot results show that VAP has learned a general operation of semantic transfer rather than memorizing a catalog of 100 known effects. This is non-trivial evidence for the in-context paradigm: the model processes the unseen reference video through the same pathway as seen conditions and extracts the transferable semantic pattern. However, the paper does not report how many zero-shot conditions were attempted, whether there were failure cases, or what the success rate is. The four shown examples may be cherry-picked from a larger set of attempts. Without quantitative zero-shot evaluation (even VLM-based semantic alignment scores on a held-out set of unseen conditions), the strength of the generalization claim cannot be calibrated.

The paper also does not test zero-shot generalization across categories—all zero-shot conditions shown are entity transformations, a subtype of concept-guided generation that VAP-Data includes extensively (24 entity transformation conditions in training). We don't know whether VAP would generalize to a radically different semantic type not represented in any training category (e.g., lighting transfer, physics simulation style, narrative pacing).

Scalability Analysis

Table 2 includes a scalability experiment where VAP is trained on subsets of VAP-Data: 1K, 10K, 50K, and the full 100K training pairs. Every metric improves monotonically with data scale:

Interpreting the scaling curve. The largest gains occur between 1K and 10K pairs: motion smoothness jumps from 92.12 to 94.89 (+2.77), dynamic degree from 60.42 to 64.58 (+4.16), and alignment from 63.91 to 66.28 (+2.37). Between 50K and 100K, gains continue but at a reduced rate: motion smoothness +1.87, dynamic degree +6.25, alignment +2.21. The curve has not clearly plateaued, suggesting that additional data might yield further improvements. The paper attributes this scalability to the unified design: "our unified design that treats reference videos as prompts without task-specific modifications, together with the MoT framework, which preserves the backbone's generative capacity while enabling plug-and-play in-context generation."

What's missing. The paper does not report scaling behavior for individual semantic categories. We don't know whether concept-guided generation, style transfer, motion transfer, and camera control all improve at similar rates with data, or whether some categories saturate early while others continue to benefit. Given that the training data distribution is imbalanced (concept: 37K videos, style: 15K, motion: 29K, camera: 19K), category-specific saturation curves would inform data collection priorities. The paper also doesn't report whether the 100 semantic conditions contribute proportionally to the gains, or whether a few high-quality conditions dominate.

Transferability Across DiT Architectures

Table 2 and Table 3 report results for VAP implemented on two different base DiTs: CogVideoX-I2V-5B (the default) and Wan2.1-I2V-14B. The Wan2.1 variant uses a distributed expert spanning only 10 of the 40 transformer layers (indices {0, 4, 8, 12, 16, 20, 24, 28, 32, 36}) to match the ~5B parameter count of the CogVideoX version. Results show:

• Wan2.1-based VAP achieves higher dynamic degree (79.17 vs. 77.08) and aesthetic quality (58.09 vs. 57.71), benefiting from Wan2.1's stronger base generation capability.
• CogVideoX-based VAP achieves slightly higher semantic alignment (70.44 vs. 70.23) and motion smoothness (98.59 vs. 97.87).
• CLIP score favors Wan2.1 (23.93 vs. 24.13—this appears to be a typo in Table 2 where CogVideoX VAP is listed as 24.13 and Wan2.1 as 23.93; Table 7 lists the same numbers, so Wan2.1 has the lower CLIP score of 23.93).

The paper's interpretation is that the reduced expert layer coverage in the Wan2.1 variant (only 1/4 of layers with in-context interaction) causes "slightly worse reference alignment" because there are fewer layers for bidirectional information exchange. The improved dynamic and aesthetic scores are attributed to the stronger base model, not to VAP's architecture. This is a qualified success for transferability: VAP works across DiT architectures, but parameter-matched variants with sparser expert coverage lose some semantic control fidelity. A fairer comparison would match expressivity (all layers with expert, requiring ~14B parameters for Wan2.1) rather than parameter count, to determine whether the transferability limitation is fundamental or a consequence of the capacity constraint.

Downstream Application Modes

Appendix B demonstrates four application modes that exercise different aspects of VAP's unified capability. These are qualitative demonstrations without quantitative metrics:

1. Multi-semantic control (Figure 10): Given a fixed reference image and eight different reference videos spanning concept, style, and camera categories, VAP produces eight distinct videos, each applying the corresponding semantic to the same subject. This demonstrates that a single model checkpoint handles heterogeneous conditions without interference.

2. Reference variation robustness (Figure 11): Given two different reference videos for the same semantic (two "Ghibli style" examples, two "expand like a balloon" examples) and the same target image, VAP produces consistent outputs aligned to the shared semantics. This suggests the model extracts the invariant semantic pattern across reference variations.

3. Subject-agnostic transfer (Figure 12): Given one reference video and multiple different target images (different subjects), VAP applies the same semantic effect to each subject appropriately. This demonstrates generalization across subject appearance, pose, and category.

4. Fine-grained text-based editing (Figure 13): With a fixed reference video and image, modifying only the color attribute in the target caption (black → green → purple → red → golden → white Ladudu) produces videos where the semantic effect is preserved but the specific attribute changes as specified. This shows VAP retains the base DiT's text-following capability and can combine it with reference-video semantics for compositional control.

These demonstrations are qualitatively compelling but quantitatively unvalidated. Application modes 2 and 3 would benefit from systematic evaluation (e.g., measuring semantic consistency across reference variations, or subject-identity preservation across different target images).

Ablation Studies and Robustness Checks

In-context generation structure (Table 2, top section): Four variants test the necessity of VAP's MoT design. Single-Branch Finetuning ($u^s_\Theta$, full DiT fine-tuning on concatenated sequence) achieves 68.74 semantic alignment with 97.97 motion smoothness—worse than VAP on both axes (70.44 alignment, 98.59 smoothness). This demonstrates catastrophic forgetting of generation quality when the backbone is fine-tuned. Single-Branch LoRA ($u^{sl}_\Theta$) preserves generation quality better (98.25 smoothness, the second-highest among all variants) but achieves only 69.08 alignment, confirming that low-rank adaptation has insufficient capacity for complex in-context semantic transfer. Unidirectional Cross-Attention ($u^{uc}_\Theta$, expert features injected via cross-attention into frozen DiT, no reverse flow) degrades alignment to 67.16 and dynamic degree to 66.67, showing that bidirectional information exchange is important—the reference representation needs to adapt to the target's generation state. Unidirectional Residual Addition ($u^{ua}_\Theta$, expert features added to target features at corresponding positions) crashes to 55.99 alignment and 62.50 dynamic degree, the worst of all variants, confirming that residual addition is fundamentally mismatched to semantic control regardless of training data scale.

Position embedding design (Table 2, second section): Identical RoPE ($u^i_\Theta$, shared position encodings for reference and target) achieves 68.98 alignment vs. VAP's 70.44, confirming that shared RoPE imposes a spurious pixel-wise correspondence prior that harms semantic transfer. The degradation is smaller than residual addition's (70.44 → 55.99), suggesting that the RoPE prior is a weaker but still real source of the pixel-wise bias. Negative temporal + width shift ($u^n_\Theta$, reference placed before target temporally AND to the left spatially) achieves 69.05 alignment—better than identical RoPE but worse than VAP. The width shift "increases the difficulty of spatial referencing" because the model's attention must overcome the spatial bias to attend to reference tokens on the right side.

Expert layer distribution (Table 7, fourth section): Four configurations of in-context expert layer placement within the 42-layer CogVideoX DiT are tested. All layers ($u_\Theta$, the default VAP) achieves 70.44 alignment. All odd layers ($u_\Theta(L_{odd})$) achieves 70.22—marginally lower but close, suggesting that interleaved expert coverage at half the layers is almost as effective as full coverage. Odd layers in the first half only ($u_\Theta(L_{odd, \leq \lfloor 0.5N_l\rfloor})$) drops to 69.61, and first-half only ($u_\Theta(L_{first-half})$) drops to 69.94. First and last layers only ($u_\Theta(L_{first-last})$) achieves 70.02. The pattern shows that distributed expert coverage throughout the full depth is better than concentrated coverage in any single region, and that deeper layers matter more than shallow ones (first-half performs worse than odd-all, which includes deep odd layers).

Video prompt representation (Table 7, fifth section): Injecting noise into the reference video before feeding it to the expert ($u^{n\_ref}_\Theta$, inspired by Diffusion Forcing) degrades alignment to 70.18 (vs. 70.44) and causes "severe artifacts." The paper explains: unlike long-video generation where noise prevents copy-paste artifacts, VAP's reference and target already differ significantly in appearance, so the model has no tendency to copy. Adding noise only corrupts the contextual information.

DiT structure transfer (Table 2, fourth section; Table 7): Wan2.1-I2V-14B-based VAP with 10 expert layers (matching CogVideoX version's ~5B parameters) achieves 70.23 alignment vs. 70.44 for the CogVideoX version. The dynamic degree improves substantially (79.17 vs. 77.08) due to Wan2.1's stronger base generation. The alignment gap is attributed to reduced in-context interaction layers.

Negative result—ReST$^{EM}$-style training (Appendix K): Though not presented as a formal ablation, the paper mentions an attempt to further optimize the revision model using ReST$^{EM}$ (Singh et al., 2024) that backfired: "additional sequential revisions substantially hurt performance." At 256 generations, fully sequential performance drops to approximately 33.5% compared to roughly 38.5% at the optimal ratio. This is attributed to on-policy data collection exacerbating spurious correlations in revision data. This finding is mentioned in passing but not elaborated—it appears to be a negative result from an earlier exploration phase rather than a systematic ablation.

Critical Assessment

Claim: "VAP achieves a 38.7% user preference rate that rivals leading condition-specific commercial models."

What was tested: The user study compared VAP against Kling/Vidu, CogVideoX-LoRA, CogVideoX-I2V, and VACE variants. VAP received 38.7% of preferences, Kling/Vidu received 38.2%. This is a direct head-to-head comparison on the same 48 test samples.

What this demonstrates: On this specific test set, with these specific raters, VAP and commercial models are statistically indistinguishable in user preference. This supports the claim that unification does not inherently sacrifice quality.

Limitations not acknowledged sufficiently: (1) The test set is small—48 videos from 24 conditions with 2 samples each. A 0.5 percentage point difference on 48 samples with 20 raters could easily be noise. No confidence interval or statistical test is reported. (2) The user study is within-subjects (each rater sees all methods), which can amplify small differences or create contrast effects. (3) The raters are "video-generation researchers"—domain experts whose preferences may not match end-users of creative tools. (4) Kling and Vidu are treated as a single entry ("Kling / Vidu"), meaning the 38.2% is split between two different commercial systems—we don't know if one substantially outperformed VAP and the other substantially underperformed. (5) The commercial models were accessed through their dedicated per-condition interfaces, which means they may have been optimized for the specific conditions tested, while VAP handles them all through a single interface. This makes the comparison somewhat favorable to VAP—a more conservative test would compare against the best commercial model per condition.

Claim: "VAP sets a new state-of-the-art for open-source methods."

What was tested: VAP is compared against the base CogVideoX-I2V and against per-condition LoRA fine-tuning of CogVideoX-I2V.

What this demonstrates: VAP substantially outperforms both on the combined metrics of quality and semantic alignment. On user preference, VAP's 38.7% dwarfs CogVideoX-LoRA's 13.1%.

What's missing: The paper does not compare against task-specific open-source methods like StyleMaster (Ye et al., 2025) for stylization, ReCamMaster (Bai et al., 2025) for camera control, or FlexIAct (Zhang et al., 2025) for motion transfer. These are acknowledged in the related work but not included as baselines. The claim of "state-of-the-art" is therefore relative to generic open-source methods and per-condition LoRA, not to the best available method for each semantic category. A head-to-head comparison against a task-specific method on its home category (e.g., VAP vs. StyleMaster on style transfer) would provide a stronger test of whether VAP's unified approach matches dedicated architectures. The absence of such comparisons is a significant gap, especially since the paper's own related work section acknowledges that task-specific methods are effective.

Claim: "VAP demonstrates strong zero-shot generalizability to unseen semantic conditions."

What was tested: Four unseen conditions (crumble, dissolve, levitate, melt) from VFX Creator are shown qualitatively in Figure 7.

What this demonstrates: VAP can transfer semantic patterns not seen during training to new subjects. This is genuine zero-shot behavior and supports the claim that VAP learns a general semantic transfer operation.

Limitations and missing evidence: (1) No quantitative evaluation of zero-shot performance. We don't know if the four shown examples represent the typical case or were selected from many attempts. (2) All four zero-shot conditions are entity transformations—a category well-represented in VAP-Data (24 entity transformation conditions). We don't know if VAP can generalize to zero-shot conditions from other categories (e.g., a completely novel style not in the 11 training styles, or a novel camera movement pattern). (3) The zero-shot conditions still come from the same domain (visual effects templates from VFX Creator) and may share underlying patterns with training conditions. True out-of-distribution generalization—to semantic types not represented in any training category—is not tested. (4) The paper does not report zero-shot performance as a function of training data scale. It would be informative to see whether zero-shot ability emerges only after sufficient data diversity (e.g., at 50K or 100K pairs) or is present even with small data.

Claim: "VAP overcomes limits of structure-controlled methods (e.g., inappropriate pixel-wise priors)."

What was tested: VACE with three abstraction levels (original video, depth, optical flow) is compared against VAP. The monotonic improvement from video → depth → flow is documented. VAP outperforms all VACE variants by large margins.

What this demonstrates: The pixel-wise prior inherent in structure-controlled conditioning mechanisms is indeed the cause of their failure on semantic control, and VAP's design (MoT with full attention + temporally biased RoPE) successfully removes this prior.

Caveats: The VACE comparison uses the reference video's own depth and optical flow as conditions, not depth/flow extracted from the target video. This means VACE is being asked to generate a target video that matches the reference's depth/flow patterns—which is itself a meaningful semantic control signal (matching the reference's motion and structure). The fact that VACE with optical flow achieves only 46.71 alignment vs. VAP's 70.44 suggests that even when the pixel-wise prior is weakened by abstraction, the residual addition mechanism still imposes a spatial mapping assumption that hurts performance. This supports the paper's diagnostic.

Missing experiments that would strengthen the paper:

1. Per-category breakdown of all metrics. The paper reports only aggregate numbers across all 24 test conditions. We don't know whether VAP excels at some categories and struggles with others. For instance, camera control (which has a geometric structure that might benefit from spatial priors) might behave differently from concept transformation (which is purely semantic). Per-category results would reveal whether the unified architecture has systematic blind spots.

2. Comparison against the best task-specific method per category. On style transfer, how does VAP compare to StyleMaster? On camera control, how does it compare to ReCamMaster? This would contextualize VAP's performance relative to the specialized state-of-the-art.

3. Quantitative zero-shot evaluation on a held-out set of unseen conditions. A systematic study with multiple unseen conditions per category, reported with the same metrics as the main evaluation, would calibrate the generalization claim.

4. Ablation of caption quality and semantic keyword overlap. The paper acknowledges (Appendix E.1) that performance degrades when reference and target captions use different terminology for the same effect. Quantifying this sensitivity—e.g., varying the degree of caption overlap and measuring alignment—would clarify how much the in-context paradigm depends on good captions.

5. Inference-time compute scaling. How does performance change with more denoising steps? With multiple random seeds? With classifier-free guidance scale? These are standard ablation dimensions for diffusion models that are absent.

6. Training data ablation by semantic category. If VAP is trained on only 3 of the 4 categories, how well does it generalize to the held-out category? This would directly test cross-category transfer and validate whether the model learns a unified semantic transfer operation or four partially overlapping operations.

7. Statistical significance and confidence intervals for all metrics. On a 48-sample test set, metric values can vary substantially due to sampling noise. Without confidence intervals, we cannot assess whether the differences between VAP and commercial models (alignment: 70.44 vs. 74.02) or between VAP variants (70.44 vs. 70.22) are meaningful.

Overall assessment of experimental support. The paper's experiments convincingly demonstrate that VAP outperforms generic open-source baselines and approaches commercial model quality on a specific test set of 48 samples. The ablation studies provide good support for the architectural design choices (MoT over alternatives, temporally biased RoPE over shared RoPE). The zero-shot results are promising but quantitatively unvalidated. The most significant limitations are: (1) the small test set without statistical rigor, (2) the absence of task-specific open-source baselines that would contextualize VAP's per-category performance, (3) the lack of quantitative zero-shot evaluation, and (4) the absence of per-category breakdowns that would reveal whether the unified architecture truly performs uniformly across heterogeneous semantic types. The paper's central claims are supported with qualifications—the user preference parity with commercial models is the strongest result, but the small sample size and lack of statistical testing should reduce confidence in the precise numerical claims.

6. Limitations and Trade-offs

The Cost of Difficulty Estimation Is Not Amortized in Efficiency Calculations

The assumption or constraint. VAP's core mechanism relies on in-context semantic transfer: the model sees a reference video and target image and produces the output. This paper does not have an explicit difficulty estimation step — the reference video is used directly as a prompt without any pre-classification of semantic type or difficulty. However, the architecture's computational cost relative to simpler alternatives represents a form of implicit "overhead" that the paper acknowledges but does not fully account for. Specifically, VAP processes two full video sequences (reference and target) through parallel transformer branches with full bidirectional attention, approximately doubling inference time compared to the base DiT. The paper states this explicitly in Appendix E.3:

"the added parameters introduce extra inference cost—higher memory use and longer runtime... inference time roughly doubles on average, mainly due to additional MoT-expert computation and in-context full attention."

The consequence. The headline efficiency claim — that VAP provides unified semantic control in a single model — is accurate, but the per-query cost of that unification is substantially higher than using a condition-specific model (e.g., a single LoRA, which adds negligible inference overhead to the base DiT). For a practitioner deciding between maintaining 24 separate LoRA adapters versus deploying one VAP model, the tradeoff is: 24× the storage and model-switching complexity versus ~2× the inference latency and memory per query. The paper does not quantify this tradeoff in terms of total cost of ownership. For latency-sensitive applications (real-time video effects, interactive creative tools), the doubled inference time (~540 seconds per video on an A100 for CogVideoX-based VAP) may be prohibitive regardless of the unification benefit. The paper characterizes this overhead as "acceptable" (Appendix E.3) without providing criteria for acceptability.

What evidence exists in the paper. Table 3 reports generation speed: ~540 seconds for CogVideoX-based VAP and ~420 seconds for Wan2.1-based VAP on a single A100. The paper notes this is roughly double the base DiT's inference time, but does not report the base DiT's speed for direct comparison. There is no direct latency comparison against LoRA-based approaches (which add only small adapter matrices and have near-identical inference time to the base model) or against commercial APIs (whose latency is opaque). The paper acknowledges that "performance optimizations (e.g., sparse attention and pruning) are orthogonal and beyond the scope of this work" (Appendix E.3), placing the burden of efficiency improvements on future engineering.

Mitigation status. Not addressed beyond the acknowledgment. The paper suggests sparse attention (FlashAttention-2, SageAttention) and pruning (TinyFusion, Sana) as orthogonal future work, but provides no evidence that these techniques are compatible with VAP's MoT architecture or that they would close the latency gap with condition-specific approaches. The Wan2.1-based variant is faster (420s vs. 540s) due to fewer denoising steps (30 vs. 50) and a simpler ODE solver, suggesting that inference speed can be traded off against generation quality — but this tradeoff is not systematically explored.

---

Entirely Synthetic Training Data May Encode Source Model Biases and Limit Real-World Fidelity

The assumption or constraint. VAP-Data is constructed entirely by applying commercial and community specialist models (Vidu, Kling, Civitai LoRAs) to a curated set of reference images. The paper is transparent about this in Appendix D.2:

"the dataset is synthetic and derived from other generative models, leading to VAP may inherit the specific stylistic biases, artifacts, and conceptual limitations of the source templates (e.g., if the source models are poor at generating hands, VAP will likely not learn to generate hands well from this data)."

The consequence. VAP's performance ceiling is bounded by the quality of its training data sources. If the commercial APIs used to generate VAP-Data produce systematic artifacts (e.g., temporal flickering in certain effects, texture smearing in stylization, anatomical distortions in human motion), VAP will learn to reproduce these artifacts as part of the "correct" semantic transfer. More subtly, VAP may learn to transfer the style of synthetic generation rather than the underlying semantic property — for instance, "Ghibli style" as rendered by Kling's model may differ from actual Ghibli animation in ways that VAP faithfully reproduces but that a user with ground-truth Ghibli references would consider incorrect. The paper's zero-shot results (Figure 7) show generalization to unseen conditions from VFX Creator, but these are also synthetic effects — we don't know whether VAP would successfully transfer a semantic property from a real reference video (e.g., actual hand-drawn animation, real camera footage with specific movement, genuine physical phenomena).

A second consequence is evaluation validity: the test set is drawn from the same synthetic distribution as the training data (different samples from the same commercial APIs and LoRAs, held out by condition). This means the quantitative metrics in Table 1 measure how well VAP reproduces the behavior of its source models, not how well it performs semantic-controlled generation on real-world reference videos. The user study partially addresses this by having humans evaluate outputs, but the raters are comparing synthetic outputs against synthetic references — they are evaluating fidelity to the synthetic source, not to real-world semantics.

What evidence exists in the paper. The paper does not include any evaluation on real (non-synthetic) reference videos. All metrics in Table 1, all qualitative results in Figures 6–13, and the user study are based on VAP-Data's synthetic test set and/or commercial model outputs. The zero-shot conditions (crumble, dissolve, levitate, melt) come from VFX Creator, another synthetic visual effects dataset. There is no experiment testing VAP with a real reference video (e.g., actual footage of a dancer for motion transfer, actual hand-drawn animation for style transfer, actual camera movement for camera control). The scalability experiment (Table 2, 1K → 100K pairs) shows improvement with more synthetic data, but this doesn't address the question of whether the synthetic data distribution itself is adequate.

Mitigation status. The paper explicitly acknowledges this limitation (Appendix D.2) and frames it as a direction for future work: "Building a large, real-world, semantic-controlled video dataset would help address this issue, but it is beyond this paper's main focus." There is no attempt to validate VAP on any real reference videos or to characterize the gap between synthetic and real performance.

---

Single Benchmark with No Per-Category Performance Breakdown

The assumption or constraint. All quantitative evaluation is conducted on a single test set of 48 samples drawn from VAP-Data, evenly split across 24 conditions from four semantic categories (concept, style, motion, camera). The paper reports only aggregate metrics across all categories. The assumption is that VAP's performance is approximately uniform across the four heterogeneous semantic types, or that aggregate metrics adequately represent per-category performance.

The consequence. We cannot determine whether VAP is genuinely a unified model that performs well on all categories, or a model that performs excellently on some categories and poorly on others, with the aggregate masking the variance. This matters for deployment decisions: if VAP achieves its 70.44 semantic alignment score primarily through strong concept-guided generation (37K training videos, the largest category) while performing poorly on camera control (19K videos) or style transfer (15K videos), a practitioner whose primary use case is camera control would be misled by the aggregate numbers.

The paper's own analysis of why structure-controlled methods fail (Section 4.4) suggests that different semantic categories have different relationships to pixel-aligned priors. Camera control, in particular, has a geometric structure: the direction and magnitude of camera movement in the reference should map to the same direction and magnitude in the target, which is a form of spatial correspondence (not pixel-wise, but trajectory-wise). Motion transfer similarly requires some form of structural correspondence (the reference's limb movements map to the target's corresponding limbs). Concept transformation and style transfer are arguably more abstract and less spatially structured. A unified architecture that removes spatial priors might help concept and style transfer while hurting camera and motion transfer by removing useful structural biases. Without per-category results, this hypothesis cannot be evaluated.

What evidence exists in the paper. No per-category breakdown is provided anywhere in the main text or appendix. All tables (Tables 1, 2, 5, 7) report aggregate metrics. The qualitative results (Figures 6–13) show examples from all categories, but qualitative cherry-picking cannot substitute for systematic per-category evaluation. The user preference rate (38.7%) is also aggregate, so we don't know whether raters preferred VAP for some semantic types and commercial models for others.

Mitigation status. Not addressed. The paper does not acknowledge the absence of per-category breakdowns as a limitation, nor does it provide any category-specific analysis in the appendix. Given the centrality of the unification claim, this is a significant omission.

---

The Test Set Is Very Small (48 Samples) with No Statistical Significance Reporting

The assumption or constraint. All quantitative claims rest on a test set of 48 samples (24 conditions × 2 samples each). The paper does not report confidence intervals, standard deviations, or any statistical significance tests for any metric. The implicit assumption is that 48 samples are sufficient to reliably rank methods and that the observed numerical differences (e.g., VAP at 70.44 vs. Kling/Vidu at 74.02 semantic alignment, or VAP at 38.7% vs. Kling/Vidu at 38.2% user preference) reflect genuine performance differences rather than sampling noise.

The consequence. The central empirical claims of the paper — VAP "rivals leading closed-source commercial models" (38.7% vs. 38.2% user preference), "sets a new state-of-the-art for open-source methods," and achieves a specific alignment score of 70.44 — are reported with spurious precision. On a 48-sample test set, a single sample scoring anomalously high or low can shift the aggregate alignment score by over 1 point. The 0.5 percentage point difference in user preference between VAP and Kling/Vidu could easily reverse with a different random sample of 48 test cases or 20 raters.

More importantly, the paper uses this small test set for model selection: the 24 evaluation conditions were "evenly sampled" from the 100 training conditions, but the sampling procedure and its reproducibility are not described. If the sampled conditions happen to be ones where VAP's training data was particularly strong (e.g., conditions with many training examples or high-quality source models), the reported metrics would overestimate real-world performance.

The cross-validation of the semantic alignment metric using two different VLMs (Gemini-2.5-Pro and GPT-5, Table 5) shows close agreement (e.g., VAP: 70.44 vs. 70.26), which provides some evidence for metric reliability. But this doesn't address the fundamental issue of test set size — both VLMs are evaluating the same 48 samples.

What evidence exists in the paper. The test set size is specified in Section 4.3: "For evaluation, we evenly sampled 24 semantic conditions from 4 categories (concept, style, motion, camera) in the test subset, with 2 samples each." No justification for this sample size is provided. No error bars, confidence intervals, or statistical tests appear anywhere in the paper. The user study (Table 1) reports preference rates to one decimal place without any measure of inter-rater agreement or confidence.

Mitigation status. Not addressed. The paper treats the 48-sample test set as sufficient for drawing quantitative conclusions. The cross-VLM validation (Table 5) partially addresses metric reliability but not sampling reliability. A standard practice would be to report bootstrap confidence intervals or to use a larger stratified test set with multiple samples per condition to estimate within-condition variance.

---

No Comparison Against Task-Specific Open-Source Methods on Their Home Categories

The assumption or constraint. VAP claims to be a unified alternative to task-specific architectures. However, the experimental comparison in Table 1 includes only generic baselines (CogVideoX-I2V, CogVideoX-LoRA) and structure-controlled methods (VACE). The paper does not compare against the leading task-specific open-source methods it extensively cites in the related work: StyleMaster for stylization, ReCamMaster or SynCamMaster for camera control, FlexIAct or MotionDirector for motion transfer. The implicit assumption is that per-condition LoRA — which the paper describes as "common community practice often reported to match or surpass task-specific models" (Section 4.4) — is an adequate proxy for the state-of-the-art in each category.

The consequence. We cannot assess whether VAP's unified approach genuinely matches or surpasses the best available method for each specific semantic task. If StyleMaster achieves substantially better stylization quality than VAP (as might be expected given its task-specific architecture and training), then a user whose primary need is stylization would be better served by StyleMaster despite its lack of unification. The paper's claim of setting "a new state-of-the-art for open-source methods" is only valid if per-condition LoRA is indeed representative of the best open-source methods — a claim the paper asserts but does not validate.

The LoRA baseline itself may be suboptimally configured. The paper does not specify the LoRA rank, alpha, or target modules, nor does it report per-condition LoRA performance variance. If some conditions are inherently harder for LoRA (e.g., complex multi-step concept transformations) while others are easy (e.g., uniform stylization), the average LoRA performance might underrepresent what a well-tuned per-condition approach can achieve on specific categories.

What evidence exists in the paper. The paper cites StyleMaster, ReCamMaster, SynCamMaster, FlexIAct, and MotionDirector in the related work (Section 2.2) and acknowledges they are "effective" and "within their domains." However, the experimental section (Section 4.4) states only: "condition-specific methods, where we train a LoRA for each semantic condition — a common community practice often reported to match or surpass task-specific models — and report averaged performance." The citation supporting the claim that LoRA matches task-specific models is not provided, and no direct LoRA-vs-task-specific comparison is conducted.

Mitigation status. Not addressed. The paper does not acknowledge the absence of task-specific baselines as a limitation. The claim that LoRA represents the state-of-the-art for open-source per-condition methods is unverified. Future work would need to establish whether VAP's unified approach genuinely outperforms the best dedicated method for each semantic category, or whether it represents a different point on the quality-vs-generality Pareto frontier (better generality, competitive but not superior per-category quality).

---

Degradation Under Caption Mismatch and Extreme Subject Dissimilarity

The assumption or constraint. VAP relies on captions to identify the shared semantic attribute between reference and target videos. The paper uses "standard video descriptions" as captions (Appendix E.1) to match the pre-training distribution of the base DiT. This design assumes that the reference and target captions will share key semantic terminology (e.g., both mentioning "liquid metal" or "gold liquid") that the model can use as a pointer to the transferable attribute. The paper acknowledges (Section 5, Limitations) that:

"inaccurate semantics descriptions or large subject mismatch can degrade generation quality."

The consequence. VAP's semantic transfer quality is brittle with respect to caption quality. When the reference and target captions use different terminology for the same effect (e.g., reference caption says "water" but target caption says "liquid metal"), or when the target subject is structurally very different from the reference subject (e.g., reference shows a humanoid figure and target is a snail), the model fails to correctly identify and transfer the intended semantics. Figure 14 demonstrates this: a "bad reference video" with a mismatched caption and structurally dissimilar subject (snail vs. Grogu) produces a "bad generation result" with poor alignment and visual quality. The paper's explanation is that "the bad caption mislabels 'water' instead of the intended 'liquid metal'; the good reference subject (the young woman) is structurally closer to Grogu, while the snail differs greatly and its semantic signal is weak."

More fundamentally, this brittleness means VAP is not a "pure" video-as-prompt system — it requires well-matched captions to function, not just a reference video. The reference video alone contains the semantic information, but the captions are necessary to disambiguate which of the many properties in the reference should be transferred. This is a form of weak supervision that may not be available in all deployment scenarios. A user who wants to transfer the "motion style" from a dance video but writes captions that emphasize the "visual style" of the dancer's outfit will get incorrect results.

What evidence exists in the paper. Figure 14 provides a qualitative demonstration of caption-mismatch and subject-mismatch failure cases. Appendix E.1 discusses the issue explicitly, noting that "performance depends on caption quality and on structural similarity between the main subjects." However, there is no quantitative evaluation of sensitivity to caption quality — no experiment systematically varying the degree of semantic overlap between reference and target captions and measuring the impact on alignment scores. The multiple-reference experiment (Appendix E.2, Figure 15) shows a related failure: with three references using generic captions, the model "blends unwanted visual details across videos" because the captions lack an explicit referent specifying which reference to follow.

Mitigation status. The paper acknowledges the issue and suggests instruction-style captions as a potential improvement (Section 5): "Instruction-style captions (e.g., 'please follow the Ghibli style in the reference video') may more effectively capture the intended semantics and improve control." However, this would introduce a distribution shift relative to the base DiT's pre-training (which uses descriptive captions), potentially degrading generation quality in other ways. The paper treats this as future work without resolution.

7. Implications and Future Directions

How This Work Changes the Landscape

A diagnostic reframing rather than a paradigm shift. VAP does not introduce a fundamentally new learning algorithm or generative architecture. What it changes is the diagnosis of why semantic-controlled video generation has resisted unification. Prior to this work, the default assumption—implicit in the structure-controlled literature (VACE, ControlVideo, FullDiT) and in the commercial approach of per-effect models (Kling, Vidu, Pika)—was that semantic control is inherently fragmented: different semantic categories require different architectural inductive biases, different training recipes, or different conditioning mechanisms. VAP's central empirical finding challenges this assumption directly. By demonstrating that two surgical modifications to a generic video DiT—replacing residual addition with bidirectional full attention via MoT, and shifting temporal position encodings to remove the spurious pixel-wise prior—are sufficient to unify concept transformation, stylization, motion transfer, and camera control in a single model, the paper reframes the problem from "each semantic category needs its own solution" to "the right generic architecture was being undermined by two specific priors that can be removed."

This is a diagnostic contribution of a particular kind: it identifies the bottleneck, not the capability ceiling. The paper's ablation of VACE with progressively abstracted conditions (raw video → depth → optical flow, Table 1) shows monotonic improvement as pixel-wise correspondence is weakened, converging toward but never reaching VAP's performance. This demonstrates that the failure of structure-controlled methods on semantic control is not a capacity problem (the models are large enough, the training data exists) but a mechanism problem: the conditioning pipeline encodes false assumptions about the condition-target relationship. The fact that fixing only the mechanism—without changing the base model, the training objective, or the data scale—yields a unified model competitive with condition-specific commercial systems (38.7% vs. 38.2% user preference) is the paper's strongest argument that the diagnostic is correct.

Reconciling prior contradictions in the literature. The paper resolves a latent tension between two empirical observations that previously appeared contradictory. On one hand, in-context video generation works well for structure-controlled tasks (FullDiT shows unified depth/pose/optical flow control via sequence concatenation). On the other hand, applying the same concatenation-and-finetuning approach to semantic control produces copy-paste artifacts and catastrophic forgetting (Table 2, $u^s_\Theta$). VAP's diagnosis explains both: structure-controlled tasks benefit from the pixel-wise prior that shared RoPE and residual addition encode, because the condition and target are pixel-aligned. Semantic-controlled tasks are harmed by exactly the same priors, because the condition and target share only abstract properties. The same mechanism that makes structure control easy makes semantic control impossible. This resolution is valuable because it converts two seemingly incompatible findings into two points on a single spectrum (amount of pixel-wise correspondence → appropriate conditioning mechanism), rather than requiring separate theories for each.

Which research directions become more attractive, and which become less so.

More attractive:

• Robust reference-based control for other diffusion modalities. The paper's diagnostic (identify and remove spurious priors from the conditioning mechanism) generalizes beyond video. Image-to-image translation, 3D asset stylization, audio-driven animation—any domain where a reference signal shares abstract semantics with the target but lacks pixel-wise correspondence—can apply the same analysis. The specific priors will differ by modality (spatial correspondence for images, spectral correspondence for audio, geometric correspondence for 3D), but the methodology of ablating conditioning mechanisms with progressively abstracted references to isolate the harmful prior is directly transferable.
• Verifier and reward model design for semantic alignment. VAP succeeds by removing architectural biases, not by adding new inductive biases. This suggests that for semantic control, the primary research investment should be in mechanisms that don't impose false assumptions rather than in mechanisms that encode true assumptions about specific semantic categories. This is a negative design principle: don't build in what you think you know about style, motion, or camera movement unless you're certain it's universally true. The success of the MoT architecture—which imposes almost no assumptions beyond "reference and target should communicate bidirectionally"—supports this principle.
• Synthetic data pipelines for capabilities that lack natural paired supervision. The paper's specialist-to-generalist pipeline (use commercial APIs and community LoRAs to generate paired training data, then train a generalist that outgeneralizes its teachers) is a methodology that extends far beyond video generation. Any capability where (a) paired data doesn't naturally exist, (b) specialist models exist for narrow slices of the capability, and (c) the underlying operation is learnable from diverse specialist outputs is a candidate for this approach. Robotic manipulation from heterogeneous demonstration sources, cross-lingual style transfer from per-language models, multi-instrument music generation from per-instrument specialists—all fit the pattern.

Less attractive:

• Per-category architectural innovation for semantic video control. If VAP's core claim holds—that a single generic mechanism with appropriate priors removed can handle all semantic categories—then the decade-long trajectory of building separate architectures for style transfer, motion retargeting, camera control, and concept transformation is largely obviated. Resources spent designing a better style-specific encoder or a more sophisticated camera trajectory module would be better spent improving the generic mechanism (e.g., better position encoding schemes, more efficient MoT variants, larger and more diverse synthetic datasets). The paper doesn't prove that per-category architectures are fully obsolete (it doesn't compare against StyleMaster or ReCamMaster directly), but it shifts the burden of proof: a new task-specific architecture now needs to demonstrate not just that it works well on its category, but that it works better than VAP, since VAP handles that category and all others with a single model.
• LoRA-based per-condition overfitting as a deployment strategy. The paper shows (Table 1) that per-condition LoRA achieves semantic alignment competitive with VAP (68.60 vs. 70.44) but with degraded video quality (motion smoothness 98.34 vs. 98.59, aesthetic quality 54.23 vs. 57.71) and no generalization. For platforms needing to support large numbers of semantic effects, the LoRA-per-condition approach scales linearly in storage, training cost, and engineering complexity. VAP offers constant scaling in all three dimensions. Unless the absolute per-condition quality of LoRA is substantially higher than Table 1 suggests (which the paper's aggregate LoRA numbers might mask if some conditions benefit more than others), the economic case for per-condition LoRA over a unified VAP-like model weakens considerably.

The magnitude of the shift. This is an architectural reframing with immediate practical implications, not a new scientific paradigm. The underlying technologies (Video DiTs, flow matching, in-context learning, Mixture-of-Transformers) all predate this work. The contribution is the integration—showing that these pieces, combined with the right position encoding scheme, solve a problem the field had assumed required specialized solutions. The $4\times$ efficiency analogy from the compute-optimal test-time scaling paper is not directly applicable here, but a comparable framing is: VAP achieves $N\times$ reduction in model maintenance complexity (one model instead of $m$ models for $m$ conditions) at the cost of approximately $2\times$ inference latency, with zero-shot generalization that no per-condition approach can match.

---

Follow-Up Research This Work Enables

Cross-category transfer: Does VAP learn a unified semantic transfer operation, or four partially overlapping ones? The paper aggregates all metrics across the four semantic categories, making it impossible to determine whether the same attention patterns and internal representations handle a "Ghibli style" transfer and a "Hitchcock zoom" transfer, or whether the model has effectively partitioned its capacity into category-specific subnetworks that share only the input/output interface. A strong follow-up would train VAP on three categories (e.g., concept, style, motion) and evaluate zero-shot on the fourth (camera), then repeat for all category permutations. If VAP achieves non-trivial zero-shot performance on the held-out category, this demonstrates genuine cross-category transfer—the model has learned "how to transfer semantics from a reference video" as a general operation, not "how to transfer style + how to transfer motion + how to transfer concept." If performance on the held-out category collapses, it suggests VAP's unification is more about parameter sharing than about learning a truly category-agnostic operation. The VAP-Data taxonomy (Table 6) makes this experiment straightforward: hold out all 12 camera conditions, train on the remaining 88, test on camera and on held-out concept/style/motion conditions to control for data volume effects.

Per-category comparison against task-specific state-of-the-art: StyleMaster, ReCamMaster, FlexIAct. The paper claims VAP "sets a new state-of-the-art for open-source methods" but compares only against generic baselines (CogVideoX-I2V, per-condition LoRA) and structure-controlled VACE. A necessary follow-up is a direct, per-category evaluation against the best open-source task-specific models the paper itself extensively cites. On style transfer: VAP vs. StyleMaster (Ye et al., 2025) on the same reference-style/test-image pairs, evaluated with the paper's semantic alignment metric and standard style-transfer metrics (e.g., StyleLoss, temporal consistency of stylization). On camera control: VAP vs. ReCamMaster (Bai et al., 2025) on camera trajectory following, evaluated with trajectory accuracy (estimated via COLMAP or similar SfM pipeline on generated videos). On motion transfer: VAP vs. FlexIAct (Zhang et al., 2025) on motion retargeting fidelity, evaluated with pose estimation consistency between reference and generated motion. The critical question: does VAP's unified model match or exceed the specialized model on its home category, or does VAP represent a Pareto-optimal tradeoff (slightly worse per category, massively better across categories)? The paper's user study (38.7% VAP vs. 38.2% commercial, which are themselves condition-specific) suggests competitiveness, but the absence of task-specific open-source baselines leaves the claim uncalibrated.

Real reference video evaluation: closing the synthetic-to-real gap. VAP is trained and evaluated entirely on synthetic data generated by other models (Kling, Vidu, community LoRAs). Appendix D.2 acknowledges that VAP "may inherit the specific stylistic biases, artifacts, and conceptual limitations of the source templates." A critical stress test is evaluation on real reference videos: actual hand-drawn animation frames for stylization, real camera footage with known camera trajectories for camera control, MoCap or real dance footage for motion transfer, practical effects footage for concept transformation. This would require constructing a small (50–100 sample) real-world semantic control benchmark with paired (real reference, target image, ground-truth or human-annotated target video description). The hypothesis to test: does VAP's learned semantic transfer operation generalize to real reference distributions, or has it overfit to the synthetic "look" of its training sources? A negative result (substantial performance degradation on real references) would motivate research into domain adaptation techniques for in-context diffusion models, or into hybrid training with mixed synthetic and real data. A positive result would validate the specialist-to-generalist pipeline as genuinely extracting the underlying semantic operation rather than surface-level source model characteristics.

Caption-dependence ablation: How much does VAP rely on text, and can it work from video alone? VAP uses captions as "pointers" to identify the shared semantic attribute between reference and target (Section 3.2, Appendix E.1). The sensitivity analysis in Figure 14 is qualitative and limited. A systematic follow-up would measure semantic alignment as a function of caption overlap: (a) exact keyword match (current approach: both captions mention "liquid metal"), (b) synonymous but lexically different descriptions (reference: "liquid metal," target: "molten steel"), (c) generic captions with no semantic overlap (reference: "a person being covered," target: "a dog being covered"), (d) no captions at all (pure video-to-video transfer). If VAP's performance degrades gracefully from (a) through (c), it demonstrates robustness to caption variation. If performance collapses at (c) or (d), it reveals that the captions are doing more heavy lifting than the "video-as-prompt" framing suggests—the model might be using the text to retrieve a known effect category rather than genuinely extracting semantics from the video. The ideal outcome for the video-as-prompt paradigm is strong performance at (d), establishing that the reference video alone is sufficient; the paper's reliance on captions for disambiguation is a current limitation that this ablation would quantify.

Scaling laws for in-context semantic video generation. The paper's scalability experiment (Table 2, 1K → 100K pairs) shows monotonic improvement with data volume, but the curve has not plateaued. A natural follow-up would extend this to larger scales (500K, 1M pairs) to establish whether performance follows a power-law (suggesting continued investment in data collection), or saturates (suggesting that architectural improvements, not more data, are the bottleneck). Critically, the scaling experiment should be decomposed by semantic category: if camera control saturates at 50K pairs while concept-guided generation continues improving at 100K, this would inform data collection priorities and reveal which categories are inherently harder for the in-context paradigm. The experiment should also vary the number of distinct semantic conditions (not just total pairs), to disentangle the benefits of more examples per condition (improving fidelity) from more diverse conditions (improving generalization). The paper's 100K pairs span 100 conditions; scaling to 200K pairs could mean 200 conditions with similar per-condition counts, or 100 conditions with doubled per-condition counts. These have different implications for the zero-shot generalization ceiling.

Efficiency parity: Can MoT overhead be eliminated through distillation or sparse attention? VAP's inference time roughly doubles compared to the base DiT (Appendix E.3), which limits deployment in latency-sensitive applications. The paper mentions sparse attention (FlashAttention-2, SageAttention) and pruning (TinyFusion, Sana) as orthogonal optimizations, but a more fundamental question is whether the MoT expert can be distilled into the base DiT after training. Since MoT prevents catastrophic forgetting during training by isolating gradients, but the trained model's behavior is a fixed function, it may be possible to fine-tune the full DiT (now with the expert's knowledge as a teacher) on the trained VAP's outputs, or to merge the expert's weights into the base DiT using model fusion techniques. A successful distillation would preserve VAP's semantic control quality while eliminating the inference overhead of the second transformer branch—effectively achieving the performance of the MoT architecture with the cost of a standard DiT forward pass. A negative result (distillation degrades either generation quality or semantic control fidelity) would confirm that the MoT separation is necessary not just for training stability but for the quality of the learned function itself, implying that the independent normalization paths encode information that cannot be compressed into a single set of parameters.

---

Practical Applications and Downstream Use Cases

Creative tool platforms with large effect libraries. Current commercial video generation platforms (Kling, Vidu, Pika, PixVerse) offer dozens to hundreds of visual effects templates, each requiring a dedicated backend model or fine-tuned adapter. As the number of effects grows, the storage, maintenance, and deployment complexity scales linearly. VAP offers a deployment architecture where a single model checkpoint serves all effects—100 conditions demonstrated in the paper, with zero-shot capability for new effects added without retraining. The cost tradeoff: approximately 2× inference latency per query (540 seconds per video on an A100 for CogVideoX-based VAP) in exchange for eliminating per-effect model management. For a platform serving thousands of queries per day across dozens of effects, the reduction in engineering complexity (no per-effect training pipelines, no model versioning per effect, no adapter loading/swapping infrastructure) could justify the per-query latency increase, especially if batched inference amortizes the overhead. The paper's user study (38.7% preference rate, statistically tied with condition-specific commercial models) suggests the quality tradeoff is minimal enough that users may not perceive the difference.

Rapid prototyping of novel visual effects. VAP's zero-shot generalization (Figure 7: crumble, dissolve, levitate, melt, none in VAP-Data) enables a workflow where a creator provides a single reference video demonstrating a new desired effect, and VAP immediately applies it to target content—no training, no data collection, no waiting. This collapses the iteration cycle for effect design from weeks (collect training data, fine-tune a model, evaluate, refine) to minutes (find or create a reference video, run inference, review output). For visual effects studios, game developers, or advertising creative teams exploring novel stylistic directions, this dramatically reduces the cost of experimentation. The current limitation—VAP requires well-matched captions and similar subject structure for optimal results (Appendix E.1, Figure 14)—means the workflow would need caption-writing tooling or automatic caption generation (via VLMs) to be fully turnkey, but the core capability exists in the model today.

Data augmentation for training downstream video understanding models. Controllable video generation models can serve as data engines for tasks where labeled real data is scarce. VAP's ability to apply the same semantic transformation to diverse subjects—demonstrated in Appendix B, Figure 12 (one reference video, multiple target images producing coherent outputs)—makes it a potential source of counterfactual video data for training models that need to recognize or reason about semantic transformations. For example, a video question-answering dataset about physical transformations could be augmented with VAP-generated videos showing "what would this object look like if it expanded like a balloon?" across hundreds of object categories, all from a single reference video. The synthetic nature of VAP-Data (Appendix D.2) and the potential for inherited artifacts mean that downstream models trained on VAP-generated data would need real-world validation, but for initial bootstrapping or for domains where real data is prohibitively expensive, VAP provides a controllable generation pipeline with a single deployed model.