SAMA: Factorized Semantic Anchoring and Motion Alignment for Instruction-Guided Video Editing

ArXiv: 2603.19228

🎯 Pitch

Video editors don't need to learn everything from scratch—by splitting semantic planning from motion modeling, SAMA discovers that just learning how videos move from raw footage unlocks powerful zero-shot editing, without ever seeing paired editing data.

1. Executive Summary

This paper introduces SAMA (Semantic Anchoring and Motion Alignment), a factorized framework for instruction-guided video editing that separates semantic planning from motion modeling within a Diffusion Transformer backbone, reducing reliance on brittle external priors like VLM features or structural conditions. Semantic Anchoring jointly predicts semantic tokens and video latents at sparse anchor frames to provide an instruction-aware structural prior, while Motion Alignment pre-trains the backbone on motion-centric video restoration pretext tasks — cube inpainting, speed perturbation, and tube shuffle — to internalize temporal dynamics directly from raw videos. SAMA achieves state-of-the-art performance among open-source video editing models on VIE-Bench, OpenVE-Bench, and ReCo-Bench, while remaining competitive with leading commercial systems such as Kling-Omni. A key finding is that the factorized pre-training stage alone induces strong zero-shot video editing behavior, establishing that robust instruction-following can emerge once a model learns to jointly reason about semantic intent and temporal dynamics — even before exposure to paired video-instruction editing data.

2. Context and Motivation

The Core Problem: The Semantic-Motion Tension in Instruction-Guided Video Editing

This paper addresses a fundamental tension that sits at the heart of instruction-guided video editing: how to simultaneously apply fine-grained semantic edits that accurately follow a text instruction while faithfully preserving the original video's motion dynamics. The authors frame this as a conflict that every practical video editor must resolve — aggressive semantic changes tend to introduce temporal artifacts (localized flickering, identity drift, texture popping across frames), while enforcing strong temporal consistency tends to dilute the intended edit and reduce instruction fidelity (Section 1, Fig. 1 top).

This is not merely an aesthetic concern. For a video editor to be practically useful — whether for content creation, VFX prototyping, or video post-production — it must operate with two properties simultaneously:

Instruction fidelity: The edited output must precisely reflect the semantic changes specified in the user's instruction, including fine-grained attributes like color, pose, position, object identity, and style. A model that "preserves motion" but fails to actually change the specified object or attribute is useless.
Motion preservation: All non-edited content — including background motion, camera movement, subject dynamics, and the trajectory of any preserved objects — must remain temporally coherent and faithful to the source video. An edit that introduces flickering, ghosting, or warping on unrelated regions destroys the perceptual plausibility of the result.

The challenge is that these two requirements are mechanically coupled in most existing approaches. A standard video diffusion model processes all spatiotemporal regions uniformly — there is no architectural separation between "regions to edit" and "regions to preserve." Consequently, when the model modifies pixels to satisfy an editing instruction, it inevitably perturbs temporal correlations across the entire frame sequence. Conversely, when the model is constrained to preserve temporal consistency, it may resist making the very changes the instruction demands. The authors observe this as a fundamental architectural limitation rather than a training data problem:

"We argue that this over-reliance [on external priors] reflects a significant bottleneck, which constrains the diffusion backbone from learning inherent semantic-motion representations for precise semantic editing and faithful motion alignment" (Section 1, emphasis mine).

The key insight is that the model's internal representations do not factorize semantic structure from motion dynamics, so the two objectives compete within a shared representation space.

Why This Problem Matters

The importance of resolving this tension extends beyond academic interest in video quality metrics. The paper's motivation touches on several practical and theoretical dimensions:

Practical deployment bottleneck. Instruction-guided video editing systems have seen explosive commercial interest — products like Runway, Kling, Pika, and MiniMax represent multi-billion-dollar markets for AI-powered creative tools. However, the authors document that even leading commercial systems struggle with the semantic-motion tradeoff. In Figure 1 (top), commercial baselines exhibit visible artifacts: Kling-Omni produces identity drift and texture inconsistencies when editing clothing, while source motion degrades under aggressive semantic changes. This suggests the problem is not solved by scale alone — simply training larger models on more data does not eliminate the tension. A principled architectural solution is needed.

Generalization across editing types. The paper's evaluation spans five editing categories on VIE-Bench (Add, Swap/Change, Remove, Style/Tone Change) and additional categories on OpenVE-Bench (Background Change, Local Change, Subtitle Edit, Creative Edit). This diversity matters because different editing types stress the semantic-motion tradeoff differently:

Object replacement requires local semantic changes but global motion preservation — the background and camera must remain untouched.
Style transfer requires global semantic changes (altering texture, color palette) while preserving fine motion dynamics.
Object removal requires precise semantic erasure with seamless spatial inpainting that maintains temporal coherence with surrounding motion.
Object addition requires placing new content that moves physically plausibly within the existing scene dynamics.

A framework that works across all these categories cannot rely on task-specific heuristics — it needs a general mechanism for decoupling semantic planning from motion modeling.

The zero-shot editing promise. A striking finding in the paper is that the factorized pre-training stage alone yields strong zero-shot editing behavior, even without any paired video-instruction editing data (Section 4.3, Figure 5). This has profound implications: it suggests that robust instruction-following is not exclusively a product of supervised training on paired edit data, but can emerge from learning factorized representations of semantics and motion. The authors explicitly frame this as evidence for their central hypothesis:

"This observation suggests that robust instruction-guided video editing can naturally emerge once a model learns to jointly reason about semantic intent and temporal dynamics" (Section 1).

If validated, this inverts the conventional wisdom that large-scale paired editing datasets are the primary bottleneck for advancing video editing capabilities. Instead, the bottleneck may be representation factorization — and paired data mainly serves to refine rather than create editing ability.

Prior Approaches and Their Shortcomings

The paper identifies three broad families of prior work and analyzes where each falls short of resolving the semantic-motion tension.

Family 1: Zero-Shot and Few-Shot Adaptation of T2I Models

Early diffusion-based video editing methods (Section 2.1) attempted to repurpose pretrained text-to-image diffusion models for video by injecting temporal modeling as an afterthought. Representative works include Tune-A-Video [10], FateZero [13], Video-P2P [12], and TokenFlow [24]. These methods typically follow a pattern: (1) use DDIM inversion to encode the source video into latent codes, (2) apply text-to-image editing operations frame-by-frame, and (3) add cross-frame attention sharing or optical flow constraints to stitch frames together.

Where they fall short. The fundamental limitation is that temporal consistency is treated as a post-hoc constraint rather than an inherent property of the model. Cross-frame attention sharing can propagate edited content across frames, but it cannot guarantee that the content will be temporally consistent — it merely copies attention patterns, which can propagate artifacts just as easily as correct edits. Optical flow constraints assume that motion can be captured by 2D displacement fields, which fails under occlusions, camera motion, and complex non-rigid deformations. More critically, these methods have no mechanism for selective editing — the text-to-image model applies semantic changes globally, and the temporal post-processing cannot separate "desired edit-induced changes" from "unwanted temporal perturbations."

The authors do not experiment extensively with this family, but the broader community consensus (which the paper implicitly relies on) is that these methods produce video with noticeable flickering, jitter, and content drift, especially under complex instructions or fast motion.

Family 2: Data-Driven End-to-End Training on Paired Editing Data

With the release of large-scale instruction-guided video editing datasets (Señorita-2M, InsViE-1M, Ditto-1M, ReCo-Data, OpenVE-3M), a second wave of methods emerged that train video editing models end-to-end on paired (source video, instruction, target video) triplets. Representative works include Ditto [31], OpenVE-Edit [33], ReCo [32], VACE [36], and InsV2V [38]. These methods typically use a video diffusion backbone and train with reconstruction-style objectives that directly supervise the edited output.

Where they fall short. The paper identifies a subtle but critical limitation: these models learn editing as a monolithic mapping from (source video, instruction) to (target video) without any explicit factorization of semantic structure and motion dynamics. The consequence is that semantic and motion representations become entangled in the model's internal features. During training, the model learns spurious correlations — for example, associating a particular editing instruction with a particular motion pattern that happened to co-occur in the training data. At test time, this entanglement produces the observed tension:

When the model modifies pixels to satisfy the instruction, the entangled representations perturb motion features, causing temporal artifacts.
When the model preserves source motion, the shared representation space constrains how much semantic change can be applied, reducing edit fidelity.

The authors frame this as a representation learning problem rather than a data or scale problem. More data or larger models might mitigate but cannot eliminate the tension, because the architecture itself provides no mechanism for the model to learn that semantics and motion should be treated as separable degrees of freedom.

"We attribute the core difficulty of instruction-guided video editing to the lack of factorization between semantic structure planning and motion modeling" (Section 1).

Family 3: External Prior Injection (VLMs, Structural Conditions)

The most recent trend — and the one the paper most directly critiques — is to inject explicit external priors into the editing pipeline. This takes two main forms:

VLM-extracted semantic conditions: Methods like InstructX [14] and Kling-Omni [15] use multi-modal large language models (MLLMs) to analyze the source video, extract semantic parsing, and provide structured conditions that guide the diffusion process. The VLM acts as an external semantic reasoner, generating bounding boxes, segmentation masks, or scene graphs that constrain where and how edits are applied.
Structural signals: Methods like ControlVideo [16] and Control-A-Video [17] inject low-level structural priors such as depth maps, skeleton keypoints, or edge maps extracted from the source video. These provide explicit geometric constraints that the edited output must respect, separate from the semantic editing instruction.

Where they fall short. The authors' critique is pointed and specific:

"We argue that this over-reliance [on external priors] reflects a significant bottleneck, which constrains the diffusion backbone from learning inherent semantic-motion representations" (Section 1).

The argument has two layers:

First, an external prior is a brittle dependency. VLM-extracted semantic conditions are only as reliable as the VLM itself. If the VLM misdetects an object, misclassifies a region, or fails to understand complex spatial relationships, these errors propagate directly into the editing output with no mechanism for correction. Similarly, structural priors like depth maps are estimated by separate models that may fail under challenging conditions (low light, fast motion, complex occlusions). The entire editing pipeline inherits the failure modes of every external model in the chain.

Second, and more fundamentally, external priors prevent the diffusion backbone from developing its own internal representations. The model learns to "lean on" the external signals rather than learning to reason about semantic structure and motion directly from video data. This is a form of representation shortcut: the model bypasses the hard work of learning factorized representations because it can rely on pre-computed features that already separate "what" (semantics) from "where/how" (structure/motion). As a result, the backbone never develops the capability to internally disentangle these factors, making it fragile when external priors are noisy or unavailable.

The authors position their work as a direct response to this limitation: rather than injecting external priors to compensate for the lack of factorization, SAMA directly builds factorization into the backbone through its training objectives. The Semantic Anchoring and Motion Alignment objectives are not external signals — they are training-time losses that shape the model's internal representations.

How This Paper Positions Itself

SAMA is positioned at the intersection of two research streams that have largely been pursued independently.

Connection to semantic alignment in generation (Section 2.2). Recent work in image generation has shown that aligning a diffusion model's internal representations with pretrained visual encoders during training improves convergence speed and generation quality. REPA [45] aligns intermediate denoising features with clean features from a pretrained image encoder (e.g., DINOv2), providing a semantic training signal without requiring paired data. Follow-up works (REPA-E [46], HASTE [47], SRA [48]) refine when and how to apply this alignment. For video generation, VideoREPA [50] extends this idea to spatio-temporal relational alignment using video foundation models.

The paper explicitly builds on this lineage:

"Inspired by recent advances in semantic alignment for image/video generation, we apply semantic-alignment regularization to instruction-guided video editing" (Section 2.2, Positioning paragraph).

However, SAMA's Semantic Anchoring goes beyond standard REPA-style alignment in a crucial way: it jointly predicts semantic tokens and video latents in the same denoising trajectory, rather than merely aligning intermediate features. This makes the semantic signal an integral part of the generation process rather than a side constraint.

Connection to self-supervised video representation learning (Section 2.3). The paper draws on decades of work in self-supervised learning from unlabeled videos, particularly three families of pretext tasks: speed-based learning (SpeedNet [52], PRP [53], Pace Prediction [54]), spatiotemporal puzzles (Space-Time Cubic Puzzles [55]), and reconstruction-based objectives (VideoMAE [56], VideoMAE V2 [64]). These methods train models to solve artificially constructed tasks — predicting playback speed, restoring shuffled frames, inpainting masked regions — that force the model to internalize temporal dynamics.

SAMA's Motion Alignment component is a direct instantiation of this paradigm within a diffusion framework:

"Motivated by this line of work, we adopt lightweight pretext tasks as motion-centric restoration objectives in our Motion Alignment to better capture coherent temporal dynamics" (Section 2.3).

The novelty is not the pretext tasks themselves, but rather their integration into a video diffusion editing pipeline. In prior self-supervised work, the pretext task was the primary objective, and the learned representations were evaluated on downstream benchmarks (action recognition, video retrieval). In SAMA, the pretext tasks are auxiliary objectives that shape the model's internal motion representations in service of the editing task. The model never outputs a "pretext prediction" during inference — the pretext training simply teaches the backbone to reason about motion, which then benefits zero-shot and supervised editing.

Two complementary axes of factorization. The paper's conceptual framework is that semantic edits are sparse and temporally stable — a small number of anchor frames is often sufficient to specify the desired visual modification — while motion coherence is dense and learned from raw dynamics — it follows physical and temporal regularities that can be internalized from large-scale unlabeled videos. This observation motivates the two-factor design:

Semantic Anchoring addresses the "sparse and stable" nature of semantics by explicitly predicting semantic tokens only at a few anchor frames, giving the model a structural planning signal that is instruction-consistent but does not over-constrain the temporal dynamics.
Motion Alignment addresses the "dense and learned" nature of motion by forcing the model to recover temporal structure from perturbed inputs, thereby internalizing motion regularities without requiring explicit motion labels or optical flow annotations.

Crucially, neither component requires paired video-instruction editing data. Both are trained with readily available signals: semantic tokens from a pretrained image encoder (SigLIP) for SA, and self-supervised pretext reconstruction for MA. This means the factorized representations can be learned at scale from image editing data (for semantic coverage) and text-to-video data (for motion diversity), without the bottleneck of scarce paired video editing datasets.

The two-stage training strategy realizes this positioning. Stage 0 (factorized pre-training) teaches the model semantic anchoring and motion alignment on broad image and text-to-video data — no video editing pairs needed. Stage 1 (supervised fine-tuning) then aligns the already-factorized representations with specific editing instructions on paired data. The authors present this as evidence for their central hypothesis: once the model has learned to represent semantics and motion separately, instruction-guided editing becomes a relatively straightforward task of mapping instructions to the appropriate semantic tokens while preserving learned motion dynamics. The zero-shot editing results (Figure 5) are the key empirical validation of this claim — the model can edit before it has ever seen an editing example.

A unified perspective on conflicting prior findings. The paper's factorization framework provides a lens for understanding why prior work has produced mixed results. Models that lack factorization (end-to-end trained on editing data) can succeed on simple edits where the semantic change is localized and the motion is simple, but fail catastrophically on complex edits with fast motion or fine-grained semantic changes. Models that rely on external priors can handle complex semantics (the VLM provides strong structural guidance) but are brittle to prior quality and fail to develop robust internal representations. SAMA's contribution is not a new architectural trick but a training philosophy: teach the model to factorize semantics and motion first, then teach it to edit. This explains both the strong zero-shot results (factorization alone enables editing) and the competitive benchmark performance (supervised fine-tuning refines what factorization enables).

3. Technical Approach

3.1 Reader Orientation

SAMA is a two-stage training framework built on top of a pretrained video diffusion model that teaches the model to edit videos according to text instructions. The system solves the semantic-motion tension in video editing — where aggressive edits cause flickering and temporal artifacts, while motion preservation suppresses the intended edit — by factorizing the learning process into two complementary capabilities: (1) semantic anchoring, which learns to plan structural edits from sparse keyframes using a joint semantic-and-video prediction objective, and (2) motion alignment, which forces the model to internalize temporal dynamics by solving motion-centric restoration puzzles on raw videos, so that at inference time the model can apply precise semantic changes while naturally preserving coherent motion without relying on brittle external priors.

3.2 Big-Picture Architecture (Diagram in Words)

The SAMA system has five major components organized into two training stages:

Video Diffusion Backbone (Wan2.1-T2V-14B) — a pretrained text-to-video Diffusion Transformer (DiT) that serves as the base model. It operates in the latent space of a VAE, processing videos as sequences of latent tokens, and is trained via flow matching to predict the vector field from noise to clean video.
Semantic Anchoring Module — an auxiliary prediction head attached to the final DiT layer that jointly predicts semantic tokens extracted from a pretrained SigLIP image encoder at sparse anchor frames. This module operates during both training stages and forces the backbone to develop instruction-consistent semantic representations.
Motion Alignment Pretext Module — a set of three motion-centric video restoration tasks (Cube Inpainting, Speed Perturbation, Tube Shuffle) applied only during Stage 0 pre-training. The model receives perturbed source videos and must restore them, learning temporal dynamics directly from raw video data without paired editing supervision.
In-Context V2V Input Format — the source and target videos are encoded as latent token sequences and concatenated, with learned type embeddings distinguishing source latents (type 0), semantic tokens (type 1), and target latents (type 2). This unified format enables the model to process source context and generate edited output within a single denoising trajectory.
Two-Stage Training Pipeline — Stage 0 (Factorized Pre-training) learns semantic anchoring and motion alignment from broad image editing and text-to-video data, without any paired video-instruction editing examples. Stage 1 (Supervised Fine-tuning) refines editing behavior on paired video editing datasets, using the factorized representations learned in Stage 0.

Information flow at inference time: A source video and editing instruction enter the system → the VAE encoder converts the source video to latent tokens (type 0) → the text encoder processes the instruction → noisy target latent tokens (type 2) are initialized from Gaussian noise and concatenated with source tokens and semantic tokens (type 1, extracted from anchor frames via SigLIP) → the DiT iteratively denoises the full concatenated sequence using flow matching → the cleaned target latents are decoded by the VAE decoder into the edited video.

3.3 Roadmap for the Deep Dive

First, the flow matching formulation — the underlying generative framework, because understanding how the model produces videos is prerequisite to understanding how editing is achieved within it.
Second, the latent tokenization and in-context input format — how videos are represented as token sequences and how source context, semantic tokens, and target tokens are combined into a single denoising stream, since this unified format enables all subsequent components.
Third, Semantic Anchoring — the mechanism for providing instruction-aware structural priors by jointly predicting semantic tokens and video latents at sparse anchor frames. Understanding SA is essential because it is active in both training stages and is the primary mechanism for decoupling semantic planning from motion modeling.
Fourth, Motion Alignment — the three pretext tasks and why they teach the model to internalize motion dynamics. This naturally follows SA because MA provides the complementary motion representation axis, and explaining it after SA clarifies how the two components factorize the editing problem.
Fifth, the two-stage training strategy — how SA and MA are combined in Stage 0 (factorized pre-training on broad data) and then refined in Stage 1 (supervised fine-tuning on paired editing data). This ordering lets us see how the components fit together into a complete training recipe, building on the already-explained individual mechanisms.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a training methodology paper whose core idea is that video editing can be factorized into semantic anchoring (sparse, stable structural planning) and motion alignment (dense, learned temporal dynamics), and that teaching a video diffusion model these two capabilities through carefully designed auxiliary objectives — before any paired editing data is introduced — yields strong zero-shot editing and state-of-the-art fine-tuned performance without brittle external priors.

Flow Matching Formulation

SAMA inherits its generative framework from the flow matching paradigm, which the Wan2.1-T2V-14B backbone uses as its training objective. Understanding flow matching is essential because all video generation in SAMA — whether for editing, pretext restoration, or standard text-to-video generation — operates through this mechanism.

The core idea of flow matching is to learn a time-dependent vector field that transports samples from a simple prior distribution (standard Gaussian noise) to a complex target distribution (natural videos) along probability paths. Unlike diffusion models that learn to predict the added noise, flow matching directly regresses the vector field that defines the ordinary differential equation governing the transport.

The training objective minimizes the expected squared error between the model's predicted vector field and the true displacement from noise to data:

$L_{\text{FM}}(\theta) = \mathbb{E}_{t, \mathbf{x}_0, \mathbf{x}_1} \left\| \mathbf{v}_\theta(\mathbf{x}_t, t) - (\mathbf{x}_1 - \mathbf{x}_0) \right\|^2_2$

where $\mathbf{v}_\theta$ is the neural network parameterized by $\theta$ (the DiT backbone), $t \in [0,1]$ is the time step along the probability path, $\mathbf{x}_0 \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$ is a sample from the standard Gaussian prior, $\mathbf{x}_1$ is a real data sample (the target video latent tokens), and $\mathbf{x}_t = t\mathbf{x}_1 + (1-t)\mathbf{x}_0$ is the intermediate state constructed by linearly interpolating between noise and data.

What it computes: The network $\mathbf{v}_\theta$ takes as input the intermediate noisy state $\mathbf{x}_t$ and the time $t$ , and outputs a vector of the same dimensionality as $\mathbf{x}$ . The loss measures how far this predicted vector is from the true displacement $(\mathbf{x}_1 - \mathbf{x}_0)$ , which is the direction and magnitude from the noise sample to the clean data sample. Since $\mathbf{x}_t$ is constructed as a convex combination of noise and data, the true displacement is exactly the difference between the two endpoints. The expectation is taken over uniformly sampled times $t$ , noise samples $\mathbf{x}_0$ , and data samples $\mathbf{x}_1$ .

Why this form: Flow matching with linear interpolation defines a straight-line probability path between the prior and data distributions. The vector field along this path is constant — at every point $\mathbf{x}_t$ , the direction toward $\mathbf{x}_1$ is the same vector $(\mathbf{x}_1 - \mathbf{x}_0)$ . This is computationally simpler than diffusion formulations that require predicting the score (gradient of the log-density) or the added noise, because the target is directly the displacement. At inference time, generation proceeds by solving the ordinary differential equation:

$\frac{d\mathbf{x}}{dt} = \mathbf{v}_\theta(\mathbf{x}, t)$

starting from $\mathbf{x}_0 \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$ at $t=0$ and integrating to $t=1$ using a numerical ODE solver. The straight-line paths allow for efficient sampling with fewer integration steps compared to curved diffusion trajectories.

How editing fits into flow matching: In the standard text-to-video setting, $\mathbf{x}_1$ is a target video latent, and the model learns to transport noise to video conditioned on a text prompt. In SAMA's editing formulation, $\mathbf{x}_1$ is the edited target video latent, but the model also receives the source video latent tokens concatenated to the noisy target tokens (see next section). This means the model learns a vector field that transports noise to the edited output while attending to the source video context, effectively learning to predict how the source video should be modified. The condition information (text instruction, source video tokens, semantic tokens) is available to $\mathbf{v}_\theta$ through the transformer's self-attention mechanism, which allows the predicted vector field to depend on the full context.

Latent Tokenization and In-Context V2V Input Format

SAMA represents videos in a compressed latent space rather than operating on raw pixel frames, following the standard latent diffusion paradigm. This section explains how videos become token sequences and how the unified input format enables in-context video-to-video editing.

VAE encoding and decoding. The base model Wan2.1-T2V-14B includes a pretrained Variational Autoencoder (VAE) that compresses video frames into a lower-dimensional latent representation. For a source video $\mathbf{V}_s$ and a target video $\mathbf{V}_t$ , the VAE encoder produces latent token sequences:

$\mathbf{z}_s = \text{VAE-Enc}(\mathbf{V}_s), \quad \mathbf{z}_t = \text{VAE-Enc}(\mathbf{V}_t)$

The VAE decoder performs the inverse operation, reconstructing video frames from latent tokens. The paper does not train or modify the VAE — it uses the frozen Wan2.1 VAE throughout all experiments. The latent representation reduces both spatial and temporal resolution compared to the raw video, making it computationally feasible to process videos within a transformer backbone. The exact compression ratio and latent dimensionality are inherited from Wan2.1's architecture.

In-context concatenation. Unlike approaches that process the source video through a separate encoder or use DDIM inversion to extract conditioning signals, SAMA directly concatenates the source and target latent token sequences to form an in-context input:

$\mathbf{z} = [\mathbf{z}_s ; \mathbf{z}_t]$

where $[\cdot ; \cdot]$ denotes sequence concatenation along the token dimension. This means the DiT sees both source and target tokens in a single unified sequence, and the self-attention mechanism can freely attend between them. The target tokens $\mathbf{z}_t$ are the ones being denoised — during training, they are constructed as $\mathbf{z}_{t,\text{noisy}} = t\mathbf{z}_t + (1-t)\mathbf{z}_0$ where $\mathbf{z}_0 \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$ , and the flow matching loss is computed only on the target token positions.

Why concatenation instead of separate encoding? This in-context formulation has several advantages over alternatives. First, it uses the exact same self-attention mechanism that processes spatial and temporal dimensions to also process source-target interactions — no new architectural components are needed. Second, the source tokens provide a full, fine-grained representation of the source video that the target generation can reference at every denoising step and at every spatial location. Cross-attention to a separately encoded source would bottleneck information through a compressed representation, potentially losing fine spatial or temporal details needed for precise editing. Third, the concatenation approach is naturally compatible with the semantic token injection described in the next section — semantic tokens are simply prepended to the target sequence, creating a three-part input that the DiT processes uniformly.

Type embeddings. To disambiguate which tokens belong to which role, SAMA adds a learned type embedding vector to each token before it enters the transformer. Three type IDs are defined:

Type 0: Source video latent tokens $\mathbf{z}_s$
Type 1: Semantic tokens (introduced in the next section)
Type 2: Target video latent tokens $\mathbf{z}_t$ (the ones being denoised)

The type embedding for each token is a learned vector of the same dimensionality as the token embeddings, added element-wise. This gives each token a distinctive signature that the transformer can use to modulate its attention patterns — for example, target tokens might learn to query source tokens for content to preserve, while source tokens should never be modified by the denoising process.

Why type embeddings over shifted RoPE? The paper explicitly compares type embeddings against the commonly used shifted Rotary Position Embedding (RoPE) scheme (Appendix A, Table 6). Shifted RoPE modifies the positional encoding of the target sequence so that its positions are offset relative to the source sequence, which implicitly distinguishes them through position-dependent attention biases. However, the authors find that type embeddings "yield faster and more stable convergence than shifted RoPE, likely because they decouple token role from token position: positional encoding continues to capture spatiotemporal structure, while token identity is modeled separately." In a LoRA fine-tuning experiment on the Ditto-1M Replace subset evaluated on VIE-Bench Replace, type embeddings achieved an overall score of 6.975 versus 6.498 without, with the largest gain in content preservation (7.533 vs. 6.257), suggesting that explicit role disambiguation helps the model better maintain non-edited regions.

Training-time vs. inference-time usage. During training, the source tokens $\mathbf{z}_s$ are clean (not noised), while the target tokens $\mathbf{z}_t$ are noised according to the flow matching schedule. Semantic tokens (type 1), when present, are also noised together with the target tokens — a design choice explained in the next section. During inference, only the target token positions are initialized from noise; the source tokens and predicted semantic tokens are placed in their respective positions, and the model denoises the full sequence to produce the edited video latents and predicted semantic tokens.

Semantic Anchoring

Semantic Anchoring (SA) is the mechanism by which SAMA provides the diffusion backbone with an instruction-consistent structural prior. Rather than relying on external VLMs to extract high-level scene understanding, SA makes semantic representation an integral part of the denoising objective: the model jointly predicts semantic tokens along with video latents at sparsely sampled anchor frames, learning to ground instruction-following in a structured semantic space.

Anchor frame selection. For each video sample (whether from editing pairs, text-to-video data, or image editing data), the system uniformly samples $N$ frames from the target video to serve as anchor frames. The paper states $N = 1$ is used in all experiments for efficiency. These anchor frames are sparse relative to the full video — for a typical 81-frame video at 480p resolution, selecting just 1 frame means the semantic signal covers only ~1.2% of the temporal extent. The key design principle is that semantic edits are temporally stable: what needs to change (an object's identity, a color, a style) is usually consistent across the entire video, so specifying it at a single anchor frame is sufficient for planning the edit. The remaining frames' semantics are inferred by the model through temporal self-attention, which propagates the anchor-frame semantic signal across time.

Semantic token extraction. Each selected anchor frame is passed through a pretrained SigLIP image encoder, which produces patch-level features. SigLIP is a vision-language model trained with sigmoid-based contrastive loss, and its image encoder produces features that capture both semantic content (what objects are present, their attributes) and spatial layout (where objects are located, their relative positions). The patch-level features are then aggregated into a compact token set through a pooling operation, producing:

$M$ local semantic tokens — each capturing region-level semantics (object identity, attribute, texture) for a specific spatial region of the anchor frame. The paper sets $M = 64$ throughout all experiments.
One global semantic token — summarizing the overall content of the anchor frame (scene type, dominant style, global attributes).

The total semantic token count is thus $N \times (M + 1) = 1 \times 65 = 65$ tokens per video in the standard configuration. This is a dramatic compression from the raw anchor frame (which might contain thousands of latent tokens), forcing the semantic representation to be a concise structural summary rather than a pixel-level copy.

Projection into the diffusion space. The extracted semantic tokens are in the feature space of the SigLIP encoder, which has a different dimensionality and statistical distribution than the VAE latent space that the DiT operates on. To bridge this gap, the semantic tokens are projected through a lightweight two-layer MLP (multi-layer perceptron) into the same embedding space as the VAE latent tokens. This MLP is trained from scratch — it is the only new parameter introduced by SA beyond the standard backbone. The projected semantic tokens are denoted as $\hat{\mathbf{s}}$ :

$\hat{\mathbf{s}} = \text{MLP}(\text{SigLIP}(\text{AnchorFrame}(\mathbf{V}_t)))$

Injecting semantic tokens into the denoising trajectory. This is the critical design decision that distinguishes SAMA's SA from standard representation alignment approaches like REPA. Rather than treating semantic features as an external regularization target that is compared against intermediate DiT features, SA injects the projected semantic tokens directly into the denoising sequence as additional tokens that are noised and denoised alongside the target latents.

Concretely, the projected semantic tokens $\hat{\mathbf{s}}$ are prepended to the target latent token sequence $\mathbf{z}_t$ , forming an augmented target sequence:

$\mathbf{z}_t^{\text{aug}} = [\hat{\mathbf{s}} ; \mathbf{z}_t]$

The type ID 1 is assigned to these semantic token positions. When constructing the noisy intermediate state for flow matching, the same forward noising process is applied to both semantic tokens and target latents:

$\mathbf{z}_{t,\text{noisy}}^{\text{aug}} = t[\hat{\mathbf{s}} ; \mathbf{z}_t] + (1-t)\mathbf{z}_0^{\text{aug}}$

where $\mathbf{z}_0^{\text{aug}} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$ includes Gaussian noise for both the semantic and latent positions. The full input to the DiT is then:

$\mathbf{z}^{\text{input}} = [\mathbf{z}_s^{\text{clean}} ; \mathbf{z}_{t,\text{noisy}}^{\text{aug}}]$

where $\mathbf{z}_s^{\text{clean}}$ are the source latent tokens at type 0 (always clean, never noised).

Why noise the semantic tokens? Noising the semantic tokens together with the target latents means that the model must learn to denoise them jointly — at each denoising step, it predicts the clean semantic tokens and the clean target latents simultaneously from the same noisy intermediate state. This creates a tight coupling between semantic understanding and video generation: the model cannot produce good video latents without also recovering the correct semantic tokens, and vice versa. The semantic prediction task is not a side constraint but an integral part of the generation process. This contrasts with REPA-style alignment, where semantic features are extracted from a separately pretrained encoder and used as a regression target for intermediate features — in that setting, the semantics are "external" and the model merely aligns to them. In SA, the semantics are "internal" — they are part of what the model generates, making semantic reasoning inseparable from video generation.

Semantic prediction head and loss. After the full denoising trajectory completes (or at any intermediate step during training), the positions corresponding to the semantic tokens are read out from the final DiT layer. A dedicated semantic prediction head — a small neural network attached to these output positions — produces the predicted clean semantic tokens $\mathbf{s}$ :

$\mathbf{s} = \text{Head}_{\text{sem}}(\text{DiT}(\mathbf{z}^{\text{input}})[\text{semantic positions}])$

The prediction is supervised with an L1 (mean absolute error) loss between the predicted tokens and the originally extracted anchor tokens:

$L_{\text{sem}} = \|\hat{\mathbf{s}} - \mathbf{s}\|_1$

What it computes: For each semantic token position, the absolute difference between the extracted SigLIP feature (projected through the MLP) and the DiT's predicted value is computed and averaged across all $M+1$ semantic tokens. The L1 loss penalizes deviations linearly, which is less sensitive to outliers than L2 (MSE) and encourages the model to match the semantic tokens in a sparse, high-dimensional feature space without over-focusing on a few large errors.

Why this form: The L1 loss is chosen over L2 (mean squared error) likely because semantic token features from SigLIP are high-dimensional and sparse — they encode discrete concepts (object presence, attribute values) in a continuous embedding space. L2 heavily penalizes large deviations, which could cause the model to "average" features across multiple valid configurations (e.g., blending features of two possible objects rather than committing to one), producing blurry semantic representations. L1 encourages sparser, more decisive predictions that better match the discrete nature of semantic categories.

Combined objective. The full training loss during both stages combines the flow matching loss on the video latents with the semantic anchoring loss:

$L = L_{\text{FM}} + \lambda \cdot L_{\text{sem}}$

where $\lambda = 0.1$ is the loss weight for the semantic anchoring term. This weight is set empirically — it balances the flow matching objective (which directly optimizes video quality) with the semantic anchoring objective (which provides structural guidance). Setting $\lambda$ too high would cause the model to prioritize semantic token matching over video fidelity, potentially producing videos with correct semantics but poor visual quality. Setting it too low would make the semantic signal too weak to influence the model's internal representations.

Design justification — why semantic anchoring instead of VLM injection. The paper's motivation for SA over external VLM priors has several layers. First, VLM-injected conditions are brittle: if the VLM misdetects an object or misclassifies a region, the error propagates directly into the edited video with no mechanism for correction. Second, VLM injection prevents the diffusion backbone from developing its own semantic understanding — the model learns to rely on the external signal rather than learning to parse semantics from video data directly. Third, SA is efficient: SigLIP feature extraction and the lightweight MLP projector are much cheaper than running a full VLM at inference time. Fourth, by making semantic prediction part of the denoising objective, SA creates a tight feedback loop between semantic understanding and video generation — improvements in one benefit the other, creating a virtuous cycle during training.

Empirical justification for SA. The paper provides both quantitative and qualitative evidence for SA's effectiveness in Section 4.4. Quantitatively, incorporating SA on a Wan2.2-T2V-5B baseline (smaller backbone for ablation efficiency) improves the overall VIE-Bench score from 6.312 to 6.696 (Table 5). Qualitatively, models trained with SA produce higher-quality edits at earlier training stages compared to the baseline without SA trained for the same number of steps (Figure 6a), with the baseline often yielding incomplete or less accurate modifications. The training loss curves (Figure 6b) show that SA "accelerates the decrease of the diffusion loss, leading to faster DiT convergence" and "stabilizes training, as evidenced by a noticeably reduced loss variance." This faster convergence is a strong signal that SA provides useful gradient information — the semantic prediction task helps the model learn a better internal representation faster, which in turn speeds up video generation learning.

Motion Alignment

Motion Alignment (MA) is the complementary axis to Semantic Anchoring, designed to teach the diffusion backbone to internalize coherent temporal dynamics. While SA addresses the "what to change" aspect of editing (semantic planning), MA addresses the "how to move" aspect (motion coherence). The mechanism is training the model on motion-centric video restoration pretext tasks — given a temporally perturbed source video, the model must generate the original unperturbed version. This forces the backbone to develop internal representations of motion dynamics without requiring any motion labels, optical flow annotations, or paired editing supervision.

Core mechanism: source-side perturbation with clean target. MA is applied exclusively to the source video stream during Stage 0 (factorized pre-training). The key design rule is:

"Given a source video $\mathbf{V}_s$ and instruction $y$ , we apply a motion-centric transformation $T$ only to the source video to obtain $\tilde{\mathbf{V}}_s = T(\mathbf{V}_s)$ , while keeping the target side unchanged (i.e., always using the original target video without augmentation)" (Section 3.3).

This means the model receives a perturbed source concatenated with a clean target that it must generate. The flow matching objective then trains the model to transport from noise to the clean original video, conditioned on the perturbed version. Because the perturbation is purely temporal (it corrupts motion but preserves semantic content and visual appearance), the only way for the model to succeed is to learn to infer the correct temporal dynamics from the disrupted input — exactly the capability needed for video editing, where the source provides motion cues that the edited output should preserve.

Computationally, this works as follows. Let $\mathbf{V}_s$ be the original source video, and let $\tilde{\mathbf{V}}_s = T(\mathbf{V}_s)$ be the perturbed version. The VAE encodes both:

$\tilde{\mathbf{z}}_s = \text{VAE-Enc}(\tilde{\mathbf{V}}_s), \quad \mathbf{z}_t = \text{VAE-Enc}(\mathbf{V}_s)$

Note that the target is the original unperturbed video — the model is essentially performing video restoration from the perturbed input. The concatenated input is $[\tilde{\mathbf{z}}_s ; \mathbf{z}_{t,\text{noisy}}]$ , and the flow matching target is the clean $\mathbf{z}_t$ (the original video). The text instruction is the pretext task prompt (described below), which explicitly tells the model what restoration operation to perform.

Three pretext perturbation types. The paper adopts three perturbation strategies motivated by self-supervised learning literature for video representation learning:

1. Cube Inpainting. A continuous spatiotemporal block of frames and spatial regions is masked out from the source video. Concretely, the paper states a "masking ratio of 30%" (Appendix C), meaning that 30% of the video volume (spanning contiguous frames and spatial patches) is zeroed out or replaced with a mask token. The remaining 70% of the video provides partial temporal and spatial context from which the model must infer the missing content. The corresponding pretext instruction is:

[Complete the missing regions in the video.]

What this teaches: Cube Inpainting forces the model to understand temporal continuity — how objects move, how backgrounds evolve, and how appearance transforms across time. To fill in a missing temporal block, the model must infer the trajectory of objects from the before-and-after frames, understand occlusion relationships, and generate physically plausible intermediate motion. This is directly relevant to video editing because many edits (object removal, inpainting-based modifications) require the model to generate content that seamlessly continues the existing temporal flow.

2. Speed Perturbation. The source video is temporally accelerated, while the target remains at normal playback speed. The paper applies a "2× temporal acceleration" (Appendix C), meaning every other frame is dropped, and the remaining frames are played at double speed. The pretext instruction is:

[Restore the video to normal playback speed.]

What this teaches: Speed perturbation forces the model to learn motion rate normalization — the ability to infer the natural speed of actions, camera movements, and scene dynamics from an unnaturally fast input. This is relevant to editing because many editing operations (adding objects, changing appearance) must respect the original motion rate: an added object must move at the correct speed to appear physically plausible, and style changes must not alter the perceived tempo of actions.

3. Tube Shuffle. The source video is divided into a grid of spatiotemporal tubes and randomly permuted. The paper divides each video into a "2×2×2 spatio-temporal tube grid" (Appendix C), producing 8 tubes (2 spatial height divisions × 2 spatial width divisions × 2 temporal segments). These 8 tubes are randomly shuffled, destroying both spatial layout and temporal order. The pretext instruction is:

[Restore the correct spatio-temporal order of the video segments.]

What this teaches: Tube shuffle forces the model to learn spatiotemporal reasoning at a global level — it must identify which spatial region belongs at which location and which temporal segment belongs at which time, reconstructing both spatial coherence (objects should not have their top half in one location and bottom half in another) and temporal continuity (frames must be in the correct order for motion to flow). This is the most challenging pretext task because it corrupts both spatial and temporal structure simultaneously, requiring the model to develop a holistic understanding of video structure. For editing, this capability is valuable because it teaches the model to detect and correct spatiotemporal inconsistencies — essentially the same skill needed to ensure edited content is spatially well-placed and temporally consistent with surrounding context.

Sampling ratio for pretext tasks. During Stage 0 training on text-to-video data, the paper uses a sampling ratio of 1:2:3:4 for no-pretext : cube inpainting : speed perturbation : tube shuffle. This means for every batch of 10 text-to-video samples, approximately 1 uses no pretext task (standard text-to-video generation), 2 use cube inpainting, 3 use speed perturbation, and 4 use tube shuffle. The increasing ratio with task complexity reflects a curriculum-like design: the model spends more training on the harder tasks (tube shuffle requires the most reasoning, cube inpainting the least among the three), ensuring it develops strong spatiotemporal reasoning before moving to simpler restoration objectives. The inclusion of no-pretext samples maintains the model's standard text-to-video generation capability, preventing catastrophic forgetting of the base prior.

Prompting formulation for pretext tasks. To make the restoration objective explicit, each pretext task is preceded by a short task token prepended to the original text caption for the video. The task tokens serve as instructional prompts that tell the model what type of restoration to perform, analogous to how editing instructions tell the model what type of edit to apply. The task tokens are fixed, short English phrases:

Cube Inpainting: [Complete the missing regions in the video.]
Speed Perturbation: [Restore the video to normal playback speed.]
Tube Shuffle: [Restore the correct spatio-temporal order of the video segments.]

These task tokens are concatenated with the original video caption to form the full text condition. The original caption provides semantic context (what objects are present, what the scene depicts), while the task token specifies the restoration operation. This format is intentionally similar to the instruction-guided editing format used during Stage 1 fine-tuning — the model learns to interpret a text instruction that specifies a transformation from source to target, whether that transformation is "restore corrupted motion" or "add a hat to the person."

Qualitative evidence for MA effectiveness. The paper provides visualizations of the model successfully performing all three pretext tasks (Appendix D, Figure 9). For Cube Inpainting, the model plausibly completes masked spatiotemporal regions, generating content consistent with the surrounding frames. For Speed Perturbation, it restores natural motion dynamics from temporally accelerated inputs. For Tube Shuffle, it reconstructs coherent spatiotemporal structure from shuffled tube grids. These results confirm that the pretext objectives achieve their intended effect — the model internalizes motion cues and temporal reasoning.

Quantitative evidence for MA effectiveness. Ablation results in Table 5 show that adding MA alone to the baseline (without SA) improves the overall VIE-Bench score from 6.312 to 6.711. When combining both SA and MA, the score reaches 7.095, a gain of 0.783 over baseline. The fact that SA alone (+0.384) and MA alone (+0.399) provide similar individual gains, and the combined gain (+0.783) is approximately the sum of the individual gains, suggests that SA and MA are complementary — they improve different aspects of editing quality with minimal overlap.

Qualitative motion preservation results. Figure 7 shows representative cases where MA improves temporal consistency. In a tennis scene with large camera motion, the model with MA maintains background sharpness and clear on-screen text, while the baseline without MA produces blurred background content. In a car replacement scene and a third dynamic example, the baseline "often loses background motion," while MA preserves coherent background dynamics. These results demonstrate that MA teaches the model to maintain temporal alignment with source motion, which is precisely the capability that instruction-guided editing demands but standard training fails to provide.

Why motion alignment instead of optical flow constraints? The paper does not explicitly compare against optical flow methods, but the design choice is consistent with the overall philosophy. Optical flow is an external prior — it must be estimated by a separate model, can fail under occlusions or fast motion, and provides only 2D displacement fields that cannot capture 3D camera motion, object interactions, or complex non-rigid deformations. MA, by contrast, teaches the model to internalize motion understanding through self-supervision — the model learns whatever representation of motion is needed to solve the pretext tasks, which may include 3D understanding, occlusion reasoning, and dynamic appearance changes that optical flow cannot represent. Moreover, because MA is trained within the same diffusion framework that is used for editing, there is no gap between the motion representations learned during pre-training and the motion reasoning needed during editing — they are the same internal features, shaped by the same objective, in the same backbone.

Two-Stage Training Strategy

SAMA's training strategy operationalizes the factorization insight: first teach the model to represent semantics and motion separately (Stage 0), then teach it to edit by mapping instructions onto the already-factorized representations (Stage 1). This section details the data, objectives, and training configurations for each stage.

Stage 0: Factorized Pre-training. Stage 0 starts from the pretrained Wan2.1-T2V-14B checkpoint and trains on a mixture of two data types, neither of which contains video editing pairs:

Image editing data (NHR-Edit: 720,087 pairs; GPT-Image-Edit: 1,015,170 pairs; X2Edit: 768,470 pairs): These are (source image, instruction, target image) triplets for instruction-guided image editing. Image editing provides broad semantic coverage — editing concepts like color change, object replacement, style transfer, and attribute modification are well-represented across these datasets. Since images are single frames, they teach the model what kinds of semantic changes instructions can specify, but provide no temporal dynamics.

Text-to-video data (Koala-36M: 1,532,716 videos; MotionBench: 53,879 videos): These are (text prompt, video) pairs for standard text-to-video generation. The text-to-video data supplies diverse real-world motion patterns — human actions, camera movements, object dynamics, physical interactions. By applying the three MA pretext perturbations to these videos (with the 1:2:3:4 sampling ratio), the model learns motion reasoning from data that spans a wide range of motion types.

During Stage 0, both SA and MA are active on appropriate samples:

SA is applied to all samples (both image editing and text-to-video), supervising semantic token prediction on $N=1$ randomly sampled anchor frame from the target.
MA is applied only to text-to-video samples, using the source-side perturbation scheme described in Section 3.3. Image editing samples use clean images (no MA), since image data has no temporal dimension to perturb.

The Stage 0 objective follows the combined loss in Equation 4:

$L = L_{\text{FM}} + \lambda \cdot L_{\text{sem}}, \quad \lambda = 0.1$

where $L_{\text{FM}}$ is the flow matching loss computed on the target latent tokens (which, for MA samples, are the recovered original video; for image editing samples, are the target edited image), and $L_{\text{sem}}$ is the L1 semantic anchoring loss.

Training hyperparameters for Stage 0: The paper states a learning rate of $2 \times 10^{-5}$ , global batch size of 448 for images and 112 for videos, training resolution of 480p, and support for multiple aspect ratios (1/2, 2/3, 3/4, 1/1 and their reciprocals). An exponential moving average (EMA) of model parameters is maintained with decay 0.9998 and updated every iteration.

Why image editing data in video pre-training? This is a non-obvious choice. Image editing data has no temporal dimension, so it cannot directly teach motion. However, the authors include it for two reasons. First, the image editing datasets provide much larger scale and broader semantic diversity than video editing datasets — the combined image data (>2.5M pairs) dwarfs the video editing data available (~1M pairs). Second, and more subtly, the SA objective operates at the anchor frame level, which is inherently frame-based — the SigLIP encoder processes each anchor frame independently, and the semantic tokens capture frame-level content regardless of temporal context. By training SA on diverse image editing concepts, the model learns a rich mapping from instructions to semantic tokens that generalizes across editing types. When video data is later introduced, the same SA mechanism transfers directly, because the semantic representation is frame-level and agnostic to whether the input is an image or video.

Stage 1: Supervised Fine-tuning (SFT). Stage 1 takes the Stage 0 checkpoint and fine-tunes it on paired video editing datasets:

Ditto-1M: only the Style subset is used (3,936 video editing pairs)
OpenVE-3M: Local Change, Background, Style, and Subtitles categories (818,232 pairs)
ReCo-Data: 206,596 pairs

A small portion of image editing data (NHR-Edit and Pico-Banana-400K, ~978K pairs) is mixed in to "preserve general instruction-following behavior" — without this image data mixture, the model might overfit to the specific editing distributions in the video datasets and lose its ability to handle unseen editing types.

During Stage 1, SA remains active but MA is not applied — the source videos are clean, not perturbed. This makes sense because Stage 1's goal is to teach the model to align its already-factorized representations with specific editing instructions and target videos. MA taught the model to reason about motion; Stage 1 teaches it to use that motion reasoning in service of editing. Applying MA perturbations during Stage 1 would create a distribution mismatch — the model would learn to edit from perturbed sources, which is not the inference-time condition.

The Stage 1 training configuration is: learning rate $2 \times 10^{-5}$ (same as Stage 0), global batch size 448 for images and 112 for videos, resolution 480p, multiple aspect ratios, EMA with decay 0.9998 updated every iteration, and $\lambda = 0.1$ for the SA loss weight.

The zero-shot editing phenomenon. One of the paper's most striking findings is that Stage 0 alone — without any paired video editing data — yields "strong zero-shot video editing behavior" (Section 4.3). Figure 5 shows qualitative results: at Stage 0, the model can already perform Replace, Add, Remove, Style, and Hybrid edits, producing consistent edits across multiple frames while largely preserving non-edited content. The Stage 1 fine-tuning then refines these zero-shot capabilities — the edits become sharper, more temporally consistent, and more instruction-faithful.

This finding validates the paper's central hypothesis: once a model has learned to separately represent semantic structure (via SA) and temporal dynamics (via MA), instruction-guided editing naturally emerges as the ability to map an instruction onto the appropriate semantic tokens while preserving the learned motion representations. The zero-shot results suggest that the bottleneck in prior work was not the lack of paired editing data per se, but the lack of factorized internal representations — when representations are entangled, the model needs paired supervision to disentangle them; when they are already factorized, editing becomes a straightforward mapping problem that can be learned from surprisingly little (or even zero) paired data.

Failure modes in zero-shot editing. The paper is candid about limitations: zero-shot editing exhibits three typical failure modes: "(i) attribute edits can be temporally inconsistent, e.g., the edited colors may vary across frames; (ii) newly added objects may appear slightly blurry; (iii) removal edits may leave residual ghosting" (Section 4.3). These failures reveal the boundaries of what factorization alone can achieve: SA provides a sparse semantic anchor (1 frame), so attributes that should be temporally consistent across frames may drift because the model must propagate the semantic signal from the single anchor frame across time purely through self-attention. Motion alignment teaches the model to reason about motion but does not provide explicit temporal consistency constraints on edited content. Stage 1 fine-tuning with paired editing data resolves these failure modes by providing direct supervision on temporally consistent editing — the paired data shows the model exactly what "consistent color across frames" or "sharp added object" means.

Data filtering. All datasets undergo a VLM-based coarse filtering stage (Appendix B) to remove low-quality or instruction-inconsistent samples. The filtering uses Qwen2.5-VL-72B as a judge, scoring each sample from 1–10 on four criteria: Instruction Following, Visual Quality, Content Preservation, and Motion Consistency (motion consistency only for videos). The judge runs three inference turns and scores are averaged. Samples are retained only if they meet thresholds: for images, Instruction Following ≥9, Visual Quality ≥9, Content Preservation ≥9; for videos, Instruction Following ≥8, Visual Quality ≥9, Content Preservation ≥8, Motion Consistency ≥8. These thresholds are deliberately higher for images (since image quality is easier to assess and editing artifacts are more visible in static frames) and include an explicit motion criterion for videos, ensuring that the training data for Stage 1 contains examples where the target video preserves source motion — reinforcing the factorization objective.

Summary of Key Design Patterns

Several design patterns recur throughout SAMA and reflect the underlying factorization philosophy:

Internal over external. Every component — SA's semantic tokens, MA's motion restoration, the type embeddings for token disambiguation — is an internal mechanism learned during training rather than an external module injected at inference. The SigLIP encoder is the only external model used, and it is applied only to extract semantic tokens, not to provide editing guidance. This contrasts with VLM-injection methods where an external model actively guides the editing process.

Joint prediction over separate objectives. SA is not a separate regularization loss applied to intermediate features — it is integrated into the denoising trajectory itself, with semantic tokens noised and denoised alongside video latents. This makes semantic reasoning inseparable from video generation, creating a tight coupling that the paper argues leads to better internal representations.

Self-supervision for representation learning. MA leverages classic self-supervised learning principles — solve a pretext task, learn useful representations — but applies them within a diffusion framework and in service of an editing downstream task. The pretext tasks are never evaluated at test time; their sole purpose is to shape the model's internal representations during training.

Sparse semantic signals. SA uses only $N=1$ anchor frame with $M=64$ local tokens plus one global token — a highly compressed semantic representation relative to the full video. This sparsity is intentional: it forces the model to extract the essential semantic content (what the edit should accomplish) without over-constraining the temporal dynamics (how the video should move). The model must propagate the sparse semantic signal across time through its own temporal reasoning, which is where the motion representations learned by MA become critical.

Two-stage factorization-then-specialization. The training pipeline first teaches general capabilities (semantic understanding through SA, motion reasoning through MA) on broad data without editing supervision, then specializes those capabilities to the editing task on paired data. This mirrors how humans learn complex skills — first develop fundamental competencies, then apply them to specific tasks — and explains why zero-shot editing emerges: the fundamental competencies are sufficient for basic editing even before task-specific fine-tuning.

4. Key Insights and Innovations

Innovation 1: Factorization as the Fundamental Bottleneck, Not Data Scarcity

The paper's most significant conceptual move is reframing the difficulty of instruction-guided video editing from a data problem to a representation problem. The dominant assumption in the field — visible in the rapid proliferation of large-scale editing datasets (Ditto-1M, OpenVE-3M, ReCo-Data, Señorita-2M) and the corresponding push toward end-to-end supervised training — is that collecting more paired (source, instruction, target) triplets is the primary path forward. SAMA directly challenges this premise by demonstrating that a model trained with zero video editing pairs can already edit (Section 4.3, Figure 5), and that the quality gap between zero-shot and fully fine-tuned editing is largely about refinement (sharpness, temporal consistency of attributes) rather than fundamental capability acquisition.

The diagnostic move is subtle but profound: the paper identifies entanglement of semantic and motion representations as the root cause of the semantic-motion tension, and argues that paired editing data is merely a bandage — it helps the model learn to suppress artifacts that arise from entangled representations, but it does not address the entanglement itself. This explains why even models trained on millions of editing pairs (VACE, Ditto, UniVideo) still exhibit temporal inconsistencies and instruction drift: they learn a mapping from (source, instruction) to (target) that works on average but fails when the semantic change requires disentangling fine-grained motion from appearance.

By contrast, SAMA's factorization-first approach decouples the learning problem into two axes that can be trained on abundant, unpaired data — semantic understanding from image editing and text-to-video data, motion reasoning from self-supervised pretext tasks on raw videos. The zero-shot editing results are the empirical linchpin: they demonstrate that factorization alone is sufficient for basic editing capability, validating the diagnosis that entanglement, not data scarcity, was the binding constraint. This inverts the conventional wisdom about what limits progress in video editing and redirects research attention from dataset construction to representation learning.

The significance extends beyond video editing. The paper identifies a general principle: when a generative task exhibits tension between two competing objectives (here, semantic change vs. motion preservation), the solution may not be more paired training data that jointly optimizes both, but rather a training strategy that first learns each objective separately on abundant unpaired data, then combines them. This is an architectural insight about how to structure learning for multi-objective generative tasks — it suggests that factorized pre-training on decoupled objectives could benefit any domain where output quality requires simultaneously satisfying conflicting constraints (e.g., style transfer that must preserve content, image inpainting that must maintain global consistency, audio separation that must isolate sources while preserving quality).

This is a fundamental reframing, not an incremental improvement. Prior work treated the semantic-motion tradeoff as an optimization problem to be solved with more data, better architectures, or external constraints. SAMA treats it as a representation learning problem whose solution is factorization, and then demonstrates that the factorization can be achieved with self-supervision rather than paired supervision. The conceptual shift from "we need more editing data" to "we need factorized representations" is the paper's deepest intellectual contribution.

Innovation 2: Joint Semantic-Video Denoising as Internal Prior

SAMA's Semantic Anchoring is superficially similar to representation alignment methods like REPA — both use pretrained visual encoders to inject semantic signals into diffusion training. But the mechanism and its implications are fundamentally different. REPA aligns intermediate DiT features with encoder features through a separate regression loss — the semantic signal is an external constraint that regularizes the model's internal representations. SA, by contrast, noises and denoises semantic tokens alongside video latents in the same trajectory, making semantic prediction an integral part of the generative process rather than a side constraint.

This design choice has two cascading conceptual implications that distinguish SA from prior alignment work.

First, it eliminates the external-internal gap. In REPA-style alignment, the semantic encoder (DINOv2, SigLIP) processes clean images, while the DiT processes noisy latents — the alignment loss tries to match features computed under fundamentally different input distributions. This creates a representational mismatch: the DiT learns to produce features that look like clean encoder features when projected through an alignment head, but there is no guarantee that those features are causally useful for generation. In SA, the semantic tokens are part of the same denoising problem as the video latents — the model must predict clean semantics from the same noisy state that it uses to predict clean video. This creates a tight causal coupling: the internal representations that are good for semantic prediction must also be good for video generation, because they are produced by the same forward pass.

Second, it converts semantic knowledge from a training-time scaffold into an inference-time capability. Because SA is not a separate loss applied to intermediate features but part of the denoised output, the model learns to generate semantic tokens as a byproduct of video generation. The authors do not exploit this at inference time (semantic tokens are extracted from the target anchor frame by SigLIP, not generated by the model), but the capability is latent — the model has learned to produce SigLIP-compatible semantic features from noisy inputs. This opens the door to semantic token prediction as a planning mechanism: in future work, the model could be prompted to generate semantic tokens first (a form of structural imagination), then condition video generation on those self-generated tokens, enabling fully internal semantic planning without any external encoder at inference time.

The ablation evidence supports the claim that SA is more than aligned features. The training loss curves (Figure 6b) show that SA accelerates DiT convergence and reduces loss variance — this is not typical for an auxiliary regularization loss, which often increases training time. The fact that SA speeds up training suggests it provides gradient signal that directly aids the primary flow matching objective, which is consistent with the interpretation that semantic prediction is genuinely helpful for learning video generation, not merely an additional constraint. This is a methodological refinement of the REPA paradigm (it builds on the same insight that semantic encoders can guide diffusion training) but a conceptual departure in how the semantic signal is integrated — from external regularization to internal generation.

Innovation 3: Self-Supervised Motion Restoration as Motion Internalization

The paper's Motion Alignment component adapts self-supervised learning pretext tasks — a decades-old idea from video representation learning — to a diffusion-based editing framework in a way that repurposes them from evaluation-oriented representation learning to capability-oriented skill acquisition. This is a subtle but important distinction.

Classic self-supervised video learning (SpeedNet, VideoMAE, Space-Time Cubic Puzzles) trains models on pretext tasks and then evaluates the learned representations on downstream benchmarks (action recognition, video retrieval). The pretext task is never used at test time; it is merely a training vehicle for learning useful features. In SAMA, the pretext tasks are similarly never used at test time — no one asks the model to unshuffle tubes or restore playback speed during editing. But the purpose is fundamentally different: the goal is not to learn general-purpose video features that transfer to action recognition, but to learn motion reasoning skills that directly improve the model's ability to preserve temporal dynamics during editing.

This shift in purpose is reflected in the design. The pretext perturbations are applied only to the source video, while the target remains the clean original — the model must restore motion from corrupted input. At editing time, the source video is clean, but the model must generate an edited target that preserves the source's motion. The skill transfer is direct: restoration from perturbation teaches the model to infer and reproduce temporal dynamics from a reference, which is exactly what editing demands (infer motion from source, reproduce it in edited target). The pretext tasks are curriculum learning for motion preservation — they teach the model what coherent motion looks like by forcing it to reconstruct it from corrupted versions.

The complementary gains from SA and MA (Table 5: SA alone +0.384, MA alone +0.399, combined +0.783) provide quantitative evidence that motion reasoning and semantic planning are indeed separate capabilities that can be independently improved. The near-additivity of the gains suggests minimal overlap — SA does not incidentally teach motion, and MA does not incidentally teach semantics, validating the factorization design.

The qualitative evidence (Figure 7) shows that MA specifically improves motion coherence under challenging conditions: large camera motion, fast dynamics, complex backgrounds. These are precisely the cases where standard training (without explicit motion objectives) fails because the model has no incentive to learn motion representations — the standard flow matching loss can be reduced by memorizing static appearance patterns without understanding temporal dynamics. MA forces the model to attend to motion by making motion the primary axis of variation in the training signal.

This is a methodological innovation in how self-supervised learning is applied to generative models. Rather than using pretext tasks for representation evaluation, SAMA uses them for capability bootstrapping — teaching a specific downstream skill (motion preservation) through carefully designed pre-training objectives that isolate that skill from other factors (semantics are held constant by using clean target videos). The approach is general: any generative task that requires preserving some aspect of the input while modifying another could benefit from pretext pre-training that isolates the preservation axis.

Innovation 4: Zero-Shot Editing as a Diagnostic for Representation Quality

The paper's finding that Stage 0 pre-training alone yields strong zero-shot video editing (Section 4.3, Figure 5) functions as more than a performance claim — it serves as a diagnostic tool that validates the factorization hypothesis. The logic is: if the core difficulty of video editing is the semantic-motion entanglement, and if SAMA's pre-training successfully disentangles these representations, then editing should emerge from the disentangled representations even without explicit editing supervision. The zero-shot results confirm this prediction, providing causal evidence for the factorization hypothesis that correlational benchmarks (SFT performance) alone could not.

This is significant because it establishes a new evaluation paradigm for video editing models. Typically, editing capability is measured by fine-tuning on paired data and evaluating on held-out editing examples. This conflates two factors: the quality of the model's internal representations and the effectiveness of the fine-tuning process. A model might achieve high SFT performance through brute-force memorization of editing patterns while having poor underlying representations. The zero-shot editing test decouples these factors — it measures what the model can do purely from its pre-trained representations, before any editing-specific supervision.

The failure modes observed in zero-shot editing are equally diagnostic. The three issues — temporally inconsistent attributes, blurry added objects, residual ghosting in removals — reveal the boundaries of factorization-based editing. SA provides a sparse semantic anchor (1 frame), so attributes that should be consistent across frames can drift — the model must propagate the semantic signal through temporal self-attention, which is an implicit rather than explicit consistency mechanism. Added objects have no motion reference in the source, so the model must generate plausible dynamics from scratch — MA teaches motion reasoning but does not provide a motion template for newly introduced content. Removals requiring background inpainting demand both semantic understanding (what should fill the removed region) and motion reasoning (how that fill should move), pushing the limits of what sparse semantic anchors can specify. These boundaries are informative: they show where factorization alone is insufficient and where paired editing data adds value (providing direct supervision on temporal consistency of attributes, sharpness of added objects, and coherence of inpainted regions).

This is a conceptual innovation in evaluation methodology. The zero-shot test transforms editing capability from a monolithic target (measured by post-SFT benchmarks) into a latent variable that can be partially measured before SFT, with the SFT gain representing the residual that paired data addresses. This decomposition enables more targeted research: improving zero-shot editing performance requires better factorization (stronger semantic anchors, more robust motion representations), while improving SFT gains requires better alignment between factorized representations and editing-specific objectives.

Innovation 5: Type Embeddings for Token Role Disambiguation as a Convergence Accelerator

The paper's use of learned type embeddings to distinguish source latents, semantic tokens, and target latents might appear to be a minor implementation detail. However, the ablation results (Appendix A, Table 6) and the authors' analysis reveal it as a non-trivial design choice with implications for how transformers process heterogeneous token sequences.

The standard approach for distinguishing sequences in concatenated transformer inputs is shifted RoPE — offset the positional encodings of the second sequence so that the model can attend position-specifically to each sequence. This works by making the positional encoding carry double duty: it encodes both spatiotemporal position (where in the video a token belongs) and sequence identity (which video — source or target — the token comes from). SAMA's type embeddings decouple these two functions: positional encoding captures spatiotemporal structure, while type embeddings capture token role. The result is faster convergence (Figure 8 qualitatively) and better content preservation (7.533 vs. 6.257 in Table 6 ablation).

The conceptual reason this matters extends beyond video editing. As models increasingly process heterogeneous inputs — multiple modalities, multiple temporal contexts, multiple semantic levels — the "what is this token" signal and the "where does this token belong" signal become increasingly conflated in standard positional encoding schemes. Type embeddings provide an explicit mechanism for the model to learn role-specific processing without sacrificing positional precision. The content preservation gain is particularly telling: by making token roles explicit, the model can learn attention patterns that respect role boundaries (e.g., "attend to source tokens for content to preserve, but do not modify them") without relying on positional heuristics that may fail under varied sequence lengths or aspect ratios.

This is an incremental architectural refinement (type embeddings are not novel per se — they are used in BERT, ViT, and many other transformers) but a conceptually significant application to the in-context video editing setting. The paper identifies that the standard approach (shifted RoPE) creates an unnecessary coupling between position and role that slows learning and degrades content preservation, and demonstrates that a simple disentanglement yields measurable gains. The broader principle — explicit role disambiguation benefits heterogeneous sequence processing — is transferable to any multi-stream transformer architecture.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on three benchmarks: VIE-Bench [14] (500 test videos, five editing categories: Add, Swap/Change, Remove, Style/Tone Change), OpenVE-Bench [33] (covers Style, Background Change, Local Change, Local Remove, Local Add, Subtitle Edit, Creative Edit), and ReCo-Bench [32] (tasks: Add, Replace, Remove, Style, evaluated on nine fine-grained metrics). Exact test set sizes are not stated beyond the VIE-Bench count of 500. All three are recent, publicly available benchmarks designed specifically for instruction-guided video editing evaluation.
Base model(s). The primary model is Wan2.1-T2V-14B [58], a 14-billion-parameter text-to-video Diffusion Transformer pretrained via flow matching. The authors state it provides a "strong text-to-video prior" (Section 3.4) with an established latent space and temporal modeling capabilities. For ablation studies (Section 4.4), the smaller Wan2.2-T2V-5B [78] is used for efficiency, trained on a subset of Ditto-1M. The choice of a text-to-video base model rather than an image diffusion model is deliberate: the base model already possesses temporal reasoning capabilities that SAMA's Motion Alignment can refine, and its video-native architecture avoids the need to retrofit temporal modules onto image backbones.
Metrics. The paper uses VLM-as-judge evaluation across all three benchmarks. For VIE-Bench, GPT-4o [75] scores each edited video on three axes: Instruction Following (how well the edit implements the instruction), Preservation (how well non-edited content and motion are preserved), and Quality (overall visual fidelity). For OpenVE-Bench, Gemini-2.5-Pro [76] provides per-category scores and a global average. For ReCo-Bench, Gemini-2.5-Flash-Thinking [77] evaluates on nine fine-grained metrics: Semantic Accuracy (SA), Scope Precision (SP), Content Preservation (CP), Appearance Naturalness (AN), Scale Naturalness (SN), Motion Naturalness (MN), Visual Fidelity (VF), Temporal Stability (TS), and Edit Stability (ES), which are then aggregated into three category scores (SEA for Edit Accuracy, SVN for Video Naturalness, SVQ for Video Quality) and an overall score S. All VLM scores are on a 1–10 scale, with higher scores indicating better quality. The choice of VLM judges is particularly appropriate for this work because the paper's central critique of prior methods is their reliance on external VLM priors during generation — using VLMs for evaluation creates a clean separation between training-time methodology and evaluation-time metrics, ensuring the evaluator does not inherit biases from the same VLM pipeline used by some baselines during editing.
Baselines. The paper compares against a comprehensive set of both closed-source and open-source systems. Closed-source baselines include: Kling1.6 [72], Kling-Omni [15], Runway (Gen-4) [23], MiniMax [73], and Pika [74]. Open-source baselines include: InsV2V [38], DiffuEraser [35], VACE [36], Omni-Video [42], UniVideo [43], InstructX [14], InsViE [30], LucyEdit [37], ICVE [39], Ditto [31], OpenVE-Edit [33], VINO [44], and ReCo [32]. Not all baselines are evaluated on all benchmarks — specific coverage is shown in Tables 2, 3, and 4. The baseline selection is deliberately broad, spanning end-to-end editing models (Ditto, OpenVE-Edit), unified generation-editing models (VACE, UniVideo, VINO), VLM-guided methods (InstructX, Omni-Video), and specialized removal methods (DiffuEraser).
Generation budget / compute accounting. The paper does not report a formal compute budget or FLOPs accounting. All models are compared at their default inference configurations — the paper does not control for inference-time compute across methods (e.g., number of denoising steps, model size, or generated frames). This is a meaningful omission in a paper whose contributions include a training methodology claim (factorized pre-training improves data efficiency). A FLOPs-matched analysis comparing SAMA's 14B backbone against smaller or larger models at equal compute, or analyzing whether the factorized pre-training reduces the training compute needed to reach a given performance, would strengthen the efficiency claims.
Cross-validation / statistical protocol. The paper does not describe a cross-validation protocol, report confidence intervals, or discuss statistical significance testing for any of the benchmark results. All numbers in Tables 2-4 are single-run evaluations. For the ablation study (Section 4.4), the paper uses a smaller backbone (Wan2.2-T2V-5B) and trains on a subset of Ditto-1M to reduce computational cost, but does not report multiple runs or error bars. The VLM-based evaluation pipeline involves averaging scores across three inference turns for data filtering (Appendix B), but it is unclear whether evaluation scores are similarly averaged or are single-turn judgments. This is a limitation — with 500 test questions on VIE-Bench, score differences of 0.1–0.3 points (which separate top methods in Table 2) may fall within VLM scoring variance, making it difficult to assess whether small gaps are statistically meaningful.

Main Quantitative Results

VIE-Bench: Broad Editing Categories

VIE-Bench provides the most comprehensive comparison, covering 15 competing methods across four editing categories with 500 test videos. SAMA's headline results are in Table 2.

Overall performance. SAMA achieves the highest average scores among open-source models across most categories, and is competitive with the best closed-source systems. On Swap/Change — arguably the most challenging category because it requires precise object replacement while maintaining pose and position — SAMA achieves an overall average of 9.340, exceeding all compared methods including closed-source leaders Kling-Omni (9.194) and Runway (9.161). This is notable because Swap/Change demands simultaneous fine-grained semantic accuracy (replacing the correct object with the correct replacement) and faithful motion preservation (maintaining the original subject's movement trajectory), the exact capability the factorization is designed to provide.

On Remove, SAMA achieves 9.144 overall, outperforming all open-source methods by a substantial margin (next best open-source: InstructX at 8.322) and exceeding the best closed-source system Runway (8.504). The gap is especially large on Instruction Following for Remove (SAMA: 9.533 vs. Runway: 8.664), suggesting that SA's semantic anchoring is particularly effective for removal edits, where the instruction must precisely specify what to remove while preserving everything else.

On Add, SAMA scores 8.711, slightly behind Kling-Omni (9.181) and UniVideo (8.656), but competitive with InstructX (8.349) and Runway (8.447). The Add category is the weakest for SAMA relative to closed-source systems — this aligns with the zero-shot failure mode observation (Section 4.3) that "newly added objects may appear slightly blurry," suggesting that MA's motion learning from source video restoration may not fully transfer to generating plausible dynamics for objects with no motion reference in the source.

On Style/Tone Change, SAMA achieves 9.259, competitive with Kling-Omni (9.341) and Runway (9.133), and exceeding InstructX (9.196). Style transfer stresses global semantic change while preserving fine motion details — a scenario where factorization should excel because SA handles the global semantic transformation and MA preserves source dynamics.

Head-to-head with Kling-Omni. Kling-Omni is the strongest closed-source competitor and represents the VLM-injection paradigm that SAMA critiques. Kling-Omni leads on Add (9.181 vs. 8.711) and Style (9.341 vs. 9.259), while SAMA leads on Swap/Change (9.340 vs. 9.194) and Remove (9.144 vs. 9.133). The margins on the categories where Kling-Omni leads are larger than those where SAMA leads, suggesting that VLM injection may provide stronger semantic guidance for additive edits (where the model must imagine and place new content) while SAMA's factorization excels at transformative edits (replacing, removing) that require tightly coupled semantic-motion reasoning.

Head-to-head with UniVideo. UniVideo is the strongest open-source competitor and a unified video understanding/generation/editing model. SAMA outperforms UniVideo on all categories except Add, where UniVideo scores 8.656 vs. SAMA's 8.711 — essentially tied within likely evaluation noise. On Remove, SAMA's lead (9.144 vs. 8.233) is the largest, consistent with the pattern observed against Kling-Omni.

Gap to closed-source on Add. The Add category reveals a persistent gap: SAMA's Instruction Following for Add is 8.467 versus Kling-Omni's 9.333, a 0.866-point difference. The Preservation scores are nearly identical (9.422 vs. 9.589), suggesting the bottleneck is not motion preservation but semantic accuracy — knowing what to add, where, and with what attributes. This may reflect a limitation of SA's sparse semantic anchoring: a single anchor frame with 64 local tokens may not capture the fine-grained spatial precision needed for accurate object placement, especially compared to VLM-extracted bounding boxes or segmentation masks that provide explicit spatial guidance.

OpenVE-Bench: Fine-Grained Editing Subcategories

OpenVE-Bench evaluates on more specific editing subcategories, using Gemini-2.5-Pro for scoring. Table 3 reports per-category and global scores.

Overall performance. SAMA achieves the highest global score of 4.05, outperforming the best open-source competitor Ditto (4.01) and the closed-source Runway (3.72). The numerical scale is different from VIE-Bench (Gemini-2.5-Pro uses different scoring calibration than GPT-4o), but the ranking is clear.

Subcategory strengths. SAMA leads or ties for best in three subcategories: Background Change (2.59 vs. Runway 2.62, essentially tied), Local Change (3.93 vs. Ditto 2.03 — a dramatic gap), and Creative Edit (3.11 vs. Ditto 1.23 — another dramatic gap). The large margins on Local Change and Creative Edit are consistent with the approach's strengths: these edits require modifying specific objects or regions while preserving everything else, which maps directly onto SA's semantic anchoring (which specifies what to change) and MA's motion preservation (which maintains temporal coherence in unedited regions).

Subcategory weaknesses. SAMA scores below the best competitor on Style (2.59 vs. OpenVE-Edit 2.36 — SAMA actually leads here per the table, so this is not a weakness), Local Remove (2.54 vs. UniVideo 2.98), and Subtitle Edit (3.63 vs. Runway 3.64, essentially tied). Local Remove is interesting because SAMA excelled at Remove on VIE-Bench — the discrepancy may reflect dataset-specific difficulty (OpenVE-Bench's removal tasks may be more challenging or involve finer spatial precision).

Peculiarities in OpenVE-Bench scores. Several baselines show anomalously low scores on specific categories that suggest evaluation inconsistency. Omni-Video scores 1.00 on Subtitle Edit — an implausibly low score that may indicate the model failed entirely on this subcategory or the VLM judge applied different standards. Ditto scores 1.23 on Creative Edit and 1.53 on Local Change, both dramatically below SAMA. These very low scores for some baselines should be interpreted cautiously — they may reflect genuine capability gaps, or they may reflect that some models were not designed for these specific editing types and produce degenerate outputs.

ReCo-Bench: Fine-Grained Quality Metrics

ReCo-Bench is the most fine-grained evaluation, splitting each editing category into nine quality dimensions scored by Gemini-2.5-Flash-Thinking. Table 4 reports per-category and per-metric results for Add, Replace, Remove, and Style.

Overall performance. SAMA achieves the highest overall scores (S column) on Replace (9.10 vs. UniVideo 8.90), Remove (8.09 vs. ReCo 7.00), and Style (9.25 vs. ReCo 9.17), and the second-highest on Add (8.92 vs. UniVideo 8.90, a marginal difference). This broad dominance across categories and metrics is the strongest quantitative evidence for the factorization approach's generality.

Metrics where SAMA leads. SAMA achieves the best or tied-for-best score on the majority of individual metrics across categories. On Replace, SAMA leads on 8 of 9 metrics (all except Appearance Naturalness, where UniVideo leads 7.73 vs. 7.77). On Remove, SAMA leads on 8 of 9 metrics (only Scope Precision trails ReCo, essentially tied at 8.71 vs. 8.71 — wait, the table shows SAMA 8.71 and ReCo 7.43 for Scope Precision, so SAMA leads). The consistency across metrics suggests that SAMA's improvements are not narrowly concentrated on one aspect of editing quality but reflect a genuinely better underlying representation.

Metrics where SAMA trails. On Add, SAMA trails UniVideo on Appearance Naturalness (7.44 vs. 7.27) and Content Preservation (9.83 vs. 9.69) — both marginal. On Style, SAMA trails ReCo on Semantic Accuracy (8.46 vs. 9.11) and several other metrics by small margins. The most persistent relative weakness is Appearance Naturalness, where SAMA trails the leader on Add and Replace. This may reflect the blurriness issue noted in zero-shot editing — objects may be slightly less crisp than in competing methods.

Comparison with ReCo (the method). ReCo is a region-constraint in-context generation method that explicitly uses region annotations for editing. SAMA outperforms ReCo on all four task categories, with the largest margin on Remove (8.09 vs. 7.00). This is significant because ReCo uses stronger spatial supervision (region constraints) than SAMA (which has no region annotations, relying purely on SA's semantic tokens for spatial grounding). The fact that SA's learned semantic anchoring outperforms explicit region constraints suggests that the factorized semantic representations capture spatial information effectively without requiring manual annotation.

Qualitative Results

Figures 4, 10, 11, and 12 provide extensive qualitative comparisons. The paper highlights several patterns:

Fine-grained instruction grounding (Figure 4, VIE-Bench). SAMA correctly handles relative position cues ("on the left"), attribute combinations ("alternating light and dark hair"), and complex replacements (pigeon-to-squirrel, seal-to-crab) that competing methods (Ditto, UniVideo, Kling) either fail to execute or execute inconsistently across frames.

Motion preservation under removal (Figure 4, ReCo-Bench). In a stroller removal example, SAMA "better preserves temporal alignment" while competing methods show background warping or residual ghosting. This is direct qualitative evidence that MA's source-motion preservation transfers to editing.

Style transfer with identity preservation (Figure 4). Stylization examples show SAMA maintaining subject identity and details while applying global style changes, whereas other methods "may drift or blur." This is consistent with SA providing a consistent semantic anchor (the subject's identity features are captured in the semantic tokens and propagated through temporal self-attention).

Zero-Shot Editing Results (Section 4.3, Figure 5)

The zero-shot results are presented qualitatively in Figure 5, comparing Stage 0 (factorized pre-training only, no paired video editing data) against Stage 1 (after SFT on paired data). Stage 0 produces edits that are "consistent over multiple frames while largely preserving non-edited content" across Replace, Add, Remove, Style, and Hybrid tasks. Stage 1 refinements are visible as sharper additions, more temporally consistent attributes, and cleaner removals.

Quantitative zero-shot results are absent. The paper does not report any numerical scores for Stage 0 editing on any benchmark. This is a notable omission — the zero-shot editing capability is presented as one of the paper's central findings ("Remarkably, this stage alone already induces strong zero-shot video editing behavior" — Section 1), but no quantitative metrics are provided to calibrate what "strong" means relative to baselines. Without benchmark scores, the reader cannot assess whether Stage 0 zero-shot editing is competitive with existing open-source models (which would be an extraordinarily strong result) or merely produces recognizable but low-quality edits.

The qualitative examples in Figure 5 are suggestive but not conclusive. The boat color change and dancing man removal look plausible but may have artifacts not visible in static frames. The illustration style transfer and girl addition show the characteristic zero-shot failure modes the authors acknowledge (temporal inconsistency, blurriness, ghosting). A quantitative comparison of Stage 0 against baselines like InsV2V or VACE would dramatically strengthen the claim that factorization alone enables editing — it would quantify the gap between zero-shot and fully supervised editing and establish a baseline for future work on factorization-based approaches.

Ablation Studies and Robustness Checks

Semantic Anchoring (SA) ablation (Table 5, Figure 6): Removing SA from SAMA reduces the overall VIE-Bench score from 7.095 to 6.711 (a drop of 0.384). The per-metric breakdown shows that SA improves all three axes (Instruction Following: 7.402 vs. 6.969; Preservation: 6.998 vs. 6.620; Quality: 6.884 vs. 6.544), with the largest relative gain in Instruction Following. Figure 6a provides qualitative evidence that SA accelerates editing quality convergence — at the same training step, the model with SA produces more complete and accurate edits, while the baseline yields "incomplete or less accurate modifications." Figure 6b shows that SA "accelerates the decrease of the diffusion loss, leading to faster DiT convergence" and "stabilizes training, as evidenced by a noticeably reduced loss variance." This is a non-obvious finding: an auxiliary semantic prediction loss does not merely add a regularizer that prevents overfitting — it actively accelerates the primary flow matching objective, suggesting that the semantic prediction task provides complementary gradient signal that aids video generation learning. The SA ablation was conducted on Wan2.2-T2V-5B (smaller backbone) trained on a subset of Ditto-1M, not the full 14B model — this is a reasonable efficiency tradeoff but may underestimate SA's impact on larger models where representation learning benefits typically scale.

Motion Alignment (MA) ablation (Table 5, Figure 7): Removing MA reduces the overall score from 7.095 to 6.696 (a drop of 0.399). MA alone (without SA) achieves 6.711, nearly identical to SA alone (6.696). The three components combine for a total gain of 0.783 over baseline, which is approximately the sum of individual gains (0.384 + 0.399 = 0.783), suggesting the two components are complementary with minimal interaction. The near-additivity of gains is the strongest evidence for the factorization hypothesis — if SA and MA were addressing overlapping aspects of editing, the combined gain would be sub-additive. Figure 7 provides qualitative evidence that MA specifically improves temporal consistency under fast motion and reduces motion blur, with representative cases including a tennis scene with large camera motion (MA maintains "background sharpness, e.g., clearer on-screen text, while the baseline appears blurred") and a car replacement scene (baseline "often loses background motion").

Type Embeddings ablation (Appendix A, Table 6, Figure 8): Removing type embeddings and using only positional encoding reduces the overall score from 6.975 to 6.498 (a drop of 0.477). The largest loss is in Preservation (7.533 vs. 6.257, a drop of 1.276), while Instruction Following (6.705 vs. 6.619) and Quality (6.686 vs. 6.619) show more modest changes. This is informative: type embeddings primarily help the model distinguish which tokens correspond to source content (which should be preserved) versus target content (which should be edited), and removing this explicit signal causes the model to inadvertently modify source content. The experiment was conducted under a LoRA setting on Wan2.2-T2V-5B trained on the Ditto-1M Replace subset and evaluated on VIE-Bench Replace, a more limited setting than the main experiments. Figure 8 provides qualitative evidence that the model without type embeddings produces worse content preservation (background changes, identity drift) on object replacement tasks.

Training stage ablation (Figure 5): Comparing Stage 0 (factorized pre-training only) against Stage 1 (after SFT) qualitatively demonstrates that SFT refines the zero-shot editing capability — edits become sharper, more temporally consistent, and exhibit fewer artifacts. However, as noted above, no quantitative scores are reported for Stage 0, making this a qualitative rather than quantitative ablation.

Critical Assessment

Claim 1: Factorization enables strong zero-shot editing

This claim is qualitatively demonstrated but quantitatively unsubstantiated. Figure 5 shows that Stage 0 produces recognizable edits without paired video editing data, which is genuinely impressive and supports the paper's central hypothesis. However, the paper reports no benchmark scores for Stage 0, making it impossible to assess whether "strong zero-shot video editing behavior" means "competitive with existing open-source models" (which would be a major finding) or "produces recognizable but low-quality edits" (which would be expected from most pre-trained video diffusion models prompted in-context). The zero-shot claim is the paper's most novel and potentially impactful finding — it effectively inverts the conventional wisdom that paired editing data is the primary bottleneck — but without quantitative evaluation, it remains a qualitative observation rather than an empirical demonstration.

A minimal experiment to strengthen this claim would be to evaluate Stage 0 on VIE-Bench and report scores alongside the open-source baselines in Table 2. If Stage 0 already approaches or exceeds methods like InsV2V or VACE (which are trained on paired editing data), this would be strong evidence for the factorization hypothesis. If Stage 0 scores are substantially lower, the claim that factorization "alone" enables editing would need qualification — factorization might provide a useful prior that SFT refines, but not independently sufficient editing capability.

Claim 2: SAMA achieves state-of-the-art among open-source models

This claim is strongly supported by the quantitative evidence. On VIE-Bench (Table 2), SAMA achieves the highest or tied-for-highest scores among open-source methods on Swap/Change (9.340), Remove (9.144), and Style (9.259), and is competitive on Add (8.711 vs. UniVideo 8.656). On OpenVE-Bench (Table 3), SAMA achieves the highest global score (4.05 vs. Ditto 4.01). On ReCo-Bench (Table 4), SAMA achieves the highest overall score on Replace (9.10), Remove (8.09), and Style (9.25), and is marginally second on Add (8.92 vs. UniVideo 8.90). The breadth of benchmarks (three), diversity of editing categories (Add, Replace, Remove, Style, Background, Local, Subtitle, Creative), and granularity of metrics (from 3 VIE-Bench axes to 9 ReCo-Bench metrics) make this a robust finding.

However, the open-source SOTA claim should be understood with two caveats. First, SAMA is built on Wan2.1-T2V-14B, which is an exceptionally strong base model (14B parameters, state-of-the-art text-to-video generation). Some competing open-source methods use smaller backbones: Omni-Video (1.3B), InsViE (2B), Lucy-Edit (5B), OpenVE-Edit (5B). The comparison is not strictly fair — SAMA may benefit substantially from its larger base model, and it is unclear how much of the performance gap is attributable to SAMA's training methodology versus the base model's scale and quality. A controlled experiment that applies SAMA's training to smaller backbones, or competing methods' training to the 14B Wan2.1 backbone, would isolate the methodological contribution.

Second, the paper does not report inference-time compute, making it possible that SAMA achieves higher quality through more expensive inference (more denoising steps, longer generation time) rather than better training. A compute-matched comparison would strengthen the SOTA claim.

Claim 3: SAMA is competitive with leading commercial systems

This claim is supported, but with clearer boundaries than the abstract suggests. On VIE-Bench Swap/Change, SAMA leads Kling-Omni (9.340 vs. 9.194) and Runway (9.161). On Remove, SAMA leads all commercial systems. On Style, SAMA trails Kling-Omni marginally (9.259 vs. 9.341). However, on Add, SAMA trails Kling-Omni by a substantial margin (8.711 vs. 9.181), and also trails Runway on Add (8.447). The Add category is the most commercially important (adding objects to videos is a common creative workflow), and SAMA's underperformance here suggests a genuine limitation rather than evaluation noise.

The gap on Add may reflect a fundamental limitation of the factorization approach. SA provides semantic anchoring on sparse frames (N=1), which specifies what to add but not precisely where with pixel-level accuracy. VLM-injection methods like Kling-Omni can extract spatial layouts (bounding boxes, segmentation masks) that provide explicit spatial guidance for object placement. Improving SA's spatial precision — perhaps through more anchor frames, finer-grained semantic tokens, or explicit position encoding in the semantic features — might close this gap.

Claim 4: Semantic Anchoring and Motion Alignment are complementary

This claim is strongly supported by the ablation study (Table 5). SA alone gains +0.384, MA alone gains +0.399, combined gains +0.783. The near-additivity of individual gains (0.384 + 0.399 = 0.783) is consistent with the hypothesis that SA and MA improve orthogonal aspects of editing — semantic planning and motion preservation, respectively — with minimal overlap in their contributions. This is the paper's cleanest quantitative evidence for the factorization design.

However, the ablation was conducted on Wan2.2-T2V-5B trained on a subset of Ditto-1M, not the full 14B model trained on the complete data mixture. It is possible that the near-additivity observed at 5B scale does not perfectly generalize to 14B scale, where larger models may learn to internally factorize semantics and motion even without explicit SA/MA objectives (reducing the marginal gain) or may find synergistic combinations not captured by simple additivity (increasing the gain). Replicating the ablation at full scale would be computationally expensive but would provide more definitive evidence.

Experimental gaps and missing analyses

No quantitative zero-shot evaluation. As discussed above, this is the most significant gap.

Single base model family. All experiments use the Wan2.1/Wan2.2 family. The paper does not demonstrate that SA and MA generalize to other diffusion backbones (e.g., Stable Video Diffusion, HunyuanVideo, Movie Gen). While the principles are architecture-agnostic, the empirical results are specific to Wan2.1's architecture and pretrained representations.

No difficulty-stratified analysis. A central insight from the paper's motivation is that the semantic-motion tension varies with editing difficulty — complex edits with fast motion stress the factorization more than simple edits. However, benchmarks report only aggregate scores. A difficulty-stratified evaluation (by motion complexity, by edit spatial extent, by number of objects) would reveal where SAMA's advantages are largest and where they diminish — information that would directly guide practitioners on when to prefer SAMA over alternatives.

No inference-time compute ablation. The paper does not report the number of denoising steps, generation time, or GPU memory requirements for any method. Efficiency claims (faster convergence during training, Figure 6b) are not complemented by inference efficiency analysis.

VLM evaluation reliability. All three benchmarks use VLM judges (GPT-4o, Gemini-2.5-Pro, Gemini-2.5-Flash-Thinking) for scoring. While VLM-as-judge is standard practice in the field, the reliability of these scores — particularly for fine-grained distinctions between top methods — is not assessed. The paper does not report inter-rater reliability, correlation with human judgments, or score variance across multiple VLM inference runs. With score differences as small as 0.1–0.3 separating top methods on VIE-Bench, the ranking of methods may be sensitive to VLM stochasticity.

Data filtering impact. The VLM-based data filtering pipeline (Appendix B) applies strict quality thresholds (Instruction Following ≥8 for videos, etc.). The paper does not ablate the filtering step to show whether it is essential for performance or merely a conservative quality control measure. This matters because the filtering may remove challenging examples from the training data, potentially inflating benchmark performance by training on an easier distribution than competitors.

No ablation of anchor frame count or semantic token count. SA uses N=1 anchor frame and M=64 local semantic tokens in all experiments. The impact of these hyperparameters on editing quality and computational cost is unexplored. A sweep over N (1, 2, 4, 8 frames) and M (16, 64, 256 tokens) would reveal the tradeoff between semantic precision and computational overhead, and would inform practical deployment decisions.

Stage 0 data mixture ablation. Stage 0 trains on a mixture of image editing and text-to-video data with specific MA pretext task ratios. It is unclear whether the image editing data is essential for SA's effectiveness, whether the specific pretext task ratios matter, or whether text-to-video data without MA would suffice. An ablation removing image editing data from Stage 0 would test whether SA's semantic understanding transfers across modalities (from image editing concepts to video editing tasks) or requires in-domain video supervision.

6. Limitations and Trade-offs

The Cost of Difficulty Estimation Is Not Accounted For

This is the most significant practical gap in the paper.

The assumption or constraint. The entire VLM-based data filtering pipeline (Appendix B) that produces high-quality training data — and by extension the quality of the factorized pre-training — depends on running Qwen2.5-VL-72B as a judge across millions of video samples, scoring each on four criteria with three inference turns per sample and averaging scores. For video data, the filtering applies thresholds of Instruction Following ≥8, Visual Quality ≥9, Content Preservation ≥8, and Motion Consistency ≥8. The paper states this filtering is applied to "all datasets" but does not report the filtering cost, the fraction of data retained, or how filtering affects downstream editing performance.

The consequence. A practitioner attempting to replicate SAMA faces an expensive and poorly-characterized data preprocessing step. Running a 72B-parameter VLM across millions of videos with multiple inference turns is computationally enormous — potentially dwarfing the cost of the actual diffusion model training. Without knowing the filtering retention rate, it is impossible to estimate how much raw data must be collected to produce the training set sizes reported in Table 1. Furthermore, the filtering criteria may systematically bias the training distribution toward "easy" edits (high instruction following, high visual quality, high motion consistency removes challenging examples that stress the semantic-motion tension). The paper provides no ablation showing that the filtering is necessary — it may be a conservative quality control that marginally improves performance at enormous cost, or it may be essential. Without this ablation, a practitioner cannot determine whether to invest in the filtering infrastructure or to proceed with unfiltered data.

What evidence exists in the paper. None. Section 4.1 mentions that "All datasets are additionally subjected to a VLM-based coarse filtering stage to remove low-quality or instruction-inconsistent samples" but provides no retention statistics, no cost estimate, and no ablation comparing filtered versus unfiltered training. Appendix B describes the filtering prompts and thresholds in detail but provides no empirical analysis of the filtering's impact.

Mitigation status. Not addressed. The authors do not acknowledge this as a limitation, do not report cost or retention rates, and do not suggest future work on cheaper filtering alternatives. This is a notable transparency gap in an otherwise well-documented paper — the filtering is a critical preprocessing step that affects both training data quality and practical replicability, yet receives no empirical analysis.

Zero-Shot Editing Claims Are Qualitatively Demonstrated but Quantitatively Unsubstantiated

The zero-shot editing result is the paper's most striking and potentially impactful finding — but the empirical support is incomplete.

The assumption or constraint. The paper claims that Stage 0 factorized pre-training "alone already induces strong zero-shot video editing behavior" (Section 1) and that "robust instruction-guided video editing can naturally emerge once a model learns to jointly reason about semantic intent and temporal dynamics" (Section 1). Section 4.3 presents qualitative examples (Figure 5) showing Stage 0 producing edits across Replace, Add, Remove, Style, and Hybrid tasks.

The consequence. Without quantitative evaluation, the claim that zero-shot editing is "strong" is uncalibrated. Does Stage 0 editing approach the quality of existing open-source models trained on paired editing data (e.g., InsV2V, VACE)? If so, this would be genuinely transformative — it would imply that paired editing data is largely unnecessary and that the field should redirect effort toward factorization-based pre-training. Or does Stage 0 produce recognizable but substantially lower-quality edits than even weak supervised baselines? If so, the zero-shot result is an interesting scientific observation about emergent behavior but not a practical editing method, and the claim of "strong" editing is overstated.

The failure modes the paper acknowledges for Stage 0 — temporally inconsistent attributes, blurry added objects, residual ghosting in removals — are precisely the types of artifacts that degrade user-perceived quality in editing applications. A quantitative comparison against baselines would reveal whether these artifacts make Stage 0 editing practically unusable or merely slightly degraded relative to Stage 1.

What evidence exists in the paper. Only Figure 5 (qualitative examples). No benchmark scores, no VLM-judge evaluations, no human study, no comparison against any baseline are reported for Stage 0. This is a significant omission given that the zero-shot result is highlighted in the abstract, introduction, and conclusion as one of the paper's key contributions.

Mitigation status. Not addressed. The paper does not acknowledge this as a limitation, does not provide quantitative metrics, and does not suggest that future work should benchmark zero-shot factorization-based editing. Section 5 (Conclusion) mentions that "SAMA exhibits strong zero-shot editing behavior" without qualification. A minimal additional experiment — running Stage 0 through the same VIE-Bench evaluation pipeline used for the main results — would cost little and would substantially strengthen (or appropriately bound) the zero-shot claim.

Results Are Validated on a Single Model Family

The paper's findings are entirely tied to the Wan2.1/Wan2.2 architecture and pretrained representations.

The assumption or constraint. All experiments use the Wan2.1-T2V-14B backbone for main results and Wan2.2-T2V-5B for ablation studies. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (wait — this quote is from the reference example, not the SAMA paper; the SAMA paper does not make a generalizability claim about the backbone). The paper does not provide evidence that SA and MA would benefit other diffusion backbones (e.g., Stable Video Diffusion, HunyuanVideo, Movie Gen, CogVideoX, or image-to-video models).

The consequence. Several aspects of SAMA's performance may be backbone-specific. Wan2.1's flow matching formulation, latent space structure, temporal attention mechanism, and pretraining data distribution all interact with SA and MA in ways that may not transfer. For instance, SA's semantic tokens are projected into Wan2.1's specific VAE latent embedding space — a different backbone with different latent dimensionality or different VAE properties might require a different projector architecture or might not benefit from joint semantic-video denoising. MA's pretext tasks assume the backbone has sufficient temporal modeling capacity to learn motion restoration — a backbone with weaker temporal attention (e.g., one originally designed for images and retrofitted for video) might not benefit or might even degrade. The type embeddings versus shifted RoPE finding (Appendix A) is particularly likely to be backbone-specific, as different architectures implement positional encoding differently.

The paper's central claim — that factorization is the key missing ingredient for video editing, and that SA + MA provide it — cannot be separated from the specific backbone without replication on other architectures. If factorization helps on Wan2.1 but not on other backbones, the contribution is a strong training recipe for one model family rather than a general principle.

What evidence exists in the paper. All experiments use Wan2.1 or Wan2.2. The paper provides no evidence on other backbones.

Mitigation status. Not addressed. The paper does not discuss backbone generalizability as a limitation and does not suggest replication on other architectures as future work. This is perhaps understandable given computational constraints — training a 14B model on multiple backbones would be extremely expensive — but it means the paper's contribution should be understood as a methodology validated on Wan2.1 specifically, not as a general architectural principle demonstrated across model families.

The Add Category Reveals a Persistent Weakness Relative to VLM-Injection Methods

The Add editing category — arguably the most commercially important for creative workflows — is where SAMA most clearly underperforms the VLM-injection paradigm it critiques.

The assumption or constraint. SA provides semantic anchoring through sparse anchor frames (N=1) with M=64 local semantic tokens and one global token. This representation captures what should be edited (object identity, attributes, style) but provides limited spatial precision for where new content should be placed. VLM-injection methods like Kling-Omni can extract explicit spatial layouts (bounding boxes, segmentation masks) that specify pixel-level placement.

The consequence. On VIE-Bench Add, SAMA scores 8.711 overall versus Kling-Omni's 9.181 — a gap of 0.47 points, which is substantially larger than SAMA's leads on other categories (Swap/Change: +0.146; Remove: +0.011). The Instruction Following gap on Add is particularly large: SAMA 8.467 versus Kling-Omni 9.333 — a 0.866-point difference. This suggests that SA's semantic representation, while effective for replacing existing content (where spatial position is inherited from the source) and removing content (where spatial precision matters less), struggles with placing entirely new objects at precise spatial locations.

This is a fundamental limitation, not merely a quantitative shortfall. The factorization philosophy — sparse semantic anchors capture what to change, motion dynamics are learned from raw video — inherently provides weaker spatial grounding than methods that extract explicit spatial information (bounding boxes, depth maps, segmentation) from the source. For editing tasks where spatial precision is paramount (adding a small object at a specific location, placing text at exact coordinates, compositing elements into specific regions), SA's implicit spatial representation may be insufficient regardless of how well it is trained.

What evidence exists in the paper. Table 2 (VIE-Bench) shows the Add gap. The zero-shot failure modes (Section 4.3) mention that "newly added objects may appear slightly blurry," which is consistent with imprecise spatial placement — blurriness can arise when the model is uncertain about the exact spatial extent or location of the added object and averages across possible placements. The paper does not provide a dedicated analysis of why Add underperforms or what specific aspect of SA limits spatial precision.

Mitigation status. Partial. The paper acknowledges the Add gap implicitly through the quantitative results but does not discuss it as a limitation of the factorization approach. Section 5 (Conclusion) mentions that future work will focus on "stronger semantic tokenization to further reduce residual artifacts and temporal inconsistencies," which could potentially address spatial precision. However, stronger semantic tokenization (e.g., more anchor frames, finer-grained tokens, position-aware semantic features) increases computational cost and may partly converge toward the VLM-injection paradigm the paper critiques — using more detailed external semantic features to guide editing.

No Inference-Time Cost Analysis or Latency Characterization

The paper provides substantial analysis of training-time convergence benefits but entirely omits inference-time practical considerations.

The assumption or constraint. SAMA generates edited videos through iterative flow matching denoising of the concatenated source + semantic + target token sequence. At inference time, this requires: (1) encoding the source video through the VAE encoder, (2) extracting SigLIP features from N anchor frames, (3) projecting features through the SA MLP, (4) concatenating all token sequences, (5) running the full DiT denoising loop (typically 50+ steps for flow matching), and (6) decoding the target latents through the VAE decoder. None of these costs are characterized in the paper.

The consequence. A practitioner choosing between SAMA and alternatives cannot make an informed cost-quality tradeoff. Does SAMA's higher quality come at the cost of substantially longer inference time? The in-context V2V format requires feeding the full source video latent sequence through the DiT at every denoising step — for long videos or high resolutions, this could dominate inference cost. The SigLIP encoder and SA MLP add inference overhead that VLM-injection methods also have (and possibly more, since they must run a full VLM), but the paper provides no numbers.

For latency-sensitive applications (interactive editing, real-time preview), the sequential nature of flow matching denoising may be a bottleneck regardless of SAMA's architectural choices. For batch processing (rendering many edited videos offline), throughput becomes the relevant metric. Without any cost characterization — FLOPs per edit, GPU-seconds per video second, peak memory usage, or latency at different resolutions and frame counts — the paper's practical value is limited to quality comparisons rather than deployment guidance.

What evidence exists in the paper. None. The paper does not report the number of denoising steps used at inference, the inference time, GPU memory requirements, or any other computational cost metric. Training-time convergence benefits (Figure 6b) are mentioned as evidence of SA's efficiency, but inference cost is not discussed.

Mitigation status. Not addressed. The paper does not acknowledge this as a limitation and does not report any inference metrics. This is a significant gap for a paper that positions itself as a practical alternative to both open-source and commercial systems. Inference cost analysis is standard in video generation papers and its absence here is notable.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new architecture, a larger dataset, or a stronger VLM integration. Instead, it makes a conceptual intervention that reframes what the field should optimize. The dominant narrative in video editing research has been: more and better paired editing data + stronger external priors = better editing. SAMA inverts this: representation factorization is the bottleneck, and paired editing data is downstream of it. This is a paradigm shift in research prioritization, not an incremental refinement.

The magnitude of the shift is substantial but bounded. It is substantial because the paper demonstrates that a model trained on zero video editing pairs can edit — a finding that directly contradicts the implicit assumption motivating rapid dataset construction (Señorita-2M, InsViE-1M, Ditto-1M, OpenVE-3M, ReCo-Data). If factorization-first pre-training on image editing and text-to-video data already induces editing capability, then the community's resource allocation — heavily tilted toward collecting and curating video editing triplets — may be suboptimal. The paper's strongest empirical support for this shift is the zero-shot qualitative result (Figure 5) and the near-additive ablation gains from SA (+0.384) and MA (+0.399) on VIE-Bench (Table 5), which suggest these components improve orthogonal axes of editing quality without relying on paired supervision.

However, the shift is bounded because the paper does not provide quantitative zero-shot editing benchmarks. Without knowing whether Stage 0 editing scores at 4 or 8 on VIE-Bench relative to supervised baselines, the field cannot assess whether factorization reduces the need for paired data by 20% or 90%. The practical implication hangs on this number, and its absence means the conceptual shift is currently a hypothesis with strong qualitative support rather than a quantitatively established fact.

The paper resolves a latent contradiction in prior editing research. Methods that worked well on localized edits (object replacement, color change) often failed on global edits (style transfer, background replacement), and vice versa — but the reason was unclear. SAMA provides a diagnostic lens: localized edits stress semantic precision (knowing exactly what to change), while global edits stress motion preservation (maintaining dynamics under widespread appearance change). Prior methods that entangled these representations performed well on one axis and poorly on the other depending on which axis their training data emphasized. SAMA's factorization explains this pattern: SA handles localized semantic changes, MA handles global motion preservation, and their combination covers both. The near-additivity of SA and MA gains (Table 5: +0.384 + 0.399 ≈ +0.783 combined) is quantitative evidence for this diagnostic — the components are independently improvable because they address separable aspects of editing.

Research directions that become more attractive after this work:

Self-supervised pre-training for generative editing tasks. SAMA demonstrates that carefully designed pretext objectives can teach a diffusion model capabilities (motion reasoning, semantic grounding) that transfer to editing without paired supervision. This opens a broad design space: what other editing-relevant capabilities can be bootstrapped from unlabeled data? Spatial reasoning (object relationships, occlusion, depth) through geometric pretext tasks? Temporal causality (cause-before-effect, action-reaction) through temporal ordering puzzles? The principle — teach a capability through self-supervision, then apply it to editing through in-context conditioning — is general and may extend beyond video editing to image compositing, 3D scene manipulation, and audio-visual editing.
Representation disentanglement as an explicit training objective. Prior work on disentangled representations in generative models has largely focused on images (β-VAE, FactorVAE, StyleGAN) and has not been systematically applied to video editing, where the semantic-motion entanglement is both more acute (temporal artifacts are more visible) and more structured (motion has physical regularities). SAMA provides a concrete recipe — joint semantic-video denoising + motion restoration pretext — that achieves disentanglement without requiring paired editing data or explicit disentanglement losses. This opens the door to more sophisticated factorization objectives: disentangling subject identity from motion, camera motion from scene dynamics, or editing intent from execution details.

Research directions that become less attractive after this work:

Incremental improvements to VLM-injection pipelines. If representation factorization is the fundamental bottleneck, then making VLMs more accurate or adding more structural priors (depth, normals, segmentation) to the injection pipeline addresses the symptom (poor editing quality) rather than the cause (entangled representations in the diffusion backbone). The paper's competitive performance against Kling-Omni — a state-of-the-art VLM-injection system — on Swap/Change and Remove (Table 2) suggests that internal factorization can match or exceed external prior injection on tasks where semantic-motion reasoning is tightly coupled. The Add category gap (SAMA 8.711 vs. Kling-Omni 9.181) suggests VLM injection retains an advantage for spatially precise placement, but this gap may close with improved SA spatial precision rather than requiring a return to the injection paradigm.
Dataset-centric scaling as the primary research strategy. If Stage 0 pre-training on image editing (2.5M pairs) and text-to-video data (1.6M videos) yields editing capability without any video editing pairs, then the marginal value of each additional video editing triplet may be lower than previously assumed. Resources currently directed toward collecting and curating video editing datasets may be better spent on improving factorization pre-training objectives, expanding the diversity of motion pretext tasks, or scaling text-to-video data (which is far more abundant than paired editing data). The paper's data composition (Table 1) — ~2.5M image editing pairs + ~1.6M text-to-video pairs for Stage 0, ~1M video editing pairs for Stage 1 — provides a concrete starting point for this reallocation.

Follow-Up Research This Work Enables

Quantitative zero-shot editing benchmarks with difficulty stratification. The paper's most glaring omission is also its most actionable follow-up. Evaluate Stage 0 (factorized pre-training only, no video editing data) and Stage 1 (after SFT) on VIE-Bench, OpenVE-Bench, and ReCo-Bench, reporting the same metrics used in Tables 2-4. The key numbers: (a) What is the absolute zero-shot editing score, and how does it compare to the weakest supervised open-source baseline (e.g., InsV2V on VIE-Bench scores ~4.3 for Swap/Change, Table 2)? (b) What fraction of the Stage 0 → Stage 1 improvement comes from SFT on paired data versus simply more training on the same data distribution? (c) How does the zero-shot-to-supervised gap vary by editing category (Add vs. Remove vs. Style) and by difficulty (motion complexity, number of edited objects, spatial extent of edit)? A difficulty-stratified analysis would reveal whether factorization provides a uniform prior across all edits or is disproportionately helpful for specific editing types — information that directly guides whether factorization-first pre-training is worth adopting.

SA with explicit spatial grounding versus VLM-injection on Add tasks. The Add category gap (Table 2: SAMA 8.711 vs. Kling-Omni 9.181) identifies a clear limitation of SAMA's implicit spatial representation. A direct follow-up would compare three conditions on a controlled Add benchmark (e.g., ReCo-Bench Add, which has fine-grained spatial metrics like Scope Precision): (a) SAMA's standard SA (N=1 anchor frame, M=64 local semantic tokens), (b) SA with increased spatial resolution (N=4 anchor frames, M=256 tokens, or per-frame bounding box tokens derived from the SigLIP features), and (c) SAMA augmented with a lightweight spatial grounding head that predicts object bounding boxes from the semantic tokens and uses them as an additional conditioning signal (but still trained end-to-end, not injected from an external VLM). The comparison would reveal whether the Add gap is due to insufficient spatial information in the semantic tokens (which more tokens or explicit spatial supervision could fix) or due to a fundamental limitation of the factorization approach (which would require external spatial priors to resolve). The experiment is computationally practical — it requires training smaller model variants (Wan2.2-T2V-5B as in the ablation studies) and evaluating on a single benchmark category.

Cross-backbone replication of SA and MA benefits. SAMA's findings are currently validated only on Wan2.1/Wan2.2. A replication study on at least one other video diffusion backbone (Stable Video Diffusion 1.1, HunyuanVideo, or CogVideoX-5B) with identical SA and MA implementations would test generalizability. The key measurements: (a) Does SA accelerate convergence (reduced flow matching loss at fixed step, Figure 6b pattern) on other backbones? (b) Is the near-additivity of SA and MA gains (Table 5) replicated? (c) Does the zero-shot editing phenomenon emerge at comparable quality? A negative result — SA/MA help Wan2.1 but not other backbones — would bound the contribution to a training recipe rather than a general principle and would motivate investigation of what backbone properties (VAE latent structure, temporal attention design, pretraining data) modulate factorization benefits. A positive result on even one additional backbone would substantially strengthen the paper's claim that factorization is architecture-agnostic.

Scaling laws for factorized pre-training data composition. SAMA's Stage 0 uses a specific mixture: ~2.5M image editing pairs + ~1.6M text-to-video pairs with a 1:2:3:4 ratio of no-pretext:cube-inpainting:speed-perturbation:tube-shuffle. How sensitive are zero-shot editing quality and SFT convergence speed to this composition? A scaling experiment that varies (a) the ratio of image editing to text-to-video data, (b) the pretext task sampling ratios, and (c) the absolute scale of Stage 0 data (e.g., 10%, 50%, 100% of the full dataset) while measuring zero-shot and fine-tuned VIE-Bench scores would provide practical guidance for practitioners. The paper's implicit claim — that image editing data teaches semantics and text-to-video data teaches motion — predicts that removing image editing data should degrade SA benefits (lower Instruction Following) while removing text-to-video data should degrade MA benefits (lower Preservation). Testing this prediction would simultaneously validate the factorization hypothesis and establish minimal data requirements. This experiment is computationally intensive (many training runs) but can be performed at smaller model scale (5B) to make it tractable.

Motion Alignment pretext tasks as a general motion prior for video generation. MA uses three specific pretext tasks (cube inpainting, speed perturbation, tube shuffle) to teach motion reasoning, and demonstrates that this reasoning transfers to motion preservation during editing. Does MA also improve motion quality in standard text-to-video generation (where there is no source motion to preserve)? An experiment comparing text-to-video generation quality (evaluated on VBench or EvalCrafter motion metrics) between the base Wan2.1 model and a version fine-tuned with MA pretext tasks (but no SA, no editing data) would test whether MA's motion representations are editing-specific or general-purpose. A positive result — MA improves text-to-video motion quality — would establish MA as a general video generation training technique rather than an editing-specific component, broadening its applicability. A negative result would suggest that MA teaches the model to preserve source motion rather than generate novel motion, which is a meaningful distinction for how the representations develop.

Adaptive semantic anchoring with difficulty-dependent anchor frame allocation. The paper uses N=1 anchor frame uniformly for all videos, motivated by the observation that semantic edits are "sparse and temporally stable." However, complex edits with multiple objects, fine-grained attributes, or extended temporal spans may benefit from more anchor frames. A dynamic anchoring scheme that estimates editing complexity (e.g., from the instruction length, number of edited objects, or initial denoising loss) and allocates more anchor frames to complex edits and fewer to simple ones would test whether the sparsity assumption can be relaxed adaptively. The experiment would compare uniform N=1, uniform N=4, and adaptive N ∈ {1,2,4,8} at equal total compute (since more anchor frames increase SigLIP encoding and SA token processing cost) on VIE-Bench, stratified by editing category and complexity. If adaptive allocation improves Add and complex Replace performance without degrading simple edits, it would provide a practical refinement of the SA mechanism that directly addresses the identified Add gap.

Practical Applications and Downstream Use Cases

On-device or edge-deployed video editing with smaller models. The paper's ablation studies use Wan2.2-T2V-5B — a 5B-parameter model — and demonstrate that SA and MA provide meaningful gains at this scale (Table 5: overall +0.783 over baseline). Combined with the finding that SA accelerates convergence (Figure 6b), this suggests that the factorization approach can extract stronger editing capability from smaller models than standard end-to-end training. A deployment scenario where a 5B SAMA model runs on a consumer GPU (e.g., RTX 4090) and provides editing quality comparable to a 14B baseline model would make video editing feasible for individual creators without cloud compute. The key adoption metric would be: can a 5B SAMA model match or exceed the VIE-Bench scores of the 14B baseline (6.312 overall in Table 5)? If so, SAMA enables a ~3× reduction in model size (and corresponding memory and latency savings) at equal editing quality.

Cost-efficient training for video editing startups and research labs. SAMA's two-stage training strategy separates expensive capabilities (semantic understanding, motion reasoning) — which are trained once on broad data and can be shared across all editing applications — from application-specific fine-tuning (which requires paired editing data). A startup building a video editing product could pre-train a single Stage 0 model on publicly available image editing and text-to-video data, then fine-tune multiple Stage 1 variants for different editing verticals (e-commerce product video editing, social media content creation, film post-production) using smaller domain-specific paired datasets. The amortized cost of Stage 0 pre-training across multiple downstream applications makes this economically attractive compared to training separate end-to-end models for each vertical. The key economic metric: if Stage 0 costs $C and each Stage 1 fine-tuning costs$ F, the crossover point where SAMA becomes cheaper than training k separate end-to-end models is when C + kF < k × (cost of end-to-end training). The paper's data composition (Table 1) and training configurations provide enough detail for a rough cost estimate.

Self-improving video editing through iterative factorization refinement. The paper's finding that Stage 0 zero-shot editing exhibits specific failure modes (temporally inconsistent attributes, blurry additions, residual ghosting) that Stage 1 SFT partially resolves suggests a bootstrapping pipeline: (1) train Stage 0 on broad data, (2) generate zero-shot edits on a large unlabeled video corpus, (3) apply the VLM filtering pipeline (Appendix B) to select high-quality zero-shot edits as pseudo-labels, (4) fine-tune on these self-generated pseudo-pairs, (5) repeat. This would reduce dependence on human-annotated or synthetically-generated paired editing data by using the model's own zero-shot capability as a data engine. The key risk — model collapse from training on self-generated outputs — is mitigated by the VLM filtering step, which acts as an external quality gate. The experiment would measure whether iterative refinement closes the gap between Stage 0 zero-shot and fully supervised Stage 1 performance, and whether the pseudo-labeled data provides editing concept coverage beyond the human-annotated datasets.