ArXiv: 2512.08765

🎯 Pitch

You don't need extra motion encoders to achieve commercial-grade motion control in video generation. Injecting point trajectories directly into a standard image-to-video model's condition features via latent-space feature propagation matches systems like Kling 1.5 Pro's Motion Brush—without changing the architecture. This simple trick eliminates the usual trade-off between control precision and training scalability.

1. Executive Summary

Wan-Move introduces a motion-controllable video generation framework that injects trajectory-guided motion signals directly into the condition features of an off-the-shelf image-to-video model (Wan-I2V-14B), eliminating the need for auxiliary motion encoders or architectural modifications. The core mechanism is latent trajectory guidance — representing object motions as point trajectories, projecting them into latent space, and replicating the first frame's latent features along each trajectory (rather than encoding tracks via separate ControlNet modules or pixel-space embeddings), which simultaneously conveys appearance and motion instructions to the diffusion backbone. Scaled fine-tuning on 2 million high-quality videos yields 5-second, 480p outputs whose motion controllability rivals commercial systems such as Kling 1.5 Pro's Motion Brush, as measured by human preference studies, while achieving the best motion accuracy (end-point error of 2.6 on MoveBench) and video quality among academic methods — a gap that widens on large-motion and out-of-distribution scenarios, establishing that precise, scalable motion control is possible without architectural overhead only when motion guidance is expressed through the same latent features the base model already consumes.

2. Context and Motivation

The Core Problem: Architectural Overhead Prevents Precise, Scalable Motion Control

The fundamental question this paper tackles is: how do we inject fine-grained motion guidance into a pretrained image-to-video (I2V) diffusion model without adding so much architectural complexity that the model becomes difficult to train at scale? This matters because motion control is what transforms video generation from a "generate whatever" capability into a tool that creators and practitioners can actually direct — specifying where objects should go, how they should move, and how the camera should behave across frames.

The gap the paper identifies is not that motion-controllable video generation doesn't exist — it does, and the paper surveys a substantial body of prior work (Section 2). Rather, the gap is that existing methods consistently sacrifice either control granularity or training scalability, and no current approach delivers both simultaneously. The paper's core claim is that this tradeoff is unnecessary: by representing motion guidance as edits to the latent features the base I2V model already consumes, one can achieve precise control without adding a single new module, enabling straightforward fine-tuning of large-scale backbones.

Why This Problem Matters

Motion-controllable I2V generation sits at the intersection of several practical and scientific demands. The paper does not itemize these explicitly as a list of motivations, but they are implicit throughout the introduction and related work:

Creative tools and production pipelines. The ability to specify where objects go in a generated video — drag a point from position A to position B, rotate an object in 3D, transfer motion from one clip to another — is precisely what video editors, animators, and content creators need to integrate generative models into their workflows. Without it, video generation is a lottery; with it, generation becomes directable. The paper's application gallery (Figure 1) showcases six qualitatively distinct control modalities (single-object, multi-object, camera + object, complex, primitive-level, 3D rotation), signaling that the goal is a general-purpose motion-control substrate, not a narrow task-specific solution.

Scalability as a first-class concern. The paper repeatedly emphasizes scalability — "scaled training," "easy fine-tuning," "fast convergence" — because the field has reached a point where the quality of generated video is strongly coupled to model scale and data volume. The current generation of foundation video models (Wan [19], CogVideoX [52], HunyuanVideo [6]) produces high-fidelity output, but only when trained on massive datasets with large transformer backbones. Any motion-control method that complicates the architecture — adding ControlNet branches, auxiliary encoders, or separate fusion modules — introduces an impedance mismatch: the control mechanism must now be retrained or carefully aligned whenever the base model is updated, and the additional parameters compete for GPU memory and optimization budget during large-scale training. A method that requires no architectural changes, by contrast, inherits the base model's scaling properties for free. The paper positions Wan-Move as exactly this: a method that "can plug straight into the I2V backbone" (Section 1).

Benchmarking gap. A less obvious but equally important motivation is the absence of rigorous, standardized evaluation for motion control. Section 4 is devoted entirely to MoveBench, and the paper explicitly calls out the deficiencies of existing benchmarks (Figure 5): DAVIS [21] provides only 50 videos with short clips (35–100 frames) and no motion annotations beyond segmentation masks; VIPSeg [22] offers more videos (343) but even shorter durations (24 frames); MagicBench [20] expands to 600 videos but relies on noisy automatic labels and categorizes only by object count. None of these benchmarks support evaluation of long-range dynamics (5 seconds, 81 frames), multi-object interactions, or diverse content categories with verified ground-truth motion tracks. MoveBench is thus positioned not as an afterthought but as a necessary piece of infrastructure that enables the paper's empirical claims — without it, comparing methods on motion accuracy would remain ad hoc and unreliable.

Where Prior Approaches Fall Short

The paper organizes prior work along two axes: (i) how motion guidance is represented and (ii) how it is integrated into the generative model (Section 1, paragraph 2). Existing methods fail on one or both axes.

Axis 1: Motion Guidance Representation — Sparse vs. Dense

Sparse representations (bounding boxes [7, 8], segmentation masks [1, 9, 10]) provide coarse control: they can specify where an object should be and roughly which direction it moves, but they cannot express fine-grained local deformations, rotation, or non-rigid motion. A bounding box tells the model "the bird moves from left to right," but not how the bird's wings flap, how its head bobs, or how individual feathers ripple. For applications like character animation, object manipulation, or expressive camera work, this is insufficient.

Dense representations solve the granularity problem but introduce different failure modes:

  • Optical flow [2, 11, 12, 3] specifies pixel-wise displacement between frames, offering very fine control. However, it requires an external flow estimation model at inference time (e.g., RAFT [83]), which introduces cumulative error across frames — flow predictions degrade with occlusion, large displacements, and out-of-distribution scenes, and these errors compound when the generated video is expected to reproduce them faithfully. Moreover, requiring a separate flow model during inference complicates the deployment pipeline and adds latency. The paper explicitly flags this as a scalability barrier (Section 1: "requires an additional model for flow estimation during inference, which adds cumulative errors across frames and hampers scalability").

  • Point trajectories [13, 14] avoid the cumulative error problem because they are sparse by design: a user or tracker specifies a set of discrete 2D tracks (e.g., "this pixel moves here, then here, then here"), and the model is free to interpolate the rest of the video. This is easier to specify at inference time (drag a point, no need for dense flow estimation). However, prior point-trajectory methods face a context problem: each track is a single-pixel thread. A pixel contains very little semantic or textural information — it's just a color value at one location. When you tell the model "this pixel moves along this path," it has no information about the neighborhood around that pixel, making it hard to align textures, maintain object boundaries, or propagate consistent motion patterns across adjacent regions. The paper describes this limitation precisely (Section 1): "each track is only a single-pixel thread and lacks surrounding spatial context. This makes it hard to align textures and motion patterns across neighboring regions."

Axis 2: Motion Signal Integration — The Encoder Bottleneck

Regardless of how motion is represented, every prior fine-tuning method introduces additional motion-processing modules to inject that signal into the base model. The paper catalogs several approaches (Section 2, paragraph on motion-controllable video generation):

  • ControlNet [18] branches: A separate copy of (part of) the base model processes the motion condition, and its outputs are injected into the main model via zero-initialized convolution layers. This is the approach used by Motion Prompting [13], LeviTor [16], MagicMotion [20], and others. The problem: ControlNet adds a substantial number of parameters — roughly a copy of the encoder portion of the base model — which must be trained from scratch while the base model is fine-tuned. At the scale of a 14B-parameter DiT backbone (as used in Wan-I2V-14B), this means adding billions of trainable parameters, increasing memory pressure during training, and requiring careful balancing of ControlNet output scales to avoid overwriting the base model's learned representations.

  • Auxiliary motion encoders with cross-attention or adaLN injection: Tora [14] uses a lightweight adaptive layer norm (adaLN) module to inject trajectory embeddings. DragAnything [1] trains an "ObjectNet" to bridge the domain gap between DINOv2 features and the UNet's feature space — but DINOv2 operates at a 14×14 patch size, which limits control granularity, and the model still requires additional identity embeddings to distinguish objects. ImageConductor [58] injects pixel-level trajectory features directly, but as the paper notes, this fails because single-pixel features lack semantic context (Table 1: ImageConductor achieves the highest EPE of 15.6 on MoveBench, indicating very poor motion control).

The common thread across all these approaches is signal degradation during encoding. Motion information — whether represented as pixel tracks, flow fields, or mask sequences — must be transformed through learned parameters before it reaches the diffusion backbone. Any error, information loss, or misalignment in this transformation directly limits control quality. The paper's key insight is that this encoding step may be entirely unnecessary if motion guidance can be expressed directly in the representation space that the base model already operates in.

The Most Direct Predecessor: Motion Prompting [13]

Motion Prompting [13] is the work most similar to Wan-Move, as both use point trajectories for motion representation. The paper explicitly distinguishes itself along two dimensions (Section 2):

  1. How tracks are embedded: Motion Prompting encodes point tracks via random embeddings in pixel space — each trajectory position is assigned a learned or random feature vector, which is then processed by a ControlNet. This inherits the single-pixel context problem: the embeddings carry no information about the appearance or texture of the region being controlled. Wan-Move instead propagates the actual latent features of the first frame along the trajectory, so each tracked position receives the full, rich feature representation (texture, color, semantic information) from the corresponding starting point.

  2. How tracks are integrated: Motion Prompting uses a ControlNet branch, which adds architectural overhead and requires training an entirely separate module. Wan-Move achieves motion injection by simply editing the condition feature (zimage) that the base model already uses — replacing zero-padded features at future-frame positions with features copied from the first frame along the trajectory path. This requires zero additional parameters.

The paper provides both quantitative and qualitative evidence for the superiority of latent feature replication over pixel-space track embeddings in Section 5.3 (Table 4, Figure 8), which we will examine in detail later.

How This Paper Positions Itself

The paper's positioning is built around a single architectural decision — "no extra modules" — and the consequences that flow from it. This is not presented as a minor implementation trick but as a design philosophy with cascading benefits:

Simplicity enables scalability. By avoiding motion encoders, the training objective reduces to fine-tuning the existing DiT backbone with a modified condition feature. The model inherits the base architecture's parallelism strategies (Ulysses sequence parallelism, FSDP), optimizer settings, and convergence properties. The paper reports training on 2 million videos using 64 A100 GPUs for 30,000 steps (Appendix 8.3) — a substantial but not extraordinary compute budget for a 14B-parameter model. If Wan-Move required a ControlNet branch, the parameter count and memory footprint would be significantly larger, potentially requiring more GPUs or a smaller batch size, which would slow convergence.

Feature propagation preserves context. The core operation — copying first-frame latent features along a trajectory in latent space — is not just parameter-free; it is deterministic and information-preserving. Unlike a learned encoder that could lose information during transformation, feature replication guarantees that the motion guidance carries exactly the same appearance information present at the trajectory's starting point. This is why the paper can claim "precise motion control" (title, abstract): the guidance signal is as rich as the first frame's VAE encoding, which captures texture, edges, color, and semantic structure at the VAE's native resolution.

The VAE's translation equivariance assumption. A critical — and unstated — assumption underlies this entire approach: that the VAE's latent features are approximately translation-equivariant, meaning a feature at spatial location (x, y) in the latent grid represents roughly the same visual content as a feature at (x+Δx, y+Δy) in a different frame. This is what justifies Eq. (3): copying zimage[0, p0] to zimage[n, pn] assumes that the feature vector from the first frame is the "correct" feature to place at the trajectory's position in frame n. If the VAE latents were not translation-equivariant, this copy operation would inject semantically mismatched features, leading to artifacts. The paper does not explicitly justify or test this assumption, but the strong empirical results suggest it holds sufficiently well for the Wan VAE in practice.

A bridge between research and commercial quality. An unusual aspect of this paper's positioning is its explicit comparison to Kling 1.5 Pro's Motion Brush [4], a commercial product from Kuaishou. The human study in Table 3 reports that Wan-Move achieves a 47.8% win rate on motion accuracy and 53.4% win rate on motion quality against Kling 1.5 Pro — essentially at parity. For visual quality, the win rate is 50.2% (statistically indistinguishable from a tie). This is significant framing: the paper is claiming not just SOTA among academic methods (which Table 1 already demonstrates) but competitiveness with a deployed commercial system that presumably benefits from proprietary data, larger-scale training, and engineering resources beyond what an academic lab typically has. The implication is that the "no extra modules" design philosophy is not just elegant — it is what makes this level of quality achievable with open-source resources.

MoveBench as a community contribution. The paper also positions MoveBench as infrastructure for the field, not just an evaluation tool for Wan-Move. The careful curation pipeline (Figure 3), the inclusion of both point trajectories and segmentation masks (so that mask-based methods can also be evaluated), the 54 content categories covering diverse motion types (Figure 4), and the human+SAM hybrid annotation process (Figure 3b) suggest an intent to set a new standard for motion-control evaluation. The paper's claim that MoveBench offers "larger data volume, longer video durations, and high-quality motion annotations" (Abstract) compared to existing benchmarks is supported by the statistics in Figure 5: 1018 videos at 81 frames each, versus 600 videos at 49 frames for the next-largest benchmark (MagicBench).

The Overarching Narrative

Reading the paper as a whole, the motivation follows a clear logical arc:

  1. Motion control is essential for practical video generation, but current methods either sacrifice precision (sparse signals) or scalability (dense signals requiring auxiliary encoders).
  2. Point trajectories offer the best balance of expressiveness and ease-of-specification, but prior work mismanages them — either by encoding them as context-free pixel embeddings (Motion Prompting) or by wrapping them in heavy ControlNet architectures.
  3. The key realization is that an I2V model already knows how to animate a first frame; motion control just needs to tell it where things should go. The most natural way to communicate this is in the model's own latent language — by editing the condition features to show where first-frame content should appear in later frames.
  4. Because this requires zero architectural changes, it scales effortlessly: fine-tune the DiT backbone on a large, high-quality video dataset with trajectory-annotated condition features, and the model learns to interpret the replicated features as motion instructions.
  5. The result is a system that matches commercial quality while being fully open-source, and the accompanying benchmark (MoveBench) provides the rigorous evaluation infrastructure the field needs to make further progress.

This narrative is compelling because it frames Wan-Move not as "we built a better motion encoder" but as "we realized you don't need a motion encoder at all" — a classic reframing that sidesteps an entire category of technical challenges rather than incrementally improving within them.

3. Technical Approach

3.1 Reader Orientation

Wan-Move is a fine-tuning framework that teaches an existing image-to-video diffusion model to accept point-trajectory motion instructions without adding any new neural network components — it achieves control purely by editing the latent features that the base model already receives as input. The problem it solves is the architectural overhead problem in motion-controllable video generation: prior methods require auxiliary motion encoders or ControlNet branches that complicate training, degrade motion signals during encoding, and resist scaling to large foundation models. The shape of the solution is a single, deterministic pre-processing step applied to the condition features before they enter the DiT backbone — copy first-frame latent features along latent-space trajectories into future-frame positions — which transforms the standard "animate this image" instruction into "animate this image such that these specific pixels end up at these specific future locations."

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components, arranged in a pipeline from inputs to generated video:

  1. Motion Specification Interface — the mechanism by which a user (or an automated system like CoTracker) provides point trajectories in pixel space. Each trajectory is a sequence of $(x, y)$ coordinates across frames indicating where a tracked point should appear.

  2. Spatial Coordinate Mapping — a deterministic mathematical function (Eq. 2) that converts pixel-space trajectory coordinates into latent-space coordinates, accounting for the VAE's spatial and temporal compression ratios. This produces $\tilde{p}[n]$ — the latent-grid position of each trajectory point at each temporal index.

  3. Latent Feature Replication Engine — the core operation (Eq. 3, Figure 2a) that reads the VAE-encoded first-frame latent feature at the starting trajectory position $\tilde{p}[0]$ and writes it into the zero-padded condition feature at every subsequent trajectory position $\tilde{p}[n]$ for $n = 1, \ldots, T/f_t$. This transforms the static first-frame condition into a spatiotemporally-aligned guide that tells the model where content should move.

  4. Condition Assembly — the process that concatenates the motion-edited latent condition $z_{\text{image}}$ with the noisy latent $x_t$, along with CLIP image embeddings and umT5 text embeddings, to form the full conditioning input to the DiT backbone. This step is architecturally identical to the base I2V model — the only difference is the content of $z_{\text{image}}$.

  5. DiT Denoising Backbone (Wan-I2V-14B) — the pretrained image-to-video diffusion transformer, fine-tuned with the flow matching objective (Eq. 4) to interpret the replicated latent features as motion instructions. The backbone remains architecturally unmodified; only its weights are updated during training.

Information flows as follows: a user provides a first-frame image, a text prompt, and point trajectories (dense or sparse) → trajectories are mapped to latent coordinates via Eq. 2 → first-frame latent features are replicated along these coordinates via Eq. 3 → the edited $z_{\text{image}}$ is concatenated channel-wise with $x_t$ → CLIP and umT5 encode the image and text → the DiT backbone processes the full conditioning input and predicts a vector field → the ODE solver integrates this vector field to produce the output latent → the VAE decoder reconstructs the 5-second, 480p video.

3.3 Roadmap for the Deep Dive

  • First, the base I2V generation framework (Section 3.1, Preliminaries) — how Wan-I2V-14B conditions on a first frame, what $z_{\text{image}}$ contains, and why the zero-padded frame slots are the natural injection point for motion guidance — because this establishes the substrate into which Wan-Move inserts its mechanism.

  • Second, latent trajectory guidance (Section 3.2) — the spatial coordinate mapping (Eq. 2) and the feature replication operation (Eq. 3) that together transform pixel-space point tracks into latent-space motion instructions, including the treatment of multiple overlapping trajectories — because this is the paper's single novel technical contribution and everything else (training, inference, scaling) follows from it.

  • Third, training data curation and the training procedure (Section 3.3) — the two-stage filtering pipeline (visual quality scoring + motion stability via SigLIP), the trajectory sampling strategy (5% dropout, 1–200 uniformly sampled tracks), and the flow matching objective — because the data and optimization choices determine whether the model actually learns to interpret replicated features as motion instructions.

  • Fourth, the inference procedure (Section 3.3) — how classifier-free guidance is applied, the three input modalities (image, text, trajectories), and the sampling schedule — because this is what a practitioner needs to actually use the trained model.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a method paper whose core idea is that motion control for I2V generation can be achieved by deterministically editing the condition latent $z_{\text{image}}$ — specifically, by copying first-frame latent features into future-frame positions along latent-space trajectories — rather than by training additional neural network modules to encode and inject motion signals.

Base I2V Framework: How Wan-I2V-14B Conditions on a First Frame

The paper builds on Wan-I2V-14B [19], a state-of-the-art image-to-video diffusion model. Understanding the base conditioning mechanism is essential because Wan-Move's entire contribution is a modification to it.

Video diffusion in latent space. Given an input video $V \in \mathbb{R}^{(1+T) \times H \times W \times 3}$ — a first frame plus $T$ subsequent frames, each of spatial dimensions $H \times W$ with 3 color channels — a pretrained VAE encoder $E$ compresses it temporally by a factor $f_t$ and spatially by a factor $f_s$, while expanding the channel dimension to $C$, yielding:

x=E(V)∈R(1+T/ft)×(H/fs)×(W/fs)×Cx = E(V) \in \mathbb{R}^{(1 + T/f_t) \times (H/f_s) \times (W/f_s) \times C}

where $x$ is the latent representation, $T/f_t$ is the number of latent frames (temporal tokens), $H/f_s$ and $W/f_s$ are the latent spatial dimensions, and $C$ is the VAE's latent channel count. The VAE decoder $D$ reconstructs the video as $\hat{V} = D(x)$.

What it computes: a compressed representation of the video in a lower-dimensional latent space. The temporal compression ($f_t$) reduces the number of frames that the diffusion model must process, while the spatial compression ($f_s$) reduces per-frame resolution. For Wan-I2V-14B generating 5-second videos at 480p resolution (832×480 pixels) with 81 frames, the specific compression ratios determine the latent tensor's shape, though the paper does not state $f_t$ and $f_s$ explicitly — they are properties of the Wan VAE.

Why this form: latent diffusion is the standard paradigm because operating in pixel space (e.g., 81 × 832 × 480 × 3 ≈ 97 million values per video) would make the attention computation in the DiT backbone prohibitively expensive. The VAE's compression (typically $f_s = 8$ and $f_t = 4$ for video VAEs, though the paper does not confirm these values for Wan specifically) reduces the token count by roughly two orders of magnitude, making transformer-based video generation feasible.

The first-frame condition feature $z_{\text{image}}$. For image-to-video generation, the model receives the first frame as a conditioning signal. The base Wan-I2V-14B encodes this by taking the first frame $I \in \mathbb{R}^{1 \times H \times W \times 3}$, concatenating it with $T$ frames of zeros $0^{T \times H \times W \times 3}$, and encoding the result with the VAE:

zimage=E(concat[I,0T×H×W×3])∈R(1+T/ft)×(H/fs)×(W/fs)×Cz_{\text{image}} = E(\text{concat}[I, 0^{T \times H \times W \times 3}]) \in \mathbb{R}^{(1 + T/f_t) \times (H/f_s) \times (W/f_s) \times C}

What it computes: a latent tensor with the same spatiotemporal dimensions as a full video latent $x$, but where only the first latent frame contains real VAE-encoded features from the input image — all subsequent latent frames ($1$ through $T/f_t$) are zero-filled, representing "unknown future content that the model must generate."

Why this form: this zero-padding convention creates a natural "canvas" for the diffusion model. The first latent frame anchors the generation to the real image content, while the zeroed slots tell the model "these positions need to be filled in, and they should be consistent with the first frame." The channel-wise concatenation of $z_{\text{image}}$ with the noisy latent $x_t$ means the model sees both "what is the truth at frame 0" and "what is the current noisy estimate of all frames" simultaneously, enabling it to denoise the future frames conditioned on the first frame. This is precisely why Wan-Move can inject motion guidance by simply replacing some of those zeros with copied first-frame features — the model already expects the zero-slots to be filled with information derived from the first frame; Wan-Move just provides stronger, spatially-localized hints about where that information should go.

The DiT backbone and conditioning. The pretrained umT5 [71] text encoder processes the user's text prompt into text embeddings $z_{\text{text}}$, and the CLIP [70] image encoder processes the first frame into a global image embedding $z_{\text{global}}$. These are injected into the DiT backbone via decoupled cross-attention [18]. The concatenated $[x_t; z_{\text{image}}]$ enters the DiT along the channel dimension, giving the model both the noisy latents to denoise and the first-frame condition to anchor to.

Latent Trajectory Guidance: The Core Mechanism

This is the paper's central technical contribution. It consists of two sequential, deterministic operations: (1) mapping pixel-space trajectories to latent coordinates, and (2) replicating first-frame latent features along those coordinates.

Motion representation: point trajectories. A point trajectory of length $1 + T$ is a sequence of 2D pixel coordinates:

p∈R(1+T)×2p \in \mathbb{R}^{(1+T) \times 2}

where $p[n] = (x_n, y_n)$ specifies the $(x, y)$ position of the tracked point in the $n$-th frame (0-indexed, with $n = 0$ being the first frame). The paper uses CoTracker [73] to extract these trajectories from training videos, placing a dense 32×32 grid of tracking points (1024 total) and retaining trajectories where the point is predicted as "visible." During inference, users can specify any number of trajectories, from a single point (drag-based control) to dense grids (motion transfer).

Why point trajectories: they offer the best tradeoff among motion representations. Unlike bounding boxes (which cannot express local deformation) or optical flow (which requires an external model during inference and accumulates errors), point trajectories are sparse, easy to specify (drag a point in an interface, use an off-the-shelf tracker, or define them programmatically for camera/rotation control), and naturally capture both global displacement and fine-grained local motion. The sparsity is a virtue: the model is free to interpolate motion in unconstrained regions, preventing over-constraint that could cause artifacts.

Step 1: Spatial coordinate mapping (Eq. 2). The trajectories exist in pixel space, but the latent features exist in VAE-compressed latent space, where spatial dimensions are downsampled by $f_s$ and temporal frames are grouped by $f_t$. The mapping is:

undefined