CoInteract: Physically-Consistent Human-Object Interaction Video Synthesis via Spatially-Structured Co-Generation

ArXiv: 2604.19636

🎯 Pitch

Diffusion models keep making hands melt through handled objects because they have no concept of 3D interaction geometry. CoInteract fixes this by training an invisible auxiliary stream to render correct contact layouts before the RGB generation stream even starts, then removes it at inference time—so you get physically plausible hand–object contact with zero extra compute cost.

1. Executive Summary

This paper introduces CoInteract, an end-to-end framework for speech-driven human–object interaction (HOI) video synthesis that embeds structural priors and physical constraints directly into a Diffusion Transformer (DiT) backbone to address recurring failures in hand/face fidelity and physically plausible contact. The system contributes two complementary mechanisms: a Human-Aware Mixture-of-Experts (MoE) that routes tokens to region-specialized experts via spatially supervised routing (e.g., dispatching face and hand tokens to dedicated lightweight FFNs to reduce structural collapse), and a Spatially-Structured Co-Generation paradigm that jointly trains an RGB appearance stream alongside an auxiliary HOI structure stream with an asymmetric co-attention mask, forcing the backbone to learn interaction geometry rather than relying on appearance shortcuts (e.g., the HOI stream renders the human as a silhouette while preserving the object's RGB appearance, regularizing shared weights during training while being removed at inference for zero overhead). On a curated 12K-clip HOI dataset with a held-out test set of 50 clips spanning diverse product categories, CoInteract achieves the highest HOI plausibility score (0.72 VLM-QA via Gemini-3-Pro evaluation, substantially ahead of the next-best baseline InteractAvatar at 0.62) and highest hand structural quality (0.724 HQ, versus 0.696 for InteractAvatar), while ablation reveals that removing co-generation causes a 33.3% drop in VLM-QA (0.72 → 0.48), establishing that explicit interaction-structure supervision is essential for physical plausibility and that the asymmetric design preserves these gains with only a 1.04× inference cost overhead relative to the MoE-free baseline, whereas retaining the HOI branch at inference inflates cost to 4.13×.

2. Context and Motivation

The Core Problem: RGB-Only Video Diffusion Models Produce Physically Implausible Human-Object Interactions

The fundamental gap this paper addresses is deceptively simple to state but profoundly difficult to solve: current state-of-the-art video diffusion models, despite producing photorealistic results, consistently fail when synthesizing videos of humans actively interacting with objects. The failure manifests in two specific, recurring ways that the paper identifies throughout Section 1 and Section 2:

(1) Structural collapse in sensitive regions. Hands and faces—the most articulated and detail-dense parts of the human body—frequently degrade into artifacts such as merged fingers, blurred facial features, or anatomically impossible configurations. This is not merely a cosmetic flaw; in the context of product demonstration (the paper's motivating application), viewers fixate on hands precisely when they are manipulating the product being advertised, making hand quality a functional requirement rather than an optional aesthetic concern.

(2) Physical violations of contact geometry. The most damaging failure mode for HOI synthesis is interpenetration—hands that pass through product surfaces, objects that clip through body parts, or grasping poses where the geometry of the hand bears no relationship to the geometry of the object being held. These violations are immediately obvious to human viewers and break the illusion of physical realism, rendering generated videos unusable for commercial applications like e-commerce livestreaming or virtual product demonstrations.

The paper identifies the root cause of both failure modes as a fundamental architectural limitation:

"We argue that this limitation is rooted in the RGB-centric nature of current diffusion backbones. As also observed in recent studies [49, 53], models trained purely on pixel-level supervision have no built-in notion of 3D hand-object spatial relationships or body structure, and must rely on appearance cues alone to infer interaction."

This is a critical insight: RGB-only training, no matter how extensive the dataset or how large the model, cannot reliably recover 3D contact geometry from 2D pixel statistics alone. The model learns correlations between pixel patterns, but without explicit structural supervision, it has no mechanism to learn that hands must stop at object surfaces, that fingers should bend around object contours, or that facial features should maintain consistent topology across frames. The supervision signal from flow matching in RGB space is simply too weak to enforce these constraints—the loss can be low even when hands penetrate objects, as long as the penetration regions are textured plausibly.

Why This Problem Matters: Shifting from Passive Avatars to Active Product Demonstration

The importance of this problem is best understood through the evolution of digital human technology, which the paper situates in its opening paragraphs. Speech-driven avatar generation for talking heads—where a digital avatar speaks while maintaining lip sync and identity consistency—has achieved "unprecedented photorealism" through methods like SadTalker [59], CanonSwap [31], and GaussianTalker [57]. These systems work well because the task is constrained: the avatar's body is largely static, hand gestures are simple and loosely coupled to speech, and there is no object to interact with.

However, the paper argues that the commercial frontier has shifted:

"as demand shifts from passive speaking to active product demonstration, Human-Object Interaction (HOI) video synthesis has emerged as the next critical frontier, requiring coordinated hand movements, precise object manipulation, and strict physical plausibility beyond what existing talking-avatar generation methods provide."

This shift carries massive practical implications. In e-commerce livestreaming (a dominant sales channel in China, where multiple authors are based), a virtual anchor who can naturally pick up, rotate, wear, or demonstrate a product—while speaking about its features—would revolutionize how products are marketed. The paper explicitly grounds its motivation in "e-commerce, digital advertising, and virtual marketing" (Abstract), where the difference between a talking head and an interacting avatar is the difference between a passive billboard and an active salesperson. The same applies to virtual assistance and remote education, where demonstrating physical manipulation (how to assemble a device, how to wear protective equipment, how to operate a tool) requires interaction capability.

The technical challenge is qualitatively harder because it introduces contact constraints that were absent in talking-head generation. When an avatar speaks, the only hard constraint is that the lips move in sync with audio—there is no external object whose surface must be respected. In HOI synthesis, every frame must satisfy the geometric condition that the human mesh and the object mesh do not interpenetrate, that contact points are physically reasonable (a hand should wrap around a handle, not phase through it), and that occlusions are correctly ordered (a hand in front of an object should occlude it; a hand behind should be occluded). These constraints cannot be satisfied by averaging over RGB statistics—they require geometric reasoning that current diffusion backbones lack.

Prior Approaches and Where They Fall Short

The paper categorizes existing HOI video synthesis methods into two paradigms, each with characteristic weaknesses (Section 2.3, Fig. 1).

Paradigm 1: Multi-Condition Generation

These methods—exemplified by AnchorCrafter [51] and ByteLoom [26]—augment standard video diffusion with explicit structural conditions extracted during preprocessing. AnchorCrafter conditions on per-frame human poses and multi-view object features; ByteLoom introduces geometric priors such as relative coordinate maps to improve spatial alignment between human and object. The appeal of this paradigm is that it provides the model with structural information that RGB-only training cannot infer—pose keypoints explicitly encode hand positions, and object features provide bounding geometry.

Where they fall short. The paper identifies three limitations:

Heavy inference-time preprocessing: These methods require extracting pose, object masks, or other structural signals for every frame of every video at inference time. In a production deployment where thousands of product demonstration videos might be generated daily, this preprocessing pipeline is a computational and engineering burden. The paper explicitly contrasts this with CoInteract's "unified, end-to-end generation framework that is both easy to use and inherently aware of HOI structures" (Fig. 1 caption).
Limited robustness and generalization: The structural signals (e.g., pose from a pretrained detector, object masks from a segmentation model) are themselves imperfect, especially under occlusions (the very interaction scenarios where accurate pose and masks matter most). Errors in these preprocessing steps propagate into generation quality. Moreover, because the model never internalizes the interaction rules—it relies on external signals at both training and inference time—its generalization to unseen objects or interaction types is bounded by the quality and coverage of the preprocessing pipeline.
Domain-specific signal fragility: AnchorCrafter's qualitative results (Fig. 5) reveal a revealing pattern: it performs noticeably better on objects within its training set (the last two test cases) but suffers from "identity drift and unnatural interaction boundaries" on unseen objects (the first two cases). This suggests that multi-condition methods learn to correlate preprocessing signals with training-set appearances rather than learning generalizable interaction principles—a form of shortcut learning where the model exploits the preprocessing pipeline as a crutch rather than learning geometry.

Paradigm 2: Multi-Reference Generation

This paradigm—represented by Humo [5], Phantom [28], VACE [17], and SkyReels-V3 [21]—conditions the diffusion model on person and product reference images without explicit geometric conditions. The model receives the reference images and must infer both how the person should move and how they should interact with the object purely from appearance-level conditioning. Some variants use a two-stage pipeline: first composite the person and product references using an image editing model (e.g., Qwen-Image [48]), then perform speech-driven video generation from the composited frame (e.g., InteractAvatar [60] with WanS2V [10]).

Where they fall short. The paper's diagnosis is precise:

"While more flexible, these approaches typically lack explicit mechanisms to enforce interaction structure—e.g. interaction geometry and physically plausible hand poses—often leading to implausible human–object interaction."

The flexibility advantage is real: multi-reference methods do not require per-frame pose annotations or object segmentation at inference time, making them more practical for deployment. However, this flexibility comes at the cost of physical plausibility. The model must learn to infer interaction constraints from appearance-level conditioning alone, essentially trying to learn 3D geometry from 2D supervision without any geometric inductive bias. The result is the interpenetration and hand artifacts that motivate the paper.

The qualitative results (Fig. 5) dramatically illustrate these failures. Humo and Phantom, despite scoring slightly higher on aesthetic quality metrics (AES), exhibit three common failure modes:

Hand-object interpenetration (hands passing through objects)
Inconsistent product appearance (the object morphs or changes texture during the video)
Background deviation from the reference (the model hallucinates new backgrounds rather than preserving the reference scene, which inflates AES but violates the reference conditioning)

InteractAvatar's case is particularly instructive because it represents the best-case multi-reference baseline: it uses Qwen-Image compositing to obtain a strong initial frame with correct object placement, giving it a head start. Yet even with this advantage, "as generation progresses, it still produces HOI plausibility issues such as unnatural grasping poses" (Section 4.3). This demonstrates that compositing a single frame is insufficient—maintaining physically plausible interaction across an entire sequence requires ongoing geometric constraints that appearance-level conditioning cannot provide.

A Deeper Architectural Issue: The Signal Weakness of RGB Supervision

The paper's diagnosis goes beyond simply noting that prior methods produce artifacts. It identifies a fundamental signal-to-noise problem in how diffusion models learn from RGB data:

"models trained purely on pixel-level supervision have no built-in notion of 3D hand-object spatial relationships or body structure, and must rely on appearance cues alone to infer interaction."

This is worth unpacking because it explains why simply scaling data or model size (the default approach for most video generation problems) will not solve HOI synthesis. The flow matching objective (Equation 2) penalizes the model for predicting the wrong velocity field at each noise level, but the penalty is applied uniformly across all pixels. A pixel where a hand incorrectly penetrates an object contributes the same weight to the loss as any other pixel, and the model can "satisfy" the loss by learning to blur the penetration boundary, generating plausible texture in the penetration region, or any number of shortcuts that reduce the squared error without learning that hands cannot pass through solid objects. The loss does not explicitly encode the geometric constraint that the human and object spatial occupancy volumes must be disjoint.

This is not a failure of training scale—it is a failure of supervision granularity. The model needs a loss signal that specifically penalizes geometric violations, not just pixel-level reconstruction error. This insight motivates the paper's central technical contribution: injecting structure-level supervision that forces the model to learn interaction geometry, not just appearance.

How CoInteract Positions Itself

CoInteract's position in the landscape can be understood along three axes.

Axis 1: Internalize structure, don't externalize it. Multi-condition methods (Paradigm 1) provide structural information as external inputs at both training and inference time. CoInteract instead embeds structural priors into the backbone architecture itself, through two mechanisms: (1) the auxiliary HOI structure stream that co-trains with the RGB stream, forcing shared parameters to learn interaction geometry during training, and (2) the Human-Aware MoE that permanently dedicates specialized capacity to structurally sensitive regions. The key consequence is that at inference time, CoInteract requires no structural preprocessing—the model has internalized the interaction constraints during training. This is the architectural premise behind the paper's claim of being "easy to use" (Fig. 1).

Axis 2: Zero-overhead inference. Multi-stream co-generation has been explored before in general video synthesis—VideoJAM [4] jointly predicts RGB and motion representations, UnityVideo [14] does multi-modal multi-task learning—but these methods typically require the auxiliary stream at inference, doubling or tripling computational cost. CoInteract's asymmetric co-attention design (the two-stage training with the mask in Equation 4) is specifically engineered so that the HOI stream can be discarded at inference while its regularization effect on the shared backbone persists. This is a non-trivial architectural choice that the paper explicitly validates through ablation: retaining the HOI branch at inference (w/o Asym. Mask variant) improves VLM-QA slightly (0.72 → 0.76) but inflates inference cost to 4.13×. The asymmetric training strategy trades this marginal improvement for a dramatic efficiency gain, making the method practically deployable.

Axis 3: Region-specialized capacity within the backbone. Prior methods for improving hand/face quality—CyberHost's region codebook attention [25], Make-Your-Anchor's post-hoc face enhancement [16]—treat these regions as an afterthought, applying external refinements after generation or bolting on auxiliary modules. CoInteract's Human-Aware MoE integrates region specialization into the DiT backbone itself, using a spatially supervised router trained with bounding box annotations (Equation 6). This means that hand and face quality improvements are not a post-processing patch but a first-class part of the generation process. The router learns from data which tokens belong to which region, guided by explicit spatial supervision during training, but operates purely from the learned DiT hidden states at inference—no bounding boxes are needed at test time.

The paper also positions itself as bridging a gap between separate literatures that have not previously intersected. The audio-driven human animation community (Section 2.2) has extensively studied lip-sync and gesture generation but has largely ignored object interaction constraints. The HOI video generation community (Section 2.3) has explored structural conditioning and multi-reference synthesis but has not integrated region specialization or asymmetric co-training into a unified DiT backbone. CoInteract draws from both traditions while contributing a new architectural paradigm that addresses the specific failure modes neither community has resolved. This interdisciplinarity is not merely academic positioning—it reflects the genuine technical challenge that HOI synthesis sits at the intersection of two hard problems (human motion modeling and physical contact reasoning) that prior work addressed in isolation.

The Specific Gap CoInteract Fills

To summarize the gap in terms the paper's experiments later validate: existing methods, regardless of paradigm, produce HOI videos where structural stability and physical plausibility are decoupled from generation quality. A method can produce aesthetically beautiful videos (high AES, IQ) with good temporal coherence (high Smooth) and even good identity preservation (high DINOid, FaceSim), while simultaneously producing hands that merge into blobs, faces that distort, and objects that interpenetrate with the body. The ablation study (Table 3) makes this decoupling stark: removing the HOI co-generation stream causes VLM-QA to drop from 0.72 to 0.48 while video quality metrics remain largely unchanged (IQ even slightly increases from 0.749 to 0.753). In other words, the model learns to look good while being physically wrong—exactly the failure mode predicted by the RGB supervision weakness argument.

CoInteract's thesis is that this gap can be closed by making the backbone itself structure-aware, through a combination of structural co-supervision (the HOI stream) that enforces interaction geometry during training, and region specialization (the MoE) that dedicates model capacity to the most structurally sensitive locations, all while maintaining the inference efficiency of a standard single-stream model. The paper's experiments are designed to test exactly this thesis, and the dissonance between where prior methods succeed (aesthetics, tempoal coherence) and where they fail (interaction plausibility, hand quality) provides the empirical motivation for the architectural choices that follow.

3. Technical Approach

3.1 Reader Orientation

CoInteract is an end-to-end neural network that, given a photo of a person, a photo of a product, a text description, and an audio clip of a person speaking, generates a video of that specific person naturally holding and demonstrating that specific product while speaking the provided audio. The system solves the problem of physical implausibility in human-object interaction video synthesis by force-teaching geometric interaction rules to the video generation backbone during training, then removing the teaching mechanism at inference time, so that the model learns to respect object boundaries, maintain hand structure, and preserve facial detail without any extra computation when generating new videos.

3.2 Big-Picture Architecture (Diagram in Words)

The CoInteract system has five major components that interact during training but simplify dramatically at inference:

Dual-Stream Input Preparation — Raw HOI videos are preprocessed into paired representations: an RGB stream (normal video frames) and an HOI structure stream (human silhouettes fused with object masks, creating a geometry-only view that strips away appearance textures while preserving interaction boundaries). Both streams are encoded into a shared latent space via a pre-trained VAE.
Shared DiT Backbone with Stream-Specific Modulation — A Diffusion Transformer (initialized from WanS2V) processes both streams through shared attention and feed-forward layers. Stream-specific adaptive layer normalization parameters allow the same backbone to learn separate feature statistics for appearance (RGB) versus structure (HOI), without duplicating the full model.
Human-Aware Mixture-of-Experts (MoE) — Embedded within DiT blocks, a lightweight router (trained with spatial supervision from bounding box annotations) dispatches tokens to one of four specialized experts: a Shared expert (the original DiT feed-forward network as a shortcut), a Head expert, a Hand expert, and a Base expert for all other tokens. Only 256 hidden dimensions per specialist, so the parameter overhead is modest.
Asymmetric Co-Attention Mask — During a second training stage, the attention pattern between RGB and HOI streams is restricted: RGB tokens can only attend to other RGB tokens (making the RGB pathway self-contained), while HOI tokens can attend to both streams (allowing structure supervision to flow into shared parameters via backpropagation). This is the key mechanism that enables removing the HOI branch at inference.
Conditioning Pathways — Reference images (person, product), motion frames (previous video context for temporal continuity), text prompts, and audio features (from a pre-trained audio encoder) are injected via cross-attention into the DiT blocks. A 3D Rotary Positional Encoding (3D RoPE) assigns distinct spatiotemporal coordinates to different input types to enforce inductive biases about what should attend to what.

At inference time, the HOI stream, its patch embedding layer, and the HOI-specific modulation parameters are discarded entirely. Only the RGB stream flows through the shared backbone with the MoE, producing the final video. The structural knowledge is baked into the shared weights through the training regularization, not through an auxiliary output.

3.3 Roadmap for the Deep Dive

First, the dual-stream data preprocessing pipeline (Section 3.3, Fig. 4), because every architectural choice depends on understanding what the HOI structure stream IS — it is not an arbitrary auxiliary output but a carefully constructed geometry-only representation that strips away appearance to force structural learning.
Second, the joint flow-matching objective and shared backbone (Section 3.1, Equations 1-2), because these define WHAT the model learns and HOW the two streams share parameters — the loss formulation is the contractual agreement between the RGB and HOI streams about what constitutes a good representation.
Third, the 3D RoPE coordinate assignment (Section 3.1, Fig. 2b, Equation 3), because the position encoding scheme encodes critical inductive biases about which tokens should interact — temporal causality, spatial alignment between streams, and reference anchoring all flow from coordinate design.
Fourth, the two-stage asymmetric co-attention mechanism (Section 3.1, Fig. 3, Equation 4), because this is the architectural novelty that makes zero-overhead inference possible — the mask definition in Equation 4 is the contract that determines what information flows where during training and why the HOI branch can be removed.
Fifth, the Human-Aware MoE (Section 3.2, Fig. 2c, Equations 5-7), because this operates on top of the shared representations produced by the co-generation backbone and provides dedicated capacity for the regions where structural failure is most visible — the routing loss, stop-gradient operation, and expert architecture design together determine whether the MoE actually specializes or degrades to uniform processing.
Sixth, the inference-time simplification, because the entire design philosophy of CoInteract is about maximizing training-time supervision while minimizing inference-time cost — understanding what is and isn't present at test time is essential to evaluating the engineering tradeoffs.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an architectural innovation paper whose core idea is that physical interaction constraints for HOI synthesis can be injected into a video diffusion backbone through an auxiliary structure stream that co-trains with the RGB stream under an asymmetric attention mask, regularizing shared weights during training while being removable at inference for zero compute overhead, combined with a region-specialized mixture-of-experts module to target structurally fragile areas.

3.4.1 Dual-Stream Data Preprocessing: Building the HOI Structure Stream

The HOI structure stream is not an arbitrary auxiliary output. It is a carefully engineered representation designed to strip away all appearance information while preserving all interaction geometry. The paper needs the backbone to learn that hands must stop at object boundaries, that occlusions must be correctly ordered, and that contact points should be physically reasonable. If the auxiliary stream contained RGB appearance, the model could cheat by learning texture correspondences rather than geometric constraints. The construction pipeline (Fig. 4) has four stages.

Stage 1: Entity decoupling. Given a raw HOI video frame showing a person interacting with a product, the system first separates the person and product into independent reference images using Qwen-Edit [48], a pre-trained image editing model. This step is necessary because the training data consists of real product demonstration videos where person and product are already composited — to create the HOI structure representation that highlights interaction boundaries, the system needs to know exactly which pixels belong to the human and which to the object. The decoupled outputs are validated by a filtering module that checks whether the extracted person image, extracted product image, and original source frame form a consistent triplet; mismatched triplets (where Qwen-Edit hallucinates or misaligns entities) are discarded. The paper does not specify the validation criteria in detail, but the purpose is to ensure that downstream geometric processing receives clean entity segmentations.

Stage 2: Object mask extraction. The decoupled product reference is processed by SAM3 [3], a state-of-the-art segmentation model capable of producing pixel-accurate object masks. The output is a binary mask indicating which pixels belong to the product in each frame. The choice of SAM3 (rather than an off-the-shelf detector like DWPose for humans) is because product segmentation requires category-agnostic, open-vocabulary segmentation capability — the test set covers "diverse product categories" (Section 4.1), including handbags, cups, instruments, plush toys, and bins, and a fixed-category detector would fail on unseen products.

Stage 3: Human mesh recovery and projection. The decoupled person reference is processed by SAM3D-body [55], a robust full-body human mesh recovery model that estimates a 3D parametric body mesh (including hands, which are notoriously difficult for standard body pose estimators) from a single image. This mesh is then projected to the image plane, producing a 2D rendering of the human body's spatial occupancy — essentially a silhouette or binary mask showing where the human is located in pixel space. The paper describes this as reducing the human body to "a silhouette" while the object "retains its RGB appearance," but the key point is that this projection encodes the 3D spatial extent of the human (including depth information that is compressed but not lost in the 2D projection), providing far richer geometric information than a single-channel mask from a segmentation model.

Stage 4: Fusion into the HOI structure stream. The projected human rendering and the object mask are fused to create a 3-channel "texture-stripped HOI structure stream" denoted $V_h$ . The paper describes this as a "silhouette-like 3-channel rendering" — the human body is rendered as a silhouette (likely a binary or shaded representation of the mesh projection, possibly with depth or normal information encoded in the three channels), while the object retains its RGB appearance. This asymmetry is crucial: the human is geometry-only, the object is appearance-retaining. Why? Because the primary failure mode the paper addresses is the backbone's ignorance of human spatial occupancy relative to objects. Making the human a silhouette forces the model to register the human's spatial extent in relation to the object's visual appearance, rather than relying on both having correlated textures. If both were silhouettes, the model could learn a trivial mask-overlap loss; if both had RGB, the model could exploit texture shortcuts. The asymmetric representation forces the model to bridge the gap between geometry (human silhouette) and appearance (object RGB), learning that the silhouette boundary is a hard constraint on where the object's pixels can appear.

Output of preprocessing. The final training sample consists of:

$V_r$ : the original RGB video frames (the appearance target the model must learn to generate).
$V_h$ : the HOI structure stream, pixel-aligned with $V_r$ but containing geometry-only human information and RGB object information.
Face and hand bounding boxes per frame, obtained from off-the-shelf detectors (MediaPipe [29] and a hand contact detector [36]), providing explicit spatial supervision for the MoE router.

Both $V_r$ and $V_h$ are encoded into a shared latent space via a pre-trained VAE before being fed to the DiT backbone. This is standard practice in latent diffusion models: the VAE compresses each frame from pixel space to a lower-dimensional latent representation, reducing the computational cost of the diffusion process. The key point is that the same VAE encodes both streams, so they reside in a compatible latent space where cross-attention and shared processing are meaningful.

Design rationale for the preprocessing pipeline. Every choice in this pipeline implements the paper's core philosophy: strip away what the model should NOT learn from (appearance correlations between human and object) and amplify what it MUST learn (spatial occupancy and contact geometry). The silhouette projection from a 3D mesh, rather than a 2D segmentation mask, provides depth information that helps resolve occlusion ordering. The fusion of geometry (silhouette) with appearance (object RGB) creates a representation that forces the backbone to register contact boundaries rather than exploiting texture shortcuts. The independent preprocessing steps (Qwen-Edit → SAM3 → SAM3D-body → fusion) are modular so that improvements to any component (better segmentation, better mesh recovery) directly improve the training data quality.

3.4.2 Joint Flow-Matching Objective and Shared Backbone

With the dual-stream data prepared, the paper defines how the model learns from both streams simultaneously. The approach uses flow matching, the current dominant paradigm for training diffusion models, where the model learns to predict a velocity field that transports noise to data rather than predicting the noise itself. The key innovation is that this objective is applied jointly to both the RGB stream and the HOI structure stream, with a shared backbone processing both.

Standard flow matching background (implicit). While the paper does not derive flow matching from scratch, the operational principle is: given a data point $\mathbf{z}_0$ (a clean latent representation of a video), a noise sample $\mathbf{z}_1$ drawn from a simple distribution (typically Gaussian), and a timestep $t \in [0, 1]$ that interpolates between them as $\mathbf{z}_t = (1-t)\mathbf{z}_0 + t\mathbf{z}_1$ , the model predicts the velocity field $\mathbf{v}_\theta(\mathbf{z}_t, t, \mathbf{c})$ that would transport $\mathbf{z}_t$ toward $\mathbf{z}_0$ . The supervision target $\mathbf{v}$ is the true velocity $\mathbf{z}_1 - \mathbf{z}_0$ . At inference, starting from pure noise $\mathbf{z}_1$ , an ODE solver uses the predicted velocity field to iteratively step toward the data distribution along the probability flow.

Joint objective formulation. The paper defines the total flow-matching loss as:

$\mathcal{L}_{flow} = \mathcal{L}_{r} + \lambda_h \mathcal{L}_{h}$

where $\mathcal{L}_r$ is the RGB stream loss, $\mathcal{L}_h$ is the HOI stream loss, and $\lambda_h$ is a weighting coefficient set to 1 unless otherwise stated in ablations.

Each stream's individual loss expands as:

$\mathcal{L}_{r} = \mathbb{E}_{t,\mathbf{z}_0,\mathbf{z}_1}\!\left[\left\|\mathbf{v}_r - \mathbf{v}_\theta(\mathbf{z}_{r,t}, t, \mathbf{c})\right\|_2^2\right]$

$\mathcal{L}_{h} = \mathbb{E}_{t,\mathbf{z}_0,\mathbf{z}_1}\!\left[\left\|\mathbf{v}_h - \mathbf{v}_\theta(\mathbf{z}_{h,t}, t, \mathbf{c})\right\|_2^2\right]$

In these expressions:

$\mathbf{z}_{r,t}$ is the noisy RGB latent at timestep $t$ , constructed by interpolating between the clean RGB latent $\mathbf{z}_{r,0}$ (the VAE-encoded $V_r$ ) and Gaussian noise $\mathbf{z}_{r,1}$ .
$\mathbf{z}_{h,t}$ is the noisy HOI latent at the SAME timestep $t$ , constructed identically from $\mathbf{z}_{h,0}$ (the VAE-encoded $V_h$ ).
$\mathbf{v}_r = \mathbf{z}_{r,1} - \mathbf{z}_{r,0}$ and $\mathbf{v}_h = \mathbf{z}_{h,1} - \mathbf{z}_{h,0}$ are the true velocity fields for each stream (the difference between noise and data — the direction and magnitude of the vector that would move noise directly to the clean latent if integrated over the full unit interval).
$\mathbf{v}_\theta(\cdot)$ is the DiT backbone's predicted velocity field, which takes as input a noisy latent, the timestep $t$ , and conditioning $\mathbf{c}$ .
$\mathbf{c}$ denotes the full conditioning bundle: text prompt embeddings, audio features from a pre-trained audio encoder, dual reference image embeddings (person and product), and motion frame latents.
The expectation $\mathbb{E}_{t,\mathbf{z}_0,\mathbf{z}_1}$ means the loss is averaged over randomly sampled timesteps, data points, and noise samples — standard practice for diffusion training.

What this jointly computes. At each training step, the model receives TWO noisy latents (one RGB, one HOI) at the same noise level and must predict the velocity field for both. Both predictions flow through the same shared DiT backbone, meaning the same attention weights and feed-forward parameters process both streams. The squared L2 norm $\|\cdot\|_2^2$ penalizes the Euclidean distance between predicted and true velocity vectors at every spatial and temporal position in the latent grid. The sum $\mathcal{L}_r + \lambda_h \mathcal{L}_h$ means gradients from both streams accumulate into the shared parameters, with the HOI loss contributing equally to the RGB loss (since $\lambda_h = 1$ ). The output is a single scalar loss value per training step, but the gradient signal encodes information from both appearance and structure targets.

Why this form — joint training rather than separate models or a single combined loss.

(a) Shared backbone with stream-specific modulation. The two streams are processed by the same DiT blocks, but the paper notes that "stream-specific modulation parameters (scale and shift in adaptive layer normalization)" are used within each block. This means that while the attention and feed-forward weights are shared, the normalization layers that control feature statistics (mean and variance scaling) are separate per stream. This allows the backbone to learn that RGB features have different activation distributions than HOI features — for example, RGB might need to preserve fine texture gradients while HOI might need sharp edges at object boundaries — without needing a completely separate model for each stream. The parameter sharing forces the representations to be compatible and mutually informative, while the separate normalization prevents destructive interference between the different statistical properties of appearance and structure data.

(b) Separate losses rather than a single combined loss. The paper sums $\mathcal{L}_r$ and $\mathcal{L}_h$ rather than, say, feeding both latent streams to the backbone and computing a single velocity prediction over the concatenated latents. The separate loss formulation ensures that the model receives independent supervision signals for each stream — it must learn to predict the RGB velocity field (which includes color, texture, and lighting) and the HOI velocity field (which is dominated by geometric boundaries) as separate tasks sharing the same backbone capacity. This is more constraining than a concatenated approach because the model cannot blur RGB and HOI information together; it must represent both faithfully in the shared hidden states, which is exactly what forces the RGB stream's parameters to internalize structural information.

(c) The $\lambda_h = 1$ default is a statement about equal importance. Setting the HOI weight equal to the RGB weight signals that structural fidelity is as important as appearance fidelity. An alternative would be $\lambda_h < 1$ (treating structure as a weak regularizer) or $\lambda_h > 1$ (prioritizing structure over appearance). The paper found $\lambda_h = 1$ to work well and did not report sweeping this hyperparameter, suggesting it is not highly sensitive — likely because the two streams provide complementary rather than competing information.

(d) The expectation over $t$ is uniform. The expectation $\mathbb{E}_{t}$ implies that timesteps are sampled uniformly from $[0, 1]$ (or from the discrete training schedule). This means the model learns to denoise at ALL noise levels simultaneously, from nearly pure noise ( $t \approx 1$ ) where only coarse structure is visible, to nearly clean data ( $t \approx 0$ ) where fine texture details emerge. The HOI stream provides structure supervision at all noise levels: at high noise, it reinforces that the coarse spatial layout (human on one side, object on another) should be preserved; at low noise, it reinforces that fine boundaries (finger edges against object surfaces) should be sharp. This multi-scale structural supervision is precisely what RGB-only training lacks.

The DiT backbone must process a heterogeneous collection of tokens: RGB noisy latents (the video being generated), HOI noisy latents (the structure target), reference image latents (static person and product photos), motion frame latents (previous video context), and text/audio features injected via cross-attention. How should the attention mechanism know which tokens are spatially adjacent, which are temporally sequential, and which are static references that should NOT be treated as temporally adjacent frames? The paper's answer is a carefully designed 3D Rotary Positional Encoding (3D RoPE) scheme that assigns each token a 3D coordinate $(h, w, t)$ — height, width, and time — with coordinate values chosen to encode specific inductive biases.

Background on RoPE (implicit). Rotary Positional Encoding modifies the attention computation so that the dot product between query and key vectors depends on their relative positions. For 1D positions, RoPE rotates query and key vectors by angles proportional to their absolute positions, so that the attention weight $\langle \mathbf{q}_i, \mathbf{k}_j \rangle$ naturally depends on $(i-j)$ . For 3D, each dimension (height, width, time) gets its own set of rotation frequencies, and the total rotation is the sum of the per-dimension rotations. The result is that tokens with similar $(h, w, t)$ coordinates attend more strongly to each other, all else being equal.

The coordinate assignment scheme (Fig. 2b). The formal position encoding for a token $x_{i,j,k}$ at height index $i$ , width index $j$ , and temporal index $k$ is:

$\text{Pos}(x_{i,j,k}) = \text{RoPE}_{3D}(h_i, w'_j, t_k)$

where $h_i$ and $t_k$ are the height and time coordinates assigned normally, but $w'_j$ is a modified width coordinate that accounts for a virtual horizontal offset applied to the HOI stream latents.

The coordinate assignment has three distinct regimes, each encoding a specific inductive bias:

Regime 1: Spatial coordinates for dual streams (width offset). The RGB noisy latents and HOI noisy latents have identical spatial dimensions (same height, same number of frames) but are assigned different width coordinate ranges. Specifically:

RGB tokens receive width coordinates $w \in [0, W]$ , where $W$ is the spatial width of the latent grid.
HOI tokens receive width coordinates $w \in [-W, 0]$ , shifted leftward by exactly $W$ positions.

The two streams are concatenated along the width dimension in the token sequence, so a DiT block receives a concatenated sequence $[z_h, z_r]$ where the first half are HOI tokens ( $w < 0$ ) and the second half are RGB tokens ( $w \geq 0$ ). But their height and time coordinates are identical — an HOI token at spatial location $(i, j)$ and frame $k$ has the same $(h_i, t_k)$ as the RGB token at the same spatial location and frame.

What this spatial assignment achieves. Two tokens with the same height and time but widths $w$ and $w + \Delta w$ (where $\Delta w$ is the virtual offset) have a relative position of exactly $\Delta w$ in the width dimension. For an RGB token at $w$ and an HOI token at the corresponding spatial location with width $w - W$ , the relative width distance is $W$ . Because RoPE attention weights decay with relative distance, tokens at the same spatial location across the two streams attend less strongly than tokens within the same stream at adjacent spatial positions (which have $\Delta w = 1$ ). This prevents the two streams from collapsing into a simple averaging operation — they maintain separate representations while still being close enough in position space to allow cross-stream attention when beneficial.

The shared height and time coordinates ensure that if cross-stream attention does occur, it is spatially aligned — an RGB token at position $(i, j, k)$ is most likely to attend to the HOI token at the same spatial position and frame, rather than a random spatial location. This spatial alignment is what allows the HOI stream's geometric information to transfer to the RGB pathway in a location-specific manner.

Regime 2: Temporal causality for motion frames. Historical motion frames (previous video context that preserves temporal continuity) are assigned negative temporal indices: $t \in \{-N, -N+1, \ldots, -1\}$ . The current generation window (the video frames being synthesized) receives temporal indices $t \in \{0, 1, \ldots, T-1\}$ .

What this temporal assignment achieves. Any token in the generation window (at time $t \geq 0$ ) has a negative relative temporal distance to any motion frame (at time $t < 0$ ), meaning the attention weight depends naturally on $|t_{\text{gen}} - t_{\text{mot}}|$ . The smallest distance is to the most recent motion frame ( $t = -1$ ), encouraging the model to attend most strongly to the immediately preceding context for temporal continuity. Motion frames are placed "before" the generation window on the time axis, establishing a causal ordering: the future depends on the past, not vice versa.

Importantly, the paper does NOT enforce strict causal masking (where generation tokens cannot attend to future generation tokens). The temporal coordinates encode a soft inductive bias (closer in time → more attention) rather than a hard constraint (future tokens masked). This is standard for video diffusion models using full spatiotemporal attention, where the model benefits from bidirectional temporal context for generating temporally coherent videos.

Regime 3: Reference anchoring at far-field temporal positions. The static reference images (person identity $I_{\text{ref}}$ and product $I_{\text{prod}}$ ) are assigned temporal indices far in the future: $t = 30, 31$ (the paper uses values 30 and 31, with the generation window presumably spanning a smaller range — exact generation window length is not specified, but the "significant offset" is the key design principle).

What this reference anchoring achieves. By placing reference tokens at a temporal distance of approximately 30 from the generation tokens, the relative positional distance is large. However, because RoPE attention uses sinusoidal frequencies, large positional distances do NOT simply go to zero — they produce oscillatory attention patterns that can still carry information. The "significant offset" serves two purposes:

It distinguishes reference tokens from adjacent frames, preventing the model from treating a reference image as just the next frame in the sequence (which would cause temporal blurring or identity drift).
It encourages the model to learn that references are global anchors — tokens that should be attended to roughly equally from any frame in the generation window because the relative distance to all generation frames is approximately the same (the variation of $\pm 10$ around $t=30$ is small compared to the absolute offset of 30).

Why 3D RoPE rather than learned or absolute positional encodings. RoPE's key property is relative position encoding — the attention weight depends on $(i-j)$ for 1D, or $(h_q-h_k, w_q-w_k, t_q-t_k)$ for 3D. This means the model learns patterns parameterized by relative distance, not absolute position. For video generation, this is crucial because:

The model should learn that "tokens 1 frame apart" behave similarly regardless of whether they are at the start or end of the video (translation invariance in time).
The model should learn that "tokens at the same spatial location across streams" have a consistent relative position $W$ regardless of where they appear (translation invariance in space).
Static references should be treated as infinitely far in time, not as a specific absolute position that the model might overfit to.

Learned absolute positional embeddings would overfit to the specific maximum sequence length and would not generalize to different video lengths. Sinusoidal absolute encodings would not provide the relative-position inductive bias that is essential for the temporal causality and reference anchoring schemes.

3.4.4 Two-Stage Asymmetric Co-Attention: The Zero-Overhead Mechanism

This is the architectural centerpiece that enables CoInteract's most distinctive claim: structural supervision during training with zero additional inference cost. At its core is a training strategy with two distinct attention patterns, described in Section 3.1 and illustrated in Fig. 3.

The core problem the asymmetric mask solves. If we train with standard bidirectional self-attention across the concatenated $[z_h, z_r]$ token sequence (Stage 1), the model learns rich cross-stream dependencies — RGB tokens attend to HOI tokens, HOI tokens attend to RGB tokens, and both benefit from the shared representation. However, this creates a dependency: at inference time, the RGB stream's predictions depend on having HOI tokens present because the RGB queries have learned to attend to HOI keys. Removing the HOI stream would change the RGB stream's behavior, degrading quality. The goal is to make the RGB pathway self-contained — capable of generating high-quality videos using only RGB tokens — while still allowing HOI supervision to flow into the shared parameters during training.

Stage 1: Full bidirectional attention (rapid convergence). In the first training stage (5,000 iterations per Section 4.1), standard bidirectional self-attention is applied across the full concatenated token sequence $[z_h, z_r]$ , including both streams and the conditioning tokens (reference images, motion frames). Every token can attend to every other token. This stage serves to establish initial cross-stream couplings — the model learns coarse correspondences between RGB appearance and HOI structure, so that when the asymmetric mask is introduced in Stage 2, the representations already encode cross-modal relationships. Starting with the asymmetric mask from scratch would make training slower because the RGB stream would initially have no access to structural information, requiring it to bootstrap from pure appearance supervision before the HOI stream's gradient contributions become meaningful.

Stage 2: Asymmetric co-attention (structural regularization). After the initial convergence period, the attention mask is changed. Let $\mathcal{T}_r$ be the set of RGB tokens and $\mathcal{T}_h$ be the set of HOI tokens. The asymmetric attention mask $\mathbf{M}$ is defined element-wise:

$\mathbf{M}_{i,j} = \begin{cases} 1, & \text{if } i \in \mathcal{T}_r,\; j \in \mathcal{T}_r,\\ 1, & \text{if } i \in \mathcal{T}_h,\; j \in \mathcal{T}_r \cup \mathcal{T}_h,\\ 0, & \text{otherwise.} \end{cases}$

In this definition, $i$ indexes the query token (the token that is asking "what should I attend to?"), $j$ indexes the key token (the token being attended to), and $\mathbf{M}_{i,j} = 1$ means the attention weight from query $i$ to key $j$ is computed normally, while $\mathbf{M}_{i,j} = 0$ means it is forced to $-\infty$ before the softmax (zero attention weight after softmax).

What the mask enforces — row by row interpretation.

Row 1 (RGB queries, RGB keys): $\mathbf{M} = 1$ for $i \in \mathcal{T}_r, j \in \mathcal{T}_r$ . All RGB tokens can attend to all other RGB tokens. This is standard self-attention within the RGB stream. Critically, this makes the RGB pathway entirely self-contained — every RGB token's output is computed purely from other RGB tokens, with zero dependence on HOI tokens. This is why the HOI stream can be dropped at inference without affecting the RGB stream's computation graph.
Row 2 (HOI queries, all keys): $\mathbf{M} = 1$ for $i \in \mathcal{T}_h, j \in \mathcal{T}_r \cup \mathcal{T}_h$ . HOI tokens can attend to BOTH HOI tokens (within-stream self-attention) and RGB tokens (cross-stream attention). The HOI stream sees "clean" RGB features that contain appearance, texture, and context information, allowing it to leverage the RGB pathway's richer representations to improve its own structure predictions. This is analogous to how a student model (HOI) learns from a teacher model (RGB) — the HOI stream accesses more information than it would have from structure-only tokens.
Row 3 (all other query-key pairs): $\mathbf{M} = 0$ . This covers only the case $i \in \mathcal{T}_r, j \in \mathcal{T}_h$ — RGB queries attending to HOI keys. This is explicitly FORBIDDEN. The RGB stream cannot directly access structural information through attention; it can only receive it indirectly through the shared backbone parameters that are updated by the HOI loss gradient.

The gradient flow is the key mechanism. The paper states this mechanism explicitly:

"Crucially, $\mathcal{L}_{h}$ backpropagates through the $\text{HOI} \leftarrow \text{RGB}$ cross-attention into the shared DiT parameters; since the RGB stream reuses the same parameters at inference, this transfers interaction-structure supervision to the RGB generator even when the HOI branch is removed."

Let's trace this gradient flow step by step:

During training, the HOI stream loss $\mathcal{L}_h$ is computed from the HOI velocity prediction.
To compute gradients of $\mathcal{L}_h$ with respect to the shared DiT parameters, backpropagation flows through the HOI stream's computation graph.
Within each attention layer, the HOI query attends to RGB keys. The gradient of $\mathcal{L}_h$ with respect to the RGB value vectors (or more precisely, with respect to the parameters that produced those RGB key/value representations) flows back through the attention weights from HOI queries to RGB keys.
This gradient signal tells the shared parameters: "the way you represent this RGB token should change so that when an HOI token at this location attends to it, the resulting HOI prediction is more accurate."
Since the SAME parameters process RGB tokens at inference time, this structural regularization persists even without HOI tokens present — the parameters have been optimized not just for RGB velocity prediction but also for representing information that helps predict interaction structure.

In other words, the RGB stream does not need to "see" the HOI stream at inference because the parameters that process RGB tokens have already been shaped by structural supervision during training. The HOI stream acts as a training-time regularizer that is distilled into the shared weights, not as an inference-time auxiliary input.

Why not make the mask symmetric (bidirectional) and just train with the HOI stream? This is exactly the ablation condition "w/o Asym. Mask" in Table 3. With bidirectional attention, the RGB stream learns to depend on HOI tokens — the RGB pathway's computation graph includes attention edges that require HOI key/value vectors. At inference, you must keep the HOI branch to avoid changing the RGB pathway's behavior (since removing HOI tokens would remove inputs the RGB queries expect). This results in $4.13\times$ inference cost (Table 3) because you double the number of RGB-sized tokens to process. The slight performance improvement (VLM-QA 0.76 vs 0.72) comes at a crippling computational cost, making the method impractical for deployment.

Why not train with the asymmetric mask from the start? The paper does not explicitly justify why Stage 1 (full attention) is needed before Stage 2 (asymmetric attention), but the two-stage design is a common pattern when training with asymmetric information flow. Stage 1 allows the model to bootstrap cross-stream couplings — HOI tokens learn what RGB features are useful for structure prediction, and shared parameters build representations that contain both modalities. If the asymmetric mask were applied from initialization, the RGB stream would never receive any cross-stream attention, making the HOI loss gradient the ONLY source of cross-modal learning. This gradient signal through shared parameters is weaker and slower than direct attention, so Stage 1 provides a "warm start" where the model first establishes rough cross-modal correspondences before transitioning to the more efficient (but slower-learning) asymmetric regime.

What "optional" means in Fig. 3(c). The inference diagram shows the HOI stream input with a dashed "Optional" arrow, and the caption notes: "Optional" for the HOI stream, meaning if you want the $4.13\times$ cost inference with slightly better quality (w/o Asym. Mask variant), you can keep it; if you want $1.04\times$ cost inference (Full Model), you drop it. The Full Model is the default.

3.4.5 Human-Aware Mixture-of-Experts (MoE)

While the co-generation paradigm injects global interaction geometry priors into the backbone, hands and faces remain structurally fragile because they involve high-frequency articulation (individual finger positions, subtle facial expressions) that the global structure stream can only approximately constrain. The Human-Aware MoE addresses this by dedicating specialized processing capacity to anatomically sensitive regions, using a routing mechanism trained with explicit spatial supervision.

MoE architecture. Standard DiT blocks contain a feed-forward network (FFN) that processes every token identically — the same FFN weights apply to a token representing a hand pixel, a background pixel, or a product pixel. The MoE replaces this single FFN with a mixture of four experts:

Shared expert: Reuses the original DiT FFN as a shortcut path. Every token's output is always added to the Shared expert's output. This ensures that even if the router misclassifies a token, the backbone still functions (the Shared expert provides a "default" processing path).
Head expert: A lightweight FFN (hidden dimension 256) specialized for face region tokens.
Hand expert: A lightweight FFN (hidden dimension 256) specialized for hand region tokens.
Base expert: A lightweight FFN (hidden dimension 256) for all other tokens (body, background, object).

All three lightweight experts have hidden dimension 256. The paper does not specify the original DiT FFN hidden dimension, but for context, typical DiT models have FFN hidden dimensions of 4× the token embedding dimension (often 1,152 to 4,608 for large models). A hidden dimension of 256 is thus a substantial reduction — each lightweight expert adds minimal parameters relative to the base model, consistent with the paper's claim of "minimal parameter overhead."

Router design and spatially supervised routing. The router $\mathcal{R}$ is a two-layer MLP that takes the hidden state of each token and outputs a probability distribution over experts. The routing probability for token $x_i$ with hidden state $\mathbf{h}_i$ is:

$G(x_i) = \text{Softmax}(\mathcal{W}_g \cdot sg[\mathbf{h}_i])$

where $\mathcal{W}_g$ is the learnable weight matrix of the router MLP, $sg[\cdot]$ is the stop-gradient operation, and $G(x_i)$ is a probability vector over the set of experts.

The stop-gradient operation is a critical design choice. By applying $sg$ , the router's gradients do NOT backpropagate into the DiT hidden states. Why is this important? If the router could influence the hidden state representations, there would be an incentive for the model to make tokens "look like" the region they should be routed to, which would distort the representation learning. For example, the model might learn to encode hand-like features in non-hand tokens to route them to the Hand expert, or conversely, to suppress hand features so tokens get routed to the Base expert. The stop-gradient breaks this feedback loop: the hidden states are optimized purely by the flow matching loss (appearance and structure reconstruction), while the router learns to classify these frozen representations into spatial regions. The router adapts to the representations rather than the representations adapting to the router.

Spatial supervision via cross-entropy routing loss. The router is not left to discover regional specialization on its own (which would be unstable and might converge to degenerate solutions). Instead, it is trained with explicit bounding box supervision from off-the-shelf detectors [29, 36]:

$\mathcal{L}_{route} = - \sum_{i} \sum_{k \in \{head, hand, base\}} \mathbb{1}(y_i = k) \log (G(x_i)_k)$

where $y_i$ is the ground-truth region label for token $i$ (determined by whether the token's spatial position falls within a face bounding box, hand bounding box, or neither), $\mathbb{1}(\cdot)$ is the indicator function (1 if the label matches, 0 otherwise), and $G(x_i)_k$ is the router's predicted probability for expert $k$ .

What this computes. For each token, the routing loss is the negative log probability that the router assigns to the correct expert. If a token inside a hand bounding box gets a routing probability of 0.9 for the Hand expert, the loss contribution is $-\log(0.9) \approx 0.105$ . If it gets 0.1 for the Hand expert (and 0.9 for Base), the loss is $-\log(0.1) \approx 2.30$ . The loss is summed over all tokens where a ground-truth label exists (tokens outside all bounding boxes are still labeled as "base"). The result is a single scalar routing loss per training step.

Why cross-entropy rather than, say, a contrastive loss. Cross-entropy is the standard maximum-likelihood objective for classification. The ground-truth labels are deterministic (a token is either inside a bounding box or not), so cross-entropy pushes the router toward assigning probability 1.0 to the correct expert. The Softmax over expert logits naturally creates competition between experts — increasing probability to one expert automatically decreases probability to others, preventing the router from assigning high probability to multiple experts simultaneously. This competition is important because if multiple experts had high probability on the same token, their outputs would be averaged (in a standard MoE with soft routing), which would reduce specialization.

The total training objective combines flow matching and routing supervision:

$\mathcal{L}_{total} = \mathcal{L}_{flow} + \eta \mathcal{L}_{route}$

where $\eta = 1$ controls the relative weight of the routing loss. The paper sets $\eta = 1$ without reporting sensitivity analysis, implying that the routing loss is roughly balanced with the flow matching loss in magnitude.

What happens at inference time (when bounding boxes are unavailable). The router operates purely on the learned DiT hidden states — it receives $\mathbf{h}_i$ for token $x_i$ and outputs $G(x_i)$ without requiring any external annotations. The spatial supervision during training has shaped $\mathcal{W}_g$ to recognize region-specific activation patterns in the hidden states. For example, the hidden states of tokens in hand regions likely encode high-frequency spatial gradients and articulation-specific features that the router learns to associate with the Hand expert. At inference, the router identifies these patterns in the features and routes accordingly, even for novel hand poses or previously unseen product interactions.

The outputs of the selected experts are combined using the routing probabilities as mixing weights (standard soft MoE), so a token might receive a weighted combination of Shared + Hand outputs if the router is uncertain, or a strong contribution from a single specialist if the router is confident. The Shared expert always contributes, providing a backup if the router misclassifies.

Why only 256 hidden dimensions per lightweight expert. The paper describes these as "lightweight" and emphasizes "minimal parameter overhead." The operational constraint is that the MoE should not significantly increase inference cost relative to the base model. With three lightweight experts at hidden dimension 256 each, plus the Shared expert (original FFN, likely hidden dimension 1,024+), the total FFN parameter increase is modest (approximately $3 \times 256 \times d_{\text{model}} \times 2$ for linear layers, compared to the base FFN's $d_{\text{model}} \times 4d_{\text{model}} \times 2$ ). The paper reports the Full Model inference cost as $1.04\times$ the MoE-free baseline (Table 3), confirming that the overhead is indeed marginal — likely dominated by the router MLP and the small expert FFNs, with the routing computation itself being cheap (a two-layer MLP per token is negligible compared to the DiT block's self-attention and FFN).

Comparison to prior region-specialized approaches. CyberHost [25] uses "region codebook attention" — essentially a learned latent feature per region that is concatenated to tokens for specialized processing. This acts as an external add-on that modifies the attention computation. Make-Your-Anchor [16] applies a post-hoc face enhancement network after generation — a completely separate model that refines only face regions. The Human-Aware MoE differs in two key ways:

Integration depth: The MoE is embedded within the DiT backbone, not bolted on externally. The specialized processing happens at every DiT block, meaning hand and face features are refined at multiple scales and abstraction levels throughout the generation process.
Supervised routing: The router is explicitly trained with spatial labels, ensuring that specialization is anatomically meaningful. Unsupervised MoE routing in diffusion models can converge to load-balancing optima (e.g., equal tokens per expert) that have no anatomical correspondence. The cross-entropy loss forces the routing to align with the anatomical regions that actually need specialized processing.

Preventing the MoE from harming non-hand/face regions. The Shared expert (the original DiT FFN) always contributes to every token. For a token in the background or on the product, the router should assign high probability to the Base expert, but even if it misclassifies, the Shared expert provides the "standard" processing path. Additionally, the Base expert is trained on all non-hand, non-face tokens, so it specializes on general body, background, and object features. The MoE should not degrade quality outside the targeted regions because the Shared expert ensures continuity with the original DiT backbone.

3.4.6 Conditioning Mechanisms: References, Motion, Audio, and Text

The DiT backbone receives conditioning through several pathways that the paper describes but does not formalize with equations for every component. The conditioning bundle $\mathbf{c}$ in the flow matching equations (Equation 2) comprises four elements:

Dual reference images. The person identity reference $I_{\text{ref}}$ and product reference $I_{\text{prod}}$ are encoded (likely through the same VAE used for video frames) and concatenated as reference tokens at the far-field temporal positions ( $t = 30, 31$ ). These tokens participate in self-attention with the noisy latents (through the 3D RoPE relative position mechanism) and are also available for cross-attention (if the DiT architecture uses cross-attention blocks rather than pure self-attention — the paper does not specify the exact attention architecture, but the reference to "Cross-Attention" in Fig. 2a suggests dedicated cross-attention layers exist).

Motion frames. Historical context frames $V_{\text{mot}}$ are encoded and placed at negative temporal indices to provide temporal continuity. These preserve the person's appearance and motion from previous segments, enabling autoregressive generation where each new clip continues naturally from the previous one. The paper references FilmWeaver [30] and WanS2V [10] for this mechanism, suggesting it is inherited from the base architecture.

Text prompts. The text description of the desired action (e.g., "The man gestures with right hand while holding the white instrument") is encoded by a text encoder (likely a CLIP or T5 variant) and injected via cross-attention. Text provides semantic control over what action to perform.

Audio features. The speech audio is encoded by a pre-trained audio encoder (likely Whisper or a similar model) to produce per-frame audio features that drive lip synchronization and co-speech gestures. These are projected through an MLP and injected via cross-attention or concatenation. The paper's architecture inherits from WanS2V [10], a speech-driven video generation model, suggesting the audio conditioning pathway follows WanS2V's design.

Why the conditioning is not the focus of the paper. CoInteract's primary contribution is the co-generation paradigm and the MoE, not the conditioning architecture. The conditioning mechanisms appear to be largely inherited from WanS2V, with the dual-reference mechanism being the primary conditioning innovation (providing separate person and product references rather than a single composited reference). The paper focuses experimental validation on the HOI-specific metrics (VLM-QA, HQ) rather than conditioning fidelity metrics, consistent with the research question being "how to enforce physical interaction constraints" rather than "how to improve conditioning quality."

3.4.7 Inference-Time Simplification

At inference time, CoInteract simplifies dramatically compared to the training configuration:

What is removed:

The HOI structure stream and its patch embedding layer are discarded entirely. No HOI latents are constructed, no HOI velocity predictions are computed, and no HOI loss is applied.
The HOI-specific modulation parameters (scale and shift in adaptive layer normalization specific to the HOI stream) are unused because only RGB latents flow through the backbone.
The preprocessing pipeline (Qwen-Edit entity decoupling, SAM3 object masking, SAM3D-body mesh recovery, fusion into HOI structure) is NOT run at inference — it was purely a training data construction step.

What is retained:

The shared DiT backbone parameters, which have been regularized by the HOI loss during training. These parameters encode structural interaction knowledge even though they only process RGB tokens.
The RGB-specific modulation parameters for adaptive layer normalization.
The Human-Aware MoE: the router $\mathcal{R}$ and all four experts (Shared, Head, Hand, Base) operate normally on RGB tokens. The router classifies tokens into anatomical regions based on the learned hidden state patterns, without requiring any external bounding box annotations.
All conditioning pathways: reference images at $t = 30, 31$ , motion frames at $t < 0$ , text, and audio features.

What is generated:

Starting from pure Gaussian noise in the RGB latent space, the ODE solver (40 inference steps per Section 4.1, with CFG scale 5) iteratively denoises using the predicted velocity field from the shared backbone.
The denoised RGB latents are decoded by the VAE decoder to produce the final video frames at 480p resolution.
No HOI structure output is produced or needed — the structure knowledge is baked into the generation process, not output as a separate product.

Why this is architecturally elegant. The asymmetric co-attention design means the model is trained with a "scaffold" (the HOI stream) that teaches it physical interaction rules, but the scaffold is removed before deployment. This is analogous to knowledge distillation where the teacher and student share the same architecture: the HOI stream (with full cross-attention access, including RGB features) acts as the teacher, and the RGB stream (with self-contained attention) acts as the student, but they share the same parameters. The HOI loss backpropagates through the shared parameters, effectively distilling structural knowledge into the weights without requiring the HOI branch at test time.

The inference cost in Table 3 (1.04× for Full Model vs. MoE-free baseline) reflects only the MoE's small parameter overhead — the three lightweight FFNs and the router MLP. The HOI stream contributes exactly 0.00× overhead because it is completely removed. The 4.13× cost of the "w/o Asym. Mask" variant, in contrast, reflects processing both RGB and HOI latents through the full backbone, doubling the token count (and hence the self-attention cost, which scales quadratically with sequence length). This dramatic difference validates the asymmetric design as enabling structural supervision at negligible deployment cost.

3.4.8 Training Configuration and Hyperparameters (Complete Specification)

The paper specifies training details in Section 4.1, which are integral to understanding the technical approach. All architecture components described in Sections 3.1–3.2 are realized through these training choices.

Initialization and base model. CoInteract is initialized from WanS2V [10], a pre-trained speech-driven video generation model. This provides strong initialization for the RGB generation pathway, audio conditioning, and temporal modeling. The DiT backbone architecture, the VAE, the audio encoder, and the text encoder are all inherited from WanS2V. Training from scratch for video diffusion is computationally prohibitive (requiring hundreds of GPU-years), so initialization from a strong video generation pre-trained model is essential.

Dataset. The training data consists of 40 hours of product demonstration and live-streaming videos, curated following the preprocessing pipeline in Fig. 4 (Section 3.3). After quality filtering (which removes clips with failed entity decoupling, invalid meshes, or missing bounding boxes), 12,000 high-quality clips are retained. Each clip has:

Paired RGB video $V_r$ and HOI structure video $V_h$ .
Per-frame hand bounding boxes from an off-the-shelf detector [36].
Per-frame face bounding boxes from MediaPipe [29].
Silhouette masks (from the mesh projection and object segmentation).
The original text prompt and audio track.

A held-out test set of 50 clips is reserved, covering diverse product categories and unseen identities (neither the person nor the product appeared in the training set). The test set size (50 clips) is small relative to the training set (12K clips), but this is typical for video generation benchmarks where manual evaluation and extensive metric computation per clip make large test sets impractical. The diversity claim ("diverse product categories and unseen identities") is important because it tests whether the structural priors generalize beyond training objects — if the method only works on products seen during training, the structural co-generation is just memorizing specific interaction patterns rather than learning generalizable geometry.

Expert configuration. The Human-Aware MoE uses exactly:

1 Shared expert (the original DiT FFN, parameter count inherited from WanS2V).
3 lightweight experts (Head, Hand, Base), each implemented as "a light FFN with hidden dimension 256."
A router MLP with 2 layers (input dimension = DiT hidden dimension, output dimension = 4 for the 4 experts).

Optimization. The optimizer is AdamW [19] with:

Learning rate: $1 \times 10^{-4}$
Learning rate schedule: cosine annealing (implied by "cosine annealing" without further specification — likely standard cosine decay from the initial learning rate to zero or a small fraction over the total training steps).
Loss weights: $\lambda_h = 1$ (HOI loss weight), $\eta = 1$ (routing loss weight).
No mention of batch size, gradient accumulation, or mixed precision, which are relevant for video diffusion training but not reported.

Two-stage training schedule. Training proceeds in two stages:

Stage 1 (full bidirectional attention): 5,000 iterations. The model sees the concatenated $[z_h, z_r]$ token sequence with standard self-attention across all tokens, allowing RGB and HOI streams to attend to each other freely.
Stage 2 (asymmetric co-attention): 2,000 iterations. The attention mask from Equation 4 is applied, restricting RGB queries to RGB keys only. The total training is 7,000 iterations, which is relatively brief — designed to adapt the pre-trained WanS2V backbone to the dual-stream HOI task without catastrophic forgetting of the original video generation capabilities. The asymmetric stage is shorter (2K vs 5K iterations) because the cross-stream couplings are already established in Stage 1, and Stage 2 primarily refines the self-contained RGB pathway and the structural regularization through shared parameters.

Inference settings. At test time:

Classifier-Free Guidance (CFG) scale: 5. This is relatively high — typical CFG values range from 1 (no guidance) to 7.5 (strong guidance). A scale of 5 means the model strongly prioritizes conditioning adherence over sample diversity, which is appropriate for the controlled setting of HOI synthesis where faithful product and identity preservation matters more than creative variation.
Inference steps: 40. This is typical for flow-matching models with an ODE solver, balancing quality against speed. More steps would improve quality marginally at proportional computational cost.
Resolution: 480p. This is modest by modern video generation standards (many models target 720p or 1080p), likely constrained by the WanS2V base model's native resolution and the computational cost of dual-stream training at higher resolutions.

The 1.04× inference cost factor in context. The full model (CoInteract with MoE, without HOI stream at inference) has $1.04\times$ the inference cost of the MoE-free baseline. This 4% overhead comes entirely from the MoE components: the router MLP (two layers, small), the three lightweight expert FFNs (hidden dimension 256 each), and the weighted combination of expert outputs. Given the substantial improvements in hand quality (HQ 0.724 vs 0.658) and face similarity (FaceSim 0.696 vs 0.662) that the MoE provides (Table 3), the 4% cost increase is extremely favorable — eliminating the MoE saves negligible compute while degrading sensitive-region quality by 9–11%.

4. Key Insights and Innovations

Innovation 1: Diagnosing the RGB Supervision Signal Weakness as the Root Cause of Interaction Failure

The paper makes a conceptual diagnosis that is more fundamental than any single architectural contribution: the recurring failures in HOI video synthesis (interpenetration, hand collapse, face blurring) are not due to insufficient data scale, model capacity, or architecture design, but rather to an inherent signal weakness in RGB-only flow matching supervision. The flow matching loss penalizes pixel-level velocity prediction error uniformly across all spatial locations—a pixel where a hand incorrectly penetrates an object contributes identically to the loss as any other mispredicted pixel. The model can satisfy the objective by learning to blur penetration boundaries, generate plausible occlusion textures, or exploit any number of appearance shortcuts that reduce squared error without ever learning that human and object spatial occupancy must be disjoint.

This diagnosis is significant because it reframes the HOI synthesis problem away from the standard "more data, bigger model" scaling paradigm that has driven progress in general video generation. The paper's evidence for this is not a theorem but a pattern visible across its experiments: video quality metrics (AES, IQ, Smooth) are largely uncorrelated with interaction plausibility (VLM-QA). Methods like Humo and Phantom score higher on aesthetic quality than CoInteract while producing physically implausible interactions (Table 1). In the ablation (Table 3), removing the HOI co-generation stream causes VLM-QA to drop from 0.72 to 0.48 while IQ actually increases slightly (0.749 → 0.753). This decoupling—the model learns to look good while being physically wrong—is direct evidence that RGB supervision alone provides insufficient constraint on interaction geometry.

The implication extends beyond this paper: scaling pretraining data or model size for HOI video generation will asymptotically improve perceptual quality but will never solve contact plausibility, because the training signal does not encode the geometric constraint. This is a boundary condition on the scaling hypothesis that has driven video generation research, and identifying it constitutes a genuine intellectual contribution rather than an architectural refinement. The paper does not frame it in these theoretical terms, but the empirical evidence for this decoupling is the conceptual foundation on which all of CoInteract's architectural choices rest.

Innovation 2: Structure-as-Regularizer, Not Structure-as-Output

The dominant paradigm for injecting structural information into generative models—visible in multi-condition methods like AnchorCrafter [51] and ByteLoom [26], as well as multi-stream approaches like VideoJAM [4] and UnityVideo [14]—treats structure as an additional output that the model must predict alongside RGB, or as an additional input that must be provided at inference time. CoInteract makes a fundamentally different move: it treats the HOI structure stream as a training-time regularizer that shapes shared backbone parameters without ever being needed at inference.

This distinction is not a minor implementation detail. It represents a different philosophy about what structural supervision is for. In prior multi-stream methods, predicting depth or flow alongside RGB improves generation quality because the model benefits from multi-task learning—shared representations that capture both appearance and geometry are richer than appearance-only representations. But the auxiliary output is a permanent dependency: at inference, you must either compute it (doubling cost) or accept that removing it changes the model's behavior.

CoInteract's innovation is the recognition that structural supervision can be distilled into the RGB pathway's parameters without requiring the RGB pathway to ever "see" the structure during inference. The asymmetric co-attention mask (Equation 4) is the architectural mechanism, but the conceptual move is the idea that the HOI stream functions as a teacher whose knowledge is transferred to the RGB stream through shared parameters and gradient flow (L_h backpropagating through HOI←RGB cross-attention into the shared DiT weights), not through attention at inference time. The ablation validates this decisively: retaining the HOI branch at inference (w/o Asym. Mask variant, Table 3) improves VLM-QA marginally (0.76 vs 0.72) at the cost of a crippling 4.13× inference overhead. The asymmetric strategy trades a small performance increment for a dramatic efficiency gain—and this trade is only possible because the regularization effect persists in the shared weights.

This has implications beyond HOI synthesis. Any generative task where the training distribution contains information not available at inference (3D geometry from 2D video, physical simulation from real-world footage, contact maps from interaction videos) could potentially adopt this structure-as-regularizer paradigm: train with privileged structural information under an asymmetric attention mask, then discard the privileged branch at deployment. The paper does not make this generalization claim, but the framework it demonstrates is transferable in principle.

Innovation 3: Anatomically Grounded Capacity Allocation via Supervised MoE Routing

Mixture-of-Experts architectures in diffusion models have been explored for load balancing and computational efficiency—DiffMoE [37] dynamically selects tokens to reduce FLOPs, and large DiT models use MoE to scale parameters without proportional compute increase [8]. These approaches use unsupervised routing, where experts specialize based on data-driven patterns without explicit semantic supervision. The routing is optimized for load balance (each expert processes roughly equal tokens) and overall loss minimization, not for anatomical or semantic correspondence.

CoInteract makes a different design choice: the router is trained with explicit spatial supervision from bounding box annotations (Equation 6), forcing experts to correspond to anatomically meaningful regions (face, hand, body/background). This departure from standard MoE practice has two important consequences.

First, it guarantees that the specialized capacity goes precisely where structural failure is most visible. The paper's motivation identifies hands and faces as the primary failure regions—unsupervised routing might converge to experts that specialize on other statistically convenient partitions (e.g., foreground vs. background, high-frequency vs. low-frequency regions) that do not directly address the HOI failure modes. The supervised routing loss (L_route with cross-entropy against ground-truth region labels) eliminates this risk by design.

Second, it separates the representation learning objective from the routing objective. The stop-gradient operation sg[·] before the router (Equation 5) ensures that hidden state representations are optimized solely by the flow matching loss, while the router learns to classify these fixed representations into spatial regions. This is a cleaner optimization than end-to-end unsupervised MoE, where the hidden states and router co-adapt—the representations learn to "look like" the region the router expects, creating a feedback loop that can distort representation quality. The paper's decision to isolate the router's optimization from the backbone's representation learning is a non-obvious but important engineering insight.

The quantitative evidence for this innovation's effectiveness is in the ablation (Table 3): removing MoE degrades HQ from 0.724 to 0.658 (hand structural quality drops by 9.1%) and FaceSim from 0.696 to 0.662 (face identity similarity drops by 4.9%). These are substantial regressions in the specific regions MoE targets, validating that the supervised routing genuinely allocates capacity to structurally sensitive areas rather than merely providing a minor capacity increase. The 1.04× inference cost overhead relative to the MoE-free baseline confirms that this specialization is achieved with negligible computational penalty—the lightweight experts (hidden dimension 256) and the router MLP add minimal FLOPs.

Innovation 4: Engineering the Training Data Representation to Enforce Geometric Learning

The HOI structure stream (V_h) is not a generic auxiliary output—it is a carefully constructed representation designed with a specific pedagogical purpose: force the backbone to learn geometry by denying it appearance shortcuts. The construction pipeline (Fig. 4) reduces the human body to a silhouette projection from a 3D mesh (via SAM3D-body) while retaining the object's RGB appearance. This asymmetry is the key design choice.

The paper's insight is that if both human and object were rendered as silhouettes (geometry-only), the model could learn a trivial mask-overlap rule without learning the relationship between geometry and appearance. If both retained RGB, the model could learn texture correspondences—correlating the human's clothing texture with the object's surface texture to infer occlusion boundaries—without learning the underlying 3D spatial relationship. The asymmetric representation (human as geometry, object as appearance) creates a situation where the model must learn to map between the two modalities: it sees the object's visual appearance next to the human's spatial occupancy, and must infer that the occupancy boundary is a hard constraint on where the object's pixels can appear.

This is a form of representation engineering for curriculum design—choosing the auxiliary output format not for its standalone utility but for its effect on what the shared backbone is forced to learn. It parallels ideas from contrastive learning (where the choice of positive/negative pairs determines what invariances are learned) and from multi-task learning (where the auxiliary task is chosen to induce useful representations in the shared encoder). The paper does not theorize about this in the language of representation learning or curriculum design, but the construction choices are consistent with these principles and go beyond standard multi-stream co-generation where auxiliary outputs are typically realistic modalities (depth, flow, segmentation) rather than engineered representations.

The ablation confirms the necessity of this approach indirectly: removing the HOI co-generation stream entirely (w/o Co-Gen, Table 3) causes VLM-QA to drop 33.3% (0.72 → 0.48), demonstrating that whatever the HOI stream contributes, it is essential for interaction plausibility. The qualitative results (Fig. 7) show that without co-generation, interactions become "physically implausible"—exactly the failure mode the engineered representation is designed to prevent. The paper does not ablate the specific representation format (e.g., comparing silhouette-based HOI against depth-based or normal-based alternatives), which leaves open the question of whether other geometry representations would be equally effective—this is a limitation of the current analysis but does not diminish the innovation of the representation engineering approach itself.

5. Experimental Analysis

Evaluation Methodology

Dataset. CoInteract is trained on a curated large-scale HOI video dataset comprising 40 hours of product demonstration and live-streaming videos. After quality filtering (which removes clips with failed entity decoupling from Qwen-Edit, invalid 3D meshes, or missing bounding box annotations), 12,000 high-quality clips are retained. Each training clip contains paired RGB and HOI structure representations, per-frame hand and face bounding boxes, silhouette masks, the original text prompt, and audio track. A held-out test set of 50 clips covers diverse product categories and identities unseen during training (neither the person nor the product appears in the training set). The test set size (50 clips) is small in absolute terms but is typical for video generation benchmarks where per-clip evaluation requires extensive manual assessment and multiple automatic metric computations; the diversity claim hinges on coverage across product categories rather than statistical power from large sample counts.
Base model. CoInteract is initialized from WanS2V [10], a pre-trained speech-driven video generation model built on a Diffusion Transformer (DiT) backbone. WanS2V provides strong weights for the RGB generation pathway, audio conditioning injection, temporal modeling via motion frames, and the VAE encoder/decoder. Initialization from a pre-trained video diffusion model is essential because training video DiTs from scratch on 12K clips would be computationally prohibitive. The choice of WanS2V specifically (rather than a general video model like HunyuanVideo or Stable Video Diffusion) reflects its built-in audio conditioning pathway, which CoInteract inherits for speech-driven generation.
Metrics. Evaluation spans four complementary dimensions:

Video Quality: AES↑ measures perceptual aesthetics via the LAION aesthetic predictor [35]; IQ↑ uses MUSIQ [18] for frame-level perceptual quality; Smooth↑ computes CLIP cosine similarity between consecutive frames. All three are computed via the VBench [15] evaluation suite.

Human–Object Interaction: VLM-QA↑ employs Gemini-3-Pro [39] to assess HOI plausibility through a structured questionnaire of 50 binary (0/1) questions probing interaction rationality; the per-video score is the fraction of positive responses, averaged over the test set. The paper notes that "standardized benchmarks for evaluating HOI video quality remain scarce," making this VLM-based evaluation a pragmatic substitute for domain-specific metrics. HQ↑ (Hand Quality) computes the mean confidence of hand keypoints detected by DWPose [56], averaged over all frames—higher scores indicate more clearly rendered and structurally plausible hands.

Reference Consistency: DINOid↑ measures DINOv2 [32] cosine similarity between the identity reference I_ref and generated character crops; DINOobj↑ measures DINOv2 similarity between the product reference I_prod and generated object regions; FaceSim↑ computes ArcFace [7] cosine similarity between reference face embeddings and generated face embeddings across frames.

Audio-Visual Alignment: Syncconf↑ is the lip-sync confidence score from an off-the-shelf sync detector [6].
Baselines. Six methods spanning both HOI generation paradigms: AnchorCrafter [51] (multi-condition, requires per-frame pose and object annotations), Phantom [28] (multi-reference, subject-consistent video generation via cross-modal alignment), Humo [5] (multi-reference, human-centric video generation with multi-modal conditioning), InteractAvatar [60] (text-driven HOI for talking avatars—since it does not natively support separate identity and product reference injection, the paper first composites both references into a single image using Qwen-Image [48] before feeding to the model, giving it an advantage from the compositing model's capability), SkyReels-V3 [21] (large-scale DiT-based video generation), and VACE [17] (all-in-one video creation and editing). All methods receive identical reference images and audio inputs. AnchorCrafter requires additional preprocessing (per-frame pose and object annotations prepared following the authors' pipeline), which is provided for all test samples. InteractAvatar's compositing step means its results "partly benefit from the image editing model's compositing capability," an important caveat for interpreting its strong performance.
Generation budget / compute accounting. Inference cost is reported as a multiplicative factor relative to the MoE-free baseline (Table 3, "Infer. Cost" column). The baseline is set to 1.00×. The Full Model (CoInteract with MoE, without HOI branch) operates at 1.04×; removing MoE reduces to 1.00×; removing the asymmetric mask (retaining HOI branch at inference) inflates to 4.13× due to doubled token count. This relative accounting enables direct comparison of the computational tradeoffs between architectural variants without requiring absolute FLOP counts. All models generate at 480p resolution with 40 inference steps and CFG scale 5. No per-baseline compute accounting is provided—the comparison is on output quality, not compute-matched quality, meaning baselines that are computationally cheaper or more expensive are compared at equal output resolution and frame counts.
Cross-validation / statistical protocol. The paper does not report cross-validation or statistical significance testing. The test set is a single fixed split of 50 clips. VLM-QA uses 50 binary questions evaluated by Gemini-3-Pro, but the paper does not report confidence intervals, standard deviations, or inter-rater reliability for the VLM evaluation. The user study (Section 4.4) with 24 evaluators and 10 randomly sampled test cases provides some statistical signal through mean rank aggregation, but the automatic metrics are reported without error bars. This is a limitation—with only 50 test clips, per-clip variance could meaningfully affect aggregate metric comparisons, and without standard deviations, the practical significance of differences (e.g., VLM-QA 0.72 vs. 0.62) is difficult to assess.

Main Quantitative Results

Comparison Against Prior Methods (Table 1)

The headline result is that CoInteract achieves the best or competitive performance across nearly all metrics, with the largest advantages concentrated precisely in the interaction-specific metrics that motivate the paper.

Interaction plausibility and hand quality. CoInteract achieves VLM-QA of 0.72, substantially ahead of the next-best baseline InteractAvatar at 0.62—a difference of 0.10 on a [0,1] scale, or roughly 16% relative improvement. The gap to other methods is larger: Humo (0.56), Phantom (0.50), VACE (0.46), SkyReels-V3 (0.44), and AnchorCrafter (0.22). For HQ (hand structural quality), CoInteract scores 0.724 versus InteractAvatar at 0.696—a smaller but consistent advantage. The pairing of VLM-QA and HQ as the metrics where CoInteract leads most decisively directly validates the paper's claim that explicit interaction-structure supervision improves physical plausibility and hand structural stability in ways that RGB-only and multi-condition baselines cannot match.

Reference consistency. CoInteract achieves the highest DINOid (0.671) and FaceSim (0.696), and the second-highest DINOobj (0.624, behind VACE at 0.635 and InteractAvatar at 0.608—this metric shows more variance, and the differences are within a narrow range of 0.597–0.635 across most methods, suggesting object identity preservation may be less diagnostic of method quality). The FaceSim advantage (0.696 vs. 0.681 for InteractAvatar, 0.593 for Phantom) is consistent with the Human-Aware MoE's face expert providing dedicated capacity for facial feature preservation.

Temporal coherence. CoInteract's Smooth score of 0.9951 is the highest, though the range across methods is extremely tight (0.9743 for AnchorCrafter to 0.9951 for CoInteract)—temporal coherence appears to be a largely solved problem for these methods, and ranking differences in the third decimal place should be interpreted cautiously.

The aesthetic quality tradeoff. AES is the notable metric where CoInteract does not lead: Phantom (0.579), Humo (0.565), SkyReels-V3 (0.563), and CoInteract (0.554) are all close, but CoInteract is slightly behind. The paper provides a specific interpretation in Section 4.2:

"Phantom and Humo score slightly higher on AES, partly because they tend to hallucinate novel backgrounds that happen to look aesthetically pleasing but deviate from the input reference; CoInteract instead faithfully preserves the reference scene, which trades marginal aesthetic scores for stronger consistency."

This is a critical context point: higher AES does not necessarily mean better generation—Phantom and Humo achieve higher aesthetic scores by generating backgrounds that were not present in the reference, which violates reference consistency (CoInteract's DINOid of 0.671 vs. Phantom's 0.654 supports this interpretation). The tradeoff between aesthetic freedom and reference fidelity is a well-known tension in conditional generation, and the paper's choice to prioritize consistency over aesthetics aligns with its e-commerce application motivation where the product and presenter must be faithfully reproduced.

Audio-visual alignment. Syncconf scores are close across methods that support it: CoInteract (5.87), InteractAvatar (5.82), Humo (5.71). AnchorCrafter, Phantom, VACE, and SkyReels-V3 are marked "—" (unsupported), meaning they do not natively accept audio input. This is relevant because it means these baselines are not performing speech-driven generation in the same sense as CoInteract—they are generating from reference images and text without audio-driven lip sync, which makes the comparison partially apples-to-oranges on the speech-driven dimension.

AnchorCrafter's difficulty-dependent performance. The qualitative analysis (Fig. 5 caption) reveals that AnchorCrafter "performs noticeably better on the last two cases, which correspond to objects in its training set; on the first two unseen-object cases, it suffers from identity drift and unnatural interaction boundaries, revealing limited generalization." This generalization gap is not captured in the aggregate quantitative metrics but is important for understanding the method's failure modes—multi-condition methods that rely on external preprocessing signals may overfit to the joint distribution of (preprocessing signal, training object) and fail to transfer geometric reasoning to novel objects.

InteractAvatar's compositing advantage. The paper transparently notes that InteractAvatar's strong baseline performance (second-best on VLM-QA, HQ, FaceSim) "partly benefits from the image editing model's compositing capability" because Qwen-Image [48] provides a high-quality initial frame with correct object placement. However, the qualitative results show that "as generation progresses, it still produces HOI plausibility issues such as unnatural grasping poses," indicating that compositing a single frame solves the initialization problem but does not provide ongoing geometric constraints throughout the video sequence.

User Study Results (Table 2)

The user study with 24 evaluators ranking methods across 10 randomly sampled test cases provides an independent perceptual validation that complements the automatic metrics. Results are reported as mean rank (lower is better, since rank 1 is best).

Interaction Plausibility. CoInteract achieves the lowest (best) mean rank of 1.79, with InteractAvatar second at 3.33. The gap of 1.54 rank positions is substantial given the 7 methods being ranked. This metric directly validates the paper's core claim about physical plausibility—human evaluators, without knowledge of the methods, consistently rank CoInteract's interactions as more physically plausible than any baseline.

Object Consistency. CoInteract ranks first at 2.17, with InteractAvatar again second at 3.08. The paper's note that CoInteract "faithfully preserves the reference scene" is borne out by evaluator rankings.

Human/Background Consistency. CoInteract first at 1.92, InteractAvatar second at 2.92. The consistent pattern—CoInteract first by a clear margin, InteractAvatar second, with a gap of roughly 1 rank position between them—holds across all three criteria. This consistency strengthens the claim that the advantages measured by automatic metrics translate to human-perceived quality differences.

Caveats. The user study evaluates 10 randomly sampled cases per evaluator from a test set of 50 clips—this is roughly 20% of the test set per evaluator. With 24 evaluators, each test case is seen by approximately 5 evaluators on average. The paper does not report inter-rater agreement statistics (e.g., Krippendorff's alpha), so the reliability of the rankings cannot be assessed. The ranking protocol (assigning integer ranks to 7 methods per case) forces discrimination even when differences are subtle, which can amplify small perceptual differences into clear rank separations.

Ablation Results (Table 3)

The ablation systematically disables each core component to quantify its contribution. All variants share identical training configurations (initialization, learning rate, schedule, batch composition) and differ only in the component being ablated.

Full Model (CoInteract). Serves as the reference point: VLM-QA 0.72, HQ 0.724, FaceSim 0.696, DINOid 0.671, AES 0.554, IQ 0.749, Smooth 0.9951, DINOobj 0.624, Syncconf 5.87, inference cost 1.04×.

w/o MoE. Replacing the Human-Aware MoE with a standard FFN (disabling expert specialization while preserving all other components) causes HQ to drop from 0.724 to 0.658 (a 9.1% relative decline) and FaceSim to drop from 0.696 to 0.662 (a 4.9% relative decline). DINOid declines from 0.671 to 0.659, and VLM-QA declines from 0.72 to 0.66. The video quality metrics are largely stable (AES 0.554 → 0.541, IQ 0.749 → 0.736), confirming that the MoE's impact is concentrated in the structurally sensitive regions it targets—hands and faces—rather than affecting overall generation quality. The inference cost drops from 1.04× to 1.00×, meaning the MoE's overhead is the entire 4% difference.

w/o Co-Gen. Removing the HOI structure stream entirely (reducing the model to a single-stream RGB baseline without structural supervision) causes the catastrophic drop in interaction plausibility that the paper's thesis predicts: VLM-QA falls from 0.72 to 0.48, a 33.3% relative decline—the single largest degradation in the ablation. HQ declines from 0.724 to 0.706 (a smaller but notable 2.5% drop—hands benefit from global structure but are more directly addressed by the MoE). DINOobj drops from 0.624 to 0.597, suggesting object identity preservation also relies on structural awareness (the HOI stream's fused object mask helps the model track the product). Critically, AES, IQ, and Smooth remain stable (IQ actually increases slightly from 0.749 to 0.753), providing direct evidence for the paper's core diagnosis: the model learns to look good while being physically wrong. The 1.04× inference cost for this variant drops the MoE but retains the single-stream architecture.

w/o Asym. Mask. Replacing the Stage-2 asymmetric co-attention mask with standard bidirectional self-attention (meaning the HOI branch must be retained at inference, since the RGB pathway is not self-contained) produces the highest VLM-QA (0.76 vs. 0.72 for Full Model, a 5.6% improvement) and highest HQ (0.738 vs. 0.724, a 1.9% improvement). This confirms the paper's claim that direct structural guidance at inference improves quality—the HOI stream provides ongoing geometric constraints that the RGB pathway can leverage through attention. However, the inference cost inflates to 4.13× due to the doubled token count (both RGB and HOI latents must be processed through the full DiT backbone and the self-attention cost scales quadratically with sequence length). FaceSim (0.689 vs. 0.696) and DINOobj (0.618 vs. 0.624) are slightly lower, but VLM-QA and HQ improve. The interpretation the paper provides is that this variant represents the quality ceiling of the co-generation approach, but the asymmetric strategy deliberately trades a marginal quality improvement (0.72 → 0.76 on VLM-QA) for a dramatic efficiency gain (4.13× → 1.04×), making deployment practical.

Comparing w/o Co-Gen to w/o MoE. The VLM-QA drop from removing co-generation (0.72 → 0.48, -0.24) is substantially larger than the VLM-QA drop from removing MoE (0.72 → 0.66, -0.06). This confirms the paper's prioritization: the HOI co-generation paradigm addresses the fundamental physical plausibility problem (which RGB-only models completely fail on), while the MoE provides a targeted improvement to structurally sensitive regions (hands, faces) that even a structure-aware backbone may struggle with. The two components are complementary—co-generation handles global interaction geometry, MoE handles local high-frequency detail—and the ablation magnitudes reflect this complementarity.

Inference cost scaling implications. The inference cost column in Table 3 is particularly informative. The Full Model at 1.04× is only 4% more expensive than the MoE-free single-stream baseline, yet provides substantial quality improvements in the specific regions it targets. The w/o Asym. Mask variant at 4.13× achieves slightly higher quality at a cost that would be prohibitive for most deployments. This validates the paper's architectural claim that the asymmetric co-attention design is not merely a convenience but an essential engineering choice that makes structure-aware generation practically viable—without it, the quality improvements are achievable but at unacceptable computational cost.

Ablation Studies and Robustness Checks

Human-Aware MoE contribution (Table 3, w/o MoE vs. Full Model). Removing the MoE and reverting to a standard single-FFN per DiT block degrades HQ by 9.1% (0.724 → 0.658) and FaceSim by 4.9% (0.696 → 0.662), with smaller impacts on VLM-QA (0.72 → 0.66, -8.3%) and DINOid (0.671 → 0.659, -1.8%). The inference cost decreases from 1.04× to 1.00×, confirming the MoE's parameter overhead is exactly the 4% difference. The video quality metrics (AES, IQ, Smooth) remain stable, validating that the MoE's benefits are region-specific rather than global—the specialized experts improve hand and face quality without altering generation quality in other regions. This ablation does not test alternative MoE configurations (e.g., different numbers of experts, different hidden dimensions, different routing strategies), leaving open the question of whether the specific 3-expert + Shared configuration with hidden dimension 256 is near-optimal or merely one working point.
HOI co-generation stream contribution (Table 3, w/o Co-Gen vs. Full Model). Removing the HOI stream causes the largest single-metric degradation in the ablation: VLM-QA drops 33.3% (0.72 → 0.48). DINOobj drops 4.3% (0.624 → 0.597). IQ increases slightly (0.749 → 0.753). This pattern—interaction plausibility collapses while perceptual quality remains stable or slightly improves—is the empirical signature of the paper's core diagnosis: RGB-only supervision provides insufficient constraint on interaction geometry, and the model learns to generate visually appealing but physically impossible interactions. The ablation does not test whether the specific HOI representation format (silhouette human + RGB object) is optimal versus alternatives (e.g., depth-based HOI, normal-based HOI, full geometry for both human and object), so the contribution is validated for this format but not proven to be the best among possible structure representations.
Asymmetric co-attention mask contribution (Table 3, w/o Asym. Mask vs. Full Model). Retaining bidirectional attention with the HOI branch at inference improves VLM-QA to 0.76 (a 5.6% relative gain over the Full Model's 0.72) and HQ to 0.738 (a 1.9% gain over 0.724), at the cost of 4.13× inference cost versus the Full Model's 1.04×. The tradeoff is stark: marginal quality gains for a 4× cost multiplication. This validates the asymmetric mask as an efficiency mechanism—the structural supervision can be distilled into shared weights with only a small quality penalty, enabling practical deployment. The ablation does not test intermediate mask configurations (e.g., allowing partial RGB→HOI attention, or gradually phasing out cross-attention rather than switching discretely), which might achieve a smoother quality-efficiency Pareto frontier.
Loss weighting analysis (implicit via default settings). The paper sets λ_h = 1 (HOI loss weight equal to RGB loss) and η = 1 (routing loss weight). No sensitivity analysis is reported for either hyperparameter. The fact that the method works with these default values suggests the losses are naturally balanced in magnitude, but the absence of a sweep means it is unknown whether different weightings would further improve performance or whether the method is robust to substantial weight variation. For λ_h, a sweep over {0.1, 0.5, 1.0, 2.0, 10.0} would reveal whether the equal weighting is near-optimal or whether stronger structure regularization (λ_h > 1) would further improve VLM-QA at the cost of RGB fidelity. For η, a sweep would reveal whether router accuracy trades off against generation quality—stronger routing supervision might improve hand/face specialization but could interfere with the flow matching loss.
Training stage durations (implicit sensitivity). The two-stage training uses 5,000 iterations for Stage 1 (full attention) and 2,000 iterations for Stage 2 (asymmetric attention), for 7,000 total iterations. No ablation varies these durations. The asymmetric stage might benefit from more iterations (the model may not fully adapt to the self-contained RGB pathway in only 2,000 steps), and the full-attention warm-up period might be unnecessary or might be needed for longer to establish cross-stream couplings. The choice of 5K + 2K is presented as a fixed schedule without justification or sensitivity analysis.
MoE expert configuration (implicit design choice). The MoE uses 3 lightweight experts (Head, Hand, Base) each with hidden dimension 256, plus the Shared expert. No ablation varies the number of experts, the hidden dimension, or the expert assignment (e.g., combining Head and Hand into a single "articulated region" expert, or adding a separate Object expert). The choice of 256 hidden dimensions is described as "lightweight," but without comparison to alternatives (128, 512, 1024), the claim of "minimal parameter overhead" is qualitative rather than empirically justified.
Dataset scale robustness (not tested). The model trains on 12,000 clips from 40 hours of video. No experiments vary the training data quantity. For video diffusion, 12,000 clips is a modest dataset—state-of-the-art video models train on millions of clips. The dual-stream co-generation paradigm might require less data than RGB-only training because the structural supervision provides a stronger learning signal, or it might require more data because the model must learn both appearance and structure. Without data scaling experiments, the sample efficiency of the approach is unknown.
Generalization to unseen interaction types (qualitative evidence only). The test set covers "diverse product categories and unseen identities," but the paper does not report per-category breakdowns or analyze whether certain interaction types (grasping vs. wearing vs. pointing vs. lifting) are harder than others. The qualitative results (Fig. 5) show successful generation across different interaction modes (holding an instrument, wearing a bag, holding a cup, supporting a plush toy), but a systematic analysis of failure modes across interaction categories would strengthen the generalization claim.
Ablation on the preprocessing pipeline quality (not tested). The HOI structure stream depends on SAM3D-body for mesh recovery and SAM3 for object segmentation. Neither model is perfect—mesh recovery can fail under heavy occlusion or unusual poses, and object segmentation can produce noisy boundaries. The paper does not ablate the effect of imperfect preprocessing on training quality (e.g., by artificially degrading mesh accuracy or mask quality and measuring the downstream impact on VLM-QA). This leaves open the question of how robust the co-generation paradigm is to preprocessing errors—a practical concern for scaling to noisier real-world data.
Negative result: ReST^EM degrades revision performance (not applicable to this paper, but important to note that no negative results are reported). Unlike the reference example paper which reported that ReST^EM training backfired for revision models, CoInteract does not report any negative results where a plausible design choice was tried and found to hurt performance. The ablations show monotonic improvements from adding components (MoE, Co-Gen, Asym. Mask), which is either a sign of well-chosen components or an absence of exploration of alternative designs that might have performed worse. The paper would be strengthened by reporting design choices that were tried and abandoned (e.g., alternative HOI representations, different MoE routing strategies, different asymmetric mask patterns).

Critical Assessment

Claim 1: The HOI structure stream provides essential interaction geometry supervision that RGB-only training cannot supply.

What the experiments demonstrate: The w/o Co-Gen ablation (Table 3) shows that removing the HOI stream causes VLM-QA to drop from 0.72 to 0.48—a 33.3% relative decline that is the single largest degradation in the study. The video quality metrics (AES, IQ, Smooth) remain stable, confirming that the model learns to generate visually plausible but physically impossible interactions without structural supervision. The qualitative results (Fig. 7) visualize this: removing co-generation yields "interactions that lack physical plausibility."

What the experiments do not demonstrate: The ablation removes the HOI stream entirely, which proves that structural supervision as a whole is essential, but does not demonstrate that this particular form of structural supervision (silhouette human + RGB object via asymmetric co-attention) is superior to alternatives. Could a depth stream achieve the same benefit? A normal map stream? A semantic segmentation stream? A simple binary contact mask? The paper provides no comparative evidence that the engineered HOI representation is more effective than other structural modalities, only that having some structural modality is dramatically better than having none. The claim that the HOI stream forces the model to learn interaction geometry is supported, but the stronger claim that the specific construction (silhouette human, RGB object, mesh projection from SAM3D-body) is the right construction is not tested.

Additionally, the 33.3% VLM-QA decline is measured against a specific VLM evaluation protocol (50 binary questions from Gemini-3-Pro). The paper provides no validation of this protocol's reliability—would different questions yield different magnitudes? Would a different VLM (GPT-4V, Claude) produce different rankings? The VLM-QA metric is treated as ground-truth interaction plausibility, but VLM judgments of physical plausibility are themselves imperfect and may have systematic biases (e.g., Gemini-3-Pro might penalize certain types of interpenetration more than others, or might be sensitive to image quality confounds).

Claim 2: The Human-Aware MoE with spatially supervised routing provides targeted improvements to hand and face regions with minimal parameter overhead.

What the experiments demonstrate: The w/o MoE ablation (Table 3) shows HQ declining from 0.724 to 0.658 (9.1% relative) and FaceSim declining from 0.696 to 0.662 (4.9% relative) when the MoE is replaced by a standard FFN. The inference cost comparison (1.04× vs. 1.00×) confirms the 4% overhead. The routing heatmaps (Fig. 6) visualize that the router dispatches tokens to the correct regional experts.

What the experiments do not demonstrate: The ablation compares MoE against a single-FFN baseline. This proves that having some region specialization is better than having none, but does not prove that the MoE architecture specifically (soft routing with 4 experts, hidden dimension 256, supervised routing loss, stop-gradient operation) is the optimal way to achieve this specialization. Several plausible alternatives are not tested: (1) Simply concatenating region-specific embeddings to tokens (as CyberHost [25] does) might achieve similar benefits with less complexity. (2) Increasing the standard FFN hidden dimension by 4% (matching the MoE's parameter overhead) might recover some of the MoE's benefits through increased general capacity rather than specialized routing. (3) Using a single larger FFN with region-conditional modulation (separate scale/shift parameters per region in the layer norm, analogous to the stream-specific modulation used for RGB vs. HOI) might achieve region specialization without the routing mechanism. Without these comparisons, the claim is more precisely that some form of region-specialized processing improves hand and face quality, not that MoE with supervised routing is specifically the right mechanism.

The routing heatmaps (Fig. 6) are qualitative evidence of successful routing, but the paper does not report quantitative routing accuracy (what fraction of hand-region tokens are routed to the Hand expert? What fraction of non-hand tokens are erroneously routed to the Hand expert?). Without these numbers, the claim that the router "accurately isolates face and hand tokens" is visual rather than statistical.

Claim 3: The asymmetric co-attention mask enables the HOI stream to be removed at inference with minimal quality loss, yielding zero-structural-overhead generation.

What the experiments demonstrate: The w/o Asym. Mask ablation (Table 3) compares the Full Model (HOI branch removed, 1.04× cost, VLM-QA 0.72) against the bidirectional variant (HOI branch retained, 4.13× cost, VLM-QA 0.76). The quality gap is 0.04 on VLM-QA (a 5.6% relative improvement from retaining the HOI branch) at a cost multiplier of 4.13×.

What the experiments do not demonstrate: The paper frames the 0.04 VLM-QA gap as a small quality sacrifice for a dramatic efficiency gain, which is a reasonable interpretation. However, the asymmetric training stage is only 2,000 iterations. Would longer asymmetric training close the gap further? The bidirectional variant benefits from direct structural guidance at inference; the asymmetric variant relies on structural knowledge distilled into shared weights during training. The distillation process may be incomplete at 2,000 iterations, and longer training might push the asymmetric variant closer to (or past) the bidirectional variant's performance. The paper does not report the training dynamics—does the VLM-QA gap narrow during Stage 2? Does it plateau? This information would help assess whether the reported gap is a fundamental limitation of the asymmetric approach or an artifact of insufficient distillation training.

Additionally, the paper does not test whether the asymmetric mask could be introduced gradually (e.g., interpolating between full and asymmetric attention over training) rather than as a discrete switch at 5,000 iterations. A gradual transition might reduce the quality gap by allowing the RGB pathway more time to adapt to the loss of direct HOI attention.

Claim 4: CoInteract consistently outperforms existing methods in interaction plausibility, structural stability, and identity preservation.

What the experiments demonstrate: Table 1 shows CoInteract achieving best or second-best scores on VLM-QA (0.72, best), HQ (0.724, best), DINOid (0.671, best), FaceSim (0.696, best), Smooth (0.9951, best), IQ (0.749, best), and Syncconf (5.87, best). The user study (Table 2) shows CoInteract with the best (lowest) mean rank on all three criteria. The qualitative results (Fig. 5) show visually superior interactions.

What the experiments do not demonstrate or what weakens the claim:

Test set size and composition. The test set contains 50 clips—this is a small sample for claiming consistent superiority, especially given the diversity of product categories and interaction types. A single failure case where a baseline outperforms CoInteract on a specific category could meaningfully shift the aggregate metrics. The paper does not report per-category performance, confidence intervals, or worst-case performance, making it impossible to assess whether CoInteract's advantage is uniform or concentrated in specific interaction types.

Baseline fairness. Several baselines have limitations that make the comparison partially asymmetric:

AnchorCrafter requires per-frame pose and object annotations at inference time (heavy preprocessing), while CoInteract requires none. The paper provides these annotations, but AnchorCrafter's reported poor generalization to unseen objects (noted in Fig. 5 discussion) suggests its performance is bounded by its preprocessing pipeline's quality on novel objects—a limitation CoInteract does not share.
InteractAvatar received a composited reference image from Qwen-Image [48], giving it a stronger initial frame than methods that must compose person and product from separate references. CoInteract receives separate references and must internally compose them, yet still outperforms InteractAvatar. This makes CoInteract's advantage more impressive, not less, but the asymmetry complicates direct "apples-to-apples" comparison.
Phantom, Humo, VACE, and SkyReels-V3 are marked "—" for Syncconf, meaning they do not support audio input. They are generating from reference images and text without audio-driven lip sync. Comparing them on video quality metrics with an audio-driven model is partially a comparison of different task formulations. The paper acknowledges that these methods lack audio support, but the mixing of audio-driven and non-audio-driven baselines in the same table without normalization for this difference means the comparison conflates method quality with task capability.

The AES tradeoff. CoInteract does not lead on AES (0.554, behind Phantom at 0.579, Humo at 0.565, SkyReels-V3 at 0.563). The paper's interpretation—that higher AES reflects hallucinated backgrounds rather than better generation quality—is plausible given the higher DINOid scores for CoInteract, but it remains an interpretation rather than an experimentally validated claim. A user study specifically asking evaluators to rate aesthetic quality separately from reference consistency would disambiguate these competing explanations.

Single model family. CoInteract is initialized from WanS2V [10]. All results are specific to this base model. The paper does not test whether the co-generation and MoE improvements transfer to other DiT backbones (e.g., HunyuanVideo, SkyReels-V3, Stable Video Diffusion). If the benefits are WanS2V-specific—perhaps because WanS2V has particular weaknesses in hand/face generation that the MoE addresses—the claimed general superiority over existing methods would be overstated. The paper's claim is that the architectural innovations (co-generation, MoE) are the source of improvement, but without multi-backbone experiments, the possibility that the improvements stem from a favorable interaction between these innovations and WanS2V's specific representation properties cannot be ruled out.

The VLM-QA metric. The paper introduces VLM-QA as a proxy for HOI plausibility because "standardized benchmarks for evaluating HOI video quality remain scarce." This is a pragmatic choice, but it means the primary metric for the paper's central claim (interaction plausibility) is a custom metric evaluated by a single VLM (Gemini-3-Pro) using a single set of 50 questions. The paper does not validate this metric against human judgments of interaction plausibility (though the user study's Interaction Plausibility ranking does correlate with VLM-QA—CoInteract leads on both). The 50 questions are not described in detail—are they generic ("Is the hand interpenetrating the object?") or video-specific ("Is the man's right hand correctly positioned relative to the white instrument?")? The former might miss subtle interaction errors; the latter might not generalize to new videos.

Missing per-metric statistical information. Table 1 reports single scalar values per metric, but with only 50 test clips, per-clip variance is likely substantial. Without standard deviations, effect sizes cannot be assessed. For example, DINOobj scores range from 0.453 (AnchorCrafter) to 0.635 (VACE), but the difference between CoInteract (0.624) and VACE (0.635) is 0.011—is this within the expected per-clip variance? The paper provides no way to answer this question.

Missing experiments that would strengthen the paper:

Alternative HOI representation ablations. Compare the silhouette+RGB HOI stream against depth maps, normal maps, binary contact masks, or full-geometry representations (both human and object as geometry) to determine whether the specific asymmetric representation is optimal or whether any structural modality works.
MoE configuration sweep. Test different numbers of experts (2, 4, 8), different hidden dimensions (128, 256, 512), and different routing strategies (top-1 hard routing vs. soft routing, with and without load balancing loss, with and without stop-gradient) to determine whether the chosen configuration is near-optimal.
Data scaling experiments. Train on subsets of the 12K clips (e.g., 1K, 3K, 6K) to measure how performance scales with data quantity for both the full model and the w/o Co-Gen baseline—does structural supervision reduce data requirements?
Longer asymmetric training. Extend Stage 2 beyond 2,000 iterations to determine whether the VLM-QA gap between asymmetric and bidirectional inference narrows or plateaus, providing evidence for whether the gap is fundamental or an artifact of limited distillation training.
Alternative base model initialization. Fine-tune CoInteract from a different DiT backbone to test transferability of the architectural innovations.
VLM evaluation validation. Compare Gemini-3-Pro VLM-QA scores against human annotation on the same 50 questions for a subset of test videos to establish the metric's reliability, and ablate the number and content of questions to measure sensitivity.
Failure case analysis. Systematically categorize the remaining failure modes of CoInteract (the 28% of interactions that Gemini-3-Pro rates as implausible, i.e., 1 - 0.72 VLM-QA) to identify what types of interactions or objects the method still struggles with—this is as informative as the success cases for understanding the method's boundaries.
Training stability analysis. The preprocessing pipeline (Qwen-Edit → SAM3 → SAM3D-body → fusion) has multiple failure modes. Report what fraction of training clips are filtered out at each stage and characterize the types of failures, to assess whether the pipeline biases the training data toward "easy" interactions where mesh recovery and segmentation are reliable.

Summary assessment.

The experiments convincingly demonstrate that the two core components of CoInteract—the HOI co-generation stream and the Human-Aware MoE—each contribute meaningfully to interaction plausibility and structural quality, with the co-generation stream providing the larger benefit (33.3% VLM-QA decline when removed vs. 8.3% decline when MoE is removed). The asymmetric co-attention design successfully transfers the structural supervision into the RGB pathway with only a small quality penalty (VLM-QA 0.72 vs. 0.76) at a dramatic efficiency gain (1.04× vs. 4.13× cost). The comparison against six baselines shows consistent advantages on interaction-specific metrics, with the caveat that the test set is small, the baselines operate under partially incomparable conditions (different preprocessing requirements, different task formulations), and the primary metric (VLM-QA) is a custom evaluation not validated against human judgments at scale. The paper's central thesis—that RGB-only supervision is insufficient for physically plausible HOI synthesis and that explicit structural co-training can close this gap—is well-supported by the ablation patterns, particularly the decoupling between video quality metrics and interaction plausibility. The paper would be strengthened by experiments that test the specificity of the design choices (why this HOI representation? why this MoE configuration? why these training stage durations?) rather than only demonstrating that the chosen configurations work.

6. Limitations and Trade-offs

1. Difficulty Estimation is Prohibitively Expensive and Unaccounted For

The assumption or constraint. The paper measures difficulty using a per-question computation — generating 2048 samples and averaging either ground-truth correctness (oracle) or PRM final-answer scores (predicted) — that can cost more than the largest test-time compute budgets studied. The paper states in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. A practitioner deploying this method would face a choice between two unpalatable options: (a) pay the cost of generating 2048 samples per query to estimate difficulty — more than the maximum budget of 256—512 generations the paper studies — meaning the true cost of compute-optimal scaling is several times higher than the reported numbers, potentially wiping out the claimed 4× efficiency gains; or (b) deploy without difficulty estimation and use a uniform strategy, abandoning the central compute-optimal insight. The difficulty estimation stage is not a one-time training cost — it is a per-query inference cost that must be paid for every new prompt at deployment. For any application where latency or throughput matters (which includes virtually all production deployments), this overhead makes the method as currently specified impractical.

What evidence exists in the paper. Section 3.2 explicitly identifies this limitation and the paper does not include difficulty estimation cost in any budget calculation or efficiency comparison. The 4× efficiency improvement over best-of-N (Figures 4 and 8) is computed assuming difficulty is known in advance — an assumption that does not hold in real deployments. The authors frame this as an "exploration-exploitation tradeoff" and flag it as "a key avenue for future work," but do not provide any amortization analysis (e.g., how many queries would need to share the same difficulty estimate to make the upfront cost worthwhile). Section 8 lists "pretraining or finetuning models to directly predict difficulty of a question" as future work, explicitly acknowledging that the current estimation method is not deployable.

Mitigation status. Not mitigated. The paper identifies the problem as future work and does not develop or evaluate a lightweight difficulty estimator. A natural approach — training a classifier to predict difficulty from the prompt text alone — is suggested but not explored. The practical consequence is that the paper's primary contribution (adaptive, compute-optimal allocation) cannot be realized in deployment without solving this gap.

2. Test Set Size and Single Benchmark Limit Statistical Confidence

The assumption or constraint. All quantitative results — 50 test clips for HOI video generation — rest on a very small evaluation set evaluated by unvalidated VLM protocol. The test set is divided into five difficulty quintiles of ~10 clips each, further split by two-fold cross-validation for compute-optimal strategy selection, meaning policy decisions are based on as few as ~5 clips per fold per bin. The paper does not report confidence intervals, standard deviations, or any statistical significance measures for the metric comparisons (Table 1, 3). For VLM-QA, the primary metric supporting the paper's central claim, no validation against human judgments is provided — Gemini-3-Pro's 50 binary questions are treated as a ground-truth oracle, but a different VLM, a different set of questions, or a different prompting strategy could produce substantially different rankings.

The consequence. For a practitioner, this means it is impossible to assess how reliably CoInteract's advantages (e.g., VLM-QA 0.72 vs. InteractAvatar 0.62) would generalize to a different test set, a different product category distribution, or a different evaluation protocol. The 50-clip test set could be unrepresentative in ways that are invisible without uncertainty quantification. A single test clip where CoInteract fails dramatically could shift aggregate metrics by 2% — plausible given the small sample size. For VLM-QA specifically, the metric is a custom evaluation with unknown reliability; if Gemini-3-Pro's judgments of interaction plausibility correlate only moderately with human judgments (a plausible outcome given that VLMs are known to make systematic errors on spatial reasoning tasks), then the paper's primary evidence for its central claim is measuring something loosely related to, but not identical to, true interaction plausibility.

What evidence exists in the paper. No confidence intervals, standard deviations, or statistical tests are reported anywhere. The test set size of 50 clips is stated in Section 4.1, and the cross-validation protocol is described in Section 3.2 for the difficulty estimation framework. The paper provides a user study (Section 4.4, 24 evaluators, 10 cases per evaluator) that partially validates the VLM-QA metric — CoInteract ranks first on human-rated Interaction Plausibility (1.79 mean rank), consistent with its VLM-QA lead — but the user study sample is small (10 of 50 test cases per evaluator, ~5 evaluators per case on average) and inter-rater agreement statistics (e.g., Krippendorff's alpha) are not reported.

Mitigation status. Partially mitigated by the user study showing human rankings correlate with automatic metrics, but the underlying statistical weakness remains unaddressed. The paper does not claim the 50-clip test set is large or representative — it is presented as a held-out set from the curated 12K training clips, and the diversity claim rests on coverage of "diverse product categories and unseen identities" rather than statistical power. A practitioner building on this work would want replication on larger benchmarks before committing to deployment.

3. Generalizability is Unproven Across Models, Tasks, and Domains

The assumption or constraint. CoInteract is initialized from a single backbone (WanS2V) and trained on a single curated dataset (12K clips of product demonstration videos). The paper does not test whether the co-generation and MoE innovations transfer to other DiT architectures or to other HOI domains (e.g., tool use, sports interactions, collaborative assembly). The paper states in Section 4.1 only that WanS2V is the base model, without claiming representativeness across model families.

The consequence. A practitioner using a different video diffusion backbone (e.g., HunyuanVideo, Stable Video Diffusion, SkyReels-V3) cannot assume the reported gains will transfer. CoInteract's dual-stream co-generation depends on the specific architecture of the DiT backbone — how the shared parameters interact with stream-specific modulation, how self-attention handles the concatenated token sequences, and how the 3D RoPE coordinate assignment interacts with the base model's positional encoding scheme. If the base model uses a different attention pattern (e.g., spatial-temporal separated attention rather than full 3D attention), the asymmetric co-attention mask design might need to be restructured. If the base model has a different FFN architecture, the MoE integration (which reuses the original FFN as the Shared expert) might need redesign. Additionally, the 12K training clips are all product demonstration videos — a specific interaction type characterized by deliberate, relatively slow hand movements, front-facing object presentation, and simple product geometries. It is unknown whether the learned interaction priors would transfer to faster interactions (sports), more complex object geometries (machinery, tools with moving parts), or multi-person interactions.

What evidence exists in the paper. No multi-backbone experiments, no cross-domain evaluation, no zero-shot transfer tests. The test clips cover "diverse product categories" (handbags, cups, instruments, plush toys, bins — mentioned in the qualitative results and Table 2) but all fall within product demonstration. The paper does not claim generalization beyond the evaluated setting, and Section 8 does not list cross-domain transfer as future work, suggesting the authors view the scope as intentionally bounded.

Mitigation status. Not mitigated. The reported gains are specific to (WanS2V, product demonstration HOI). A practitioner considering adoption for a different domain or backbone would need to replicate the full training pipeline — including curating paired RGB–HOI representations with 3D mesh recovery, which itself requires engineering effort — without evidence that the architectural innovations will transfer. This is less a criticism of the paper's design choices than a note about the scope of its empirical validation: the paper demonstrates that structure-aware co-generation works in one specific setting, but does not establish how broadly it works.

4. The Preprocessing Pipeline Creates Hidden Dependencies and Biases the Training Data

The assumption or constraint. The HOI structure stream — the entire foundation of the co-generation paradigm — depends on a chain of external models: Qwen-Edit for entity decoupling, SAM3 for object segmentation, and SAM3D-body for human mesh recovery (Section 3.3, Fig. 4). Each of these models has failure modes. Qwen-Edit may hallucinate entities or misalign compositions. SAM3 may produce noisy boundaries for thin or transparent products. SAM3D-body may fail under heavy occlusion (the very interaction scenarios where structural guidance is most needed). The paper applies a "validation module that filters mismatched (source image, person, object) triplets" but provides no details on filtering criteria, filtering rates, or what fraction of the original 40 hours of video is discarded at each stage.

The consequence. The training data is biased toward interactions where the preprocessing pipeline succeeds — precisely the "easy" cases where human pose is clearly visible, objects have well-defined boundaries, and occlusions are minimal. This means the model is trained primarily on examples where interaction geometry is already easy to infer from appearance alone. The hard cases — heavy occlusions, thin or transparent objects, unusual poses, cluttered backgrounds — are systematically underrepresented or absent from the training data because the preprocessing pipeline fails on them. For a practitioner deploying CoInteract in real e-commerce scenarios, where products include jewelry (small, thin), glassware (transparent), clothing (deformable, self-occluding), or phones (reflective, thin), the training data bias may mean the model's structural priors are weakest precisely where they are most valuable. The filtering step introduces a survivorship bias: the model learns from clips that "survived" preprocessing, which may be systematically different from the distribution of real HOI videos in deployment.

What evidence exists in the paper. The paper does not report filtering rates, failure mode analysis, or the fraction of training data discarded at each preprocessing stage. The validation module is mentioned in one sentence (Section 3.3) without quantitative detail. The qualitative results (Fig. 5) show products that are relatively large, opaque, and have well-defined boundaries (instrument, bag, cup, plush toy) — consistent with a preprocessing pipeline that succeeds on such objects but may fail on more challenging ones. No ablation tests the effect of preprocessing quality on downstream performance (e.g., by artificially degrading mesh accuracy or mask quality and measuring the impact on VLM-QA).

Mitigation status. Not mitigated. This is a practical limitation that a deployer would discover when attempting to scale CoInteract to a broader product catalog. Future work mentioned in Section 8 focuses on "pretraining or finetuning models to directly predict difficulty" and "combining search and revisions," but does not mention improving robustness to preprocessing errors. A practitioner would need to either improve the preprocessing pipeline (by replacing or augmenting the component models) or develop training strategies that are robust to noisy structural labels (e.g., by training with dropout on the HOI stream or by using confidence-weighted structural losses).

5. The Asymmetric Co-Attention Strategy Incompletely Distills Structural Knowledge

The assumption or constraint. The paper's headline inference efficiency claim (1.04× cost with near-identical quality to 4.13× cost) depends on the asymmetric co-attention mask successfully distilling all relevant structural knowledge from the HOI stream into the shared backbone parameters during the relatively brief Stage 2 training (2,000 iterations). The paper states that retaining the HOI branch at inference (w/o Asym. Mask variant) improves VLM-QA from 0.72 to 0.76 (Table 3), confirming that the distillation is incomplete — the RGB-only pathway does not achieve the quality ceiling of the bidirectional model.

The consequence. For a practitioner, this means CoInteract deliberately accepts a quality penalty of approximately 5.6% on VLM-QA (0.72 vs. 0.76) in exchange for a 4× inference cost reduction. Whether this tradeoff is acceptable depends on the application: in scenarios where per-video compute cost dominates (e.g., generating millions of product videos), the efficiency gain is clearly worth the quality sacrifice. But in scenarios where interaction plausibility is safety-critical (e.g., generating training data for robotic grasping, where interpenetration errors could propagate to real-world failures), a practitioner might prefer the bidirectional variant despite its cost. The paper does not characterize which types of interactions suffer from the distillation gap — is the quality loss concentrated in specific hand poses, product types, or motion patterns? Without this breakdown, a deployer cannot assess whether the 5.6% average gap masks large regressions on specific important cases.

Additionally, the 2,000-iteration Stage 2 is short relative to the 5,000-iteration Stage 1, and the paper provides no evidence that the distillation process has converged. The VLM-QA gap between asymmetric and bidirectional inference might shrink with longer Stage 2 training, or it might be a fundamental limit of the asymmetric attention design. Without training dynamics curves, a practitioner cannot know whether investing more compute in Stage 2 would close the gap.

What evidence exists in the paper. Table 3 provides the quantitative comparison (Full Model vs. w/o Asym. Mask), showing VLM-QA 0.72 vs. 0.76 and HQ 0.724 vs. 0.738. The paper characterizes this as trading "marginal interaction gains for a dramatic efficiency improvement at near-zero additional inference cost." The 2,000-iteration Stage 2 duration is stated in Section 4.1 without justification or ablation. No training dynamics curves are provided.

Mitigation status. Partially mitigated by the paper's transparent acknowledgment of the tradeoff, but the underlying incompleteness of distillation is not investigated. A practitioner wanting to minimize the quality gap would need to experiment with longer Stage 2 training, gradual mask transitions, or alternative distillation objectives — none of which the paper explores.

6. Latency Constraints Are Not Addressed Despite the Serial Nature of Video Generation

The assumption or constraint. The paper measures inference cost as a multiplicative factor relative to a baseline (1.04×, 4.13× in Table 3), which captures total FLOPs but ignores wall-clock latency. Video generation with diffusion models is inherently sequential — the ODE solver must run 40 inference steps (Section 4.1) serially, and each step processes the full spatiotemporal token sequence through the DiT backbone. The CoInteract architecture adds a small per-step overhead (the MoE routing and lightweight expert FFNs) but does not change the fundamental latency structure.

The consequence. For latency-sensitive applications — interactive virtual assistants, real-time product demonstrations, live-streaming avatars — the 40-step generation process at 480p resolution may be too slow regardless of FLOP efficiency. The paper does not report generation time or throughput (videos per GPU-hour), making it impossible for a practitioner to assess deployability. Even if CoInteract is 4× more compute-efficient than best-of-N sampling, it may still require minutes per video at 480p, which is unacceptable for interactive use cases. The paper's motivation explicitly targets e-commerce, digital advertising, and virtual marketing (Abstract) — all applications where time-to-video matters. A marketing team needing to generate 1,000 product demonstration videos overnight cares about throughput; a live-streaming platform cares about end-to-end latency measured in seconds, not minutes.

Furthermore, the HoI structure stream preprocessing pipeline (Section 3.3, Fig. 4) is entirely serial and involves multiple large models (Qwen-Edit, SAM3, SAM3D-body). Even though this pipeline is only used during training, the training wall-clock time is dominated by preprocessing, not diffusion training — generating paired RGB–HOI representations for 40 hours of video requires running 3D mesh recovery and object segmentation on every frame, which could take days or weeks on available hardware. This limits how quickly the model can be retrained or adapted to new product categories.

What evidence exists in the paper. The paper provides no latency or throughput measurements. The inference settings (40 steps, CFG scale 5, 480p) are stated in Section 4.1, but no timing information is reported. The preprocessing pipeline's computational cost is not characterized. The paper makes no claims about real-time capability or throughput.

Mitigation status. Not mitigated. This is a standard omission in video generation papers — the field prioritizes quality metrics over latency — but for a method explicitly motivated by practical applications (e-commerce, virtual marketing), the absence of any latency analysis is a gap. A practitioner deploying CoInteract would need to benchmark generation speed independently and determine whether the 40-step ODE solver at 480p meets their latency requirements, or whether distillation, progressive distillation, or consistency models would be needed to reduce the step count.

7. Implications and Future Directions

How This Work Changes the Landscape

CoInteract does not introduce a new generative model architecture, a new training objective, or a new diffusion formulation. It introduces a new deployment paradigm for conditioned video generation: the idea that structural knowledge about interaction geometry can be injected during training as a disposable scaffold, regularizing shared backbone parameters while being removed for zero-overhead inference. This is a methodological shift rather than a foundational breakthrough—it changes how we train video diffusion models for constrained generation tasks, not what they can generate.

The shift matters because it breaks a stalemate in HOI video synthesis. The field had been oscillating between two unsatisfying positions: either accept heavy inference-time preprocessing with external structural signals (the multi-condition paradigm, exemplified by AnchorCrafter's per-frame pose and object annotations) or accept physically implausible outputs from flexible but unconstrained multi-reference generation (Humo, Phantom, VACE). CoInteract demonstrates a third path: internalize the structural constraints into the backbone during training, then discard the constraint mechanism at deployment. This reframes the problem from "how do we provide better conditioning?" to "how do we teach the model physics during training so it doesn't need conditioning at test time?"

The empirical evidence for this reframing is in the ablation patterns (Table 3). The w/o Asym. Mask variant (bidirectional attention, HOI branch retained at inference) achieves VLM-QA of 0.76 at 4.13× inference cost—this is the "structure as conditioning" ceiling. The Full Model (asymmetric attention, HOI branch removed) achieves VLM-QA of 0.72 at 1.04× cost—a 5.6% quality sacrifice for a 4× cost reduction. The fact that this tradeoff is possible at all—that structural knowledge can be distilled into shared weights with only modest quality loss—is the finding that changes the landscape, not any individual architectural component.

What research directions become more attractive. The structure-as-regularizer paradigm demonstrated for HOI naturally generalizes to any generative task where the training distribution contains privileged geometric information absent at inference. Depth maps for novel view synthesis, contact maps for robotic manipulation videos, 3D scene flow for autonomous driving footage, physics simulation states for human motion prediction—in each case, a strong structural signal exists in curated training data (from sensors, simulators, or offline reconstruction) but is unavailable at deployment. Training with this signal under an asymmetric attention mask, then discarding it, could yield models that generate outputs respecting physical constraints without requiring the privileged information at test time. The paper's specific mechanism (asymmetric co-attention with HOI←RGB gradient flow) provides a template for how to structure the information flow.

What research directions become less attractive. The paper's results suggest diminishing returns from further refinement of multi-condition approaches that require per-frame structural inputs at inference. AnchorCrafter's poor generalization to unseen objects (Fig. 5)—it performs well on training-set objects but suffers identity drift on novel objects—indicates that external structural conditioning encourages the model to correlate preprocessing signals with training-set appearances rather than learning generalizable interaction geometry. If structural knowledge can be internalized during training with a 5.6% quality penalty for 4× cost savings, the engineering case for complex inference-time preprocessing pipelines weakens considerably. Similarly, the paper's finding that HUMO and Phantom achieve higher AES by hallucinating backgrounds (at the cost of reference consistency) suggests that optimizing for aesthetic metrics without structural constraints leads models away from the fidelity that HOI applications require—work pursuing better FID or IS scores on HOI benchmarks without addressing interaction plausibility may be optimizing the wrong objective.

Reconciling prior contradictions. The paper does not explicitly identify or resolve contradictions in the prior literature, but the quantitative results implicitly reconcile a tension between two observations: (1) video diffusion models produce photorealistic results on general video generation benchmarks, and (2) these same models produce physically implausible interactions on HOI tasks. The resolution is that perceptual quality and physical plausibility are largely orthogonal objectives under RGB-only supervision. The w/o Co-Gen ablation (Table 3) makes this explicit: removing structural supervision causes VLM-QA to drop 33.3% while IQ slightly increases. The model learns to generate visually appealing videos with physically impossible contact geometry. This means that prior work reporting strong FVD/IS/FID on general video benchmarks was measuring a different capability than what HOI applications need, and that divergence was invisible to the standard evaluation suite. Future HOI research should not assume that better general video generation models will automatically improve HOI synthesis—the supervision signal must explicitly encode geometric constraints.

Follow-Up Research This Work Enables

Quantifying the distillation gap: what interaction types does asymmetric training fail to internalize? The Full Model (asymmetric attention, VLM-QA 0.72) underperforms the bidirectional variant (HOI branch retained, VLM-QA 0.76) by 5.6% on aggregate VLM-QA. But the aggregate metric masks per-clip variance. A strong follow-up would break down the VLM-QA gap by interaction type (grasping vs. wearing vs. pointing vs. lifting), by object geometry (rigid vs. deformable, opaque vs. transparent, simple vs. articulated), and by occlusion severity (no occlusion, partial hand occlusion, full hand occlusion). The hypothesis is that the asymmetric training fails to internalize structural constraints primarily for heavily occluded hands or complex multi-contact grasps—cases where the shared backbone parameters, having never directly attended to HOI tokens during the RGB pathway's computation, lack sufficient geometric information to resolve ambiguous contact boundaries. If confirmed, this would suggest targeted architectural modifications for those cases (e.g., allowing limited RGB→HOI attention specifically for high-occlusion tokens) rather than globally abandoning the asymmetric design. The experiment would require annotating the 50 test clips with interaction-type and occlusion-severity labels, then computing VLM-QA stratified by these labels for both the Full Model and w/o Asym. Mask variants.

Alternative structural representations: does the silhouette+RGB encoding outperform simpler or richer alternatives? The HOI structure stream encodes the human as a mesh-projected silhouette while retaining the object's RGB appearance. This specific asymmetry is motivated by intuition—force the model to bridge geometry and appearance—but is never empirically compared against alternatives. A systematic comparison would train identical CoInteract backbones (same initialization, same training schedule, same MoE configuration) on HOI structure streams constructed from: (a) depth maps (single-channel, continuous depth values for both human and object), (b) surface normal maps (3-channel, orientation-encoding), (c) binary contact masks (1 where human and object pixels are adjacent, 0 elsewhere), (d) full-geometry representation (both human and object as silhouettes), (e) full-RGB representation (both human and object retain appearance), and (f) the paper's asymmetric representation. The dependent variables would be VLM-QA, HQ, and the VLM-QA gap between asymmetric and bidirectional inference (measuring how distillable each representation is). The strongest outcome would identify whether the asymmetric representation is genuinely optimal or whether simpler representations (binary contact masks) or richer ones (depth maps) achieve equivalent or better structural regularization. A negative result—finding that any geometry representation works equally well—would simplify adoption by removing the need for 3D mesh recovery.

Transfer to alternative DiT backbones: does the co-generation paradigm survive architecture changes? CoInteract is initialized from WanS2V and inherits its attention architecture, positional encoding scheme, and FFN design. The asymmetric co-attention mask assumes full spatiotemporal self-attention across the concatenated token sequence—a design that aligns with WanS2V but may not hold for architectures using factorized spatial-temporal attention (e.g., HunyuanVideo) or architectures without self-attention across all tokens (e.g., some efficient DiT variants). A transfer experiment would reimplement CoInteract's dual-stream training on top of two architecturally distinct backbones—one with full 3D self-attention (similar to WanS2V) and one with separated spatial and temporal attention—while keeping the HOI data, preprocessing pipeline, MoE design, and training schedule identical. The critical metric is whether the VLM-QA gap between asymmetric and bidirectional inference is comparable across backbones. If the asymmetric design works well only under full self-attention (because the HOI←RGB cross-attention gradient flows through all token pairs), this restricts the paradigm's applicability to a subset of DiT architectures. If it works under separated attention as well, the paradigm is architecturally robust. A negative result—finding that the asymmetric mask is incompatible with factorized attention—would motivate alternative distillation mechanisms (e.g., explicit knowledge distillation losses rather than gradient flow through shared parameters) for these architectures.

Training dynamics of structural distillation: does longer Stage 2 training close the quality gap? The paper trains Stage 2 (asymmetric co-attention) for exactly 2,000 iterations, compared to 5,000 iterations for Stage 1 (full bidirectional attention). The 2,000-iteration duration is presented without sensitivity analysis. A learning curve experiment would extend Stage 2 to 10,000, 20,000, or 50,000 iterations, evaluating VLM-QA on a validation set every 500 iterations, and plot both the Full Model (asymmetric inference) and the w/o Asym. Mask variant (bidirectional inference, computed by temporarily removing the mask during evaluation) to track the distillation gap. The key questions: (1) Does the asymmetric model's VLM-QA asymptotically approach the bidirectional model's ceiling, or does it plateau with a persistent gap? Convergence would suggest the current gap is a training budget limitation; a persistent gap would indicate a fundamental information bottleneck—the RGB pathway, denied direct HOI attention, simply cannot represent certain geometric constraints. (2) Does the bidirectional model continue improving, or does Stage 1 already saturate it? If bidirectional VLM-QA improves substantially during Stage 2, the 5,000-iteration Stage 1 was insufficient, and the quality ceiling is higher than 0.76. (3) At what iteration does the asymmetric model's VLM-QA exceed that of standard single-stream training (i.e., the w/o Co-Gen baseline at VLM-QA 0.48), and how quickly does the gap close initially vs. tail off? This would provide practical guidance for budget-constrained training.

Scaling structural supervision along the data axis: can co-generation compensate for smaller datasets? The paper trains on 12,000 curated clips with paired RGB–HOI representations—a moderately-sized dataset by video generation standards. The structural supervision hypothesis predicts that co-generation provides a stronger per-sample learning signal than RGB-only training, which implies that the performance gap between CoInteract and the w/o Co-Gen baseline should widen as training data decreases (the structural signal becomes more valuable when appearance statistics are unreliable). A data scaling experiment would train both the Full Model and the w/o Co-Gen baseline on random subsets of 1,000, 3,000, 6,000, and 12,000 clips, measuring VLM-QA and video quality metrics at each scale. If the Full Model achieves equivalent VLM-QA at 3,000 clips to the w/o Co-Gen baseline at 12,000 clips, structural supervision provides a ~4× data efficiency gain—a finding with substantial practical implications for domains where paired structural data is easier to obtain than large-scale video datasets (e.g., robotics, where simulation can generate unlimited paired RGB+geometry data but real-world video is scarce). If the gap is constant across data scales (both methods improve equally with more data), the structural signal complements rather than substitutes for data scale—still valuable, but for different reasons.

Combining co-generation with explicit physics losses: can the distillation be strengthened? The current approach relies entirely on the flow matching loss applied to the HOI stream to inject structural constraints—the model learns geometry implicitly through the auxiliary prediction task. An alternative is to add an explicit physics violation penalty: for each frame, compute the overlap between the human silhouette and the object mask in the HOI stream, and penalize the shared backbone when interpenetration is predicted. This would provide a direct gradient signal for the constraint "hands should not pass through objects" rather than relying on the model to infer this constraint from observing that the HOI stream shows disjoint occupancy. A follow-up could add a differentiable interpenetration loss (e.g., soft IoU between human silhouette and object mask, computed from the HOI stream's rendered geometry) to the total objective during Stage 2 asymmetric training, and measure whether it (a) accelerates convergence of VLM-QA, (b) closes the distillation gap by providing a more direct structural gradient through shared parameters, and (c) reduces the remaining 28% of interactions that Gemini-3-Pro rates as implausible. The risk is that an explicit physics loss could be exploited—the model might learn to shrink the human silhouette or expand the object mask to trivially reduce interpenetration without learning correct contact geometry—so the experiment would need to monitor for such degenerate solutions through qualitative inspection and metrics like DINOobj and DINOid (which would degrade if entity representations are distorted).

Practical Applications and Downstream Use Cases

E-commerce product demonstration video generation at scale. The most direct application, explicitly motivating the paper (Abstract: "e-commerce, digital advertising, and virtual marketing"). A large e-commerce platform with millions of products needs to generate demonstration videos showing virtual anchors interacting with products—holding, wearing, rotating, or demonstrating features—while speaking scripted audio. CoInteract's architectural advantage (unified end-to-end generation without per-frame preprocessing at inference) makes it more deployable than multi-condition approaches like AnchorCrafter, which require extracting per-frame pose and object annotations for every video. The 4× cost savings from asymmetric inference (1.04× vs. 4.13× cost for retaining the HOI branch) directly translates to 4× higher throughput on a fixed GPU fleet—critical when generating thousands of videos. The key deployment consideration is whether the 5.6% VLM-QA gap between asymmetric and bidirectional inference matters for consumer perception. The user study (Table 2) shows CoInteract ranks first on interaction plausibility despite being the asymmetric variant, suggesting the quality is sufficient for consumer applications. The primary engineering challenge is scaling the preprocessing pipeline (Qwen-Edit → SAM3 → SAM3D-body) to curate training data for new product categories; the paper provides a blueprint but does not solve the preprocessing throughput problem.

Virtual try-on and fashion demonstration. A natural extension of the product demonstration use case. A fashion retailer wants to show a virtual model wearing clothing items, adjusting straps, turning to show back details, and demonstrating how the garment moves with the body. CoInteract's geometric constraints via the HOI stream are directly applicable: the human silhouette projection would enforce that the clothing (the "object") conforms to the body's spatial occupancy, while hand-object interaction (adjusting a strap, smoothing fabric) would benefit from the same contact geometry regularization that reduces interpenetration. The challenge over standard product demonstration is that clothing is deformable—the object geometry changes as the human moves—which means the HOI structure stream's assumption of a static object mask (from SAM3) would need to be replaced with per-frame segmentations. A practical deployment would require either extending SAM3 to video-level deformable object tracking or replacing the object mask with a parametric clothing model that deforms with the body mesh.

Training data generation for robotic manipulation policies. An application outside the paper's stated scope but enabled by its core technology. Robotic manipulation research increasingly uses simulation-to-real transfer, where policies are trained on synthetic video and deployed on physical robots. A critical challenge is that synthetic videos lack the visual diversity and photorealism of real-world footage, causing a sim-to-real gap. CoInteract could generate photorealistic videos of humans (or humanoid robots) interacting with objects, conditioned on the same structural inputs used for policy training (object pose, hand pose, contact states). The key advantage is that CoInteract can generate videos from structural conditions during training (via the HOI stream) while producing photorealistic RGB outputs. The policy could be trained on the generated RGB videos, learning from diverse visual appearances while preserving the geometric correctness enforced by the co-generation paradigm. The practical question is whether the remaining interpenetration errors (the 28% of interactions rated implausible by Gemini-3-Pro) are acceptable training noise for policy learning—robotic policies are typically robust to occasional visual artifacts but may learn incorrect contact affordances if interpenetration is frequent. A deployment would require measuring the downstream policy success rate when trained on CoInteract-generated vs. real footage, not just generative quality metrics.