ArXiv: 2604.11804

🎯 Pitch

For the first time, a single video generation model simultaneously controls character identity, object appearance, lip-synced speech, and explicit motion—all injected into a 12B backbone through mechanisms that treat each condition as a native temporal stream, not an architectural afterthought.

1. Executive Summary

This paper introduces OmniShow, an end-to-end framework for Human-Object Interaction Video Generation (HOIVG) that unifies four multimodal conditioning signals—text, reference images, audio, and pose—within a single 12.3B MMDiT-based model built on Waver 1.0. The framework achieves this unification through two named mechanisms: Unified Channel-wise Conditioning, which injects reference images and pose via a channel-concatenation strategy that preserves the base model's native input structure (e.g., appending pseudo-frame tokens alongside noisy video tokens rather than constructing hybrid tokens), and Gated Local-Context Attention, which enforces fine-grained audio-visual synchronization through masked cross-attention with learnable gating vectors (e.g., restricting each video frame to attend only to its corresponding audio window). A Decoupled-Then-Joint Training strategy addresses data scarcity by first training separate R2V and A2V specialist models on heterogeneous sub-task datasets, then merging their weights and jointly fine-tuning—the merged model exhibits emergent RA2V capabilities before any joint training. On the authors' newly introduced HOIVG-Bench (135 curated samples with paired text, human and object reference images, audio, and pose), OmniShow achieves state-of-the-art or competitive results across R2V, RA2V, and RP2V settings while remaining ~2.5% larger than its 12B backbone (12.3B total), and is the only model supporting the full RAP2V setting. Human evaluation shows a clear preference for OmniShow over baselines (e.g., 51.2% vs. 21.2% favoring HuMo-17B in overall video quality for RA2V), establishing that simultaneous multimodal conditioning can be achieved without the quality degradation that typically accompanies aggressive architectural modification—provided the injection mechanisms respect the pretrained model's native input distribution.

2. Context and Motivation

The Practical Problem: Real-World Video Production Needs Diverse, Simultaneous Control

The paper is motivated by a concrete, commercially significant gap: current video generation systems cannot simultaneously handle the four conditioning signals that real-world human-object interaction content creation demands. If an e-commerce brand wants to generate a promotional video, they need the system to (1) understand a textual description of the scene ("a woman demonstrating a new smartphone in a bright modern kitchen"), (2) preserve the exact appearance of a specific human model (from a reference photo) and the specific product (from a product image), (3) synchronize the human's lip movements and gestures with a provided audio narration track, and (4) follow an explicit pose sequence that defines precise hand-object interactions (e.g., picking up the phone, turning it to show the back, gesturing toward the camera). Section 1 frames this combination of requirements as the Human-Object Interaction Video Generation (HOIVG) task.

This is not an academic exercise in modality stacking. The paper explicitly grounds its motivation in three application domains (Section 1, paragraph 1):

  • E-commerce demonstrations: Automated product showcase videos where the same human model demonstrates different products across hundreds of SKUs, each requiring distinct object reference images, synchronized voice-over audio, and precisely choreographed handling motions.
  • Short video production: Creators who need to rapidly iterate content by recombining existing assets—a human reference from one shoot, an object from a product database, a voiceover recorded separately, and a pose track from a motion-capture library.
  • Interactive entertainment: Applications where user-provided inputs (a selfie, a chosen virtual object, a recorded voice line, and a captured body motion) drive real-time or near-real-time character animation.

The critical observation is that none of these applications can be fully realized by systems that handle only subsets of these conditions. A system that does Reference+Audio-to-Video but lacks pose control cannot execute precise hand-object interactions—the human might speak naturally but handle the product incorrectly or unnaturally. A system that does Reference+Pose-to-Video but lacks audio produces "silent" demonstrations that feel unnatural and fail the basic requirement of spoken narration. The HOIVG task is therefore defined by the simultaneity requirement: all four conditions must be honored in a single generated output, not composited from separate generation steps.

The Gap: No Unified Framework Exists

Section 2 systematically surveys the landscape of related work, and the conclusion is unambiguous: there is currently no unified framework capable of harmonizing text, reference image, audio, and pose conditions in an end-to-end manner. The paper identifies four categories of prior approaches, each falling short of the full HOIVG requirement:

Reference-to-Video (R2V) methods focus on subject identity preservation from reference images. Systems like SkyReels-A2 (Fei et al., 2025), Phantom (Liu et al., 2025), and various video customization approaches (Zhou et al., 2025; Yuan et al., 2025) can generate videos where a specific person and/or object appears consistently. However, these methods are fundamentally silent: they have no mechanism for incorporating audio, meaning they cannot produce the synchronized speech that is essential for product demonstrations or conversational scenarios. The paper characterizes this as producing "silent interactions" (Section 1, paragraph 2). Moreover, these methods typically do not accept explicit pose control, relying instead on the text prompt alone to guide motion—a level of control that is inadequate when specific hand movements and object manipulations are required.

Audio-to-Video (A2V) methods have converged around portrait animation and talking-head generation. Systems like Hallo3 (Cui et al., 2025), Anytalker (Zhong et al., 2025), MultiTalk (Kong et al., 2025), and OmniAvatar (Gan et al., 2025) can animate a single input image with synchronized speech and body movements driven by an audio track. However, these methods have a critical constraint: they accept only a single initial frame (or a single reference image) as the visual condition, not separate human and object reference images. This means you cannot independently specify "this particular human model" and "this particular product"—the input is a single pre-composited image, which severely limits the re-composability that e-commerce and content creation workflows require. Furthermore, these methods typically do not accept pose sequences as input, operating in an "audio-to-motion" paradigm where the generated motion is inferred entirely from the audio signal. For tasks requiring specific hand-object trajectories that are unrelated to the speech content (e.g., demonstrating how to assemble a product while describing its features verbally), this is insufficient.

Methods targeting Human-Object Interaction Video Generation specifically exist—AnchorCrafter (Xu et al., 2024), HunyuanVideo-HOMA (Huang et al., 2025), and DreamActor-H1 (Wang et al., 2025)—but each imposes requirements that limit their applicability. AnchorCrafter requires predefined body skeletons, hand meshes, and object depth maps that are difficult to obtain in practice. HunyuanVideo-HOMA uses sparse human poses and object trajectory dots, a simplified representation that may not capture fine-grained interactions. DreamActor-H1 relies on body mesh templates and object bounding boxes, which are powerful but restrictive input formats not available in typical content creation pipelines. More fundamentally, none of these methods support audio conditioning. They are visual-only systems that cannot respond to speech input, meaning they produce silent demonstrations regardless of the application.

Joint audio-video generation models like Seedance 1.5 Pro (Chen et al., 2025), Jova (Huang et al., 2025), and LTX-2 (HaCohen et al., 2025) represent the frontier of multimodal generation, but they operate primarily from text prompts. They lack the reference image conditioning that would allow specifying a particular person or object identity—they can generate "a woman" holding "a smartphone," but not this specific woman holding this specific product.

Attempts at combining audio and reference images do exist: HuMo (Chen et al., 2025) and HunyuanCustom (Hu et al., 2025) support Reference+Audio-to-Video. However, the paper points out that these methods (1) overlook pose conditions entirely, making them inadequate for complex interactions that text cannot fully specify, and (2) are not tailored for the HOIVG task, often falling short of the high-fidelity subject preservation that e-commerce applications demand. VACE (Jiang et al., 2025) and Kling-Omni (Kling Team, 2025) support multiple modalities including reference images and pose, but lack audio capabilities.

The central gap is therefore clear: the research community has produced strong specialized models for individual conditioning tasks (R2V, A2V, pose-guided generation), but no single model brings all four modalities together. The prevailing approach in practice would be to cascade systems—generate a video using R2V+pose, then apply a separate lip-synchronization model to add audio—but the paper shows in Section C that this cascaded approach introduces artifacts, degrades visual quality, and fails to achieve the synchronized motion that joint conditioning enables.

Why Is Data a Barrier? The Heterogeneous Data Problem

A reader might reasonably ask: if the task is so practically important, why has no one built a unified framework before? The paper identifies a data scarcity problem (Section 1, Challenge ii) that makes naive approaches infeasible. High-quality data for HOIVG requires quintuplet alignment: a video where the text caption, human reference image, object reference image, audio track, and pose sequence are all valid and mutually consistent. If any one condition fails—the audio is misaligned, the pose is incomplete, the reference images don't match the subjects in the video—the entire sample must be discarded.

The consequence is that curated, fully-paired HOIVG datasets are extremely rare and expensive to produce. What is readily available are fragmented, sub-task-specific datasets: large collections of R2V data (image-video pairs with captions), separate collections of A2V data (talking-head videos with synchronized audio), and smaller collections of pose-annotated human motion videos. These datasets exist in relative abundance because they are natural byproducts of existing content (social media videos, movie clips, product photography) and require only the annotation modalities relevant to their specific sub-task.

The paper's insight—which motivates the Decoupled-Then-Joint Training strategy—is that throwing away this heterogeneous data and only training on fully-paired samples would be enormously wasteful. A unified HOIVG model needs to learn rich representations of human appearance, object appearance, audio-visual correspondence, and pose-conditioned motion, and the lion's share of available training signal for each of these skills exists in separate, non-overlapping datasets. The challenge is methodological: how do you train a single model to handle all four conditions simultaneously when your training data is fragmented across sub-tasks, with each sample providing only a subset of conditions?

The Controllability–Quality Trade-off

A deeper technical challenge that Section 1 identifies (Challenge i) is the tension between adding conditioning mechanisms and preserving generation quality. Modern video diffusion transformers (DiTs) like Waver 1.0, upon which OmniShow is built, achieve their impressive generation quality through pretraining on enormous corpora of video data. Their internal representations—the token distributions, the attention patterns, the learned flow dynamics—are finely calibrated to produce coherent, high-fidelity videos from text (and optionally, a first-frame image).

Introducing new conditioning modalities typically requires architectural modification: adding new cross-attention layers for audio, creating new input pathways for pose, or concatenating additional conditioning tokens to the noisy latent representation. Each such modification disrupts the pretrained feature distributions in ways that can be difficult to recover from, especially when training on the limited data available for the new modalities. The paper frames this concretely: "Naively introducing aggressive modifications to handle multimodal inputs typically disrupts the base model's pretrained generative priors" (Section 1). The result is a model that may respect the new conditions but at the cost of degraded overall visual quality—blur, temporal artifacts, or loss of the rich detail that the base model originally possessed.

This trade-off is not hypothetical. The paper notes in Section 3.3 that prior approaches like HuMo increase model parameters by ~21.4% (totaling 17B from a 14B base) to add audio conditioning, suggesting that their architectural modifications are substantial. OmniShow's design is explicitly motivated by minimizing this disruption: the goal is to add conditioning capability while preserving the native input structure and token distributions of the pretrained base model as closely as possible.

The Evaluation Gap

A final motivation for the paper is the absence of a standardized benchmark for evaluating HOIVG systems (Section 1, Challenge iii). Existing benchmarks evaluate individual conditioning dimensions in isolation: video quality metrics (VBench), subject consistency metrics (OpenS2V, FaceSim), audio-visual synchronization metrics (Sync-C, Sync-D), and pose accuracy metrics (AKD, PCK) all exist as separate evaluation protocols. But there is no benchmark that tests all of these simultaneously on a curated set of samples designed to stress-test the synergy between modalities. Without such a benchmark, it is impossible to determine whether a system that performs well on, say, R2V and A2V separately actually maintains that performance when all conditions are applied simultaneously—whether interactions between conditioning signals degrade individual modality alignment.

The paper's creation of HOIVG-Bench is thus a contribution that sits alongside the model itself: it provides the infrastructure for the community to systematically and fairly compare future HOIVG systems under standardized conditions. The benchmark's 135 samples are carefully constructed to include diverse human attributes, object categories, motion intensities, and interaction types, with all conditions (text, human reference, object reference, audio, pose) fully paired and validated. Importantly, the synthetic generation of human and object reference images (using Nano Banana and ElevenLabs) is not a shortcut but a deliberate choice for privacy compliance and legal safety in a public benchmark release.

How OmniShow Positions Itself

The paper does not claim to have invented any individual conditioning mechanism that is entirely without precedent. Channel-concatenation for reference images, cross-attention for audio, and flow matching for video diffusion are all techniques with prior art. What OmniShow contributes is the architectural philosophy and training methodology that makes simultaneous, non-disruptive unification possible:

  • Philosophy of Minimalist Intervention (explicitly named in Section 5, Discussion): Rather than adding new specialized modules for each modality, OmniShow reuses and minimally extends the base model's existing mechanisms. Reference images and pose are injected through the same channel-concatenation pathway that the base model already uses for first-frame conditioning—just extended with pseudo-frame tokens. Audio is injected through cross-attention layers, but the gating mechanism and the strategic placement (only in dual-stream blocks, informed by empirical gate activation analysis in Figure 5) kepps the architectural footprint small. The result is a model that is only ~2.5% larger than its 12B backbone (12.3B total), compared to ~21.4% inflation in competing approaches.

  • Evolution from Specialists to a Generalist (also named in Section 5): The Decoupled-Then-Joint Training strategy rejects the idea that a unified model must be trained from scratch on unified data. By first training separate R2V and A2V specialist models on abundant heterogeneous data, then fusing them via weight interpolation and joint fine-tuning, the approach leverages the full available training signal from each sub-task. The emergent RA2V capability observed in the merged-but-not-yet-joint-trained model (Figure 4) is presented as evidence that weight-space fusion of specialist models can combine capabilities in ways that exceed the sum of training data—a finding the paper explicitly flags as "an inspiring discovery that we hope will motivate further exploration."

  • Benchmark-driven evaluation: By releasing HOIVG-Bench alongside the model, the paper establishes evaluation standards that future work can adopt, enabling apples-to-apples comparisons and accelerating progress on the task.

In the broader context of controllable video generation research—which the paper surveys in Section 2 as having evolved from single-modality control (R2V, A2V, pose-guided) toward multi-condition integration—OmniShow represents the endpoint of that trajectory for the specific domain of human-object interaction. It is not positioned as a conceptual breakthrough in a single conditioning mechanism, but rather as a systems-level contribution that demonstrates how to achieve robust, quality-preserving unification of multiple conditions through careful architectural decisions and a data-efficient training paradigm.

3. Technical Approach

3.1 Reader Orientation

OmniShow is a video diffusion transformer that, given a text description, photos of a specific person and object, a spoken audio track, and a stick-figure motion sequence, produces a video of that exact person interacting with that exact object while speaking in sync with the audio and moving exactly as the pose sequence dictates. The problem it solves is the simultaneous injection of four fundamentally different types of conditioning information into a pretrained video generation model without destroying the model's ability to produce visually coherent, high-quality output—and it does this by reusing and minimally extending the base model's existing conditioning pathways rather than grafting on large new specialized modules for each modality.

3.2 Big-Picture Architecture (Diagram in Words)

The system processes a video through six major components arranged in a single end-to-end pipeline:

  1. VAE Encoder: Compresses all visual inputs (noisy video tokens, reference images, pose videos) into a shared latent space. The same encoder handles everything—no modality-specific encoders.
  2. Channel-wise Condition Injector: Takes the noisy video tokens (what will become the output), pads them with extra "pseudo-frame" slots, and concatenates the encoded reference image tokens and pose video tokens directly into the channel dimension alongside the noisy tokens. This is the sole entry point for visual conditioning.
  3. Audio Feature Extractor (Wav2Vec 2.0 + projector): Converts raw audio waveforms into dense feature vectors, applies sliding-window context packing to give each feature local temporal context, and projects these into the transformer's hidden dimension.
  4. Gated Local-Context Cross-Attention (inserted into dual-stream MMDiT blocks): Performs masked cross-attention where each video frame token attends only to its temporally corresponding audio window, with a learnable scalar gate that controls how strongly the audio signal influences the video features.
  5. MMDiT Backbone (inherited from Waver 1.0): The core diffusion transformer with dual-stream (video+text) and single-stream blocks that iteratively denoises the concatenated [pseudo-frames | noisy video tokens] using flow matching.
  6. VAE Decoder: Converts the denoised latent tokens back into pixel-space video frames.

Information flows sequentially: conditioning signals are encoded and injected before the denoising process begins, then the transformer iteratively refines the entire concatenated token sequence, with audio cross-attention modulating features at every dual-stream block. The output is a single coherent latent that decodes into the final video.

3.3 Roadmap for the Deep Dive

  • First, the base model (Waver 1.0) and its native conditioning paradigm, because OmniShow's entire design philosophy is to extend rather than replace this mechanism.
  • Second, Unified Channel-wise Conditioning for reference images and pose, since this is the paper's core injection strategy and establishes the "minimalist intervention" pattern.
  • Third, Gated Local-Context Attention for audio, which introduces the only new attention modules and is the most architecturally invasive component—understanding why it works requires seeing how it is constrained.
  • Fourth, the Decoupled-Then-Joint Training strategy, because the training data problem (heterogeneous, sub-task-specific datasets) is what makes naive joint training impossible and motivates the specialist-generalist approach.
  • Fifth, HOIVG-Bench construction and evaluation metrics, since the benchmark is both a contribution and the lens through which all experimental results are interpreted.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and architecture paper whose core idea is that simultaneous multimodal conditioning in video generation can be achieved without quality degradation if you (1) reuse the base model's native conditioning structure for visual modalities, (2) constrain audio injection through temporal masking and learned gating, and (3) train through a specialist-then-generalist curriculum that extracts maximal signal from heterogeneous sub-task data.

The Base Model: Waver 1.0 and Its Native Conditioning Paradigm

OmniShow is built on Waver 1.0, a 12-billion-parameter video generation model that uses the Multimodal Diffusion Transformer (MMDiT) architecture with flow matching training. Understanding OmniShow requires understanding exactly what conditioning capabilities Waver 1.0 already provides and how it provides them, because OmniShow's central design decision is to extend this mechanism rather than replace it.

Latent space and VAE. Waver 1.0 operates in a compressed latent space produced by Wan 2.1 VAE. Raw video frames are encoded into latent tokens $x_1 \in \mathbb{R}^{N \times D}$, where $N$ is the number of latent tokens (determined by the VAE's spatial and temporal compression) and $D$ is the channel dimension. This keeps all subsequent operations in the latent space, with only the final VAE decoder converting back to pixels. The VAE's compression ratio is critical because it determines the temporal granularity at which later components must align audio features with video frames—audio features must be downsampled to match the VAE's temporal compression (a stride of $s=4$, as specified in Section 3.3).

Flow Matching training objective. The model is trained to predict a velocity field that transports Gaussian noise to clean video latents. The training objective (Equation 1 in the paper) is:

LFM=Et,x0,(x1,e)[∥vθ(t,xin,e)−u(xt∣x1)∥2]\mathcal{L}_{\text{FM}} = \mathbb{E}_{t, \mathbf{x}_0, (\mathbf{x}_1, \mathbf{e})}\left[\|\mathbf{v}_\theta(t, \mathbf{x}_{\text{in}}, \mathbf{e}) - \mathbf{u}(\mathbf{x}_t | \mathbf{x}_1)\|^2\right]

where $t$ is the diffusion timestep, $\mathbf{x}_0 \sim \mathcal{N}(0, \mathbf{I})$ is sampled Gaussian noise, $\mathbf{x}_1$ is the clean video latent tokens, $\mathbf{e}$ is the paired text embedding, $\mathbf{x}_{\text{in}}$ is the model's input (which includes conditioning—see below), $\mathbf{v}_\theta$ is the model's predicted velocity, and $\mathbf{u}(\mathbf{x}_t | \mathbf{x}_1) = \mathbf{x}_1 - \mathbf{x}_0$ is the ground-truth flow velocity from the current noisy state $\mathbf{x}_t$ to the clean target $\mathbf{x}_1$.

What it computes: The model takes a noisy version of the video latent $\mathbf{x}_t$ (constructed by interpolating between $\mathbf{x}_0$ and $\mathbf{x}_1$ according to $t$), along with conditioning information, and predicts the velocity vector that points from the current noisy state toward the clean video. The loss is simply the squared Euclidean distance between the predicted and ground-truth velocities, averaged over timesteps, noise samples, and training examples.

Why this form: Flow matching is chosen over the more common diffusion (score-matching) formulation because it provides a simpler, more direct training objective—the model predicts a deterministic vector field rather than a noisy score function. This matters for OmniShow because the additional conditioning mechanisms (reference images, pose, audio) are trained with the same flow matching loss, meaning they must learn to shift the velocity prediction in ways that respect the conditioning without disrupting the underlying transport dynamics. The squared-error form is standard for flow matching and ensures the model learns the conditional expectation of the velocity, which is the optimal predictor under the $L_2$ loss.

Native channel-concatenation conditioning. Waver 1.0 is described as a "task-unified model" supporting both Text-to-Video (T2V) and Image-to-Video (I2V) tasks through a single conditioning mechanism (Equation 2):

xin=Concat(xt,c,m)\mathbf{x}_{\text{in}} = \text{Concat}(\mathbf{x}_t, \mathbf{c}, \mathbf{m})

where $\mathbf{c} \in \mathbb{R}^{N \times D}$ are condition tokens (same spatial-temporal shape as $\mathbf{x}_t$), and $\mathbf{m} \in [0, 1]^{N \times 4}$ is a binary mask indicating which of the 4 possible condition "slots" are active.

What it computes: The model input is formed by concatenating the noisy video tokens $\mathbf{x}_t$, the condition tokens $\mathbf{c}$, and the binary mask $\mathbf{m}$ along the channel dimension. This means each spatial-temporal position has its video features, condition features, and a mask indicator stacked into a single vector. For T2V, $\mathbf{c}$ is filled with black image tokens (all zeros) and $\mathbf{m}$ is all zeros—the model receives no visual condition. For I2V, the first frame's position in $\mathbf{c}$ is replaced with that frame's VAE-encoded tokens, and the corresponding mask entries are set to 1, telling the model "here is a real conditioning signal."

Why this form: Channel concatenation preserves the native input structure of the transformer—the token count $N$ does not change between T2V and I2V, and the model's self-attention layers see a fixed-size sequence. This is in contrast to alternatives like token concatenation along the sequence dimension (which would change $N$ and require the model to learn new positional relationships) or separate cross-attention pathways (which add architectural complexity). The binary mask acts as a learned switch: the model can learn to ignore condition tokens when the mask is zero and attend to them when the mask is one. This is the design pattern that OmniShow extends from 2 tasks (T2V, I2V) to 4+ conditioning modalities.

Dual-stream and single-stream blocks. The MMDiT architecture contains two types of transformer blocks. Dual-stream blocks process video tokens and text tokens through separate attention pathways before mixing them. Single-stream blocks jointly process both modalities. The specific arrangement of these blocks matters because OmniShow inserts audio cross-attention only into the dual-stream blocks—a decision justified empirically by observing that the learned gating parameters $g$ (explained in the audio section below) show much larger activation norms in dual-stream blocks than in single-stream blocks (Figure 5).

Unified Channel-wise Conditioning: Injecting Reference Images and Pose

This mechanism is how OmniShow adds reference image conditioning (for both human and object appearance) and pose conditioning (for explicit motion control) to the base model's existing I2V pathway. The core insight is that both modalities can be injected through the same channel-concatenation interface that the base model already uses for first-frame conditioning, provided the capacity is expanded with pseudo-frame tokens.

Pose encoding as RGB video. The paper renders the target pose sequence as an RGB video—skeleton keypoints drawn as colored lines on a black background—and encodes this through the same VAE used for the output video. This produces pose video tokens $\mathbf{p} \in \mathbb{R}^{N \times D}$ that have exactly the same shape as the noisy video tokens $\mathbf{x}_t$. Using the same VAE for pose and video is critical: it ensures the pose tokens live in the same latent distribution as the video tokens, so the model can process them through identical pathways without needing a separate pose encoder or projection network.

Reference image encoding. Reference images (one for the human subject, one for the object) are also VAE-encoded, producing reference image tokens $\mathbf{r} \in \mathbb{R}^{N' \times D}$. Here $N'$ is the number of tokens for a single frame (the reference images are still images, so $N' < N$ where $N$ covers multiple frames). The paper does not explicitly state $N'$, but it corresponds to the latent dimensions of a single frame under the Wan 2.1 VAE's spatial compression.

Pseudo-frame augmentation. The base model's channel-concatenation interface provides $N$ conditioning slots—one per video token position. But OmniShow needs to inject reference images (which are single-frame) and pose (which is full-video) simultaneously. Simply averaging or summing them would lose information. The solution is to expand the token capacity by appending $N'$ pseudo-frame tokens $\mathbf{x}' \in \mathbb{R}^{N' \times D}$ to the noisy video tokens along the temporal axis. The full injection formula (Equation 3) is:

xin=Concat([x′∥xt],[r∥p],[m′∥m])\mathbf{x}_{\text{in}} = \text{Concat}([\mathbf{x}' \| \mathbf{x}_t], [\mathbf{r} \| \mathbf{p}], [\mathbf{m}' \| \mathbf{m}])

where $\|$ denotes concatenation along the temporal (token-count) axis, and $\mathbf{m}'$ is the corresponding augmented binary mask for the pseudo-frame positions.

What it computes: The input to the model is now a sequence of $N + N'$ tokens. The first $N'$ positions correspond to "pseudo-frames": their noisy video tokens are $\mathbf{x}'$ (initialized as noisy versions of the reference images—see below), their condition tokens are the reference image tokens $\mathbf{r}$, and their mask is $\mathbf{m}'$ (set to 1 where reference images are present). The remaining $N$ positions correspond to the actual video frames: their noisy tokens are $\mathbf{x}_t$, their condition tokens are the pose video tokens $\mathbf{p}$, and their mask is $\mathbf{m}$. The model sees one unified sequence where pseudo-frames carry reference appearance information and real frames carry pose motion information. Self-attention operates across the entire sequence, so pseudo-frame tokens can influence real-frame tokens and vice versa.

Why this form: The key alternative the paper considers and rejects is token concatenation along the sequence dimension (also described as "hybrid tokens" in Section 3.2). In that approach, reference image tokens would be concatenated directly to the noisy video token sequence (changing the sequence length) or interleaved with them. The paper argues this would introduce "substantial learning costs" because the base model's self-attention patterns and positional embeddings are calibrated for videos of a specific structure. By keeping the channel-concatenation structure intact and merely expanding the sequence with pseudo-frames positioned as a temporal prefix, the model can "efficiently transfer the pretrained I2V capability"—the pseudo-frames act as an extended first-frame conditioning that the model already knows how to handle. The quantitative ablation (Table 2a, Figure 3) confirms this: the channel-concatenation approach achieves FaceSim = 0.707 vs. 0.601 for token concatenation, a substantial improvement in identity preservation.

Pseudo-frame initialization and the reference reconstruction loss. Leaving $\mathbf{x}'$ as all-zero tensors would give the model no signal about what should appear in the pseudo-frame positions. The paper instead initializes $\mathbf{x}'$ with noisy reference image tokens—the clean reference image tokens $\mathbf{r}$ perturbed by the same diffusion timestep $t$ used for the main video tokens. This means the pseudo-frames start from a noisy version of what they are supposed to reconstruct, and the model's denoising process must recover them.

To explicitly enforce that the model preserves reference image details, the paper adds a second flow matching loss $\mathcal{L}_{\text{FM-ref}}$ computed only on the pseudo-frame positions, with a loss weight of 1 (equal to the main video loss). This is the "Reference Reconstruction Loss" mentioned in Section 3.2.

What it computes: During training, the model predicts velocity for all $N + N'$ positions. For the pseudo-frame positions, there is a ground-truth target: the clean reference image tokens. So the standard flow matching loss is applied to the pseudo-frames, exactly as it is for real frames. The total loss is $\mathcal{L}_{\text{FM-video}} + \mathcal{L}_{\text{FM-ref}}$.

Why this form: Without the reconstruction loss, the pseudo-frames would be "dead" positions that the model might learn to ignore. By forcing the model to reconstruct the reference images at those positions, the loss ensures that (1) the pseudo-frame tokens carry meaningful visual information about the reference subjects, and (2) this information flows through the self-attention layers to influence the real video tokens in ways that preserve appearance. The ablation in Table 2a confirms this: removing the reconstruction loss drops FaceSim from 0.707 to 0.678, showing that explicit reconstruction supervision is necessary for the highest-fidelity identity preservation.

Positional encoding for pseudo-frames. The base model uses 3D Spatiotemporal Rotary Position Embeddings (RoPE) that encode a token's position in the (time, height, width) grid. For the pseudo-frames, the paper adopts the native strategy: treating pseudo-frames and original video frames as a continuous temporal sequence starting from $T=0$. Alternative strategies considered in an ablation (Table 5a) include "Temporal Shift" (using negative temporal indices like $T=-1, -2$ for pseudo-frames) and "Spatiotemporal Shift" (adding spatial offsets following HunyuanCustom's approach). The native strategy wins (FaceSim = 0.707 vs. 0.675 for Temporal Shift and 0.279 for Spatiotemporal Shift) because "the model's pre-training on standard continuous video" creates an expectation of monotonically increasing temporal indices, and artificial shifts create mismatches that the channel-conditioning interface cannot easily compensate for.

Summary of the injection flow: (1) Pose sequence is rendered as RGB video and VAE-encoded to $\mathbf{p} \in \mathbb{R}^{N \times D}$. (2) Reference images are VAE-encoded to $\mathbf{r} \in \mathbb{R}^{N' \times D}$. (3) Pseudo-frame tokens $\mathbf{x}'$ are created by noising $\mathbf{r}$ to timestep $t$. (4) The full input is formed by temporal concatenation: $[\mathbf{x}' ; \mathbf{x}_t]$ for noisy tokens, $[\mathbf{r} ; \mathbf{p}]$ for condition tokens, $[\mathbf{m}' ; \mathbf{m}]$ for masks. (5) The MMDiT backbone denoises all $N+N'$ tokens jointly, with self-attention allowing cross-talk between pseudo-frames and real frames. (6) The VAE decoder reconstructs only the real video frames from the last $N$ positions.

Gated Local-Context Attention: Injecting Audio with Temporal Precision

Audio is fundamentally different from visual conditions: it is a 1D temporal signal that must align with specific frames (lip movements to phonemes, gestures to speech rhythm) but has no spatial structure. Injecting it through channel concatenation (as is done for reference images and pose) would be awkward because audio tokens would need to be broadcast or repeated to match the video token shape. Instead, OmniShow uses cross-attention, which is the natural mechanism for attending from one modality (video) to another (audio) with different sequence lengths and spatial layouts.

However, naive global cross-attention—where every video token attends to every audio token—would be both computationally wasteful (audio sequences can be long) and semantically problematic (a video frame at $t=3$ seconds should not be influenced by audio from $t=8$ seconds). The Gated Local-Context Attention mechanism solves this through three coordinated design choices: context packing, attention masking, and adaptive gating.

Audio feature extraction. Raw audio is first processed by Wav2Vec 2.0, a pretrained self-supervised speech model that produces frame-level audio representations capturing both phonetic content (what sounds are being produced) and prosodic features (rhythm, intonation, energy). The paper specifies that representations from "multiple layers" of Wav2Vec 2.0 are merged, which is a common technique to capture both low-level acoustic features (from earlier layers) and higher-level phonetic/semantic features (from later layers). The exact merging strategy (concatenation, summation, learned weighting) is not specified.

Temporal alignment with video FPS. The raw Wav2Vec 2.0 features operate at the model's native frame rate (typically 50Hz or 100Hz), which does not match the video frame rate. The paper applies linear interpolation to resample the audio features to exactly match the original video's FPS. This means that for each video frame, there is now exactly one corresponding audio feature vector.

Audio context packing. A single audio feature vector at time $t$ carries only instantaneous acoustic information—it knows what phoneme is currently being spoken but not what came before or what comes next. Lip movements for a phoneme, however, depend on coarticulation: how the mouth transitions between sounds. To provide this context, the paper uses a sliding window strategy. For each frame's audio feature, it stacks the features from $w$ neighboring frames (where $w=5$) along an extra dimension, creating a "packed" feature that spans a local temporal window. These packed features are then sampled with a stride of $s=4$ to match the VAE's temporal compression (the VAE compresses video temporally by a factor of 4, so one latent frame corresponds to 4 original video frames). Finally, the contextual features are flattened in chronological order, yielding dense 2D features where each position carries $w$ frames of audio context.

What it computes: For a video with $T$ original frames, Wav2Vec 2.0 produces $T$ audio feature vectors. The context packing creates $T$ vectors each of dimension $w \times d_{\text{audio}}$ (where $d_{\text{audio}}$ is the Wav2Vec 2.0 feature dimension). Stride-4 sampling reduces this to $T/4$ vectors, which matches the number of VAE latent frames. The flattening produces a sequence of length $T/4$ where each element is a vector of dimension $w \cdot d_{\text{audio}}$.

Why this form: The window size $w=5$ is chosen empirically through an ablation (Table 5b). A window of $w=1$ (no context) achieves Sync-C = 8.872 and Sync-D = 7.878—already decent synchronization because the attention masking (below) handles temporal alignment, but the model produces "jittery transitions between phonemes and overreaction" because it cannot anticipate upcoming sounds. A window of $w=11$ causes "over-smoothing" where "broad context dilutes the fine-grained cues needed for precise lip synchronization," resulting in worse Sync-D = 9.588. The value $w=5$ strikes the optimal balance, providing enough phonetic context for smooth coarticulation without drowning out the instantaneous signal needed for precise lip shapes.

Audio projector. The packed audio features are fed through a shared audio projector (a learned linear or small MLP) that maps them from Wav2Vec 2.0's feature dimension to the transformer's hidden dimension $H$. This projector is trained from scratch as part of the audio modules. The output is a sequence of audio tokens $\mathbf{a} \in \mathbb{R}^{N_a \times H}$, where $N_a$ is the number of latent frames (the video frame count after VAE temporal compression plus the extra pseudo-frames—see the zero-padding note below).

Attention map constraints (masked cross-attention). The audio tokens $\mathbf{a}$ serve as keys $K$ and values $V$ in a cross-attention layer where the queries $Q$ come from the video (and pseudo-frame) tokens $\mathbf{h}_i$. The crucial modification is the attention mask $\mathbf{M}$, which is a binary matrix that restricts each query position to attend only to its corresponding audio window. The masked attention formula (Equation 4) is:

Attn(Q,K,V,M)=softmax(QKTdk+log⁡M)V\text{Attn}(Q, K, V, \mathbf{M}) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}} + \log \mathbf{M}\right)V

where $d_k$ is the key dimension, and $\log \mathbf{M}$ uses the convention that $\log 0$ is implemented as a large negative constant (effectively $-\infty$) to zero out attention weights for masked positions.

What it computes: For each video token at latent frame position $i$, the attention mask $\mathbf{M}$ has 1s for the $w$ audio tokens corresponding to that frame's temporal window (positions $[i - \lfloor w/2 \rfloor, i + \lfloor w/2 \rfloor]$) and 0s everywhere else. The $\log \mathbf{M}$ term adds 0 for allowed positions and $-\infty$ for disallowed positions. After softmax, the attention weights for disallowed positions become exactly zero, and weights for the $w$ allowed positions sum to 1. This enforces strict temporal locality: a video frame can only "listen to" the audio from its immediate temporal neighborhood.

Why this form: Without the mask, the cross-attention would be global—every video token could attend to every audio token. This would allow information leakage across time (e.g., a frame at 1 second learning from audio at 8 seconds), which would produce temporally incoherent lip movements because the model would have no incentive to learn precise frame-level audio-visual alignment. The mask acts as a strong inductive bias: it tells the model that audio-visual correspondence is temporally local, forcing it to learn fine-grained synchronization rather than relying on spurious long-range correlations. The ablation (Table 2b) confirms this is essential: removing the attention map constraints causes Sync-C to drop from 9.023 to 2.201 and Sync-D to increase from 7.419 to 13.01—a catastrophic collapse in synchronization.

Zero-padding for pseudo-frames. The audio sequence must match the temporal length of the full token sequence, which includes the $N'$ pseudo-frame tokens prepended for reference image injection. Since pseudo-frames correspond to a single reference image (not a temporal sequence), there is no meaningful audio to align with them. The paper's solution is to zero-pad the audio tokens at the positions corresponding to pseudo-frames. This is described as "a trick essential for accommodating reference image injection." Without this padding, the sequence lengths would mismatch and the masked attention would fail. With zero-padding, the pseudo-frame video tokens attend to zero-valued audio tokens, which the softmax effectively ignores (zero keys produce uniform attention weights), allowing the pseudo-frames to process reference image information without audio interference.

Adaptive gating. Directly inserting newly initialized cross-attention layers into a pretrained model can destabilize training because the random initial weights produce noisy outputs that corrupt the pretrained feature distributions before the optimizer has a chance to tune them. The standard solution in the control literature (e.g., ControlNet) is to zero-initialize the weights of the new layers, so they initially output zero and the model behaves exactly like the pretrained version, then gradually learn to produce meaningful outputs.

OmniShow uses a different approach: a learnable gating vector $\mathbf{g} \in \mathbb{R}^{H}$ (where $H$ is the token hidden dimension) initialized to a near-zero value of $1 \times 10^{-5}$. The audio injection is formulated as (Equation 5):

ho=hi+FAttn(hi,a)⊙g\mathbf{h}_o = \mathbf{h}_i + \mathcal{F}_{\text{Attn}}(\mathbf{h}_i, \mathbf{a}) \odot \mathbf{g}

where $\mathbf{h}_i, \mathbf{h}_o \in \mathbb{R}^{(N'+N) \times H}$ are the input and output video token sequences, $\mathbf{a}$ is the audio token sequence, $\mathcal{F}_{\text{Attn}}$ represents the complete attention operation (including the masked attention of Equation 4 and all associated linear projections), and $\odot$ is element-wise (Hadamard) multiplication.

What it computes: The cross-attention output $\mathcal{F}_{\text{Attn}}(\mathbf{h}_i, \mathbf{a})$ is a tensor of the same shape as $\mathbf{h}_i$. The gating vector $\mathbf{g}$ is broadcast across the token dimension and multiplied element-wise, scaling each feature dimension by a learned scalar. The result is added to the residual stream $\mathbf{h}_i$, so when $\mathbf{g} \approx 0$, the model is essentially the pretrained backbone without audio influence.

Why this form: The paper argues that this offers "distinct advantages" over zero-initializing weights within $\mathcal{F}$. First, it provides training stability: the near-zero initialization means early training steps are dominated by the pretrained backbone, preventing the newly inserted attention from corrupting feature distributions. Second, and more importantly, $\mathbf{g}$ acts as an explicit indicator of audio impact. After training, one can inspect the average norm of $\mathbf{g}$ for each transformer block to understand where audio information is most influential. Figure 5 shows that the average norm of $\mathbf{g}$ is substantially higher in dual-stream blocks than in single-stream blocks (the paper shows a plot with dual-stream norms starting around 0.03 and rising to 0.12 over 5000 training steps, while single-stream norms stay below 0.01). This empirical observation directly informs the architectural decision to insert audio cross-attention only into the dual-stream blocks, reducing the parameter overhead from what would be needed for all-block injection.

Strategic placement of audio attention. The decision to insert audio cross-attention only into dual-stream blocks is explicitly motivated by the gating analysis. Dual-stream blocks process video and text through separate pathways before mixing them, which the paper interprets as a natural location for modality-specific conditioning (text in one stream, audio in another). Single-stream blocks jointly process all modalities, and the low gate norms suggest that audio information would not be effectively utilized there. This placement is what keeps the model size increase to approximately 2.5%: the audio modules (projector, cross-attention weights, gating vectors) add only 0.3B parameters to the 12B backbone, totaling 12.3B. For comparison, HuMo's approach of adding audio conditioning increases model parameters by approximately 21.4% (from a 14B base to 17B total).

Ablation of gating. Table 2b shows that removing adaptive gating (i.e., setting $\mathbf{g} = \mathbf{1}$ or removing the multiplication) degrades video quality: AES drops from 0.540 to 0.529. This confirms that the gating mechanism is not just a training stabilizer—it learns a meaningful modulation that improves final generation quality.

Decoupled-Then-Joint Training: Learning from Heterogeneous Data

The central data challenge for HOIVG is that fully-paired quintuplet data (text + human reference + object reference + audio + pose, all aligned with the output video) is extremely scarce. A sample is valid only if every condition is present and correct; a video with great audio synchronization but missing pose annotations must be discarded from a fully-paired training set. However, abundant data exists for individual sub-tasks: R2V datasets (image-video pairs with captions), A2V datasets (talking-head videos with synchronized audio), and smaller collections with pose annotations.

The Decoupled-Then-Joint Training strategy solves this by training on heterogeneous data in a curriculum: first train specialist models on abundant sub-task data, then merge them and fine-tune jointly on the scarce fully-paired data.

Data collection pipeline (Appendix A, Figure 9). The paper constructs four datasets from an "in-house human-centric video pool":

  1. Shot segmentation: Videos are decomposed into individual shots using PySceneDetect, removing scene transitions to ensure each clip contains a single uninterrupted narrative. This is necessary because long videos contain cuts where the visual content changes abruptly, and training on such discontinuities would teach the model to produce jarring transitions.

  2. Multi-dimensional filtering: Clips are filtered by (a) video resolution (ensuring high visual fidelity), (b) aesthetic appeal score, (c) motion intensity (selecting clips with sufficient movement—static clips provide no signal for motion learning), and (d) OCR score (removing clips with excessive on-screen text or watermarks that would add spurious visual patterns). The total curated collection amounts to O(1m) clips, approximately 3500 hours.

  3. Specialized dataset construction:

    • R2V Dataset: Synthesized data using an internal object image database. Human images are retrieved, image editing composes human and object concepts, and an internal image-to-video model generates corresponding videos. Human evaluators filter samples with distortions or "AI-ness." Additionally, reference images are extracted from real videos, super-resolved, and checked for image-video consistency.
    • A2V Dataset: Audio-visual synchronization assessment selects clips where speech audio aligns with visual actions. Misaligned samples are discarded.
    • RA2V Dataset: The most stringently curated dataset, combining criteria from R2V and A2V and adding an expert selection phase. Each clip is independently reviewed for visual quality and semantic consistency across both reference image and audio modalities.
    • RAP2V Dataset: A high-quality subset of RA2V extended with per-frame pose extraction using DWPose, providing the ground truth for fine-tuning the full four-condition model.

Decoupled Training Phase (Figure 2c, left side). Two specialist models are trained independently:

  • R2V Specialist: Trained on the R2V dataset with audio modules discarded—the model retains the exact architecture of the base Waver 1.0 model, learning only text + reference image conditioning. This model never sees audio data, so its weights are optimized purely for visual identity preservation.

  • A2V Specialist: Trained on the A2V dataset with the Gated Local-Context Attention modules active. Following the common A2V paradigm, this model incorporates the first-frame image as an additional condition (using the native channel-concatenation mechanism). It learns audio-visual synchronization but only handles a single reference image (the first frame), not separate human and object reference images.

Why decouple: If one tried to train a unified model from scratch on the heterogeneous mixture, the gradients from different sub-tasks would interfere. R2V training would push weights toward visual identity preservation, while A2V training would push toward audio-responsive motion—and these objectives might conflict in shared parameters. By training specialists independently, each model can fully optimize for its sub-task without gradient interference. Moreover, each specialist sees the full volume of its respective dataset, extracting maximal signal from the heterogeneous data.

Joint Training Phase (Figure 2c, right side). Once specialists are trained, they are fused into a unified model through three steps:

1. Weight interpolation (model merging). The audio modules (cross-attention weights, projector, gating vectors) are inherited entirely from the A2V specialist—the R2V specialist has no audio modules to contribute. The remaining shared parameters (the MMDiT backbone weights, VAE, text encoder) are linearly interpolated:

θmerged=α⋅θA2V+(1−α)⋅θR2V\theta_{\text{merged}} = \alpha \cdot \theta_{\text{A2V}} + (1 - \alpha) \cdot \theta_{\text{R2V}}

with $\alpha = 0.6$ for the A2V model and $0.4$ for the R2V model.

Why this ratio: The paper states that "audio synchronization (relying on fine-grained temporal alignment) is significantly more sensitive to weight disruption than visual identity (relying on global appearance features)." In practice, this means that diluting the A2V weights too much would destroy the hard-won audio-visual synchronization, while the R2V model's visual identity capabilities are more robust to interpolation. The 60/40 split preserves the delicate temporal alignment while still incorporating the R2V model's visual knowledge. The specific values (0.6, 0.4) appear to be empirically chosen rather than optimized through a sweep; the paper does not report ablation over the interpolation ratio.

2. Emergent RA2V capability. Remarkably, the merged model—before any joint training—already exhibits zero-shot Reference+Audio-to-Video generation (Figure 4). The paper describes this as the merged model "successfully generates videos that respect both reference images and audio inputs" despite never being explicitly trained on the RA2V task. This is presented as evidence that the weight-space combination of specialist models produces a representation that generalizes beyond either specialist's training distribution. The paper explicitly flags this as "an inspiring discovery that we hope will motivate further exploration."

3. Joint fine-tuning and pose introduction. The merged model is then trained on the full RA2V dataset, followed by fine-tuning on a high-quality subset (the "Aesthetic Tuning" step) to enhance naturalness and visual quality. Finally, pose conditioning is introduced in the last stage using the RAP2V dataset. The paper explicitly states that pose is introduced only in the final fine-tuning stage to prevent overfitting, because pose is "a strong supervision signal" that could dominate the training objective and cause the model to ignore the other conditions (text, reference images, audio). By delaying pose introduction, the model first learns to balance text, image, and audio conditioning, then adds the explicit motion signal as a refinement.

Comparison with naive training strategies (Table 2c). The ablation compares three alternatives, all run on 32 GPUs for fairness:

  • Single-stage (Only RA2V): Train directly on RA2V data without the specialist phase. This achieves NexusScore = 0.345, Sync-D = 13.11, and AES = 0.453—all substantially worse than the proposed approach, due to "poor convergence" from training a complex multi-condition model on limited fully-paired data.
  • Multi-stage (R2V → RA2V): Pretrain on R2V data, then fine-tune on RA2V. This improves AES to 0.473 but Sync-D remains poor at 13.23 because the model never learns audio-visual synchronization.
  • Multi-stage (A2V → RA2V): Pretrain on A2V data, then fine-tune on RA2V. This improves Sync-D to 7.38 (good synchronization) but NexusScore drops to 0.342 because the model never learns strong reference image conditioning.

The proposed Decoupled-Then-Joint Training achieves NexusScore = 0.364 and Sync-D = 8.14 with AES = 0.474—the best overall trade-off, demonstrating that the specialist fusion captures the complementary strengths of both sub-task models.

HOIVG-Bench: A Standardized Evaluation Suite

The paper constructs HOIVG-Bench as both a contribution and the experimental platform for all quantitative results in Section 4. Its design reflects the specific challenges of evaluating simultaneous multimodal conditioning.

Sample construction pipeline. The 135 samples are created through a multi-stage process designed to produce realistic, diverse, and legally safe evaluation data:

  1. Video curation: Raw videos are selected from an in-house library based on three criteria: duration exceeding 4 seconds, clear human-object interactions, and diversity in human attributes (gender, age, ethnicity) and object categories (daily necessities, tools, etc.). The paper specifies that videos are standardized to 5-second clips at 720p resolution in portrait mode for evaluation, even though OmniShow supports generation up to 10 seconds—this standardization ensures fair comparison with baselines that only support short clips.

  2. Object image acquisition: Rather than simply cropping objects from the original video (which would create an unrealistic "oracle" setting where the reference image perfectly matches the video's object appearance), the paper uses Nano Banana (Google's image generation API) to modify the original objects' textures and colors while adding fine-grained details. This simulates real-world generation scenarios where the provided reference image may differ in lighting, angle, or texture from how the object appears in the desired video.

  3. Human image acquisition: For privacy and identity protection, human reference images are generated from video screenshots using Nano Banana. These generated images maintain stylistic similarity to the original subjects (pose, clothing style, general appearance) while altering identity features (facial structure, skin details). This serves dual purposes: legal compliance for public benchmark release and testing the model's ability to generalize from synthetic reference images to natural video output. The paper acknowledges this as a "carefully deliberated choice" and notes that manual checks filter out images with "noticeable AI-ness."

  4. Pose extraction: DWPose extracts per-frame human pose skeletons from the original videos, producing the ground-truth pose sequence that serves as both a conditioning input and an evaluation target.

  5. Audio synthesis: A two-stage process using GPT-4o and ElevenLabs: first, GPT-4o generates a speech script describing the target object; then, GPT-4o analyzes the human reference image's gender and age attributes, and ElevenLabs synthesizes speech audio with matching timbre. This produces semantically consistent audio that tests both speech synchronization and natural voice-quality matching.

Evaluation metrics across five dimensions:

  • Text Alignment (TA): VideoReward predicts a text alignment score. Higher is better.
  • Reference Consistency: FaceSim (facial identity similarity, from OpenS2V) and NexusScore (general subject consistency, also from OpenS2V). Higher is better.
  • Pose Accuracy: Average Keypoint Distance (AKD, lower is better) and Percentage of Correct Keypoints (PCK) at a 5% error threshold (higher is better), both computed using DWPose on generated frames.
  • Audio-Visual Synchronization: Sync-C (synchronization confidence, higher is better) and Sync-D (synchronization distance, lower is better) from the SyncNet literature (Chung and Zisserman, 2016).
  • Video Quality: Aesthetic score (AES) and Image Quality Assessment (IQA) from VBench, plus overall visual quality (VQ) and motion quality (MQ) from VideoReward. Higher is better for all four.

What this benchmark tests that prior benchmarks don't: HOIVG-Bench evaluates the simultaneous satisfaction of all conditions. A model might achieve high FaceSim by copying reference appearance but produce unnatural motion (captured by low MQ). It might achieve high Sync-C by animating lips but fail to follow the pose sequence (low PCK). The multi-dimensional metric suite captures these trade-offs, and the paper's main results (Table 1) show that OmniShow avoids the catastrophic failures in any single dimension that plague competing methods—it achieves balanced performance across all five axes.

4. Key Insights and Innovations

Innovation 1: The Philosophy of Minimalist Intervention as a Unifying Architectural Principle

The paper's most intellectually distinctive contribution is not any single conditioning mechanism but the explicit articulation and validation of a design philosophy for multimodal video generation: that the way to add controllability without sacrificing quality is to reuse and minimally extend the base model's native conditioning pathways, rather than grafting on modality-specific modules. This is named in Section 5 as the "Philosophy of Minimalist Intervention" and is enacted through two concrete design choices—channel concatenation for visual modalities (reference images + pose) and strategically placed, gated cross-attention for audio—but the novelty lies in the diagnosis of why prior approaches fail, not just in the mechanisms themselves.

The field's dominant approach to multimodal conditioning has been additive: add a new cross-attention layer for each new modality (as in HuMo, which grows ~21.4% in parameter count from a 14B base to 17B), or construct hybrid token sequences that interleave conditioning tokens with video tokens (the token-concatenation approach that the ablation in Table 2a and Figure 3 shows underperforms). These approaches treat each modality as requiring its own architectural subsystem. The unstated assumption is that a pretrained DiT's internal representations are general-purpose enough that new pathways can be attached without disrupting them.

OmniShow's counter-thesis is that the base model's native input structure is a fragile, finely calibrated prior that aggressive architectural modification disrupts. Waver 1.0 already has a conditioning interface—channel concatenation with binary masks—designed to toggle between T2V and I2V. Rather than building alongside this interface, OmniShow extends it: reference images and pose are injected through the same channel-concatenation pathway, just with expanded capacity (pseudo-frame tokens). Audio, which cannot fit naturally into channel concatenation due to its 1D temporal structure, gets cross-attention—but its placement is determined not by architectural convenience but by empirical analysis of learned gate activations (Figure 5), which reveals that audio information flows primarily through dual-stream blocks. The result is a model only ~2.5% larger than its 12B backbone (12.3B total), compared to ~21.4% inflation in HuMo.

This is not merely an efficiency win. The conceptual move is from treating multimodal conditioning as an architectural integration problem (how do we connect new modules?) to treating it as a representation preservation problem (how do we inject new signals without corrupting existing ones?). The evidence for this framing's validity comes from the ablation of conditioning methods (Table 2a, Figure 3): token concatenation—the intuitive "just add more tokens" approach—achieves FaceSim of 0.601 vs. 0.707 for channel concatenation, a substantial gap in identity preservation. The paper attributes this to the model's ability to "efficiently transfer the pretrained I2V capability" when the native input format is preserved.

This is a fundamental reframing rather than an incremental improvement. It changes the question from "what new modules do we need?" to "what is the minimal perturbation to the base model that still conveys the conditioning signal?" and provides an empirical methodology (gate activation analysis) for answering that question. It also implies a counterintuitive prescription: when adding a new conditioning modality, the best architecture may be one that looks like it shouldn't work—repurposing a channel-concatenation slot meant for first-frame conditioning to carry pose sequences, or inserting cross-attention only into a subset of transformer blocks based on post-hoc gate analysis—because respecting the pretrained feature distribution matters more than architectural expressiveness.

Innovation 2: Weight-Space Merging of Specialist Models as a Data-Efficient Training Paradigm

The Decoupled-Then-Joint Training strategy contains a finding that the paper correctly identifies as "an inspiring discovery" (Section 5): that merging separately trained R2V and A2V specialist models via simple weight interpolation produces a model with emergent zero-shot RA2V capability (Figure 4), without any joint training on paired Reference+Audio-to-Video data. This is conceptually distinct from standard multi-stage training or curriculum learning, and its significance extends beyond the specific application.

Prior approaches to training models on multiple conditioning modalities fall into roughly three categories. The first is joint training from scratch on fully-paired data, which the paper shows is infeasible for HOIVG due to data scarcity—the single-stage baseline in Table 2c achieves the worst results across the board (NexusScore = 0.345, Sync-D = 13.11). The second is sequential fine-tuning (train on one sub-task, then fine-tune on another), which suffers from catastrophic forgetting: the R2V→RA2V baseline preserves visual identity but never learns audio synchronization (Sync-D = 13.23), while the A2V→RA2V baseline preserves synchronization but loses reference consistency (NexusScore = 0.342). The third is joint training on a mixture of heterogeneous sub-task data, which the paper argues suffers from gradient interference between conflicting objectives—though this is asserted rather than demonstrated empirically.

OmniShow's approach—training specialists independently, then fusing their weights via linear interpolation (with a 60/40 A2V/R2V ratio, chosen because "audio synchronization is significantly more sensitive to weight disruption than visual identity") and jointly fine-tuning the merged model—is qualitatively different. The critical observation is that the merged model, before any joint training, already generates videos that simultaneously respect reference images and audio inputs. This means the weight-space interpolation itself synthesizes a representation that combines capabilities from both specialists, without requiring explicit paired training data to teach the combination.

This is a fundamental finding with implications beyond HOIVG. It suggests that for tasks requiring the simultaneous satisfaction of multiple conditions, one can amortize the cost of data collection: instead of needing expensive N-way paired data for all condition combinations, one needs only abundant single-condition data (to train specialists) plus a modest amount of N-way paired data (for joint fine-tuning). The weight interpolation step bridges the gap, producing a model that already approximately satisfies all conditions before seeing any jointly-annotated examples. This finding connects to the broader literature on model merging and weight interpolation in LLMs and vision models, but applies it to the specific challenge of conditioning composability—demonstrating that controllability itself can emerge through weight-space arithmetic.

The paper is appropriately cautious about this claim: it presents the emergent RA2V result as a qualitative observation (Figure 4) rather than a quantitative ablation, and does not explore the sensitivity of emergence to the interpolation ratio, the choice of which specialists to merge, or the architectural properties that enable it. These are flagged as future work. The novelty here is in identifying the phenomenon and demonstrating its practical utility, not in fully characterizing it. For practitioners, the immediate implication is a training recipe: when faced with heterogeneous sub-task data for a multimodal generation problem, train specialists first and merge them—don't try to train jointly from the start.

Innovation 3: Difficulty-Equivalent Diagnosis of Conditioning Interference via Gate Activation Analysis

Section 3.3 introduces a small but conceptually elegant diagnostic tool: the use of learned gating vectors not just as a training stabilizer but as a post-hoc indicator of modality impact, which directly informs architectural decisions. This is a methodological innovation rather than an architectural one, and it solves a concrete design problem that is typically addressed through expensive ablation sweeps.

When adding a new conditioning pathway (e.g., audio cross-attention) to a pretrained DiT, the designer faces a placement question: in which transformer blocks should the new attention layers be inserted? The naive approach is to insert them everywhere (all blocks), which maximizes capacity but also maximizes disruption and parameter overhead. The conservative approach is to insert them in a specific layer range (e.g., the middle blocks) based on intuition about where semantic vs. perceptual processing occurs. Both approaches require training multiple model variants and comparing them—an expensive proposition at the 12B scale.

OmniShow's alternative is to instrument the model with learnable gating vectors initialized to near-zero (1e-5) and placed after each candidate audio cross-attention layer. During training, these gates learn to scale the audio influence per feature dimension per block. After training, one can simply inspect the average L2 norm of the gate vector for each block. Figure 5 shows a clear pattern: gates in dual-stream blocks develop substantially larger norms (~0.12 after 5000 steps) than gates in single-stream blocks (<0.01). This tells the designer, without running any ablation, that audio information is predominantly utilized in dual-stream blocks, and that audio cross-attention can be removed from single-stream blocks with minimal performance impact.

The elegance of this approach is that it converts a design choice (where to place modules) into a measurement problem (where does the model actually use the modality). The gates act as learned sensitivity indicators: if a block's gate stays near zero throughout training, the model is effectively ignoring audio at that location, and the cross-attention layer there is dead weight. This is analogous to pruning based on weight magnitudes, but applied at the level of modality-specific architectural components rather than individual parameters.

This is an incremental methodological advance rather than a fundamental conceptual shift, but it is practically valuable because it generalizes: any future work adding any new conditioning modality to a DiT can use the same gating instrumentation to determine optimal placement without running a combinatorial ablation over all possible insertion points. The paper itself uses this insight to justify inserting audio attention only into dual-stream blocks, keeping the parameter overhead to ~2.5%. The ablation in Table 2b confirms that removing adaptive gating degrades quality (AES drops from 0.540 to 0.529), validating that the gates are learning meaningful modulation, not just acting as dead-code detectors.

Innovation 4: HOIVG-Bench as a Standardized Multi-Dimensional Stress Test

The creation of HOIVG-Bench (Section 3.5) is a contribution to evaluation infrastructure, but its intellectual significance lies in what it reveals about the limitations of existing evaluation paradigms and the specific failure modes it is designed to surface. This is not just "here is a new dataset"—it is a diagnostic instrument for a specific pathology that single-modality benchmarks miss.

Existing video generation benchmarks evaluate conditioning dimensions independently. VBench measures overall video quality and aesthetics. OpenS2V measures subject consistency in the R2V setting. Sync-C and Sync-D measure audio-visual synchronization in the A2V setting. AKD and PCK measure pose accuracy in pose-guided generation. A model can score well on all of these in isolation and still fail catastrophically when asked to satisfy them simultaneously—because the conditioning signals can interfere. A model that tightly follows a pose sequence might distort the reference subject's appearance. A model that perfectly synchronizes lips to audio might ignore the text prompt's specification of the scene. A model that preserves reference identity might produce frozen, unnatural motion.

HOIVG-Bench is designed as a simultaneous stress test: every sample contains all four conditioning signals (text, human reference, object reference, audio, pose), and the evaluation suite measures all five quality dimensions (text alignment, reference consistency, audio-visual sync, pose accuracy, video quality) on the same generated outputs. This allows the benchmark to surface cross-modal interference effects that single-modality evaluation misses. The paper's results in Table 1 demonstrate exactly this: HuMo-17B achieves strong FaceSim (0.805) in the RA2V setting but this comes at some cost to audio-visual synchronization (Sync-C = 8.013 vs. OmniShow's 8.612); AnchorCrafter achieves reasonable pose accuracy (AKD = 0.229) in the RP2V setting but at catastrophic cost to reference consistency (FaceSim = 0.404, NexusScore = 0.215) and video quality (AES = 0.499). These trade-offs would be invisible if one only evaluated on single-modality benchmarks.

The construction methodology is also notable for its realism-oriented design choices. The decision to use AI-generated human and object reference images (via Nano Banana) rather than cropped frames from the target video is explicitly motivated by deployment realism: in real e-commerce workflows, the provided reference image will differ in lighting, angle, and texture from how the subject appears in the desired video. A benchmark that uses perfectly matched reference images (cropped from ground-truth frames) would create an artificially easy setting that does not reflect practical use. Similarly, the use of synthesized audio (GPT-4o script → ElevenLabs speech) rather than the original video's audio track ensures that the audio condition is semantically consistent but acoustically novel—the model cannot cheat by memorizing audio-visual correlations from training.

This is a fundamental infrastructure contribution rather than an algorithmic one, but its significance for the field is substantial: it establishes a standardized, reproducible, and diagnostically rich evaluation protocol for a task that previously had no shared benchmark, enabling fair comparison between future HOIVG systems. The inclusion of privacy-compliant, AI-generated reference images also sets a precedent for how to construct publicly releasable human-centric video benchmarks without running afoul of identity and consent concerns—a non-trivial practical contribution given increasing regulatory and ethical scrutiny of facial data in research datasets.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. All experiments are conducted on HOIVG-Bench, the benchmark introduced in this paper (Section 3.5). It consists of 135 carefully curated samples, each equipped with a detailed text caption, a human reference image, an object reference image, semantically aligned audio, and a coherent pose sequence. Samples were constructed from an in-house video library through a pipeline involving shot segmentation, multi-dimensional filtering, AI-based reference image generation (Nano Banana for images, ElevenLabs for audio), and DWPose-based pose extraction. The benchmark is standardized to 5-second video clips at 720p resolution in portrait mode for all quantitative evaluations. A word cloud of text prompts (Figure 6a) and a motion intensity distribution histogram (Figure 6b) demonstrate diversity in interaction scenarios and dynamic range across the 135 samples.

  • Base model(s). All experiments use Waver 1.0, a 12B-parameter MMDiT-based video generation model (Zhang et al., 2025) trained with flow matching. OmniShow extends this backbone to 12.3B parameters (a ~2.5% increase) through the addition of Gated Local-Context Attention modules. The paper argues Waver 1.0 is representative of state-of-the-art video diffusion transformers and its native channel-concatenation conditioning paradigm (supporting both T2V and I2V) provides the foundation that OmniShow's minimalist intervention strategy extends. For comparison, the paper also references the base model's scale relative to competitors (HuMo-17B at 17B parameters, Phantom-14B at 14B).

  • Metrics. Evaluation spans five dimensions with ten metrics drawn from established prior work:

    • Text Alignment (TA): VideoReward (Liu et al., 2025) predicts a text alignment score. Higher is better.
    • Reference Consistency: FaceSim and NexusScore from OpenS2V (Yuan et al., 2025), measuring facial identity similarity and general subject consistency respectively. Higher is better for both.
    • Audio-Visual Synchronization: Sync-C (synchronization confidence, higher is better) and Sync-D (synchronization distance, lower is better) from SyncNet (Chung and Zisserman, 2016). These are standard metrics in the talking-head and A2V literature.
    • Pose Accuracy: Average Keypoint Distance (AKD, lower is better) and Percentage of Correct Keypoints (PCK) at a 5% error threshold (higher is better), both computed using DWPose (Yang et al., 2023) on generated frames.
    • Video Quality: Aesthetic score (AES) and Image Quality Assessment (IQA) from VBench (Huang et al., 2024), plus overall visual quality (VQ) and motion quality (MQ) from VideoReward. Higher is better for all four.

    The symbol "—" in Table 1 indicates that a metric is not applicable in the corresponding setting (e.g., Sync-C is "—" for R2V and RP2V baselines that lack audio conditioning).

  • Baselines. The paper compares against six existing methods across their supported conditioning settings:

    • HunyuanCustom (Hu et al., 2025): 13B parameters, supports R2V and RA2V.
    • HuMo (Chen et al., 2025): Evaluated in both 1.7B and 17B variants, supports R2V and RA2V.
    • VACE (Jiang et al., 2025): 14B parameters, supports R2V and RP2V.
    • Phantom (Liu et al., 2025): Evaluated in 1.3B and 14B variants, supports R2V.
    • AnchorCrafter (Xu et al., 2024): 1.5B parameters, supports RP2V.

    For the RAP2V setting where no existing method supports all four conditions, the paper constructs a cascaded baseline: VACE (for RP2V generation) followed by LatentSync (Li et al., 2024) (for audio-driven lip synchronization), evaluated in Appendix C (Table 4). For the A2V-specific evaluation, comparisons include FantasyTalking (Wang et al., 2025), HunyuanVideo-Avatar (Chen et al., 2025), Hallo3 (Cui et al., 2025), MultiTalk (Kong et al., 2025), and OmniAvatar (Gan et al., 2025), evaluated on the EMTD benchmark (Meng et al., 2025) in Table 3.

  • Generation budget / compute accounting. The paper does not use a generation budget metric (e.g., number of samples or FLOPs) in the manner of text-to-image or LLM scaling analyses. Instead, all methods are compared at their default inference settings with standardized output specifications: 5-second video clips at 720p resolution in portrait mode. This ensures that the quality of a single generated output is compared fairly, but it does not account for differences in inference cost or speed across models. Training compute is reported in Section 4.1: large-scale training uses 128 GPUs (each 80GB) with FSDP and Ulysses-style sequence parallelism (size 8), utilizing BF16 mixed precision. The AdamW optimizer uses a learning rate of 3 × 10⁻⁵ and weight decay of 0.01. Ablation experiments (Section 4.3) are conducted on 8 GPUs for fairness. Training spans two resolutions (480p and 720p) with the 720p phase following an initial 480p phase. All features are extracted offline to maximize training efficiency.

  • Cross-validation / statistical protocol. No cross-validation protocol is described. The HOIVG-Bench test set of 135 samples is used for all quantitative evaluations in Tables 1, 3, and 4. Human evaluation (Figure 8) engages 30 participants for RA2V and 33 participants for RP2V, assessing randomly selected subsets of 20 samples each in side-by-side comparisons. Evaluators judge four dimensions: text alignment, reference consistency, audio-visual synchronization (or pose accuracy, depending on setting), and overall video quality. No confidence intervals or statistical significance tests are reported for any quantitative metric.

Main Quantitative Results

R2V (Text + Reference Image to Video)

The R2V setting evaluates how well models preserve subject identity from reference images while following text prompts, without audio or pose conditioning. Results are reported in the top section of Table 1.

OmniShow achieves competitive reference consistency with state-of-the-art specialized models while delivering the best video quality. FaceSim reaches 0.874, statistically tied with Phantom-14B at 0.876 and HuMo-17B at 0.843, and substantially ahead of HunyuanCustom (0.440) and HuMo-1.7B (0.647). NexusScore is 0.389, the highest among all methods (next best: VACE at 0.368, Phantom-14B at 0.366). This means OmniShow preserves both facial identity and general subject appearance at or above the level of models specialized for R2V.

On video quality metrics, OmniShow leads across the board: AES = 0.468 (vs. next best Phantom-1.3B at 0.459), IQA = 0.740 (vs. Phantom-14B at 0.741, effectively tied), VQ = 11.12 (vs. Phantom-14B at 10.93), and MQ = 5.885 (vs. Phantom-14B at 5.517). The motion quality advantage (5.885 vs. 5.517) is particularly notable given that OmniShow achieves this while matching reference consistency—there is no evidence of a quality-consistency trade-off.

Text alignment shows an interesting pattern: OmniShow achieves TA = 7.746, which is lower than Phantom-14B (8.609), VACE (8.413), and Phantom-1.3B (8.342). This is the only dimension where OmniShow does not lead or tie for the lead in R2V. The paper does not discuss this gap explicitly, but it may reflect the fact that OmniShow's architecture is optimized for multimodal conditioning and the text pathway may receive relatively less dedicated capacity compared to models designed purely for text+image-to-video tasks.

The qualitative comparison (Figure 7a) provides context: unlike baselines that "often rigidly paste objects onto human subjects at implausible sizes," OmniShow "ensures both visual fidelity and realistic composition." This suggests that the slightly lower TA score may reflect a different text interpretation strategy—prioritizing visual plausibility over literal text adherence—though this interpretation is not explicitly stated in the paper.

RA2V (Text + Reference Image + Audio to Video)

The RA2V setting tests the simultaneous satisfaction of reference identity preservation and audio-visual synchronization. This is the most practically demanding setting where multiple baselines exist for comparison. Results are in the middle section of Table 1.

OmniShow achieves the best audio-visual synchronization while matching or exceeding reference consistency of dedicated baselines. Sync-C reaches 8.612, outperforming HuMo-17B (8.013) and HuMo-1.7B (7.234). Sync-D is 7.608, the lowest (best) among all methods (HuMo-17B: 8.316, HunyuanCustom: 10.08). The gap in Sync-C of approximately 0.6 points over the 17B HuMo variant and 1.4 points over the 1.7B variant represents a meaningful improvement in lip synchronization quality.

Reference consistency is competitive: FaceSim = 0.810 (HuMo-17B: 0.805, effectively tied) and NexusScore = 0.369 (HuMo-17B: 0.344, HunyuanCustom: 0.350). OmniShow achieves this while being substantially smaller than HuMo-17B (12.3B vs. 17B parameters), suggesting more efficient use of model capacity.

Video quality metrics show OmniShow leading: AES = 0.465 (vs. HuMo-17B at 0.439), VQ = 10.86 (vs. HuMo-17B at 10.27), MQ = 5.554 (vs. HuMo-17B at 4.269). The motion quality gap is substantial—OmniShow's MQ of 5.554 versus HuMo-17B's 4.269 suggests that the Gated Local-Context Attention enables more natural body movement alongside speech, avoiding what the qualitative analysis (Figure 7b) describes as the "overreaction" and "frozen body" issues in prior methods. HunyuanCustom achieves the highest TA at 7.289 but shows markedly worse audio-visual synchronization (Sync-D = 10.08), suggesting a text-prioritization strategy that doesn't transfer well to multimodal settings.

Human evaluation (Figure 8, left panel) reinforces these quantitative results. In side-by-side comparisons with HuMo-17B, evaluators preferred OmniShow on video quality (51.2% vs. 21.2%), audio-visual sync (41.0% vs. 30.7%), reference consistency (42.2% vs. 30.3%), and text alignment (34.0% vs. 42.7%). Note that text alignment is the only dimension where HuMo-17B was preferred (42.7% vs. 34.0%), consistent with the quantitative TA scores where HuMo-17B (8.146) slightly edges OmniShow (8.093). The paper attributes the overall human preference to "smoother temporal dynamics and richer visual details" that "might be overlooked by frame-level metrics."

RP2V (Text + Reference Image + Pose to Video)

The RP2V setting tests the combination of reference identity preservation with explicit pose-driven motion control. Results are in the bottom section of Table 1.

OmniShow achieves substantially better pose accuracy than baselines while maintaining competitive text alignment and the best video quality. Pose metrics: AKD = 0.174 (vs. VACE at 0.206, AnchorCrafter at 0.229) and PCK = 0.460 (vs. VACE at 0.336, AnchorCrafter at 0.176). The PCK advantage is large—OmniShow correctly places 46% of keypoints within a 5% error threshold versus 33.6% for VACE and only 17.6% for AnchorCrafter. This demonstrates that the Unified Channel-wise Conditioning effectively transmits precise pose information through the channel-concatenation pathway.

However, reference consistency drops relative to the R2V setting: FaceSim = 0.474 and NexusScore = 0.418. These are lower than VACE (FaceSim 0.600, NexusScore 0.352) and substantially below OmniShow's own R2V FaceSim of 0.874. The paper explicitly acknowledges this trade-off: "precise pose adherence introduces viewpoint shifts and facial morphology changes that affect FaceSim" (Section 4.2). This is an honest admission of a genuine tension—when the pose sequence specifies viewpoints or head orientations different from the reference image, the model must distort the reference appearance to follow the pose, and FaceSim penalizes this distortion.

Text alignment also drops: TA = 6.526 vs. VACE at 7.690 and AnchorCrafter at 2.669. AnchorCrafter's catastrophic TA (2.669) is consistent with its poor reference consistency—the model appears to largely ignore both text and reference conditions, producing low-quality outputs that happen to follow the pose. OmniShow's lower TA relative to VACE may reflect the same pose-following tension: strong pose conditioning can override text-specified scene details.

Video quality tells a different story: OmniShow achieves VQ = 10.28 (vs. VACE at 10.14) and IQA = 0.722 (vs. VACE at 0.712), with MQ at 4.937 (vs. VACE at 5.393). The motion quality is slightly lower than VACE, possibly because VACE is less constrained by pose following and can produce more fluid (if less accurate) motion.

Human evaluation (Figure 8, right panel) shows a clear OmniShow advantage: evaluators preferred OmniShow on video quality (52.3% vs. 18.3%), pose accuracy (56.8% vs. 13.2%), reference consistency (30.5% vs. 44.8%), and text alignment (36.2% vs. 30.6%). The reference consistency preference is notable: human evaluators preferred VACE on this dimension (44.8% vs. 30.5%) despite OmniShow's higher NexusScore (0.418 vs. 0.352). This reinforces that FaceSim, which penalizes viewpoint-induced appearance changes, may not align with human perception of "does this look like the reference person" when the pose is different.

RAP2V (Full Four-Condition Setting)

No existing method supports all four conditions, so OmniShow is compared against a cascaded baseline (VACE for RP2V + LatentSync for lip synchronization) in Appendix C, Table 4. This is a critical experiment because it tests whether end-to-end joint conditioning outperforms sequential composition of specialized tools—the default practical alternative.

OmniShow outperforms the cascaded baseline across all evaluation dimensions. On audio-visual synchronization: Sync-C = 7.699 vs. 7.016 for the cascaded baseline, and Sync-D = 7.674 vs. 7.823. The improvement is modest but consistent, suggesting that joint conditioning enables better lip-sync than post-hoc correction. More importantly, the cascaded baseline introduces artifacts: the qualitative comparison (Figure 10) shows that "conducting isolated lip-synchronization generation as a post-processing step... leads to visual artifacts and blur, especially under mouth occlusions." These artifacts manifest quantitatively in lower video quality: OmniShow's VQ = 11.06 vs. 10.05, MQ = 5.880 vs. 3.911, and AES = 0.424 vs. 0.417.

Pose accuracy also improves: AKD = 0.172 (cascaded: 0.198) and PCK = 0.478 (cascaded: 0.340). The PCK improvement from 34% to 47.8% is substantial and suggests that adding lip-synchronization as a post-processing step can degrade the pose accuracy achieved by the RP2V model—likely because the lip-sync model modifies facial regions in ways that shift keypoint positions.

Reference consistency shows a similar pattern: FaceSim = 0.645 vs. 0.591, NexusScore = 0.353 vs. 0.341. End-to-end training preserves subject appearance better than cascaded post-processing.

A2V Evaluation on EMTD Benchmark (Audio-Driven Setting)

To validate that the audio conditioning mechanism itself is effective before combination with other modalities, the paper evaluates the A2V specialist model (OmniShow-A2V, which uses only the Gated Local-Context Attention on a first-frame image) against dedicated audio-to-video methods on the EMTD benchmark. Results are in Table 3.

OmniShow-A2V achieves state-of-the-art audio-visual synchronization. Sync-C reaches 6.49, outperforming MultiTalk (6.34), OmniAvatar (5.40), Hallo3 (4.26), and HunyuanVideo-Avatar (4.89). Sync-D is 8.97, competitive with MultiTalk (8.47) and better than Hallo3 (10.22). Video quality metrics: AES = 1.51 (best among compared methods, next best Hallo3 at 1.48) and IQA = 2.26 (competitive with Hallo3 at 2.31 and OmniAvatar at 2.16).

This result is important because it isolates the audio conditioning component and demonstrates it is genuinely state-of-the-art, not just "good enough for the unified model." The Gated Local-Context Attention, with its context packing (w=5), temporal masking, and adaptive gating, produces better synchronization than purpose-built A2V architectures. This validates that the audio injection design is not compromised by its minimalist philosophy—it is genuinely competitive with (and in some metrics superior to) larger, more architecturally invasive approaches.

Ablation Studies and Robustness Checks

All ablation experiments are conducted on 8 GPUs for fairness, as specified in Section 4.3. The full model (OmniShow) is included in each ablation table for reference.

Unified Channel-wise Conditioning Ablations (Table 2a, Figure 3)

Token concatenation vs. channel concatenation: Replacing the proposed channel-concatenation injection with the token-concatenation approach ("hybrid tokens" from OmniControl, Tan et al., 2025) degrades FaceSim from 0.707 to 0.601 and NexusScore from 0.353 to 0.344, with AES also dropping slightly (0.471 to 0.466). This is the paper's key architectural claim: preserving the native input structure matters more for quality than the expressiveness of the conditioning representation. The qualitative example in Figure 3 shows that token concatenation fails to accurately preserve the visual appearance of the reference subject, producing a noticeably different face.

Reference reconstruction loss: Removing the flow matching loss on pseudo-frame tokens reduces FaceSim from 0.707 to 0.678 and NexusScore from 0.353 to 0.352, with AES unchanged at 0.466. The FaceSim drop confirms that explicit reconstruction supervision on the pseudo-frames is necessary to force the model to retain high-fidelity semantic details from the reference images. Without this loss, the pseudo-frames can become "dead" positions that the model learns to route around rather than actively using for appearance conditioning.

Gated Local-Context Attention Ablations (Table 2b)

Audio context window: Removing audio context packing (equivalent to w=1, no temporal neighborhood) produces Sync-C = 8.872 and Sync-D = 7.878—both worse than the full model (9.023 and 7.419). The paper describes the resulting behavior as "jittery transitions between phonemes and overreaction," suggesting that the model overfits to instantaneous audio cues without the smoothing effect of temporal context. This is a non-obvious finding: one might expect that removing context would hurt synchronization (which it does), but the mechanism is interesting—the model becomes too responsive, producing exaggerated movements that follow every audio fluctuation rather than natural coarticulated speech.

Attention map constraints: Removing the temporal masking (allowing each video frame to attend to all audio tokens globally) causes a catastrophic collapse in synchronization: Sync-C drops from 9.023 to 2.201 and Sync-D increases from 7.419 to 13.01. This is the largest single-factor degradation in any ablation and confirms that temporal locality is not just a helpful inductive bias—it is essential. Without masking, the model has no incentive to learn fine-grained frame-level alignment and defaults to producing lip movements uncorrelated with the audio.

Adaptive gating: Removing the learnable gating vector (setting g = 1 or removing the multiplication in Equation 5) keeps synchronization metrics essentially unchanged (Sync-C = 8.872 vs. 9.023 for the full model, Sync-D = 7.819 vs. 7.419) but degrades video quality: AES drops from 0.540 to 0.529. This suggests the gating mechanism primarily serves as a quality stabilizer—it prevents the audio injection from corrupting visual features—rather than being necessary for synchronization itself. The gates learn to modulate how much audio influence reaches each feature dimension, and removing this modulation introduces subtle artifacts that AES (aesthetic quality) detects but Sync-C (a binary synchronization signal) does not.

Decoupled-Then-Joint Training Ablations (Table 2c)

Single-stage training (Only RA2V): Training directly on RA2V data without the specialist pre-training phase yields NexusScore = 0.345, Sync-D = 13.11, and AES = 0.453—all substantially worse than the proposed approach (0.364, 8.14, 0.474). This confirms the data scarcity problem: when training data is limited to fully-paired samples, the model fails to converge to good solutions for both reference consistency and audio synchronization simultaneously.

Multi-stage (R2V → RA2V): Pretraining on R2V data then fine-tuning on RA2V produces NexusScore = 0.360 (competitive with the full model at 0.364) and AES = 0.473 (also competitive at 0.474), but Sync-D remains at 13.23—essentially no better than single-stage training. The model never learns audio-visual synchronization because the R2V pretraining phase doesn't expose it to audio, and the RA2V fine-tuning phase cannot overcome this gap with limited data. This is a classic catastrophic forgetting / insufficient adaptation problem: the R2V specialist is a local minimum that doesn't easily accommodate new modalities.

Multi-stage (A2V → RA2V): The reverse curriculum—pretraining on A2V then fine-tuning on RA2V—produces good synchronization (Sync-D = 7.38, actually better than the full model's 8.14) but poor reference consistency: NexusScore = 0.342 vs. 0.364. The model learns audio synchronization well but the single-reference-image conditioning from A2V doesn't transfer to the multi-reference-image setting, and the limited RA2V fine-tuning data can't fully teach this capability. AES also drops to 0.456.

The key insight from this ablation triplet is that sequential multi-stage training produces asymmetric knowledge transfer: each curriculum excels at what it learned first and struggles with what it learns second. The Decoupled-Then-Joint approach avoids this by training specialists independently and then merging them in weight space—both capabilities transfer simultaneously because both sets of weights are combined before any joint training occurs.

Positional Encoding for Pseudo-Frames (Table 5a, Appendix D.1)

RoPE strategy: Three strategies for assigning positional embeddings to pseudo-frame tokens are compared:

  • Native (continuous indexing from T=0): FaceSim = 0.707, NexusScore = 0.353, AES = 0.471.
  • Temporal Shift (negative indices T=-1, -2): FaceSim = 0.675, NexusScore = 0.351, AES = 0.468.
  • Spatiotemporal Shift (spatial offsets following HunyuanCustom): FaceSim = 0.279, NexusScore = 0.339, AES = 0.456.

The spatiotemporal shift is catastrophic for face identity preservation (0.279 vs. 0.707), which the paper attributes to the model's pretraining on standard continuous video creating an expectation of monotonically increasing temporal indices. The native strategy aligns with "the model's inherent expectation of temporal continuity, allowing it to effectively utilize the reference context." This ablation validates that something as seemingly minor as positional encoding assignments can have outsized effects when working with pretrained video diffusion models—a finding that supports the paper's broader "minimalist intervention" philosophy: even the positional encoding scheme should respect the pretrained model's expectations.

Window Size of Audio Context Packing (Table 5b, Appendix D.2)

Context window sweep: Three window sizes for the audio context packing are compared:

  • w = 1 (no context): Sync-C = 8.872, Sync-D = 7.878, AES = 0.533.
  • w = 5 (proposed): Sync-C = 9.023, Sync-D = 7.419, AES = 0.540.
  • w = 11 (excessive context): Sync-C = 7.020, Sync-D = 9.588, AES = 0.527.

The pattern is an inverted-U for synchronization quality: too little context (w=1) produces jittery, overreactive lip movements; too much context (w=11) causes "over-smoothing" where "broad context dilutes the fine-grained cues needed for precise lip synchronization"; w=5 represents the optimal balance. The degradation from w=5 to w=11 is substantial (Sync-C drops from 9.023 to 7.020, Sync-D increases from 7.419 to 9.588), demonstrating that the window size is not merely a tuning parameter—getting it wrong can cause near-complete loss of the synchronization benefit.

Critical Assessment

Claim 1: OmniShow achieves state-of-the-art performance across multimodal conditioning settings.

The evidence in Table 1 supports this claim for the settings where baselines exist (R2V, RA2V, RP2V), but with important qualifications:

In R2V, OmniShow leads on reference consistency (NexusScore 0.389) and video quality (AES 0.468, VQ 11.12, MQ 5.885) but trails on text alignment (TA 7.746 vs. Phantom-14B's 8.609). This is a genuine trade-off, not a clean sweep. The paper does not ablate or explain this text alignment gap, which raises the question: is OmniShow's architectural philosophy (minimalist intervention on the base model) systematically trading text-following ability for multi-condition robustness? The base Waver 1.0 model's text alignment is not reported, so we cannot determine whether this gap is inherited from the base model or introduced by OmniShow's conditioning mechanisms.

In RA2V, OmniShow leads convincingly on audio-visual synchronization (Sync-C 8.612, Sync-D 7.608) and video quality while matching reference consistency. HuMo-17B edges ahead on text alignment (TA 8.146 vs. 8.093), but the difference is small (0.053) and likely not statistically significant given the 135-sample test set. The human evaluation strengthens this: evaluators preferred HuMo-17B on text alignment (42.7% vs. 34.0%) but preferred OmniShow on every other dimension, including overall video quality (51.2% vs. 21.2%).

In RP2V, the picture is mixed. OmniShow leads on pose accuracy (PCK 0.460 vs. VACE's 0.336) and several video quality metrics, but FaceSim drops substantially to 0.474 (vs. 0.600 for VACE). The paper correctly attributes this to viewpoint shifts caused by pose following, but this is a real limitation: when users provide both a reference image and a pose sequence that implies different head orientations, OmniShow prioritizes pose accuracy over identity preservation. Whether this is the "right" trade-off depends on the application, but the paper does not provide a mechanism to control it (e.g., a weight parameter balancing pose adherence vs. identity preservation).

Missing comparison: The paper does not provide an apples-to-apples comparison of inference speed, memory usage, or FLOPs per generated second of video. The parameter count comparison (OmniShow 12.3B vs. HuMo 17B) suggests efficiency, but the architectural differences (cross-attention placement, number of transformer blocks with audio modules) mean FLOPs may not scale proportionally with parameters. A table of inference cost would substantially strengthen the practical case for OmniShow.

Claim 2: The minimalist intervention philosophy preserves generation quality while enabling multimodal control.

The strongest evidence for this claim is the comparison with HuMo: OmniShow is 12.3B vs. HuMo's 17B (a ~28% reduction in parameters) yet achieves comparable or better results across most metrics. The ablation of channel vs. token concatenation (FaceSim 0.707 vs. 0.601, Table 2a) provides direct evidence that the specific injection mechanism matters.

However, a critical control experiment is missing: the paper never evaluates the base Waver 1.0 model on HOIVG-Bench. This means we cannot quantify how much generation quality is lost relative to the unconditioned base model. The minimalist intervention claim is that quality is preserved, but to test this we would need to see: (a) Waver 1.0's video quality metrics when generating videos from text alone (without any reference images, audio, or pose), and (b) OmniShow's video quality metrics when generating with all conditions. If the quality drop from (a) to (b) is small, the minimalist claim is supported. If it's large, then "minimal intervention" still causes degradation—it's just less degradation than more invasive approaches.

The paper also does not compare OmniShow against a variant where audio attention is inserted into all transformer blocks (not just dual-stream). The gate activation analysis (Figure 5) justifies the dual-stream-only placement, but an ablation showing that all-block insertion either doesn't improve metrics or actually degrades them would strengthen the "minimalist = better" argument from correlation to causation.

Claim 3: The Decoupled-Then-Joint Training strategy effectively leverages heterogeneous data.

The ablation in Table 2c provides strong evidence: single-stage training on RA2V data alone produces the worst results (Sync-D 13.11), sequential multi-stage training produces asymmetric knowledge transfer, and the proposed approach achieves the best balance (NexusScore 0.364, Sync-D 8.14, AES 0.474).

However, two limitations temper this claim:

The emergent RA2V capability (Figure 4) is presented as a key finding but is only shown qualitatively. No quantitative metrics are reported for the merged-but-not-joint-trained model, so we cannot assess how close it is to the final jointly-trained model or to the single-stage baseline. If the merged model achieves, say, Sync-C = 7.0 and FaceSim = 0.75 (substantial but not state-of-the-art), that would still demonstrate emergence. Without numbers, the claim remains anecdotal.

The interpolation ratio (0.6/0.4 A2V/R2V) is not ablated. The paper states it was chosen because "audio synchronization is significantly more sensitive to weight disruption than visual identity," but sweeping the ratio (e.g., 0.3/0.7, 0.5/0.5, 0.7/0.3) and measuring both Sync-D and FaceSim would reveal how sensitive the final model is to this choice and whether the stated rationale holds empirically. A practitioner trying to replicate this strategy on a different base model and different modalities would need to determine the appropriate ratio through trial and error.

Claim 4: OmniShow is the first framework supporting the full RAP2V setting.

This claim is factually correct within the paper's scope: none of the compared baselines support all four conditions simultaneously. The cascaded baseline comparison (Table 4) demonstrates that end-to-end joint conditioning outperforms sequential composition of specialized tools (VQ 11.06 vs. 10.05, PCK 0.478 vs. 0.340, Sync-C 7.699 vs. 7.016).

However, the cascaded baseline is only one specific composition (VACE + LatentSync). Other compositions—e.g., a strong R2V model with a separate pose-guided model and audio-driven model combined through test-time optimization or latent fusion—might perform differently. The paper acknowledges this implicitly by describing the cascaded baseline as "a representative approach" rather than the optimal sequential pipeline. The claim should be qualified: OmniShow outperforms this specific cascaded baseline; it may or may not outperform all possible sequential compositions.

General Experimental Weaknesses

Test set size: HOIVG-Bench contains only 135 samples. For a benchmark intended to support community-wide comparison and future research, this is quite small. A single anomalous sample can shift aggregate metrics by nearly 1 percentage point. The paper does not report confidence intervals, standard deviations, or any measure of statistical reliability for the quantitative results. Given the 135-sample size and the number of metrics reported, many of the numerical differences in Table 1 are likely within sampling error. For example, the difference between OmniShow's FaceSim of 0.874 and Phantom-14B's 0.876 in the R2V setting (a 0.002 gap) is almost certainly not statistically significant at this sample size. The paper would benefit from bootstrap confidence intervals or at minimum a statement about the expected variance of each metric.

Single model family: All experiments use Waver 1.0 as the base model. The minimalist intervention philosophy's transferability to other DiT architectures (e.g., Wan 2.1, HunyuanVideo, Stable Video Diffusion) is not tested. The specific mechanisms—channel concatenation for visual conditions, masked cross-attention for audio—are architecturally generic, but their effectiveness depends on the base model's pretrained feature distributions and conditioning interface. A positive result on a second base model family would substantially strengthen the generalizability claim.

No real-user study with the target application: Human evaluation (Figure 8) compares OmniShow against baselines on generic dimensions (video quality, audio-visual sync, etc.) but does not test the actual downstream task—e.g., "would you use this generated video as a product demonstration for an e-commerce listing?" Task-specific evaluation (product visibility, interaction naturalness, suitability for commercial use) would connect the technical metrics to the application motivations laid out in Section 1.

Missing difficulty analysis: Unlike the reference paper analysis (which stratified results by problem difficulty), OmniShow does not analyze performance as a function of input complexity. One would expect that HOIVG performance varies with factors like: motion intensity of the pose sequence, visual complexity of the reference object, speaking rate in the audio, and degree of occlusion in the interaction. Breaking down HOIVG-Bench results by these factors would reveal where OmniShow excels and where it fails—for instance, does performance degrade on fast-paced motion? On complex articulated objects? On rapid speech? Such an analysis would guide practitioners on when to use OmniShow vs. alternative approaches.

Limited resolution and duration: All experiments are standardized to 5-second 720p clips. The paper states OmniShow supports generation up to 10 seconds, but longer-form generation is not evaluated. For real e-commerce applications, product demonstration videos are often 30-60 seconds. How OmniShow's conditioning mechanisms scale with video length—whether attention masking remains effective for longer audio sequences, whether reference appearance drifts over extended generations—is untested.

6. Limitations and Trade-offs

6.1 The Difficulty Estimation Cost for Adaptive Strategy Selection Remains Unaccounted

The assumption or constraint. The Decoupled-Then-Joint Training strategy requires estimating which training curriculum (single-stage, multi-stage R2V→RA2V, multi-stage A2V→RA2V, or the proposed specialist-then-generalist approach) will work best before committing to the full training pipeline. The paper implicitly treats the optimal training strategy as known through the ablations in Table 2c, but in a practical deployment—where a team is training a HOIVG model from a different base architecture with different heterogeneous data sources—there is no cheap way to determine whether Decoupled-Then-Joint Training will outperform simpler alternatives without running all candidate curricula at scale. The paper's ablation in Table 2c required training four separate model variants on 32 GPUs each. This is not a modest diagnostic experiment; it is a substantial fraction of the final training cost.

The paper does not explicitly acknowledge this as a limitation, but it is a direct consequence of the approach's dependence on heterogeneous data. The entire methodology rests on the premise that joint training from scratch on limited fully-paired data will fail (validated in Table 2c: single-stage achieves Sync-D = 13.11 vs. 8.14) and that sequential multi-stage training produces asymmetric knowledge transfer (validated by the R2V→RA2V and A2V→RA2V rows). But the paper provides no lightweight signal, metric, or diagnostic that would allow a practitioner to predict in advance whether a given heterogeneous dataset requires the decoupled-then-joint approach or whether simpler curricula would suffice. The choice of 0.6/0.4 interpolation ratio is also presented without an ablation sweep—a practitioner replicating this on a different base model would need to determine the appropriate ratio through trial and error, which is expensive at the 12B+ scale.

The consequence. The headline result—that Decoupled-Then-Joint Training achieves the best tradeoff (NexusScore 0.364, Sync-D 8.14, AES 0.474)—is a post-hoc measurement, not a prospective guarantee. A team with different heterogeneous data (e.g., different ratios of R2V to A2V data, different audio quality, different object categories) might find that the single-stage approach works adequately or that the optimal interpolation ratio differs. Without a cheap diagnostic for strategy selection, the practical barrier to adopting the decoupled-then-joint paradigm is substantially higher than the paper suggests: you must first train multiple 12B-scale models to determine whether the approach is even necessary for your setting. For resource-constrained practitioners, this effectively means committing to a single strategy without evidence that it is optimal for their data.

What evidence exists in the paper. Table 2c provides the only direct comparison of training strategies, and it validates that the proposed approach works best for the paper's specific data mixture. The emergent RA2V result (Figure 4) is presented qualitatively but without quantitative metrics for the merged-but-not-joint-trained model, so its diagnostic value for strategy selection is unclear. Section 5 (Discussion) acknowledges the broader insight—"one can fully leverage heterogeneous data through a decoupled-then-joint training paradigm"—but does not discuss the cost of determining whether this paradigm applies. The paper also does not report the total GPU-hours for each ablation variant, making it difficult to assess whether the diagnostic cost of strategy selection is comparable to, or exceeds, the cost of simply running the most expensive strategy.

Mitigation status. Not addressed. The paper does not propose a lightweight method for predicting when decoupled training is needed or for selecting the interpolation ratio without a sweep. The emergent RA2V phenomenon hints that the merged model's zero-shot performance could serve as a cheap diagnostic (if merging produces decent but not great results, joint fine-tuning will help; if merging produces catastrophic results, the interpolation ratio or the strategy itself may need revisiting), but this is not explored quantitatively. A future version of this work could report the merged model's metrics on a small validation set and correlate them with post-joint-training improvement, providing a practical signal for strategy selection.

6.2 Reference Identity Preservation Degrades Significantly When Pose Control Is Active

The assumption or constraint. The paper's central claim is that OmniShow simultaneously satisfies all four conditioning signals without quality degradation. However, the RP2V results in Table 1 reveal a substantial drop in reference consistency when pose conditioning is active: FaceSim falls from 0.874 in R2V to 0.474 in RP2V, and NexusScore drops from 0.389 to 0.418 (the NexusScore actually increases slightly, but FaceSim nearly halves). The paper acknowledges this explicitly in Section 4.2:

"precise pose adherence introduces viewpoint shifts and facial morphology changes that affect FaceSim"

This is an honest admission, but it reveals a fundamental trade-off that the architecture does not resolve: when the pose sequence specifies head orientations or body configurations that differ from the reference image's viewpoint, the model must distort the reference appearance to follow the pose, and this distortion is penalized by reference consistency metrics. The same trade-off is visible in the text alignment metric: TA drops from 7.746 in R2V to 6.526 in RP2V, suggesting that strong pose conditioning can override text-specified scene details as well.

The paper does not provide a mechanism for controlling this trade-off—no hyperparameter or loss-weight that allows a user to specify, for a given generation, whether pose accuracy or identity preservation should take priority. The model makes this trade-off implicitly, and the results in Table 1 suggest it leans toward pose adherence at the expense of identity.

The consequence. For practical applications, this limitation means that OmniShow may not be suitable when the reference image and the desired pose sequence imply substantially different viewpoints or body configurations. An e-commerce brand that provides a front-facing product photo as the reference image but wants the generated video to show the model turning the product to the side (as specified by the pose sequence) will likely see degraded identity preservation in the profile views. The human evaluation results (Figure 8, right) confirm that human evaluators notice this: in the RP2V setting, 44.8% of evaluators preferred VACE on reference consistency (vs. 30.5% for OmniShow), despite OmniShow's higher NexusScore (0.418 vs. 0.352). This suggests that FaceSim, which penalizes viewpoint-induced appearance changes, tracks human perception more closely than NexusScore in this regime.

The paper demonstrates robust identity preservation within a single viewpoint regime (R2V, where no pose forces viewpoint changes), but the generalization to multi-viewpoint generation under pose control introduces a degradation that is not bounded or characterized beyond the aggregate FaceSim drop. A practitioner cannot predict whether a specific combination of reference image and pose sequence will yield acceptable identity preservation without generating the video.

What evidence exists in the paper. Table 1 provides the core evidence: FaceSim at 0.874 (R2V) vs. 0.474 (RP2V), a ~46% relative drop. The human evaluation (Figure 8, right) shows that 44.8% of evaluators prefer VACE on reference consistency in RP2V, despite VACE having lower NexusScore (0.352 vs. 0.418). The qualitative examples in Figures 7, 11, 12, and 13 provide visual evidence of identity shifts under pose control, though the curated nature of these examples makes it difficult to assess prevalence.

Mitigation status. The paper does not attempt to resolve this trade-off. It acknowledges the FaceSim drop (Section 4.2) and attributes it to pose-induced viewpoint shifts, but does not propose a solution—no loss-weight hyperparameter, no architectural modification, no inference-time technique for balancing pose adherence against identity preservation. Section 5 (Future Work) mentions "better align the generation with real-world dynamics and mitigate artifacts under extreme conditions" through RL-based post-training, but this is not specific to the identity-pose trade-off. The limitation is architectural: the Unified Channel-wise Conditioning injects pose and reference images through a shared pathway with fixed relative influence, giving the model no signal about which condition should take priority when they conflict.

6.3 The Benchmark's AI-Generated Reference Images Create an Unrealistic Evaluation Gap

The assumption or constraint. HOIVG-Bench uses AI-generated human and object reference images (produced by Nano Banana, a Google image generation API) rather than real photographs. The paper explicitly states this was a "carefully deliberated choice to comply with privacy, ethical, and legal guidelines for public release" (Section 3.5), and notes that "manual checks to filter out images with noticeable 'AI-ness'" were performed to "ensure the benchmark closely reflects real-world data."

However, AI-generated images—even those that pass visual inspection for "AI-ness"—differ from real photographs in ways that are statistically detectable by vision models. They may have different noise characteristics, color distributions, lighting consistency, and texture patterns. A model trained on real video data (as OmniShow is) and evaluated on AI-generated reference images is being tested on a distribution that is not identical to the one it was trained on—and more importantly, not identical to the one it would encounter in real deployment, where reference images are typically real photographs (product photos, model headshots).

The consequence. The benchmark results may overestimate or underestimate OmniShow's real-world reference consistency. If the AI-generated images are easier to match (because they have simplified textures or more uniform lighting), the reported FaceSim and NexusScore would be optimistic relative to real-photo performance. Conversely, if the AI-generated images have subtle artifacts that confuse the model (because the model was trained on real video and has never seen synthetic-image conditioning during training), the reported metrics would be pessimistic. The paper provides no analysis of which direction the bias goes, and no comparison of performance on real vs. AI-generated reference images from the same video sources.

The privacy justification is legitimate—releasing a benchmark with real human faces raises consent and legal issues—but it creates a systematic evaluation gap that the paper does not quantify. The cascaded baseline comparison (Table 4) and the main results (Table 1) are all evaluated on AI-generated reference images, so the relative rankings among methods may be preserved even if absolute metrics are shifted. But a practitioner cannot infer from the reported FaceSim of 0.874, for example, what FaceSim they should expect when conditioning on real product photography.

What evidence exists in the paper. Section 3.5 describes the AI generation pipeline and notes the manual filtering step, but no experiment compares OmniShow's performance on real vs. AI-generated reference images. The paper does not report any analysis of distribution shift between the AI-generated reference images and the real video frames they are meant to condition. The only quantitative evidence of the benchmark's relationship to real data comes from the motion intensity distribution (Figure 6b) and the word cloud of text prompts (Figure 6a), which characterize the video content but not the reference image characteristics.

Mitigation status. The paper acknowledges the AI generation choice and justifies it on privacy grounds, but does not attempt to quantify or bound the resulting distribution shift. Section 5 (Limitations) states:

"the human reference images in our benchmark is AI-generated, which might introduce slight distribution biases compared to purely real-world photos"

The qualifier "slight" is asserted without evidence. A future version of the benchmark could include a small held-out set of real reference images (with consent from the subjects) to calibrate the AI-generated results against real-photo performance, providing an explicit correction factor. The paper does not propose this.

6.4 Model Scaling and Long-Form Video Generation Capability Are Unexplored

The assumption or constraint. All quantitative results in the paper are standardized to 5-second video clips at 720p resolution in portrait mode (Section 3.5). The paper states that OmniShow "supports video generation of up to 10 seconds" and that the 5-second standardization was done "to ensure a fair comparison with baseline methods that only support short-clip generation." This is a practical constraint imposed by the baseline comparison, but it means the paper provides no evidence about OmniShow's performance beyond 5 seconds, despite the stated e-commerce and content creation applications where product demonstration videos routinely run 30-60 seconds.

The model scale (12.3B parameters) is also a single data point. The paper makes efficiency claims based on parameter count (OmniShow is 12.3B vs. HuMo's 17B, a ~28% reduction), but does not investigate how performance scales with model size. Would a smaller OmniShow variant (e.g., 7B) retain most of the multimodal conditioning capability? Would a larger variant (e.g., 24B) resolve the identity-pose trade-off discussed in Limitation 6.2? The paper's minimalist intervention philosophy implies that conditioning mechanisms should transfer across scales, but this is untested.

The consequence. For deployment planning, the lack of long-form generation evaluation leaves critical questions unanswered:

  • Temporal consistency over extended durations: The Gated Local-Context Attention enforces fine-grained local alignment (w=5 frames of audio context), but does it prevent reference appearance drift over 30+ seconds of generation? The pseudo-frame tokens that carry reference image information are placed at the sequence start (as a temporal prefix) and influence the entire generation through self-attention, but attention to early tokens can attenuate with sequence length—a known limitation of transformer architectures that the paper does not address.
  • Audio-visual synchronization over long speech: The attention masking restricts each frame to attend to a fixed window of w audio tokens. For long audio sequences (2+ minutes of speech), this locality constraint could cause accumulated drift where lip movements gradually desynchronize from audio, since there is no global synchronization signal.
  • Pose sequence length: The paper does not specify the maximum pose sequence length supported or how pose conditioning degrades for very long sequences with many keyframes.
  • Cost scaling: At 720p, 5 seconds of generation requires a certain compute budget. Extrapolating to 10 seconds, 30 seconds, or 60 seconds—and to higher resolutions (1080p, 4K)—is not straightforward without data on how generation time and memory scale with output length and resolution. The paper's discussion of training infrastructure (128 GPUs, FSDP, sequence parallelism) concerns training cost, not inference cost.

The parameter count comparison is also incomplete as a measure of efficiency. Architectural differences (where audio attention is placed, how many blocks contain cross-attention, the dimension of the audio projector) mean that FLOPs per generated frame may not scale linearly with parameter count. A model with fewer parameters but deeper attention pathways could be slower at inference than a larger model with shallower pathways. The paper does not report inference time, memory usage, or FLOPs for any setting.

What evidence exists in the paper. The 5-second standardization is stated in Section 3.5 and justified in Section 5 (Limitations): "our current evaluation focuses on 5-second clips to ensure a fair comparison with baselines that only support short-clip generation." The 10-second capability is mentioned but never evaluated. The training infrastructure details (128 GPUs, Section 4.1) provide some sense of scale but are not translated to inference cost. The parameter count comparisons (12.3B vs. 17B for HuMo, Section 4.1) are the only efficiency metric reported.

Mitigation status. The paper acknowledges the duration limitation (Section 5) but does not provide even a preliminary long-form evaluation. Section 5 (Future Work) mentions "we aim to scale up the training data and model capacity to push the boundaries of the model's generalization ability in complex scenarios," but this is phrased as an aspirational goal rather than a concrete next step. No inference cost profiling, no scaling law analysis, and no temporal consistency evaluation beyond 5 seconds are provided.

6.5 The Evaluation Benchmark Is Too Small to Support Reliable Statistical Inference

The assumption or constraint. HOIVG-Bench contains 135 samples (Section 3.5). All quantitative results in Tables 1, 3, and 4 are reported as scalar means over these 135 samples, with no confidence intervals, standard deviations, or statistical significance tests. Given the number of metrics (11) and the number of model comparisons (6 baselines × multiple settings), the sample size is small enough that many of the reported numerical differences may fall within sampling error.

Consider the R2V setting in Table 1. OmniShow's FaceSim is 0.874 vs. Phantom-14B's 0.876—a difference of 0.002 that is almost certainly not statistically significant at n=135. Yet the paper reports these as scalar values without qualification, and the abstract claims "overall state-of-the-art performance." In the RA2V setting, OmniShow's TA is 8.093 vs. HuMo-17B's 8.146 (difference: 0.053). In the RP2V setting, OmniShow's IQA is 0.722 vs. VACE's 0.712 (difference: 0.010). These are small gaps that could easily reverse with a different random sample of 135 videos.

The problem compounds when comparing across settings. The paper makes claims like "OmniShow achieves the best audio-visual synchronization" (Section 4.2) based on Sync-C differences of ~0.6 points (8.612 vs. 8.013 for HuMo-17B) on 135 samples. Without knowing the variance of Sync-C across samples, a 0.6-point gap could be anywhere from strongly significant to well within noise.

The consequence. The paper's central comparative claims—that OmniShow achieves state-of-the-art or competitive performance—are qualitatively supported but not statistically validated. A practitioner choosing between OmniShow and HuMo-17B for an RA2V deployment cannot determine whether OmniShow's Sync-C advantage of 0.6 points is reliable or a sampling artifact. The human evaluation (30-33 evaluators, 20 samples each) provides a complementary signal that partially mitigates this—evaluators preferred OmniShow on most dimensions—but the small sample size means the quantitative Table 1 results and the human evaluation in Figure 8 are both subject to substantial sampling variance.

This also limits the benchmark's utility for future research. If a subsequent paper reports a Sync-C of 8.7 on HOIVG-Bench (vs. OmniShow's 8.612), the community cannot determine whether this represents a genuine improvement or noise without standard errors. The benchmark's construction methodology (curated diverse samples, AI-generated references) is high-quality, but the sample size makes it unsuitable as a primary evaluation benchmark for methods that differ by small margins.

What evidence exists in the paper. The sample size of 135 is stated in Section 3.5. No standard deviations, confidence intervals, or p-values appear anywhere in the paper. The human evaluation (Figure 8) reports raw percentages but no inter-evaluator agreement statistics (e.g., Fleiss' kappa) that would indicate whether evaluator preferences are consistent or noisy. The paper does not discuss the statistical reliability of its results.

Mitigation status. Not addressed. The paper treats the 135-sample benchmark as sufficient for drawing comparative conclusions, with no discussion of statistical power or measurement uncertainty. The benchmark's construction (Section 3.5) emphasizes the quality and diversity of the samples but not their quantity. For a benchmark intended to "set a solid standard for the emerging HOIVG task" (Section 1), the sample size is a significant limitation that future work expanding HOIVG-Bench would need to address. The paper does not suggest expansion or acknowledge the sample size as limiting.

6.6 The Minimalist Intervention Philosophy Lacks a Causal Ablation Against the "Aggressive" Counterfactual

The assumption or constraint. The paper's central design philosophy—stated in Section 5 as the "Philosophy of Minimalist Intervention" and operationalized through channel-concatenation for visual modalities and strategically placed gated cross-attention for audio—rests on the claim that preserving the base model's native input structure prevents quality degradation that aggressive architectural modification causes. The evidence for this claim comes from (1) the channel vs. token concatenation ablation (Table 2a: FaceSim 0.707 vs. 0.601), (2) the parameter count comparison with HuMo (12.3B vs. 17B), and (3) the gate activation analysis (Figure 5) justifying dual-stream-only audio attention.

However, the paper never compares the proposed minimalist design against an "aggressive" version of OmniShow itself. The relevant ablation would be: What happens if audio cross-attention is inserted into all MMDiT blocks (both dual-stream and single-stream) rather than just dual-stream? What happens if reference images and pose are injected through dedicated, modality-specific cross-attention layers rather than sharing the channel-concatenation pathway? What happens if the unified channel-concatenation pathway is replaced with separate injection mechanisms for each visual modality?

Without these ablations, the claimed benefit of minimalism is confounded with the specific injection mechanisms chosen. The gate activation analysis (Figure 5) shows that audio is predominantly utilized in dual-stream blocks after training with gates, but this does not prove that audio could not be utilized in single-stream blocks if cross-attention were placed there without gates. The gates may suppress audio in single-stream blocks because that is the path of least resistance during optimization, not because single-stream blocks are inherently unsuited for audio processing. An ablation where audio attention is placed everywhere (without gating) and compared against the proposed gated, dual-stream-only design would isolate the causal effect of the minimalist placement strategy.

The consequence. The paper's strongest rhetorical claim—that the minimalist philosophy is why OmniShow succeeds—is supported by evidence that is more correlational than causal. The comparison with HuMo (12.3B vs. 17B) shows that OmniShow's specific design is parameter-efficient, but HuMo is a different architecture from a different lab, with different training data, different base model, and different optimization hyperparameters. The parameter count difference could be due to any number of confounds unrelated to minimalism. The channel-concatenation ablation (Table 2a) shows that channel concatenation outperforms token concatenation, but this is a comparison of two minimalist strategies (both within OmniShow's framework)—it does not test against an aggressive counterfactual.

A practitioner trying to design a multimodal conditioning system for a different base model (e.g., HunyuanVideo, Stable Video Diffusion) cannot conclude from this paper that minimalist intervention is always better than aggressive modular design. The paper demonstrates that a specific set of minimalist choices works well for Waver 1.0 on HOIVG, but the generalizability of the philosophy to other architectures, other modalities, and other tasks is assumed rather than tested.

What evidence exists in the paper. The gate activation analysis (Figure 5) is the closest the paper comes to a causal test of the minimalist placement strategy, but as noted, it is confounded by the presence of the gates themselves. The ablations in Table 2a, 2b, and 2c test components within the minimalist framework but not the framework itself against an aggressive alternative. The paper does not report any experiment where OmniShow's architecture is made more aggressive and performance is compared.

Mitigation status. The paper does not acknowledge this as a limitation. Section 5 presents the minimalist intervention philosophy as a confirmed insight ("We demonstrated that deeply understanding the underlying input structure and learning dynamics of DiTs enables the design of efficient injection methods") without noting that the counterfactual—an aggressive version of OmniShow—was never tested. The "Evolution from Specialists to a Generalist" insight is similarly presented as a general finding without a negative control showing that the specialist approach fails for an alternative architecture or modality set. Future work could strengthen the minimalist claim by designing and evaluating an OmniShow variant with deliberately aggressive architectural modifications (e.g., all-block audio attention, separate reference and pose injection pathways) and demonstrating that it underperforms the minimalist version.

7. Implications and Future Directions

How This Work Changes the Landscape

OmniShow shifts the framing of multimodal controllable video generation from an architectural integration problem (how do we connect new modality-specific modules to a pretrained backbone?) to a representation preservation problem (how do we inject conditioning signals into the pathways the base model already uses, without corrupting its pretrained feature distributions?). This is not a paradigm shift in the sense of introducing a new generative framework—the underlying flow matching, MMDiT architecture, and VAE compression are all inherited from Waver 1.0—but it is a methodological reframing with immediate practical consequences for how future multimodal systems should be designed.

The paper's central finding is that this reframing works: a model that is only ~2.5% larger than its 12B backbone (12.3B total) can simultaneously handle text, reference images, audio, and pose conditioning while matching or exceeding the performance of models that expand parameter count by ~21.4% (HuMo, 14B → 17B) and achieving state-of-the-art audio-visual synchronization (Sync-C 8.612 vs. 8.013 for the 17B HuMo variant, Table 1 RA2V). The implication for the field is that parameter efficiency in multimodal conditioning is not just a nice-to-have—it is a signal that the conditioning mechanism is correctly aligned with the base model's representations. Architects of future multimodal video models should prioritize understanding and reusing the base model's native interfaces (channel concatenation, existing attention pathways) rather than designing bespoke injection modules for each new modality.

This reframing also reconciles a latent contradiction in the controllable generation literature. Prior work on combining multiple conditions fell into two camps: one that grafts on large, specialized modules for each new modality (as in HuMo, which adds substantial parameter overhead for audio), and one that avoids architectural modification by cascading separate specialist models (the VACE + LatentSync baseline in Table 4). The first camp risks disrupting pretrained priors; the second camp introduces compounding errors and artifacts from sequential processing (the cascaded baseline achieves VQ 10.05 vs. OmniShow's 11.06 and PCK 0.340 vs. 0.478). OmniShow demonstrates a third path: simultaneous end-to-end conditioning through minimally invasive extensions of existing pathways, achieving both better quality than cascading and better efficiency than modular grafting. This resolves the apparent tradeoff between controllability and quality described in Section 1: the tradeoff exists when conditioning mechanisms are treated as independent architectural additions, but it can be largely circumvented when they are designed as extensions of the base model's native input structure.

The paper also introduces a diagnostic methodology that changes how multimodal conditioning systems should be developed. The use of learned gating vectors as post-hoc indicators of modality impact (Figure 5)—revealing that audio information flows predominantly through dual-stream blocks—converts a design choice (where to insert new attention layers) into a measurement problem (where does the model actually use the modality?). This is a transferable technique: any future work adding any conditioning modality to a DiT-based model can instrument the candidate injection locations with learnable gates, train once, and read off the gate norms to determine optimal placement, avoiding expensive combinatorial ablations over insertion points. This lowers the barrier to designing multimodal systems on new base architectures.

The weight-merging discovery—that separately trained R2V and A2V specialist models, when linearly interpolated at a 60/40 ratio, exhibit emergent zero-shot RA2V capability (Figure 4) without any joint training—has implications beyond HOIVG. It suggests that controllability can emerge through weight-space arithmetic, not just through training on jointly-annotated data. For the broader field of multimodal generation, this means that the expensive bottleneck of collecting N-way paired data can be partially circumvented: train specialists on abundant single-modality data, merge them, and fine-tune on a modest amount of jointly-annotated data. The paper does not characterize the limits of this phenomenon (which specialist pairs merge successfully? what interpolation ratios work? does it generalize to other modalities?), but it establishes the phenomenon's existence in a 12B-scale video model, which makes it worth investigating systematically.

Finally, HOIVG-Bench—despite its small size (135 samples)—establishes a multi-dimensional evaluation template that surfaces cross-modal interference effects invisible to single-modality benchmarks. The fact that AnchorCrafter achieves reasonable pose accuracy (AKD 0.229) at catastrophic cost to reference consistency (FaceSim 0.404, Table 1 RP2V) would be invisible on a pose-only benchmark. The fact that OmniShow's FaceSim drops from 0.874 (R2V) to 0.474 (RP2V) reveals a genuine identity-pose tradeoff that single-modality evaluation would mask. By making simultaneous multi-condition evaluation the standard, HOIVG-Bench changes what it means to "solve" controllable video generation: future systems must demonstrate balanced performance across all conditioning dimensions, not excellence in one at the expense of others.

Follow-Up Research This Work Enables

Causal isolation of the minimalist intervention philosophy. The paper's central claim—that preserving the base model's native input structure prevents quality degradation—rests on correlational evidence: channel concatenation outperforms token concatenation (FaceSim 0.707 vs. 0.601, Table 2a), and OmniShow achieves competitive performance with fewer parameters than HuMo. But the aggressive counterfactual is never tested. A follow-up study should construct an "OmniShow-Aggressive" variant where: (1) reference images and pose are injected through dedicated modality-specific cross-attention layers rather than sharing the channel-concatenation pathway, (2) audio cross-attention is inserted into all MMDiT blocks (not just dual-stream), and (3) the gating mechanism is removed. Training this variant on the same data as OmniShow and comparing on HOIVG-Bench would isolate the causal effect of the minimalist design choices. If OmniShow-Aggressive underperforms (lower AES, higher Sync-D), the minimalist philosophy is validated. If it performs comparably, the philosophy is not the active ingredient—the specific injection mechanisms (channel concatenation, masked attention) are, and they could be deployed in more aggressive architectures without penalty.

Characterizing the limits and mechanism of weight-merging emergence. The paper reports that merging R2V and A2V specialists at a 60/40 ratio produces emergent RA2V capability (Figure 4), but provides only qualitative evidence. A systematic follow-up should: (a) quantify emergence by reporting the merged-but-not-fine-tuned model's full metrics on HOIVG-Bench (FaceSim, Sync-C, Sync-D, AES, etc.) and comparing against both the specialists and the final jointly-trained model—this establishes how much capability weight merging alone provides vs. how much joint training adds; (b) sweep the interpolation ratio from 0.1 to 0.9 in 0.1 increments, measuring RA2V metrics at each point to identify whether emergence is robust to ratio choice or fragile; (c) test cross-architecture merging by training R2V and A2V specialists on a different base model (e.g., HunyuanVideo) and checking whether merging still produces emergent joint capability—if it fails, the phenomenon may depend on Waver 1.0-specific properties (MMDiT dual-stream structure, specific pretraining distribution); (d) test merging of non-audio modalities (e.g., R2V + Pose-to-Video specialists) to determine whether emergence is specific to the audio-visual combination or generalizable. This would convert an anecdotal observation into a characterized technique with known boundary conditions.

Cost-effective difficulty estimation for training strategy selection. The Decoupled-Then-Joint Training strategy is validated as superior to simpler curricula (Table 2c: Sync-D 8.14 vs. 13.11 for single-stage), but the paper provides no lightweight signal for predicting in advance whether a given heterogeneous dataset requires this approach. A follow-up should develop a cheap diagnostic: after training separate R2V and A2V specialists but before committing to the full joint-training pipeline, measure the merged model's zero-shot metrics on a small validation set (e.g., 30 samples). If the merged model's Sync-D is below some threshold (suggesting the specialists' capabilities are already composable in weight space), joint training is likely to help; if Sync-D is catastrophically high (suggesting the specialists' representations are incompatible), the interpolation ratio or the training data mixture needs revisiting. Reporting this diagnostic on the paper's existing ablations (computing merged-model metrics for the Table 2c variants) would provide initial calibration. A subsequent study could deliberately construct "hard" heterogeneous datasets (e.g., R2V data from real photos + A2V data from cartoon avatars) and test whether the diagnostic predicts training success or failure.

Scaling behavior of multimodal conditioning with video length and model size. OmniShow is evaluated only on 5-second 720p clips (Section 3.5), despite supporting up to 10 seconds. A scaling study should: (a) evaluate temporal consistency by generating videos at 5s, 10s, 20s, and 30s (if technically feasible) and measuring how reference consistency (FaceSim, NexusScore) and audio-visual synchronization (Sync-C, Sync-D) degrade with duration—does the pseudo-frame attention mechanism suffer from the known transformer limitation of attention attenuation over long sequences? (b) profile how the identity-pose tradeoff evolves with duration: does FaceSim continue to drop beyond 5 seconds, or does it stabilize? (c) train OmniShow variants at different scales (e.g., 3B, 7B, 12B, 24B) using the same data and training recipe, measuring how each conditioning dimension scales with parameter count—does audio-visual synchronization scale more steeply than reference consistency? Does the identity-pose tradeoff improve at larger scales (suggesting it is a capacity limitation) or persist (suggesting it is an architectural constraint)? This would provide the first multimodal conditioning scaling laws for video generation.

Benchmark expansion with real reference images and calibration. HOIVG-Bench's 135 AI-generated reference images raise distribution-shift concerns that the paper acknowledges but does not quantify (Section 5: "might introduce slight distribution biases"). A follow-up should: (a) construct a calibration set of 20-30 real reference images (with explicit consent from photographed subjects) paired with the same videos, audio, and pose sequences as a subset of HOIVG-Bench; (b) evaluate OmniShow and baselines on both the AI-generated and real reference images for the same underlying videos, computing the difference in FaceSim, NexusScore, and TA between the two conditions; (c) report this difference as a distribution-shift correction factor that future HOIVG-Bench users can apply to interpret results. For example, if FaceSim on real images is consistently 0.05 lower than on AI-generated images, then OmniShow's reported FaceSim of 0.874 corresponds to approximately 0.82 in real deployment. This would substantially increase the benchmark's practical utility without requiring a complete rebuild. Additionally, expanding HOIVG-Bench to 500+ samples (with the same construction pipeline) would bring statistical power into a range where the paper's comparative claims can be validated with confidence intervals.

Cross-modal interference as a controllable parameter. The identity-pose tradeoff (FaceSim 0.874 in R2V → 0.474 in RP2V, Table 1) and the text alignment drop (TA 7.746 → 6.526) suggest that OmniShow implicitly prioritizes conditioning signals in a fixed hierarchy when they conflict. A follow-up should develop a user-controllable tradeoff mechanism: introduce a scalar weight λ_pose ∈ [0, 1] that modulates the relative influence of pose vs. reference conditions during inference (e.g., by scaling the pose condition tokens or adjusting the pseudo-frame mask). A sweep over λ_pose on HOIVG-Bench would produce a Pareto frontier showing the achievable (FaceSim, PCK) pairs. This would allow practitioners to select an operating point based on application needs—prioritizing identity for virtual try-on demos, prioritizing pose accuracy for sports training videos. The mechanism could be implemented without retraining by modifying the channel-concatenation masks at inference time, making it immediately practical if it works.

Practical Applications and Downstream Use Cases

E-commerce product demonstration at scale. The paper's explicit motivating application (Section 1) is directly served by OmniShow's capabilities. A brand with a catalog of 1,000 SKUs currently produces individual product videos through manual filming—a process costing hundreds of dollars per video and requiring coordination between models, videographers, and editors. With OmniShow, the brand provides: (1) one reference image of their human model, (2) one product photo per SKU, (3) a script-generated audio narration track (using ElevenLabs with the model's voice profile), and (4) a set of pose templates for common demonstration motions (pick up, turn, show back, gesture to features). The system generates a 5-second product video per SKU at 720p resolution. The paper's quantitative results support viability: in the RAP2V setting (Table 4), OmniShow achieves VQ 11.06, FaceSim 0.645, and PCK 0.478, with Sync-C 7.699 ensuring the narration is lip-synchronized. The key practical benefit is recomposability: the same human model reference, a library of pose templates, and per-product audio tracks can generate videos for an entire catalog without additional photography or filming, reducing per-video production cost to the inference compute budget. The limitation to 5-10 second clips (Section 5) means this currently serves short-form product highlights (social media ads, listing thumbnails) rather than full demonstrations, but these are high-volume commercial formats.

Privacy-compliant avatar animation for content creators. The paper's use of AI-generated reference images in HOIVG-Bench (Section 3.5) was motivated by privacy concerns, but this actually demonstrates a deployment pathway: a content creator who wants an animated avatar for voiceover content provides a synthetic or stylized reference image (generated from their likeness via Nano Banana or similar) rather than a real photograph. OmniShow then drives the avatar with their recorded audio (Sync-C 8.612 in RA2V, Table 1) and optional pose sequences for expressive gestures. Because the reference image is synthetic, there is no privacy risk from model inversion or unauthorized likeness use—the generated video features an AI-constructed face that is stylistically similar to the creator but not biometrically identifiable. The paper's A2V results on EMTD (Table 3: Sync-C 6.49 for the A2V specialist, best among compared methods) validate that the audio driving mechanism is sufficiently precise for professional content. This application is production-ready today for the 5-second clip regime, supporting short-form platforms where creators need rapid, lips-synchronized avatar clips at scale.

Data generation for self-improving video models. The paper's Decoupled-Then-Joint Training strategy demonstrates that heterogeneous, imperfectly-paired data can train a unified model, but the trained model itself can then generate higher-quality training data for future iterations. Specifically: use OmniShow to generate RAP2V videos for a large set of text prompts, reference images, and audio tracks; automatically filter generated videos by the paper's evaluation metrics (FaceSim, Sync-C, PCK); retain only videos exceeding quality thresholds; and use these as additional training data for the next model iteration. This is a standard self-improvement loop (à la STaR or ReST^EM), but OmniShow makes it newly tractable for multimodal video because: (a) the unified conditioning means generated data satisfies all four modalities simultaneously, (b) the evaluation metrics provide multi-dimensional quality filtering, and (c) the model's efficiency (12.3B parameters) makes large-scale generation feasible. The paper's finding that the ReST^EM-trained revision model degraded (Appendix K of the reference paper, though not directly applicable to OmniShow) serves as a cautionary note—naive self-improvement can backfire—but the availability of multiple conditioning signals (not just correctness, but also synchronization, consistency, and quality metrics) provides richer filtering than text-only self-improvement loops.

When to Prefer This Method

The paper does not position OmniShow against a clear "prefer-A-over-B" decision boundary with named competitors and explicit switching conditions. It demonstrates that OmniShow is the only method supporting the full RAP2V setting (Section 1, Table 4), so for applications requiring simultaneous text + reference image + audio + pose conditioning, there is no alternative to choose between—OmniShow is the only option. For sub-task settings (R2V, RA2V, RP2V), the paper's results in Table 1 show OmniShow is competitive with or superior to specialized baselines across most metrics, but it does not articulate specific conditions under which a practitioner should choose, say, Phantom-14B over OmniShow for R2V or HuMo-17B over OmniShow for RA2V. The paper's framing is that OmniShow is a unified replacement for multiple specialized models, not a method that is optimal in some regimes and suboptimal in others. The closest the paper comes to a tradeoff discussion is the acknowledged identity-pose tension in RP2V (FaceSim 0.474 vs. 0.600 for VACE, Section 4.2), which implies that if reference identity preservation under pose variation is the sole priority and audio is not needed, a specialized R2V+Pose model might be preferable—but the paper does not make this recommendation explicit, and it provides no mechanism for controlling the tradeoff. A practitioner choosing between OmniShow and a specialized alternative must therefore rely on the quantitative Table 1 differences for their specific setting, understanding that OmniShow's advantage is in balanced multi-condition performance rather than dominance on any single conditioning axis.