ArXiv: 2511.21579

🎯 Pitch

Jointly generating video and audio from noise causes a 'Correspondence Drift' where two chaotic signals fatally destabilize alignment learning. Harmony fixes this by co-training with unidirectional tasks, then explicitly amplifies the sync signal during inference, achieving state-of-the-art lip-sync that finally rivals the closed-source giants.

1. Executive Summary

This paper proposes Harmony, a joint audio-video generation framework that mechanistically enforces cross-modal synchronization by addressing three identified failures of standard joint diffusion: Correspondence Drift (the instability of aligning two concurrently evolving noisy latents), an architectural conflation of global style and local temporal alignment, and the intra-modal bias of conventional Classifier-Free Guidance. The model is trained and evaluated on a newly introduced benchmark, Harmony-Bench—a 150-case test suite spanning ambient sounds, human speech, and complex mixed scenes—using a Wan2.2-5B-initialized dual-branch Diffusion Transformer. Harmony introduces a Cross-Task Synergy training paradigm (co-training joint generation with audio-driven video and video-driven audio tasks to provide stable, uni-directional supervisory signals), a Global-Local Decoupled Interaction Module (separating RoPE-aligned frame-wise attention for fine-grained temporal synchronization from a reference-latent-based global style module for holistic consistency), and Synchronization-Enhanced CFG (SyncCFG) (repurposing "mute audio" and "static video" negative anchors to explicitly amplify the cross-modal alignment vector during inference). On the speech-video subset, progressively integrating all components lifts Sync-C from 4.20 to 6.51 and drops Sync-D from 10.93 to 8.63, establishing a new state-of-the-art in audio-visual synchronization that holds across both ambient and speech domains while also attaining competitive video quality and audio fidelity metrics.

2. Context and Motivation

The Core Problem: Joint Audio-Video Generation Consistently Fails at Fine-Grained Synchronization

The fundamental challenge this paper tackles is deceptively simple to state but extraordinarily difficult to solve: generating video and audio simultaneously in a way that genuinely coheres — where a speaker's lips move in exact correspondence with their voice, where a door slam's sound lands precisely on the frame where the door makes contact, where background music's emotional tenor matches the visual mood. Current open-source joint generation models, despite rapid progress in individual modality quality, systematically fail at this. The authors document this failure concretely in Table 1, where existing models show a stark pattern of partial capabilities: MM-Diffusion [42], JavisDiT [34], and AnimateSI [52] can generate environmental sounds but not human speech; JAM-Flow [30] handles speech but not ambient sounds; and even the more general UniVerse-1 [50] and Ovi [35] — which can generate both modalities — exhibit poor to fair alignment quality (✗ and ❍ ratings in Table 1 for speech-video and sound-video alignment respectively).

This gap is not a minor quality-of-life issue. It represents a fundamental barrier to deploying joint generation in any application where the output must feel real. A talking head with mismatched lip movements is uncanny — it triggers the same cognitive dissonance as poorly dubbed film. Environmental video with asynchronous sound effects breaks immersion immediately. Music video generation where the performer's motions don't track the beat feels amateurish. The paper's qualitative examples (Figure 5 and the supplementary Figures 7, 8) make this visceral: competing models produce frozen or minimally-moving faces during speech, generate static waves while ocean sounds play, or output sound whose waveform bears no structural relationship to the visual event.

Why This Problem Is Both Urgent and Foundational

The significance of solving audio-visual synchronization extends along several axes that the paper surfaces both explicitly and implicitly:

The open-source gap. The authors point to proprietary systems — Veo 3 [8] and Sora 2 [37] — as having "set a high benchmark" with "remarkable fidelity and substantial practical utility." The unstated implication is that these closed-source models have largely solved the synchronization problem (or at least reached commercially acceptable levels), while open-source alternatives remain far behind. This creates a landscape where academic research, startup innovation, and applications requiring transparency or customization are locked out of the most compelling generative capabilities. Closing this gap is both a scientific challenge and a matter of democratizing access to capable audio-visual AI.

The completeness problem in video generation. The paper situates itself against the broader video generation literature, noting that models like AnimateDiff [13], SVD [1], HunyuanVideo [29], and Wan [49] have achieved state-of-the-art visual quality — but produce silent videos. These outputs "feel incomplete and lack the immersive quality of real-world experiences." This is not merely an aesthetic critique; it's a functional one. Real video almost never exists without audio (surveillance footage is a rare exception), and humans are exquisitely sensitive to the absence of expected sound. A door closing on screen should produce a corresponding audio event — when it doesn't, the experience feels wrong in a way that undermines the entire generation's perceived quality.

The application pull. While the paper doesn't extensively enumerate applications, the implications are clear from context: digital avatars requiring synchronized speech, automated content creation for social media and advertising, video game asset generation where sound effects must match visual events, film pre-visualization, and accessibility tools for the hearing impaired that require precise audio-visual alignment. Each of these domains demands not just any audio paired with video, but audio that is causally and temporally bound to the visual stream.

A diagnostic, not just engineering, contribution. Perhaps most significantly, the paper frames its contribution not as "we built a better model through more compute and data" but as "we identified why joint diffusion systematically fails at synchronization and designed targeted interventions." This diagnostic stance — the identification of Correspondence Drift, the conflation of global and local alignment, and the inadequacy of standard CFG — represents a theoretical contribution to understanding multimodal generative models, not just an engineering improvement.

Prior Approaches and Their Systematic Shortcomings

The paper organizes prior work along several axes and identifies specific failure modes for each:

Generation-specialized models (limited modality coverage). Early open-source joint generation models — MM-Diffusion [42], JavisDiT [34], AnimateSI [52] — focused on environmental sound generation, treating the audio stream as a coarse accompaniment to visual events. These models were architecturally incapable of generating human speech because speech requires modeling fine-grained phonetic structure (sub-second timing, specific mouth shapes) that environmental sound models were not designed to capture. Conversely, JAM-Flow [30] focused exclusively on speech-video synthesis, achieving good lip-sync but having no capability for ambient sounds, music, or mixed acoustic scenes. This specialization reflects a deeper issue: the architectural and training demands of speech generation (phonetic precision, speaker identity disentanglement) differ substantially from those of environmental sound generation (event detection, temporal onset precision, spectral realism).

Generalist models (alignment quality failures). The more recent UniVerse-1 [50] and Ovi [35] represent attempts at general joint generation — models that can produce both speech and environmental sounds. Table 1 rates their speech-video alignment as "Poor/Unsupported" (UniVerse-1) and "Fair/Limited" (Ovi), and their sound-video alignment as "Fair/Limited" and "Fair/Limited" respectively. These are not models that occasionally misalign — they systematically underperform on the core synchronization metric. The paper's qualitative evidence (Figure 5) makes this concrete: competing models produce talking heads where the mouth doesn't track the speech, or music performances where the visual motion is minimal and the audio-video coupling is loose at best.

The deeper methodological gap. The paper's key insight — and what distinguishes it from prior work — is that these failures are not accidental or fixable through simple scaling. They are the downstream symptoms of three fundamental, unaddressed problems in the joint diffusion formulation itself:

  1. Correspondence Drift (Section 3.2.1): No prior work has diagnosed the inherent instability of trying to learn cross-modal correspondence when both modalities are simultaneously noisy. The standard joint diffusion objective (Equation 1) optimizes noise prediction for both streams simultaneously. In early training, both latent representations are dominated by Gaussian noise, meaning the model is attempting to learn an alignment mapping between two essentially random variables. The "optimal" correspondence in this regime shifts iteration to iteration as both representations evolve, creating a learning target that fundamentally drifts. This explains the slow convergence and poor final alignment observed empirically (Figure 3).

  2. Architectural conflation of objectives (Section 3.3): Prior models typically use a single global cross-attention mechanism between audio and video streams. This forces one attention module to simultaneously solve two qualitatively different problems: (a) establishing frame-level temporal correspondence (which lip frame maps to which phoneme segment), and (b) propagating global stylistic attributes (the overall emotional tenor, the ambient acoustic character of the scene). These objectives have conflicting demands — frame-level alignment benefits from local, high-resolution attention windows, while global style requires aggregation across the entire sequence. A single mechanism forced into this trade-off achieves neither objective well.

  3. CFG's intra-modal bias (Section 3.4): Standard Classifier-Free Guidance (Equation 8) amplifies the conditioning signal — pushing the denoising process away from an unconditional (null-text) prediction and toward a text-conditioned one. But this guidance vector is purely intra-modal: it strengthens how well each modality separately matches the text prompt, without any mechanism for promoting cross-modal consistency. A generation can be highly text-adherent for both audio and video individually (the audio matches the audio prompt, the video matches the video prompt) while being completely desynchronized between modalities. Standard CFG has no term that penalizes such desynchronization.

How Harmony Positions Itself

The paper positions Harmony not as an incremental improvement over prior models, but as a methodological intervention that directly targets each of these three root causes. This positioning is explicit in the paper's abstract and introduction, which frame the contributions as solutions to identified problems rather than as abstract architectural innovations:

  • Cross-Task Synergy targets Correspondence Drift by introducing a training paradigm where the model alternately conditions on clean (noise-free) latents from one modality while denoising the other. This provides stable, uni-directional supervision that the purely joint setting cannot offer. The paper shows empirically (Figure 3) that an audio-driven model (clean audio, noisy video) converges to high alignment scores rapidly, while the joint model (both noisy) lags substantially. The Cross-Task Synergy training objective (Equation 3) integrates these stable signals as a weighted auxiliary loss alongside the primary joint loss.

  • Global-Local Decoupled Interaction addresses the architectural conflation by splitting the interaction into two specialized sub-modules: RoPE-Aligned Frame-wise Attention (Section 3.3.1) handles fine-grained temporal synchronization using local attention windows with rescaled rotary position embeddings to account for the mismatched sampling rates of audio and video latents; Global Style Alignment (Section 3.3.2) handles holistic consistency by modulating a reference audio latent with global video context, keeping the style injection separated from the temporal alignment pathway.

  • SyncCFG (Section 3.4) directly addresses CFG's intra-modal bias by redesigning the negative anchor. Instead of using an unconditional (null-text) prediction as the baseline, SyncCFG uses a cross-modal negative: for video guidance, the negative anchor is the prediction conditioned on "mute audio" (Equation 9); for audio guidance, it's the prediction conditioned on "static video" (Equation 10). The guidance vector then amplifies specifically the difference attributable to cross-modal coupling — the visual dynamics caused by sound, and the acoustic events caused by motion.

The paper also positions itself practically through the introduction of Harmony-Bench, which it argues is necessary because existing benchmarks are inadequate. JavisBench [34] lacks human speech evaluation entirely. Verse-Bench [50] has low-quality labels and limited synchronization coverage. Harmony-Bench is designed with three explicitly tiered subsets (ambient-only, speech-only, complex mixed scenes) to disentangle evaluation of different capability dimensions, and with a comprehensive metric suite covering video quality, audio fidelity, and synchronization.

Importantly, the paper frames its contribution as validating a principle — that targeted methodological interventions at the level of training paradigm, architectural design, and inference-time guidance can overcome the synchronization barrier — rather than claiming that more data or larger models alone would suffice. The ablation study (Table 3), which shows monotonic improvements as each component is added (from Sync-C 4.20 to 6.51), is presented as evidence that each identified root cause is real and that each proposed solution addresses it effectively.

3. Technical Approach

3.1 Reader Orientation (Approachable Technical Breakdown)

The system being built is a unified generative model that can produce synchronized audio and video together — given a reference image, text descriptions, and optionally a speech transcript and reference voice sample, it outputs a video clip with perfectly matched sound. It solves the problem that existing joint audio-video models produce desynchronized outputs — lip movements that don't track speech, sound effects that land at the wrong moment, ambient audio that feels disconnected from the visual scene — by identifying and fixing three specific mechanistic failures in how joint diffusion models learn cross-modal alignment.

3.2 Big-Picture Architecture (Diagram in Words)

The Harmony system has five major structural components:

  1. Dual-branch Diffusion Transformer backbone: A video generation branch (initialized from the pre-trained Wan2.2-5B model) and a symmetric audio generation branch, each processing their respective modality's noisy latents through parallel transformer stacks. Both branches operate in a shared latent space produced by modality-specific VAEs.

  2. Multi-encoder conditioning frontend: Three separate encoders preprocess conditioning inputs before they reach the transformer — a speech encoder for phonetic transcripts (preserving fine-grained pronunciation information), a T5 text encoder for descriptive prompts (capturing semantic scene content), and an audio VAE for reference audio (capturing speaker timbre).

  3. Global-Local Decoupled Interaction Module: At every transformer layer, the two branches exchange information through two specialized attention mechanisms — one handles precise frame-level temporal correspondence (which video frame maps to which audio moment), the other handles holistic stylistic consistency (the overall emotional tone, the ambient character of the scene). These are deliberately separated so they don't interfere with each other.

  4. Cross-Task training controller: During training, the system alternates between three generation modes — standard joint generation (both modalities denoised from noise simultaneously), audio-driven video generation (clean audio conditions noisy video), and video-driven audio generation (clean video conditions noisy audio). A weighted loss combines all three objectives.

  5. SyncCFG inference guidance: At inference time, rather than using standard classifier-free guidance (which amplifies text-adherence only), the system uses specially constructed negative anchors — "mute audio" and "static video" — to compute guidance vectors that specifically amplify the cross-modal synchronization signal.

Information flows as follows: conditioning inputs (reference image, text prompt, transcript, reference audio) are encoded → noisy latents for both modalities are initialized from Gaussian noise → at each denoising step, both branches process their respective latents through self-attention, then exchange information through the Global-Local Decoupled Interaction Module, then produce noise predictions → these predictions are modified by SyncCFG guidance → the latents are updated toward less noisy states → after all steps, the final clean latents are decoded through VAEs into synchronized video frames and an audio waveform.

3.3 Roadmap for the Deep Dive

  • First, the Correspondence Drift diagnosis (Section 3.2.1) — what it is, the empirical evidence for it in Figure 3, and why it motivates the entire Cross-Task Synergy approach. This is foundational because it explains why joint training alone fails at synchronization.
  • Second, the Cross-Task Synergy training objective (Section 3.2.2) — the exact loss formulation (Equations 3 and 4), what each term contributes, and how the training hyperparameters are set. This is the mechanism that directly addresses drift.
  • Third, the dual-branch architecture and audio conditioning pipeline (also Section 3.2.2) — the VAE encoders, the separate speech and text encoders, the reference audio prepending, and why these design choices matter for synchronization. This establishes the structural substrate on which the interaction modules operate.
  • Fourth, the Global-Local Decoupled Interaction Module (Section 3.3) — first the RoPE-Aligned Frame-wise Attention (Section 3.3.1) for fine-grained temporal synchronization, explaining the sampling rate mismatch problem, the RoPE rescaling trick, and the local windowed cross-attention mechanism; then the Global Style Alignment module (Section 3.3.2) for holistic consistency, explaining how the reference audio latent serves as a style carrier without disrupting fine-grained denoising.
  • Fifth, Synchronization-Enhanced CFG (Section 3.4) — the limitation of standard CFG (Equation 8), the construction of "mute audio" negative anchors for video guidance (Equation 9), the construction of "static video" negative anchors for audio guidance (Equation 10), and why isolating the cross-modal component of the prediction difference amplifies synchronization specifically.
  • Sixth, the training curriculum and hyperparameters (from Appendix B) — the three-stage training protocol, the data mixture strategy, the exact learning rates, batch sizes, iteration counts, loss weights, and inference settings. These operational details are essential for understanding the full system.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a methodological intervention paper whose core idea is that joint audio-video diffusion fails at synchronization due to three specific, identifiable mechanistic flaws — Correspondence Drift in the training objective, architectural conflation of global and local alignment objectives, and CFG's inability to promote cross-modal consistency — and that targeted fixes for each (Cross-Task Synergy training, a Global-Local Decoupled Interaction Module, and Synchronization-Enhanced CFG) collectively resolve these failures.

The Correspondence Drift Diagnosis (Section 3.2.1)

The paper's central diagnostic claim is that the standard joint diffusion training objective (Equation 1) contains an inherent instability that prevents the model from learning robust cross-modal alignment. The authors name this phenomenon Correspondence Drift and provide both a theoretical characterization and empirical evidence.

The standard joint audio-video diffusion loss is:

L=∣∣ϵv−ϵ^v(zv,t,za,t,t)∣∣2+∣∣ϵa−ϵ^a(za,t,zv,t,t)∣∣2\mathcal{L} = ||\epsilon_v - \hat{\epsilon}_v(z_{v,t}, z_{a,t}, t)||^2 + ||\epsilon_a - \hat{\epsilon}_a(z_{a,t}, z_{v,t}, t)||^2

where $\epsilon_v$ and $\epsilon_a$ are the actual noise added to the video and audio latents respectively, $\hat{\epsilon}_v(z_{v,t}, z_{a,t}, t)$ is the model's predicted video noise given the noisy video latent $z_{v,t}$ and the noisy audio latent $z_{a,t}$ at timestep $t$, and $\hat{\epsilon}_a(z_{a,t}, z_{v,t}, t)$ is the corresponding predicted audio noise.

What it computes: For each modality, the squared L2 distance between the true injected noise and the model's prediction of that noise. The total loss is the sum of these two modality-specific terms. Minimizing this loss trains the model to denoise both streams simultaneously, with each modality's denoising conditioned on the current state of the other modality.

Why this form causes Correspondence Drift: During early training (high $t$, corresponding to heavily noised latents), both $z_{v,t}$ and $z_{a,t}$ are dominated by Gaussian noise. The model is being asked to learn a conditional mapping $\hat{\epsilon}_v(z_{v,t}, z_{a,t}, t)$ where $z_{a,t}$ is essentially random and changing rapidly as the model's own parameters update and shift the distribution of $z_{a,t}$ at each training iteration. The "optimal" video noise prediction for a given audio latent keeps changing because the audio latent itself keeps changing — both because the audio branch is also learning and because the noise sampling is stochastic. This creates a moving target for the cross-modal interaction: the correspondence the model should learn today is different from the correspondence it should have learned yesterday, causing the learning signal to drift. The same logic applies symmetrically to the audio noise prediction.

Evidence for drift: Figure 3 provides the empirical smoking gun. The authors compare two training setups that use an identical network architecture: (1) an audio-driven video generation task, where the audio timestep is fixed at $t_a = 0$ (clean audio conditioning a noisy video), and (2) the standard joint generation task, where both modalities are denoised from noise simultaneously. The audio-driven model "rapidly converges to a high alignment score" (as measured by Sync Confidence, rising from approximately 1.5 to 4.5-5.0 over 5,000 iterations) while the joint generation model "exhibits markedly slower convergence" (Sync Confidence rising only to approximately 2.5-3.0 over the same period). The Sync Distance metric shows the inverse pattern (lower is better): the audio-driven model drops from roughly 14 to 10.5, while the joint model only reaches roughly 12.5.

The interpretation is straightforward: when one modality is clean (deterministic, noise-free), the cross-modal interaction module receives a stable gradient signal — it learns "given this specific audio pattern, the corresponding video should look like this." When both modalities are noisy, the gradient signal is noisy in both directions — the interaction module receives conflicting updates because neither the input nor the target is stable. The Cross-Task Synergy training strategy is the direct response to this diagnosis.

Cross-Task Synergy Training Objective (Section 3.2.2)

Overall training strategy. The Cross-Task Synergy training paradigm combines the standard joint generation loss with two auxiliary losses from uni-directional generation tasks. The key insight is that these auxiliary tasks — audio-driven video generation and video-driven audio generation — provide high-quality, stable supervisory signals because in each case one modality's latent is clean (timestep zero) while the other is noisy. These stable signals "instill robust alignment priors" that then accelerate and stabilize the learning of alignment in the primary joint generation task.

Total training objective:

L=Ljoint+λvLdrivenaudio+λaLdrivenvideo\mathcal{L} = \mathcal{L}_{\text{joint}} + \lambda_v \mathcal{L}_{\text{driven}}^{\text{audio}} + \lambda_a \mathcal{L}_{\text{driven}}^{\text{video}}

where $\lambda_v = 0.1$ is the weight for the audio-driven video generation loss, and $\lambda_a = 0.3$ is the weight for the video-driven audio generation loss. These hyperparameter values are specified in Appendix B and reflect an asymmetry: the video-driven audio loss receives triple the weight of the audio-driven video loss, suggesting that providing clean video supervision to the audio branch is particularly important for learning synchronization.

What it computes: A weighted sum of three noise-prediction losses. The joint loss $\mathcal{L}_{\text{joint}}$ penalizes error in predicting noise for both modalities simultaneously (as in standard joint training). The audio-driven loss $\mathcal{L}_{\text{driven}}^{\text{audio}}$ penalizes error in predicting video noise when conditioned on a clean (noise-free) audio latent — this forces the video branch to learn precise audio-to-video mappings using a stable audio input. The video-driven loss $\mathcal{L}_{\text{driven}}^{\text{video}}$ penalizes error in predicting audio noise when conditioned on a clean video latent — this forces the audio branch to learn precise video-to-audio mappings using a stable video input.

Component losses (Equation 4):

Ljoint=∣∣ϵv−ϵ^v(zv,t,za,t,c,t)∣∣2+∣∣ϵa−ϵ^a(za,t,zv,t,c,t)∣∣2\mathcal{L}_{\text{joint}} = ||\epsilon_v - \hat{\epsilon}_v(z_{v,t}, z_{a,t}, \mathbf{c}, t)||^2 + ||\epsilon_a - \hat{\epsilon}_a(z_{a,t}, z_{v,t}, \mathbf{c}, t)||^2

where $\mathbf{c}$ represents the set of all auxiliary conditions (text prompts, speech embeddings, reference audio latents). This is the standard joint training loss: both modalities are at the same noise level $t$, and each denoising step is conditioned on the other modality's current noisy state.

Ldrivenaudio=∣∣ϵv−ϵ^v(zv,t,za,0,c,t)∣∣2\mathcal{L}_{\text{driven}}^{\text{audio}} = ||\epsilon_v - \hat{\epsilon}_v(z_{v,t}, z_{a,0}, \mathbf{c}, t)||^2

where $z_{a,0}$ is the clean (noise-free) audio latent, achieved by setting the audio timestep $t_a = 0$. The model predicts video noise $\epsilon_v$ given the noisy video latent $z_{v,t}$ and the clean audio latent $z_{a,0}$. This provides a deterministic, stable audio conditioning signal — the video branch learns that specific audio patterns correspond to specific video patterns without the added stochasticity of a simultaneously noisy audio input.

Ldrivenvideo=∣∣ϵa−ϵ^a(za,t,zv,0,c,t)∣∣2\mathcal{L}_{\text{driven}}^{\text{video}} = ||\epsilon_a - \hat{\epsilon}_a(z_{a,t}, z_{v,0}, \mathbf{c}, t)||^2

where $z_{v,0}$ is the clean video latent (video timestep $t_v = 0$). The model predicts audio noise $\epsilon_a$ given the noisy audio latent $z_{a,t}$ and the clean video latent $z_{v,0}$. This provides a deterministic, stable video conditioning signal for the audio branch.

Why this form: The critical design choice is that the auxiliary tasks always condition one modality's denoising on the clean latent of the other modality. This is fundamentally different from simply adding more joint training data — it's a qualitatively different learning signal. In the audio-driven task, the model sees: "when the audio is definitively X, the video should look like Y." This is a supervised learning signal for cross-modal correspondence. In the joint task, the model sees: "when the audio is probably-kind-of X but also noisy, the video should probably-kind-of look like Y." This is a weakly supervised signal at best. The auxiliary tasks provide the strong signal, and the joint task ensures the model can generalize to the regime where both modalities are noisy (the actual inference condition). The asymmetric weights ($\lambda_v = 0.1$, $\lambda_a = 0.3$) suggest the authors found that video-to-audio supervision was more valuable than audio-to-video — possibly because learning to generate temporally precise audio (which must track sub-frame-level events) from clean video is harder and benefits more from explicit training.

Training implementation. During training, the system samples which task to run per batch or per example (the paper doesn't specify the exact sampling strategy, but the implication from Figure 2 and the loss formulation is that all three losses are computed and weighted). The audio-driven task sets $t_a = 0$ while $t_v$ is sampled normally from the noise schedule. The video-driven task sets $t_v = 0$ while $t_a$ is sampled normally. The joint task samples the same timestep $t$ for both modalities.

The convergence benefit (validated in Figure 3). The paper shows in Figure 3 that a Cross-Task Synergy model (red/dashed line) achieves Sync Confidence scores that track the audio-driven baseline much more closely than the pure joint generation baseline — rising to approximately 4.0-4.5 by 5,000 iterations versus the joint baseline's roughly 2.5-3.0. The Sync Distance for the Cross-Task Synergy model drops to roughly 10.5, close to the audio-driven model's performance and substantially better than the joint baseline's roughly 12.5. This validates the core hypothesis: the stable supervisory signals from the uni-directional tasks act as a catalyst, transferring alignment knowledge to the joint generation pathway.

Dual-Branch Architecture and Audio Conditioning Pipeline (Section 3.2.2)

Video branch. The video generation pathway adapts the pre-trained Wan2.2-5B model [49], a state-of-the-art open-source video diffusion transformer. The adaptation involves adding cross-modal interaction layers (the Global-Local Decoupled Interaction Module) at each transformer layer while preserving the pre-trained visual generation capabilities. The video branch is conditioned on a reference image (providing initial visual content and identity) and a descriptive text prompt (describing the scene content, action, and style). These are the standard Wan2.2 conditioning inputs.

Audio branch. The audio generation pathway is a symmetric transformer designed to mirror the video branch's architecture for structural parity — both branches have the same number of layers, the same hidden dimension, and the same interaction module placement, enabling bidirectional cross-modal communication at every layer. The audio branch synthesizes an audio clip $A$ conditioned on three inputs:

  1. Speech transcript $T_s$: The phonetic content to be spoken. Crucially, the paper departs from prior works [34, 35] by using a dedicated speech encoder [3] rather than a general-purpose text encoder. This speech encoder is specifically designed to capture fine-grained pronunciation information (phoneme sequences, prosodic features) that are essential for accurate lip-sync. The output is the speech embedding $\mathbf{e}_{\text{speech}}$.

  2. Descriptive audio caption $T_a$: A text description of the acoustic scene — for example, "a man speaking in a calm voice with gentle background music" or "gunshots firing rapidly with sharp loud cracking sounds." This is processed through a T5 text encoder [6], producing the prompt embedding $\mathbf{e}_{\text{prompt}}$. Using separate encoders for phonetic content and semantic description ensures that speech precision and scene semantics don't interfere with each other — the speech encoder specializes in pronunciation, the T5 encoder specializes in semantic understanding.

  3. Reference audio $A_r$: A short audio clip (typically 1-3 seconds, as specified in Appendix B) that captures the desired speaker's voice timbre — the unique acoustic signature of their voice (pitch range, spectral characteristics, speaking style). Both the target audio $A$ and the reference audio $A_r$ are encoded through the same audio VAE [5], producing latents $z_a$ and $z_r$ respectively. The reference latent $z_r$ captures time-invariant speaker characteristics while discarding the original phonetic content.

Audio latent construction for denoising. During the denoising process, the reference audio latent $z_r$ is prepended to the noisy target audio latent $z_{a,t}$, forming a composite input latent $z'_{a,t} = \text{concat}(z_r, z_{a,t})$. This composite latent is what the audio branch's transformer processes. Prepending (rather than, say, adding or cross-attending the reference separately) means the reference information is available through the transformer's self-attention mechanism at every layer, providing a persistent timbre conditioning signal that doesn't interfere with the cross-modal interaction modules (which are reserved for audio-video synchronization).

Audio noise prediction (Equation 2):

ϵ^a=MM-DiT(concat(za,t′,espeech,eprompt),ta)\hat{\epsilon}_a = \text{MM-DiT}(\text{concat}(z'_{a,t}, \mathbf{e}_{\text{speech}}, \mathbf{e}_{\text{prompt}}), t_a)

where $\hat{\epsilon}_a$ is the predicted audio noise, $\text{MM-DiT}$ is the Multi-Modal Diffusion Transformer (the audio branch), $z'_{a,t}$ is the composite latent (reference prepended to noisy target), $\mathbf{e}_{\text{speech}}$ is the speech embedding, $\mathbf{e}_{\text{prompt}}$ is the text prompt embedding, and $t_a$ is the audio timestep.

What it computes: The audio branch takes the concatenation of the composite audio latent (reference + noisy target) with the conditioning embeddings, processes them through its transformer layers (including cross-modal interaction with the video branch), and outputs a prediction of the noise that was added to the target audio latent. This prediction is then used to update the noisy latent toward a cleaner state.

Why this design: Several deliberate architectural choices manifest here. First, using separate text encoders for speech versus description is a critical departure from prior work [34, 35] which used a single text encoder for both. The motivation is that phonetic precision and semantic understanding are qualitatively different linguistic tasks: the speech encoder needs to capture sub-phoneme timing information relevant to mouth shapes, while the T5 encoder needs to capture high-level semantic content. A single encoder forced to do both would likely sacrifice phonetic precision for semantic breadth or vice versa. Second, prepending the reference latent to the target latent ensures the reference information is accessible throughout the entire transformer depth via self-attention, providing a persistent timbre conditioning that doesn't require additional architectural modifications. Third, structural parity between the video and audio branches (same depth, same interaction module placement) enables symmetric bidirectional communication — the interaction modules at layer $i$ in the video branch can directly exchange with the interaction modules at the same layer $i$ in the audio branch, maintaining aligned representational depth.

Cross-modal interaction placement. The paper specifies that "a bidirectional global-local decoupled interaction module" is instantiated at each layer of both branches. This means that at every transformer layer, after the self-attention and feed-forward operations, the video branch's latent representation and the audio branch's latent representation exchange information through the Global-Local Decoupled Interaction Module before proceeding to the next layer. This deep, layer-wise interaction is more expressive than shallow interaction (e.g., only at the final layer) because it allows the modalities to influence each other's representations at multiple levels of abstraction — early layers can align low-level features (local textures, spectral patterns), while later layers can align high-level features (semantic content, emotional tone).

RoPE-Aligned Frame-wise Attention (Section 3.3.1)

The temporal resolution mismatch problem. Audio and video latents operate at fundamentally different temporal resolutions. The video latent $z_v$ has $T_v$ frames (temporal positions), while the audio latent $z_a$ has $T_a$ frames, where $T_v \neq T_a$ due to the different sampling rates of the video and audio VAEs. This means a discrete event — say, a lip movement occurring at video frame 12 — has no exact corresponding audio frame if $T_a$ is not an integer multiple of $T_v$. The audio frame that temporally aligns with video frame 12 might fall at position 12 · $(T_a/T_v)$ = 28.3 — between audio frames 28 and 29. A standard discrete attention mechanism would have to attend to either frame 28 or frame 29, both of which are temporally imperfect. This "forced approximation introduces temporal jitter and fundamentally degrades fine-grained synchronization."

The RoPE rescaling solution. Rotary Position Embeddings (RoPE) encode position information by rotating the query and key vectors in attention based on their absolute positions. The core insight is that RoPE provides a continuous positional encoding space — two positions that are close in time have similar RoPE rotations, regardless of whether they correspond to discrete frame boundaries. The authors exploit this by rescaling the positional indices before computing RoPE.

For audio-to-video (A2V) attention, an audio frame at index $j$ is mapped to a virtual position:

j′=j⋅TvTaj' = j \cdot \frac{T_v}{T_a}

This virtual position $j'$ is used to compute the RoPE encoding for the audio frame's query/key vectors. In effect, the audio frame is treated as if it exists at a continuous position $j'$ on the video's timeline. When the video branch computes attention with this audio frame, the RoPE encoding naturally aligns it with the video frames closest to position $j'$ — the attention scores will be highest for video frames whose RoPE positions are nearest to $j'$, achieving sub-frame temporal alignment without requiring interpolation of the latent representations themselves.

Symmetrically, for video-to-audio (V2A) attention, a video frame at index $i$ is mapped to a virtual position $i' = i \cdot (T_a/T_v)$ on the audio timeline.

Why RoPE rescaling rather than interpolation: An alternative approach would be to temporally interpolate one modality's latent to match the other's frame rate. However, this would introduce interpolation artifacts (blurring) and would require choosing which modality to interpolate (breaking symmetry). The RoPE rescaling approach is non-destructive — it doesn't modify the latent representations at all, it only adjusts the positional encoding used in the attention computation. This preserves the full information content of each modality's latent while enabling precise temporal correspondence through the attention mechanism's natural soft-alignment behavior. The approach is also symmetric and computationally cheap — it's just an index remapping before the RoPE computation.

Frame-wise cross-attention mechanism (A2V direction). After RoPE rescaling, the actual attention computation operates within local temporal windows. Rather than allowing each video frame to attend to all audio frames (global attention), each video frame attends only to a small context window of adjacent audio frames. The latent $z_v$ is first reshaped to expose its temporal dimension, yielding $z'_v$ where the temporal axis is explicit. For each video frame $i \in [0, T_v-1]$:

Δzv′[:,i,:,:]=Cross-Attn(Qv,i,Ka,i,Va,i)\Delta z'_v[:, i, :, :] = \text{Cross-Attn}(Q_{v,i}, K_{a,i}, V_{a,i})

where $Q_{v,i} = z'_v[:, i, :, :] W^Q_{v,i}$ is the query vector for video frame $i$, $K_{a,i} = C_{a,i} W^K_{a,i}$ and $V_{a,i} = C_{a,i} W^V_{a,i}$ are the key and value matrices constructed from a local context window $C_{a,i}$ of audio frames surrounding the temporally corresponding position (after RoPE rescaling), and $W^Q_{v,i}$, $W^K_{a,i}$, $W^V_{a,i}$ are learned projection matrices.

What it computes: For each video frame individually, a cross-attention operation where the query comes from that single video frame and the keys/values come from a temporally local window of audio frames. The RoPE rescaling ensures that the "temporally corresponding position" is computed in continuous time, so the local audio window is centered on the correct sub-frame position. The output $\Delta z'_v[:, i, :, :]$ is an update to that video frame's representation that incorporates information from the temporally aligned audio frames.

Why frame-wise and local: Global cross-attention (where each video frame attends to all audio frames) would allow any video frame to potentially attend to any audio frame, regardless of temporal distance. While this sounds flexible, it actually makes the temporal correspondence problem harder — the model has to learn to ignore temporally distant audio frames from scratch, and this learning signal is weak (there are many more temporally irrelevant than temporally relevant audio frames for any given video frame). Local windowed attention imposes a strong inductive bias: video frame $i$ should only be influenced by audio frames temporally near position $i' = i \cdot (T_a/T_v)$. This dramatically reduces the search space for learning temporal alignment and prevents the model from developing spurious long-range cross-modal dependencies. It is also more computationally efficient — the attention complexity drops from $O(T_v \cdot T_a)$ (global) to $O(T_v \cdot w)$ where $w$ is the window size (local).

Residual integration (Equation 6). After computing the cross-attention updates for all frames in both directions:

zvupdated=zv+Δzv′,zaupdated=za+Δzaz_v^{\text{updated}} = z_v + \Delta z'_v, \quad z_a^{\text{updated}} = z_a + \Delta z_a

The updates $\Delta z'_v$ and $\Delta z_a$ (the V2A analog of $\Delta z'_v$) are added back to the original latents via residual connections. This means the frame-wise attention refines the representations rather than replacing them — the model can preserve information from the self-attention and feed-forward layers that is modality-specific while incorporating cross-modal temporal alignment information.

Why residual rather than replacement: Residual connections provide a gradient highway that allows the model to learn the cross-modal mapping gradually. If the cross-attention output replaced the latent entirely, the model would be forced to immediately learn useful cross-modal representations or risk losing all modality-specific information. The residual formulation lets the model start with near-zero cross-modal influence and ramp it up as training progresses, which is more stable. It also preserves the ability to represent modality-specific features that have no cross-modal counterpart (e.g., background details in video that don't produce sound).

Global Style Alignment Module (Section 3.3.2)

The problem with monolithic attention. Frame-wise local attention excels at establishing precise temporal correspondence, but its localized nature inherently limits the propagation of global stylistic attributes. The overall emotional tone of a scene, the ambient acoustic character of an environment, the speaking style of a narrator — these are properties that require aggregating information across the entire sequence, not just a local window. Prior methods that used a single global attention mechanism forced one module to simultaneously learn both fine-grained temporal alignment (benefiting from locality) and holistic style propagation (requiring globality), creating a fundamental architectural tension.

The decoupling strategy. The paper's solution is principled decomposition: the RoPE-Aligned Frame-wise Attention (Section 3.3.1) is exclusively responsible for precise temporal correspondence, while a separate Global Style Alignment module handles holistic consistency. The two modules operate independently and their outputs are combined through the residual connections in the main transformer — there is no direct competition between them.

The reference latent as style carrier. The global style alignment module's key insight is to use the reference audio latent $z_r$ as the vehicle for style information transfer. Recall that $z_r$ encodes the reference speaker's timbre — their unique vocal signature. This latent is compact (it encodes a short reference clip, not the full generated audio) and is already integrated into the audio generation pipeline (it's prepended to $z_{a,t}$). The global style module modulates $z_r$ with global visual context, producing a visually-informed reference latent that guides the audio generation toward stylistic consistency with the video.

Global style attention operation (Equation 7):

zrupdated=zr+Cross-Attn(Qr,Kv,Vv)z_r^{\text{updated}} = z_r + \text{Cross-Attn}(Q_r, K_v, V_v)

where $Q_r = z_r W^Q_r$ is the query matrix derived from the reference audio latent, $K_v = z_v W^K_v$ and $V_v = z_v W^V_v$ are the key and value matrices derived from the full video latent (all temporal positions), and $W^Q_r$, $W^K_v$, $W^V_v$ are learned projection matrices.

What it computes: The reference audio latent $z_r$ acts as a query, attending to the entire video latent $z_v$ (all video frames) as key-value pairs. This is a global attention operation — unlike the frame-wise module which uses local windows, here the query (reference latent) can attend to any video frame regardless of temporal position. The output $\Delta z_r = \text{Cross-Attn}(Q_r, K_v, V_v)$ captures global visual context (the overall scene lighting, the dominant colors, the motion energy, the emotional register of facial expressions) and injects it into the reference latent. The updated reference latent $z_r^{\text{updated}}$ now encodes both the speaker's timbre and the visual scene's global style.

Downstream effect. The updated reference latent $z_r^{\text{updated}}$ is prepended to the noisy audio latent $z_{a,t}$ (as described in Section 3.2.2), so it conditions every step of the audio denoising process. The audio branch generates speech or environmental sounds that are timbre-matched to the reference speaker and stylistically consistent with the visual scene — for example, if the video shows a tense, dimly-lit scene with rapid camera movements, the audio generation will be biased toward tense, rapid, lower-pitched sounds even if the reference speaker's natural timbre is calm and measured.

Why reference-latent-based rather than direct audio modulation: An alternative approach would be to directly modulate the target audio latent $z_a$ with global video context — for instance, via a global cross-attention where $z_a$ queries $z_v$. The paper argues this would "disrupt fine-grained denoising" because $z_a$ is the primary signal being denoised and its content includes both global style and fine-grained temporal structure. Injecting global video context directly into $z_a$ through attention would mix style information with the temporal structure that the frame-wise attention module is trying to preserve, creating interference. By confining the global style injection to the reference latent $z_r$, the module keeps the style pathway separate from the temporal synchronization pathway. The target audio latent $z_a$ receives style information indirectly — via the prepended reference latent — without having its internal temporal structure modified by global attention.

Why this prevents interference: The decoupling is clean: the frame-wise attention module modifies $z_v$ and $z_a$ directly to establish temporal correspondence (local, high-resolution), while the global style module modifies only $z_r$ to establish stylistic consistency (global, low-resolution). The two modifications occur in parallel at each transformer layer and are combined through the standard residual connections and the prepending mechanism without ever competing for the same representational substrate. This is the architectural realization of the paper's claim that "each component can excel at its specific task."

Synchronization-Enhanced CFG — Analysis of Standard Limitations (Section 3.4.1)

Standard CFG formulation (Equation 8):

ϵ~=ϵ^θ(zv,t,za,t,∅c)+s(ϵ^θ(zv,t,za,t,c)−ϵ^θ(zv,t,za,t,∅c))\tilde{\epsilon} = \hat{\epsilon}_{\theta}(z_{v,t}, z_{a,t}, \emptyset_c) + s \Big( \hat{\epsilon}_{\theta}(z_{v,t}, z_{a,t}, c) - \hat{\epsilon}_{\theta}(z_{v,t}, z_{a,t}, \emptyset_c) \Big)

where $\tilde{\epsilon}$ is the guided noise prediction, $\hat{\epsilon}_{\theta}(z_{v,t}, z_{a,t}, c)$ is the model's noise prediction conditioned on the text prompt $c$, $\hat{\epsilon}_{\theta}(z_{v,t}, z_{a,t}, \emptyset_c)$ is the unconditional prediction (null-text condition $\emptyset_c$), and $s$ is the guidance scale (typically > 1).

What it computes: The guided prediction starts from the unconditional prediction $\hat{\epsilon}_{\theta}(z_{v,t}, z_{a,t}, \emptyset_c)$ and extrapolates away from it in the direction of the text-conditioned prediction. The difference vector $(\hat{\epsilon}_{\theta}(z_{v,t}, z_{a,t}, c) - \hat{\epsilon}_{\theta}(z_{v,t}, z_{a,t}, \emptyset_c))$ represents the effect of conditioning on text — it's the modification to the noise prediction that makes the output text-adherent. Multiplying by $s > 1$ and adding it back amplifies this effect, making the output more strongly text-conditioned.

The limitation: CFG is blind to cross-modal synchronization. The subtraction in Equation 8 isolates the text-conditioning signal: "what changes in the prediction when we tell the model what the content should be?" But it provides no information about cross-modal consistency. A generation can have perfect text adherence for both modalities independently — the video perfectly matches the video prompt, the audio perfectly matches the audio prompt — while being completely desynchronized. Standard CFG has no term that captures or amplifies the relationship between the two modalities. The guidance vector $(s \cdot (\hat{\epsilon}_{\theta}(c) - \hat{\epsilon}_{\theta}(\emptyset_c)))$ pushes the generation toward better conditioning, but conditioning and synchronization are orthogonal objectives.

Why this matters in practice: In joint audio-video generation, the text prompt typically describes both modalities (e.g., "a person speaking while birds chirp in the background"). Standard CFG will strengthen both the visual depiction of the person and the acoustic rendering of speech and birdsong, but it has no mechanism to ensure the lip movements line up with the speech or that the birdsong coincides with visual bird activity. The model can achieve low loss (high text adherence) while having terrible synchronization, because the loss function (Equation 1) doesn't directly penalize desynchronization — it penalizes per-modality noise prediction error, which can be low even when the cross-modal temporal alignment is poor if each modality individually is high-quality.

SyncCFG — Video Guidance Formulation (Section 3.4.2)

The core insight for SyncCFG. Rather than using an unconditional prediction as the negative anchor (which isolates the text-conditioning signal), SyncCFG uses a cross-modal negative: a prediction conditioned on one modality being "silent" or "static." The subtraction then isolates not the effect of text conditioning, but the effect of cross-modal coupling — specifically, how the presence of sound modifies the visual prediction, and how the presence of motion modifies the audio prediction. The guidance amplifies this cross-modal effect.

Constructing the "mute audio" negative anchor for video guidance. The audio-driven pathway of the model (trained during Cross-Task Synergy) can predict video noise conditioned on any audio latent, including a specially constructed "silent" audio. The negative anchor is:

ϵ^θdriven(zv,t,za,0null)\hat{\epsilon}_{\theta}^{\text{driven}}(z_{v,t}, z_{a,0}^{\text{null}})

where $z_{a,0}^{\text{null}}$ is a "muted audio" latent — a clean (timestep 0) audio latent representing silence. The paper doesn't specify exactly how this latent is constructed, but the operational meaning is clear: it's the audio latent that corresponds to a completely silent audio track (no speech, no ambient sound, no music). This represents the model's prediction of what the video should look like if there were no sound at all.

SyncCFG video guidance (Equation 9):

ϵ~v=ϵ^θdriven(zv,t,za,0null)+sv(ϵ^θjoint(zv,t,za,t)−ϵ^θdriven(zv,t,za,0null))\tilde{\epsilon}_v = \hat{\epsilon}_{\theta}^{\text{driven}}(z_{v,t}, z_{a,0}^{\text{null}}) + s_v \Big( \hat{\epsilon}_{\theta}^{\text{joint}}(z_{v,t}, z_{a,t}) - \hat{\epsilon}_{\theta}^{\text{driven}}(z_{v,t}, z_{a,0}^{\text{null}}) \Big)

where $\tilde{\epsilon}_v$ is the SyncCFG-guided video noise prediction, $\hat{\epsilon}_{\theta}^{\text{joint}}(z_{v,t}, z_{a,t})$ is the joint-generation prediction (conditioned on the actual noisy audio latent), $\hat{\epsilon}_{\theta}^{\text{driven}}(z_{v,t}, z_{a,0}^{\text{null}})$ is the audio-driven prediction conditioned on mute audio, and $s_v = 3$ is the video guidance scale (from Appendix B).

What it computes: The base prediction is the "silent video" prediction — what the video would look like if there were no sound. The guidance vector is $(\hat{\epsilon}_{\theta}^{\text{joint}}(z_{v,t}, z_{a,t}) - \hat{\epsilon}_{\theta}^{\text{driven}}(z_{v,t}, z_{a,0}^{\text{null}}))$, which subtracts the silent-video prediction from the joint prediction. This difference vector isolates the visual modifications that are directly caused by the presence of audio — the mouth movements driven by speech, the object motions driven by impact sounds, the environmental dynamics driven by ambient audio. Multiplying by $s_v$ and adding it to the silent-video baseline amplifies these audio-driven visual dynamics specifically, without affecting video features that are independent of audio (like static background elements or audio-irrelevant motions).

Why this isolates synchronization: The key property is that both predictions use the same noisy video latent $z_{v,t}$. The only difference between $\hat{\epsilon}_{\theta}^{\text{joint}}(z_{v,t}, z_{a,t})$ and $\hat{\epsilon}_{\theta}^{\text{driven}}(z_{v,t}, z_{a,0}^{\text{null}})$ is the audio conditioning — one uses the actual audio, the other uses silence. Any difference in the predicted video noise must therefore be attributable to the audio signal. Features that are audio-irrelevant (like background rendering quality, static scene elements, motion that would occur regardless of sound) will appear in both predictions and cancel out in the subtraction. The guidance vector is a pure measure of "how should the video change because of the sound," and amplifying it specifically enhances synchronization.

Concrete example: For a speaking person, $\hat{\epsilon}_{\theta}^{\text{driven}}(z_{v,t}, z_{a,0}^{\text{null}})$ would predict a video with a still face and a closed mouth (the silent baseline), while $\hat{\epsilon}_{\theta}^{\text{joint}}(z_{v,t}, z_{a,t})$ would predict a video with the mouth opening and closing to form speech sounds. The difference vector captures the mouth movement patterns, and amplifying it makes the mouth movements more pronounced and more precisely tied to the audio.

SyncCFG — Audio Guidance Formulation (Section 3.4.3)

Constructing the "static video" negative anchor for audio guidance. Symmetrically, the video-driven pathway predicts audio noise conditioned on a "static video" latent — a clean video latent representing a completely motionless scene:

ϵ^θdriven(zv,0null,za,t)\hat{\epsilon}_{\theta}^{\text{driven}}(z_{v,0}^{\text{null}}, z_{a,t})

where $z_{v,0}^{\text{null}}$ is a static video latent. This represents the model's prediction of what the audio should be in a motionless scene — ambient background sound only, with no sound events triggered by movement.

SyncCFG audio guidance (Equation 10):

ϵ~a=ϵ^θdriven(zv,0null,za,t)+sa(ϵ^θjoint(zv,t,za,t)−ϵ^θdriven(zv,0null,za,t))\tilde{\epsilon}_a = \hat{\epsilon}_{\theta}^{\text{driven}}(z_{v,0}^{\text{null}}, z_{a,t}) + s_a \Big( \hat{\epsilon}_{\theta}^{\text{joint}}(z_{v,t}, z_{a,t}) - \hat{\epsilon}_{\theta}^{\text{driven}}(z_{v,0}^{\text{null}}, z_{a,t}) \Big)

where $\tilde{\epsilon}_a$ is the SyncCFG-guided audio noise prediction, $\hat{\epsilon}_{\theta}^{\text{joint}}(z_{v,t}, z_{a,t})$ is the joint-generation prediction, $\hat{\epsilon}_{\theta}^{\text{driven}}(z_{v,0}^{\text{null}}, z_{a,t})$ is the video-driven prediction conditioned on static video, and $s_a = 2$ is the audio guidance scale (from Appendix B).

What it computes: The base prediction is the "still-scene audio" prediction — what the audio would be if nothing were moving in the video. The guidance vector is $(\hat{\epsilon}_{\theta}^{\text{joint}}(z_{v,t}, z_{a,t}) - \hat{\epsilon}_{\theta}^{\text{driven}}(z_{v,0}^{\text{null}}, z_{a,t}))$, which subtracts the still-scene prediction from the joint prediction. This isolates the acoustic events that are directly caused by visual motion — the sound of footsteps from walking, the impact sound from a door closing, the rustle of leaves from wind visible in the video. Multiplying by $s_a$ and adding it amplifies these motion-driven sounds specifically.

Why the guidance scales differ ($s_v = 3$, $s_a = 2$): The paper doesn't explicitly explain this asymmetry, but a plausible interpretation is that video generation benefits more from amplified synchronization guidance than audio generation does — or equivalently, that the silent-video negative anchor provides a cleaner isolation of the cross-modal signal than the static-video negative anchor does. A static video is unambiguous (nothing moves), but a "silent audio" might be less cleanly defined (what is the spectral signature of true silence in a learned latent space?). The higher video guidance scale compensates for potentially noisier negative anchors or for video's greater inherent complexity (more dimensions to synchronize).

Why two separate guidance equations rather than one joint guidance: SyncCFG applies guidance independently to each modality's noise prediction using modality-specific negative anchors. This decoupling is important because the cross-modal effects are asymmetric: sound→video modifications (mouth movements, object vibrations) are qualitatively different from video→audio modifications (impact sounds, motion-triggered audio events). Using separate guidance formulations allows different scales ($s_v \neq s_a$) and different negative anchor constructions for each direction.

The crucial training prerequisite. SyncCFG fundamentally depends on the Cross-Task Synergy training. The model must be capable of producing $\hat{\epsilon}_{\theta}^{\text{driven}}$ predictions — that is, it must have learned to generate video from clean audio and audio from clean video during training. Without this capability, there would be no way to compute the audio-driven or video-driven predictions that form the negative anchors. This creates a tight coupling between the training innovation (Cross-Task Synergy) and the inference innovation (SyncCFG) — the former enables the latter.

Why not just increase standard CFG scale: A natural question is whether SyncCFG is fundamentally different from simply using a higher standard CFG scale. The answer is yes: standard CFG with a higher scale would amplify the text-conditioning signal more aggressively, making both modalities more text-adherent but potentially worse at synchronization (because text adherence and synchronization are orthogonal, and over-emphasizing text adherence could drown out the cross-modal signal). SyncCFG instead amplifies a different vector — the one that specifically captures cross-modal coupling — without touching the text-conditioning signal. The two forms of guidance are complementary and could in principle be combined (though the paper doesn't explore this).

Training Curriculum and Hyperparameters (Appendix B)

Three-stage training protocol. The full Harmony model is trained in three sequential stages, each with a specific purpose:

Stage 1: Foundational Audio Pre-training. Duration: 100,000 iterations. Batch size: 1,536 (global). Data: balanced 1:1 mixture of human speech and environmental sound datasets, clips with maximum duration of 10 seconds. Reference audio: randomly selected 1-3 second segment from the ground-truth clip. Learning rate: $1 \times 10^{-5}$ (constant). Purpose: The audio model learns to replicate both the timbre and content from the provided reference audio. This stage ensures the audio branch has strong generative capabilities before it needs to learn cross-modal alignment.

Stage 2: Timbre Disentanglement Finetuning. Duration: 20,000 additional iterations. Data processing modification: For human speech, the reference audio and target audio are different utterances from the same speaker (cross-utterance). For environmental sounds, the reference audio is a non-overlapping segment from the same long recording as the target. Purpose: By providing mismatched reference and target content, the model is forced to disentangle the speaker's invariant acoustic signature (timbre) from the specific phonetic content. This enables voice cloning — the model learns that the reference audio provides who is speaking, while the transcript provides what they're saying.

Stage 3: Cross-Task Audio-Visual Training. Duration: 10,000 iterations. Batch size: 128. Data: again a 1:1 mixture of human speech and environmental sound. Learning rate: $1 \times 10^{-5}$ (constant for all parameters). Loss weights: $\lambda_v = 0.1$ (audio-driven video loss), $\lambda_a = 0.3$ (video-driven audio loss). Objective: Flow Matching with shift of 5. Purpose: The full Cross-Task Synergy training that integrates the audio and video branches and learns cross-modal alignment.

Inference hyperparameters (also from Appendix B). Integration steps: 40 (the number of denoising steps at inference time). CFG scales: $s_v = 3$ for video, $s_a = 2$ for audio. Sampler shift parameter: 5 (matching the training shift). Flow Matching objective: maintained from training.

Why a three-stage curriculum rather than end-to-end training: The progressive curriculum addresses the cold-start problem in multimodal learning. If the audio branch were randomly initialized and immediately trained with cross-modal losses, it would produce poor-quality audio latents that provide uninformative conditioning signals to the video branch, and the cross-modal interaction modules would learn from noise. By first training the audio branch to generate high-quality audio independently (Stage 1), then teaching it voice cloning (Stage 2), the audio latents entering Stage 3 are already meaningful, providing a strong foundation for cross-modal alignment learning. The video branch benefits from Wan2.2-5B pre-training analogously — it starts Stage 3 with strong visual generation capabilities. This means the cross-modal interaction modules in Stage 3 learn to align already-competent modality representations rather than having to simultaneously learn generation and alignment from scratch.

Flow Matching objective. The paper specifies training uses a Flow Matching objective with a shift of 5. Flow Matching is an alternative to score-based diffusion that models a continuous transformation between the noise distribution and the data distribution. The "shift" parameter controls the noise schedule — higher shift values put more emphasis on later (cleaner) timesteps. The shift of 5 (maintained at both training and inference) is a specific design choice that likely prioritizes learning fine details in the later denoising stages, which is consistent with the paper's emphasis on fine-grained synchronization.

Summary of Design Choices and Their Justifications

  • Cross-Task Synergy with asymmetric loss weights ($\lambda_v = 0.1$, $\lambda_a = 0.3$) over purely joint training: provides stable, deterministic cross-modal supervision that counteracts Correspondence Drift, with higher weight on video→audio supervision reflecting its greater difficulty.
  • Separate speech and text encoders over a single unified encoder: preserves phonetic precision (via dedicated speech encoder) and semantic understanding (via T5) without forcing a trade-off between them.
  • Reference audio prepending over separate cross-attention or addition: provides persistent timbre conditioning throughout the transformer depth via self-attention without consuming cross-modal interaction capacity.
  • RoPE rescaling for temporal alignment over latent interpolation: achieves sub-frame temporal correspondence in a continuous positional space without modifying or blurring the latent representations themselves.
  • Local windowed attention for temporal synchronization over global attention: imposes a strong locality inductive bias that dramatically reduces the search space for learning temporal correspondence and prevents spurious long-range dependencies.
  • Reference-latent-based global style injection over direct audio latent modulation: keeps global style and local temporal alignment in separate representational pathways, preventing interference between them.
  • SyncCFG with cross-modal negative anchors over standard CFG or higher CFG scales: isolates and amplifies specifically the cross-modal coupling vector rather than the text-conditioning vector, targeting synchronization directly rather than hoping it emerges from stronger text adherence.
  • Three-stage training curriculum over end-to-end training: ensures both modality branches have strong independent generation capabilities before cross-modal alignment learning begins, preventing the interaction modules from learning from poor-quality latent representations.

4. Key Insights and Innovations

Innovation 1: Correspondence Drift as a Diagnosed Failure Mode of Joint Diffusion

The field has largely treated joint audio-video generation as an architectural scaling problem — the implicit assumption being that with enough data, enough parameters, and enough compute, cross-modal synchronization would emerge naturally from the standard joint denoising objective. Prior work on models like MM-Diffusion [42], JavisDiT [34], UniVerse-1 [50], and Ovi [35] all operated under variants of this assumption, designing ever-larger dual-stream architectures with cross-attention at various depths, while retaining essentially the same training paradigm: jointly denoise both modalities from noise and hope the interaction modules learn alignment.

This paper makes a genuinely novel diagnostic move by identifying that the training objective itself contains an inherent instability that prevents robust alignment learning, independent of model capacity. The concept of Correspondence Drift — the idea that attempting to learn a mapping between two concurrently evolving, highly stochastic latent variables creates a fundamentally unstable learning target because the optimal correspondence shifts as both representations change — is not an architectural critique but a training-theoretic one. It explains why all prior architectures, regardless of their sophistication, converge slowly and plateau at suboptimal synchronization: the gradient signal for cross-modal alignment is corrupted at the source, not in the architecture that processes it.

What makes this intellectually distinctive is that it reframes the synchronization problem from "we need better interaction mechanisms" (the dominant prior approach) to "we need better supervision for the interaction mechanisms we already have." The empirical evidence in Figure 3 is elegantly constructed to isolate this effect: by comparing an audio-driven model (deterministic, clean audio conditioning noisy video) against a joint model (both noisy) using an identical architecture, the authors demonstrate that architectural capacity is not the bottleneck — the nature of the conditioning signal is. The audio-driven model converges rapidly to high alignment (Sync Confidence rising from ~1.5 to ~5.0 over 5,000 iterations), while the joint model lags substantially (~2.5–3.0), despite both having access to the same interaction modules. This is a clean ablation that separates training dynamics from architectural expressivity.

This diagnostic contribution is fundamental rather than incremental because it identifies a failure mode that was previously invisible to the field. Prior work attributed poor synchronization to insufficient architectural sophistication or insufficient data; Harmony argues that even with perfect architecture and infinite data, the standard joint training objective would still struggle because the learning target itself drifts. If correct, this diagnosis has implications beyond audio-video generation — any joint diffusion process attempting to learn fine-grained cross-modal correspondences (video-text, audio-text, multi-sensor) would suffer from the same instability if both modalities are simultaneously denoised.

Innovation 2: Cross-Task Synergy as a Training-Theoretic Intervention (Not an Architectural One)

The natural response to Correspondence Drift would be architectural — design more sophisticated interaction modules, add auxiliary losses, or increase model capacity. Harmony's Cross-Task Synergy training paradigm instead makes a methodological intervention at the level of the training objective by augmenting the joint loss with auxiliary losses from uni-directional generation tasks where one modality's latent is clean (timestep zero) and deterministic.

This is fundamentally distinct from multi-task learning as commonly practiced. Standard multi-task learning adds auxiliary objectives (e.g., "also predict the audio from video") to improve representation learning, but typically all tasks operate in the same noisy regime. Cross-Task Synergy is specifically designed to provide stable cross-modal supervision — the auxiliary tasks create a learning regime where the interaction modules can learn "given this specific audio pattern, the video should look exactly like this" without the added stochasticity of a simultaneously noisy partner. The joint loss term then ensures the model can generalize this learned correspondence to the regime where both modalities are noisy at inference time.

What makes this innovative is not the idea of auxiliary tasks per se, but the targeted use of clean-latent conditioning as a training scaffold to solve an identified training instability. The authors are not simply adding more losses to regularize or improve representations; they are deliberately engineering a training signal that counteracts a specific diagnosed failure mode. The asymmetric loss weights (λv = 0.1 for audio-driven video, λa = 0.3 for video-driven audio) further suggest that the intervention is calibrated — video-to-audio supervision receives triple the weight, possibly because learning to generate temporally precise audio (which must track sub-frame-level visual events like impacts and lip movements) from clean video is harder than the reverse and benefits more from explicit training.

This is a conceptual advance because it opens a design space that prior work hadn't explored: using uni-directional generation tasks not as standalone capabilities to be optimized, but as training scaffolds whose primary purpose is to stabilize and accelerate the learning of a different (joint) task. The fact that the uni-directional tasks also become deployable capabilities (audio-driven video generation, video-driven audio generation) is presented as a beneficial side effect, not the primary motivation. This inversion of priorities — training for joint generation, using uni-directional tasks as scaffolding — represents a shift in thinking about how to structure multi-modal training.

Innovation 3: Architectural Decoupling of Global Style and Local Temporal Alignment

Prior work on audio-video interaction — from early models like MM-Diffusion to recent ones like Ovi — used a single, monolithic cross-attention mechanism between the audio and video streams. The implicit assumption was that a sufficiently expressive attention module could simultaneously learn to align fine-grained temporal correspondences (lip-to-phoneme mapping, event-to-sound onset timing) and propagate global stylistic attributes (emotional tone, ambient acoustic character, overall scene mood).

Harmony identifies this as a fundamental architectural conflation — two objectives with qualitatively different computational requirements are forced to share a single mechanism, creating an unavoidable trade-off where neither is achieved optimally. Fine-grained temporal alignment benefits from locality (attending to temporally nearby frames, high resolution in the time dimension) and symmetry (bidirectional, frame-level correspondence). Global style propagation benefits from globality (attending across the entire sequence) and compression (aggregating holistic features into a compact representation). A single attention module cannot simultaneously be local and global, high-resolution and compressive.

The innovation is not the specific modules (local windowed attention and reference-latent cross-attention are individually well-known techniques) but rather the decomposition insight — recognizing that these two objectives are in tension and designing a principled separation that allocates each to a specialized component. The RoPE-Aligned Frame-wise Attention (Section 3.3.1) handles only temporal synchronization, using local windows and continuous positional encoding to achieve sub-frame alignment without interference from global style considerations. The Global Style Alignment module (Section 3.3.2) handles only holistic consistency, using the reference audio latent as a compact carrier for style information that is modulated by global video context without touching the fine-grained temporal structure of the target audio.

The decoupling is clean in a way that prior hybrid approaches (which might, for instance, combine local and global attention in a single module) are not: the two components operate on different latent representations (frame-wise attention modifies zv and za directly; global style modifies only zr) and their outputs are integrated through the architecture's natural residual connections and prepending mechanism rather than through learned gating or blending. This prevents the components from competing for representational capacity and allows each to specialize fully.

This is a fundamental architectural insight with implications beyond audio-video generation. Any multi-modal interaction that requires both fine-grained temporal alignment and holistic style propagation — video-text alignment in long-form generation, multi-sensor fusion in robotics, cross-modal retrieval — faces the same tension. The paper's solution (separate pathways for local temporal and global stylistic interaction, operating on different representational substrates) provides a template for addressing this tension in other domains.

Innovation 4: SyncCFG as a Mechanism for Isolating and Amplifying Cross-Modal Coupling

Classifier-Free Guidance is a standard inference-time technique across diffusion models, uniformly applied as a way to amplify conditioning adherence. In the audio-video domain, prior work like Ovi [35] used standard CFG (Equation 8) to strengthen how well each modality separately matched its text prompt. The guidance vector in standard CFG is computed by contrasting a text-conditioned prediction with an unconditional (null-text) prediction, isolating the effect of text conditioning.

Harmony's key insight is that this guidance vector is orthogonal to synchronization — it amplifies text adherence for each modality independently, but provides no signal about cross-modal consistency. A generation can have perfect text adherence for both audio and video while being completely desynchronized, and standard CFG would happily amplify it toward that desynchronized state. The innovation of SyncCFG is that it redesigns the negative anchor to isolate a fundamentally different signal: not "what changes when we add text conditioning" but "what changes when we add cross-modal coupling."

The construction is conceptually elegant: for video guidance, the negative anchor uses a "mute audio" latent (silence) so the subtraction isolates visual dynamics caused specifically by sound — mouth movements, object vibrations, event-triggered motions. For audio guidance, the negative anchor uses a "static video" latent so the subtraction isolates acoustic events caused specifically by visual motion — impact sounds, footsteps, motion-triggered audio. In both cases, features that are independent of cross-modal coupling (static background rendering, ambient background noise that would exist regardless of motion) appear in both predictions and cancel out, leaving a pure cross-modal signal.

This represents a conceptual shift in what CFG means: from a generic conditional amplifier to a targeted mechanism for enhancing specific types of conditioning. The standard CFG formulation assumes there is one conditioning signal (text) to be amplified; SyncCFG recognizes that in multi-modal generation, there are multiple conditioning signals (intra-modal text adherence, cross-modal synchronization) and that these should be amplified through different guidance vectors computed from different negative anchors.

What makes this more than an incremental trick is its dependence on and completion of the Cross-Task Synergy training. The model can only compute SyncCFG because it was trained to perform audio-driven and video-driven generation during the Cross-Task Synergy stage — without those capabilities, there would be no way to produce the negative anchor predictions (specifically, the "mute audio" and "static video" conditioned predictions). This creates a tight coupling between the training innovation and the inference innovation: Cross-Task Synergy enables SyncCFG, and SyncCFG exploits Cross-Task Synergy. The two are not independent contributions but two halves of a single methodological thesis — that training a model to generate in multiple cross-modal regimes creates capabilities that can be recomposed at inference time to isolate and amplify specific aspects of the generation.

The ablation study (Table 3) provides compelling evidence for this coupling: adding SyncCFG on top of a model already trained with Cross-Task Synergy produces the single largest improvement (Sync-C jumps from 5.09 to 6.51, Sync-D drops from 10.16 to 8.63) — larger than the gains from the Global-Local Decoupled Interaction module or from Cross-Task Synergy training alone. This suggests that Cross-Task Synergy training creates latent capabilities (the ability to generate in audio-driven and video-driven modes) that are not fully exploited until SyncCFG extracts them during inference.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The paper introduces Harmony-Bench, a new benchmark of 150 test cases designed to evaluate joint audio-video generation across three progressively difficult subsets of 50 items each: (1) Ambient Sound-Video — AI-generated scenarios conditioned on separate audio and video captions to test non-speech temporal alignment; (2) Speech-Video — a mix of 25 real-world and 25 AI-synthesized samples in English and Chinese, conditioned primarily on a transcript, to assess lip-sync and speech quality; (3) Complex Scene (Ambient + Speech) — co-occurring speech and ambient sounds requiring the model to process transcripts, ambient sound descriptions, and video captions simultaneously. The benchmark is motivated by the inadequacy of existing benchmarks: JavisBench [34] lacks human speech evaluation, and Verse-Bench [50] has low-quality labels and limited synchronization coverage (Appendix C.1). The training corpus for the model spans over 4 million audio-visual clips from public sources (OpenHumanVid [31], AudioCaps [28], WavCaps [36], Emilia [15]) supplemented with 2 million curated clips for human speech and 2 million for environmental sounds, all annotated using Gemini [12] (Appendix B).

  • Base model. The video branch is initialized from the pre-trained Wan2.2-5B model [49], a state-of-the-art open-source video diffusion transformer. The audio branch is a symmetric transformer trained from scratch through a three-stage curriculum (Appendix B). The paper argues this model is representative of current open-source video generation capabilities and that the video branch's strong visual priors from Wan2.2 pre-training provide a solid foundation for cross-modal alignment learning (Stage 3 of training).

  • Metrics. The paper employs a comprehensive three-category metric suite (Appendix C.2). Video Quality and Coherence: Aesthetic Quality (AQ, via aesthetic-predictor-v2-5 [48]), Imaging Quality (IQ, via MUSIQ [27]), Dynamic Degree (DD, via RAFT [46]), Motion Smoothness (MS, via a pre-trained video frame interpolation model), and Identity Consistency (ID, via DINOv3 [44] feature similarity between reference and generated frames). Audio Fidelity and Quality: AudioBox-Aesthetics [47] across four dimensions — Production Quality (PQ), Production Complexity (PC, lower is better), Content Enjoyment (CE), and Content Usefulness (CU); Word Error Rate (WER, via Whisper-large-v3 [40]) for speech intelligibility; and IB-A Score [10] for semantic audio-text alignment. Audio-Visual Synchronization: Sync-C and Sync-D [7] for lip-sync accuracy; DeSync Score (predicted by Synchformer [25]) for temporal misalignment in seconds; and ImageBind (IB) Score [10] for overall audio-visual consistency via cosine similarity of feature embeddings.

  • Baselines. Five prior methods are compared: MM-Diffusion [42] (unconditional generation, evaluated only on ambient sound-video since it cannot generate speech), JavisDiT [34] (hierarchical spatio-temporal prior synchronization), UniVerse-1 [50] (unified generation via stitching of experts), Ovi [35] (twin backbone cross-modal fusion), and indirectly JAM-Flow [30] and AnimateSI [52] (referenced in Table 1 for capability comparison but not quantitatively evaluated). MM-Diffusion lacks several metrics since it produces unconditional outputs that cannot be evaluated for speech-specific synchronization.

  • Generation budget / compute accounting. The paper uses a Flow Matching objective with 40 integration steps at inference time. Guidance scales are s_v = 3 for video and s_a = 2 for audio SyncCFG (Appendix B). The sampler's shift parameter is 5, matching the training configuration. Compute is not measured in FLOPs or wall-clock time; the fair comparison is implicit in using the same number of inference steps and comparable model scales across baselines. The training budget is specified in iterations: Stage 1 audio pre-training for 100,000 iterations (batch size 1,536), Stage 2 timbre disentanglement for 20,000 iterations, and Stage 3 cross-task training for 10,000 iterations (batch size 128). All stages use a constant learning rate of 1 × 10^{-5}.

  • Cross-validation / statistical protocol. The paper does not report cross-validation or statistical significance testing. The 150-case Harmony-Bench test set is evaluated once per model. The ablation study (Table 3) is evaluated on a human-speech-only subset, separate from the full benchmark evaluation, and no train/validation/test splits within Harmony-Bench are described.

Main Quantitative Results

Aggregate Comparison Across All Three Subsets (Table 2)

Table 2 reports the average performance across all three Harmony-Bench subsets (ambient sound-video, speech-video, and complex scene). Harmony achieves state-of-the-art or competitive results across all metric categories, with its primary advantage concentrated in audio-visual synchronization:

  • Video quality: Harmony achieves the best AQ (0.59 vs. Ovi's 0.57), tied-best MS (0.99, with most methods at 0.99), best DD (0.36 vs. Ovi's 0.34), and best ID (0.91 vs. Ovi's 0.90). IQ is tied with Ovi at 0.65. The margins in visual quality are modest but consistent.
  • Audio fidelity: Harmony achieves best PQ (6.39 vs. Ovi's 6.19), best PC (2.05 vs. Ovi's 2.13, lower is better), best CE (4.73 vs. Ovi's 4.44), and competitive CU (5.67 vs. Ovi's 5.84). Audio quality is strong but not dramatically ahead of Ovi on all sub-metrics.
  • Audio-visual synchronization: This is where Harmony separates decisively. Sync-C reaches 5.61 (vs. Ovi's 4.04 and UniVerse-1's 0.97). Sync-D drops to 7.53 (vs. Ovi's 9.62 and UniVerse-1's 10.71). DeSync drops to 0.92 (vs. Ovi's 1.14 and JavisDiT's 1.13). IB score reaches 0.19 (tied with JavisDiT at 0.18, ahead of Ovi at 0.18 and UniVerse-1 at 0.12). The Sync-C improvement of 1.57 over Ovi and Sync-D reduction of 2.09 over Ovi are the largest inter-method gaps in the table, validating the paper's central claim that Harmony significantly advances synchronization.

These aggregate numbers mask important subset-level variation, which the detailed per-subset tables (Tables 4–6 in the appendix) unpack.

Speech-Video Generation (Table 4, Appendix D)

On the 50-item speech-video subset, Harmony's synchronization advantage is most pronounced:

  • Sync-C: Harmony achieves 6.51 vs. Ovi's 5.13, UniVerse-1's 1.22, and JavisDiT's 1.20. The 6.51 Sync-C represents a 27% improvement over Ovi and a more than 5× improvement over UniVerse-1.
  • Sync-D: Harmony achieves 8.63 vs. Ovi's 10.38, UniVerse-1's 13.10, and JavisDiT's 12.73 — reductions of 1.75, 4.47, and 4.10 respectively.
  • WER: Harmony achieves the lowest Word Error Rate at 0.15, compared to Ovi's 0.19, UniVerse-1's 0.23, and JavisDiT's 1.00. This is speech intelligibility essentially at ceiling.
  • IB: Harmony achieves 0.18, slightly ahead of Ovi's 0.17 and UniVerse-1's 0.16, and behind JavisDiT's 0.22.
  • Video dynamism: Harmony achieves DD of 0.20, higher than UniVerse-1's 0.15 and Ovi's 0.17, indicating noticeably more natural motion — the paper notes that competing methods often achieve good ID scores (UniVerse-1 at 0.89, Ovi at 0.88) by generating near-static videos where frame-to-frame identity is trivially high. Harmony's ID of 0.93 is the best despite its higher dynamism.

Chinese Speech Cross-Lingual Evaluation (Table 7, Appendix D)

A dedicated Chinese speech test reveals Harmony's cross-lingual generalization advantage:

  • WER: Harmony achieves 0.92 vs. Ovi's 9.10 (nearly 10× higher), UniVerse-1's 2.32, and JavisDiT's 4.84. The paper notes that standard WER is not perfectly optimized for Chinese tokenization, making relative performance the meaningful indicator.
  • Sync-C: Harmony achieves 5.05 vs. Ovi's 4.45, JavisDiT's 1.27, and UniVerse-1's 0.91.
  • Sync-D: Harmony achieves 9.38 vs. Ovi's 10.79 and UniVerse-1's 11.02.
  • IB: Harmony at 0.22, tied with UniVerse-1 and slightly ahead of Ovi and JavisDiT at 0.20.

The pronounced gap in Chinese WER (0.92 vs. 9.10 for Ovi) suggests that Harmony's dedicated speech encoder and separate phonetic conditioning pathway (versus the single-encoder approach of prior work) generalize substantially better to non-English phonetics.

Environmental Sound-Video Generation (Table 5, Appendix D)

On the 50-item ambient sound subset (which has no speech, so WER and lip-sync metrics are not applicable):

  • Video dynamism: Harmony achieves DD of 0.56, significantly higher than Ovi's 0.44, UniVerse-1's 0.16, and JavisDiT's 0.33. This confirms the qualitative observation that competing methods generate static or minimally dynamic videos for environmental scenes.
  • Audio fidelity: Harmony leads on PQ (6.53 vs. Ovi's 6.45), CE (4.12 vs. Ovi's 3.78), and CU (6.22 vs. Ovi's 5.98), though trails on PC (2.68 vs. Ovi's 2.46, higher is worse).
  • Synchronization: DeSync drops to 0.70 for Harmony, compared to Ovi's 1.06, UniVerse-1's 1.10, and JavisDiT's 0.94. IB score at 0.21 edges out Ovi's 0.20.
  • Visual quality: AQ of 0.64 vs. Ovi's 0.62, though ID drops to 0.90 (vs. Ovi's 0.93 and UniVerse-1's 0.92) — consistent with Harmony generating more dynamic motion that reduces trivial frame-to-frame identity consistency.

Complex Scene Generation (Table 6, Appendix D)

On the most challenging 50-item subset combining speech and ambient sounds:

  • Synchronization: Sync-C achieves 4.70 (vs. Ovi's 2.94, UniVerse-1's 0.72). Sync-D drops to 6.43 (vs. Ovi's 8.86 and UniVerse-1's 8.32). The 1.76 Sync-C improvement and 2.43 Sync-D reduction demonstrate that Harmony's synchronization advantage persists even when the generation task requires simultaneous speech and ambient sound synthesis.
  • WER: Harmony achieves 0.15 vs. Ovi's 0.79 and UniVerse-1's 0.25, showing that speech intelligibility is maintained even in acoustically complex mixed scenes.
  • Audio quality: Harmony achieves best PQ (6.43 vs. Ovi's 5.94) and CE (4.76 vs. Ovi's 4.14), though CU drops to 4.86 (vs. Ovi's 5.33).
  • Video dynamism: DD of 0.32 is lower than Ovi's 0.41, a reversal from the other subsets. The paper doesn't comment on this specifically.

The complex scene results are particularly significant because they demonstrate that Harmony's synchronization improvements are not achieved by simply ignoring competing acoustic events — the model simultaneously maintains speech intelligibility (WER 0.15), ambient sound fidelity (PQ 6.43), and cross-modal alignment (Sync-C 4.70) in the most demanding generation scenario.

Qualitative Results (Figures 5, 6, and Appendix Figures 7–10)

Figure 5 provides side-by-side comparisons on two challenging cases:

  • Talking head (left panel): A man speaking the phrase "The agreement covers anything purchased. The rule is strictly enforced. That contract is active." Competing methods (UniVerse-1, Ovi) produce still or minimally-moving faces where lip movements do not track the speech. Harmony generates fluid, naturalistic facial motion with lip movements visibly synchronized to the utterance.
  • Music performance (right panel): A folk musician playing mandolin. UniVerse-1 generates irrelevant noise (visually and acoustically disconnected). Ovi produces music that is "musically correct, but less dynamic" — the waveform visualization shows simpler structure. Harmony generates a fluid video of a person playing with "motions that are dynamically synchronized with the rich, corresponding music, as evidenced by the more complex audio waveform."

Figure 6 visualizes the audio-to-video frame-wise cross-attention maps:

  • For human speech, attention maps show the model "precisely localizes its attention on the speaker's oral region." Critically, in multi-person scenes, the model "can distinguish between them, focusing exclusively on the active speaker."
  • For environmental sounds, attention localizes to the primary sound source (an animal) while also attending to ambient environmental sounds (rain in the cat example, birdsong in the crocodile case). This demonstrates that the decoupled interaction module achieves fine-grained, contextually-aware alignment that distinguishes sound sources.

Appendix Figure 7 (more human speech comparisons) shows that Harmony generates fluid, naturalistic motion with clear, synchronized audio, while Ovi and UniVerse-1 tend toward static "talking head" effects.

Appendix Figure 8 (environmental sound comparisons) shows that JavisDiT produces low-quality video with unstable audio (in the "gunfire" example, the audio waveform is highly irregular). UniVerse-1 and Ovi generate static or partially static scenes — in the "ocean waves" case, "the main waves remain frozen while only the water surface shows minimal movement." Harmony generates dynamic video with realistic motion and high-fidelity audio precisely synchronized to visual events.

Appendix Figure 10 (audio-driven generation) demonstrates that Harmony's Cross-Task Synergy training enables native audio-driven video generation as a deployment capability — video generated solely from a target speech audio clip exhibits high visual quality, natural facial expressions, coherent head movements, and precisely synchronized lip movements.

Ablation Studies and Robustness Checks

Progressive component ablation (Table 3): The paper reports a systematic ablation on the human-speech dataset, starting from a baseline model that replaces the Global-Local Decoupled Interaction (GLDI) module with a standard global cross-attention mechanism (similar to Ovi [35]) and is trained without Cross-Task Synergy (CTS). The baseline achieves Sync-C of 4.20, Sync-D of 10.93, and IB of 0.13.

  • Adding GLDI (decoupling local and global interactions but without RoPE alignment): Sync-C rises to 4.29 (+0.09), Sync-D drops to 10.67 (−0.26), IB rises to 0.14 (+0.01). The gain is modest, suggesting that mere decoupling without proper temporal alignment provides limited benefit.
  • Adding RoPE Alignment to GLDI: Sync-C jumps to 4.80 (+0.51 over GLDI-only), Sync-D drops to 10.30 (−0.37), IB at 0.14. The substantial Sync-C improvement confirms that resolving the audio-video timescale mismatch via RoPE rescaling is critical for fine-grained synchronization — decoupling alone is insufficient without proper temporal alignment.
  • Adding Cross-Task Synergy (CTS) training: Sync-C rises to 5.09 (+0.29), Sync-D drops to 10.16 (−0.14), IB rises to 0.15 (+0.01). The additional improvement demonstrates that CTS provides complementary benefits beyond architectural improvements, consistent with the paper's claim that Correspondence Drift is a training-dynamic problem, not just an architectural one.
  • Adding Synchronization-Enhanced CFG (SyncCFG) at inference: Sync-C catapults to 6.51 (+1.42), Sync-D drops to 8.63 (−1.53), IB rises to 0.18 (+0.03). This is the single largest improvement in the ablation, validating that SyncCFG extracts latent cross-modal capabilities that CTS training builds but doesn't fully exploit.

The trajectory from 4.20 to 6.51 Sync-C (a 55% improvement) and from 10.93 to 8.63 Sync-D (a 21% improvement) provides evidence that each component addresses a distinct bottleneck — the improvements are largely additive rather than overlapping, suggesting the three identified root causes (Correspondence Drift, architectural conflation, CFG's intra-modal bias) are indeed independent problems.

Cross-Task Synergy convergence analysis (Figure 3): The paper compares training dynamics across three configurations: (1) pure audio-driven generation (clean audio conditioning noisy video), (2) standard joint generation (both modalities jointly denoised), and (3) Cross-Task Synergy (joint + auxiliary uni-directional losses). The audio-driven model converges rapidly to Sync Confidence of approximately 4.5–5.0 by 5,000 iterations, while joint generation only reaches approximately 2.5–3.0. The Cross-Task Synergy model tracks the audio-driven baseline much more closely, reaching approximately 4.0–4.5 and substantially closing the gap to the oracle (audio-driven) convergence trajectory. Sync Distance shows the inverse pattern: audio-driven drops to approximately 10.5, joint plateaus at approximately 12.5, and Cross-Task Synergy approaches approximately 10.5. This validates the core hypothesis that the stable, uni-directional supervision from clean-latent conditioning transfers alignment knowledge to the joint generation pathway.

Voice cloning capability (Appendix F, Figure 9): The paper demonstrates that the reference audio prepending mechanism enables voice cloning — generating new speech content in a target speaker's voice from a 1–3 second reference clip — without degrading visual quality or lip-sync. Figure 9 shows examples across multiple distinct voice timbres. This is not an ablation per se but a capability validation: the three-stage training curriculum (Stage 1 audio pre-training, Stage 2 timbre disentanglement, Stage 3 cross-task training) successfully produces a model that disentangles speaker identity from phonetic content.

Diverse visual styles (Appendix H, Figure 12): Harmony generates speech-video across distinct artistic styles (Disney-style animation, traditional ink wash painting), maintaining high visual quality, sharp details, and temporally coherent motion consistent with the target aesthetic. This validates that the interaction modules and conditioning pipeline do not overfit to photorealistic styles and that the Wan2.2 initialization provides sufficient visual priors for style generalization.

Diverse ambient sounds (Appendix H, Figure 13): Harmony generates a wide spectrum of ambient sounds beyond simple environmental noise — "from the sharp, percussive bursts of fireworks to the structured harmonies of music" — with each sound rendered at high fidelity and synchronized to its visual source. This demonstrates that the model generalizes across acoustic event types rather than specializing in narrow sound categories.

What is NOT ablated (notable gaps):

  • No ablation of the separate speech and text encoders: The paper claims this is a crucial design choice (Section 3.2.2) but provides no experiment comparing separate vs. unified encoding. Without this ablation, it's unclear whether the phonetic precision benefit the paper argues for is real or whether a single sufficiently large encoder could achieve similar performance.
  • No ablation of loss weights (λ_v = 0.1, λ_a = 0.3): The asymmetric weighting is a specific design choice that the paper doesn't experimentally justify. Without sweeps over these hyperparameters, it's unknown whether the asymmetry is important or whether any reasonable weighting would work similarly.
  • No ablation of the three-stage training curriculum: The full three-stage protocol is evaluated, but there's no comparison to end-to-end joint training from scratch or to a two-stage variant (skipping timbre disentanglement). The claimed benefit of progressive training is not empirically validated.
  • No SyncCFG scale sweeps: The guidance scales (s_v = 3, s_a = 2) are specified as fixed values with no reported sweep or sensitivity analysis. Given that SyncCFG provides the single largest improvement in Table 3, understanding how performance varies with guidance scale would be valuable for practitioners.
  • No ablation of frame-wise attention window size: The local window size for the RoPE-Aligned Frame-wise Attention is not specified or ablated, making it unclear whether the benefit comes from locality per se or from some specific window configuration.
  • No ablation on the number of integration steps: The paper uses 40 steps at inference but doesn't explore whether the synchronization benefit is robust to fewer steps (important for latency-sensitive applications).

Critical Assessment

Claim 1: "Harmony establishes a new state-of-the-art, significantly outperforming existing methods in both generation fidelity and, critically, in achieving fine-grained audio-visual synchronization."

Assessment: Strongly supported for synchronization; conditional for generation fidelity.

The synchronization claim is well-supported. On the speech-video subset (Table 4), Sync-C of 6.51 represents a 27% improvement over Ovi (5.13) and a more than 5× improvement over UniVerse-1 (1.22). Sync-D of 8.63 vs. Ovi's 10.38 is a 17% reduction. On complex scenes (Table 6), Sync-C of 4.70 vs. Ovi's 2.94 (60% improvement) and Sync-D of 6.43 vs. Ovi's 8.86 (27% reduction) demonstrate that the advantage persists in the most challenging setting. The Chinese cross-lingual results (Table 7) further extend this: WER of 0.92 vs. Ovi's 9.10 is an order-of-magnitude improvement. These gaps are large enough to be practically meaningful even without reported confidence intervals.

However, the "generation fidelity" claim requires qualifications. In video quality (Table 2 aggregate), AQ improves from 0.57 (Ovi) to 0.59 — a 3.5% relative improvement. IQ is tied at 0.65. ID improves from 0.90 to 0.91. These are marginal gains, not "significant outperformance." In audio fidelity, PQ improves from 6.19 to 6.39 (3.2%), and CE from 4.44 to 4.73 (6.5%). On the environmental sound subset (Table 5), PC actually degrades (2.68 vs. Ovi's 2.46, higher is worse). The paper's claim that Harmony significantly outperforms on generation fidelity overstates the case; the model is competitive with Ovi on quality metrics, but the primary and decisive advantage is limited to synchronization.

The limitation to five baseline comparisons on a 150-case benchmark also warrants caution. The field is young and baselines may not be fully optimized. Ovi and UniVerse-1 are the only recent generalist models evaluated; JavisDiT and MM-Diffusion represent earlier, more limited approaches. A comparison against proprietary systems (Veo 3, Sora 2) would be more informative about the absolute state of the art but is precluded by their closed-source nature.

Claim 2: "Cross-Task Synergy training instills robust alignment priors to counteract [Correspondence] drift."

Assessment: Supported with strong evidence from Figure 3 and Table 3, but with an important caveat.

Figure 3 provides direct empirical evidence for the Correspondence Drift hypothesis and for Cross-Task Synergy's mitigation of it. The audio-driven baseline (clean conditioning) converges rapidly while joint training stagnates — this isolates the effect of conditioning signal stability on alignment learning. The Cross-Task Synergy model substantially closes the gap, tracking the audio-driven convergence trajectory. This is a clean experiment that controls for architecture (all configurations use the same network).

Table 3 further shows that adding CTS training to an already-strong architecture (GLDI + RoPE) yields additional gains: Sync-C from 4.80 to 5.09, Sync-D from 10.30 to 10.16. While these gains are smaller than those from SyncCFG (+1.42 Sync-C), they demonstrate that CTS contributes beyond what architecture alone achieves.

The caveat is that the paper never demonstrates that Correspondence Drift as a phenomenon (the shifting optimal mapping due to concurrent noise evolution) is the specific mechanism causing slow convergence, rather than other correlates of joint training (e.g., the joint loss simply being a harder optimization problem with more local minima). Figure 3 shows that joint training converges slowly and that CTS helps — this is consistent with the drift hypothesis but doesn't uniquely confirm it. An experiment that varied the noise correlation between modalities (e.g., independent vs. shared noise schedules) or that measured the stability of the learned cross-modal mapping over training iterations would provide more direct evidence for drift specifically.

Claim 3: "A Global-Local Decoupled Module resolves the architectural conflict [between global style and local temporal alignment] by handling style and timing separately."

Assessment: Supported but the evidence is indirect.

The ablation in Table 3 shows that adding GLDI (decoupling local and global interactions) yields Sync-C improvement from 4.20 to 4.29 — a small gain. Adding RoPE alignment on top of GLDI yields the larger jump to 4.80. This suggests that the decoupling itself provides limited benefit without proper temporal alignment, and that the primary value of the module may come from the RoPE rescaling (which resolves timescale mismatch) rather than from the separation of global and local pathways.

The paper does not provide an ablation that isolates the contribution of the global style alignment module from the frame-wise attention module — for example, comparing the full GLDI against a version with only frame-wise attention (no global style module) would reveal whether the decoupling specifically helps or whether a local-attention-only design would perform similarly. The qualitative attention visualizations in Figure 6 demonstrate that the frame-wise attention localizes to sound sources, but they don't demonstrate that the global style module provides benefits beyond what frame-wise attention alone would achieve.

The architectural decoupling argument is conceptually compelling, but the experimental evidence for it is weaker than for the other two contributions (CTS and SyncCFG).

Claim 4: "SyncCFG provides an explicit mechanism to amplify the alignment signal during inference."

Assessment: Strongly supported by the ablation study.

Adding SyncCFG to a model already trained with CTS, GLDI, and RoPE produces the largest single improvement in Table 3: Sync-C jumps from 5.09 to 6.51 (+1.42), Sync-D drops from 10.16 to 8.63 (−1.53). This 28% Sync-C improvement from a single inference-time technique is substantial and validates that the negative anchor construction successfully isolates and amplifies cross-modal coupling.

However, the paper does not ablate the specific negative anchor design against plausible alternatives. For instance, would a simpler approach — using a higher standard CFG scale or computing the guidance vector by contrasting joint generation with an independently generated (rather than driven) negative — achieve similar benefits? The SyncCFG formulation requires the model to have been trained with Cross-Task Synergy (to enable the driven predictions), so the coupling between CTS training and SyncCFG inference is tight — but whether this specific coupling is necessary or whether other recouplings would work equally well is not tested.

Additionally, the SyncCFG scales (s_v = 3, s_a = 2) are presented as fixed values. Without scale sweeps, it's impossible to know whether SyncCFG benefits are robust to scale variation or whether the reported numbers represent cherry-picked optimal values. Given that CFG scale sensitivity is a known issue in diffusion models, this is a meaningful gap.

Methodological Strengths

Comprehensive benchmark design. Harmony-Bench's three-tier structure (ambient-only, speech-only, complex mixed) disentangles capability dimensions that prior benchmarks conflated. This allows the paper to show that Harmony's synchronization advantage holds across all three difficulty levels and that it doesn't sacrifice ambient sound quality for speech quality or vice versa. The inclusion of Chinese speech evaluation is particularly valuable for testing cross-lingual generalization.

Transparent ablation that validates additive contributions. The progressive ablation in Table 3 demonstrates that each component contributes independently — the improvements are largely monotonic and non-overlapping. This is methodologically stronger than a set of leave-one-out ablations because it shows the trajectory of improvement as components accumulate, making it clear that the components address different bottlenecks.

Qualitative evidence aligned with quantitative trends. The attention map visualizations (Figure 6), the side-by-side comparison stills (Figure 5), and the supplementary waveform comparisons (Figures 7, 8) provide intuitive validation of the quantitative metrics. The observation that competing methods achieve good identity scores by generating static videos (while Harmony maintains higher dynamism) is an important qualitative insight that contextualizes the quantitative results.

Methodological Weaknesses and Missing Evidence

No confidence intervals or statistical testing. All Tables (2, 4–6, 7) report point estimates without variance. On a 50-item per-subset benchmark, differences of a few tenths in AQ or IQ may not be statistically significant. The synchronization metric gaps are large enough to likely be robust, but the quality metric comparisons (where Harmony's advantage is marginal) would benefit from error bars.

Difficulty estimation for real-world deployment is unaddressed. Unlike some papers that develop explicit difficulty estimation protocols, Harmony provides no mechanism for estimating whether a given prompt falls into the "easy" or "hard" regime for the model. However, this criticism is less applicable here than in papers that claim test-time compute allocation benefits — Harmony's contribution is architectural/training methodology, not inference-time resource allocation, so difficulty estimation is less critical.

The Wan2.2-5B initialization advantage is not isolated. The video branch starts from a strong pre-trained model, while the audio branch is trained from scratch. The paper does not compare against a baseline that uses the same Wan2.2 initialization with a simpler interaction mechanism (e.g., standard cross-attention with joint training only). It's possible that some of Harmony's visual quality advantage comes from the Wan2.2 initialization rather than from the proposed methods. A baseline that pairs Wan2.2 with standard joint training and a simple cross-attention interaction module would help isolate how much of the improvement is attributable to the base model versus the methodological innovations.

No evaluation on real-world distribution shift. All evaluation is on Harmony-Bench, which was constructed by the authors. There's no evaluation on third-party benchmarks or in-the-wild data. The benchmark was designed to test specific capabilities, which is appropriate for a methods paper, but the generalization to uncurated real-world prompts remains unvalidated.

Training data overlap with evaluation is unclear. The training corpus includes data from OpenHumanVid [31] and other public sources, and the test set includes "real-world" samples. The paper doesn't specify whether the real-world test samples are drawn from the same distribution as the training data or whether they are held-out from different sources. If there is training-distribution overlap, the reported metrics may overestimate real-world performance.

The paper doesn't compare against training a larger model without the proposed methods. The central thesis is that targeted methodological interventions solve synchronization better than scaling. But the paper never tests a scaled-up baseline — e.g., a version of Ovi with 2× or 4× the parameters, trained on the same data. Without this comparison, the claim that Harmony is "more effective than simply scaling up models" (Section 5) remains an assertion rather than an empirical finding.

The ablated models are trained on speech-only data (Table 3), while the final model is trained on speech + environmental sounds. The ablation study uses a human-speech-only dataset, which means the component contributions are measured in a narrower training regime than the final model. The relative importance of components might differ when environmental sound data is included. The Sync-C of 6.51 in the ablation (Table 3) is higher than the Sync-C of 6.51 reported for the speech-video subset (Table 4), suggesting the ablation may overestimate speech-only performance relative to the full model.

Summary: What the Experiments Establish and What They Don't

Established convincingly:

  • Harmony achieves substantially better audio-visual synchronization than Ovi, UniVerse-1, and JavisDiT on a new 150-case benchmark spanning speech, environmental sounds, and complex mixed scenes.
  • The combination of Cross-Task Synergy training, Global-Local Decoupled Interaction with RoPE alignment, and SyncCFG yields additive improvements in synchronization, with SyncCFG providing the single largest gain.
  • Cross-Task Synergy training with clean-latent conditioning converges faster and to higher alignment than standard joint training (Figure 3).
  • The model generalizes cross-lingually (Chinese speech) and across diverse visual styles (cartoon, ink wash) and ambient sound types (music, fireworks).

Not established or partially established:

  • Whether Correspondence Drift specifically (as opposed to general joint training difficulty) causes slow convergence — the evidence is consistent with the drift hypothesis but does not uniquely confirm it.
  • Whether the Global-Local Decoupling specifically (as opposed to the RoPE rescaling within it) resolves an architectural conflict — the ablation shows limited gain from decoupling alone without RoPE alignment.
  • Whether the separate speech and text encoders improve phonetic precision beyond what a single unified encoder would achieve — this design choice is unablated.
  • Whether Harmony's visual quality advantage over Ovi is meaningful or within noise — the margins are small and no confidence intervals are provided.
  • How SyncCFG performance varies with guidance scale — no sweeps are reported.
  • Whether Harmony scales better than simply training a larger Ovi-style model — no scaled baseline is tested.

6. Limitations and Trade-offs

6.1 Difficulty Estimation and Problem Routing Are Not Addressed

The assumption or constraint. The paper's entire framework assumes that all generation prompts are treated uniformly — there is no mechanism for estimating whether a given generation task is "easy" or "hard" for the model, and no adaptive allocation of compute or strategy based on difficulty. This is in contrast to some test-time compute scaling papers that condition on estimated difficulty, but it reflects a genuine gap in Harmony: the model applies the same architecture, same training, and same inference procedure to all inputs regardless of their inherent complexity. The paper does not explicitly acknowledge this as a limitation, but it is a structural consequence of the design — Harmony improves synchronization universally but provides no tools for understanding when it will succeed or fail on a particular input.

The consequence. A practitioner deploying Harmony cannot predict whether a given prompt — particularly one that falls outside the training distribution (unusual acoustic scenes, rare sound-visual couplings, prompts requiring fine-grained synchronization of multiple simultaneous sound sources) — will produce well-synchronized output or degraded results. The complex scene subset of Harmony-Bench (Table 6) represents the most challenging test condition, but it is a fixed 50-case set that may not cover edge cases. If a user submits a prompt requiring, say, three simultaneous sound sources with distinct visual correlates, there is no principled basis for estimating whether the model will handle it. More critically, there is no mechanism for routing difficult cases to a fallback strategy (e.g., generating audio and video separately with a post-hoc alignment step, or using a more capable proprietary model).

What evidence exists in the paper. The per-subset results in Tables 4, 5, and 6 show that Harmony's performance varies across difficulty levels. On the speech-video subset (Table 4), Sync-C is 6.51 and WER is 0.15 — near-ceiling performance. On the complex scene subset (Table 6), Sync-C drops to 4.70 and CU drops to 4.86 (below Ovi's 5.33). The Chinese cross-lingual evaluation (Table 7) shows WER of 0.92 — higher than the English WER of 0.15, though still far better than baselines. These numbers demonstrate that difficulty matters, but the paper provides no analysis of which specific prompt features correlate with degraded performance, no per-example error analysis on Harmony-Bench, and no difficulty prediction model.

Mitigation status. Not addressed. The paper does not propose a difficulty estimator, a confidence score, or a routing mechanism. Future work on this would require either (a) training a lightweight classifier to predict generation difficulty from prompt features (analogous to difficulty estimation in the test-time compute scaling literature), or (b) developing a generation-time confidence metric (e.g., based on the consistency of SyncCFG guidance vectors across denoising steps) that could flag potentially desynchronized outputs for human review or alternative processing.

6.2 The 2048-Sample Difficulty Estimation Overhead Is Not Present — But a Different Practical Overhead Is

The constraint. Unlike some papers that require expensive oracle difficulty estimation at inference time, Harmony does not face this specific issue. However, it has its own practical overhead that is not accounted for in any efficiency analysis: the three-stage training curriculum and the Cross-Task Synergy objective multiply the effective training cost compared to standard joint training. During Stage 3 cross-task training, each training iteration computes three forward passes (joint, audio-driven, video-driven) and three corresponding losses. The loss weights (λv = 0.1, λa = 0.3) determine the gradient contribution, but the computational cost of the forward passes is incurred regardless. Stage 1 (100,000 iterations at batch size 1,536) and Stage 2 (20,000 iterations) add further pre-training cost before any cross-modal learning occurs.

The consequence. A practitioner deciding whether to adopt Harmony over a simpler joint-training baseline (e.g., Ovi with a single global cross-attention module) cannot determine whether the synchronization improvements justify the training cost. The paper provides no training cost comparison: no FLOP counts, no GPU-hours, no wall-clock training time, no comparison of total training compute between Harmony and the baselines. For an organization with limited compute, the three-stage curriculum plus the per-iteration overhead of Cross-Task Synergy could mean the difference between being able to train the model and not. Additionally, the asymmetric loss weights (0.1 and 0.3) were presumably determined through hyperparameter sweeps, which add further unaccounted tuning cost.

What evidence exists in the paper. The paper specifies the training configuration — Stage 1: 100K iterations, batch size 1,536, 10-second clips; Stage 2: 20K iterations; Stage 3: 10K iterations, batch size 128 — but provides no FLOP or time measurements. The main paper (Section 4.1) and Appendix B describe the training protocol without any efficiency analysis. The Flow Matching objective with shift 5 and 40 inference steps is specified, but inference latency (wall-clock time per generation) is not reported.

Mitigation status. Not addressed. The paper makes no attempt to quantify training or inference efficiency, and does not discuss whether the three-stage curriculum is strictly necessary or whether a streamlined training protocol could achieve comparable results. A cost-benefit analysis (synchronization improvement per unit of training compute) is entirely absent. This is a significant practical gap for anyone considering reproducing or deploying the method.

6.3 Single Model Family and Single Dataset: Generalization to Other Architectures and Domains Is Unvalidated

The assumption or constraint. All experiments use a single base architecture: a Wan2.2-5B-initialized video branch paired with a symmetric audio diffusion transformer. The training data is a specific curated corpus of ~4 million clips. The evaluation is exclusively on Harmony-Bench, a 150-case test set constructed by the authors. The paper states (Section 4.1) that data comes from "public sources like OpenHumanVid, AudioCaps, and WavCaps" supplemented with "our own curated high-quality collections," but does not report results on any third-party benchmark or out-of-distribution evaluation set.

The consequence. Several aspects of Harmony's performance may not transfer to other settings. First, the architecture-specific claims — particularly the Global-Local Decoupled Interaction Module's benefit over monolithic cross-attention, and the RoPE rescaling trick for temporal alignment — may depend on properties of the Wan2.2 transformer (its layer count, hidden dimension, positional encoding scheme) that do not hold for other video diffusion architectures (e.g., UNet-based models, models with different temporal processing, models that don't use RoPE). Second, the training data specificity — the curated 2M-speech and 2M-environmental clip supplements to public data — may contain distributional biases (overrepresentation of certain acoustic scenes, speaker demographics, visual styles) that inflate Harmony-Bench performance relative to in-the-wild deployment. Third, the benchmark specificity — Harmony-Bench was designed by the same team that built the model, and the 150 cases may inadvertently favor Harmony's design choices. The paper acknowledges that existing benchmarks are "inadequate" (Appendix C.1), but replacing inadequate benchmarks with a custom one creates a risk of overfitting to the evaluation set even without explicit tuning — simply through architectural and training decisions made with awareness of what the benchmark tests.

What evidence exists in the paper. The paper provides no cross-architecture evaluation (e.g., applying the GLDI module to a different video diffusion backbone). The Chinese speech evaluation (Table 7) provides some evidence of cross-lingual generalization, but this is within the same model trained on multilingual data, not a test of whether the method transfers to a different base architecture. The diverse visual styles (Figure 12) and diverse ambient sounds (Figure 13) are qualitative demonstrations, not systematic out-of-distribution evaluations.

Mitigation status. Not addressed. The paper does not discuss architectural generalization, does not evaluate on third-party benchmarks, and does not analyze failure cases on Harmony-Bench that might reveal systematic generalization gaps. The authors position Harmony as a methodological contribution (Section 5: "more effective than simply scaling up models"), which implies the methods should transfer, but this transfer claim is entirely unevaluated. A minimal step would be to report results on at least one external benchmark (even if limited — e.g., the speech-video subset of an existing dataset with ground-truth alignment annotations) to establish baseline generalizability.

6.4 SyncCFG Depends on Cross-Task Synergy Training — It Is Not a Standalone Inference Technique

The constraint. Synchronization-Enhanced CFG (Equations 9 and 10) requires the model to be capable of producing audio-driven video predictions ($\hat{\epsilon}_{\theta}^{\text{driven}}(z_{v,t}, z_{a,0}^{\text{null}})$) and video-driven audio predictions ($\hat{\epsilon}_{\theta}^{\text{driven}}(z_{v,0}^{\text{null}}, z_{a,t})$). These capabilities exist only because the model was trained with the Cross-Task Synergy objective (Equations 3 and 4), which includes explicit auxiliary losses for audio-driven and video-driven generation. A standard joint-generation model — even one with the same GLDI architecture — cannot compute SyncCFG because it has no driven-generation pathway.

The consequence. SyncCFG cannot be applied as a drop-in inference-time improvement to existing joint audio-video models (Ovi, UniVerse-1, JavisDiT). It is not comparable to standard CFG scale tuning or other inference-only techniques. To benefit from SyncCFG, a practitioner must (a) train the model with the Cross-Task Synergy objective from scratch, which requires implementing the auxiliary tasks and the weighted loss, and (b) construct the negative anchor latents ($z_{a,0}^{\text{null}}$ and $z_{v,0}^{\text{null}}$), which the paper describes conceptually but does not specify algorithmically (how is "mute audio" represented in the VAE latent space? how is "static video" constructed?). This makes SyncCFG a tightly coupled training-inference innovation rather than a modular inference technique, which substantially increases the barrier to adoption for practitioners who already have trained joint-generation models.

What evidence exists in the paper. The ablation in Table 3 demonstrates this coupling directly: SyncCFG is added last (row 5), on top of a model already trained with CTS. Without CTS (row 1–3), SyncCFG cannot be applied. The paper provides no experiment that attempts to approximate SyncCFG for a non-CTS model (e.g., by using an independently trained audio-to-video model as the negative anchor source).

Mitigation status. The paper does not discuss this coupling as a limitation. It presents SyncCFG as a novel CFG formulation (Section 3.4) without explicitly noting that it requires a specific training regime inaccessible to existing models. The coupling is architecturally elegant — Cross-Task Synergy training creates capabilities that SyncCFG exploits — but it also means the contributions are not separable. A practitioner cannot adopt just SyncCFG or just Cross-Task Synergy; they must adopt both or neither. This is a tradeoff the paper should acknowledge but does not.

6.5 No Ablation of the Reference Audio Mechanism or the Three-Stage Curriculum

The constraint. The paper's training pipeline involves a three-stage curriculum (Appendix B): Stage 1 audio pre-training (100K iterations), Stage 2 timbre disentanglement (20K iterations), and Stage 3 cross-task training (10K iterations). The reference audio latent $z_r$ is prepended to the noisy audio latent and also serves as the query in the Global Style Alignment module (Equation 7). These design choices are presented as integral to the system, but neither the curriculum staging nor the reference audio mechanism is ablated.

The consequence. Several important questions are left unanswered: (1) Is the three-stage curriculum necessary? Could a two-stage variant (skipping timbre disentanglement) achieve comparable synchronization? Could end-to-end joint training from scratch (with Cross-Task Synergy losses) match the staged approach given enough iterations? (2) Does the reference audio prepending actually contribute to synchronization, or only to voice cloning? The paper demonstrates voice cloning as a capability (Figure 9), but never ablates the reference audio pathway to determine whether it helps or hinders synchronization — it's possible that the reference latent introduces an additional conditioning pathway that the model must learn to integrate, potentially competing with the cross-modal synchronization signal. (3) Is the Global Style Alignment module's use of $z_r$ as query (Equation 7) superior to alternative designs, such as using a learned style token or directly aggregating global video features into the audio branch?

What evidence exists in the paper. None of these ablations are present. The ablation study (Table 3) starts from a baseline that already includes reference audio conditioning, and the progressive additions are GLDI → RoPE → CTS → SyncCFG. The three-stage curriculum is described in Appendix B but never compared against a shorter pipeline. The voice cloning results (Figure 9) demonstrate that reference audio works, but not whether it is architecturally optimal or whether synchronization would improve if the reference pathway were simplified.

Mitigation status. Not addressed. The paper treats the three-stage curriculum and reference audio mechanism as settled design decisions without empirical justification. The timbre disentanglement stage (Stage 2) is motivated conceptually — "to enable the model to disentangle general acoustic characteristics from specific content" — but its contribution to final synchronization is unmeasured. For practitioners, this means the training protocol must be adopted in full, with no guidance on which stages are essential versus which could be compressed or omitted under compute constraints.

6.6 Hard Synchronization Cases and Failure Modes Are Not Analyzed

The constraint. The paper reports aggregate and per-subset metrics on Harmony-Bench but provides no analysis of which specific cases Harmony fails on, what the failure modes look like, or whether there are systematic categories of prompts where synchronization degrades. The complex scene subset (Table 6) shows that Sync-C drops from 6.51 (speech-video) to 4.70, and DeSync rises from 8.63 to 6.43 (lower is better, so this is an improvement, but the numbers are not directly comparable across subsets since the subsets contain different prompts). There is no per-example breakdown, no stratification by prompt characteristics (e.g., number of sound sources, speech rate, visual complexity), and no qualitative failure cases in the paper or appendix — all qualitative examples (Figures 5, 7, 8, 9, 10, 11, 12, 13) are success cases.

The consequence. A practitioner cannot anticipate when Harmony will fail. Will it struggle with very fast speech? With overlapping sound events? With prompts where the visual event that causes a sound is small in the frame (e.g., a distant door closing)? With non-human vocalizations (animal sounds, singing)? Without a failure analysis, the model's capability boundaries are unknown, and deployment in high-stakes applications (where a desynchronized output could be misleading or harmful) is difficult to justify. The paper's claim that Harmony "establishes a new state-of-the-art" is supported for aggregate synchronization metrics, but aggregate metrics can mask catastrophic failures on important subcategories.

What evidence exists in the paper. The only evidence about difficulty-dependent performance is the per-subset breakdown (Tables 4, 5, 6, 7). The complex scene subset (Table 6) is described as "the most challenging" and shows degraded CU (4.86 vs. Ovi's 5.33) — suggesting that audio "usefulness" drops in mixed scenes. The DD on complex scenes (0.32) is below Ovi's 0.41, a reversal from other subsets. These hints suggest that complex scenes introduce tradeoffs, but the paper does not investigate further. The attention visualizations (Figure 6) show successful localization, but no attention maps for failure cases are shown.

Mitigation status. Not addressed. The paper does not provide a failure analysis, error categorization, or per-example results. The Harmony-Bench construction (Appendix C.1) describes the test cases as "meticulously designed" and "progressively challenging," but the evaluation is purely aggregate. A minimal step would be to report per-example Sync-C distributions (e.g., histograms or box plots) to show whether the improvement over baselines is uniform or driven by a subset of cases, and to show at least one qualitative failure case with analysis of what went wrong.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper advances the field of joint audio-video generation not by proposing a larger model or a novel architecture in the conventional sense, but by making a diagnostic contribution — it identifies three specific, mechanistic failure modes in the standard joint diffusion training paradigm and designs targeted interventions for each. This shifts the research conversation from "we need better cross-modal architectures" (the dominant prior assumption) to "we need to understand why cross-modal alignment fails during training and inference, then fix those root causes."

The magnitude of this shift is moderate but foundational — it is not a paradigm overthrow (joint diffusion remains the generative framework), but it provides a conceptual toolkit that reorients how researchers should think about synchronization failures. Prior to this work, the implicit hypothesis was that synchronization quality would scale naturally with model capacity, data scale, and architectural sophistication — just as image quality in text-to-image models improved dramatically with DiT architectures and larger datasets. Harmony demonstrates that this hypothesis is incomplete: the standard joint training objective (Equation 1) contains an inherent instability (Correspondence Drift) that no amount of scaling can fully resolve, because the fundamental problem is not insufficient capacity but an unstable learning signal. This is analogous to the shift in thinking that occurred when the RLHF community recognized reward hacking not as a model-size problem but as an objective-design problem — it redirects effort from "build bigger" to "build smarter training and inference procedures."

The paper also resolves a tension in the prior literature that was visible but undiagnosed. Table 1 shows existing models specialized in either speech or environmental sounds, with generalist models (UniVerse-1, Ovi) achieving poor alignment quality on both. The field had tacitly accepted that joint generation of high-quality speech and high-quality environmental sounds with precise synchronization was an unsolved problem, but the explanations offered were vague: "it's hard," "the modalities are different," "more data is needed." Harmony provides a concrete, testable explanation: joint training is inherently unstable for learning fine-grained correspondence because both modalities are noisy during training, and standard CFG amplifies intra-modal conditioning without touching cross-modal consistency. This transforms the problem from an amorphous challenge into a set of specific, addressable issues.

Several research directions become more attractive in light of this work:

  • Training-objective design for multi-modal diffusion. The Correspondence Drift diagnosis suggests that any joint diffusion process where two modalities must learn fine-grained temporal correspondence will face similar instability. This makes training-objective interventions (analogous to Cross-Task Synergy) a high-priority research area for video-text alignment, audio-text alignment, multi-sensor fusion, and robotics domains where joint diffusion is applied.
  • CFG redesign for cross-modal tasks. SyncCFG demonstrates that the negative anchor in classifier-free guidance can be repurposed to amplify specific aspects of conditioning — cross-modal synchronization in this case, but the principle generalizes. This opens a design space for modality-specific CFG formulations that isolate different conditioning signals (style, content, temporal alignment, spatial correspondence) through different negative anchors.
  • Architectural decomposition for multi-modal interaction. The Global-Local Decoupled Interaction Module validates the strategy of separating global and local cross-modal interactions into specialized components that operate on different representational substrates. This decomposition principle is likely applicable beyond audio-video generation to any multi-modal architecture where fine-grained alignment and holistic consistency impose conflicting computational demands.

Conversely, some research directions become less attractive:

  • Monolithic cross-attention for multi-modal synchronization. The paper provides evidence (Table 3, baseline vs. +GLDI) that a single cross-attention mechanism conflates objectives and underperforms decoupled designs. Future work that proposes "better cross-attention" as a synchronization solution without addressing the local-vs-global tension will need to confront this evidence.
  • Scaling as the primary path to synchronization. The ablation study shows that targeted methodological interventions (GLDI + RoPE + CTS + SyncCFG) yield a 55% Sync-C improvement (4.20 → 6.51) without increasing model size. This provides strong evidence that architectural and training methodology improvements are higher-leverage than parameter scaling for synchronization — a finding that should shift resource allocation in both academic and industrial labs working on this problem.
  • Standard CFG as sufficient for cross-modal tasks. SyncCFG's 28% Sync-C improvement (+1.42) over a model already using standard CFG demonstrates that standard CFG leaves substantial cross-modal performance on the table. Future joint-generation work should not treat CFG as a solved inference component.

Follow-Up Research This Work Enables

1. Directly testing the Correspondence Drift hypothesis through noise-correlation experiments. The paper argues that Correspondence Drift occurs because both modalities' latents are simultaneously noisy and evolving, creating an unstable learning target. A direct test would manipulate the noise correlation between modalities during training: compare (a) independent noise sampling for audio and video (current practice, maximal drift), (b) perfectly correlated noise (same noise applied to both modalities after appropriate rescaling), and (c) anti-correlated noise, measuring convergence speed and final Sync-C for each condition. If drift is the mechanism, correlated noise should accelerate convergence because the "optimal correspondence" shifts less when both modalities are perturbed similarly. This experiment requires no architectural changes — only modifying the noise sampling procedure in the training loop — and would provide definitive evidence for or against the drift hypothesis. A negative result (no difference across conditions) would suggest that the benefit of Cross-Task Synergy comes from some other property of clean-latent conditioning, not from stabilizing the correspondence mapping.

2. Ablating Global-Local Decoupling vs. RoPE rescaling independently. The paper's ablation (Table 3) shows GLDI alone provides minimal gain (Sync-C +0.09), while adding RoPE alignment yields a substantial jump (+0.51). This pattern leaves unclear whether the decoupling itself matters or whether the entire benefit comes from RoPE rescaling. A targeted experiment would compare four conditions on the speech-video subset: (a) standard global cross-attention (baseline), (b) local windowed attention without RoPE rescaling (testing whether locality alone helps), (c) global attention with RoPE rescaling applied to the full sequence (testing whether continuous positional encoding alone helps even without locality), and (d) the full GLDI with RoPE (current design). If (c) performs nearly as well as (d), it would demonstrate that RoPE rescaling — not decoupling — is the active ingredient, and the paper's architectural decoupling argument would need revision. The attention visualizations (Figure 6) would be particularly informative for this ablation: does global attention with RoPE still localize to sound sources, or does locality provide a necessary inductive bias for interpretable attention patterns?

3. Extending SyncCFG to standard (non-CTS) joint-generation models via approximate negative anchors. SyncCFG requires Cross-Task Synergy training because the negative anchor predictions ($\hat{\epsilon}_{\theta}^{\text{driven}}$) are only available if the model was trained with uni-directional tasks. A practical research question is whether approximate negative anchors can provide similar benefits for existing models that weren't trained with CTS. For example: train a lightweight adapter that maps a standard joint-generation model's internal representations to approximate audio-driven and video-driven noise predictions, using a small amount of paired uni-directional training data. Or, construct negative anchors by running a pre-trained single-modality diffusion model (e.g., an audio-to-video model, a video-to-audio model) in parallel and using its predictions as the negative baseline. The key measurement would be how much of the 1.42 Sync-C improvement is recoverable through approximate anchors vs. requiring full CTS training — this would determine whether SyncCFG's benefit can be partially decoupled from its training prerequisite, making it accessible to practitioners with existing models.

4. Stress-testing Harmony on systematically varied acoustic complexity. The paper demonstrates Harmony's advantage on three fixed-difficulty subsets (ambient, speech, complex scene), but provides no analysis of which specific prompt features correlate with synchronization degradation. A systematic stress test would construct a controlled benchmark varying one dimension at a time: (a) speech rate (words per second, from slow to very fast), (b) number of simultaneous sound sources (1, 2, 3, 4+), (c) visual salience of the sound-producing object (large/central vs. small/peripheral in the frame), (d) audio-visual temporal offset (sounds that precede, coincide with, or follow visual events by varying delays). For each dimension, measure Sync-C, Sync-D, and WER to identify Harmony's capability boundaries. This would produce a capability profile that practitioners can use to estimate whether their target application falls within Harmony's reliable operating range. The finding that complex-scene DD drops below Ovi (0.32 vs. 0.41, Table 6) already hints at a tradeoff — systematic stress testing would reveal whether this is a general pattern or specific to the benchmark's complex scene construction.

5. Combining Harmony's methodological innovations with a scaled-up baseline to isolate scaling vs. methodology contributions. The paper claims Harmony is "more effective than simply scaling up models" (Section 5), but never tests a scaled-up baseline without the proposed methods. A crucial experiment would train two models on identical total compute: (a) an Ovi-style model (monolithic cross-attention, standard joint training, standard CFG) with 2-4× the parameters of Harmony, and (b) Harmony at its current scale. If the scaled-up baseline matches or exceeds Harmony's Sync-C, it would undermine the paper's central claim that methodology beats scaling for synchronization — the observed improvement might be achievable through either path. If Harmony outperforms the scaled baseline, it provides strong evidence for the methodology-over-scaling thesis. This experiment is expensive (training a scaled baseline) but is the cleanest way to validate the paper's most ambitious claim. A negative result (scaling works) would not invalidate Harmony's contributions — the method could still be the more compute-efficient path — but would reframe them as an efficiency improvement rather than a unique capability unlock.

6. Applying Cross-Task Synergy to other cross-modal diffusion domains. The Correspondence Drift diagnosis and Cross-Task Synergy solution are not specific to audio-video generation — any joint diffusion process learning fine-grained cross-modal correspondence from simultaneously noisy latents should face similar instability. Promising domains for replication include: (a) joint text-video generation (do text-video models trained with clean text conditioning noisy video converge faster to spatio-temporal alignment?), (b) joint audio-motion generation (can clean motion conditioning accelerate audio-driven gesture synthesis?), and (c) multi-sensor fusion in robotics (does conditioning vision on clean proprioceptive signals improve visuomotor alignment?). For each domain, the minimal experiment replicates Figure 3: compare a uni-directional baseline (clean conditioning), standard joint training, and Cross-Task Synergy training using identical architectures, measuring convergence speed and final alignment quality. A consistent pattern of Cross-Task Synergy improving convergence across domains would elevate Correspondence Drift from an audio-video specific observation to a general principle of multi-modal diffusion training. A null result in some domain (e.g., text-video, where the modalities already share substantial semantic structure) would help delineate the boundary conditions — drift may primarily affect domains where the cross-modal mapping is fine-grained and non-redundant (like audio-video alignment, where sub-frame timing precision matters) rather than domains where the mapping is coarser and more semantic.

Practical Applications and Downstream Use Cases

Digital avatar and virtual presenter systems. The combination of Harmony's voice cloning capability (Appendix F, Figure 9), cross-lingual speech generation (Table 7: Chinese WER of 0.92), and precise lip synchronization (Sync-C of 6.51 on speech-video, Table 4) makes it directly applicable to virtual avatar systems where a single reference image and a short reference voice clip are used to generate talking-head video in multiple languages. The key practical advantage over existing pipelines (which typically chain separate TTS, lip-sync, and video generation models) is that Harmony produces the synchronized output in a single forward pass, eliminating cascading errors where TTS output doesn't match the lip-sync model's expected input, or where the video generation model's facial dynamics don't track the TTS prosody. The diverse style capability (Appendix H, Figure 12: Disney-style, ink wash painting) further enables branded virtual presenters with consistent visual identity. The 40-inference-step cost is a latency consideration, but for applications where generation can be done offline (pre-recorded presentations, localized content), the quality improvement over cascaded systems — particularly the 27% Sync-C improvement over Ovi — is likely worth the compute.

Automated video content creation with synchronized Foley and ambient sound. For social media content, advertising, and short-form video production, Harmony's ability to generate environmental sounds precisely synchronized to visual events (Table 5: DeSync of 0.70 vs. Ovi's 1.06) addresses a major pain point in current workflows. Existing approaches typically require either (a) manual Foley artistry (expensive, time-consuming), (b) library sound effects with manual alignment (labor-intensive), or (c) separate video-to-audio generation followed by alignment post-processing (error-prone cascading). Harmony's joint generation produces the synchronized audio-visual output directly, with the model handling both the acoustic content (what sound to generate) and the temporal alignment (when it should occur). The complex scene results (Table 6) are particularly relevant here: real-world content often combines speech with ambient sounds (e.g., a vlogger speaking while cooking, with sizzling sounds synchronized to visual cooking actions), and Harmony maintains WER of 0.15 and Sync-C of 4.70 in this regime — speech remains intelligible while ambient sounds stay synchronized. For high-volume content production where manual audio work is the bottleneck, this could substantially reduce post-production time.

Accessibility tooling: automated audio description and sign language video generation. While the paper focuses on joint generation from scratch, the Cross-Task Synergy training gives Harmony native audio-driven video generation capability (Appendix G, Figure 10). This enables a workflow where an existing audio track (e.g., a podcast, an audiobook, a lecture) drives the generation of a synchronized talking-head video — the model produces lip movements, facial expressions, and head motion that correspond to the input speech. For deaf and hard-of-hearing users, this could generate sign language or lip-readable video avatars from audio content. The cross-lingual capability (Chinese WER 0.92, Table 7) extends this to multilingual content, and the voice cloning (Appendix F) allows the generated avatar to use a consistent, familiar speaker identity across multiple pieces of content. The key technical requirement here is robustness to out-of-distribution audio — the model was trained on curated speech data and may degrade on noisy, conversational, or heavily accented audio — so deployment would require careful evaluation of the WER-to-real-audio-quality relationship that the paper does not provide (only clean transcript-conditioned WER is reported).

Data generation for self-improving audio-visual models. The Harmony framework — particularly the Cross-Task Synergy training paradigm — provides a recipe for generating high-quality synchronized training data that could bootstrap further model improvements. A self-improvement loop would: (a) use Harmony to generate synchronized audio-visual clips from text prompts spanning diverse acoustic scenes, (b) filter the generated clips using the synchronization metrics (Sync-C, Sync-D, DeSync) to retain only high-alignment outputs, and (c) fine-tune the model (or train a larger model) on this filtered synthetic data. This is analogous to rejection sampling fine-tuning in language models, but applied to the synchronization dimension. The key enabler is that Harmony's Sync-C of 6.51 on speech (Table 4) is high enough that the generated data is largely correct — the model is not learning from desynchronized outputs. The risk is that training on self-generated data could amplify any systematic biases in Harmony's synchronization (e.g., slightly delayed lip movements that are consistently wrong but not detected by the metrics), so human evaluation on a subset of generated training data would be essential before scaling this loop.

When to Prefer This Method

The paper does not articulate an explicit tradeoff matrix against named alternatives under specific operating conditions — its positioning is as a universally better approach for synchronization rather than as a method suited to particular deployment regimes. The comparison to Ovi, UniVerse-1, and JavisDiT is presented as superiority, not as a contingent tradeoff. Therefore, a conditional "prefer A when / prefer B when" decision rule would be an imposition rather than an extraction from the paper's stated claims. The appropriate guidance for practitioners is simpler: if audio-visual synchronization quality is the primary objective (as it is for talking-head generation, Foley-synchronized content, and music performance video), Harmony's 27-60% Sync-C advantage over the next best open-source alternative (Ovi) makes it the clear choice among available open-source methods, provided the training cost of the three-stage curriculum and the CTS objective overhead is acceptable. If generation fidelity alone (visual quality, audio production quality) is the primary objective and synchronization is secondary, Ovi remains competitive — the visual quality margins (AQ 0.59 vs. 0.57) are small enough that other factors (inference latency, training cost, ease of implementation) may dominate the decision.