MoGA: Mixture-of-Groups Attention for End-to-End Long Video Generation

ArXiv: 2510.18692

🎯 Pitch

MoGA ditches blockwise sparsity in favor of a learned token router that achieves over 3× FLOPs reduction versus full attention, enabling minute-long 480p video generation with a ∼580k context. It outperforms both block-sparse and full-attention baselines on VBench while running 1.7× faster, showing that routing accuracy, not just sparsity, is what unlocks end-to-end long video.

---

1. Executive Summary

This paper introduces Mixture-of-Groups Attention (MoGA), a sparse attention mechanism for end-to-end long video generation with Diffusion Transformers that replaces coarse block-level similarity estimation with precise, learned group assignments via a lightweight token router (a single linear layer followed by softmax gating, analogous to Mixture-of-Experts routing). Fine-tuning on the open-source Wan2.1 models (1.3B and 14B) and an MMDiT-based architecture, MoGA enables generation of minute-long, multi-shot, 480p videos at 24 fps with a context length of approximately 580k tokens, achieving over 3× FLOPs reduction versus full attention (2.26 PFLOPs vs. 6.94 PFLOPs for 30-second videos) and a 1.7× training and inference speedup. The paper further demonstrates that MoGA combined with complementary local spatiotemporal window attention (STGA) consistently outperforms both training-free sparse baselines (SVG, VMoBA) and full-attention multi-shot methods (EchoShot) on VBench metrics, establishing that precise token-level routing preserves shot-spanning identity and background consistency without the accuracy–efficiency trade-offs inherent in blockwise estimation—though hard boundary transitions still require explicit shot-level textual conditioning to maintain scene-level coherence.

2. Context and Motivation

The Core Problem: Full Attention Is a Non-Starter for Long Video Generation

The fundamental problem this paper tackles is computational in nature but has deep architectural implications: standard self-attention in Diffusion Transformers (DiTs) scales quadratically with sequence length, making it prohibitive for end-to-end long video generation. To understand the scale of this problem, consider the numbers the paper provides: generating a 1-minute video at 480p resolution with roughly 1,600 tokens per frame across 961 frames (at 16 fps) yields a total token count of approximately 384,000 tokens. Since self-attention complexity is $\mathcal{O}(N^2)$, processing such a sequence requires computing roughly $1.5 \times 10^{11}$ pairwise interactions — per attention layer, per diffusion timestep. This is not merely a performance inconvenience; it is an architectural barrier that prevents DiTs from being applied in any end-to-end fashion to minute-scale video generation.

The paper quantifies this in Section 6 (Appendix): for the Wan2.1-1.3B model, full attention on a 30-second video costs 6.94 PFLOPs — an amount that would require enormous computational resources for even a single generation. The problem compounds as video duration increases: full attention grows quadratically, meaning a 60-second video would cost roughly 4× more than a 30-second one, rapidly exceeding any practical compute budget.

This matters because video generation is inherently temporal. Unlike image generation, where each token relates to others within a single spatial frame, video tokens must attend across time to maintain object persistence, character identity, background consistency, and narrative coherence. The paper explicitly frames this in terms of scaling laws (Section 1):

"In video generation, given the inherently temporal nature, progress requires not only scaling parameters and data but, more critically, scaling the effective context length. This need is especially salient for long-form video generation (e.g., movies), where persistent memory is essential for maintaining consistency of environments and characters."

The key phrase is "persistent memory." In short video generation (2–5 seconds), a model can rely on local temporal coherence — nearby frames are similar, and attention can be restricted to a sliding window without much loss. But for minute-scale videos with multiple shots, characters reappearing after being off-screen for 20 seconds, and environments persisting across scene transitions, the model genuinely needs to attend across large temporal gaps. If attention is made too local or too sparse without careful design, this long-range information is lost, and consistency degrades — characters change appearance, backgrounds shift, and the narrative falls apart.

The Gap: Existing Efficiency Solutions Either Lose Information or Make Coarse Approximations

Prior to MoGA, the field had developed three broad strategies for handling long video contexts, each with significant limitations that the paper identifies and addresses.

---

Multi-Stage Pipelines: Disjoint Objectives and Error Accumulation

The first paradigm, which the paper discusses in Section 2.1, decomposes long video generation into multiple stages — typically generating keyframes first, then synthesizing intermediate frames between them. Representative works include Captain Cinema (hierarchical planning with top-down keyframe generation and bottom-up synthesis), VideoTetris (compositional generation), and Vlogger. The paper's critique of this approach is pointed:

"This design yields disjoint objectives that are not directly optimized for the end task, leading to error accumulation across stages. It also introduces hand-crafted inductive biases, hindering scalability."

The problem is subtle but important. In a multi-stage pipeline, the keyframe generation model is trained to produce plausible individual frames, but it has no direct signal about how those frames will be used in the interpolation stage. The interpolation model, in turn, must work with whatever keyframes it receives, even if they contain artifacts or inconsistencies that make smooth interpolation difficult. Errors compound: a slightly inconsistent keyframe leads to an awkward interpolation, which becomes input to the next keyframe prediction, and so on. This is fundamentally a credit assignment problem — the end-to-end objective (producing a coherent long video) is not backpropagated through all stages.

Moreover, these pipelines introduce strong inductive biases about what a "shot" looks like, how transitions work, and what frame rate to use. While these biases help when they match the target distribution, they limit the model's ability to learn novel structures or scale to arbitrary video formats. The paper positions MoGA as an end-to-end alternative that avoids these issues entirely.

---

Autoregressive Generation: Sequential Segment Synthesis with Limited Duration

A second paradigm generates videos autoregressively: produce a short segment, then condition on it to generate the next segment, repeating to extend duration. Notable examples include Diffusion Forcing (adapting denoising schedules for variable sequence lengths), StreamingT2V (combining short- and long-term memory for streaming extension), and MAGI-1 (demonstrating the scaling capability of this approach).

The paper acknowledges the promise of this direction but identifies two key limitations. First, these methods "produce videos of limited duration" — the autoregressive nature means errors propagate forward, and without global context, later segments can drift from earlier ones. Second, they "fail to generate multi-shot videos in real-world scenes" — the autoregressive conditioning mechanism is typically designed for continuous video, not for the abrupt transitions between different scenes that characterize multi-shot content.

This second point is particularly important. In a multi-shot video, the model must generate a completely new scene (different location, possibly different characters, different lighting) after a shot boundary, while still maintaining global consistency (the protagonist should look the same, the overall style should match). An autoregressive model conditioned on the previous segment will naturally try to continue what it was doing, which is the opposite of what a shot transition requires. This is an architectural mismatch that MoGA sidesteps by processing the entire video sequence jointly through sparse attention.

---

Context Compression: Inevitable Information Loss

A third paradigm compresses historical content to fit within a fixed computational budget. TTT uses a bidirectional recurrent layer to compress long context into a fixed-size representation. FramePack employs importance-based frame compression, selectively retaining the most informative frames while discarding others. The paper's critique is direct: this approach "inevitably results in information loss."

The compression is lossy by design — you cannot compress 384,000 tokens worth of video information into a small fixed-size representation without discarding some details. The question is which details are discarded and whether they matter. For maintaining character consistency across shots, a compression-based model might retain the general appearance of a character (their clothing color, approximate hairstyle) but lose fine-grained details (the specific pattern on their scarf, the exact shape of earrings). These fine-grained details are precisely what distinguishes consistent identity from approximate similarity, and their loss becomes increasingly visible as video duration increases.

The paper positions MoGA as a complementary approach to compression: rather than discarding information, it selectively routes attention so that the model processes all tokens but only computes attention for the most relevant pairs. This preserves information while reducing computation — a fundamentally different trade-off.

---

Sparse Attention: The Accuracy–Efficiency Trade-off in Block-Level Selection

The most directly relevant prior work, and the primary target of MoGA's innovation, is sparse attention for video generation. The paper distinguishes between two categories (Section 2.2):

Static (prior-driven) sparsity uses hand-designed attention patterns based on domain knowledge. STA employs 3D sliding windows with hardware-aware implementation. SVG classifies attention heads as spatial or temporal sparse attention using online pattern selection. Radial Attention introduces a static attention mask with $\mathcal{O}(n \log n)$ complexity. These methods are efficient because the attention pattern is fixed and requires no computation to determine, but the paper identifies a fundamental limitation:

"These methods have difficulty modeling evolving long-range dependencies, which are crucial for maintaining cross-shot consistency."

The problem is that static patterns are, by definition, not adaptive to content. A sliding window can only attend within a fixed temporal radius — if a character leaves the frame and returns 10 seconds later, a sliding window cannot connect the two appearances. Static sparsity sacrifices long-range interactions for efficiency, which is acceptable for short clips but breaks down for multi-shot content where the most important attention edges are precisely the long-range ones.

Coarse-to-fine dynamic selection attempts to address this by scoring which tokens are important and attending only to those. The dominant approach, exemplified by MoBA (Mixture of Block Attention) and its video-specific variant VMoBA, works in two stages:

1. Coarse stage: Partition the key-value sequence into blocks (contiguous groups of tokens, typically along the spatiotemporal dimensions), compute block-level importance scores (often by averaging token-level features within each block), and select the top-k blocks for each query.
2. Fine stage: Perform full attention within the selected blocks.

The paper's central critique of this approach is illustrated in Figure 1(b) and articulated clearly:

"Coarse-to-fine dynamic selection... introduces an efficiency-performance trade-off: using larger blocks with a small top-k reduces the computational cost of the coarse stage but reduces selection performance."

This trade-off is inescapable within the block-level framework. Here's why:

• Larger blocks mean fewer blocks to score and select from, reducing the computational cost of the coarse stage. But large blocks are coarse-grained: a block might contain tokens from multiple semantically distinct regions (a character's face and the background behind them), and the average importance score blurs these distinctions. The model ends up attending to entire blocks that are only partially relevant, wasting computation on irrelevant token pairs and potentially missing fine-grained correspondences.

• Smaller blocks provide more precise selection — you can pick out exactly the relevant spatiotemporal regions. But smaller blocks mean more total blocks, which means the coarse scoring stage becomes more expensive, eating into the efficiency gains that motivated sparsity in the first place.

Figure 1(b) illustrates the failure mode concretely: when block-level similarity is "confused" (i.e., the average score doesn't accurately reflect which individual tokens within the block are relevant), the resulting attention is unreliable. The block might be selected based on one highly relevant token even though the other 99 tokens in the block are irrelevant, or it might be rejected based on an average score that masks a few critically important tokens.

The paper also discusses SVG2, a training-free dynamic sparse attention method that performs online k-means clustering over tokens during inference and selects the top-k clusters based on centroids. This is conceptually closer to MoGA's grouping approach, but the paper identifies a critical distinction:

"Online clustering in SVG2 introduces additional k-means computations during the forward pass and is not straightforward to differentiate through. In contrast, MoGA employs trainable cluster centroids to enable simple and efficient routing with minimal computational overhead, making it suitable for end-to-end training."

This is a key insight: if you want the grouping to be optimized for the end task (video generation quality), the grouping mechanism must be differentiable and trained end-to-end. Online k-means during inference is a post-hoc heuristic — it groups tokens based on whatever similarity metric is used for clustering, not based on what grouping would minimize the video generation loss. MoGA's router, being a learned linear layer trained with the rest of the model, learns to group tokens in whatever way best serves the denoising objective.

---

The Deeper Architectural Insight: Attention Is Inherently Sparse

Beyond the computational argument, the paper makes a deeper claim about the structure of attention in videos that motivates MoGA's design:

"In videos, softmax attention is inherently sparse because nearby tokens exhibit strong local spatiotemporal correlation, while only a few globally shared, dynamic semantics persist across frames. Most query–key pairs contribute little, whereas a small subset dominates."

This claim, which the paper supports by citing prior work on attention sparsity (Ge et al., 2023; Xi et al., 2025), suggests that full attention is not just computationally wasteful but actually suboptimal for video generation. Computing attention over all token pairs introduces noise from irrelevant interactions — a background pixel attending to another background pixel from a different shot provides no useful signal but still contributes to the softmax normalization, potentially diluting the influence of genuinely important interactions.

The implication is profound: a well-designed sparse attention mechanism should not merely approximate full attention at lower cost, but could potentially outperform full attention by suppressing these irrelevant interactions. The paper provides evidence for this in the quantitative results: MoGA matches or surpasses the original Wan model (full attention) on multiple VBench metrics despite higher sparsity (Table 1), and in some qualitative cases "delivers superior performance" to full attention (Figure 10).

This reframes sparse attention from a necessary compromise (we wish we could use full attention but it's too expensive, so we approximate it) to a positive design choice (attention should be sparse because videos are structured, and enforcing the right sparsity pattern improves quality).

---

The Missing Piece: Precise Token-Level Routing Without Global Similarity Estimation

The paper's key insight, illustrated in Figure 1(c) and formalized in Section 3.2, is that block-level coarse estimation is unnecessary — tokens can be directly and precisely assigned to groups using a learned router, without ever computing global token-to-token similarity scores.

The intuition draws from Mixture-of-Experts (MoE), where a router assigns tokens to specialized expert networks. In MoGA, the router assigns tokens to groups, and attention is performed within each group. The router's weights effectively serve as learnable cluster centers — implicit prototypes that the router learns to recognize during training. A token is assigned to the group corresponding to whichever cluster center it most resembles (as measured by the learned linear projection followed by softmax).

This has several critical advantages over block-level selection:

1. Precision without granularity trade-off. Each token is individually routed based on its own features, not averaged within a block. This means the routing is as fine-grained as possible (per-token) without incurring the per-block scoring cost. The router is a single linear layer followed by softmax, costing $\mathcal{O}(M \cdot d)$ per token where $M$ is the number of groups and $d$ is the hidden dimension — negligible compared to the attention computation itself.

2. Semantic, not spatial, grouping. The router groups tokens based on learned semantic features, not spatiotemporal proximity. Tokens representing the same character's face across different shots and different time points can be routed to the same group, enabling long-range attention across shot boundaries. Figure 3 demonstrates this concretely: after training, the router assigns "the man's head, hands, and portions of his clothing to the same group" even though these tokens span multiple shots and are not spatiotemporally contiguous.

3. End-to-end differentiability. Unlike post-hoc clustering (SVG2), the router is trained alongside the rest of the model. The gradients from the denoising loss flow through the attention computation and into the router, teaching it to group tokens in whatever way best serves video generation quality. The auxiliary group balancing loss (Equation 6) prevents the router from collapsing to trivial solutions (routing all tokens to one group).

4. Kernel compatibility. Because MoGA reduces to standard self-attention within each group, it integrates seamlessly with optimized attention kernels like FlashAttention. Algorithm 1 shows the elegant implementation: group tokens by their routing assignments, permute the QKV tensors so tokens in the same group are contiguous, call FlashAttention on each contiguous block, and unpermute the outputs. No custom CUDA kernels, no block-sparse matrix multiplications, no mask-based attention implementations.

---

The Multi-Shot Challenge: Single Global Prompts Are Insufficient

The paper identifies a second, complementary challenge that is specific to multi-shot video generation. Even with perfect long-range attention, a single global text prompt cannot reliably control scene transitions or orchestrate events at precise time points in a long video. The paper articulates this clearly:

"Extended context alone is insufficient because a single global prompt cannot reliably control scene transitions or orchestrate events at precise time points in long videos."

Consider a one-minute video with 10 shots: the first shot might be a character in a kitchen, the second shot in a garden, the third in a car. A single prompt ("A woman goes about her day, moving through different locations") provides no mechanism to tell the model when to switch scenes, what each scene should look like, or which characters appear in which shot. The model might blend scenes together, produce transitions at arbitrary times, or lose track of which character is where.

The paper addresses this through shot-level textual conditioning via cross-modal attention, where each shot is guided by a concise description. This requires a data pipeline that produces minute-level video samples with dense, multi-shot captions and reliable shot segmentation — which the paper constructs (Section 3.3) but positions as supporting infrastructure rather than a core contribution.

Importantly, this shot-level conditioning interacts with the attention mechanism: the STGA component (spatial-temporal group attention) explicitly prevents inter-shot attention in the local window, relying on MoGA to handle cross-shot information flow through its semantic groups. The shot boundaries are therefore handled by a combination of architectural design (no local cross-shot attention) and conditioning (explicit shot-level text).

---

How MoGA Positions Itself Relative to Existing Work

The paper positions MoGA at the intersection of two research threads — sparse attention and multi-shot video generation — while claiming to resolve a fundamental tension in the former and enable a new capability in the latter.

Relative to sparse attention methods (MoBA, VMoBA, SVG, STA): MoGA is presented not as another point on the accuracy–efficiency Pareto frontier, but as a method that sidesteps the trade-off entirely by eliminating block-level estimation. The claim is not "our block selection is better" but "block selection is unnecessary — we route tokens directly." This is a categorical difference in mechanism, not an incremental improvement.

Relative to multi-stage pipelines (Captain Cinema, VideoTetris): MoGA offers end-to-end optimization with a single objective, eliminating the error accumulation and hand-crafted inductive biases that the paper critiques in multi-stage approaches. The quantitative comparison in Table 3 (30-second multi-shot generation) shows MoGA substantially outperforming IC-LoRA+Wan (a keyframe-based pipeline) under the same backbone, which the paper interprets as evidence for the benefits of end-to-end modeling.

Relative to autoregressive methods (StreamingT2V, MAGI-1): MoGA processes the entire video sequence jointly rather than sequentially, which the paper argues is better suited to multi-shot content where abrupt scene transitions are required. The autoregressive conditioning mechanism, designed for continuous video, doesn't naturally accommodate the discontinuity of shot boundaries.

Relative to context compression (TTT, FramePack): MoGA avoids information loss by attending to all tokens (not compressing historical content) while maintaining efficiency through selective attention computation. The paper doesn't directly compare against compression methods quantitatively, but the conceptual distinction is clear: MoGA preserves information and reduces computation through routing, while compression reduces both information and computation through summarization.

Relative to full attention (Wan, EchoShot): The paper makes the somewhat provocative claim that MoGA can outperform full attention, not just approximate it. The rationale is that full attention introduces noise from irrelevant token pairs, and MoGA's grouping suppresses this noise by restricting attention to semantically coherent token subsets. The evidence in Tables 1 and 2 supports this claim, with MoGA matching or exceeding full-attention baselines on most VBench metrics despite substantially lower FLOPs.

The data contribution. While the paper's primary contribution is the MoGA attention mechanism, it also addresses a practical barrier to long video generation: the lack of high-quality multi-shot video training data. The data pipeline (Section 3.3, Figure 4) converts raw long videos into one-minute, multi-shot clips with dense annotations and clean shot boundaries. This is positioned as enabling infrastructure — without it, the architectural innovations would lack the training data to realize their potential — but the paper is clear that the core intellectual contribution is the attention mechanism, not the data pipeline per se.

In summary, MoGA is motivated by a concrete computational barrier (quadratic self-attention in DiTs for long video sequences), informed by a structural insight about video (attention is inherently sparse; most query-key pairs are uninformative), and addresses a specific failure mode in the dominant sparse attention paradigm (the accuracy–efficiency trade-off of block-level estimation). The paper's position is that learned, per-token group assignment via a lightweight router is both simpler and more effective than block-level selection, and that this mechanism — combined with local spatiotemporal attention and shot-level conditioning — enables minute-scale, multi-shot video generation that was previously infeasible with end-to-end DiT architectures.

3. Technical Approach

3.1 Reader Orientation

MoGA is a sparse attention mechanism for Diffusion Transformers (DiTs) that replaces the quadratic-cost full self-attention of long video sequences with a learned, group-based routing system. It solves the problem that generating minute-long, multi-shot videos requires computing attention over ~580k tokens, which is computationally intractable with standard attention — by assigning each token to one of several semantic groups using a lightweight trained router, then performing full attention only within each group, reducing FLOPs by over 3× while preserving the long-range cross-shot interactions that maintain character identity and scene consistency.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five intertwined components, spanning both the model architecture and the data pipeline that enables training:

1. Base DiT Model — a pretrained Diffusion Transformer for short video generation (e.g., Wan2.1 at 1.3B or 14B parameters, or an MMDiT-based architecture) that serves as the starting point for fine-tuning. It operates on latent video representations produced by a VAE, denoising them via the rectified flow objective.

2. Mixture-of-Groups Attention (MoGA) — the core innovation. A lightweight token router (a single linear layer followed by softmax gating, producing a probability distribution over $M$ groups) assigns every token in the sequence to exactly one group based on its learned semantic features. Full self-attention is then computed independently within each group, reducing complexity from $\mathcal{O}(N^2)$ to approximately $\mathcal{O}(N^2 / M)$ under uniform assignment. The router is trained end-to-end with the denoising objective, learning to group tokens that need to attend to each other (e.g., tokens representing the same character across different shots) without any block-level similarity pre-computation.

3. Spatial-Temporal Group Attention (STGA) — a complementary local attention mechanism that operates alongside MoGA. It partitions the latent video into fixed spatial windows and groups frames along the temporal axis, restricting self-attention to these local neighborhoods. This captures short-range spatial and temporal dependencies that MoGA's semantic groups may miss. STGA uses static, predefined groups — unlike MoGA's learned routing, these groups are determined by spatiotemporal proximity. Notably, frames from different shots are assigned to distinct temporal groups, preventing inter-shot attention in the local windows (cross-shot information flows exclusively through MoGA).

4. Cross-Modal Attention Blocks — interleaved with visual attention blocks, these enable shot-level textual conditioning. Each shot is guided by a concise text description, and cross-modal attention (instantiated as either cross-attention or MMDiT-style multi-modal attention) injects this conditioning into the visual features. This addresses the limitation that a single global text prompt cannot reliably control scene transitions or orchestrate events at precise time points in long videos.

5. Multi-Shot Data Pipeline — a two-stage preprocessing system (video-level and shot-level) that converts raw long videos into one-minute, multi-shot training clips with dense captions and clean shot boundaries. This is not a model component but is essential infrastructure: without it, the architectural innovations lack the training data to learn multi-shot consistency.

Information flow at inference time: A sequence of shot-level text descriptions enters the model → each text token conditions visual features via cross-modal attention → the latent video representation (a 3D grid of tokens across space and time) passes through alternating visual and cross-modal attention blocks → within each visual attention block, tokens are processed by STGA (local spatiotemporal windows) and MoGA (learned semantic groups) in parallel → the outputs of both attention mechanisms are averaged → the denoised latent is decoded by the VAE into a video.

3.3 Roadmap for the Deep Dive

• First, the formal definition of vanilla self-attention and its quadratic complexity in video DiTs, establishing the computational baseline that MoGA must improve upon.
• Second, the MoGA router mechanism — how tokens are assigned to groups, the mathematical formulation of routing, and how grouped attention reduces complexity — since this is the core contribution.
• Third, the MoGA pseudocode and kernel integration (FlashAttention compatibility, sequence parallelism), because practical efficiency depends on implementation details as much as algorithmic complexity.
• Fourth, the group balancing loss — why it's necessary, what failure mode it prevents, and its mathematical form — since without it the router collapses and MoGA degenerates.
• Fifth, the Spatial-Temporal Group Attention (STGA) mechanism, its relationship to MoGA, and the architectural decisions that enable them to complement each other.
• Sixth, the overall DiT architecture and cross-modal conditioning, showing how MoGA fits into a complete video generation system.
• Seventh, the data pipeline, since the architectural innovations require specific training data characteristics that did not previously exist at scale.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a method paper whose core idea is that block-level coarse estimation for sparse attention is an unnecessary constraint, and that a learned, per-token group assignment mechanism can achieve both higher efficiency and better video generation quality by forming semantically coherent attention groups directly.

---

Vanilla Self-Attention and Its Computational Barrier in Video DiTs

The paper builds on Diffusion Transformers (DiTs), where the denoising network is a Transformer that operates on latent video representations. The input to the attention layers is a sequence of tokens $\boldsymbol{X} \in \mathbb{R}^{N \times d}$, where $N = h \times w \times t$ is the total number of tokens across the latent spatial dimensions ($h \times w$) and the latent temporal dimension ($t$), and $d$ is the model's hidden dimension.

For context, the paper provides a concrete scale example following the Wan architecture: the VAE downsampling factors for (t, h, w) are (4, 8, 8) and the patchify sizes are (1, 2, 2). For a 1-minute video at 480p with approximately 1,600 tokens per frame across 961 frames (at 16 fps), this yields $N \approx 1,600 \times 961 = 384,000$ tokens. (The paper's final model actually operates at 24 fps with 1,441 frames and a context length of 578k tokens, but the 384k figure establishes the baseline scale.)

For a single query token $\boldsymbol{x}$ with query vector $\boldsymbol{q}$, vanilla self-attention computes:

$\operatorname{SA}(\boldsymbol{q}, \boldsymbol{K}, \boldsymbol{V}) = \text{softmax}\left(\frac{\boldsymbol{q} \boldsymbol{K}^{\top}}{\sqrt{d}}\right) \cdot \boldsymbol{V}$

where $\boldsymbol{K} \in \mathbb{R}^{N \times d}$ is the key matrix (all tokens' keys), $\boldsymbol{V} \in \mathbb{R}^{N \times d}$ is the value matrix (all tokens' values), and the softmax is taken over the $N$ key positions.

What it computes: For a single query token, this produces a weighted sum of all $N$ value vectors, where the weights are determined by the scaled dot-product similarity between the query and each key. The $\sqrt{d}$ scaling prevents the dot products from growing too large in magnitude as the dimensionality increases, which would push the softmax into a near-one-hot regime. The output is a single vector of dimension $d$ that aggregates information from every token in the sequence.

Why this form: The dot-product attention mechanism allows every token to directly interact with every other token, regardless of their positions in the sequence. This is what gives Transformers their signature ability to model long-range dependencies — a token at frame 1 can attend to a token at frame 961 with the same mechanism as a token at frame 480 attending to frame 481. However, this universality comes at the cost of $\mathcal{O}(N^2)$ complexity: for each of the $N$ queries, we compute $N$ dot products with keys, plus the weighted sum over $N$ values, yielding roughly $2N^2 d$ operations per attention layer.

For $N = 384,000$ and a typical $d$ of a few thousand, this translates to trillions of operations per attention layer, per diffusion timestep. The paper quantifies this in Appendix Table 5: for the Wan2.1-1.3B model generating a 30-second video, full attention costs 6.94 PFLOPs. This is not merely slow — it is prohibitive for training and inference at scale.

Beyond the computational cost, the paper makes a structural argument: full attention is not ideally aligned with video structure. In video data, softmax attention is inherently sparse because:

• Nearby tokens (within the same spatial region across consecutive frames) exhibit strong local spatiotemporal correlation — a pixel at position (x, y) in frame t is almost certainly relevant to the pixel at (x, y) in frame t+1.
• Only a small number of globally shared, dynamic semantics persist across long temporal distances — the appearance of a character's face, the texture of a consistently appearing object, the overall lighting and color palette.
• Most query–key pairs contribute negligible attention weights, while a small subset dominates the softmax distribution.

The paper cites prior work (Ge et al., 2023; Xi et al., 2025) supporting this sparsity claim. The implication is that computing and storing all $N^2$ attention weights is not only expensive but wasteful — the model structure should reflect the data structure.

---

MoGA Router: Learned Per-Token Group Assignment

The central mechanism of MoGA is a token router that assigns each token in the sequence to one of $M$ groups, where $M$ is a predetermined hyperparameter. The router is a single linear projection followed by softmax gating, directly inspired by the routing mechanism in Mixture-of-Experts (MoE) architectures.

Formally, given a token $\boldsymbol{x} \in \mathbb{R}^d$ and a number of groups $M$, the router computes routing scores $\boldsymbol{r} \in \mathbb{R}^M$ as:

$\boldsymbol{r} = \operatorname{Router}(\boldsymbol{x})$

where $\operatorname{Router}$ is a learned linear transformation — effectively a matrix $\boldsymbol{W}_r \in \mathbb{R}^{d \times M}$ (plus an optional bias) that maps the token's $d$-dimensional representation to $M$ unnormalized scores.

What it computes: For a single token, the router produces $M$ raw scores, one per group. These scores represent the unnormalized affinity between the token and each of the $M$ learnable "cluster centers" implicitly encoded in the router's weight matrix. Intuitively, each column of $\boldsymbol{W}_r$ can be thought of as a prototype vector in the token representation space, and the dot product $\boldsymbol{x} \cdot \boldsymbol{W}_r[:, i]$ measures how well the token matches prototype $i$.

The group assignment probabilities are then computed via softmax:

$p(i \mid \boldsymbol{x}) = \operatorname{softmax}(\boldsymbol{r})_i$

where $p(i \mid \boldsymbol{x})$ is the probability that token $\boldsymbol{x}$ is assigned to group $i$. The softmax ensures that these probabilities sum to 1 across the $M$ groups and that they are strictly positive, providing a differentiable assignment mechanism.

The actual (hard) group assignment takes the argmax:

$g(\boldsymbol{x}) = \arg\max_{i \in [M]} \; p(i \mid \boldsymbol{x})$

where $g(\boldsymbol{x}) \in \{1, \ldots, M\}$ is the integer index of the group to which the token is assigned. This is a hard decision — each token goes to exactly one group — which is necessary for the subsequent grouped attention computation. The argmax is not differentiable, but this is handled by using the soft probabilities in the auxiliary loss (described below) and by using straight-through estimation or simply letting gradients flow through the attention computation given the hard assignment.

Why this form: The router design has several important properties that make it well-suited for this task:

1. Per-token precision without global similarity estimation. Unlike block-level methods that must first compute average similarity scores for entire blocks of tokens, the router makes an independent decision for each token based solely on that token's own features. The cost is $\mathcal{O}(M \cdot d)$ per token, which is negligible compared to the attention computation itself (since $M$ is small, typically 5–20, while $d$ is in the thousands). This eliminates the accuracy–efficiency trade-off of block size: there are no blocks, so there is no block size parameter to tune.

2. Learned, semantic grouping. Because the router's weight matrix is trained end-to-end with the denoising objective, it learns to group tokens based on features that are useful for video generation, not based on hand-crafted similarity metrics. The paper provides a compelling visualization in Figure 3: after training, the router assigns "the man's head, hands, and portions of his clothing to the same group," demonstrating that it has learned to identify semantically coherent entities (body parts of the same person) that span shot boundaries. This is the key property that enables long-range cross-shot consistency — tokens representing the same character in different shots get routed to the same group and can therefore attend to each other.

3. Differentiability and end-to-end training. The softmax probabilities provide a differentiable signal for training. When combined with the auxiliary group balancing loss (Equation 6), the router receives gradients that encourage both accurate grouping (through the denoising loss, which flows through the attention computation) and balanced grouping (through the auxiliary loss). This is a critical distinction from post-hoc clustering methods like SVG2's online k-means, which cannot be optimized for the end task.

4. Simplicity and minimal overhead. The router is a single linear layer — the simplest possible learnable transformation. There are no attention computations, no similarity matrices, no iterative refinement. This keeps the computational overhead minimal and makes the mechanism easy to implement and integrate with existing attention kernels.

---

Grouped Attention: Computing Self-Attention Within Assigned Groups

Once tokens are assigned to groups, MoGA performs standard self-attention independently within each group. For a token $\boldsymbol{x}$ assigned to group $g(\boldsymbol{x})$, the MoGA output is:

$\operatorname{MoGA}(\boldsymbol{x}) = p(g(\boldsymbol{x}) \mid \boldsymbol{x}) \cdot \operatorname{SA}(\boldsymbol{q}, \boldsymbol{K}_{g(\boldsymbol{x})}, \boldsymbol{V}_{g(\boldsymbol{x})})$

where $\boldsymbol{K}_{g(\boldsymbol{x})}$ and $\boldsymbol{V}_{g(\boldsymbol{x})}$ are the key and value matrices restricted to tokens in group $g(\boldsymbol{x})$, $\boldsymbol{q}$ is the query vector of $\boldsymbol{x}$, and $\operatorname{SA}$ is the standard self-attention operation defined in Equation 1.

What it computes: For a token assigned to group $g$, the output is the standard attention-weighted sum of the value vectors of all tokens in group $g$, multiplied by the router's confidence $p(g \mid \boldsymbol{x})$ in that assignment. The multiplication by $p(g \mid \boldsymbol{x})$ is a form of soft gating — if the router is uncertain about the assignment (i.e., the token lies near the boundary between two groups), the attention output is downweighted, giving less influence to tokens that might not be the best match. If the router is confident, $p(g \mid \boldsymbol{x})$ is close to 1 and the attention output is used at full strength.

Why this form: The multiplication by the routing probability serves two purposes:

1. Gradient flow. Since the argmax in the hard assignment is not differentiable, multiplying by the softmax probability $p(g \mid \boldsymbol{x})$ (which is differentiable) provides a path for gradients to flow from the attention output back to the router parameters. If the assignment was good (the attention output was useful for denoising), the gradient increases $p(g \mid \boldsymbol{x})$ for that group. If the assignment was poor, the gradient decreases it. This is similar to how MoE models use the gating probabilities to provide a differentiable training signal despite hard expert selection.

2. Uncertainty-aware aggregation. When a token genuinely belongs to multiple semantic groups (e.g., a token representing a character's hand that is touching a background object), the router may produce a split probability distribution. Multiplying by $p(g \mid \boldsymbol{x})$ naturally downweights the contribution from any single group, reflecting the token's ambiguous membership. This is a more nuanced treatment than hard assignment alone.

The key computational property of this grouped attention is its complexity reduction. Under the assumption of uniform group assignment (each group receives approximately $N/M$ tokens), the attention cost per group is $\mathcal{O}((N/M)^2)$, and since there are $M$ groups, the total cost is:

$\mathcal{O}(M \cdot (N/M)^2) = \mathcal{O}(N^2 / M)$

This is a factor of $M$ reduction compared to full attention's $\mathcal{O}(N^2)$. For $M = 5$ (the setting used with Wan2.1), this gives a theoretical 5× reduction. The actual reduction is somewhat less due to non-uniform assignment and the overhead of the router itself, but the paper reports a 3× reduction in practice (2.26 PFLOPs vs. 6.94 PFLOPs for 30-second videos) and a 1.7× wall-clock speedup in both training and inference.

A crucial implementation detail is that MoGA does not introduce additional memory overhead. The paper explicitly states that "unlike alternative sparse attention such as VMoBA, which incur additional memory overhead due to their block-based mechanisms, our approach maintains memory efficiency without additional memory consumption." This is because MoGA only reorganizes existing tensors (via permutation) rather than creating additional data structures like block indices or importance score caches.

---

MoGA Pseudocode and Kernel Integration

The paper provides Algorithm 1, which describes the practical implementation of MoGA using FlashAttention. The algorithm consists of four steps:

Step 1: Compute routing assignments. Given the input tokens $\boldsymbol{X} \in \mathbb{R}^{N \times d}$, compute $\boldsymbol{g} = \text{router}(\boldsymbol{X})$, where $\boldsymbol{g} \in \{1, \ldots, M\}^N$ is the vector of hard group assignments.

Step 2: Permute tokens by group. The QKV tensors are rearranged so that all tokens belonging to group 1 come first, followed by all tokens in group 2, and so on. This produces $\hat{\boldsymbol{Q}}, \hat{\boldsymbol{K}}, \hat{\boldsymbol{V}}$ and metadata cu_seqlen (cumulative sequence lengths marking group boundaries), max_seqlen (the size of the largest group), and permute_index (the mapping from original to permuted positions, needed for unpermuting later).

Step 3: Apply FlashAttention per group. The permuted tensors are passed to flash_attn, which processes each contiguous block (corresponding to one group) using the cu_seqlen metadata to identify group boundaries. FlashAttention computes exact attention within each group using its tiled, IO-aware algorithm — the same computation as standard attention but optimized for GPU memory hierarchy.

Step 4: Unpermute outputs. The attention outputs are rearranged back to the original token order using permute_index, producing the final output $\boldsymbol{O}$.

What this algorithm accomplishes: It efficiently implements grouped attention without writing any custom CUDA kernels. The key insight is that grouping is reduced to a permutation operation, and attention is reduced to standard FlashAttention on variable-length sequences. The cu_seqlen / max_seqlen interface — which FlashAttention already supports for handling sequences of different lengths within a batch — is repurposed to handle groups of different sizes.

Why this form: This design has two major practical advantages:

1. Kernel compatibility. By reducing MoGA to standard attention on permuted sequences, the method inherits all the optimizations in FlashAttention (tiled computation, recomputation of attention weights in backward pass, optimized memory access patterns) without modification. There is no need for custom sparse attention kernels that may be hardware-specific or difficult to maintain across GPU generations.

2. Sequence parallelism compatibility. The paper explicitly notes that MoGA is compatible with sequence parallelism (DeepSpeed Ulysses). In a sequence-parallel setting, tokens are distributed across multiple GPUs along the sequence dimension. MoGA computes routing scores over tokens on each GPU (with all attention heads), then aggregates routing results across GPUs before the sequence gather and head scatter operations. This is described as a natural integration rather than a workaround, since the routing computation is lightweight and the aggregation is a simple all-to-all communication of group assignments.

The output of each visual attention block is the mean of the MoGA output and the STGA output (described below), providing both long-range semantic attention (MoGA) and local spatiotemporal attention (STGA) in parallel.

---

Group Balancing Loss: Preventing Router Collapse

A known failure mode of learned routing mechanisms (well-documented in the MoE literature) is router collapse: the router learns to send all tokens to one or a few groups, which minimizes the immediate denoising loss (because all tokens can attend to all other tokens, effectively reverting to full attention) but eliminates the efficiency benefits of sparsity. The paper addresses this with an auxiliary group balancing loss inspired by the load balancing loss used in Switch Transformers and other MoE architectures.

The loss is defined as:

$\mathcal{L}_{\text{gb}} = \alpha \cdot M \sum_{i=1}^{M} F_i P_i$

where $\alpha$ is a hyperparameter controlling the loss weight (set to $0.1$ in experiments), $M$ is the number of groups, $F_i$ is the fraction of tokens assigned to group $i$, and $P_i$ is the mean routing probability allocated for group $i$.

$F_i$ is computed as:

$F_i = \frac{1}{N} \sum_{\boldsymbol{x}} \mathbf{1}(g(\boldsymbol{x}) = i)$

where $\mathbf{1}$ is the indicator function and the sum counts how many of the $N$ tokens are assigned to group $i$. $F_i$ is simply the empirical frequency of group $i$ in the current batch.

$P_i$ is computed as:

$P_i = \frac{1}{N} \sum_{g(\boldsymbol{x}) = i} p(g(\boldsymbol{x}) \mid \boldsymbol{x})$

where the sum runs only over tokens that were assigned to group $i$, and $p(g(\boldsymbol{x}) \mid \boldsymbol{x})$ is the softmax probability from Equation 3. $P_i$ is the average confidence the router has in its assignments to group $i$, averaged only over tokens actually assigned to that group.

What it computes: $\mathcal{L}_{\text{gb}}$ multiplies, for each group, the fraction of tokens assigned ($F_i$) by the average routing probability for those assignments ($P_i$), sums across groups, and scales by $\alpha M$. The product $F_i P_i$ is small when either few tokens are assigned to group $i$ ($F_i$ is small) or the router has low confidence in those assignments ($P_i$ is small). The sum $\sum_i F_i P_i$ is minimized under a uniform distribution — when $F_i = 1/M$ for all groups and the router is maximally confident — which yields a value of $1/M$ for the sum and thus a total loss of $\alpha \cdot M \cdot 1/M = \alpha$ after scaling.

Why this form: The design follows directly from the MoE load balancing loss introduced by Fedus et al. (2022) and has several desirable properties:

1. Encourages uniform assignment. The product $F_i P_i$ penalizes both under-utilization (small $F_i$) and uncertainty (small $P_i$). A group that receives no tokens ($F_i = 0$) contributes nothing to the loss but also contributes nothing to the model's capacity — the penalty comes indirectly because other groups must have $F_j > 1/M$, increasing their $F_j P_j$ terms. The overall minimum is achieved when all $F_i = 1/M$.

2. Does not force exact uniformity. The loss is a soft penalty, not a hard constraint. If the denoising objective strongly favors sending more tokens to certain groups (because certain semantic clusters are larger), the router can deviate from uniformity, paying a penalty proportional to $\alpha$. The hyperparameter $\alpha$ controls this trade-off: larger $\alpha$ enforces stricter balancing at the potential cost of denoising quality; smaller $\alpha$ allows more flexibility but risks collapse. The paper's chosen value of $0.1$ represents an empirical balance point.

3. Differentiable. Both $F_i$ and $P_i$ are differentiable with respect to the router parameters (through the softmax probabilities), so $\mathcal{L}_{\text{gb}}$ provides a direct training signal that can be backpropagated alongside the denoising loss.

The paper validates the necessity of this loss in Appendix Figure 12. Without $\mathcal{L}_{\text{gb}}$, the balancing metric (the same sum $\sum_i F_i P_i$ monitored during training) rapidly increases as the router funnels tokens into a few groups to gain short-term advantages in the diffusion MSE loss. With $\mathcal{L}_{\text{gb}}$, the metric quickly converges to near 1, reflecting balanced assignments across groups. The paper states: "Because our goal is to separate weakly related tokens and maintain balanced grouping, the additional group balance loss is necessary to enforce the desired assignments."

---

Spatial-Temporal Group Attention (STGA): Complementary Local Attention

While MoGA captures long-range coherence through semantic grouping, it lacks an explicit mechanism for local spatiotemporal continuity — nearby tokens in space and time naturally need to attend to each other to model motion, textures, and fine-grained details. The paper complements MoGA with Spatial-Temporal Group Attention (STGA) , a static (non-learned) local attention mechanism.

STGA partitions the latent video representation into fixed groups along two dimensions:

1. Spatial partitioning. The latent spatial grid is divided into fixed windows. For the Wan2.1-based model, the spatial grid is partitioned into $2 \times 2$ groups. For the MMDiT-based model, the finer $4 \times 4$ partitioning is used (to accommodate the longer context length).

2. Temporal partitioning. Frames are grouped along the temporal axis, with a critical design decision: frames from different shots are assigned to distinct temporal groups. This means that within STGA's local windows, there is no attention across shot boundaries — cross-shot information is handled exclusively by MoGA. The paper states this explicitly: "Frames from different shots are assigned to distinct temporal groups."

The paper identifies a subtle issue with this strict shot separation: "We empirically find that completely removing inter-shot interactions causes flicker in the first frame after a shot cut." To address this, when computing STGA within a shot's temporal group, the keys and values are augmented with two latent frames from adjacent shots — but importantly, the queries are not augmented. This means tokens within a shot can attend to the boundary frames of neighboring shots (providing continuity at transitions), but tokens in neighboring shots cannot attend into the current shot (preserving the one-way information flow that prevents shot blending).

Additionally, STGA includes per-frame attention by grouping tokens within each latent frame, enabling intra-frame spatial information exchange that neither MoGA nor the shot-based temporal groups provide.

What this produces: Each token receives outputs from three group attention computations:

1. Its MoGA group (learned semantic group, providing long-range cross-shot interactions).
2. Its STGA spatial-temporal group (providing local motion and texture information within its shot).
3. Its per-frame group (providing intra-frame spatial context).

The final visual attention output for each token is the mean of these outputs. The paper does not use learned weights for this aggregation — it is a simple arithmetic mean, which the experiments show is sufficient.

Why this form: The separation of responsibilities between MoGA and STGA is a deliberate architectural choice:

• MoGA handles what is semantically coherent but spatially distributed — character identity across shots, background consistency, global style. It groups tokens by what they are (a person's face, a specific object), not where they are.
• STGA handles what is spatially and temporally local — motion continuity, texture consistency, fine-grained spatial details. It groups tokens by where they are in the video grid.
• The shot boundary treatment in STGA ensures that local attention doesn't inadvertently blend different scenes together (which would cause ghosting or morphing artifacts), while MoGA's semantic groups provide the only path for cross-shot consistency (preventing identity drift or background shifts).

This complementary design is validated in Figure 11: "Using MoGA alone lacks local information exchange and fails to produce meaningful visual content. Conversely, using only STGA limits long-range shot interactions, leading to poor cross-shot consistency and weakened narrative coherence. When combined, the model achieves strong cross-shot consistency."

---

Overall DiT Architecture and Cross-Modal Conditioning

MoGA is integrated into a complete DiT-based video generation system, illustrated in Figure 2. The architecture follows the standard DiT pattern with two types of blocks interleaved:

Visual Attention blocks process the latent video representation exclusively. Each block contains both MoGA and STGA as described above, with their outputs averaged. These blocks handle all within-modality (visual-to-visual) information flow.

Cross-Modal Attention blocks inject text conditioning into the visual features. The paper supports two instantiations:

1. Cross-attention (as used in Wan2.1): Text tokens from the shot descriptions serve as keys and values, and visual tokens serve as queries. This allows visual features to selectively attend to relevant textual information.
2. MMDiT-style multi-modal attention (as used in the MMDiT-based model): Text and visual tokens are concatenated and processed jointly through self-attention, following the approach of models like HunyuanVideo and SD3.

The critical difference from standard short-video DiTs is that the text conditioning is shot-level rather than global. Each shot in the multi-shot video has its own concise text description, and the cross-modal attention ensures that visual tokens within a given shot attend to their corresponding text description. This is what enables the model to handle scene transitions: shot 1's visual tokens attend to "A woman in a kitchen, morning light," while shot 2's visual tokens attend to "The same woman in a garden, afternoon sun."

The paper states that this shot-level conditioning is necessary because "extended context alone is insufficient because a single global prompt cannot reliably control scene transitions or orchestrate events at precise time points in long videos." Without it, the model would have no mechanism to know when to switch scenes or what the new scene should contain — the visual features alone do not carry this information.

The training objective is the rectified flow objective — a standard choice for flow-matching-based diffusion models. The model is fine-tuned (not trained from scratch) on existing DiT-based short video generation checkpoints, specifically the open-source Wan2.1 models at 1.3B and 14B parameter scales, and an MMDiT-based model for the longest context experiments.

Training hyperparameters (verbatim from Section 4):

• Learning rate: constant $1 \times 10^{-5}$
• Loss weight $\alpha$ for group balancing: $0.1$
• Number of MoGA groups: $M = 5$ for Wan2.1-based model; $M = 20$ for MMDiT-based model
• STGA spatial partition: $2 \times 2$ for Wan2.1; $4 \times 4$ for MMDiT
• Multistage training: 3,000 steps on 10-second clips, followed by 1,000 steps on 30-second clips (for Wan2.1-1.3B)
• The MMDiT-based model generates 1,441 frames at 24 fps (60 seconds) at 480p, with a context length of 578k tokens

The Wan2.1-1.3B model stably generates 477 frames at 16 fps (30 seconds) and 480p resolution, with a context length of 187k tokens. The larger context in the MMDiT-based model is enabled by the increased number of groups ($M = 20$ vs. $M = 5$), which provides a greater theoretical efficiency factor.

---

Data Pipeline: Creating Multi-Shot Long Video Training Data

The architectural innovations described above require training data with specific properties: minute-long videos with multiple shots, clean shot boundaries, and dense per-shot captions. Such data does not exist in standard video datasets (which typically contain short, single-shot clips with global captions), so the paper constructs a two-stage pipeline to produce it from raw long videos.

Video-Level Stage (first pass over raw videos):

1. Quality assessment. Each raw video is analyzed using visual quality assessment (VQA) models that evaluate aesthetics (e.g., using LAION aesthetic predictors), clarity, and exposure. Simple operators also detect black borders.

2. Filtering. Videos are filtered using source-specific, calibrated thresholds to remove low-quality content. The paper makes an important distinction between clip-level and source-level filtering: "Because long video samples require temporal coherence, we relax clip-level filtering while applying stricter filtering at the source (raw-video) level." The rationale is that individual clips might score poorly on certain quality metrics (e.g., a deliberately dark scene might fail exposure thresholds) but be perfectly valid within the context of the longer video — strict clip-level filtering would discard these. Source-level filtering ensures that only generally high-quality video sources are used.

3. Shot segmentation. Each filtered video is segmented into single-shot clips using two complementary tools: AutoShot (Zhu et al., 2023) and PySceneDetect. AutoShot shows higher sensitivity to fades and gradual transitions, while PySceneDetect is more reliable for hard cuts. By combining predictions from both tools, the pipeline labels each boundary as either "clean" (agreed upon by both tools) or "affected by transition overlap" (detected by one but not the other). This metadata is used later to ensure clean shot boundaries in training samples.

This stage produces a pool of single-shot clips with quality metadata, shot boundary labels, and information about transition types.

Shot-Level Stage (processing individual clips):

1. Quality filtering per clip. Single-shot clips are analyzed using VQA and optical character recognition (OCR) models. Low-quality clips (poor aesthetics, excessive blur, etc.) are discarded.

2. Watermark and subtitle removal. Based on OCR results, the pipeline computes a maximum-area crop that excludes detected watermarks and subtitles while preserving the original aspect ratio. Clips where the retained area after cropping is insufficient (too much of the frame is occupied by text or watermarks) are discarded entirely.

3. Caption generation. Cropped clips are captioned using a multimodal large language model (specifically Qwen2.5-VL, cited as Bai et al., 2025). Each clip receives a concise description of its content, which will serve as the shot-level text conditioning during training.

4. Multi-shot assembly. Temporally adjacent single-shot clips (from the same original video) are merged into multi-shot training samples of up to 65 seconds. Critically, a few frames are trimmed from clips affected by transition overlap (as identified in the video-level stage) to ensure clean shot boundaries. This means the training data contains hard cuts between shots with no transition effects, which is what the model learns to generate.

What this pipeline produces: Minute-long video samples where each shot has a clean boundary, a high-quality caption, and all frames are free of watermarks and subtitles. The pipeline is not claimed as a core contribution but rather as necessary infrastructure — without it, MoGA's shot-level conditioning and cross-shot consistency mechanisms would lack appropriate training data.

Why this design: Several choices reflect practical constraints and empirical findings:

• Two-stage processing (video-level then shot-level) is more efficient than processing every frame of a long video at full resolution. The video-level stage quickly identifies promising raw videos; the shot-level stage does expensive VQA and captioning only on the segments that pass initial filtering.
• Dual shot detection (AutoShot + PySceneDetect) provides robustness that neither tool alone achieves. AutoShot's sensitivity to gradual transitions is important for identifying "soft" boundaries that a simpler detector would miss, while PySceneDetect's reliability on hard cuts anchors the segmentation.
• Trimming transition frames is a practical simplification. Real video transitions (dissolves, wipes, fades) are complex to model and would require the model to learn transition synthesis. By trimming these frames, the training data contains only hard cuts, which simplifies the learning problem and matches the target use case (generating videos with clear shot boundaries, not transition effects).
• Captioning with MLLMs (rather than using existing metadata or CLIP-based methods) ensures that captions are dense and descriptive, capturing details relevant to video generation (character appearance, actions, setting, lighting, camera angle) rather than just high-level topics.

4. Key Insights and Innovations

Innovation 1: Block-Level Coarse Estimation Is an Unnecessary Constraint in Sparse Attention — Token-Level Routing Is Both Simpler and More Effective

The dominant paradigm in dynamic sparse attention for video generation — exemplified by MoBA, VMoBA, and related methods — rests on a specific assumption: that to efficiently select which query-key pairs to compute, you must first aggregate tokens into blocks, estimate block-level importance scores, and then select the top-k blocks for fine-grained attention. This assumption was so deeply embedded that the research conversation had focused entirely on navigating the resulting accuracy–efficiency trade-off (larger blocks reduce coarse-stage cost but blur selection; smaller blocks sharpen selection but increase overhead). The paper's first and most fundamental conceptual move is to reject the premise that block-level estimation is necessary at all.

MoGA's router is not an improved block scorer — it is a categorical departure from the block-based framework. Instead of estimating similarity between queries and key blocks (which requires computing and comparing scores, even if at a coarse granularity), MoGA assigns each token to a group based solely on its own features, using learned cluster centers encoded in a single linear layer's weights. This eliminates the coarse estimation stage entirely: there are no blocks to score, no top-k to select, and no block size hyperparameter to tune. The computational cost of the router — $\mathcal{O}(M \cdot d)$ per token, where $M$ is a small constant (5–20) — is negligible compared to the attention computation itself.

What makes this genuinely innovative rather than an obvious application of MoE routing to attention is the conceptual reframing it enables. Block-level methods implicitly assume that attention relevance is a function of spatiotemporal proximity — tokens that are nearby in the video grid are more likely to attend to each other, so block-level selection along spatiotemporal dimensions is a natural approximation. MoGA's routing makes no such assumption. The router groups tokens based on learned semantic features that emerge from end-to-end training, not on where tokens are located in the video tensor. This is why the router in Figure 3 groups "the man's head, hands, and portions of his clothing to the same group" across shot boundaries — these tokens are not spatiotemporally contiguous (they span different frames and different spatial positions), but they share a semantic identity (parts of the same person) that the router has learned to recognize.

This is a fundamental shift from spatial selection (attend to tokens in nearby blocks) to semantic selection (attend to tokens that belong to the same learned concept), and it is this shift — not merely the efficiency gain — that makes MoGA suitable for multi-shot long video generation. Cross-shot character consistency requires attending across large spatiotemporal gaps, which block-based methods (even if perfectly accurate) would have difficulty capturing because the blocks containing a character in different shots are not spatiotemporally adjacent.

The paper's evidence that this conceptual move matters is not just the FLOPs reduction (though the 3× savings versus full attention on 30-second videos is significant) but the qualitative properties it enables: MoGA matches or exceeds full attention on VBench consistency metrics (Table 1) and produces stable character identity across shots that block-based sparse methods and even full-attention methods struggle with (Figures 5, 8). The fact that a highly sparse method can match full attention on consistency metrics is evidence that MoGA's grouping is not merely approximating full attention at lower cost — it is performing a genuinely different computation that may suppress noise from irrelevant token pairs that full attention necessarily includes.

The comparison to SVG2 is particularly instructive for understanding what makes MoGA distinctive. SVG2 also recognizes that tokens should be grouped into semantically coherent clusters, and it also avoids block-level estimation. But SVG2 performs online k-means clustering during inference — a post-hoc heuristic that cannot be optimized for the end task. MoGA's router, by contrast, is trained end-to-end with the denoising objective. Its cluster centers (the rows of the router weight matrix) are optimized by gradient descent to produce groupings that minimize video generation loss. This is a learned representation of what constitutes a useful attention group, not a geometric clustering of token features. The difference is subtle but consequential: a k-means cluster might group tokens by color or texture similarity, but MoGA learns to group them by whatever features are predictive of beneficial attention interactions for the denoising task.

This innovation is foundational rather than incremental — it opens an entirely new design space for sparse attention mechanisms based on learned, per-token routing, which the block-based paradigm could not explore.

---

Innovation 2: The Accuracy–Efficiency Trade-off in Sparse Attention Is Not Inherent — It Is a Byproduct of Block-Level Approximation

The paper's second conceptual contribution is a diagnostic insight rather than a technical mechanism: the widely observed trade-off between attention accuracy and computational efficiency in sparse video generation methods is not a fundamental property of sparse attention but a specific consequence of block-level coarse estimation. By demonstrating that a method with no blocks and no coarse scoring achieves both higher sparsity and better or comparable generation quality, the paper effectively dissolves what the field had treated as an inescapable tension.

To appreciate why this is a conceptual contribution rather than merely an empirical result, consider how prior work framed the problem. VMoBA explicitly parameterizes the trade-off through block size and top-k selection: smaller blocks with larger k improve accuracy at the cost of efficiency; larger blocks with smaller k improve efficiency at the cost of accuracy. This framing — that you must choose a point on the Pareto frontier — was so natural that it went unquestioned. It followed directly from the mechanism: if you estimate importance at the block level, you incur a granularity-dependent estimation error that trades off against the computational savings from scoring fewer blocks.

MoGA's router sidesteps this mechanism entirely. There is no granularity parameter to tune because there are no blocks — each token is individually assigned. There is no top-k selection because tokens are not selecting which other tokens to attend to; they are all assigned to groups, and attention is computed within each group. The efficiency comes from the reduction in attention group size (each group contains approximately $N/M$ tokens), not from selectively dropping query-key pairs. Every token attends to every other token in its group — there is no estimation error from missed connections because no connections are estimated and discarded; they are simply assigned to different computational partitions.

The paper's evidence for this claim is structural rather than purely quantitative. If the trade-off were inherent, we would expect MoGA to either (a) underperform full attention on quality metrics (accepting a quality loss for efficiency gain) or (b) be less efficient than block-based methods (accepting an efficiency loss for quality gain). Instead, MoGA simultaneously achieves 3× FLOPs reduction versus full attention (Figure 6, Table 5) while matching or exceeding full attention on VBench quality metrics (Table 1). On 10-second multi-shot generation, MoGA outperforms EchoShot (a full-attention method) on subject consistency, background consistency, and aesthetic quality (Table 2). On 30-second generation, MoGA substantially outperforms the keyframe-based pipeline IC-LoRA+Wan (Table 3) while being an end-to-end method rather than a multi-stage pipeline.

This simultaneous improvement on both efficiency and quality is what distinguishes this innovation from a mere engineering optimization. It implies that the block-level paradigm was not just inefficient — it was architecturally suboptimal for video generation. The accuracy losses from block-level estimation were not an acceptable cost of sparsity; they were an artifact of a specific mechanism that could be replaced entirely.

The significance extends beyond video generation. Block-based sparse attention (MoBA) was proposed as a general method for long-context LLMs as well, and DeepSeek has adopted related approaches. MoGA's demonstration that learned per-token routing can outperform block-level selection suggests that the same conceptual move might benefit other domains where sparse attention is used — language modeling, protein structure prediction, or any sequence modeling task where attention is known to be sparse but the sparsity pattern is content-dependent.

This innovation is a reframing of the problem space rather than a point solution. It changes what researchers should be optimizing: not the block size or the top-k threshold, but the grouping mechanism itself.

---

Innovation 3: Semantic Grouping for Attention Can Emerge from End-to-End Training with a Simple Routing Objective — It Does Not Require Explicit Clustering or Similarity Supervision

The third innovation concerns how the router learns to produce useful groupings. MoGA's router receives no direct supervision about which tokens should be grouped together — there are no ground-truth group labels, no auxiliary task predicting semantic categories, and no explicit similarity objective (beyond the group balancing loss, which only enforces uniform utilization, not correctness). The router is trained solely through the gradients that flow from the denoising loss, through the attention computation within each group, and back to the router parameters.

Despite this indirect training signal, the router converges to groupings that are interpretably semantic. Figure 3 shows that after training, one group contains the man's head, hands, and portions of his clothing — body parts that belong to the same person and that span multiple shots. This is not a trivial consequence of the architecture; it represents an emergent organizational principle that the model discovers through the training dynamics.

What makes this conceptually significant is what it reveals about the relationship between attention and semantics in video generation. The router does not group tokens by visual similarity (head tokens look different from hand tokens, yet they are grouped together) or by spatiotemporal proximity (tokens from different shots are grouped together). It groups tokens by mutual attention benefit — the router learns that routing these tokens to the same group allows them to attend to each other, and that this cross-attention improves denoising quality. In effect, the router learns a task-specific notion of semantic coherence: what matters is not whether tokens "look like" they belong together by any pre-defined metric, but whether allowing them to attend to each other helps the model produce better videos.

This stands in contrast to methods that explicitly construct semantic groupings using pre-trained feature extractors (e.g., clustering DINOv2 features to identify object instances) or that use auxiliary losses to encourage semantic grouping (e.g., contrastive losses that pull together tokens from the same object). MoGA shows that these explicit mechanisms are unnecessary — the denoising objective alone provides sufficient signal for a simple linear router to discover task-relevant groupings, provided that the grouping is sufficiently balanced (enforced by the lightweight auxiliary loss) and that the architecture allows gradients to flow through the grouping decision (enabled by multiplying attention outputs by the softmax routing probability).

The theoretical implication is subtle but important. It suggests that the inductive bias of grouped attention — the architectural constraint that tokens can only attend within their assigned groups — is itself sufficient to drive the emergence of semantic organization, without any explicit semantic supervision. The router is forced to assign tokens to groups, and the model is forced to generate videos using only within-group attention. The only way to achieve good generation quality under this constraint is to route tokens that need to attend to each other into the same group. The gradients from the denoising loss naturally push the router toward this solution, even though no component of the loss explicitly says "group semantically related tokens together."

This is not merely a convenient training property — it is evidence for a deeper claim about attention in video DiTs: that the attention patterns needed for effective denoising are structured enough that a simple linear router with M groups can capture them, and that this structure can be discovered through gradient-based optimization without explicit semantic guidance.

The practical significance is that MoGA requires no additional annotation, no pre-trained semantic segmentation models, and no auxiliary training objectives beyond the lightweight group balancing loss. This makes it straightforward to apply to new domains or model architectures (as demonstrated by the paper's application to both Wan2.1 and MMDiT backbones) without engineering domain-specific grouping heuristics.

---

Innovation 4: Sparse and Full Attention Mechanisms Can Play Complementary, Architecturally Separated Roles in a Single Model — Semantic Grouping for Long-Range Coherence, Static Windows for Local Fidelity

The fourth innovation is an architectural design principle rather than a mechanism: MoGA and STGA are not alternative attention patterns that you choose between (as in prior work that uses a single sparse pattern) but complementary components whose responsibilities are deliberately partitioned. MoGA handles semantically coherent but spatially distributed interactions (cross-shot character identity, global style), while STGA handles spatiotemporally local but semantically agnostic interactions (motion continuity, texture detail). Neither mechanism alone suffices — Figure 11 shows that MoGA alone cannot produce meaningful visual content (lacking local information exchange) and STGA alone cannot maintain cross-shot consistency. Together, they achieve what neither can individually.

What makes this distinctive is not the idea of combining multiple attention patterns (prior work like StreamingT2V uses both short-term and long-term memory), but the principled separation along the semantic vs. spatiotemporal axis. In prior work that combines local and global attention, the "global" component is typically still spatiotemporally structured — it attends to a sparse set of key frames, or compresses historical tokens, or selects tokens based on spatiotemporal heuristics. MoGA's "global" component (the learned semantic groups) is not spatiotemporally structured at all — it groups tokens based on what they represent, not where or when they occur in the video.

This separation has a specific architectural consequence that the paper exploits: in STGA, frames from different shots are explicitly assigned to different temporal groups, preventing any local inter-shot attention. This is a deliberate choice — the paper states that "frames from different shots are assigned to distinct temporal groups" — and it means that cross-shot information flows exclusively through MoGA. The architecture enforces that long-range, cross-shot interactions are handled by the semantic grouping mechanism, not by the local spatiotemporal mechanism.

Why is this principled rather than ad hoc? Because it reflects a hypothesis about the structure of video data: that the interactions needed for local continuity (motion, texture, fine detail) are fundamentally different from the interactions needed for global consistency (identity, style, background), and that these two types of interactions are best handled by attention mechanisms with different inductive biases — one that is spatially localized and temporally bounded, and one that is semantically organized and globally scoped. The paper does not state this hypothesis explicitly as a theoretical claim, but the architecture embodies it, and the ablation in Figure 11 provides empirical support.

The shot-boundary treatment in STGA — augmenting keys and values with two latent frames from adjacent shots, but not augmenting queries — is a particularly elegant instantiation of this separation. It provides just enough cross-shot information to prevent flicker at shot boundaries (a local artifact) without allowing the local attention mechanism to blend content across shots (which would undermine shot separation). The one-way information flow (queries cannot cross shot boundaries, but keys and values from adjacent shots are visible) is a carefully calibrated architectural choice that the paper justifies through empirical observation rather than theory, but that reflects a deeper principle: the architecture should provide mechanisms for both separation (shots should be distinct) and continuity (transitions should be smooth), and these mechanisms should be allocated to different computational pathways rather than forced into a single attention pattern.

This innovation is methodological — it provides a template for how to think about designing attention architectures for structured data (not just videos, but any data with both local and global structure). The key question is not "which sparse attention pattern is best?" but "what are the distinct types of interactions the model needs, and which attention mechanisms should handle each type?"

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper uses three test sets tailored to specific generation durations: for 5-second single-shot video generation, a diverse test set of 300 prompts; for 10-second multi-shot video generation, the 100 multi-shot prompt sets from EchoShot (Wang et al., 2025b); for 30-second multi-shot long video generation, 11 scripts comprising 105 prompts, where each script contains 8–10 shots. All prompts produce short-level textual conditioning for multi-shot experiments.

• Base model(s). MoGA is fine-tuned on two DiT-based pretrained checkpoints: the open-source Wan2.1 models at 1.3B and 14B parameters (Wan et al., 2025), and an MMDiT-based architecture (following Esser et al., 2024; Kong et al., 2024; Gao et al., 2025). The Wan2.1 models are chosen for fair comparison with baselines (the paper explicitly states "we train MoGA on the open-source Wan2.1 models for a fair comparison"); the MMDiT-based model enables the longest context (578k tokens, 60-second generation). All training uses the rectified flow objective and fine-tunes from short-video-generation checkpoints.

• Metrics. The primary evaluation framework is VBench (Huang et al., 2024b), which provides automated metrics for video generation quality. Specifically:

  • Subject consistency and background consistency: measure how well the main subjects and backgrounds of sampled frames are preserved throughout the video, computed using feature similarity across frames.
  • Motion smoothness: measures motion fluidity, penalizing jitter and abrupt transitions between frames.
  • Aesthetic quality and image quality: quantify the visual appeal and technical fidelity of individual frames using pretrained aesthetic and quality assessment models.

  Additionally, the paper introduces cross-shot consistency metrics for multi-shot evaluation:

  • Cross-Shot CLIP and Cross-Shot DINO: sample a fixed number of frames from different shots, then compute feature similarities across shots using CLIP (Radford et al., 2021) and DINOv2 (Oquab et al., 2023). These specifically measure whether character identity and scene properties persist across shot boundaries, which is the key challenge in multi-shot generation.

• Baselines. The paper compares against multiple categories:

  • Full-attention methods: The original Wan2.1 models (full attention) for single-shot generation; EchoShot (Wang et al., 2025b), which natively supports multi-shot generation with full attention.
  • Multi-stage pipeline methods: IC-LoRA+Wan (Huang et al., 2024a; Wan et al., 2025), a keyframe-based pipeline that uses In-Context LoRA with the Wan backbone.
  • Training-based sparse attention: DiTFastAttn (Yuan et al., 2024), which uses attention compression for DiT models.
  • Training-free sparse attention: SVG (Xi et al., 2025), which classifies attention heads as spatial or temporal sparse attention using online pattern selection; VMoBA (Wu et al., 2025), which adapts MoBA's block-based dynamic routing to video generation.

  Not all baselines are evaluated in all settings. For 5-second single-shot generation (Table 1), comparisons include Wan (full attention), DiTFastAttn, SVG, and VMoBA. For 10-second multi-shot generation (Table 2), comparisons include IC-LoRA+Wan and EchoShot. For 30-second long generation (Table 3), comparisons include IC-LoRA+Wan under the Wan2.1-1.3B and Wan2.1-14B backbones, plus an MMDiT-based MoGA variant.

• Generation budget / compute accounting. The paper measures compute in two complementary ways. For algorithmic complexity, FLOPs are computed analytically for different numbers of groups and video durations, with Table 5 reporting full attention at 6.94 PFLOPs vs. MoGA at 2.26 PFLOPs for 30-second videos under Wan2.1-1.3B (a ~3× reduction). For wall-clock efficiency, the paper reports a 1.7× speedup in both training and inference. Importantly, the paper emphasizes that MoGA introduces "no additional memory consumption" compared to full attention, unlike block-based methods such as VMoBA that incur additional memory overhead for block indices and importance score caches. The key parameter controlling the efficiency-quality trade-off is the number of groups M, swept at M = 5 for Wan2.1-based models and M = 20 for the MMDiT-based model.

• Cross-validation / statistical protocol. The paper does not report cross-validation or statistical significance testing. Results are presented as single-number metrics without confidence intervals or error bars. The test sets are described as "a diverse test set comprising 300 prompts" (5-second single-shot), "the 100 multi-shot prompt sets from [EchoShot]" (10-second multi-shot), and "a test set of 11 scripts comprising 105 prompts" (30-second long generation). Given the small sample size for the 30-second benchmark (11 scripts), the reported metrics should be interpreted as indicative rather than statistically robust — the paper does not discuss variance across prompts or report any measure of result stability. For the compute-optimal strategy selection that MoGA does not require (unlike the reference example paper), there is no cross-validation protocol to report.

---

Main Quantitative Results

Single-Shot Short Video Generation (5 seconds)

Table 1 reports the comparison between MoGA and sparse attention baselines on single-shot 5-second video generation. The headline result is that MoGA achieves consistent improvements over existing sparse baselines despite higher sparsity, and in several cases matches or surpasses the original full-attention Wan model.

Specific numbers from Table 1 (the paper does not provide exact values in the text, but describes patterns):

• Subject consistency: MoGA outperforms DiTFastAttn, SVG, and VMoBA. The paper states MoGA "achieves consistent improvements over existing sparse baselines across metrics" and "despite higher sparsity, MoGA can still match or surpass the original Wan (full attention) on multiple metrics."
• Background consistency: The pattern mirrors subject consistency, with MoGA exceeding sparse baselines and matching full attention.
• Motion smoothness: MoGA maintains competitive motion smoothness, indicating that the semantic grouping does not introduce temporal artifacts (despite not explicitly modeling temporal continuity — that is handled by STGA).
• Aesthetic quality and image quality: MoGA achieves comparable or better scores than sparse baselines, and is competitive with full attention.

The key interpretive point is that MoGA's superior performance on consistency metrics comes despite higher sparsity than the alternatives — the paper's framing is that MoGA's semantic grouping is not merely an efficiency hack but can actually improve quality by suppressing irrelevant token interactions.

Multi-Shot Video Generation (10 seconds)

Table 2 shifts to multi-shot generation, comparing MoGA against EchoShot (full attention, natively multi-shot) and IC-LoRA+Wan (keyframe pipeline). The headline: MoGA outperforms both baselines on most metrics despite relying on sparse attention.

The paper states: "Despite relying on sparse attention, our method outperforms the full attention baseline (EchoShot) on most metrics, indicating that preserving interactions among salient tokens not only reduces FLOPs but also suppresses noise from irrelevant content. This leads to stronger character identity consistency and improved temporal scene coherence."

Specific metric patterns from Table 2:

• Subject consistency: MoGA exceeds both EchoShot and IC-LoRA+Wan. This is the most important finding for multi-shot generation — maintaining character identity across different scenes is the primary challenge, and MoGA's semantic grouping directly targets this.
• Background consistency: MoGA outperforms both baselines. EchoShot's full attention, which computes all pairwise interactions including cross-shot token pairs, does not help background consistency — in fact, MoGA's sparse grouping produces better results, consistent with the claim that full attention introduces noise from irrelevant cross-shot interactions.
• Motion smoothness: Competitive across methods (specific relative ordering not detailed).
• Aesthetic quality: MoGA scores higher than both baselines.
• Image quality: Similar advantage for MoGA.

The paper explicitly interprets this as evidence for the "suppress noise from irrelevant content" hypothesis: full attention computes all N² interactions, including semantically meaningless ones (e.g., a background pixel from shot 1 attending to a background pixel from shot 3), which can dilute the influence of genuinely important interactions. MoGA's grouping restricts attention to tokens within the same semantic group, which naturally filters out cross-shot interactions unless the router has learned that they are beneficial.

Long Video Generation (30 seconds)

Table 3 reports the most challenging setting: 30-second multi-shot generation. Because "few open-source methods can produce 30-second, multi-shot videos," the comparison is primarily against IC-LoRA+Wan under the same backbones (Wan2.1-1.3B and Wan2.1-14B), plus an MMDiT-based MoGA variant.

The headline: MoGA substantially outperforms IC-LoRA+Wan across all metrics for both backbones. Specific claims from the paper:

• Under the Wan2.1-1.3B backbone, MoGA exceeds IC-LoRA+Wan on all reported VBench metrics (subject consistency, background consistency, motion smoothness, aesthetic quality, image quality). The paper states this "highlights the benefits of end-to-end modeling over multistage pipelines."
• Under the Wan2.1-14B backbone, the same pattern holds — MoGA outperforms IC-LoRA+Wan across metrics.
• The MMDiT-based MoGA model (M=20, 578k context length) maintains "high visual fidelity" even under "aggressive sparsity." The paper frames this as "indicating a scalable path to longer context lengths."

What makes these results notable is not just that MoGA outperforms a multi-stage pipeline (end-to-end methods generally have an advantage), but that it does so with substantially lower computation: the IC-LoRA+Wan pipeline uses the full Wan backbone per stage, while MoGA uses sparse attention throughout. The paper does not report a FLOPs comparison specifically for the IC-LoRA+Wan vs. MoGA setting, but the per-generation savings are documented in Figure 6 and Table 5.

The paper also makes an implicit comparison between the two MoGA variants. The MMDiT-based model (M=20, spatial partition 4×4) achieves longer context (60 seconds vs. 30 seconds) while "maintaining high visual fidelity," which the paper presents as evidence for scalability — increasing the number of groups allows proportionally longer sequences at similar per-token compute cost.

Computational Efficiency Scaling

Figure 6 plots the relationship between the number of groups M and FLOPs for the Wan2.1-1.3B model across video durations. The key data points:

• Full attention at 30 seconds: 6.94 PFLOPs.
• MoGA at M=5, 30 seconds: 2.26 PFLOPs (a ~3.1× reduction).
• MoGA at M=10, 30 seconds: Approximately 1.5 PFLOPs (roughly 4.6× reduction, from the trend in Figure 6).
• MoGA at M=20, 30 seconds: Below 1 PFLOPs (nearly 7× reduction).

The FLOPs reduction scales roughly as 1/M, consistent with the theoretical analysis that MoGA reduces complexity from O(N²) to O(N²/M) under uniform assignment. The paper notes that the actual reduction is somewhat less than the theoretical maximum due to non-uniform group assignment and the overhead of the router itself, but the scaling behavior follows the predicted trend.

Importantly, the paper also reports that MoGA achieves a 1.7× wall-clock speedup in both training and inference. This is smaller than the FLOPs reduction (3.1×) because wall-clock time is affected by factors beyond raw FLOPs — memory bandwidth, kernel launch overhead, and the permutation/unpermutation operations in Algorithm 1. Nevertheless, a 1.7× speedup is practically significant for training long-video models.

The paper explicitly contrasts this with alternative sparse attention: "unlike alternative sparse attention such as VMoBA, which incur additional memory overhead due to their block-based mechanisms, our approach maintains memory efficiency without additional memory consumption." This is a practical advantage — memory, not just FLOPs, is often the bottleneck for long-sequence training, and MoGA's zero additional memory overhead means it can be applied to longer sequences than block-based methods on the same hardware.

---

Ablation Studies and Robustness Checks

Routing Group Number M: Table 4 reports ablation results on the number of groups for MoGA with Wan2.1-1.3B on 10-second videos. The key finding is a non-monotonic relationship: cross-shot DINO and CLIP scores exhibit a "rise-then-fall trend as the number of groups increases." Specifically:

• At small M (too few groups), each group is large and diverse, reducing the selectivity of attention — tokens that should be separated may end up in the same group, leading to weaker consistency.
• At large M (too many groups), groups become too small and fragmented, preventing tokens that should attend to each other (e.g., different views of the same character) from being in the same group.
• The optimal M balances these forces, achieving "near-optimal consistency while maintaining computational efficiency."

The paper interprets this as evidence that "a moderate level of grouped sparsity strikes a balance between global consistency and efficiency." This is a practically important finding — M is the single most important hyperparameter in MoGA, and the optimal value depends on the trade-off between group purity (keeping semantically related tokens together) and group coverage (ensuring each group is large enough to capture all relevant tokens for a given concept).

The specific M values used in the main experiments — M=5 for Wan2.1 and M=20 for MMDiT — represent empirically determined operating points for their respective context lengths and model scales, but the paper does not report a systematic sweep to justify these specific choices. The M=20 setting for MMDiT is notably higher, which the paper attributes to enabling "a much longer context length" — more groups means smaller groups, reducing the per-attention cost and allowing proportionally longer sequences at similar total FLOPs.

Group Balancing Loss: Appendix Figure 12 validates the necessity of the group balancing loss (Equation 6). The paper monitors the balancing metric (∑ᵢ FᵢPᵢ) during training under two conditions: with and without the auxiliary loss. Findings:

• Without the loss: The metric increases as training progresses, indicating that the router funnels tokens into a few groups. The paper explains: "the router funnels tokens into a few groups to gain short-term advantages in the diffusion MSE loss" — routing all tokens to one group is equivalent to full attention, which initially yields lower denoising loss but eliminates the efficiency benefits.
• With the loss (α=0.1): The metric "rapidly converges to around 1, reflecting globally balanced assignments."

The paper draws a clear conclusion: "Because our goal is to separate weakly related tokens and maintain balanced grouping, the additional group balance loss is necessary to enforce the desired assignments." This is non-trivial — one might have hoped that the denoising objective alone would encourage diverse grouping (since different semantic concepts require different attention patterns), but the training dynamics favor collapse. The group balancing loss is therefore not optional; it is a required component of the training procedure.

MoGA vs. STGA Ablation: Figure 11 provides a qualitative ablation isolating the contributions of MoGA and STGA. The paper reports three configurations:

• MoGA only (no STGA): "Lacks local information exchange and fails to produce meaningful visual content." This is expected — MoGA groups tokens semantically, but without any mechanism for local spatial continuity (adjacent pixels attending to each other), the model cannot produce coherent textures, edges, or motion.
• STGA only (no MoGA): "Limits long-range shot interactions, leading to poor cross-shot consistency and weakened narrative coherence." Because STGA explicitly separates frames from different shots into distinct temporal groups (and only provides two-frame boundary augmentation), there is essentially no mechanism for maintaining character identity or background consistency across shots. Cross-shot consistency is lost.
• MoGA + STGA combined: "The model achieves strong cross-shot consistency."

This ablation validates the paper's core architectural claim: MoGA and STGA are not alternative sparse attention patterns but complementary mechanisms that handle fundamentally different types of interactions. The failure modes are cleanly separated — STGA failure manifests as broken local structure; MoGA failure manifests as broken global consistency. Neither alone suffices, and their combination (via simple output averaging) produces results that neither achieves individually.

The paper frames this as evidence that "MoGA effectively routes and preserves shot-spanning identity and context at relatively low computational cost," since STGA's local attention is cheap (bounded window size) and MoGA's global attention is efficient (factor of M reduction from full attention).

Controllability of Subject Consistency: Figure 10 compares MoGA and full attention (both trained on 10-second data with Wan2.1-14B) on a specific capability: maintaining subject identity while responding to text instructions to change appearance. The setup involves two scenarios:

• Left panel: The subject wears the same outfit across different shots. Both MoGA and full attention maintain consistent identity, but the paper notes that MoGA achieves this with "71.25% sparsity" — roughly 3.5× fewer FLOPs than full attention.
• Right panel: The subject changes outfits at shot transitions according to text instructions. Both methods successfully execute the clothing change while maintaining the subject's facial identity.

The key finding is that MoGA "achieves narrative coherence and content editability on par with full attention, and in some cases delivers superior performance." This is important because it demonstrates that MoGA's sparsity does not come at the cost of controllability — the model can still follow detailed shot-level text instructions, and the semantic grouping does not prevent the appearance changes that the text specifies.

The 71.25% sparsity figure is notable: it means only ~29% of the full attention computation is being performed, yet the qualitative results are comparable or better. This is the strongest evidence for the paper's claim that full attention computes many unnecessary interactions, and that MoGA's grouping suppresses these without losing the important ones.

Multi-Style Generation: Figure 9 demonstrates MoGA's ability to generate in non-photorealistic styles (specifically 2D animation). The paper states MoGA "not only performs strongly in realistic spaces but also excels in stylized domains such as animation" and "can produce high-quality, long-form 2D videos while maintaining temporal coherence, identity consistency, and scene continuity across diverse styles." This is not a controlled ablation but a qualitative demonstration that the method generalizes beyond the photorealistic training distribution — the learned semantic grouping mechanism is not specific to realistic textures or lighting.

Emergence of Background Consistency: Figure 8 demonstrates a qualitative phenomenon the paper terms "emergence of background consistency." After training on long, multi-shot videos, MoGA exhibits "implicit control over consistency in both the environment and the characters" — even when specific details are not explicitly specified in the shot-level text captions (e.g., cabinet shape, position of an IV drip bottle), different shots "automatically maintain coherent, temporally consistent depictions."

The paper frames this as an emergent property rather than an explicitly trained behavior: the model learns from the multi-shot training data that background elements should persist across shots unless the text explicitly changes them, and MoGA's cross-shot attention mechanism provides the architectural pathway for this persistence. No auxiliary loss or explicit conditioning enforces background consistency — it emerges from the combination of MoGA's semantic grouping and the multi-shot training data.

Training-Free vs. Trained Sparse Attention: The paper implicitly compares MoGA (trained) against SVG and VMoBA (training-free) in Table 1, where MoGA outperforms both. However, this comparison conflates two dimensions: the routing mechanism (learned semantic groups vs. block-level scoring vs. head classification) and the need for training. VMoBA is specifically designed to work without training, which is an advantage for plug-and-play deployment on existing checkpoints but limits its ability to optimize routing for the specific generation task. MoGA requires fine-tuning (3,000 + 1,000 steps in the multistage training), which gives it more capacity to adapt routing to the video generation objective. The paper does not ablate this separately — we cannot isolate how much of MoGA's advantage comes from the routing mechanism itself versus from the fact that it is end-to-end trained while VMoBA is not.

Kernel Compatibility: The paper emphasizes FlashAttention integration (Algorithm 1) and sequence parallelism compatibility as practical advantages, but does not provide an ablation comparing MoGA's permute-based implementation against alternative implementations (e.g., custom sparse kernels, mask-based attention). The 1.7× speedup is reported relative to full attention, not relative to other sparse implementations. It is therefore unclear whether MoGA's efficiency advantage over VMoBA or SVG in practice comes from the grouping mechanism itself or from better kernel utilization.

---

Critical Assessment

Does MoGA genuinely demonstrate that block-level coarse estimation is unnecessary?

What the experiments show: Table 1 demonstrates that MoGA outperforms VMoBA (the primary representative of block-level dynamic routing) on standard VBench metrics for 5-second single-shot generation. Table 2 and Table 3 extend this to multi-shot settings where VMoBA is not directly evaluated. Figure 10 shows MoGA matching full attention with 71.25% sparsity.

What the experiments do not show: The paper never isolates the contribution of per-token routing versus block-level routing while controlling for all other variables. VMoBA and MoGA differ in multiple dimensions simultaneously: VMoBA is training-free, uses blocks with a top-k selection, and computes block-level similarity scores. MoGA is trained, uses per-token routing with softmax gating, and has no top-k selection. Any of these differences — not just the block-level vs. per-token distinction — could account for the performance gap. A controlled experiment would fix the training regime (both methods trained or both training-free), vary only the routing mechanism, and compare. This experiment is not reported.

Additionally, the paper's central conceptual claim — that block-level estimation introduces an inherent accuracy–efficiency trade-off that per-token routing transcends — requires demonstrating that MoGA can achieve both better quality AND better efficiency than block-based methods simultaneously. Table 1 shows better quality, and Figure 6 shows better FLOPs scaling, but these are on different settings (different video durations, different backbones). A single experiment showing MoGA with better VBench scores AND lower FLOPs than VMoBA on the same task would more directly support the claim.

The paper's claim is plausible but not rigorously demonstrated in the experiments as reported. The qualitative evidence (Figure 3: semantic grouping emerges; Figure 10: matching full attention at high sparsity) is suggestive that the routing mechanism is doing something meaningful, but the causal link between "eliminating block-level estimation" and "better performance" is not isolated.

Does MoGA genuinely outperform full attention, or just match it with less compute?

What the experiments show: Table 1 states MoGA "can still match or surpass the original Wan (full attention) on multiple metrics." Table 2 states MoGA "outperforms the full attention baseline (EchoShot) on most metrics."

What the experiments do not show: The comparison against EchoShot (Table 2) is not a pure comparison of attention mechanisms. EchoShot and MoGA are different models with different architectures, training data, and training procedures — they share the Wan backbone only in the sense that both are DiT-based. EchoShot was not designed for sparse attention and may not have been optimized for the specific VBench metrics. The claim that MoGA's sparsity "suppresses noise from irrelevant content" (the proposed mechanism for why MoGA might outperform full attention) would be more convincingly demonstrated by comparing MoGA against the identical model with full attention (trained on the same data with the same hyperparameters), showing that MoGA's grouping provides a quality benefit beyond the efficiency gain. The Wan comparison in Table 1 is closer to this ideal, but the paper reports it as "matching or surpassing" without specifying which conditions produce which outcome.

A cautious assessment: The evidence supports the claim that MoGA achieves comparable quality to full attention at substantially lower compute cost. The claim that MoGA surpasses full attention on quality is suggested but not robustly established — it depends on specific metrics and baselines, and the experiments do not cleanly isolate the "noise suppression" mechanism from other confounding factors (different training data for EchoShot, different optimization for the Wan baseline).

Does MoGA scale to minute-long generation in practice?

What the experiments show: The MMDiT-based MoGA model (M=20) generates 1,441 frames at 24 fps (60 seconds) at 480p with a context length of 578k tokens. Table 3 reports VBench metrics for 30-second generation; Figure 7 shows qualitative results for a 1-minute video with 22 shots.

What the experiments do not show: The quantitative evaluation in Table 3 is for 30-second videos, not 60-second videos. The 60-second result in Figure 7 is a single qualitative example — there is no quantitative evaluation of 60-second generation quality, no comparison against any baseline at 60 seconds, and no demonstration that the quality metrics remain stable as duration increases from 30 to 60 seconds. The paper's claim of "minute-level" generation is therefore supported by existence proof (the model can run and produce a coherent 60-second video) but not by rigorous evaluation.

Additionally, the MMDiT-based model is a different architecture from the Wan2.1-based models used in most experiments. The paper does not provide a direct comparison between the MMDiT-based MoGA and a full-attention MMDiT baseline at the same duration, so we cannot assess how much quality is sacrificed for the extended context length. The claim of "high visual fidelity" is qualitative and not benchmarked.

The 60-second generation is a proof of concept, not a rigorously evaluated capability. The paper's primary quantitative evidence supports 30-second generation quality; the extension to 60 seconds is demonstrated but not measured.

Does the data pipeline enable the reported results, or are the architectural innovations sufficient with any multi-shot data?

What the experiments show: The data pipeline is described in detail (Section 3.3, Figure 4) and used to produce training data. The paper states that "extended context alone is insufficient because a single global prompt cannot reliably control scene transitions," motivating shot-level conditioning.

What the experiments do not show: There is no ablation comparing models trained on the pipeline's multi-shot data versus models trained on simpler multi-shot data (e.g., concatenated short videos with synthetic shot boundaries, or videos with automatically generated but noisier shot segmentations). Without this ablation, we cannot assess whether the data pipeline's sophisticated filtering, dual shot detection, and careful transition trimming are necessary for the results, or whether the architectural innovations (MoGA + STGA + shot-level conditioning) would work with any reasonably clean multi-shot data. The data pipeline is a substantial engineering contribution, but its necessity is asserted rather than tested.

Are the evaluation metrics sufficient to capture long-video quality?

The paper relies on VBench metrics (subject consistency, background consistency, motion smoothness, aesthetic quality, image quality) plus cross-shot CLIP and DINO similarity. These are frame-level and pairwise metrics — they assess whether sampled frames are consistent with each other and visually appealing, but they do not assess:

• Narrative coherence: Does the sequence of shots tell a coherent story? Do events unfold logically? CLIP similarity between frames from different shots does not capture whether the sequence makes sense as a narrative.
• Temporal dynamics: Motion smoothness penalizes jitter but does not assess whether the motion is realistic or appropriate for the described content.
• Long-range consistency failure modes: The metrics sample a "fixed number of frames from different shots" for cross-shot evaluation, but do not specify how many frames or which shots. If only a few frames per shot are sampled, subtle consistency failures (e.g., object gradually changing color over 10 seconds within a single shot) may be missed.
• Human preference: No user study is reported. VBench metrics correlate with human judgments but are not a substitute, especially for complex constructs like "narrative coherence" or "character identity consistency" that may require fine-grained human evaluation.

The paper's strongest claims — about "emergent background consistency" and "maintaining identity across shots" — are supported primarily by qualitative examples (Figures 5, 7, 8, 9, 10, 11) and aggregate metrics that may not capture the specific failure modes of long video generation. A more rigorous evaluation would include temporal consistency metrics over longer windows, human evaluations, or targeted probes of specific consistency challenges (e.g., does the model preserve a character's specific clothing pattern when the character reappears after 5 shots?).

Missing Comparisons and Experiments

Several experiments would strengthen the paper's claims:

1. Direct FLOPs-matched comparison between MoGA and VMoBA at equal sparsity levels. This would isolate whether the per-token routing mechanism provides quality benefits beyond what block-level routing achieves at the same compute budget.

2. Full-attention baseline trained on the same multi-shot data as MoGA. The comparisons against EchoShot and Wan use models trained on different data distributions, confounding the attention mechanism comparison.

3. Quantitative evaluation of 60-second generation against a baseline (even if that baseline is a simple extension of a 30-second method, such as concatenated IC-LoRA+Wan outputs). The 1-minute generation is the paper's headline capability, but it is evaluated only qualitatively with a single example.

4. Ablation of the data pipeline quality. Training MoGA on simpler multi-shot data (e.g., AutoShot-only segmentation, no watermark removal, no transition trimming) would reveal whether the pipeline's sophistication is necessary.

5. Memory consumption benchmarks comparing MoGA against VMoBA and full attention at equal sequence lengths. The paper claims zero additional memory overhead but provides no measurements.

6. Scaling behavior of cross-shot consistency with video duration. Does MoGA's cross-shot DINO score remain stable as videos grow from 10 to 30 to 60 seconds, or does it degrade? This would test the paper's claim that MoGA provides a "scalable path to longer context lengths."

7. Ablation of the STGA shot-boundary augmentation (the two-frame keys/values from adjacent shots). The paper mentions this prevents flicker but does not quantify the improvement or show what happens without it.

Summary of Experimental Support

The experiments provide solid support for MoGA's efficiency claims: MoGA achieves approximately 3× FLOPs reduction and 1.7× speedup versus full attention on 30-second videos (Figure 6, Table 5), while matching or exceeding full-attention quality on standard VBench metrics (Tables 1, 2). The ablation in Figure 11 convincingly demonstrates that both MoGA and STGA are necessary, and that neither alone suffices.

The experiments provide moderate support for MoGA's quality claims: MoGA outperforms sparse baselines (Table 1), outperforms EchoShot on multi-shot consistency (Table 2), and substantially outperforms the IC-LoRA+Wan pipeline (Table 3). However, these comparisons are confounded by differences in training data, architecture, and optimization that prevent clean attribution of the quality improvements to the MoGA routing mechanism specifically.

The experiments provide weak support for the paper's strongest conceptual claims: the claim that block-level estimation is "unnecessary" is not tested through controlled ablation; the claim that MoGA surpasses full attention by "suppressing noise" is not isolated from other factors; the claim of "minute-level" generation is supported by a single qualitative example without quantitative benchmarks. The paper's primary contribution — the MoGA routing mechanism — is well-motivated and the experiments demonstrate it works effectively, but the experiments do not provide the rigorous causal evidence that would fully validate the paper's strongest interpretive claims about why it works better than alternatives.

6. Limitations and Trade-offs

6.1 The Difficulty Estimation Cost Is Not Accounted for in the Reported Efficiency Gains

The assumption or constraint. The core of MoGA's advantage — the learned routing mechanism that assigns tokens to semantic groups — requires the router to be trained end-to-end on multi-shot long video data. This training data, in turn, requires the sophisticated data pipeline described in Section 3.3. The pipeline relies on multiple external models (VQA models for aesthetics, clarity, and exposure; OCR models for watermark and subtitle detection; AutoShot and PySceneDetect for shot boundary detection; Qwen2.5-VL for captioning) and multiple processing stages (video-level filtering, shot-level filtering, captioning, multi-shot assembly with transition trimming). The paper does not account for the computational cost, engineering effort, or model dependencies of this pipeline in any of its efficiency calculations.

The paper acknowledges the pipeline's existence and describes it in detail, but does not frame it as a cost or limitation. The only explicit acknowledgement of the pipeline as infrastructure (rather than contribution) is the description of MoGA's primary contributions in Section 1, which omits the data pipeline from the list of contributions entirely.

The consequence. A practitioner seeking to apply MoGA to a new video domain (e.g., medical imaging, robotics video, sports footage) cannot simply fine-tune on available data — they must first construct a comparable multi-shot data pipeline. This requires sourcing and running multiple pretrained models, tuning their outputs, and designing filtering heuristics. The total computational and engineering cost of this pipeline is unmeasured and may rival or exceed the cost of MoGA training itself. The paper's headline efficiency gains (3× FLOPs reduction, 1.7× speedup) are computed relative to the model training and inference costs only, excluding the substantial one-time cost of preparing suitable training data. In domains where high-quality multi-shot data already exists (perhaps feature films with shot annotations), this limitation is minor. In domains where such data must be created from scratch, it may be prohibitive.

What evidence exists in the paper. None. The paper does not report the computational cost of running the data pipeline, the storage requirements for intermediate outputs, the wall-clock time for processing raw videos into training samples, or the failure rate (what fraction of raw videos pass all quality filters and produce usable training clips). The paper also does not ablate the necessity of the pipeline's sophistication — could a simpler pipeline (e.g., AutoShot-only segmentation, no OCR-based watermark removal, no captioning) produce training data of sufficient quality? Without this ablation (discussed in Section 5), the pipeline's necessity is assumed rather than demonstrated.

Mitigation status. Not addressed. The paper positions the data pipeline as supporting infrastructure and does not discuss its cost, engineering requirements, or dependency on external models. Section 8 (Conclusion) does not mention the pipeline as a limitation or area for future work. A practitioner would need to independently assess whether comparable multi-shot training data exists for their domain or build the pipeline from scratch, with no guidance from the paper on effort or cost.

---

6.2 The Hardest Shots — Where Cross-Shot Consistency Matters Most — Remain Effectively Unquantified at Scale

The assumption or constraint. The paper's evaluation of cross-shot consistency relies on two mechanisms: (1) VBench metrics (subject consistency, background consistency) computed by sampling frames across the video, and (2) cross-shot CLIP and DINO similarity scores computed by sampling "a fixed number of frames from different shots" (Section 4, Evaluation Metrics). Both mechanisms sample frames — they do not densely track consistency across every frame of every shot. The 30-second evaluation uses 11 scripts comprising 105 prompts (8–10 shots per script), and the 60-second evaluation is a single qualitative example with no quantitative metrics reported.

Furthermore, the paper's primary concurrency claim — that MoGA's semantic grouping preserves shot-spanning identity better than full attention — is supported by aggregate metrics that average across all shots and all videos. This averaging obscures the distribution of consistency: does MoGA perform well on most shots but fail catastrophically on a few? Or does it provide modest improvement across all shots? The distinction matters because long video generation is a weakest-link problem — a single inconsistent shot breaks the viewing experience regardless of how consistent the other shots are.

The paper does not report per-shot consistency metrics, worst-case consistency, or the variance of consistency scores across shots. The 60-second generation (22 shots, Figure 7) is a single example selected for presentation, with no indication of how representative it is.

The consequence. A practitioner deploying MoGA for production video generation cannot predict whether the model will maintain consistency for the most challenging shots in a sequence — the ones with large temporal gaps, occlusions, or lighting changes. Aggregate metrics may mask a bimodal distribution where MoGA performs excellently on 90% of shots but loses identity on the remaining 10%, which would be unacceptable for applications requiring reliable consistency (e.g., virtual production, character-driven storytelling). The single 60-second example is anecdotal — without quantitative benchmarks or failure analysis at this duration, the claim of "minute-level" generation is a capability demonstration, not a reliability guarantee.

What evidence exists in the paper. None that addresses the distribution of consistency. Table 2 reports aggregate subject consistency and background consistency for 10-second multi-shot generation (100 prompt sets). Table 3 reports the same for 30-second generation (11 scripts, 105 prompts). Neither table reports standard deviations, minimum/maximum values, or per-script breakdowns. The cross-shot CLIP and DINO metrics are introduced but their distributions (variance, worst-case values) are not discussed. Figure 7 shows a single 60-second example; there is no quantitative evaluation at this duration.

Mitigation status. Not addressed. The paper does not acknowledge the limitation of aggregate metrics for assessing consistency reliability, does not propose per-shot or worst-case evaluation protocols, and does not discuss the small sample size for the 30-second benchmark (11 scripts) as a statistical limitation. The 60-second generation is presented as a qualitative showcase without any accompanying quantitative evaluation or failure analysis. Future work on long video generation evaluation — particularly metrics that measure consistency degradation over increasing temporal gaps — would be needed to characterize the reliability boundaries that aggregate metrics cannot capture.

---

6.3 The Revision Model's Reversion Problem Has a Direct Analog: Shot Boundaries Require Explicit Conditioning to Prevent Scene Blending

The assumption or constraint. MoGA's architecture enforces a specific separation of responsibilities: STGA prevents local inter-shot attention by assigning frames from different shots to distinct temporal groups, and MoGA provides the only path for cross-shot information flow through its learned semantic groups. The paper also introduces shot-level textual conditioning via cross-modal attention, where each shot is guided by its own concise text description.

This architecture assumes that shot boundaries can and should be explicitly specified — both through the STGA isolation mechanism (which prevents the local attention from blending shot content) and through the shot-level text conditioning (which tells the model what each shot should contain). Without this explicit conditioning and architectural isolation, there is no mechanism to tell the model when one shot ends and another begins — the visual features alone do not encode shot boundary information.

The paper acknowledges this dependency implicitly when it states: "Extended context alone is insufficient because a single global prompt cannot reliably control scene transitions or orchestrate events at precise time points in long videos" (Section 1). The shot-level conditioning and the STGA shot isolation are both presented as solutions to this problem, but the dependency on these mechanisms is a constraint on MoGA's applicability: the model requires structured input (shot-segmented text descriptions) that may not be available in all use cases.

The consequence. MoGA cannot generate coherent multi-shot videos from a single global prompt — it requires per-shot textual descriptions. This means the model is not applicable to use cases where the user wants to specify a high-level narrative ("A day in the life of a cat") and have the model autonomously decide shot boundaries, scene compositions, and transitions. The paper's approach shifts the creative burden from the model to the user or an upstream planner: someone or something must segment the narrative into shots and write a description for each shot before MoGA can generate the video.

Additionally, the dependency on explicit shot conditioning means that errors in shot segmentation or description propagate directly to the generated video. If the upstream planner produces an awkward shot boundary (e.g., cutting in the middle of an action) or an ambiguous shot description, MoGA has no mechanism to override or refine it — the architecture enforces the shot structure it receives.

What evidence exists in the paper. The paper does not ablate shot-level conditioning. There is no experiment showing what happens if MoGA is given a single global prompt for a multi-shot video — would it fail to produce distinct shots? Would shots blend together? Would the model ignore the prompt structure? Without this ablation, we cannot assess how critically MoGA depends on explicit shot conditioning versus whether the semantic grouping could partially compensate. The paper's statement that "extended context alone is insufficient" is asserted, not experimentally demonstrated for MoGA specifically.

The qualitative results (Figures 5, 7, 8, 9, 10) all use shot-level conditioning, as this is the standard operating mode. The poor performance of STGA-only in Figure 11 (which limits long-range shot interactions and leads to "poor cross-shot consistency") demonstrates the necessity of MoGA for cross-shot consistency given the architectural shot isolation, but does not test the necessity of the shot isolation itself.

Mitigation status. Not addressed. The paper does not discuss the dependency on explicit shot conditioning as a limitation, does not propose mechanisms for autonomous shot boundary detection or generation, and does not experiment with less structured conditioning formats (e.g., a single narrative paragraph, timestamps within a global prompt). The shot-level conditioning is presented as a feature (enabling precise control) rather than a constraint (requiring structured input that may not be available). Future work on integrating MoGA with automated script-to-shot decomposition or on learning shot boundaries from visual features directly would address this limitation.

---

6.4 The Method Has Only Been Validated on a Single Domain (General Video) with Two Model Families, Leaving Generalization to Other Video Types and Architectures Uncharacterized

The assumption or constraint. All experiments in the paper use one of two DiT-based architectures (Wan2.1 at 1.3B and 14B; an MMDiT-based architecture) fine-tuned on general video data processed through the paper's custom pipeline. The evaluation uses one benchmark suite (VBench) plus the paper's custom cross-shot metrics, applied to general-domain video generation (people, scenes, animation style). The paper asserts MoGA is "a general sparse attention" (Section 4) and demonstrates its application to two different DiT variants, but does not test it on other video domains, other model architectures, or other generation paradigms.

The paper does not claim generalizability beyond what is tested — but it also does not acknowledge the scope limitation explicitly. The claim in the abstract that "comprehensive experiments on various video generation tasks validate the effectiveness of our approach" refers to various durations and settings (single-shot, multi-shot, 5–60 seconds, different styles) within the general video domain, not various domains or architectures.

The consequence. Several aspects of MoGA's design may be domain-specific in ways that are not tested:

• The semantic grouping learned by the router depends on the training data distribution. In general video data, "semantic" means things like "person identity," "object category," "background texture." In a different domain (e.g., medical ultrasound video, satellite imagery time series, molecular dynamics simulations), what constitutes a useful semantic group may be fundamentally different, and the router's capacity (a single linear layer with M=5–20 groups) may be insufficient or inappropriate.
• The sparsity of attention in videos — the claim that "most query–key pairs contribute little" — is domain-specific. In domains with global physical constraints (e.g., fluid dynamics simulations where every spatial point influences every other point through pressure fields), attention may be genuinely dense, and sparsification would degrade quality. The paper's assumption of inherent sparsity is validated on general videos but not on other video-like data.
• The STGA design (2D spatial windows, temporal grouping by shot) assumes a particular video structure (roughly static camera within shots, semantic continuity within temporal windows). For domains with rapid camera motion, frequent cuts, or non-standard frame rates, the fixed window sizes and shot-based temporal grouping may be misaligned with the data structure.
• The MMDiT-based architecture is a substantially different design from the Wan2.1 backbone (it uses MMDiT-style cross-modal attention instead of cross-attention, finer spatial partitioning, more groups), and the paper provides only limited quantitative evaluation of this variant (Table 3 for 30-second generation, qualitative 60-second example). We do not know whether MoGA's benefits transfer to other common DiT variants (e.g., U-Net-based DiTs, cascade models) or to non-DiT video generation architectures (e.g., autoregressive transformers, GAN-based generators).

What evidence exists in the paper. The paper tests two model families (Wan2.1 and MMDiT) and one domain (general video, including both realistic and animated styles). The consistency of results across the two architectures (MoGA outperforms baselines with both backbones in Table 3) provides some evidence of architectural robustness, but both are DiT-based and share fundamental design assumptions. The animation style results (Figure 9) provide some evidence of stylistic generalization within the general video domain. There is no evidence regarding generalization to other video domains, other model architectures, or other data modalities that share the "sequence of frames" structure.

Mitigation status. Not addressed as a limitation. The paper does not discuss domain or architectural generalization as an open question. Section 5 (Conclusion) states that "diverse experiments in video generation further demonstrate the effectiveness of our approach" but does not qualify the scope of "diverse" or suggest that testing in other domains is needed. The application of MoGA to two DiT variants is a reasonable start, but the leap from "works on these two architectures for general video" to "MoGA is a general sparse attention mechanism" requires broader validation that the paper does not provide.

---

6.5 Sequential Dependencies in the Data Pipeline and Training Curriculum Create an Unquantified Cold-Start Cost for New Domains

The assumption or constraint. Training a MoGA model for a new domain requires, in sequence: (1) sourcing raw long videos in the target domain, (2) running the multi-stage data pipeline (VQA filtering, shot detection, OCR-based watermark removal, MLLM captioning, multi-shot assembly with transition trimming), (3) fine-tuning from a pretrained short-video DiT checkpoint using a multistage curriculum (3,000 steps on 10-second clips, then 1,000 steps on 30-second clips), and (4) tuning hyperparameters including the number of groups M, the group balancing loss weight α, and the STGA spatial partition size. Each of these steps has dependencies on the previous step, and failure at any stage (e.g., the MLLM produces poor captions for domain-specific content) propagates forward.

The paper does not characterize the sensitivity of final video quality to any of these pipeline or curriculum choices. What if the 10-second pretraining stage is shortened or skipped? What if the caption quality is lower (e.g., using a smaller MLLM)? What if the shot detection is noisier (e.g., AutoShot-only, without PySceneDetect ensembling)? What if the watermark removal is omitted? The paper provides no guidance on which components of the pipeline are most critical versus which are nice-to-have.

Additionally, the paper does not report the yield of the data pipeline — what fraction of raw video footage survives all filtering stages and becomes usable training data. This yield determines how much raw video must be sourced to produce a training set of a given size, which directly affects the feasibility of applying MoGA to domains where video data is scarce or expensive (e.g., medical procedures, scientific experiments, sports analytics with licensing restrictions).

The consequence. A practitioner seeking to apply MoGA to a new domain faces substantial uncertainty about the required investment. They must commit to building the full data pipeline before knowing whether the resulting training data will be sufficient to train a high-quality MoGA model. If the pipeline yield is low (e.g., 10% of raw footage passes filtering), they may need 10× more raw footage than anticipated, which could be prohibitively expensive or impossible to obtain. If the curriculum is critical (skipping the 10-second stage degrades quality substantially), they must budget for the full multistage training, which increases compute cost. Without ablations of pipeline components and curriculum stages, practitioners cannot make informed decisions about where to invest effort versus where to cut corners.

What evidence exists in the paper. None that quantifies pipeline yield, component sensitivity, or curriculum necessity. The paper describes the pipeline and curriculum as implemented but does not report: the fraction of raw videos retained after video-level filtering; the fraction of single-shot clips retained after shot-level filtering; the number of raw video hours required to produce the training set used in experiments; the quality of captions produced by Qwen2.5-VL (e.g., human evaluation of caption accuracy); or any ablation of pipeline components (e.g., training with AutoShot-only vs. AutoShot+PySceneDetect segmentation). The multistage curriculum (3k steps at 10 seconds, 1k steps at 30 seconds) is stated without justification or ablation — we do not know whether a single-stage curriculum on 30-second clips would work, or whether the 10-second stage is essential for convergence.

Mitigation status. Not addressed. The paper presents the data pipeline as a fixed recipe and the training curriculum as a given schedule, without discussing sensitivity, failure modes, or cost. Section 5 (Conclusion) does not mention the pipeline complexity or curriculum design as areas needing further study. For the paper's contribution — demonstrating that MoGA works given suitable training data — this is acceptable. For a practitioner seeking to replicate or adapt the method, the unquantified pipeline and curriculum dependencies represent a significant practical risk.

---

6.6 The Reported Wall-Clock Speedup Is 1.7×, Not the 3× Suggested by FLOPs Reduction, and Latency-Critical Applications May Be Constrained by Serial Dependencies

The assumption or constraint. The paper reports two efficiency metrics: a ~3× FLOPs reduction (2.26 PFLOPs vs. 6.94 PFLOPs for 30-second videos under Wan2.1-1.3B, Table 5) and a 1.7× wall-clock speedup in both training and inference (Section 4.3). The gap between these numbers — FLOPs reduction is nearly 2× larger than the wall-clock speedup — is attributed implicitly to overhead from the permutation/unpermutation operations in Algorithm 1, non-uniform group sizes (which reduce the effective parallelism of attention computation), and memory bandwidth limitations that prevent the FLOPs reduction from translating linearly to wall-clock time.

The paper emphasizes MoGA's kernel compatibility (FlashAttention, sequence parallelism) and zero additional memory overhead as practical advantages, but does not characterize the latency implications of grouped attention. The permutation step in Algorithm 1 requires a global synchronization: all tokens must be routed before any attention can be computed, and the output must be unpermuted before the next layer. Additionally, the router computation, while lightweight (a single linear layer), adds a serial step before the attention computation that is not present in full attention.

For the Wan2.1-1.3B model with ~187k tokens (30 seconds), the permutation overhead and router computation are amortized over the large attention savings. For shorter sequences or smaller models where attention is less dominant in the overall runtime, the relative overhead may be larger, reducing or eliminating the speedup. The paper only reports speedup for the specific model and sequence length used in the main experiments — the scaling of overhead with sequence length and model size is not characterized.

The consequence. For interactive or real-time applications (e.g., video editing tools with live preview, streaming video generation), the 1.7× speedup may be insufficient to meet latency requirements, even if the FLOPs reduction is substantial. The serial dependencies in MoGA — routing before attention, unpermuting after attention — mean that the critical path length is not reduced by the same factor as the total FLOPs. A full-attention model could theoretically be parallelized more aggressively (all N² attention computations are independent), while MoGA introduces a sequential bottleneck at the group assignment stage.

Additionally, the gap between FLOPs reduction and wall-clock speedup means that the theoretical scaling with M (more groups → proportionally less FLOPs) may not translate to proportional wall-clock improvements. As M increases, groups become smaller, and the permutation becomes more fine-grained (more groups to sort tokens into), potentially increasing the relative overhead. The paper does not report wall-clock speedups for different values of M, so we cannot assess whether the FLOPs-to-wall-clock efficiency gap widens or narrows as M grows.

What evidence exists in the paper. The 1.7× training and inference speedup is reported in Section 4.3 without further breakdown (e.g., what fraction of the remaining time is spent in attention vs. routing vs. permutation?). Figure 6 reports FLOPs scaling with M and video duration, but not wall-clock time. Table 5 reports PFLOPs for different (M, duration) pairs, but not corresponding wall-clock benchmarks. The paper acknowledges the gap between FLOPs and wall-clock implicitly by reporting both numbers, but does not analyze the sources of overhead or characterize their scaling behavior. The paper does not report latency (time-to-first-frame or time-to-completion) as distinct from throughput (frames per second of generation).

Mitigation status. Not addressed. The paper reports the 1.7× speedup as a positive result alongside the 3× FLOPs reduction, without noting the gap as a limitation or analyzing its causes. For batch processing or offline generation (where throughput matters more than latency), a 1.7× speedup is genuinely useful. For latency-sensitive applications, the uncharacterized overhead and serial dependencies may make MoGA less attractive than the FLOPs numbers suggest. The paper does not propose optimizations to close the FLOPs-to-wall-clock gap (e.g., fusing the permutation with the attention kernel, overlapping routing with previous layer computations) or characterize the latency implications of the group assignment mechanism.

7. Implications and Future Directions

How This Work Changes the Landscape

MoGA introduces a categorical reframing of sparse attention for video generation — it is not an incremental improvement within the block-level paradigm but a rejection of that paradigm's central premise. The field had converged on a specific assumption: that efficient sparse attention requires first estimating importance at a coarse granularity (blocks), then refining to token-level computation within selected blocks. This assumption was so embedded that the research conversation — as reflected in MoBA, VMoBA, SVG2, and related work — focused entirely on tuning the block size and top-k selection to navigate the resulting accuracy–efficiency trade-off. MoGA's core move is to demonstrate that this trade-off is not a property of sparse attention but an artifact of block-level approximation, and that per-token learned routing can simultaneously improve both efficiency and generation quality by forming semantically coherent attention groups directly.

The magnitude of this shift is methodological rather than paradigmatic. MoGA does not change what video generation models do (they still denoise latent representations via DiTs with attention) or what the field values (consistency, fidelity, efficiency). But it changes how researchers should think about designing attention mechanisms for structured sequence data: not as a problem of approximating full attention at lower cost, but as a problem of architecting different attention pathways for different types of token interactions, where the grouping mechanism itself is learned end-to-end rather than engineered through geometric heuristics.

Several specific shifts follow from this reframing:

From spatial selection to semantic selection. Block-based methods implicitly assume that attention relevance correlates with spatiotemporal proximity — tokens in nearby blocks are more likely to be relevant. MoGA makes no such assumption. The router groups tokens that are mutually beneficial for the denoising objective, regardless of where they appear in the video grid. Figure 3's visualization — where the router groups "the man's head, hands, and portions of his clothing" across shot boundaries — is not a cherry-picked example but a direct consequence of the mechanism. This suggests that future sparse attention research should focus on what tokens represent rather than where tokens are located, which opens connections to representation learning, self-supervised clustering, and prototype-based routing that the block-level paradigm could not explore.

The accuracy–efficiency trade-off is dissolved, not optimized. Prior work (VMoBA, MoBA) presented the block size as a tunable knob on a Pareto frontier: smaller blocks improve accuracy at the cost of efficiency, larger blocks do the reverse. MoGA eliminates the block size knob entirely. The efficiency comes from the factor of M reduction in attention group size (O(N²/M) vs. O(N²)), not from selectively dropping query–key pairs. This means the efficiency gain is architectural rather than selective — every token computes attention within its group, and no relevance estimation or top-k filtering is required. The practical consequence is that MoGA's efficiency is more predictable and less dependent on the accuracy of a coarse scoring stage, which the paper shows can fail confusingly (Figure 1b).

Full attention is reframed as potentially harmful, not ideal. The paper provides evidence that MoGA can match or exceed full attention on consistency metrics (Tables 1, 2), and offers an explicit mechanism: full attention computes all N² interactions, including semantically meaningless ones, which can dilute the softmax normalization and introduce noise. This is a provocative claim — it suggests that sparse attention is not a compromise forced by computational constraints but a positive design choice that reflects the structure of video data. If this claim holds under broader testing, it would invert the default assumption in the field from "use full attention unless you can't afford it" to "use structured sparse attention unless you have evidence that your domain requires dense interactions."

The paper also helps reconcile conflicting intuition in the sparse attention literature. Prior methods that used block-level selection (VMoBA, SVG) achieved efficiency but struggled with long-range dependencies. Methods that used full attention (EchoShot, Wan) achieved consistency but were computationally prohibitive for long videos. MoGA demonstrates that long-range consistency and computational efficiency are not opposing goals — the architectural separation of semantic (MoGA, long-range) and spatiotemporal (STGA, local) attention provides both simultaneously. This resolves the tension by showing that the apparent trade-off was an artifact of using a single attention mechanism for both types of interactions.

Finally, MoGA's demonstration that a single linear layer with M=5–20 groups can learn useful semantic routing — without explicit clustering, without auxiliary semantic losses, without pretrained feature extractors — suggests that the inductive bias of grouped attention is stronger than previously recognized. The router converges to interpretable semantic groupings (Figure 3) solely from the denoising gradient flowing through the attention computation. This has implications beyond video: any domain where attention is sparse and content-dependent (long-context language modeling, protein folding, graph neural networks) might benefit from replacing block-level or heuristic sparsity with learned per-token routing trained end-to-end through the primary task objective.

The paper does not resolve which aspects of MoGA are essential versus incidental. The training-free vs. trained distinction is confounded with the routing mechanism distinction in the VMoBA comparison. The shot-level conditioning dependency, the data pipeline complexity, and the STGA design choices are not ablated separately. As a result, MoGA opens a design space but does not fully characterize its boundaries — this is appropriate for a method paper introducing a new mechanism but means the landscape shift is direction-setting rather than definitive.

---

Follow-Up Research This Work Enables

Direct FLOPs-matched comparison between learned per-token routing and block-level routing with training controlled. The paper's central claim — that block-level coarse estimation is unnecessary — cannot be fully validated without isolating the routing mechanism from other confounds. A controlled experiment would: take the same base DiT checkpoint (e.g., Wan2.1-1.3B), train two variants with matched FLOPs budgets — one using MoGA (M groups, trained router) and one using VMoBA-style block routing (with comparable sparsity, but trained end-to-end rather than applied training-free), both on the same multi-shot data pipeline with the same curriculum — and compare on VBench metrics and cross-shot consistency scores. If MoGA still outperforms, the claim is supported. If the trained block-level method closes the gap, then the advantage is from end-to-end training rather than the routing mechanism per se. If the trained block-level method surpasses MoGA, then the field should revisit block-based routing with learned block scoring rather than adopting per-token routing. This experiment would also clarify whether the block size hyperparameter can be eliminated entirely (as MoGA claims) or merely shifted to a different design choice (number of groups M vs. block size).

Characterizing the scaling behavior of MoGA's cross-shot consistency with increasing temporal gaps. The paper reports aggregate cross-shot consistency metrics (Tables 2, 3) but does not analyze how consistency degrades as the temporal distance between shots increases. A targeted experiment would: generate videos with controlled temporal gaps (e.g., a character appears in shot 1, then reappears after 2, 5, 10, or 20 intervening shots), measure identity consistency (CLIP/DINO similarity of the character's features across the gap) for MoGA vs. full attention vs. block-level sparse methods, and plot the consistency vs. gap curve. The key question is whether MoGA's semantic grouping actually preserves identity across arbitrary gaps (because the router groups all instances of the same character together regardless of temporal distance) or whether consistency still degrades with gap but more slowly than alternatives. If MoGA's consistency is gap-independent, that would be strong evidence for the semantic grouping mechanism working as claimed. If it degrades, that reveals a boundary condition for the method and suggests that larger M or multi-router designs might be needed for very long videos with widely separated character reappearances.

Router capacity and architecture ablation: Is a single linear layer sufficient, or would deeper routers, multi-head routing, or top-k soft assignment improve grouping? MoGA uses the simplest possible router: one linear layer, hard argmax assignment, M=5–20 groups. This simplicity is elegant but raises the question of whether more expressive routers could improve semantic grouping quality. Concrete experiments: (a) a 2-layer MLP router vs. the current linear router, measuring both VBench metrics and the interpretability of learned groups (via visualization as in Figure 3); (b) multi-head routing where each attention head has its own router (currently, the paper computes routing scores "over tokens with whole heads," meaning one routing assignment is shared across all heads — this may force heads with different attention patterns to use the same groups); (c) soft top-k assignment where each token attends to its top-2 groups (weighted by routing probability) rather than a single hard group, trading off some efficiency for potentially better coverage of tokens that span semantic boundaries. The group balancing loss (Equation 6) assumes uniform assignment is optimal — testing whether non-uniform assignment (e.g., allowing some groups to be larger for more common semantic categories) improves quality without the load balancing penalty would test whether the balancing loss is a necessary regularization or a constraining simplification.

Stress-testing MoGA on domains where attention is genuinely dense — when does semantic sparsity fail? The paper's central assumption is that video attention is inherently sparse. This assumption needs to be tested at its breaking point. Construct a synthetic video dataset where attention must be dense for correct generation: e.g., videos of coupled physical systems (multiple pendulums connected by springs, fluid flow with global pressure constraints) where every spatial location influences every other location through physical laws. Train MoGA and full-attention DiTs on this data. The prediction: MoGA should underperform full attention because its grouped structure prevents necessary dense interactions. Measuring exactly how much it underperforms and whether the degradation is graceful (modest quality loss) or catastrophic (model cannot learn the dynamics at all) would characterize the boundary of MoGA's applicability. This experiment would also test whether the router can learn to assign tokens to groups in a way that approximates dense attention (e.g., by making groups very large, effectively reverting to full attention) or whether the group balancing loss prevents this adaptation. If MoGA fails catastrophically, it would establish that the method should not be applied blindly to any video domain — a practitioner would need evidence of attention sparsity before adopting MoGA.

Router interpretability as a diagnostic tool for video generation models. Figure 3 demonstrates that the learned groups are human-interpretable (grouping body parts of the same person). This suggests a research direction: can the router's grouping assignments be used to diagnose consistency failures in video generation? For example, if a character's identity drifts across shots, does the router still assign the drifted and original tokens to the same group (implicating the attention computation within the group as the failure point) or does it assign them to different groups (implicating the router as the failure point)? A diagnostic tool that visualizes per-frame group assignments for a generated video and highlights where group membership changes unexpectedly could help model developers identify why a particular video has consistency failures — is the router failing to recognize semantic continuity, or is attention failing to maintain identity even when tokens are correctly grouped? This would transform MoGA's router from a mechanism for efficiency into a mechanism for interpretability and debugging, which is an underexplored application of learned routing in generative models.

Extending MoGA to autoregressive video generation and other sequence modeling paradigms. MoGA is developed and tested exclusively on DiT-based diffusion models with the rectified flow objective. Autoregressive video generation (MAGI-1, FAR) is an increasingly competitive paradigm that faces the same quadratic attention bottleneck. Applying MoGA to an autoregressive video transformer — where the router must operate causally (tokens at time t can only be grouped with tokens from times ≤ t, unless bidirectional attention is used within segments) — would test whether the semantic grouping mechanism transfers across generation paradigms. The key adaptation would be: does the router still learn useful semantic groups when the attention is causal, or does the causal mask restrict the formation of groups that span shot boundaries (since future tokens cannot influence past group assignments)? If MoGA works causally, it would enable efficient autoregressive generation of minute-scale videos. If it fails, that reveals a fundamental coupling between MoGA's grouping and bidirectional attention that would motivate research into causal semantic routing mechanisms. This direction also connects to the broader question of whether learned sparse attention patterns can be applied to any Transformer-based sequence model (language, audio, biological sequences) or whether they are specifically suited to generation tasks where bidirectional context is available during training.

---

Practical Applications and Downstream Use Cases

Cost-efficient long-form content creation for independent creators and small studios. MoGA with the Wan2.1-1.3B backbone reduces FLOPs for 30-second video generation from 6.94 PFLOPs to 2.26 PFLOPs (a ~3.1× reduction) and delivers a 1.7× wall-clock speedup. For a content creator generating a 5-minute video (ten 30-second segments), full attention would require ~69 PFLOPs per segment, or ~690 PFLOPs total — a compute budget accessible only to organizations with large GPU clusters. MoGA reduces this to ~226 PFLOPs total, bringing minute-scale video generation within reach of a single high-end GPU (e.g., an H100 can deliver ~1 PFLOP/s of effective throughput for attention-dominated workloads, making 226 PFLOPs feasible in a few minutes rather than tens of minutes). The shot-level text conditioning interface (each shot described by a concise caption) maps naturally to storyboard-based workflows already used in animation and film pre-production, where a director writes per-shot descriptions before any animation begins. MoGA could serve as a rapid prototyping tool: a creator writes 10–20 shot descriptions, MoGA generates a draft 30–60 second video in minutes, and the creator iterates on the shot descriptions rather than on individual frames. The 71.25% sparsity figure (Section 4.3) suggests that the model can handle appearance changes specified in the shot descriptions (e.g., "the character now wears a red jacket") while maintaining facial identity — a capability that is essential for narrative video but difficult to achieve with image-based or single-shot methods.

Automated data augmentation for video understanding models. Training robust video understanding models (action recognition, temporal localization, video question answering) requires diverse training data with controlled variations in background, lighting, camera angle, and character appearance. Current data augmentation techniques (spatial transforms, color jitter, frame dropping) cannot produce the semantic variations needed for robustness — they cannot change a character's clothing while preserving their identity, or change the background while preserving the action. MoGA's demonstrated ability to generate multi-shot videos with controlled appearance changes (Figure 10: same character, different outfits in different shots, specified by shot-level text) enables a new form of semantic data augmentation. A practitioner could: (1) take a training video of an action, (2) write a series of shot descriptions that vary background ("in a kitchen" → "in a garden"), lighting ("morning" → "afternoon"), and character attributes while specifying that the action remains the same, (3) use MoGA to generate variations, and (4) train the video understanding model on the augmented dataset. The key advantage over existing augmentation is that the variations are semantically controlled through the shot-level text interface — the generated videos preserve the action semantics while varying irrelevant attributes, which is precisely the variation pattern that improves model robustness. The 1.7× speedup over full attention makes this pipeline practical for generating thousands of augmented videos, which would be prohibitively expensive with full-attention generation at the same duration.

Enabling on-device or edge-deployed long video generation through model compression synergies. MoGA's efficiency comes from reducing attention computation (the dominant cost in DiTs for long sequences), which is complementary to other efficiency techniques that reduce parameter count or feature dimension (quantization, pruning, distillation). A compressed model (e.g., Wan2.1-1.3B quantized to INT8, or distilled to a smaller architecture) combined with MoGA's sparse attention could potentially enable minute-scale video generation on consumer GPUs or edge devices. The paper's memory efficiency claim — "no additional memory consumption" compared to full attention — is critical here because memory (not just FLOPs) is the bottleneck for edge deployment. A block-based sparse method like VMoBA incurs additional memory for block indices and importance caches, which could push a memory-constrained edge device over its limit even if the FLOPs budget is sufficient. MoGA's permute-based implementation (Algorithm 1) adds no persistent memory overhead beyond the router parameters (M × d additional weights, negligible compared to the model size). A practical experiment: take the Wan2.1-1.3B MoGA model, apply INT8 quantization, and benchmark 30-second generation on a single RTX 4090 (24 GB VRAM). If MoGA's memory efficiency enables generation within 24 GB that full attention cannot fit, the combination of MoGA + quantization could make long video generation accessible on hardware that is currently limited to short (<5 second) clips. The paper's specific numbers — 2.26 PFLOPs for 30 seconds, 187k tokens — provide the baseline for such an experiment.