ArXiv: 2601.20499

🎯 Pitch

A quarter of attention heads in autoregressive video diffusion models ignore history almost entirely, yet naïvely pruning them causes quality collapse. By adaptively classifying heads into dedicated roles—sink, neighbor, or dummy—and packing one extra frame into dummy-head context, this paper achieves 2× inference speedup with under 0.5% quality loss, enabling real-time 24 FPS video generation without any retraining.

1. Executive Summary

This paper identifies and exploits a structural inefficiency in autoregressive video diffusion models—approximately 25% of attention heads attend almost exclusively to the current frame rather than aggregating historical context—and proposes Dummy Forcing, a training-free inference acceleration method built on Heterogeneous Memory Allocation (assigning head-specific context lengths: sink-only, neighbor-window, or current-frame-only), Dynamic Head Programming (a greedy optimization that adaptively classifies heads by maximizing retained attention scores), and Packed Attention Forward (extending dummy-head context by one packing frame to shift the classification boundary and enable more aggressive pruning). Evaluated on Self Forcing and LongLive models across VBench and VBench-Long benchmarks, Dummy Forcing delivers up to 2.0× end-to-end speedup with less than 0.5% quality drop—achieving real-time generation at 24.3 FPS on short video and 1.4× acceleration on 30-second long video—while also enabling 6.58× longer effective cache for long-context generation, establishing that head-wise context redundancy can be compressed aggressively without retraining only when heads are classified adaptively rather than naïvely by a fixed ratio.

2. Context and Motivation

The Core Problem: Autoregressive Video Diffusion Is Slow Despite Caching

The fundamental problem this paper tackles is deceptively simple: autoregressive video diffusion models, despite their architectural support for KV caching, remain inefficient at inference time because their internal attention mechanisms waste substantial computation on redundant historical context. This matters because video generation is one of the most compute-intensive tasks in modern machine learning, and autoregressive models—while theoretically more efficient than bidirectional ones—still struggle to achieve interactive or real-time generation speeds.

To understand why this is significant, we need to grasp the architectural shift that has occurred in video generation. Early video diffusion models (like those built on the DiT architecture) process all video frames simultaneously using bidirectional attention: every frame attends to every other frame during the denoising process. This has two major drawbacks. First, users must wait until the entire video is generated before seeing any output—there's no streaming, no preview, no interactivity. Second, the attention computation scales quadratically with the total number of frames, making long video generation prohibitively expensive. If a video has TT frames with HWHW tokens each, bidirectional attention computes O((T⋅HW)2)O((T \cdot HW)^2) operations.

Autoregressive video diffusion models (Chen et al., 2024; Yin et al., 2025; Huang et al., 2025) address both problems by decomposing video synthesis into a frame-by-frame process. At each autoregressive (AR) step, the model generates one frame (or a small chunk of frames) conditioned on all previously generated frames. The key advantage is that past frames, once generated, can be stored in a KV cache—their key and value representations from the self-attention layers are computed once and reused across all subsequent steps. This transforms the generation process from batch processing to streaming: frames appear sequentially, and the computational cost per step grows linearly with the number of cached frames rather than quadratically with all frames.

However, linear growth still becomes unwieldy quickly. In a 30-second video at 24 FPS, after generating 720 frames, the KV cache contains representations for all previous 719 frames. Even with the sliding window strategy that most current methods adopt—retaining only the most recent LL frames plus one "sink frame" (a fixed anchor frame that provides global context via the attention sink mechanism)—the cache length per AR step remains substantial. For a typical sliding window of L=36L = 36 frames, each self-attention layer must process queries against keys and values from 36 cached frames plus the current frame, and this computation repeats across all layers of a deep transformer. When generating high-resolution video (720P, 1080P), the number of visual tokens per frame grows quadratically with resolution, making the problem even more acute.

The paper frames this efficiency challenge explicitly in Section 1:

"existing methods still face efficiency challenges when processing long visual token sequences. For instance, the KV cache length for past frames increases significantly in computation-dense tasks such as long videos or high-resolution videos."

This is not merely an inconvenience. It sets hard practical limits on deployment scenarios: real-time interactive video generation (where sub-second latency is expected), high-resolution synthesis (where token counts explode), and long-form storytelling (where the model needs access to distant history to maintain visual consistency across scene transitions).

Why Prior Efficiency Approaches Fall Short

The paper identifies three categories of prior work that attempt to address video generation efficiency, each with significant limitations that Dummy Forcing overcomes.

Input-level windowing is a black box. Most current autoregressive video diffusion systems (Yang et al., 2025a; Liu et al., 2025; Millon, 2025) employ a sliding window strategy that restricts the model's attention to only the most recent LL frames plus a sink frame. This is an architectural constraint applied uniformly at the input level—the model simply cannot see beyond the window boundary. While effective at bounding computational cost, this approach treats all historical information as equally important (or equally disposable), making no distinction between heads that genuinely need long-range context and those that don't. As the paper puts it:

"the model's internal utilization on contextual frames still remains a black box and has been largely unexplored."

The consequence is that the window size must be chosen conservatively: make it too small, and the model loses temporal consistency; make it too large, and inference becomes slow. There's no mechanism for differential allocation where some attention heads get long context and others get none.

LLM KV cache compression methods don't translate well to video. The KV cache pruning literature from large language models has developed sophisticated token-level compression strategies. Methods like H2O (Zhang et al., 2023) identify "heavy hitter" tokens based on cumulative attention scores and preserve only the most important ones. StreamingLLM (Xiao et al., 2023) retains initial "sink tokens" plus a recent window. DuoAttention (Xiao et al., 2024) classifies heads into "retrieval heads" (which need full context) and "streaming heads" (which only need recent tokens and attention sinks), compressing the latter. FastGen (Ge et al., 2023) proposes even finer-grained per-head context lengths.

However, as the paper demonstrates through direct comparison in Table 2, these methods underperform on video for two critical reasons:

  1. Per-step token selection overhead: Methods like R-KV (Cai et al., 2025) and Infinipot-V (Kim et al., 2025) compute token importance scores at every AR step to decide which tokens to evict. The paper reports that this additional computation "undermines the benefits gained from reduced cache length, resulting in a marginal 1.1× overall speedup ratio." The selection algorithm's runtime nearly cancels out the attention savings.

  2. Frame-level redundancy, not token-level: LLM compression methods operate at the granularity of individual tokens—they might keep some tokens from a paragraph while discarding others. But in video, the paper's key insight is that certain attention heads don't meaningfully use any historical tokens at all. As the paper explicitly states:

"we find that KV cache compression for video models can be more aggressive, with the cache of dummy heads potentially all removed."

Token-level methods that carefully select which tokens to keep within each head are solving the wrong granularity of problem. The real opportunity is at the head level: identify which heads need history and which don't, then prune aggressively on the latter.

Video-specific sparse attention methods aren't designed for autoregressive models. Prior work on accelerating video generation has focused on sparse attention patterns for bidirectional diffusion transformers, where attention masks are fixed and known in advance. Methods like Sparse VideoGen (Xi et al., 2025) introduce separate spatial and temporal heads with predetermined sparsity patterns. However, in autoregressive models, the attention mask shifts from bidirectional to causal, and the context length varies with each AR step (growing as more frames are generated). As the paper notes:

"This makes existing video sparse attention methods difficult to apply directly, as variable-length attention would require recompiling kernels at each step."

Kernel compilation is a one-time cost in bidirectional models where the attention shape is constant. In autoregressive models, recompiling at every step would dominate runtime.

Diffusion step skipping is orthogonal but limited. Methods like TeaCache (Liu et al., 2024) accelerate generation by skipping certain denoising timesteps—they observe that the model's predictions converge after a few steps and the later steps can be reused. The paper acknowledges this approach but shows in Table 2 that "the acceleration gain is limited given that current base models are already few-step diffusion models." When the denoising schedule already uses only 4–8 steps, there's simply not much to skip. This makes TeaCache complementary to Dummy Forcing rather than competitive—the paper demonstrates their combination in Table 7.

The Gap This Paper Identifies: Nobody Has Looked Inside the Attention Heads

The unifying shortcoming across all prior work is that no one has systematically studied how individual attention heads utilize historical context in autoregressive video diffusion models. The self-attention mechanism is treated as a monolithic operation—either all heads see the same window, or compression decisions are made token-by-token without considering that some heads might not need any history at all.

This is the specific gap the paper fills. Section 3.2 presents a detailed profiling study of attention head behavior, asking a question that seems obvious in retrospect but had been unexplored: when attending to past frames, do all heads contribute equally, or do some heads dominate while others are effectively dormant?

The answer—that approximately 25% of heads allocate over 80% of their attention to the current frame, and that their historical KV caches can be pruned with only 0.26% accuracy loss—is both surprising and actionable. It reveals that pre-trained autoregressive video diffusion models have learned a shortcut: they specialize certain heads for context aggregation (attending heavily to sink and neighbor frames) while leaving other heads to focus almost exclusively on refining the current frame. The paper names these latter heads "dummy heads" —a term that captures their functional role: they exist in the architecture but don't meaningfully participate in the cross-frame information flow that justifies the KV cache's computational cost.

This observation fundamentally reframes the efficiency problem. Rather than asking "which tokens can we discard?" (the LLM compression approach) or "how small can we make the window?" (the sliding window approach), the paper asks "which heads need any cache at all?" This enables a level of compression—removing all KV cache for entire categories of heads—that prior methods couldn't consider because they weren't looking at head-wise utilization patterns.

Why This Matters: The Practical Stakes

The paper situates its contribution at the intersection of several high-impact deployment scenarios, each of which is blocked by current efficiency limitations:

Real-time interactive video generation. If a model can generate video at 24 FPS or faster, it can produce output at display rate—creating the possibility of truly interactive video experiences where users' inputs modify the generation in real time. The paper's baseline Self Forcing model achieves 17.6 FPS, which is below this threshold for smooth playback. Dummy Forcing pushes this to 24.3 FPS, crossing the real-time barrier without quality degradation.

High-resolution video synthesis. Modern displays are 1080P or higher, but autoregressive video diffusion models are typically developed and evaluated at lower resolutions (e.g., 480P) because the quadratic token growth with resolution makes high-resolution generation extraordinarily expensive. At 1080P, the number of visual tokens per frame can be 16× that of a 480P frame. The paper's experiments on 720P and 1080P video show that Dummy Forcing's speedup increases with resolution—reaching 2.0× at 1080P—because the savings from pruning dummy heads' caches scale with the number of tokens being processed.

Long-context video generation for narrative consistency. In storytelling scenarios with scene transitions, models need to maintain visual memory of characters and objects that may disappear and reappear across shots. Current sliding window approaches can only look back ~36 frames (roughly 1.5 seconds at 24 FPS), which is insufficient for maintaining identity across longer gaps. The paper's long-context experiments in Section 4.4 demonstrate that Dummy Forcing can redirect the cache budget saved from dummy and sink heads to neighbor heads, achieving 6.58× longer effective cache at similar or better speed—directly enabling better shot-to-shot consistency without additional computational cost.

Deployment on constrained hardware. All experiments in the paper are conducted on a single H100 GPU. Achieving real-time or near-real-time video generation on a single consumer-grade or edge device remains far out of reach. Methods that improve efficiency without retraining—as Dummy Forcing does—lower the barrier to deployment by reducing the hardware requirements for a given quality-speed tradeoff.

How This Paper Positions Itself

The paper explicitly frames its contribution not as a new model architecture or a new training paradigm, but as a training-free inference optimization enabled by a previously overlooked structural property of autoregressive video diffusion models. The key positioning moves are:

It is an analysis-first contribution. Unlike many efficiency papers that start with a proposed method and then evaluate it, Dummy Forcing begins with a detailed empirical study of attention head behavior (Section 3.2) that establishes the existence, stability, and performance impact of dummy heads before any method is introduced. The three observations—that certain heads under-utilize past context, that dummy head positions are stable across conditions, and that pruning their caches incurs only slight degradation—form the evidentiary foundation that makes the subsequent method design feel inevitable rather than speculative.

It generalizes across model families and training paradigms. The paper demonstrates dummy heads not just in Self Forcing (Huang et al., 2025) but also in CausVid (Yin et al., 2025, built on Diffusion Forcing) and Rolling Forcing (Liu et al., 2025). Appendix B provides attention score distributions showing the same pattern—consistent fractions of heads with >0.8 attention on the current frame—across these architecturally distinct models. This suggests that dummy heads are a convergent property of autoregressive video diffusion training, not an idiosyncrasy of any particular implementation. The paper speculates in Section 5 that this arises from a natural division of labor: shallow layers aggregate current-frame information into high-level features, intermediate layers query past frames in this abstracted space, and deep layers refine the current frame for decoding—with dummy heads concentrated in the first and last few layers (Figure 9).

It is orthogonal to and compatible with other acceleration methods. Rather than competing with diffusion step skipping or other inference optimizations, the paper positions Dummy Forcing as operating on a different axis (attention head structure vs. denoising schedule) and demonstrates this compatibility explicitly in Table 7, where combining Dummy Forcing with TeaCache achieves over 30 FPS—faster than either method alone.

It doesn't claim to solve the fundamental capability ceiling. The paper is careful not to overstate what efficiency gains can achieve. The dummy head phenomenon reflects an architectural inefficiency in how models use their capacity, not a limitation in what they can represent. Pruning dummy caches doesn't make the model more capable—it makes it faster at its existing capability level. This is why the paper's long-context experiments are particularly important: by reallocating saved cache budget to non-dummy heads, the method actually improves functional capability (longer effective memory) rather than just accelerating computation, demonstrating that the efficiency isn't just about doing less work—it's about doing the right work.

A Note on Terminology: Why "Dummy" Heads?

The paper's naming choice is deliberate and informative. These heads are not "broken" or "useless" —they still perform computation (the current-frame self-attention), and they contribute to the model's output. They are "dummy" in a specific sense: they are non-participatory in cross-frame context aggregation. Their attention scores to historical frames are so low that the KV cache they receive from past frames is essentially wasted computation—the attention-weighted sum over keys and values is dominated by the current frame's contribution regardless of what's in the cache. This distinguishes them from other head classification schemes in the literature (like DuoAttention's retrieval/streaming split) where even the compressed heads still retain some access to historical tokens. A dummy head's cache can be entirely removed, reducing it to self-attention within the current frame only, with negligible quality impact.

This framing sets up the method's central design tension that the paper navigates through its three components: if 25% of heads can have their caches removed with almost no penalty, what about 50%? What about 75%? The answer—developed through Heterogeneous Memory Allocation, Dynamic Head Programming, and Packed Attention Forward—is that aggressive pruning is possible, but only if you (1) subdivide the remaining functional heads into specialized types (sink vs. neighbor), (2) make classification decisions adaptively rather than by a fixed threshold, and (3) provide a minimal safety margin (the packing frame) to prevent the pruning boundary from cutting through context-critical heads.

3. Technical Approach

3.1 Reader orientation (approachable technical breakdown)

Dummy Forcing is a training-free inference acceleration method that restructures how autoregressive video diffusion models manage their key-value (KV) caches by identifying and selectively compressing attention heads that do not meaningfully aggregate historical frame information. It solves the problem of KV cache redundancy—about 25% of attention heads attend almost exclusively to the current frame, wasting computation on cached history—by adaptively classifying heads into three specialized types (dummy, sink, neighbor), assigning each type a tailored context length, and executing them through fused attention calls to achieve up to 2.0× end-to-end speedup without additional training.

3.2 Big-picture architecture (diagram in words)

The Dummy Forcing system operates as a pre-processing and execution modification layer inserted into the standard autoregressive video diffusion inference pipeline. It has three major components, applied in sequence:

  1. Attention Profiling (one-time per video shot): The system samples attention maps from the pre-trained model at a representative AR step and denoising timestep, computing per-head "frame attention scores" that quantify how strongly each head distributes attention across sink frames, neighbor frames, and the current frame. This profiling runs once and produces a matrix $\mathcal{F} \in \mathbb{R}^{\text{num\_head} \times 3}$ capturing each head's attention allocation pattern.

  2. Dynamic Head Programming (DHP): Taking $\mathcal{F}$ and a target dummy head count $N$ as input, this optimization module solves a constrained maximization problem to classify every head as either dummy, sink, or neighbor. The classification maximizes the total retained attention score—heads that heavily weight the current frame become dummy candidates; heads that heavily weight sink frames become sink heads; heads that heavily weight neighbor frames become neighbor heads. This produces an index mapping $c_h \in \{\text{sink}, \text{neighbor}, \text{dummy}\}$ for all $\text{num\_head}$ heads.

  3. Heterogeneous Memory Allocation with Packed Attention Forward (HMA + PAF): During actual inference, the multi-head self-attention operation is split according to the head classification. Sink heads see only 1 sink frame + 1 current frame; neighbor heads see a local sliding window of $L-1$ recent frames + 1 current frame; dummy heads see 1 packing frame (the immediately preceding frame) + 1 current frame. These head groups are executed through two fused attention kernel calls (sink+dummy heads packed together, neighbor heads separate) rather than three separate calls or one monolithic call, minimizing kernel launch overhead.

Information flows as follows: a video prompt enters the system → the model begins autoregressive generation → at the designated profiling step, attention maps are collected and frame attention scores are computed → DHP classifies all heads → the head classification mapping is fixed for all subsequent AR steps and denoising timesteps of that video shot → each self-attention layer applies heterogeneous context lengths according to head type → outputs are reassembled into the standard attention output shape → the rest of the diffusion transformer proceeds unchanged.

3.3 Roadmap for the deep dive

  • First, the profiling methodology and the frame attention score metric (Equation 3), because this metric is the quantitative foundation for both the motivating observations and the DHP optimization. Understanding how frame-level attention is measured is prerequisite to understanding why dummy heads are identifiable.
  • Second, the three motivating observations from the profiling study (Observation 1–3), which establish the existence, stability, and performance impact of dummy heads. These are not part of the method per se but form the empirical justification that makes the method design coherent.
  • Third, Heterogeneous Memory Allocation (HMA), which defines the three head types and their corresponding context lengths. This is the "what" of the method—the structural modification to self-attention.
  • Fourth, Dynamic Head Programming (DHP), which provides the adaptive mechanism for deciding which heads get which context lengths. This is the "how" of classification, formalizing head assignment as an optimization problem with a proven optimal greedy solution.
  • Fifth, Packed Attention Forward (PAF), which addresses the practical challenge of achieving aggressive dummy head ratios by extending dummy-head context slightly and fusing attention calls to reduce kernel launch overhead. This is the "how to make it fast" component that turns good classification into actual speedup.
  • Sixth, the implementation practicalities, including query subsampling for efficient profiling, the selection of profiling steps, and the Triton-based kernel implementation.

3.4 Detailed, sentence-based technical breakdown

This is primarily an empirical analysis and systems optimization paper whose core idea is that autoregressive video diffusion models exhibit a consistent structural property—certain attention heads ("dummy heads") do not meaningfully aggregate cross-frame context—and that exploiting this property through adaptive head-wise KV cache management enables substantial inference acceleration without retraining.

The Frame Attention Score: Measuring Head-Level Context Utilization

The paper introduces a simple but essential metric to quantify how each attention head distributes its attention across different categories of frames: sink frames (the fixed anchor frame preserved across all AR steps), neighbor frames (the $L-1$ most recent frames in the sliding window), and the current frame (the frame being generated at the current AR step). This metric is computed from the raw attention map $\mathcal{A}$ produced by each self-attention head.

For a single attention head, the raw attention map $\mathcal{A}$ is a matrix of size $HW \times (L+1)HW$, where $HW$ is the number of visual tokens per frame (height times width of the latent representation) and $(L+1)HW$ is the total number of key tokens: $HW$ from the sink frame, $(L-1)HW$ from the neighbor frames, and $HW$ from the current frame. Each entry $\mathcal{A}_{uv}$ represents the attention weight from query token $u$ (from the current frame) to key token $v$ (from any of the cached frames).

The frame attention score for frame category $r$ (where $r \in \{\text{sink}, \text{neighbor}, \text{current}\}$) is:

αr=1HW∑u=1HW∑v∈JrAuv\alpha^{r} = \frac{1}{HW} \sum_{u=1}^{HW} \sum_{v \in \mathcal{J}_{r}} \mathcal{A}_{uv}

where $\mathcal{J}_{\text{sink}} = [0, HW)$ indexes the key positions belonging to the sink frame, $\mathcal{J}_{\text{neighbor}} = [HW, L \cdot HW)$ indexes the key positions belonging to all neighbor frames (collectively), and $\mathcal{J}_{\text{current}} = [L \cdot HW, (L+1) \cdot HW)$ indexes the key positions belonging to the current frame. Note that $\sum_{r} \alpha^{r} = 1$ by construction, since every attention weight across all key tokens sums to 1 for each query, and the $\frac{1}{HW}$ factor averages this across all $HW$ queries.

What this computes: For a single attention head, $\alpha^{\text{sink}}$ is the average fraction of attention that queries from the current frame direct toward the sink frame's tokens; $\alpha^{\text{neighbor}}$ is the average fraction directed toward all neighbor frames' tokens combined; and $\alpha^{\text{current}}$ is the average fraction directed toward the current frame's own tokens (self-attention within the frame). The three values sum to 1 and together characterize the head's attention allocation pattern.

Why this form: Aggregating at the frame level (rather than tracking individual token-level attention patterns) is motivated by the paper's goal of making frame-level cache decisions—the method removes or retains entire frames' worth of KV cache for each head, not individual tokens. A head that allocates 80% of its attention to the current frame's tokens is functionally not using historical context, regardless of how it distributes the remaining 20% across sink and neighbor frames. The frame-level aggregation produces a compact, interpretable signature that directly informs the head classification decisions made by Dynamic Head Programming. The choice to not separate individual neighbor frames further reflects that the paper's HMA scheme treats all neighbor frames as a single sliding window unit.

In the main paper, profiling uses the Self Forcing model (Huang et al., 2025) as the representative architecture. Profiling is conducted at the third AR step because at this step the lengths of key sequences for sink, neighbor, and current frames are identical—earlier steps have fewer total frames, breaking the symmetry that makes comparisons clean. Attention maps are collected at the last denoising timestep to avoid the influence of noise on attention patterns (early denoising steps operate on noisy latents where attention may be less interpretable). Results are averaged across 100 text prompts sampled from VBench to ensure statistical stability.

The Three Observations That Motivate Dummy Forcing

The frame attention score metric enables three empirical findings, presented in Section 3.2, that collectively justify the design of Dummy Forcing:

Observation 1: Certain heads under-utilize past context. When the frame attention scores $\alpha^{\text{current}}$ are computed for all heads and sorted, approximately 25% of heads exhibit $\alpha^{\text{current}} > 0.8$ —that is, they allocate over 80% of their attention weight to the current frame's own tokens, leaving less than 20% distributed across all historical frames combined (Figure 4c). The paper's language is precise: these heads attend "almost exclusively on $K_i$"—the key vectors from the current frame—"even though $K_s$ and $K_{i-L+1:i-1}$ are available." Since these heads fail to perform meaningful cross-frame information aggregation, the paper names them dummy heads. The identification procedure for dummy heads given a target count $N$ is straightforward: compute $\alpha^{\text{current}}$ for every head, sort descending, and select the top-$N$ heads with the largest $\alpha^{\text{current}}$ values. These are the heads that attend most heavily to the current frame and least to history.

Observation 2: Dummy head positions are stable across conditions. A natural question is whether the set of dummy heads depends strongly on the specific conditions under which profiling is conducted—different text prompts might cause different attention patterns, different AR steps have different amounts of accumulated history, and different denoising timesteps have different noise levels. The paper investigates this by computing the dummy head sets $\mathcal{I}_c = \{(l_n, h_n)\}_{n=1}^{N}$ under $C$ varying conditions, where each $\mathcal{I}_c$ contains the $(layer, head\_index)$ positions of the top-$N$ dummy heads identified under condition $c$. The stability is quantified using the core set ratio, defined as:

1N∣I1∩I2∩⋯∩IC∣∈[0,1]\frac{1}{N} \left| \mathcal{I}_1 \cap \mathcal{I}_2 \cap \cdots \cap \mathcal{I}_C \right| \in [0, 1]

What this computes: The intersection $\mathcal{I}_1 \cap \mathcal{I}_2 \cap \cdots \cap \mathcal{I}_C$ is the set of head positions that appear in every condition's dummy head set. Dividing its size by $N$ gives the fraction of positions that are universally dummy across all $C$ conditions. A value of 1 means the exact same $N$ heads are dummy under all conditions; a value near 0 means dummy head identity varies dramatically.

Why this form: The core set ratio directly measures the worst-case agreement—a head must be dummy under all conditions to count, not just most. This is a conservative metric appropriate for a method that fixes head classification after profiling: if there's substantial disagreement, fixing the classification based on one condition risks misclassifying heads under other conditions.

The paper reports that across different AR steps (with $C = 5$), the core set ratio is 0.92 —meaning 92% of dummy head positions appear in the dummy set at every AR step (Figure 4d, first bar group). Across different denoising timesteps, the ratio is also high. Across different text prompts, the ratio drops to approximately 0.75 —still substantial, but indicating that the dummy head set has some prompt dependence. This partial instability motivates the paper to develop Dynamic Head Programming (the adaptive classification method) rather than relying on a fixed, pre-computed dummy head set. The paper explicitly states:

"there is about 25% discrepancy when using varying text prompts"

and positions DHP as the mechanism to handle this residual variability.

Observation 3: Pruning dummy head caches incurs only slight degradation. The most directly actionable finding: when the KV caches of approximately 25% of heads (identified as dummy using the top-$N$ $\alpha^{\text{current}}$ method under a single profiling condition) are entirely removed—so those heads see only the current frame—the resulting performance drop on the VBench benchmark is 0.26% (from 84.0 to 83.78, as shown in Table 1 and Figure 1). In contrast, randomly selecting 25% of heads and evicting their caches causes severe degradation. This demonstrates that (a) the dummy heads as identified by $\alpha^{\text{current}}$ genuinely don't need their historical cache, and (b) there exist other "non-dummy" heads whose caches are critical—pruning them randomly destroys performance.

This observation is the crucial link between analysis and method: it proves that head-wise cache reduction is possible without retraining and that the $\alpha^{\text{current}}$ metric is a valid signal for identifying which heads can be pruned. It also establishes that 25% dummy heads yields a speedup from 17.6 FPS to 19.6 FPS (an 11% improvement), but the speedup is modest—motivating the more sophisticated HMA and PAF components that push dummy head ratios higher.

Importantly, this observation extends beyond Self Forcing. Appendix B (Figures 10 and 11) replicates the profiling on CausVid (Yin et al., 2025, built on the Diffusion Forcing framework) and Rolling Forcing (Liu et al., 2025), showing the same pattern: a consistent fraction of heads with $\alpha^{\text{current}} > 0.8$ across varying AR steps and denoising timesteps. The paper also generates videos from these models with 50% of heads treated as dummy (Figures 15 and 16), showing comparable quality to the original models. This cross-model replication establishes that dummy heads are a general property of autoregressive video diffusion training, not an artifact of a specific architecture or training recipe.

Heterogeneous Memory Allocation (HMA): Defining Head Types and Their Context Windows

Building on the dummy head observation, HMA extends the head classification from a binary (dummy vs. non-dummy) to a ternary scheme: sink heads, neighbor heads, and dummy heads. The motivation comes from additional analysis (referenced in Section 4.5) showing that among non-dummy heads, there is further specialization: some heads attend disproportionately to the sink frame while largely ignoring neighbor frames, and others do the opposite. Simply merging all non-dummy heads into a single category that sees the full $1$ sink frame + $L-1$ neighbor frames + $1$ current frame would leave redundancy: sink-focused heads would receive neighbor-frame caches they don't substantially use, and vice versa.

For a video at AR step $i$, with sink frame index $s$ and sliding window size $L$, the three head types receive the following context:

Sink heads see only the sink frame and the current frame:

sink:softmax(Qi[Ks,Ki]⊤)[Vs,Vi]\text{sink}: \text{softmax}\left(Q_i [K_s, K_i]^\top\right) [V_s, V_i]

where $Q_i \in \mathbb{R}^{HW \times d}$ is the query from the current frame, $K_s, V_s \in \mathbb{R}^{HW \times d}$ are the key and value from the sink frame, and $K_i, V_i \in \mathbb{R}^{HW \times d}$ are the key and value from the current frame. The $[\cdot]$ notation denotes concatenation along the token sequence dimension, so the attention is computed over $2HW$ total key tokens.

Neighbor heads see the local sliding window of recent frames plus the current frame, but NOT the sink frame:

neighbor:softmax(QiKi−L+1:i)Vi−L+1:i\text{neighbor}: \text{softmax}\left(Q_i K_{i-L+1:i}\right) V_{i-L+1:i}

where $K_{i-L+1:i} \in \mathbb{R}^{L \cdot HW \times d}$ represents keys from the $L$ frames in the window: indices $i-L+1$ through $i-1$ (the $L-1$ recent frames) plus index $i$ (the current frame). The sink frame is explicitly excluded from the neighbor heads' context because it is "exclusively modeled through sink heads." This design choice means the neighbor heads' cache is actually smaller than the traditional sliding window by exactly one frame (the sink frame), providing additional compression.

Dummy heads see only the current frame, with no historical context:

dummy:softmax(QiKi⊤)Vi\text{dummy}: \text{softmax}\left(Q_i K_i^\top\right) V_i

This is the most aggressive compression: the dummy head's self-attention reduces to current-frame-only self-attention. The KV cache from historical frames is entirely removed for these heads across all layers.

The key architectural modification to the transformer is in how multi-head self-attention is executed. In a standard implementation, all $\text{num\_head}$ heads process the same full key-value sequence—the concatenation of sink, neighbor, and current frames. Under HMA, the heads are partitioned into three groups by indexing along the head dimension:

  1. Extract the query, key, and value slices for sink heads, neighbor heads, and dummy heads.
  2. Execute the three attention computations with their respective context lengths (as specified above).
  3. Place the attention outputs back into their original positions in the head dimension to reconstruct the standard output shape.

The paper implements this partitioning using Triton (OpenAI, 2021), a GPU programming framework, rather than standard PyTorch operations. The reason is that "additional overhead caused by the above head indexing and placement" would otherwise negate the speedup from reduced attention computation. Triton enables custom kernel fusion where the indexing, attention computation, and output placement happen in a single optimized GPU kernel rather than multiple separate operations with intermediate memory transfers.

Why this ternary split rather than binary (dummy vs. non-dummy): Ablation results in Table 6 (row 1) show that merging sink and neighbor heads into a single non-dummy class yields "reasonable performance, but its acceleration is suboptimal." The paper explains this is "because although both sink head and neighbor head are non-dummy heads, they focus on distinct parts of the context, and simply merging them would result in redundancy." In other words, a sink-focused head receiving the full sliding window pays the computational cost of attending to neighbor frames it doesn't meaningfully use; a neighbor-focused head receiving the sink frame pays for a frame it doesn't need. The ternary split eliminates this intra-non-dummy redundancy.

Why remove the sink frame from neighbor heads: This is a non-obvious design choice with both compression and functional rationale. The sink frame serves as a global anchor—it is typically the first frame of the video and remains constant as the sliding window moves. By assigning sink-frame modeling exclusively to sink heads, the method ensures that global temporal consistency is maintained through a dedicated set of heads. Meanwhile, neighbor heads focus exclusively on local temporal dynamics (the $L$ most recent frames). This specialization means neighbor heads don't waste computation on a frame type (sink) that they may not be optimized to process, and it reduces the neighbor heads' effective context length by one frame—a modest but compounding savings across all neighbor heads in all layers.

Dynamic Head Programming (DHP): Optimal Head Classification as an Optimization Problem

Given the ternary head type scheme from HMA and a target number of dummy heads $N$, the problem becomes: which specific heads should be sink, which should be neighbor, and which should be dummy? The classification must work adaptively because (as Observation 2 showed) the dummy head set is not perfectly invariant across text prompts—there's approximately 25% variance in which heads are dummy across different prompts.

DHP formalizes this as a constrained optimization problem over the head type assignments $c_h \in \{\text{sink}, \text{neighbor}, \text{dummy}\}$ for $h = 1, \ldots, \text{num\_head}$. The objective is to maximize the total retained attention score across all heads, subject to the constraint that exactly $N$ heads are classified as dummy.

To set up the optimization, the paper first computes the per-head frame attention scores using Equation 3 but with a computational efficiency modification: instead of using all $HW$ query tokens, it uniformly samples 25% of query tokens and computes an approximate attention map against all key tokens. The paper reports this "achieves good performance while taking less than 10ms in practice." This subsampling yields $\mathcal{F} \in \mathbb{R}^{\text{num\_head} \times 3}$, where $\mathcal{F}_{h,0} = \alpha_h^{\text{sink}}$, $\mathcal{F}_{h,1} = \alpha_h^{\text{neighbor}}$, and $\mathcal{F}_{h,2} = \alpha_h^{\text{current}}$ for the $h$-th head.

The value function $f_h(c_h)$ quantifies how much attention score is retained when head $h$ is assigned to type $c_h$. Retention means the attention to frame categories that the head type preserves in its context:

  • If $c_h = \text{sink}$: the head retains attention to sink frames and the current frame, but loses attention to neighbor frames. So $f_h(\text{sink}) = \mathcal{F}_{h,0} + \mathcal{F}_{h,2}$.
  • If $c_h = \text{neighbor}$: the head retains attention to neighbor frames and the current frame, but loses attention to the sink frame. So $f_h(\text{neighbor}) = \mathcal{F}_{h,1} + \mathcal{F}_{h,2}$.
  • If $c_h = \text{dummy}$: the head retains attention to only the current frame, losing both sink and neighbor attention. So $f_h(\text{dummy}) = \mathcal{F}_{h,2}$.

The optimization problem is then:

max⁡∑h=1num_headfh(ch)s.t.∑h=1num_headI(ch=dummy)=N\max \sum_{h=1}^{\text{num\_head}} f_h(c_h) \quad \text{s.t.} \quad \sum_{h=1}^{\text{num\_head}} \mathbb{I}(c_h = \text{dummy}) = N

What this computes: Find an assignment of every head to one of three types that maximizes the total amount of attention score that will survive the heterogeneous memory allocation, while forcing exactly $N$ heads into the dummy category. The surviving attention score is what the model can still "pay attention to" after cache pruning—the lost attention score ($\mathcal{F}_{h,0}$ or $\mathcal{F}_{h,1}$) represents information from historical frames that the head would have attended to but can no longer access.

Why this form: This formulation directly connects the optimization objective to the model's functional behavior. The three value functions encode what each head type loses: a sink head loses $\mathcal{F}_{h,1}$ (neighbor attention), a neighbor head loses $\mathcal{F}_{h,0}$ (sink attention), and a dummy head loses $\mathcal{F}_{h,0} + \mathcal{F}_{h,1}$ (all historical attention). The constraint that exactly $N$ heads must be dummy enforces the desired compression ratio. Maximizing retained attention ensures that the $N$ dummy heads are those whose historical attention is smallest—heads for which the loss $\mathcal{F}_{h,0} + \mathcal{F}_{h,1}$ is minimal. This is a principled alternative to simply thresholding $\alpha^{\text{current}}$, which would not account for the sink/neighbor distinction among non-dummy heads.

The paper proves in Appendix A that this optimization problem has a greedy optimal solution. The proof proceeds as follows:

  1. Define the opportunity cost of forcing head $h$ to be dummy: $\ell_h = \max(\mathcal{F}_{h,0}, \mathcal{F}_{h,1})$. This is the maximum attention score that would be retained if head $h$ were not dummy (either as sink or neighbor)—essentially, what you give up by making it dummy.

  2. Sort all heads by $\ell_h$ in ascending order (smallest opportunity cost first).

  3. Assign the $N$ heads with the smallest $\ell_h$ as dummy heads. These are the heads where the cost of making them dummy is minimal—either because their historical attention is naturally low, or because the retained attention from one frame type (sink or neighbor) wouldn't have been large anyway.

  4. For the remaining $\text{num\_head} - N$ heads, assign each to sink if $\mathcal{F}_{h,0} \geq \mathcal{F}_{h,1}$ (it attends more to sink than to neighbor), or to neighbor otherwise.

The optimality proof (Appendix A) uses a swap argument: consider any alternative assignment $\mathcal{I} \neq \mathcal{I}^*$ (where $\mathcal{I}^*$ is the greedy solution). There must exist a head $i$ that is dummy in $\mathcal{I}$ but not in $\mathcal{I}^*$, and a head $j$ that is dummy in $\mathcal{I}^*$ but not in $\mathcal{I}$. By construction of the greedy algorithm, $\ell_j \leq \ell_i$. Swapping their assignments (making $i$ non-dummy and $j$ dummy) changes the total objective by $\ell_i - \ell_j \geq 0$ —the objective cannot decrease. Repeated swaps transform any assignment into the greedy assignment without decreasing the objective, proving optimality.

Why greedy rather than exhaustive search: With $\text{num\_head}$ typically being 360 or more (Self Forcing has 360 heads; RealTime-14B has 1600 heads), exhaustive enumeration of all $3^{\text{num\_head}}$ assignments is impossible. The greedy algorithm runs in $\mathcal{O}(\text{num\_head} \log \text{num\_head})$ time—dominated by the sorting step—and is guaranteed optimal because the problem exhibits the matroid structure necessary for greedy optimality. The key property is that the constraint (exactly $N$ dummy heads) and the objective (sum of per-head retained scores, with each head's contribution independent of others) decompose cleanly: there are no interactions between heads that would make the assignment of one head depend on the assignment of another. Each head's contribution to the objective depends only on its own classification, making the problem separable and greedy-optimal.

Implementation detail: The profiling to obtain $\mathcal{F}$ is performed at the third AR step and the last denoising timestep, with query token subsampling at 25%. The resulting head classification $c_h$ is then fixed for all subsequent AR steps and denoising timesteps within that video shot. A new profiling and classification is triggered only when the video shot changes (i.e., when a new text prompt is provided for interactive generation). The paper reports that this one-time classification "can complete within 100ms," making it a negligible overhead compared to the multi-second video generation process.

Contrast with prior work: Previous head classification schemes like DuoAttention (Xiao et al., 2024) perform binary classification (retrieval vs. streaming) using heuristics or learned classifiers. DHP provides a principled optimization formulation with provable optimality, and it handles the ternary classification (sink/neighbor/dummy) that the paper shows is necessary for maximum compression. The optimization directly ties the classification to the empirical attention patterns of the specific model instance, making it adaptive without requiring any training.

Packed Attention Forward (PAF): Enabling Aggressive Dummy Head Ratios

The naive approach to implementing HMA would be to launch three separate attention kernel calls: one for sink heads, one for neighbor heads, and one for dummy heads. Each kernel call incurs GPU launch overhead, and the dummy head kernel—processing only the current frame—is extremely fast but still costs a launch. This setup limits how aggressively dummy head counts can be increased because:

  1. Classification boundary errors: When $N$ is too large (e.g., 300 out of 360 heads, or 83%), the greedy DHP algorithm starts classifying heads with non-trivial historical attention as dummy. These "boundary" heads—heads that are borderline between dummy and non-dummy—may actually contribute meaningful cross-frame information. Naively removing their historical caches degrades performance (as shown in Figure 8: performance drops sharply when $N$ exceeds ~240 out of 360 heads).

  2. Kernel launch overhead: Three separate attention calls incur more launch overhead than two or one, partially offsetting the computational savings from reduced context lengths.

PAF addresses both issues with a single design change: extend the context of dummy heads to include the immediately preceding frame (frame $i-1$). The modified dummy head attention becomes:

pack_dummy:softmax(Qi[Ki−1,Ki]⊤)[Vi−1,Vi]\text{pack\_dummy}: \text{softmax}\left(Q_i [K_{i-1}, K_i]^\top\right) [V_{i-1}, V_i]

where $K_{i-1}, V_{i-1}$ are the key and value from the $(i-1)$-th frame (the nearest neighbor).

What this changes: A packed dummy head now sees 2 frames (the current frame plus one preceding frame) instead of 1 frame. Its context is still dramatically smaller than a standard attention head (which would see $L$ cached frames plus the current frame), but it's no longer completely history-free.

Why this helps with classification boundaries: The paper argues that the classification boundary between dummy and non-dummy is somewhat arbitrary—heads near the boundary have low but non-zero historical attention. By giving dummy heads access to the immediately preceding frame, the method effectively shifts the classification boundary: a head that would have been classified as "neighbor" (and received $L$ frames of context) under the strict dummy definition can now safely be classified as "packed dummy" (receiving 2 frames of context) without losing all of its modest cross-frame functionality. The $(i-1)$-th frame captures the strongest temporal dependencies (adjacent frames are most correlated), so it provides the most important historical information at minimal context cost. Empirically, this allows the dummy head ratio to increase to 50% or more (the paper's default is 50%) without the quality degradation that occurs with purely context-free dummy heads.

Why this helps with kernel launch overhead: A packed dummy head and a sink head now have the same effective context length (both process 2 frames: sink heads process $K_s$ and $K_i$; packed dummy heads process $K_{i-1}$ and $K_i$). This enables a critical optimization: sink heads and packed dummy heads can be fused into a single attention kernel call. The fused kernel computes attention for both head types simultaneously, with softmax normalization applied separately per head. The number of kernel launches drops from three (sink, neighbor, dummy) to two (sink+dummy, neighbor). The paper reports:

"Empirically, the reduction in kernel launches largely compensates for the slightly enlarged context length, while enabling more than 50% of heads to be configured as dummy heads without noticeable quality degradation."

Why not also fuse neighbor heads: Neighbor heads have a different and variable context length ($L$ frames in the sliding window vs. 2 frames for sink/packed-dummy). Fusing them into a single kernel call would require either padding their key-value sequences to match, negating the compression benefit, or implementing a kernel that handles variable-length attention within a single launch—which is complex and may not map efficiently to GPU hardware.

Interaction with the cache budget: The packing frame $i-1$ requires that this frame's KV cache be retained for dummy heads—it cannot be pruned. However, since $i-1$ is typically the most recent frame and would be in the neighbor heads' sliding window anyway, its KV cache is already being stored and computed. The marginal cost is the attention computation itself (softmax over 2 frames rather than 1 frame for dummy heads), which is small compared to the savings from not processing $L$ frames. And critically, the fusion of sink and dummy heads into one kernel call means the increased attention cost for dummy heads is partially offset by reduced launch overhead.

The 2-kernel architecture: After PAF, each self-attention layer executes exactly two kernel calls:

  1. Kernel 1 (sink + packed dummy): Processes all sink heads and all packed dummy heads in a single call. The key-value sequence is $[K_{\text{reference}}, K_i]$, where $K_{\text{reference}}$ is $K_s$ for sink heads and $K_{i-1}$ for packed dummy heads. This requires the kernel to handle two different key-value sequences within the same call, which is feasible because the sequence length is the same (2 frames) for both head types—only the specific frame indices differ. The Triton implementation achieves this by indexing into the KV cache buffer with head-type-dependent offsets.

  2. Kernel 2 (neighbor): Processes all neighbor heads with the full sliding window $K_{i-L+1:i}$. The key-value sequence length is $L$ frames.

The output of both kernels is concatenated along the head dimension and passed to the subsequent output projection, matching the standard transformer's expected input shape.

Ablation evidence for PAF (Table 6, row 2): When the packing frame is removed and three separate attention calls are used (sink, neighbor, dummy), the paper reports that "it degrades performance while offering negligible speed gains." The performance degradation comes from naive boundary misclassification—without the packing frame's safety margin, some heads near the classification boundary lose all historical context, causing quality drops. The negligible speed gains come from the added kernel launch overhead of the third attention call negating the minor per-head savings from processing 1 frame instead of 2.

Implementation Practicalities: Profiling, Classification, and Execution

The paper's method operates as a one-time profiling and classification phase followed by repeated execution using the fixed classification across all subsequent AR steps and denoising timesteps of a video shot. This section consolidates the practical implementation details threaded through Sections 3.2, 3.3, and Appendix C.

Profiling step selection. The frame attention scores $\mathcal{F}$ are computed at a single representative point in the generation process: the third AR step and the last denoising timestep. The third AR step is chosen because at this step, "the lengths of attention key for the sink/neighbor/current frames are identical"—earlier steps have fewer total frames, making frame-count-dependent comparisons asymmetric. The last denoising timestep is chosen "to avoid the impact from noise"—early denoising steps operate on noisy latents where attention patterns may reflect noise structure rather than semantic frame relationships.

Query token subsampling. To avoid the full $O(HW \times L \cdot HW)$ cost of computing attention maps for profiling, the paper "uniformly sample[s] 25% of the original query tokens" and computes approximate attention maps using only these queries against all key tokens. The paper reports this approximation "achieves good performance while taking less than 10ms in practice." The subsampled scores $\mathcal{F}_{h,0}, \mathcal{F}_{h,1}, \mathcal{F}_{h,2}$ are then used directly in the DHP optimization.

Classification fixation. Once DHP produces the head type assignments $c_h$, this classification is fixed—it is not recomputed at subsequent AR steps or denoising timesteps within the same video shot. Observation 2 demonstrated that dummy head positions are largely stable across AR steps (92% core set ratio) and timesteps, so repeated classification would add overhead without meaningful benefit. The classification is recomputed only when a new text prompt begins a new video shot in interactive generation scenarios.

Triton kernel implementation. The paper uses Triton (OpenAI, 2021) rather than standard PyTorch attention implementations. The rationale is explicit: "to reduce the additional overhead caused by the above head indexing and placement." Standard PyTorch would require (1) slicing query/key/value tensors along the head dimension, (2) launching separate scaled_dot_product_attention calls for each head group with different key-value sequences, and (3) concatenating results. Each of these operations involves GPU kernel launches and intermediate memory allocations. The Triton implementation fuses indexing, attention computation, and placement into custom kernels that operate directly on the original memory buffers with head-type-dependent offsets, avoiding intermediate tensor allocations and reducing launch overhead.

Default hyperparameters. Unless specified otherwise, the paper's default configuration sets the dummy head count $N$ to 50% of total heads. The sliding window size $L$ is model-dependent: Self Forcing and LongLive have their own default window sizes (typically around 36 frames). The packing frame is always the $(i-1)$-th frame. The sink frame is "the first frame" in most configurations, following the standard attention sink convention.

Head distribution across layers (Figure 9). The paper observes that dummy heads and neighbor heads are not uniformly distributed across transformer layers. Averaged across 100 prompts, dummy heads "primarily appear in the first and last few layers, while neighbor heads cluster in intermediate layers." The paper's interpretation is functional: shallow layers first aggregate information from the current frame to abstract high-level features, intermediate layers then query past frames in this semantic space for cross-frame information aggregation, and deep layers refine the current frame and project back to the low-level representation space for decoding. This layered structure suggests that dummy heads serve a real purpose (current-frame feature extraction) in early and late layers, while cross-frame aggregation is concentrated in middle layers—consistent with the finding that pruning dummy head caches causes minimal degradation because the historical context in those layers was barely being used.

Cross-model applicability. The paper applies Dummy Forcing without modification to Self Forcing (Huang et al., 2025), LongLive (Yang et al., 2025a), CausVid (Yin et al., 2025, built on Diffusion Forcing), Rolling Forcing (Liu et al., 2025), and RealTime-14B (Millon, 2025). The method is architecture-agnostic as long as the model uses standard multi-head self-attention with KV caching. The specific $\text{num\_head}$ values vary: Self Forcing has 360 heads, RealTime-14B has 1600 heads. The dummy head count $N$ scales proportionally (50% of total heads in each case).

Integration with sliding window and KV re-caching. Dummy Forcing operates on top of the existing sliding window strategy used by the base models. The sink frame selection and window size $L$ are inherited from the base model's configuration. For long video generation with KV re-caching (Yang et al., 2025a)—a technique that recomputes KV caches when the sliding window shifts discontinuously to mitigate temporal distribution shift—the head classification remains fixed but is applied to the recomputed caches. The paper reports in Appendix D (Table 8) that the combination works without degradation, achieving comparable quality with higher speed on 60-second interactive generation.

4. Key Insights and Innovations

Innovation 1: The "Dummy Head" as a Diagnostic Concept — Reframing KV Cache Efficiency from Token-Level to Head-Level Redundancy

The most distinctive intellectual contribution of this paper is not the acceleration method itself, but the diagnostic concept that enables it: the identification of "dummy heads" as a structural property of autoregressive video diffusion models. Prior to this work, the dominant framing for KV cache compression—inherited from the LLM efficiency literature—was token-level: which individual tokens in the cache are important enough to keep? Methods like H2O (Zhang et al., 2023) compute per-token importance scores; StreamingLLM (Xiao et al., 2023) retains initial sink tokens plus recent windows; DuoAttention (Xiao et al., 2024) classifies heads but still preserves some tokens for every head. The fundamental assumption across all these approaches is that every attention head participates in cross-context aggregation to some meaningful degree, and the optimization problem is which tokens each head needs.

The dummy head concept overturns this assumption entirely. By profiling frame-level attention scores (Equation 3), the paper demonstrates that a substantial fraction of heads—approximately 25% in the default analysis, and up to two-thirds under aggressive configurations (Figure 8)—allocate over 80% of their attention weight to the current frame. These heads are not "bad" or "broken"; they are functionally specialized for intra-frame processing, not inter-frame aggregation. The diagnostic insight is that the KV cache for these heads is not just compressible at the token level—it is entirely unnecessary. The paper proves this with the striking result in Table 1: removing all historical KV caches from 25% of heads (identified by the α_current metric) causes only a 0.26% quality drop on VBench, while randomly pruning 25% of heads' caches causes severe degradation.

This reframing has cascade effects throughout the paper. It shifts the efficiency question from "which tokens can we discard?" (a continuous optimization over cache contents) to "which heads need any cache at all?" (a discrete classification problem). It explains why prior LLM compression methods underperform on video: they're solving the wrong granularity of problem, carefully selecting tokens within heads when entire heads can have their caches zeroed out. Most importantly, it reveals that pre-trained autoregressive video diffusion models have learned an implicit division of labor across attention heads—some specialize in cross-frame aggregation, others in current-frame refinement—that the architecture and training objective never explicitly enforced. The paper's Figure 9, showing that dummy heads concentrate in early and late layers while neighbor heads cluster in middle layers, provides a layered interpretation: shallow layers extract current-frame features, middle layers perform cross-frame semantic aggregation, and deep layers refine the current frame for decoding. This layered specialization is an emergent property of autoregressive video diffusion training that no prior work had documented.

The dummy head concept is a fundamental shift in how to think about attention head utilization, not an incremental refinement. It belongs to the class of diagnostic contributions—analogous to the discovery of "attention sinks" (Xiao et al., 2023) or "induction heads" in language models—that change the research community's mental model of what's happening inside transformers. The paper's demonstration that dummy heads appear across three architecturally distinct models (Self Forcing, CausVid/Diffusion Forcing, Rolling Forcing; Figures 10–11, Figures 15–16) establishes this as a general property of autoregressive video diffusion training, not a quirk of any specific implementation. The implication is that future work on video model efficiency should start from the assumption that head-level redundancy exists and design methods to exploit it, rather than treating all heads as equally participating in cross-frame computation.

This innovation is evidenced primarily by the profiling analysis in Section 3.2 (Figures 4a–c, Table 1) and the cross-model replication in Appendix B (Figures 10–11). The 0.26% quality drop from pruning 25% of heads' caches (Table 1, Figure 1) is the critical empirical anchor—it converts the diagnostic observation from an interesting pattern into an actionable efficiency lever.

Innovation 2: Ternary Head Specialization as a Compression Architecture — Beyond Binary Dummy/Non-Dummy

The paper's second conceptual move is the recognition that binary classification (dummy vs. non-dummy) leaves substantial redundancy on the table, and that optimal compression requires a ternary scheme (sink, neighbor, dummy) that reflects the functional specialization within the non-dummy head population. This is not an obvious extension. A natural first approach after discovering dummy heads would be to identify and prune them, then leave the remaining "useful" heads with their full sliding-window context intact. This binary approach corresponds to the combine ablation in Table 6 (row 1), which the paper reports "achieves reasonable performance, but its acceleration is suboptimal."

The insight driving the ternary split is that non-dummy heads themselves are not monolithic. Through the same frame attention score profiling, the paper observes that among heads that do engage with historical context, some allocate disproportionate attention to the sink frame (the global anchor) while largely ignoring neighbor frames, and others show the reverse pattern. Merging these into a single "non-dummy" class that receives the full concatenation [sink, neighbor_window, current] means that sink-focused heads waste computation attending to neighbor frames they barely use, and neighbor-focused heads waste computation on the sink frame. The ternary split eliminates this intra-class redundancy by assigning each head type only the context it demonstrably uses: sink heads get [sink, current], neighbor heads get [neighbor_window, current], and dummy heads get [current] (or [prev_frame, current] after packing).

This is a moderate conceptual advance over prior head classification work. DuoAttention (Xiao et al., 2024) proposed a binary retrieval/streaming split for LLMs, where streaming heads keep only recent tokens and attention sinks. FastGen (Ge et al., 2023) proposed finer-grained per-head context lengths but using heuristic pattern matching rather than a principled optimization. Dummy Forcing's ternary scheme is more specialized to video structure—recognizing that sink frames and neighbor frames serve distinct functional roles (global temporal anchor vs. local motion dynamics) that are handled by different heads—and is derived from empirical attention patterns rather than architectural assumptions. The paper's ablation (Table 6, row 1) provides direct evidence that the ternary split yields better speedup than binary at comparable quality, validating the intuition that intra-non-dummy redundancy is real and exploitable.

The design choice to remove the sink frame from neighbor heads' context is a subtle but telling detail. Most sliding window systems include the sink frame in every head's context by default—it's the conventional architecture. The paper argues that since sink heads are dedicated to sink-frame modeling, neighbor heads can safely drop it, saving one frame's worth of cache per neighbor head without losing functionality. This saving compounds across all neighbor heads in all layers and is an example of how the ternary scheme enables compression that a binary scheme cannot express.

Innovation 3: Formulating Head Classification as a Provably Optimal Greedy Assignment Problem

While the dummy head observation and ternary scheme are empirical discoveries, Dynamic Head Programming (DHP) represents a theoretical contribution: the formulation of head classification as a constrained maximization problem over retained attention scores, with a proof that a simple greedy algorithm achieves optimality in O(n log n) time (Appendix A). This stands in contrast to prior head classification methods, which typically use heuristic thresholds (e.g., "heads with cumulative attention below τ are streaming"), learned classifiers trained on proxy tasks, or manual inspection of attention patterns.

The intellectual novelty here is twofold. First, the objective function—maximize retained attention scores subject to a dummy head count constraint—directly connects the optimization to what the model functionally loses when context is pruned. Each head's value function f_h(c_h) encodes exactly which attention components survive under each classification. A head classified as dummy loses α^sink + α^neighbor; a head classified as sink loses α^neighbor. The optimization selects the classification that minimizes total attention loss for a given compression budget. This is more principled than thresholding α_current because it jointly considers the sink/neighbor distinction: a head with high α_current but also high α_sink might be better kept as a sink head than forced to dummy, even though a naive current-frame threshold would prune it.

Second, the proof of greedy optimality (Appendix A) establishes that the problem has the matroid structure necessary for greedy algorithms to be globally optimal—specifically, that the per-head contributions are independent and the constraint is a simple cardinality bound. This is not a deep theoretical result (the proof uses a standard swap argument), but it provides a correctness guarantee that heuristic methods lack. It also has practical implications: DHP runs deterministically and produces a unique optimal solution (up to ties in ℓ_h), removing the need for hyperparameter tuning or validation-set optimization that learned classifiers require.

This is an incremental theoretical advance rather than a fundamental one—the optimization problem is simple and the proof is straightforward—but it fills a gap in the literature where head classification has been treated as an engineering heuristic rather than a formal problem. The practical significance is that DHP adapts automatically to any model instance without training, making the method truly plug-and-play: profile attention once, run the greedy algorithm, and deploy the classification.

The innovation is evidenced by the DHP algorithm description in Section 3.3 (Equations 5–6), the optimality proof in Appendix A, and the ablation in Table 6 showing that dynamic (adaptive) classification outperforms fixed-threshold alternatives. The 100ms profiling overhead reported in Appendix C confirms the method's practical viability.

Innovation 4: The Packing Frame as a Classification Boundary Shift — Enabling Aggressive Compression Without Quality Cliff

The paper's fourth conceptual contribution addresses a practical barrier: why can't we simply increase the dummy head count to achieve arbitrary speedup? Figure 8 provides the empirical answer: performance is stable as dummy heads increase from 0 to ~240 (out of 360 total heads), then drops sharply from ~240 to 360. The cliff occurs because at very high dummy ratios, DHP is forced to classify heads with non-trivial historical attention as dummy—heads near the classification boundary that actually contribute meaningful cross-frame information.

The standard solution to this problem would be to accept a quality-speed tradeoff (fewer dummy heads, less speedup) or to retrain the model to tolerate higher pruning ratios. Packed Attention Forward (PAF) offers a different solution: shift the classification boundary itself by slightly extending what "dummy" means. By giving dummy heads access to the immediately preceding frame (i-1) rather than only the current frame, PAF creates a safety margin at the boundary. Heads that were borderline (low but non-zero historical attention) now retain the most temporally proximate historical information—adjacent frames are maximally correlated—while still being dramatically cheaper than full neighbor heads. This shifts the boundary so that more heads can be safely classified as dummy without the quality cliff.

This is an incremental innovation in method design but a conceptually elegant one: it recognizes that the classification boundary between head types is inherently fuzzy, not sharp, and that a small context extension for the pruned class can substantially widen the safe pruning range. The evidence is in the ablation (Table 6, row 2): removing the packing frame and using three separate attention calls "degrades performance while offering negligible speed gains," confirming that PAF is responsible for the method's ability to reach 50%+ dummy head ratios without quality loss.

The secondary benefit—that packing enables fusing sink and dummy heads into a single kernel call because they now share the same context length—is a systems-level insight that demonstrates attention to practical deployment constraints. It converts what could have been a quality-speed tradeoff (give dummy heads 2 frames → slightly more computation → slightly less speedup) into a net win (fused kernel launches → reduced overhead → net speedup maintained or improved). This kind of cross-layer optimization—where an algorithmic choice (packing frame) enables a systems optimization (kernel fusion) that more than compensates for the algorithmic cost—is characteristic of well-engineered inference methods.

Evidence: Figure 8 (dummy ratio sweep), Table 6 (ablation), and the 2.0× speedup at 1080P (Table 4) where aggressive dummy ratios are most impactful.

Innovation 5: Long-Context Generation via Cache Budget Reallocation — Efficiency Gains as Capability Gains

The paper's final conceptual move transcends pure acceleration: the efficiency savings from dummy head compression can be reinvested to extend functional capability rather than merely reducing runtime. In the long-context video generation experiments (Section 4.4), the paper takes the cache budget saved by pruning dummy and sink heads' contexts and reallocates it to neighbor heads, extending their effective sliding window. This achieves a 6.58× longer cache (Table 5) at comparable or better speed than the baseline with a standard window.

This is significant because it reframes efficiency not as a quality-neutral cost reduction but as a capability amplifier: the same computational budget can be spent on strictly better temporal context rather than faster generation. The paper demonstrates this through an "A-B-A" narrative consistency task (Appendix E), where a character disappears in shot B and must reappear identically in shot C. Baseline models with standard 36-frame windows (~1.5 seconds of history) cannot bridge the gap and "re-generate" a new character identity. Dummy Forcing's extended cache preserves identity across the transition. This is evidenced in the qualitative comparison (Figure 7) and the quantitative VBench scores (Table 5, 68.45 vs. 69.48).

This finding has implications beyond video generation. It suggests that in any autoregressive transformer with KV caching, efficiency optimizations that identify and compress redundant attention heads can be strategically reinvested to improve model capability on tasks that benefit from longer context—storytelling, dialogue, document understanding, code generation with long dependencies. The paper does not develop this into a general framework, but the long-context experiments demonstrate the principle concretely: efficiency gains are fungible, and the optimal allocation of cache budget across heads may differ by task (favoring speed for real-time generation, favoring context length for narrative consistency).

This is a moderate conceptual advance that elevates Dummy Forcing from a pure acceleration method to a resource allocation framework. It connects to a broader theme in efficient ML: that identifying and eliminating waste creates a budget that can be spent on quality improvements, not just cost reduction. The evidence is concentrated in Section 4.4 and Table 5, with the 6.58× cache extension figure serving as the headline metric.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The primary benchmark is VBench (Huang et al., 2024a) for 5-second short video generation and VBench-Long (Huang et al., 2024b) for 30-second long video generation. VBench provides comprehensive evaluation dimensions including subject consistency, background consistency, motion smoothness, aesthetic quality, and imaging quality, aggregated into a total score. For long-context generation (Section 4.4), the paper designs a custom "A-B-A" narrative task using 3 text prompts per test case—the first and third prompts are identical, with a scene transition in between—to probe whether the model preserves character and background identity across the gap. For 60-second interactive generation (Appendix D), the paper uses 12 prompt suites released by LongLive (Yang et al., 2025a), each consisting of 6 sequentially fed prompts controlling 10-second video segments, evaluated on VBench-Long dimensions plus CLIP similarity between each segment and its prompt.

  • Base model(s). The main experiments use Self Forcing (Huang et al., 2025) and LongLive (Yang et al., 2025a) as the state-of-the-art autoregressive video diffusion baselines. Self Forcing bridges the train-test gap by conditioning on frames from the model's own outputs; LongLive introduces KV re-caching for interactive long video generation. Additional results in Appendix B extend to CausVid (Yin et al., 2025, built on the Diffusion Forcing framework) and Rolling Forcing (Liu et al., 2025, which employs joint denoising across a rolling window). Appendix D reports results on RealTime-14B (Millon, 2025), a 14-billion-parameter model with 1600 total heads, to validate scalability. The models span multiple training paradigms (Diffusion Forcing, Self Forcing, Rolling Forcing) and scales, providing evidence that dummy heads are a general property rather than model-specific. For the high-resolution experiments (Section 4.3), the paper exploits the models' zero-shot capability by modifying only the shape of the initial Gaussian noise at each AR step—no fine-tuning or architectural changes are applied.

  • Metrics. For short and long video generation, the paper reports VBench total score (a composite combining subject consistency, background consistency, motion smoothness, aesthetic quality, and imaging quality) and FPS (frames per second) measured on a single H100 GPU as the end-to-end generation speed metric. For long-context generation (Section 4.4, Table 5), the evaluation uses VBench dimensions computed on the concatenated first and third video segments (to measure identity preservation across the intervening scene transition), plus overall consistency via ViCLIP-based video-text similarity. For 60-second interactive generation (Appendix D, Table 8), CLIP similarity between each 10-second segment and its corresponding prompt is added to VBench-Long dimensions. For the dummy head profiling (Observation 3, Table 1), performance is reported as VBench total score before and after cache pruning (84.0 baseline vs. 83.78 with dummy head pruning, a 0.26% drop).

  • Baselines. The paper compares against three categories of acceleration methods: (1) KV cache compression methods from LLMs: R-KV (Cai et al., 2025), which employs token-level KV cache compression for LLM reasoning, and Infinipot-V (Kim et al., 2025), which prunes caches for streaming video understanding. Both compute token importance at each AR step. (2) Diffusion step skipping: TeaCache (Liu et al., 2024), which skips denoising timesteps for DiT acceleration by reusing cached predictions. (3) Ablation variants of Dummy Forcing itself: the binary dummy/non-dummy classification (combining sink and neighbor heads into one non-dummy type), and the three-kernel variant without packing (sink, neighbor, and context-free dummy heads in separate attention calls). For long-context generation, baselines include LongLive with and without the sliding window strategy. For the FLOPs-matched comparison (Section 7, if present), the paper would compare against a ~14× larger model—however, this paper does not include a pretraining-vs-inference FLOPs comparison; the only scale comparison is applying Dummy Forcing to RealTime-14B (Appendix D) to verify the method scales to larger models.

  • Generation budget / compute accounting. Compute is measured primarily as end-to-end FPS on a single H100 GPU—this captures the total wall-clock time including both attention computation and all other model components. At the module level, the paper also reports single attention layer runtime under different context lengths (Appendix D, Figure 12) to isolate the attention-specific speedup. For the dummy head ratio sweep (Figure 8), the independent variable is the number of dummy heads N (out of 360 total heads for Self Forcing), with runtime per AR step reported alongside VBench score. For cache compression comparison (Appendix D, Table 9), the metric is KV cache length as a percentage of the baseline: Infinipot-V and R-KV reduce to 16.7% (1.5 frames: 1 sink + 0.5 neighbor), while Dummy Forcing reduces to 27.8% (1 packing frame for 50% of heads, 4 frames for the other 50%) but achieves better speedup because it avoids per-step token selection overhead. For long-context generation (Table 5), the budget metric is the number of cached frames (#cache): Dummy Forcing achieves 36 cached frames (6.58× longer than the baseline sliding window of ~5.5 effective frames) while running 1.93× faster than LongLive without sliding window.

  • Cross-validation / statistical protocol. The paper does not employ cross-validation in the traditional machine learning sense—since Dummy Forcing is a training-free method applied to pre-trained models, there is no train/validation/test split for the acceleration technique itself. Instead, the paper's robustness protocol involves: (1) Multi-condition profiling: attention maps are collected under varying text prompts (100 prompts from VBench), AR steps, and denoising timesteps to assess head position stability (core set ratio, Figure 4d). (2) Fixed classification evaluation: the head classification derived from a single profiling step (third AR step, last denoising timestep) is fixed and applied unchanged across all subsequent evaluation on VBench, testing whether the classification generalizes. (3) Cross-model replication: the dummy head phenomenon and Dummy Forcing are validated on Self Forcing, LongLive, CausVid, Rolling Forcing, and RealTime-14B (Appendices B and D), providing evidence that results are not specific to one model or training paradigm. (4) Prompt-level averaging: unless specified, results are averaged across 100 text prompts from VBench. (5) For long-context generation, the custom A-B-A evaluation uses all prompts from VBench as the first/third prompt, with the second prompt generated automatically by Qwen2.5-72B-Instruct to introduce a scene transition (Figure 14, Appendix E).

Main Quantitative Results

Short Video Generation (5 seconds, VBench)

Headline result: On Self Forcing with 5-second short video generation, Dummy Forcing achieves 1.4× end-to-end speedup (from 17.6 FPS to 24.3 FPS) with only a 0.1% quality drop on VBench total score (Table 2). The method crosses the real-time threshold of 24 FPS, generating video at display rate.

Comparison with KV cache compression baselines (R-KV, Infinipot-V): These methods achieve only a marginal 1.1× overall speedup (Table 2) despite compressing the KV cache more aggressively in terms of cache length (to 16.7% of baseline vs. Dummy Forcing's 27.8%, as shown in Table 9). The paper attributes this to per-step token selection overhead: "the additional time introduced by token selection algorithm undermines the benefits gained from reduced cache length" (Section 4.2). In contrast, Dummy Forcing's head classification runs once per shot (~100ms) and is fixed for all subsequent steps, eliminating per-step overhead.

Comparison with TeaCache (diffusion step skipping): TeaCache achieves a smaller speedup because "the acceleration gain is limited given that current base models are already few-step diffusion models" (Section 4.2). This establishes Dummy Forcing and TeaCache as operating on complementary axes—attention structure vs. denoising schedule—which the paper later exploits in the combined variant (Table 7).

Comparison across base models: On LongLive for short video generation, Dummy Forcing achieves a 1.38× speedup from 12.3 FPS to 17.0 FPS with only a 0.5% quality drop (82.06 vs. 81.68 VBench total, Table 2). The speedup is model-dependent because it depends on the fraction of dummy heads and the base model's per-frame computational profile.

Qualitative evidence: The paper provides qualitative comparisons in Figures 19 and 20 (Appendix G) showing that videos generated with Dummy Forcing are visually indistinguishable from the baseline, consistent with the sub-0.5% quantitative quality drop.

Long Video Generation (30 seconds, VBench-Long)

Headline result: On Self Forcing with 30-second long video generation, Dummy Forcing achieves 1.4× end-to-end acceleration with up to a 0.4% quality drop (Table 3). The VBench-Long total score for Dummy Forcing is 80.16 vs. 80.48 for the baseline—a 0.32-point difference.

Comparison with baselines: R-KV shows a larger quality drop (79.87 vs. 80.48 baseline, a 0.61-point difference) with less speedup; TeaCache shows minimal speedup with slight quality degradation (80.30 vs. 80.48). Dummy Forcing achieves the best efficiency-quality tradeoff among all compared methods.

Quantitative comparison across dimensions: Figure 6 presents a radar chart comparing baseline and Dummy Forcing across VBench-Long dimensions (subject consistency, background consistency, motion smoothness, dynamic degree, aesthetic quality, imaging quality). The two radar plots nearly overlap, with Dummy Forcing showing marginally lower scores on some dimensions but maintaining strong motion and visual quality. The paper notes that the method "achieves faster generation while maintaining strong motion and high visual quality."

Qualitative evidence: Figures 21 and 22 (Appendix G) provide frame sequences from 30-second videos, showing temporal coherence maintained across long durations with the accelerated model.

High-Resolution Video Generation (720P and 1080P)

Headline result: Dummy Forcing's speedup increases with resolution because the savings from pruning dummy heads' caches scale with the quadratic growth in visual tokens. At 720P (1280×720), Dummy Forcing achieves approximately 1.7× speedup on LongLive and Self Forcing (Table 4, FPS columns). At 1080P (1920×1088), the speedup reaches 2.0× on LongLive (from 6.1 FPS to 12.2 FPS) without quality drop (VBench scores are essentially unchanged: 82.28 baseline vs. 82.25 Dummy Forcing for LongLive at 720P; 82.52 vs. 82.50 at 1080P).

Key mechanism: The paper explains that as resolution increases, "the number of cached visual tokens grows quadratically, leading to inefficiency in existing methods" (Section 4.3). Dummy Forcing's head-wise cache pruning removes a fixed fraction of heads' KV caches, and the per-head savings scale with the token count—more tokens per frame means more computation saved per pruned head. This is why the speedup factor increases from ~1.4× at standard resolution to 2.0× at 1080P.

Zero-shot high-resolution generation: The paper discovers that "current autoregressive video diffusion models exhibit strong zero-shot capabilities for low-to-high resolution video generation" and leverages this by simply modifying the shape of the initial Gaussian noise without fine-tuning. This means the 2.0× speedup at 1080P is achieved without any resolution-specific training or architectural modification—purely through inference-time cache management.

Quality preservation: Across both resolutions and both base models, the VBench total scores with Dummy Forcing are within 0.03–0.38 points of the baseline, confirming that the aggressive cache compression (50% of heads as dummy) does not damage high-resolution generation quality.

Long-Context Video Generation

Experimental design: The paper designs an "A-B-A" narrative task (Section 4.4 and Appendix E) where 3 text prompts control consecutive 5-second video segments: P₁ describes a scene, P₂ introduces a completely different scene (generated by Qwen2.5-72B-Instruct to ensure a genuine transition), and P₃ is identical to P₁. The model must reproduce the original scene's characters and backgrounds in the third segment despite the intervening transition—testing whether historical context persists across the gap. Evaluation uses VBench dimensions computed on the concatenated first and third segments (to measure identity preservation) plus overall ViCLIP consistency.

Headline results (Table 5):

  • LongLive without sliding window: Can access all historical frames (36 cached frames) but suffers from quadratic computational complexity, running at only 5.2 FPS.
  • LongLive with sliding window: Fast (10.0 FPS) but can only access ~5.5 effective cached frames, leading to identity regeneration after scene transitions (VBench total: 68.45).
  • Dummy Forcing (cache reallocation): Achieves 36 cached frames (same as full-history LongLive, 6.58× longer than the sliding window baseline) while running at 10.0 FPS (same speed as the sliding window baseline, 1.93× faster than full-history LongLive). VBench total improves to 69.48 (vs. 68.45 for sliding window, a 1.03-point gain), demonstrating that the extended cache improves narrative consistency.

The reallocation mechanism is critical: Dummy Forcing doesn't just accelerate—it reallocates the cache budget saved from pruning dummy and sink heads to neighbor heads, extending their effective window. As the paper states: "we allocate cache budget saved from the dummy&sink heads to the neighbor heads, thereby enabling longer effective cache" (Section 4.4).

Qualitative evidence (Figure 7): The paper shows a character that appears in segment 1, disappears during segment 2's scene transition, and reappears in segment 3. The baseline with sliding window "re-generates" a different-looking character in segment 3—identity is lost. Dummy Forcing with extended cache accurately reproduces the original character and background, demonstrating functional improvement in long-context memory.

60-second interactive generation (Appendix D, Table 8): Extending to 6 prompts × 10 seconds each, Dummy Forcing combined with KV re-caching (Yang et al., 2025a) achieves comparable video quality (VBench dimensions: subject consistency 96.19 vs. 96.18, background consistency 97.39 vs. 97.27, etc.) and CLIP semantic scores (ranging from 25.65 to 28.28 across segments) while maintaining higher generation speed.

KV Cache Compression Ratio Comparison (Appendix D, Table 9)

Headline finding: Dummy Forcing achieves a 27.8% cache length relative to baseline—meaning it compresses the KV cache to about one-quarter of its original size. Specifically, the method retains 1 packing frame for 50% of heads and 4 frames (1 sink + 3 neighbor) for the remaining 50% of heads, compared to the baseline's uniform 5 frames (1 sink + 4 neighbor, assuming L=5 for this calculation). In contrast, Infinipot-V and R-KV compress to 16.7% (1.5 frames: 1 sink + 0.5 neighbor frames on average). However, Dummy Forcing achieves better end-to-end speedup (1.4× vs. 1.1×) despite compressing less aggressively, because prior methods' per-step token selection overhead negates the benefits of shorter cache length.

Cache compression vs. speedup decoupling: This result reveals an important practical lesson: cache compression ratio is not a reliable proxy for speedup. The per-step overhead of the compression algorithm itself (token importance scoring, selection, cache reallocation) can dominate runtime, especially when the base model's attention computation is already efficient (as in few-step autoregressive diffusion). Dummy Forcing's design avoids this trap by making classification decisions once per shot rather than per step.

Scalability to Larger Models (Appendix D, Table 10)

RealTime-14B results: Applied to a 14B-parameter model with 1600 total heads (setting 800 as dummy heads, maintaining the 50% ratio), Dummy Forcing achieves:

  • 5-second short video (VBench): 83.15 total score (baseline: 83.20, a 0.05-point drop) with speedup from 7.1 FPS to 9.5 FPS (1.34×).
  • 30-second long video (VBench-Long): 80.73 total score (baseline: 80.67, a 0.06-point improvement—within noise, indicating no degradation) with speedup from 5.2 FPS to 7.0 FPS (1.35×).

The paper notes that "pruning all KV cache of 50% heads achieves even better total scores while delivering acceleration" on VBench-Long, demonstrating that the method scales to models an order of magnitude larger than Self Forcing without modification.

Combination with Orthogonal Methods (Table 7)

Dummy Forcing + TeaCache: Combining the two orthogonal acceleration methods (attention head compression + diffusion step skipping) yields a variant "Ours+TeaCache" that achieves over 30 FPS on short video generation—faster than either method alone (24.3 FPS for Dummy Forcing alone; TeaCache alone shows limited speedup on few-step models). The paper reports this in Table 7 (Discussion section) as evidence that Dummy Forcing "operates on a different axis" and is complementary to other inference optimizations.

Single Attention Layer Profiling (Appendix D, Figure 12)

Headline result: On a single self-attention layer, Dummy Forcing achieves a 1.7× speedup when processing attention across 15 frames (with HW visual tokens per latent frame). The speedup increases with context length because the baseline's quadratic attention complexity grows faster than Dummy Forcing's pruned attention, which processes only a subset of heads with full context.

Runtime scaling: Figure 12 shows that the baseline attention runtime grows roughly quadratically with the number of frames, while Dummy Forcing's runtime grows more slowly due to the fixed 50% head pruning. At very short context lengths (1–2 frames), the speedup is smaller (~1.1–1.2×) because kernel launch overhead dominates. As context length increases, the attention computation savings from pruned heads increasingly outweigh fixed overhead, yielding the 1.7× speedup at 15 frames.

Implication: This per-layer analysis explains why end-to-end speedup is lower than per-attention speedup: Dummy Forcing accelerates only the self-attention modules, while other components (cross-attention, feed-forward networks, layer norm, etc.) are unchanged. The per-attention 1.7× speedup translates to end-to-end ~1.4× speedup once non-attention components are factored in.

Dummy Head Distribution Across Layers (Figure 9)

Observational result: Averaged across 100 prompts, dummy heads "primarily appear in the first and last few layers, while neighbor heads cluster in intermediate layers" (Section 5, Discussion). The paper interprets this as reflecting a functional division of labor: shallow layers extract current-frame features (hence many dummy heads), intermediate layers perform cross-frame semantic aggregation in high-level feature space (hence many neighbor heads), and deep layers refine the current frame for decoding (hence many dummy heads again).

Evidence: Figure 9 presents bar charts showing the number of neighbor and dummy heads per layer. The pattern—dummy heads peaking in layers ~0–5 and ~25–30, neighbor heads peaking in layers ~10–20—is consistent across the 100-prompt average. The paper does not provide per-layer VBench ablation to confirm the functional interpretation, but the distribution pattern aligns with the broader finding that dummy heads are not "useless" but rather specialized for intra-frame processing.

Ablation Studies and Robustness Checks

Ternary vs. binary head classification (Table 6, row 1: "Combine sink and neighbor"): Merging sink and neighbor heads into a single non-dummy type (receiving 1 sink frame + L-1 recent frames + current frame) yields "reasonable performance, but its acceleration is suboptimal." The paper explains this is because sink-focused and neighbor-focused heads "focus on distinct parts of the context, and simply merging them would result in redundancy." The ternary split eliminates this intra-non-dummy redundancy, achieving better speedup at comparable quality. Table 6 quantifies this, showing that the binary variant achieves lower speedup for the same quality level.

Packing frame and kernel fusion (Table 6, row 2: "Remove packing frame, three attention"): Removing the packing frame from dummy heads (so they see only the current frame) and executing three separate attention kernel calls (sink, neighbor, context-free dummy) "degrades performance while offering negligible speed gains." Two effects are at play: (1) without the packing frame's safety margin, naive boundary misclassification causes heads with non-trivial historical attention to be incorrectly pruned, degrading quality; (2) the third kernel launch adds overhead that offsets the minor per-head savings from processing 1 frame instead of 2. This ablation validates that PAF is responsible for the method's ability to reach 50%+ dummy head ratios without the quality cliff seen in Figure 8.

Dummy head ratio sweep (Figure 8): Varying the dummy head count N from 0 to 360 (out of 360 total heads) reveals a stability-plateau then sharp-decline pattern. For N ≤ 240 (67% of heads), VBench total score remains relatively stable compared to N = 0 (the baseline). The paper interprets this as evidence that "nearly 2/3 heads do not fully utilize past frames." For N = 240 to 300, performance begins to degrade noticeably. For N > 300 (especially 300 to 360), there is "significant degradation" because "KV cache pruning harms neighbor heads, which are crucial for context aggregation." The runtime per AR step (right y-axis) decreases approximately linearly with N up to ~240, then flattens as the remaining attention computation is dominated by the neighbor heads' sliding window.

Query subsampling ratio for profiling (Appendix C): The paper uses 25% subsampling of query tokens for efficient frame attention score computation during profiling, reporting it "achieves good performance while taking less than 10ms in practice." No ablation of different subsampling ratios is provided, so the sensitivity of head classification to this parameter is not characterized. This is a minor missing ablation: if subsampling to 10% also works, profiling could be even cheaper.

Profiling step selection (temporal stability, Figure 4d and Appendix C): The paper profiles at the third AR step and last denoising timestep, then fixes the classification across all subsequent steps. The core set ratio across AR steps is 0.92 (92% of dummy heads are consistent), and across denoising timesteps the ratio is also high. Across text prompts, the ratio is ~0.75. The paper does not ablate alternative profiling choices (e.g., profiling at step 1 vs. step 3 vs. step 5) to quantify how sensitive the resulting speedup-quality tradeoff is to the profiling configuration. The 0.75 prompt-stability suggests that a one-shot classification per prompt is adequate, but the paper doesn't compare fixed-per-prompt classification vs. prompt-adaptive classification.

Cross-model generalization (Appendices B and D): Dummy Forcing is applied without modification to Self Forcing, LongLive, CausVid (Diffusion Forcing), Rolling Forcing, and RealTime-14B. The dummy head phenomenon is replicated in all cases (Figures 10–11), and generated videos with 50% dummy heads show "no noticeable degradation in video quality" (Appendix B, Figures 15–16). Quantitative VBench results are provided for RealTime-14B (Table 10), confirming the method scales to 14B parameters. This is a strong robustness check: the method generalizes across different training paradigms (Diffusion Forcing, Self Forcing, Rolling Forcing), model scales (from base Self Forcing to 14B RealTime), and base architectures. The consistent presence of dummy heads across these models supports the claim that this is a convergent property of autoregressive video diffusion training.

Sink frame exclusion from neighbor heads (implicit in HMA design): The paper's design removes the sink frame from neighbor heads' context, allocating it exclusively to sink heads. The ablation of binary vs. ternary classification (Table 6, row 1) indirectly tests this: the binary variant includes the sink frame in all non-dummy heads' context, while the ternary variant removes it from neighbor heads. The ternary variant achieves better acceleration, consistent with the claim that sink-frame attention is concentrated in a subset of heads and forcing all non-dummy heads to process it is wasteful.

Oracle vs. predicted difficulty: Not applicable—this paper does not use difficulty estimation or oracle information. Head classification is based purely on observed attention patterns from the model itself, with no ground-truth labels involved. This eliminates the oracle-vs-predicted gap that complicates some other adaptive inference methods.

Critical Assessment

Claim 1: "Dummy Forcing delivers up to 2.0× speedup over the baseline, supporting video generation at 24.3 FPS with less than 0.5% quality drop."

What the experiments demonstrate: The 2.0× figure is achieved specifically at 1080P resolution on LongLive (Table 4: 6.1 → 12.2 FPS), and the 24.3 FPS figure is achieved on Self Forcing for 5-second short video (Table 2: 17.6 → 24.3 FPS, a 1.38× speedup, not 2.0×). The "less than 0.5% quality drop" is supported across multiple configurations: 0.1% on Self Forcing short video (Table 2), 0.4% on Self Forcing long video (Table 3), ~0% on LongLive high-resolution (Table 4). However, these are different configurations achieving different speedups—no single configuration achieves both 2.0× and 24.3 FPS simultaneously. The claim is technically accurate as an "up to" statement spanning the best results across configurations, but readers might misinterpret it as describing a single operating point. The 2.0× speedup at 1080P is particularly notable because it leverages the zero-shot high-resolution capability, which is an unexpected bonus property of the base models rather than something Dummy Forcing itself enables—the method amplifies an existing capability rather than creating it.

Genuine weaknesses: (1) All FPS measurements are on a single H100 GPU. The speedup factor on consumer GPUs (with different memory bandwidth and compute characteristics) may differ—the paper provides no multi-GPU or consumer-hardware measurements. (2) The quality metric is VBench total score, which aggregates multiple dimensions. It's possible that some individual dimensions degrade more than others while the aggregate masks this—the paper shows radar charts (Figure 6) that suggest uniform preservation, but per-dimension numerical breakdowns are not provided for all experiments. (3) The 24.3 FPS figure is achieved on a model (Self Forcing) that the paper itself is evaluating—there's no independent verification or third-party benchmark.

Claim 2: "Approximately 25% of heads attend almost exclusively to the current frame, and discarding their KV caches incurs only minor performance degradation."

What the experiments demonstrate: This is the paper's strongest-supported claim. Figure 4c shows the distribution of α_current across heads, with ~25% of heads above 0.8. Table 1 shows that pruning these heads' caches causes a 0.26% VBench drop (84.0 → 83.78), while random pruning causes severe degradation. Figures 10–11 replicate the phenomenon on CausVid and Rolling Forcing. The 25% figure is based on the α_current > 0.8 threshold; Figure 8 shows that even at 50% dummy heads (N = 180), performance is stable, and at 67% (N = 240), there's still only moderate degradation, suggesting the "25%" is conservative and the actual redundancy might be larger. The cross-model replication is the strongest evidence: three architecturally distinct models trained with different paradigms all exhibit the same pattern, making it unlikely to be an artifact of a particular training run or architecture.

Genuine weaknesses: (1) The analysis is based on Self Forcing with L ≈ 36 frame window. It's possible that the dummy head fraction depends on the window size—with a smaller window, more heads might need to engage with history because there's less redundancy in the available context. The paper does not investigate this dependence. (2) The "25%" figure comes from a threshold (α_current > 0.8) that is itself arbitrary—a different threshold would give a different fraction. The paper doesn't provide a principled justification for the 0.8 threshold beyond it being "over 80%." (3) The stability analysis (Observation 2) shows 92% core set ratio across AR steps but only ~75% across prompts. This means that for a given prompt, ~25% of the dummy heads identified from a different prompt might not actually be dummy—this is the motivation for DHP's adaptive classification, but it also means that any fixed-threshold method (like the naive pruning in Table 1) is suboptimal by construction. The paper is transparent about this, but a reader might miss that the 0.26% quality drop in Table 1 is achieved with a prompt-specific head selection (profiled on the evaluation prompt), not a universal one—the quality drop with a universal dummy head set would likely be larger.

Claim 3: "Dummy Forcing enables 6.58× longer effective cache for long-context generation."

What the experiments demonstrate: Table 5 shows that Dummy Forcing with cache reallocation achieves 36 cached frames at 10.0 FPS, compared to the sliding window baseline's ~5.5 effective cached frames at 10.0 FPS—a 6.58× increase. The A-B-A narrative task (Figure 7, Table 5) shows that this translates to better identity preservation (VBench total 69.48 vs. 68.45). The 60-second generation results (Table 8) provide additional evidence at longer timescales.

Genuine weaknesses: (1) The A-B-A evaluation is custom-designed by the paper, not a standard benchmark. The prompts are generated automatically by Qwen2.5-72B-Instruct, and the quality of the evaluation depends on how well the generated transitions test identity preservation. The paper provides the prompt template (Figure 14) but does not validate the transition quality independently. (2) The "6.58× longer cache" figure compares Dummy Forcing's effective cache to the sliding window baseline's effective cache, not its nominal window size. The sliding window nominally uses L = 36 frames, but the A-B-A gap spans the entire second segment (5 seconds, ~120 frames at 24 FPS), so the effective cache for the identity task is much smaller than 36. The 6.58× figure is task-specific—it reflects the cache length relative to the A-B-A gap, not a universal cache extension ratio. (3) The comparison is against LongLive, not against a version of LongLive with the same total compute budget but a larger window—the speedup from cache compression is reinvested into cache length, but the paper doesn't explore alternative reinvestment strategies (e.g., more denoising steps, higher resolution). (4) Only 3-prompt A-B-A sequences are tested; longer narrative chains with multiple transitions are not evaluated.

Claim 4: "Dummy heads are a general property of autoregressive video diffusion models."

What the experiments demonstrate: The paper shows dummy heads in five models spanning three training paradigms: Self Forcing (Huang et al., 2025), LongLive (Yang et al., 2025a), CausVid/Diffusion Forcing (Yin et al., 2025; Chen et al., 2024), Rolling Forcing (Liu et al., 2025), and RealTime-14B (Millon, 2025). Quantitative VBench results with Dummy Forcing are provided for Self Forcing, LongLive, and RealTime-14B. Qualitative results (no degradation) are shown for CausVid and Rolling Forcing. This is strong evidence for generality across current autoregressive video diffusion methods.

Genuine weaknesses: (1) All models are from the same research lineage and time period (2024–2025). The paper does not test on older autoregressive video models, bidirectional models converted to autoregressive, or models trained with substantially different objectives. It's possible that dummy heads are an artifact of specific training recipes common in this generation of models (e.g., the specific noise schedules, frame chunking strategies, or the use of sliding windows during training) rather than a universal property of autoregressive video diffusion. (2) The paper does not investigate whether dummy heads emerge during training or are present from initialization—a training dynamics analysis would strengthen the "general property" claim. (3) All models are text-to-video; unconditional video generation or other conditioning modalities are not tested.

What would strengthen the paper:

  • Per-dimension VBench breakdowns for all experiments: The aggregate total score could mask uneven degradation across dimensions (e.g., motion smoothness might degrade more than subject consistency).
  • Ablation of window size L: Does the dummy head fraction change with L? If smaller windows force more heads to engage with history, the method's effectiveness might vary with the base model's window configuration.
  • Profiling step sensitivity: Does profiling at AR step 1 vs. step 3 vs. step 10 change the resulting head classification and speedup-quality tradeoff?
  • Multi-GPU and consumer-hardware benchmarks: The H100 results may not translate to deployment scenarios with different memory bandwidth characteristics.
  • Training dynamics analysis: When do dummy heads emerge? Are they present at initialization, or do they develop during training? This would clarify whether they're a fundamental property or a training artifact.
  • Comparison against a "head pruning during training" baseline: The paper suggests (Appendix F) that fine-tuning with dummy head caches removed could further improve performance. Without this experiment, it's unclear whether the training-free approach leaves performance on the table.
  • Standardized long-context benchmark: The custom A-B-A evaluation is clever but ad-hoc. Evaluation on a standard long-video consistency benchmark would improve comparability.
  • Per-layer ablation: Does pruning dummy heads in specific layer ranges (early, middle, late) have different effects on quality and speedup, consistent with the functional interpretation in Figure 9?

6. Limitations and Trade-offs

The Difficulty Estimation Cost is Not Amortized in the Headline Speedup

The paper reports that the profiling step to compute frame attention scores $\mathcal{F}$ for Dynamic Head Programming "can complete within 100ms" (Appendix C), but this cost is not included in the end-to-end FPS measurements that produce the headline 2.0× speedup figure. The profiling involves computing attention maps for 25% of query tokens against all key tokens at one AR step and one denoising timestep, which the paper describes as taking "less than 10ms" for the subsampled attention computation. However, the full DHP pipeline also requires aggregating results across all heads, running the greedy classification algorithm, and applying the classification mapping to restructure subsequent attention calls. The paper explicitly states:

"our method requires only a single head classification computation call which can complete within 100ms"

A 100ms one-time overhead is negligible for a 5-second video (if the video takes multiple seconds to generate), but for very short clips, interactive applications where shots change frequently, or scenarios where the model receives many short prompts, this overhead could become non-trivial. If a new classification is performed per shot and each shot is 1–2 seconds, the profiling overhead could consume 5–10% of the generation time, reducing the effective speedup.

The more fundamental concern is whether the fixed-per-shot classification is genuinely adequate for all conditions within a shot. The paper demonstrates that dummy head positions have a 0.92 core set ratio across AR steps and a ~0.75 ratio across text prompts (Figure 4d). The 0.92 step-stability is high—only 8% of dummy head positions change as generation progresses. But the paper does not measure whether this 8% variation matters for quality: if some heads switch from dummy to non-dummy behavior mid-generation, the fixed classification would either waste computation on heads that have become dummy-like, or (worse) prune context from heads that have become context-dependent. The paper's stability analysis treats all conditions as independent profiling runs rather than tracking whether the same head changes behavior within a single generation—this is a gap in the evidence for the core design choice of fixing classification after one profiling step.

Mitigation status: The paper acknowledges that the profiling step "incurs additional computation cost" (Section 3.3 discussion of Dynamic Head Programming) but does not quantify this cost relative to total generation time or amortize it into the FPS figures. The fixed-per-shot classification is presented as a feature (avoiding per-step recomputation) rather than evaluated for mid-shot drift. The paper does not ablate alternative strategies such as periodic re-profiling or dynamic per-step classification.

The Method Is Validated on a Narrow Slice of Video Generation—Single Model Family, Single Benchmark Family, Single Task Modality

All quantitative experiments are conducted on either VBench or VBench-Long, which, while comprehensive, are text-to-video benchmarks focused on general visual quality and temporal consistency. The base models—Self Forcing, LongLive, CausVid, Rolling Forcing, and RealTime-14B—all belong to the same research lineage of autoregressive video diffusion models trained on similar data distributions with similar architectural choices (DiT-based backbones, sliding window strategies, few-step denoising schedules). The paper does not evaluate on:

  • Video understanding tasks (action recognition, temporal localization, video QA) where dummy heads identified during generation might behave differently during discriminative processing
  • Other video generation benchmarks with different evaluation criteria (e.g., UCF-101 for class-conditional generation, MSR-VTT for caption-conditioned generation, EvalCrafter for compositional evaluation)
  • Bidirectional video diffusion models converted to autoregressive inference—the paper mentions these (Section 2) but does not test Dummy Forcing on them
  • Non-DiT architectures such as U-Net-based video diffusion or state-space models being adapted for video
  • Image generation models with autoregressive components, where the frame-level reuse patterns might differ

The paper acknowledges this scope limitation implicitly in Appendix F:

"since this work extensively explores the applicability of the proposed method to autoregressive video generation models... further investigation into inference acceleration for other model categories, such as world models, represents a meaningful direction"

The consequence is that a practitioner working with a different model architecture, training paradigm, or video domain cannot assume that dummy heads will be present, stable, or prunable with the same thresholds. The cross-model replication within the autoregressive diffusion family is strong evidence for that specific class, but the paper makes a leap in its framing when it characterizes dummy heads as a "general property" (abstract, Section 3.2)—the evidence supports "general across current autoregressive video diffusion models using DiT backbones and similar training recipes," not "general across video generation."

Mitigation status: The paper replicates the dummy head observation across five models spanning three training paradigms (Diffusion Forcing, Self Forcing, Rolling Forcing), which is substantially more cross-model validation than most efficiency papers provide. The limitation is primarily in the scope of models tested, not in the thoroughness within that scope. The paper suggests future work on "other model categories" in Appendix F.

The Relationship Between Dummy Heads and Video Quality Dimensions Is Not Characterized

The paper consistently reports VBench total score as the quality metric, which aggregates multiple dimensions (subject consistency, background consistency, motion smoothness, aesthetic quality, imaging quality, dynamic degree, etc.) into a single number. While the radar chart in Figure 6 suggests that Dummy Forcing preserves all dimensions roughly equally for one configuration (30-second Self Forcing), the paper does not provide per-dimension breakdowns for most experiments. This matters because different applications prioritize different quality dimensions:

  • Real-time interactive video might tolerate slightly degraded aesthetic quality if motion smoothness is preserved
  • Cinematic generation might require high aesthetic quality and accept slower generation
  • Character-driven narratives depend heavily on subject consistency across frames
  • High-resolution generation may expose different failure modes (e.g., texture inconsistency in pruned heads) than standard resolution

If dummy heads are unevenly distributed across layers that process different types of features (as Figure 9's functional interpretation suggests—shallow layers for current-frame feature extraction, intermediate layers for cross-frame aggregation, deep layers for decoding refinement), then pruning them might disproportionately affect specific quality dimensions. A head in a shallow layer that processes low-level texture features, when pruned, might cause subtle texture flickering that VBench's aggregate score masks. A head in an intermediate layer that handles motion correspondence, when incorrectly classified as dummy, might degrade motion smoothness more than other dimensions.

The paper's qualitative results (Figures 15–22) show frame sequences that appear visually comparable to baselines, but these are selected examples—the paper does not report worst-case failure modes, per-dimension degradation patterns, or the variance of quality scores across prompts. A practitioner deploying this method on a content-sensitive application (where a single jarring frame transition is unacceptable even if average quality is high) cannot assess the tail risk from the provided evidence.

Mitigation status: The radar chart in Figure 6 provides partial evidence for uniform quality preservation in one configuration. The paper does not provide per-dimension breakdowns for the short video, high-resolution, or long-context experiments. Appendix F acknowledges that post-training "could potentially achieve further performance gains or improved compression rates," implicitly acknowledging that the training-free approach may leave some quality on the table, but does not characterize where that quality is lost.

The 50% Dummy Head Default Is Not Theoretically Justified—It Is an Empirical Sweet Spot That May Not Transfer

The paper sets 50% of heads as dummy by default and demonstrates in Figure 8 that performance is stable up to ~67% (240 out of 360 heads for Self Forcing) before degrading sharply. However, the paper provides no principled method for selecting the dummy head count $N$ for a new model or task—the 50% figure is an empirical observation on Self Forcing tested at 5-second video generation. Several factors could shift the optimal $N$:

  • Model scale: RealTime-14B has 1600 heads vs. Self Forcing's 360. Does the stable dummy fraction scale linearly with total heads, or does larger capacity change the division of labor across heads? The paper tests only $N = 800$ (50%) for RealTime-14B—it does not sweep to find the optimal ratio for the larger model.
  • Window size $L$: If the sliding window is small, heads might need to engage with history more because there's less total context available—the dummy fraction could decrease. If the window is large, more redundancy might exist, and the dummy fraction could increase. The paper does not ablate $N$ against $L$.
  • Task characteristics: The long-context experiments use cache reallocation rather than pure acceleration, meaning the "budget" from dummy heads is reinvested into neighbor heads rather than taken as speedup. The optimal $N$ for a reinvestment strategy might differ from the optimal $N$ for pure speedup, since the tradeoff is between dummy-head current-frame processing (still needed for visual quality) and neighbor-head context length (needed for temporal consistency).
  • Resolution and frame rate: At higher resolutions, the per-frame token count increases, and the relative cost of neighbor heads' sliding window attention grows. This might make more aggressive dummy ratios more attractive (since the savings per pruned head increase), but it might also make any quality degradation more visible (since each frame has more detail that could be inconsistent).

The paper's approach to selecting $N$ is essentially: "sweep and pick the knee of the curve." For Self Forcing, Figure 8 shows that $N$ can go to ~240 with minimal degradation, and the paper selects 180 (50%) as a conservative default. But there's no guidance for a practitioner with a different model who doesn't want to run a full sweep: can they assume 50% is safe? Should they profile α_current distribution and set $N$ based on some statistics of that distribution? The Dynamic Head Programming optimization takes $N$ as an input constraint—it does not determine the optimal $N$. This leaves the most impactful hyperparameter to manual tuning.

Mitigation status: Figure 8 provides the sweep for Self Forcing, giving a model-specific answer. The paper does not propose a method for automatically selecting $N$, nor does it analyze whether features of the α_current distribution (e.g., the fraction of heads above some threshold, the shape of the sorted α_current curve) predict the optimal $N$ across models. The long-context experiments demonstrate that $N$ can be chosen based on a target cache budget rather than a target speedup, but this requires knowing the cache budget needed for the task, which is itself an empirical question.

The Method Does Not Address Latency—It Measures Throughput, Not Time-to-First-Frame or Interactive Responsiveness

Dummy Forcing's primary metric is Frames Per Second (FPS), measured as total frames generated divided by total wall-clock time on a single H100 GPU. This is a throughput metric—it captures the average generation speed over an entire video. It does not capture:

  • Time-to-first-frame (TTFF): The latency before the user sees any output. In autoregressive models, the first frame requires a full denoising process and cannot benefit from KV caching. Dummy Forcing's profiling step adds ~100ms to the TTFF, which is modest but non-zero.
  • Frame jitter or uneven frame pacing: If the attention computation varies in cost across AR steps (because context length grows, or because different heads are active at different steps), the output frame rate may not be constant. The paper reports average FPS but not frame-level timing.
  • Interactive round-trip latency: In scenarios where a user provides a new prompt mid-generation (e.g., the 60-second interactive generation in Appendix D), the model must respond quickly to maintain the illusion of interactivity. The paper does not report the latency between prompt input and the first generated frame for the new shot.
  • Memory bandwidth bottlenecks on non-H100 hardware: FPS is measured on an H100, which has high memory bandwidth (3.35 TB/s for H100 SXM). On consumer GPUs with lower bandwidth (e.g., RTX 4090 at ~1 TB/s), the attention computation may be less of a bottleneck relative to memory movement, and the speedup from reducing attention FLOPs may not translate proportionally. The paper provides no multi-GPU or consumer-hardware benchmarks.

For applications like real-time video generation for gaming, streaming, or interactive storytelling, the distinction between throughput and latency is critical. A method that improves average FPS from 17.6 to 24.3 but introduces occasional frame delays (e.g., from kernel launch variability in the fused attention calls) might produce a stutter that is more perceptually jarring than a consistently lower frame rate. The paper's implementation details (Triton kernels, two-kernel launch architecture) suggest attention to latency, but the paper does not measure or report latency-focused metrics.

Mitigation status: The paper reports only average end-to-end FPS. The Triton kernel implementation (Section 3.3) and the reduction from three to two kernel launches (PAF) are explicitly motivated by reducing overhead, which should improve both throughput and latency, but the latency impact is not measured. The 100ms profiling overhead for TTFF is mentioned in Appendix C but not incorporated into any startup-latency metric. The paper does not discuss or measure frame pacing consistency.

The Method Requires Access to Model Internals—It Cannot Be Applied to API-Only or Black-Box Models

Dummy Forcing operates by modifying the internal multi-head self-attention computation: it requires extracting per-head attention maps (for profiling), indexing into the head dimension to partition queries, keys, and values into groups (for HMA), and launching custom Triton kernels with head-type-dependent context windows (for execution). This means the method can only be applied to models where the practitioner has:

  • Full access to the model weights and architecture (white-box deployment)
  • The ability to modify the inference code (not a fixed optimized serving framework)
  • Sufficient GPU programming expertise to implement or adapt the Triton kernels
  • A deployment environment that supports custom CUDA/Triton kernels (not all serving platforms do)

This excludes several important deployment scenarios:

  • API-based video generation services (e.g., Runway, Pika, Kling) where the model is accessed through a black-box interface
  • Managed cloud inference platforms that provide optimized serving containers with fixed inference implementations
  • Mobile or edge deployment where Triton or CUDA kernel compilation may not be available or performant
  • Quantized or compiled model formats (e.g., TensorRT, ONNX Runtime) where the attention computation is fused into a black-box graph that cannot be easily partitioned by head type

The paper presents Dummy Forcing as a "training-free" method, which is accurate in the sense that no weight updates are required, but "training-free" does not mean "engineering-free." Implementing the method requires substantial systems engineering: profiling infrastructure to capture and analyze attention maps, a Triton-based attention implementation that supports head-type-dependent KV indexing, and modifications to the model's forward pass to route heads to different attention calls. The paper does not provide an estimate of the engineering effort required, nor does it release code (at the time of writing) that would reduce this barrier.

Mitigation status: The paper acknowledges in Appendix F that "post-training could potentially achieve further performance gains or improved compression rates" and that dummy head fine-tuning is reserved for future work—a fine-tuned model with dummy heads structurally removed (weights pruned or attention heads deleted) would be deployable in standard serving frameworks without custom kernels, partially addressing the deployment barrier. The paper does not discuss API-only or black-box deployment scenarios at all. The Triton implementation is described but not released, and the paper does not report the lines of code or engineering complexity of the implementation.

7. Implications and Future Directions

How This Work Changes the Landscape

Dummy Forcing introduces a diagnostic reframing rather than a paradigm shift: it redirects the KV cache compression conversation from token-level importance scoring to head-level functional specialization. This is not a new architecture or training objective—it is a new way of seeing what autoregressive video diffusion models have already learned. The discovery that approximately 25% of attention heads (and up to two-thirds under aggressive configurations, Figure 8) attend almost exclusively to the current frame, and that their entire historical KV caches can be removed with only 0.26% quality degradation (Table 1), establishes that cross-frame aggregation is not uniformly distributed across attention heads in these models. This is a genuine empirical finding that prior work had not documented, and it has immediate methodological consequences.

The most significant landscape change is in the granularity at which KV cache compression is conceptualized. Before this work, the efficiency literature—inherited from LLMs (H2O, StreamingLLM, DuoAttention, FastGen)—treated attention heads as universal participants in context processing, differing only in which tokens they attend to. The optimization problem was continuous: compute token importance scores, keep the top-k, discard the rest. Dummy Forcing demonstrates that for autoregressive video diffusion, the more natural optimization problem is discrete: classify heads into functional types, then assign per-type context lengths. This reframing enables compression ratios that token-level methods cannot express: entirely removing KV caches for entire categories of heads, rather than carefully selecting tokens within every head. The practical consequence is that Dummy Forcing achieves better end-to-end speedup (1.4× to 2.0×) than token-level methods (1.1× for R-KV and Infinipot-V, Table 2) despite compressing less aggressively in terms of cache length (27.8% vs. 16.7% of baseline, Table 9), because the per-step overhead of token selection is eliminated.

This work also reconciles a latent tension between two efficiency strategies that previously appeared in tension: the observation that LLM KV cache compression methods underperform on video (Section 2, Tables 2–3), and the intuition that video models should be more compressible than language models because of frame-to-frame redundancy. Dummy Forcing resolves this by showing that video models are more compressible, but at the head level rather than the token level—the redundancy is structural (entire heads doing work that doesn't require history) rather than token-level (specific tokens being unimportant). Prior LLM methods were solving the right general problem (cache compression) at the wrong granularity (token-level) for video, which explains their limited speedup.

The paper also shifts the research community's mental model of what happens inside autoregressive video diffusion transformers. The layered distribution of head types (dummy heads in early and late layers, neighbor heads in intermediate layers, Figure 9) suggests an emergent functional division: shallow layers extract current-frame features, intermediate layers perform cross-frame semantic aggregation, and deep layers refine the current frame for decoding. This is not proven—it is an interpretation—but it provides a testable hypothesis about transformer internals that goes beyond "attention heads learn to attend to relevant information." The cross-model replication (Self Forcing, LongLive, CausVid, Rolling Forcing, RealTime-14B) establishes that this pattern is not idiosyncratic, making it a plausible convergent property of autoregressive video diffusion training that future models are likely to exhibit as well.

Research directions that become more attractive after this work include: head-level structural analysis of video transformers (not just attention patterns but functional specialization across layers), training-time interventions that explicitly encourage or discourage head specialization (knowing that specialization emerges naturally), and cache budget reallocation strategies (following Section 4.4's demonstration that efficiency savings can be reinvested into capability improvements). Research directions that become less attractive include: token-level KV cache compression for video that does not account for head-wise utilization patterns (since Dummy Forcing shows these methods leave substantial performance on the table), and sparse attention patterns for autoregressive video models that don't leverage head-type classification (since the paper shows that simpler head-wise pruning can be more effective than complex sparsity patterns).

Follow-Up Research This Work Enables

Training with dummy head caches structurally removed. The paper explicitly identifies this as the most immediate follow-up (Appendix F): "we can first identify the dummy heads in pre-trained models using our proposed Dummy Forcing. Subsequently, by fine-tuning on a small dataset, we can completely remove the KV cache for dummy heads during training, forcing the model to concentrate its context aggregation capabilities on the few non-dummy heads." A strong follow-up would take a pre-trained Self Forcing or LongLive model, run DHP to classify heads, then fine-tune with dummy heads' KV cache inputs zeroed out (or with those heads structurally removed from the attention computation). The key metrics would be: (1) whether fine-tuning recovers the ~0.1–0.5% quality gap Dummy Forcing incurs, (2) whether fine-tuning enables even higher dummy ratios (e.g., 67% or 75%) without quality degradation, and (3) whether the fine-tuned model exhibits different head specialization patterns (do formerly non-dummy heads become more context-dependent to compensate?). The negative result would be if fine-tuning causes the model to redistribute context aggregation across heads, making previously prunable heads essential and reducing the effective compression ratio—this would suggest that the dummy head pattern is a fragile equilibrium that doesn't survive retraining.

Does the dummy head fraction depend on the sliding window size L? The paper profiles at a fixed window size (L ≈ 36 for Self Forcing) but never ablates whether the fraction of dummy heads changes when L varies. If the window is very small (L = 4), heads might be forced to use all available history because there's less redundancy—the dummy fraction might decrease. If the window is very large (L = 128), more heads might become dummy because the available context exceeds what's needed. A follow-up would profile the same model with varying L, compute the α_current distribution at each L, and measure whether the dummy head fraction (heads with α_current > 0.8) shifts. The hypothesis is that the dummy fraction increases with L because larger windows provide more redundant information that heads can learn to ignore. Confirming this would mean that Dummy Forcing's speedup is understated for models with larger windows, and that the method becomes more valuable as context lengths grow. Disconfirming it (dummy fraction constant across L) would suggest that head specialization is a fixed architectural property rather than an adaptive response to available context.

Dummy heads in bidirectional video diffusion models converted to autoregressive inference. The paper tests only natively autoregressive models. However, many production video models (e.g., HunyuanVideo, CogVideoX) use bidirectional attention during training and are converted to autoregressive or semi-autoregressive inference via causal masking at test time. Do these converted models exhibit dummy heads? If bidirectional training encourages all heads to use bidirectional context, then after conversion to causal attention, the attention patterns might be different from natively autoregressive models—heads might attempt to attend to "future" frames that are now masked, leading to different specialization patterns. A follow-up would profile a bidirectional model under causal masking, compute the same α_current metric, and apply Dummy Forcing. If dummy heads do not emerge in converted models, it would suggest that head specialization is a consequence of the autoregressive training objective specifically, not a general property of video transformers—this would bound the method's applicability. If dummy heads do emerge (perhaps at different fractions or in different layers), it would extend the method's applicability to a much broader class of models.

Per-dimension quality degradation analysis with varying dummy ratios. The paper reports only aggregate VBench total scores. A follow-up study would generate videos at multiple dummy head ratios (0%, 25%, 50%, 67%, 75%) and report per-dimension VBench scores (subject consistency, background consistency, motion smoothness, dynamic degree, aesthetic quality, imaging quality) to identify which quality dimensions are most sensitive to dummy head pruning. The paper's layered specialization hypothesis (Figure 9) predicts that pruning dummy heads in early layers might degrade low-level visual quality (texture, imaging quality) while pruning in intermediate layers might degrade temporal dimensions (motion smoothness, subject consistency). A per-layer ablation—applying Dummy Forcing only to specific layer ranges—would directly test this. The practical output would be a sensitivity map telling practitioners which dimensions to monitor when tuning the dummy head ratio for their application. The negative result would be if all dimensions degrade uniformly at similar dummy ratios, suggesting that dummy head pruning causes a general capacity reduction rather than affecting specific computational roles.

Cache budget reallocation as a general framework beyond video. The long-context experiments (Section 4.4) demonstrate that efficiency savings can be reinvested into capability improvements: cache budget saved from dummy and sink heads is reallocated to neighbor heads, achieving 6.58× longer effective cache. This principle—identify redundant capacity in one component, prune it, and reallocate the budget to a bottleneck component—is not specific to video. A follow-up would test the framework on autoregressive language models: profile attention heads in a long-context LLM (e.g., Llama-3, Qwen-2.5) for α_current-like metrics identifying heads that attend primarily to the current token rather than history, prune their KV caches, and reallocate the saved memory to extend the context window of retrieval-critical heads. The experiment would measure: (1) whether LLMs exhibit dummy-head-like specialization (some heads with very high α_current), (2) whether pruning their caches causes minimal perplexity degradation, and (3) whether extending non-dummy heads' context windows improves long-context benchmarks (e.g., needle-in-haystack, LongBench). This would test whether the dummy head concept generalizes beyond the video domain.

Why do dummy heads emerge? A training dynamics study. The paper observes dummy heads as a static property of trained models but provides no evidence about their origin. A training dynamics study would checkpoint a model periodically during autoregressive video diffusion training and compute the α_current distribution, the core set ratio of dummy heads (to measure when positions stabilize), and the per-layer dummy head distribution. This would answer: (1) Are dummy heads present at initialization (random weights), or do they emerge during training? (2) If they emerge, at what point in training—early (when the model learns basic frame structure) or late (when the model optimizes its capacity allocation)? (3) Does the dummy head fraction increase monotonically during training, or does it fluctuate? If dummy heads are present at initialization, they reflect an architectural inductive bias (certain head positions in certain layers naturally receive less cross-frame gradient signal). If they emerge mid-training, they reflect a capacity optimization process where the model discovers that delegating cross-frame aggregation to a subset of heads is more efficient. The answer has implications for training recipe design: if the emergence is late, training could be accelerated by explicitly encouraging head specialization earlier; if it's present from initialization, structural pruning at initialization could reduce training cost without sacrificing final quality.

Practical Applications and Downstream Use Cases

Real-time interactive video generation for consumer GPUs. The paper's headline 24.3 FPS on Self Forcing (Table 2) is measured on a single H100 GPU—a datacenter-class accelerator. However, the demonstrated 1.4× to 2.0× speedup factors are multiplicative with hardware-appropriate optimizations. If a base model achieves 8 FPS on an RTX 4090, a 1.4× speedup brings it to 11.2 FPS—still below real-time, but substantially closer. A deployment pipeline would: (1) profile the model once per architecture variant (not per prompt) using DHP, (2) hard-code the head classification into a compiled inference graph (e.g., TensorRT with custom attention plugins implementing the two-kernel PAF architecture), and (3) deploy at reduced resolution with the speedup enabling real-time or near-real-time generation. The key benefit is that no retraining is required, so existing pre-trained models can be accelerated immediately. The primary risk is that the speedup factor on consumer GPUs may differ from H100 due to memory bandwidth bottlenecks—the paper provides no consumer-hardware benchmarks, so initial deployment would require validation.

Cost reduction for cloud video generation APIs. For API-based video generation services (e.g., Runway, Pika) that deploy models at scale on GPU clusters, the 1.4× to 2.0× speedup translates directly to serving cost reduction: 1.4× speedup means 40% fewer GPU-seconds per video, which at datacenter scale represents substantial savings. A service generating 1 million videos per day at 5 seconds each, with baseline 17.6 FPS (~0.28 seconds per frame, ~1.4 seconds total per video on H100), would consume approximately 389 GPU-hours daily. A 1.4× speedup reduces this to ~278 GPU-hours, saving 111 GPU-hours per day. At ~2/GPU−hour(typicalH100cloudpricing),thissaves 2/GPU-hour (typical H100 cloud pricing), this saves ~81,000 annually—not transformative for large services, but meaningful. More importantly, the 2.0× speedup at 1080P (Table 4) matters for premium high-resolution tiers where per-video costs are much higher. The practical deployment path requires that the service have white-box access to model internals (ruling out API-only model providers) and that the Triton kernel modifications be compatible with the serving infrastructure's batching and scheduling logic.

Long-form narrative video generation with character consistency. The long-context experiments (Section 4.4) demonstrate a concrete use case: generating multi-shot videos where a character appears, disappears during a scene transition, and must reappear identically. Current autoregressive models with standard sliding windows (~36 frames, ~1.5 seconds) cannot bridge transition gaps longer than the window, forcing them to regenerate characters from scratch—breaking visual continuity. By reallocating cache budget from dummy and sink heads to neighbor heads, Dummy Forcing extends the effective cache to 36 frames at the same generation speed, enabling the model to "remember" characters and backgrounds across transitions spanning up to ~1.5 seconds. The practical deployment would target storytelling and content creation tools where users chain multiple prompts to create coherent narratives. The 1.03-point VBench improvement (68.45 to 69.48, Table 5) is modest in absolute terms but is measured on a challenging identity-preservation task—the qualitative difference (Figure 7, character identity preserved vs. regenerated) is visually striking. The limitation is that the extended cache is still bounded (36 frames), so transitions longer than that would still break continuity—this use case benefits from Dummy Forcing but does not solve the fundamental long-context problem.

Mobile and edge video generation via combined optimization strategies. While Dummy Forcing alone cannot make current video diffusion models run on mobile devices (baseline 17.6 FPS on H100 translates to perhaps 0.5–1 FPS on a flagship smartphone GPU), the method is orthogonal to and combinable with other optimizations. The paper demonstrates this with the Dummy Forcing + TeaCache combination achieving >30 FPS (Table 7). A deployment stack combining Dummy Forcing (head-wise cache compression), TeaCache or similar (diffusion step skipping), quantization (INT8/INT4 weights), distillation (student model with fewer parameters), and hardware-specific kernel optimizations could bring video generation closer to mobile feasibility. Dummy Forcing's specific contribution to such a stack is reducing the attention FLOPs without retraining, making it a drop-in component. The key practical insight is that these optimizations are independent axes: Dummy Forcing operates on attention structure, TeaCache on denoising schedule, quantization on weight precision, distillation on model capacity. Practitioners can combine them multiplicatively. The paper's compatibility demonstration (Table 7) with just one orthogonal method suggests that a full stack could yield speedups well beyond 2.0× without architectural changes to the base model.