Architecture2025~105 minarXiv:2509.24695

SANA-Video: Efficient Video Generation with Block Linear Diffusion Transformer

score 8Junsong Chen, Yuyang Zhao, Jincheng Yu +17NVIDIA↑ 46★ 4,985Sep 29, 2025cs.CVhttps://github.com/NVlabs/Sana

🎯 Pitch

SANA-Video generates minute-long, 720p videos with a constant memory footprint by replacing quadratic attention with linear attention, enabling a fixed-size KV cache that never grows with video length—slashing training costs to just 1% of MovieGen's while matching the quality of 14B-parameter models at 16× the speed.

1. Executive Summary

This paper introduces SANA-Video, a small diffusion model that efficiently generates high-resolution (up to 720×1280), minute-length videos with strong text-video alignment at remarkably fast inference speeds, deployable on consumer GPUs. The system is built on PaLM-derived linear attention and evaluated against state-of-the-art video diffusion models (Wan 2.1, SkyReel-V2, CogVideoX) on the VBench benchmark. Two core designs drive the efficiency: Linear DiT, which replaces quadratic self-attention with ReLU-based linear attention integrated with Rotary Position Embeddings and temporal convolutions (reducing complexity from O(N²) to O(N) for a 4× speedup at 720p), and a Constant-Memory KV Cache for Block Linear Attention, a block-wise autoregressive generation scheme that exploits the cumulative-state property of causal linear attention to maintain a fixed O(D²) memory footprint regardless of video length (eliminating the growing KV cache of vanilla attention and enabling minute-long generation without memory explosion). SANA-Video achieves 16× faster latency than competing models while matching their VBench scores (84.05 total score on 720p T2V, versus 83.73 for Wan 2.1-14B), and trains for only 12 days on 64 H100 GPUs — approximately 1% of MovieGen's training cost. The paper's FLOPs-matched analysis establishes that test-time efficiency gains from linear attention are most pronounced on high-resolution, token-intensive generation tasks, where full attention becomes the dominant bottleneck — but that deep-compression VAEs (DCAE-V with F32T4C32 compression) remain essential for scaling to 720p, as even linear attention slows 2.3× without sufficient token reduction.

2. Context and Motivation

The Core Problem: Video Generation Is Token-Extensive and Computation-Bound

The fundamental problem SANA-Video addresses is straightforward to state but has resisted practical solution: video generation at high resolution and long duration is prohibitively expensive in both training and inference, rendering state-of-the-art models inaccessible to most researchers and users. The paper quantifies this starkly in Section 1: generating a single 5-second video at 720p resolution with Wan 2.1-14B requires processing over 75,000 tokens and takes 32 minutes on an H100 GPU. This is not a minor inefficiency — it is a gulf between what models can achieve in principle and what can be deployed in practice.

The root cause is the quadratic complexity of self-attention (O(N²) in sequence length N), which forms the backbone of every modern diffusion transformer (DiT). In image generation, where a 1024×1024 image might produce ~4,096 tokens, this is manageable. In video generation, where a 5-second 720p clip produces tens of thousands of tokens after VAE encoding, the attention computation dominates both memory and latency. The paper's Figure 1(d) illustrates this concretely: for a 480p video, causal full attention requires 46 GB of VRAM at 60 seconds and runs out of memory beyond 65 seconds, while linear attention stays flat at 7.2 GB regardless of duration.

This gap matters for three reasons the paper makes clear:

Accessibility and democratization. When training a competitive video model requires hundreds of GPUs for months (MovieGen, Open-Sora) and inference requires datacenter-class hardware with 30+ minute latencies, the technology is locked behind institutional-scale resources. The paper frames this explicitly: "Can we develop a high-quality and high-resolution video generator that is computationally efficient and runs very fast on both cloud and edge devices?" (Section 1). The goal is not merely academic — it is to make video generation deployable on consumer GPUs like the RTX 5090.

Long video generation remains unsolved. Even with large computational budgets, generating videos longer than ~5–10 seconds is fundamentally hard for full-attention models. The reason is architectural, not just budgetary: the growing KV cache in causal full attention consumes linearly increasing memory (O(N×D) for N cached tokens), making long-context generation a memory bottleneck that cannot be solved by simply adding more GPUs. The paper notes in Section 1 that "generating long video (>10 s) is hard to realize with these large models due to the full-sequence processing operation." This is a capability gap — not just an efficiency gap — because it means certain applications (minute-long scene generation, continuous streaming) are structurally impossible with vanilla attention regardless of hardware.

The training-inference cost asymmetry. The paper draws attention to an often-overlooked fact: video generation's token volume makes training costs equally prohibitive. SANA-Video's training requires 64 H100 GPUs for 12 days — which the paper frames as remarkably low, at approximately 1% of MovieGen's training cost and 10% of Open-Sora's. This isn't just bragging; it means that the cost of experimentation — trying new architectures, training on new datasets, iterating on designs — is orders of magnitude lower. Research velocity in video generation is throttled by training cost, and reducing that cost by 10–100× enables a fundamentally different research paradigm.

Where Prior Approaches Fall Short

The paper identifies several categories of prior work and their specific limitations, summarized in Section 6 (Related Work):

Large-Scale Full-Attention DiT Models (Wan, MovieGen, Veo3, Sora)

The dominant paradigm scales up standard diffusion transformers with full self-attention. These models achieve remarkable quality — Wan 2.1-14B scores 83.73 on VBench, Veo3 integrates audio for state-of-the-art performance — but their computational cost scales quadratically with token count. The paper's Table 2 shows Wan 2.1-14B requires 1,897 seconds on H100 for a single 720p 5-second video. This is not a model you can iterate on or deploy broadly.

The paper's criticism here is not about quality but about practicality at scale. These models work, but their cost structure means they are restricted to well-resourced industrial labs and cloud deployments. More fundamentally, their quadratic attention creates a hard ceiling on video length: the memory requirements grow without bound, making minute-long generation infeasible even with unlimited hardware budget.

Efficient Attention Mechanisms for Video

A substantial body of work attempts to reduce attention complexity, which the paper surveys in Section 6.3. The approaches fall into several families:

Factorized spatial-temporal attention (Make-A-Video, VideoCrafter, LaVie, Imagen Video): These decompose 3D video attention into separate spatial and temporal attention operations, reducing complexity from O((T×H×W)²) to O((H×W)² + T²). This is a significant improvement, but it retains quadratic complexity within each dimension — spatial attention still scales quadratically with spatial tokens, and temporal attention quadratically with frames. For high-resolution or long videos, this partial solution still bottlenecks.

Sparse attention with token skipping (Sparse VideoGen, Radial Attention, SpargeAttn, VSA, Sliding Tile Attention): These methods selectively skip certain token interactions based on heuristics (spatial locality, temporal proximity, semantic-aware permutation). While effective at reducing FLOPs, they share two limitations: first, they typically retain some quadratic operations (global attention layers or local windows), and second, the sparsity patterns are hand-designed or heuristically determined, potentially discarding important long-range dependencies. The paper notes that these methods "either retain some quadratic complexity due to global self-attention layers or are limited to local attention," explicitly positioning linear attention as a cleaner solution that provides global context without quadratic cost.

State-space models and Mamba-based architectures (LinGen, Matten): These replace attention entirely with linear-complexity state-space mechanisms. The paper acknowledges these as relevant but positions them as alternative linear-complexity approaches rather than direct competitors — the key claim is that SANA-Video's linear attention maintains global context (not just local) with constant memory, which some Mamba variants may not guarantee.

Autoregressive Long Video Generation (MAGI-1, SkyReel-V2, Self-Forcing, CausVid)

For generating videos longer than the model's native context window, the dominant paradigm is block-wise autoregressive generation combined with diffusion (Section 2.2 and 6.2). The idea: decompose a long video into blocks of 5–10 seconds, generate each block with a diffusion model conditioned on previous blocks, and chain them autoregressively.

This approach has two critical failure modes the paper identifies:

Memory growth from the KV cache. Vanilla causal full attention stores a KV cache that grows with the number of tokens. For block-wise autoregressive generation, each new block must attend to all previous blocks' keys and values. As Table 1 shows, this costs O(N×D) memory for N cached tokens — meaning the cache grows linearly with video length, eventually exceeding GPU memory. Recent works (MAGI-1, SkyReel-V2, Self-Forcing) address this by restricting attention to a local window, so each block only attends to the most recent W tokens. The trade-off is explicit: stable memory cost at the expense of losing global context. The paper calls this out directly: "While this maintains a stable cost, it comes at the expense of losing global-context information" (Section 3.3.1). For videos where continuity and long-range coherence matter (a character walking across multiple scenes, a consistent art style), this local window is a genuine quality limitation.

Exposure bias from the train-test gap. In block-wise autoregressive training, earlier blocks are ground-truth videos (from the dataset), but at inference they are model-generated outputs with artifacts and errors. This mismatch — called exposure bias — causes error accumulation: small errors in early blocks compound across the generation chain. Self-Forcing addresses this by training with model-generated (self-generated) conditioning blocks via autoregressive rollout. However, as the paper notes in Section 3.3.2, this is "limited by the increasing VRAM requirement of causal vanilla attention" — Self-Forcing uses local attention within a designed window and restricts self-generation to the pre-trained model's native length (5 seconds). LongLive extends this to 1-minute streaming training but still relies on local attention with a sink token. The paper's critique is that these solutions solve exposure bias at the cost of global attention, creating a trade-off between training-inference alignment and long-range coherence that has not been satisfactorily resolved.

Deep Compression VAEs for Video

The paper's analysis in Section 3.4 and Appendix C.1 identifies an additional bottleneck that prior work partially addresses: VAE compression ratio directly determines the number of tokens the DiT must process. The standard Wan 2.1-VAE uses F8T4C16 (8× spatial, 4× temporal, 16 channels) — a 16× compression that still leaves 75,000+ tokens for a 720p 5-second video. The concurrent Wan 2.2-VAE improves to F16T4C48 (21× compression) by combining a 16× spatial VAE with 2× patch embedding. LTX-Video uses F32T8C128 (64× compression).

The paper identifies two issues with these approaches for small diffusion models:

Channel dimension mismatch with pre-trained image models: Wan 2.2-VAE's 48 or 192 latent channels (after patch embedding) do not align with SANA's pre-trained T2I latent space (32 channels), slowing convergence during video adaptation.
Diffusion over large latent dimensions: "to achieve the same compression ratio, Wan2.2-VAE would require the model to predict a much larger latent dimension (192 vs. 32 in DCAE-V), a task that is difficult for a small diffusion model" (Section 3.4).

So even when compression ratios are comparable, the dimensionality of the compressed latent matters — a 32-channel latent is easier for a small model to predict than a 192-channel one, even if the total number of tokens is similar.

How SANA-Video Positions Itself

The paper's positioning can be understood through three lenses:

Architectural Positioning: Linear Attention as the Universal Primitive

Unlike prior work that treats linear attention as one of many efficiency tricks, SANA-Video makes it the universal attention mechanism — replacing all attention modules (self-attention, cross-attention) with ReLU-based linear attention, applied at every resolution and every generation length. This is a stronger claim than prior linear attention work: SANA showed this works for images; SANA-Video extends it to the far more token-intensive video domain and demonstrates that the efficiency advantages actually grow with resolution (2× speedup at 480p, 4× at 720p, as shown in Figure 6(c)). The paper frames this as a property of the attention mechanism itself: since O(N) grows more slowly than O(N²), the relative advantage of linear attention increases as N increases — exactly the regime video generation operates in.

The paper also positions linear attention as the enabler of global context with constant memory (Table 1, Section 3.3.1). This is a specific, non-obvious property that the paper derives from the reformulation of causal linear attention (Equation 3). Because the attention state can be accumulated as a running sum of K^T V ∈ R^(D×D) and K^T ∈ R^(D×1), the memory cost is O(D²) regardless of sequence length N. Since D (the head dimension) is a fixed architectural constant, this means the KV cache is genuinely constant-memory — not just "slower-growing" or "truncated to a window." The paper exploits this to enable minute-long generation with global attention, which no prior autoregressive video method achieves. This is not just an efficiency claim; it is a capability claim: SANA-Video can do things (global attention over 1-minute videos at constant memory) that full-attention models structurally cannot.

Training Paradigm Positioning: Efficient Adaptation, Not Training from Scratch

The paper's three-stage training pipeline (Section 3.1) is a deliberate positioning against the "train from scratch" approach of many large video models:

Stage 1 (VAE Adaptation): Adapt a pre-trained T2I model to a new video VAE in just 5-10k steps. This is possible because the Linear DiT architecture preserves the image model's macro-architecture, with video-specific modifications (temporal conv, 3D RoPE) added in a way that allows zero-initialization with skip connections.
Stage 2 (Coarse-to-Fine Pre-Training): Starting from the adapted T2I model, train on low-resolution short videos first, then progressively increase resolution and duration. This is not novel in itself — many models use progressive training — but the paper claims it is especially efficient because the image-pretrained weights already contain strong visual and semantic priors, so video training focuses on learning motion and temporal coherence rather than re-learning visual concepts.
Stage 3 (Autoregressive Block Training): For long video generation, the paper introduces a two-step autoregressive training approach (monotonically increasing SNR sampler + improved self-forcing) that simultaneously addresses distribution alignment and exposure bias — something prior work solves partially or at the cost of global context.

This positions SANA-Video not as a new architectural paradigm but as a demonstration that smart adaptation of efficient image models can match or exceed purpose-built video models at a fraction of the cost. The 12-day, 64-GPU training budget is both a practical result and a conceptual argument: video generation does not require video-native architectures trained from scratch on video data; it requires efficient attention and good initialization.

Practical Positioning: Consumer-Grade Deployment

The paper's focus on RTX 5090 deployment with NVFP4 quantization (Section 5) is a clear signal about its intended audience and use case. SANA-Video is not positioned as the highest-quality video generator — it positions itself as the one that is fast enough and small enough to run on a single consumer GPU while still being competitive with much larger models. The quantitative framing is specific: 29 seconds for a 5-second 720p video on an RTX 5090 (Figure 7), compared to 32 minutes for Wan 2.1-14B on H100. This is a 66× practical latency improvement for a model that achieves comparable VBench scores (84.05 vs. 83.73 total score, Table 2).

This positioning matters because it changes the value proposition: SANA-Video is not claiming to beat the largest models on raw quality (though it matches them on several metrics); it is claiming to deliver competitive quality at a radically different point on the cost-quality Pareto frontier. For applications where 30-second generation latency is acceptable but 30-minute latency is not — creative tools, real-time applications, on-device generation — this shifts what is possible.

3. Technical Approach

3.1 Reader Orientation

SANA-Video is a video diffusion model pipeline that takes a text prompt (and optionally a starting image) and produces a high-resolution video up to a minute long — all running on a single consumer GPU in under 30 seconds. The core problem it solves is that standard diffusion transformers use quadratic-complexity self-attention ( $O(N^2)$ in the number of video tokens), making high-resolution or long video generation either impossibly slow or outright memory-infeasible; SANA-Video replaces every attention operation with a ReLU-based linear attention ( $O(N)$ ) and exploits a mathematical property of causal linear attention to build a constant-memory KV cache — meaning the GPU memory footprint stays fixed regardless of how long the video gets, while still attending to every token ever generated (global context).

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components arranged in a pipeline:

Deep Compression Video Autoencoder (DCAE-V or Wan-VAE) — compresses raw video frames into a compact latent representation (reducing token count by 16× to 128×) and decompresses latents back to pixels at the end.
Text Encoder and Re-writer — a small decoder-only LLM that encodes the user's text prompt into conditioning embeddings, optionally rewritten by a VLM (Qwen-2.5-VL-7B) for richer captions.
Linear DiT Backbone — the core diffusion transformer where all standard self-attention and cross-attention modules are replaced with $O(N)$ ReLU linear attention, augmented with 3D Rotary Position Embeddings (RoPE) and a temporal convolution inside the Mix-FFN for motion modeling. This is initialized from a pre-trained text-to-image SANA model.
Block Linear Attention with Constant-Memory KV Cache — for long video generation (>5 seconds), a block-wise autoregressive inference scheme that accumulates a fixed-size state ( $O(D^2)$ memory) across blocks, providing global context without a growing KV cache.
Training and Data Pipeline — a multi-stage training strategy (VAE adaptation → coarse-to-fine pre-training → autoregressive block training) fed by a curated dataset filtered for motion quality, aesthetic quality, and saturation, with SFT on human-preferred samples.

Information flows as follows: a text prompt enters → the re-writer optionally enhances it → the text encoder produces embeddings → noise is sampled in the VAE latent space → the Linear DiT iteratively denoises the latent over multiple timesteps, conditioned on the text embeddings (and optionally the first frame for I2V) → for short videos, this produces the full latent in one pass; for long videos, blocks are generated autoregressively, each block attending to all previous blocks through the accumulated KV cache state → the final latent is decoded by the VAE into pixel-space video frames.

3.3 Roadmap for the Deep Dive

First, the Rectified Flow training objective (Equation 1), because all model training — short video, long video, T2I, T2V, I2V — shares this foundation, and $u$ -prediction is the core supervision signal.
Second, the Linear DiT architecture (Section 3.2), covering the ReLU linear attention mechanism, the integration of 3D RoPE (including the critical numerator/denominator split for stability, Equation 2), the spatial-temporal Mix-FFN, and why each design choice was made over alternatives.
Third, the Block Linear Attention with Constant-Memory KV Cache (Section 3.3), deriving the key property that makes constant memory possible (Equation 3), explaining the block causal Mix-FFN, and walking through the autoregressive inference algorithm (Algorithm 1).
Fourth, the autoregressive training paradigm for long video generation (Section 3.3.2), covering the monotonically increasing SNR sampler, the exposure bias problem, and the improved self-forcing approach enabled by constant-memory global attention.
Fifth, the deep compression VAE (Section 3.4) — why DCAE-V was chosen, the F32T4C32 compression, and the robustness-to-perturbation analysis that justifies it for small diffusion models.
Sixth, the multi-stage training strategy and data pipeline (Sections 3.1, 3.5) — the three training stages, the data filtering criteria (motion, aesthetics, saturation, captioning), and the SFT data curation process.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and architecture paper whose core idea is that replacing quadratic attention with linear attention throughout the diffusion transformer — and exploiting the cumulative-state property of causal linear attention for constant-memory autoregressive generation — enables video generation that is simultaneously high-resolution, long-duration, and fast enough for consumer GPUs, without sacrificing competitive quality.

Rectified Flow Training Objective

All variants of SANA-Video — T2I, T2V, I2V, short video, long video — are trained under the same Rectified Flow (RF) framework with an SNR sampler. The paper states the objective in Equation 1:

$\mathbb{E}_{c, t, x_0} \left\| u(x_t \mid t, c; \theta) - v(x) \right\|^2$

where $c$ is the conditioning embedding (text prompt, and optionally first-frame latent for I2V), $\theta$ represents the model parameters, $x_0$ is a clean data sample (image or video latent from the VAE), $x_t$ is the noised version at timestep $t$ , $u(x_t \mid t, c; \theta)$ is the vector field predicted by the diffusion model (the $u$ -prediction), and $v(x)$ is the target velocity — the direction from the noised sample back toward the clean sample along the rectified flow path.

What it computes: for each training sample, we take a clean latent $x_0$ , add noise to produce $x_t$ (using the SNR sampler to determine how much noise at which timestep), feed $x_t$ and the conditioning $c$ through the Linear DiT to get a predicted velocity field $u$ , and compute the $L_2$ distance between this prediction and the true velocity $v(x)$ that would transport the noised sample to the clean sample. The expectation is taken over the data distribution ( $x_0$ ), the conditioning ( $c$ ), and the timestep distribution ( $t$ ).

Why this form: Rectified Flow is chosen over standard DDPM or score-matching objectives because it frames generation as learning a straight-line transport from noise to data, which (a) allows fewer sampling steps at inference (the straight path is simpler to integrate) and (b) unifies image and video generation under the same $u$ -prediction framework — the model predicts a velocity field regardless of modality. The SNR sampler, inherited from SANA [9] and Stable Diffusion 3 [14], weights timesteps according to their signal-to-noise ratio rather than uniformly, focusing training on the most informative noise levels. This matters because the later stages of pre-training and autoregressive block training build directly on this objective — the monotonically increasing SNR sampler for long video generation (Section 3.3.2) is a structured variant of this same framework.

Unified conditioning for T2I, T2V, and I2V. The paper emphasizes that Equation 1 is a unified objective: for T2I and T2V, $c$ is the text prompt embedding and $x$ is an image or video latent. For I2V, $c$ includes both the text prompt and the first frame condition — and the implementation simply sets the noise on the first frame to zero during the noising process. This means $x_t$ has the clean first frame concatenated with noised subsequent frames, and the model learns to complete the video conditioned on that first frame. The paper notes this requires "no model modification" — it is purely a data-level conditioning trick that works because the Linear DiT processes all tokens (image and video, clean and noised) through the same attention mechanism.

Linear DiT Architecture: Replacing Quadratic Attention with $O(N)$ Linear Attention

The architectural core of SANA-Video is the Linear DiT, which extends the SANA image-generation architecture [9] to video by making two key modifications while keeping the macro-architecture structurally identical. The design strategy is deliberate: by preserving the overall block structure (self-attention → cross-attention → Mix-FFN with time conditioning), the video model can be initialized from pre-trained image weights and adapted efficiently rather than trained from scratch.

ReLU Linear Attention Mechanism. The standard attention operation in a transformer computes, for each query token $Q_i$ , a weighted sum of all value vectors $V_j$ where the weights are softmax-normalized dot products between $Q_i$ and all $K_j$ . This has $O(N^2)$ complexity because every query must interact with every key. The ReLU linear attention used in SANA-Video replaces the softmax kernel with a ReLU activation $\varphi(\cdot) = \text{ReLU}(\cdot)$ applied element-wise:

$O_i = \frac{\varphi(Q_i) \left( \sum_{j=1}^{N} \varphi(K_j)^T V_j \right)}{\varphi(Q_i) \left( \sum_{j=1}^{N} \varphi(K_j)^T \right)}$

where $O_i \in \mathbb{R}^d$ is the output for token $i$ , $Q_i, K_i, V_i \in \mathbb{R}^d$ are the query, key, and value vectors for token $i$ (after linear projections from the input), and $\varphi$ is the ReLU non-linearity applied element-wise. The sums $\sum_{j=1}^{N} \varphi(K_j)^T V_j \in \mathbb{R}^{d \times d}$ and $\sum_{j=1}^{N} \varphi(K_j)^T \in \mathbb{R}^d$ are computed once for all tokens — they do not depend on $i$ .

What it computes: every token's key and value are passed through ReLU (zeroing out negative components), then the outer product $\varphi(K_j)^T V_j$ (a $d \times d$ matrix per token, representing that token's "state contribution") and the key sum $\varphi(K_j)^T$ are summed across all tokens to produce two global accumulators. For each query token $Q_i$ , its ReLU-activated query $\varphi(Q_i)$ is multiplied against the accumulated state (numerator) and the accumulated key sum (denominator), computing a weighted average of all value vectors — but the weights come from the dot product $\varphi(Q_i) \cdot \varphi(K_j)$ rather than softmax attention scores. The denominator normalizes the output, analogous to the sum-to-one property of softmax weights.

Why this form: the key computational property is that the two accumulators $\sum_j \varphi(K_j)^T V_j$ and $\sum_j \varphi(K_j)^T$ can be computed in $O(N d^2)$ total — $O(d^2)$ per token to update the accumulators — and then each query interacts with the already-accumulated state in $O(d^2)$ , giving overall $O(N d^2)$ complexity rather than $O(N^2 d)$ for standard attention. Since $d \ll N$ for video (the head dimension is 112, while token counts can exceed 75,000 for a 720p video), this is a massive reduction. The choice of ReLU over other kernels (softmax, ELU, GeLU) follows SANA [9]: ReLU is cheap to compute, guarantees non-negative key and query representations (stabilizing the denominator), and has been shown to work well as an attention kernel in prior linear attention work.

Integration of 3D Rotary Position Embeddings (RoPE). Standard linear attention with a ReLU kernel lacks positional information — every query-key interaction is position-agnostic. The paper integrates 3D RoPE (encoding temporal and spatial positions) to provide the model with sequence ordering. However, naively applying RoPE in the same way as vanilla attention (rotating Q and K before the kernel) causes two problems for linear attention: (1) the ReLU kernel filters out negative components, but RoPE rotations can produce negative values, causing position information to be partially erased, and (2) when RoPE is applied to both Q and K in the denominator, the rotation can make the denominator sum approach zero or negative values, causing numerical instability.

The solution, formalized in Equation 2, applies RoPE after the ReLU kernel (so $\text{RoPE}(\varphi(Q_i))$ and $\text{RoPE}(\varphi(K_j))$ ), ensuring position encoding is applied to already non-negative vectors. For the stability issue: if both Q and K in the denominator are RoPE-rotated, the sum $\sum_{j=1}^{N} \text{RoPE}(\varphi(K_j))^T$ can contain negative values after rotation, potentially summing to zero. The paper modifies the denominator to use unrotated keys (or queries):

$O_i = \frac{\text{RoPE}(\varphi(Q_i)) \left( \sum_{j=1}^{N} \text{RoPE}(\varphi(K_j))^T V_j \right)}{\varphi(Q_i) \left( \sum_{j=1}^{N} \varphi(K_j)^T \right)}$

Why this split: the numerator uses RoPE on both Q and K, giving the attention computation access to relative position information (RoPE encodes relative distances through dot products of rotated vectors). The denominator strips RoPE from the keys (keeping only ReLU-activated K), ensuring the sum remains positive — since $\varphi(K_j) \geq 0$ element-wise after ReLU, $\sum_j \varphi(K_j)^T$ is a vector of non-negative entries, and when dotted with the non-negative $\varphi(Q_i)$ , the denominator is guaranteed positive. This split — RoPE in numerator, no RoPE in denominator — is the key insight that makes linear attention with positional encoding numerically stable. Figure 3(b) provides empirical evidence: removing RoPE from the denominator (green line) yields stable training, while including it causes loss spikes.

Figure 3(a) visualizes the effect: linear attention without any positional encoding produces a dense, unfocused attention map. Applying RoPE after ReLU ( $\text{RoPE}(\varphi(x))$ ) produces "a sparser, more localized attention pattern" — the model learns to attend preferentially to nearby spatial positions and proximate frames, which is exactly what video motion modeling requires (local temporal coherence, local spatial features). Applying RoPE before ReLU ( $\varphi(\text{RoPE}(x))$ ) would lose positional information because ReLU would zero out negative components of the rotated vectors.

Spatial-Temporal Mix-FFN. The Mix-FFN in SANA's original design includes a 3×3 depthwise convolution after the first linear layer to inject spatial locality into the features — addressing the observation that linear attention's attention maps are denser and less locally focused than softmax attention's (visible in Figure 3(a) left panels). For video, the paper extends this with a 1D temporal convolution appended via a shortcut connection at the end of each DiT block (Figure 2(c)):

The block's standard processing (linear self-attention, cross-attention, first MLP layer, spatial 3×3 convolution, second MLP layer) produces an intermediate output $x_{\text{spatial}}$ .
A temporal 1D convolution (kernel size 3 along the time axis, applied as $3 \times 1 \times 1$ in the 3D tensor of shape $[T, H, W, D]$ ) processes $x_{\text{spatial}}$ to aggregate information across adjacent frames.
The temporal convolution output is added to $x_{\text{spatial}}$ via the shortcut connection: output = $x_{\text{spatial}} + \text{TemporalConv}(x_{\text{spatial}})$ .

Why this design: the shortcut connection with zero-initialization (the temporal convolution weights start at zero) ensures that at the start of video training, the block behaves identically to the pre-trained image model — the temporal conv adds nothing initially. As training progresses, the temporal conv learns to aggregate motion information across frames. This is crucial because it allows the model to leverage pre-trained image weights without disruption. The separation into spatial processing (3×3 conv inside Mix-FFN) and temporal processing (1D conv at the block output) follows a factorized spatial-temporal design that avoids the $O(T^2 \times H^2 \times W^2)$ complexity of full 3D attention while still modeling both spatial texture and temporal motion.

Macro-architecture specifications (Table 6). SANA-Video-2B has:

Width (hidden dimension): 2240
Depth (number of DiT blocks): 20
FFN dimension: 6720 (increased from SANA's 5600 to accommodate 3D RoPE)
Number of attention heads: 20
Head dimension: 112 (increased from SANA's 32 to accommodate 3D RoPE, which applies rotations in head-dimensional space)
Total parameters: 2,056M (~2B)

The increase in head dimension from 32 to 112 is specifically because RoPE operates in pairs of dimensions within each head — a larger head dimension allows more fine-grained positional encoding. The increase in FFN dimension compensates for the additional representation capacity needed for spatio-temporal processing.

Block Linear Attention with Constant-Memory KV Cache

Section 3.3 introduces the mechanism that enables SANA-Video to generate arbitrarily long videos at constant memory cost. The foundation is the cumulative-state property of causal linear attention, which the paper derives from the basic linear attention formula.

Derivation from causal linear attention. In the causal (autoregressive) setting, the linear attention output for token $i$ can only attend to tokens $j \leq i$ (previous tokens and itself). Starting from the linear attention formula:

$O_i = \frac{\varphi(Q_i) \left( \sum_{j=1}^{i} \varphi(K_j)^T V_j \right)}{\varphi(Q_i) \left( \sum_{j=1}^{i} \varphi(K_j)^T \right)}$

(RoPE is omitted for clarity). Define the attention state for token $j$ as $S_j = \varphi(K_j)^T V_j \in \mathbb{R}^{d \times d}$ — the outer product of the ReLU-activated key and the value. This $d \times d$ matrix captures everything token $j$ contributes to future attention computations. The cumulative sum of states up to token $i-1$ is $\sum_{j=1}^{i-1} S_j \in \mathbb{R}^{d \times d}$ , and the cumulative sum of keys is $\sum_{j=1}^{i-1} \varphi(K_j)^T \in \mathbb{R}^d$ . Then for token $i$ :

$O_i = \frac{\varphi(Q_i) \left( \sum_{j=1}^{i-1} S_j + S_i \right)}{\varphi(Q_i) \left( \sum_{j=1}^{i-1} \varphi(K_j)^T + \varphi(K_i)^T \right)}$

The crucial insight: to compute $O_i$ , we only need (a) $\sum_{j=1}^{i-1} S_j$ (a $d \times d$ matrix — the accumulated state), (b) $\sum_{j=1}^{i-1} \varphi(K_j)^T$ (a $d$ -dimensional vector — the accumulated key sum), and (c) the current token's $Q_i, K_i, V_i$ . After computing $O_i$ , we update the accumulators:

$\sum_{j=1}^{i} S_j \leftarrow \sum_{j=1}^{i-1} S_j + \underbrace{\varphi(K_i)^T V_i}_{\text{new state } S_i}$ $\sum_{j=1}^{i} \varphi(K_j)^T \leftarrow \sum_{j=1}^{i-1} \varphi(K_j)^T + \underbrace{\varphi(K_i)^T}_{\text{new key sum}}$

Why this is constant-memory: the storage requirement is these two accumulators — a $d \times d$ matrix (the state sum) and a $d$ -dimensional vector (the key sum) — totaling $O(d^2 + d) = O(d^2)$ memory. Since $d$ (head dimension = 112) is a fixed architectural constant, this memory is independent of the sequence length $N$ . You could process 100 tokens or 100,000 tokens — the KV cache size stays at $O(112^2) \approx 12,544$ floating-point numbers (multiplied by the number of heads and layers). In contrast, standard causal full attention requires $O(N \times d)$ memory to cache the key and value vectors of all previous tokens — for $N = 75,000$ tokens at $d = 112$ across 20 layers and 20 heads, this balloons to gigabytes of memory, growing without bound as the video lengthens.

Table 1 comparison formalizes this across three attention types:

Attention Type	Memory for $N$ cached tokens	Compute per new token	Total compute for $N$ tokens
Causal Full Attention	$O(N \times D)$	$O(N \times D)$	$O(N^2 \times D)$
Causal Local Attention (window $W$ )	$O(W \times D)$	$O(W \times D)$	$O(N \times W \times D)$
Causal Linear Attention	$O(D^2)$	$O(D^2)$	$O(N \times D^2)$

Since $N \gg W \gg D$ for long video generation, causal linear attention dominates both in memory (constant vs. growing) and total compute (linear vs. quadratic in $N$ ). The local attention compromise — capping memory at $O(W \times D)$ by restricting the attention window — achieves bounded memory but at the cost of global context: tokens more than $W$ positions away are invisible to the current token. Linear attention provides both bounded memory and global context — the accumulated state $\sum_j S_j$ implicitly encodes information from every token ever seen, since it is a sum over all tokens $j \leq i-1$ .

Block Causal Mix-FFN. The Mix-FFN's temporal convolution (kernel size 3) requires the previous frame to compute the current output for temporal boundary tokens. In the block-wise autoregressive setting, this means block $n$ needs access to the last frame of block $n-1$ . The paper implements this causally with two operations (Figure 4(b)):

Zero Padding: during training, an all-zero token of shape $1 \times H \times W \times D$ is appended to the end of each block. This prevents the temporal convolution from leaking information from block $n+1$ into block $n$ during training (since block $n$ 's temporal conv would otherwise see the first frame of block $n+1$ ).
Last-Frame Caching: the last frame of the previous block ( $\text{Token}_{-1} \in \mathbb{R}^{1 \times H \times W \times D}$ ) is cached and prepended to the next block before its temporal convolution. This provides the required temporal context without needing the full previous block in memory.

Together, the KV cache for the Linear DiT in the autoregressive setting stores: $\sum_j S_j$ (attention state accumulator), $\sum_j \varphi(K_j)^T$ (key sum accumulator), and the last-frame token for the temporal conv (conv cache $f$ ). All three have fixed size: $O(d^2)$ , $O(d)$ , and $O(H \times W \times D)$ respectively — none grow with video length.

Inference Algorithm (Algorithm 1). The autoregressive inference procedure for $M$ blocks is:

Initialize KV cache as $[\text{None}, \text{None}, \text{None}]$ (empty state sum, empty key sum, empty conv cache).
For each block $i = 1, \dots, M$ :
- Initialize the block's noise latent $x_i^{t_T} \sim \mathcal{N}(0, I)$ .
- Run the denoising loop for $T$ timesteps (e.g., $T = 50$ ): at each step $j$ , the model $G_\theta$ takes the current noisy latent, the timestep $t_j$ , and the KV cache to produce a clean prediction $\hat{x}_0$ , which is then used to compute the next noisy latent via the noise scheduler $\Psi$ (all steps except the last).
- At the final denoising step ( $j = 1$ ), the clean prediction $\hat{x}_0$ is appended to the output, and the KV cache is updated: $G_\theta^{KV}$ computes the new cumulative attention state, cumulative key sum, and conv cache from the clean latent of block $i$ , updating the KV entries.
Return all generated blocks concatenated.

Why this works in practice: the KV cache update at the end of each block is what enables global context. When block $i+1$ is denoised, its attention operations see the accumulated state $\sum_{j=1}^{\text{end of block } i} S_j$ — which includes contributions from every token in every previous block. There is no window truncation, no sink token, no approximation. The $O(d^2)$ memory cost is genuinely constant, and the compute per new token is $O(d^2)$ — meaning generating the 1,000th token costs the same as generating the 10th token. The paper reports that this enables a 4-step LongSANA to generate a 1-minute, 16 FPS, 480p video in 35 seconds on an H100 GPU — a real-time generation rate of 27 FPS (generating faster than the video plays back).

Autoregressive Block Training for Long Video Generation

Training the autoregressive variant (LongSANA) builds on the pre-trained 5-second SANA-Video model. The challenge is to teach the model to generate coherent videos across block boundaries while maintaining quality within each block.

Monotonically Increasing SNR Sampler. Standard diffusion training samples a random timestep $t$ for each sample. In block-wise autoregressive training, each block $i$ has its own timestep $t_i$ . The paper proposes a monotonically increasing SNR schedule: later blocks always have larger timesteps (more noise) than earlier blocks. Concretely:

Randomly select one block $b$ and sample its timestep $t_b$ using the standard SNR sampler [14].
For all other blocks, sample timesteps via a propagated probability distribution [19] that enforces $t_1 \leq t_2 \leq \dots \leq t_N$ .

Why increasing timesteps: this mirrors the inference process, where earlier blocks are cleaner (fully denoised) and later blocks are progressively noisier (diffused from random noise). At inference, block 1 is denoised from pure noise; block 2 is conditioned on the clean block 1 and denoised from noise; block 3 is conditioned on clean blocks 1–2 and denoised from noise; and so on. The monotonically increasing timestep schedule ensures training matches this — earlier blocks have small $t$ (little noise, close to clean, like they would be at inference after denoising), later blocks have large $t$ (much noise, like they would be at the start of inference).

The paper identifies two benefits: (1) faster convergence because the monotonically increasing space is much smaller than random independent timesteps for each block, and (2) better consistency because the model learns the causal dependency structure (earlier blocks provide clean conditioning for later blocks). Figure 6(d) confirms this visually — monotonically increasing timesteps produce higher-quality, more consistent outputs across blocks compared to random sampling.

Exposure Bias and Improved Self-Forcing. Exposure bias is the mismatch between training conditions (earlier blocks are ground-truth, clean videos from the dataset) and inference conditions (earlier blocks are model-generated outputs with imperfections). In standard autoregressive models, this causes error accumulation: small artifacts in block 1 propagate to block 2, get amplified, propagate to block 3, and so on. Self-Forcing [17] addresses this by training with model-generated conditioning — at training time, the model generates its own earlier blocks (via autoregressive rollout) and then conditions on those imperfect outputs. However, as the paper notes, Self-Forcing in full-attention DiT is "limited by the increasing VRAM requirement of causal vanilla attention" (Figure 1(c)) — the growing KV cache from self-generated content quickly exceeds GPU memory. Consequently, Self-Forcing uses a local attention window and restricts self-generated content to 5 seconds.

SANA-Video's constant-memory KV cache removes this limitation. Because the KV cache for linear attention stays at $O(d^2)$ regardless of sequence length, the model can self-generate much longer conditioning content — the paper mentions generating up to 1 minute during self-forcing training. This means the training signal includes long-range error propagation: the model sees how its own errors in block 1 affect block 2, how those compound in block 3, and so on, across many blocks. This better aligns the training condition distribution with the inference condition distribution, reducing error accumulation. LongLive [20] similarly explores streaming long training on 1-minute video, but is "still limited to local attention with sink" — so SANA-Video's improvement is specifically combining the self-forcing paradigm with global attention enabled by the constant-memory linear KV cache.

Deep Compression Video Autoencoder (DCAE-V)

The VAE serves as the bridge between pixel-space videos (high-dimensional, redundant) and latent-space representations (compressed, structured) that the DiT operates on. The compression ratio of the VAE directly determines the number of tokens the DiT must process.

Token count budgeting. A 5-second 720p video at 16 FPS produces $5 \times 16 = 80$ frames of $720 \times 1280$ pixels. With standard Wan-VAE (F8T4C16 compression): spatial $1280/8 = 160$ , $720/8 = 90$ , temporal $80/4 = 20$ — producing $160 \times 90 \times 20 = 288,000$ latent positions, each with 16 channels. This is the "75,000+ tokens" the paper refers to (after patch embedding, token counts vary). With DCAE-V (F32T4C32): spatial $1280/32 = 40$ , $720/32 = 22.5$ (approximately 22 or 23), temporal $80/4 = 20$ — producing ~18,000 latent positions. The token count drops dramatically, reducing the DiT's computational load proportionally.

DCAE-V specifications. The paper fine-tunes DCAE [21] into DCAE-V with:

Spatial downsampling factor $F = 32$ (compresses $720 \times 1280$ to ~ $22 \times 40$ )
Temporal downsampling factor $T = 4$ (compresses 80 frames to 20 latent frames)
Latent channels $C = 32$

The ratio of $F \times F \times T / C$ gives a compression ratio metric — approximately 128× for DCAE-V versus 16× for Wan-VAE. The 32 latent channels deliberately match SANA's pre-trained T2I latent space, enabling fast adaptation: the model's input/output dimensionality is unchanged, so pre-trained weights map directly.

Why not Wan 2.2-VAE (F16T4C48, then 2× patch embedding)? The paper identifies two issues for small diffusion models:

Latent dimension mismatch: Wan 2.2-VAE's 48 channels combined with 2× patch embedding produces an effective latent dimension of 192 (48 channels × 2×2 patch = 192). SANA's pre-trained model expects 32 channels. Adapting would require remapping 192 → 32 channels, losing the benefit of weight initialization.
Prediction difficulty: A small diffusion model (2B parameters) must predict the clean latent at each denoising step. Predicting a 192-dimensional latent per token is significantly harder than predicting 32 dimensions — the output space is 6× larger, requiring more model capacity to capture the same distribution.

Robustness to perturbation (Appendix C.1, Table 7). The paper's key VAE selection criterion is robustness under Gaussian perturbation — how well the decoder reconstructs a clean video from a noisy latent. The rationale is that during diffusion inference, the model's predicted latent $\hat{x}_0$ is not perfectly clean; it has residual noise or approximation error. A VAE that degrades severely under small perturbations will amplify these errors into visible artifacts.

Table 7 adds noise $\epsilon z$ (where $z \sim \mathcal{N}(0, I)$ and $\epsilon = 0, 0.1, 0.2$ ) to the encoded latent before decoding and measures PSNR, SSIM, and LPIPS. At $\epsilon = 0.1$ :

Wan 2.1-VAE: PSNR drops from 34.41 to 28.61 (degradation of 5.8 dB)
Wan 2.2-VAE: PSNR drops from 35.61 to 30.12 (degradation of 5.49 dB)
DCAE-V: PSNR drops from 33.25 to 31.91 (degradation of only 1.34 dB)

At $\epsilon = 0.2$ , DCAE-V maintains 29.34 dB and 0.90 SSIM, while Wan-VAEs drop to 24–26 dB and 0.78–0.84 SSIM. The paper interprets this as superior "reconstruction generalization" — DCAE-V's decoder is less sensitive to latent noise, making it more forgiving of the diffusion model's imperfect predictions. This is a critical property for small diffusion models, which have less capacity to precisely match the VAE's training distribution.

Multi-Stage Training Strategy

The paper trains SANA-Video in three stages (Section 3.1), each building on the previous:

Stage 1: VAE Adaptation on Text-to-Image. The pre-trained T2I model was trained with a specific VAE latent space. When switching to a video VAE (Wan-VAE for 480p, DCAE-V for 720p), the latent distribution changes — different compression ratios, different channel dimensions, different reconstruction characteristics. Rather than training from scratch, the paper fine-tunes the T2I model on image data encoded with the video VAE. This converges in "5-10k training steps" because only the VAE decoder distribution has shifted, while the semantic conditioning (text-to-content mapping) transfers directly. This stage uses the same architecture as the pre-trained model (no temporal modifications yet) and only processes images, not videos.

Stage 2: Coarse-to-Fine Video Pre-Training. The VAE-adapted model receives its video-specific modifications (3D RoPE, temporal convolution in Mix-FFN) and is trained on video data. The temporal components are zero-initialized with skip connections: the temporal convolution weights start at zero, and the skip connection means the block output is initially $x_{\text{spatial}} + 0 = x_{\text{spatial}}$ , identical to the image model. As training progresses, the temporal components learn to model motion without disturbing the pre-trained spatial representations.

The coarse-to-fine progression: start with low-resolution, short videos (192p, 2.5 seconds) with abundant data and relaxed filtering criteria, letting the model learn basic motion patterns efficiently. Progressively increase to higher resolution, longer videos (480p, 5 seconds) with stricter data filtering (higher aesthetic scores, better motion quality). The intuition: motion is a coarse property — you can learn "a cat jumps" at low resolution; details like fur texture and eye movement need higher resolution. Training the expensive high-resolution stage on already-motion-aware weights is more efficient than learning motion and details simultaneously from scratch. The paper mentions "different data filtering criteria" at each stage (Appendix D), suggesting looser filters early (more data, faster motion learning) and tighter filters late (higher quality, better aesthetics).

Training hyperparameters (Appendix B.1): AdamW optimizer, weight decay 0.03, constant learning rate $5 \times 10^{-5}$ , using Accelerate FSDP for sharded data parallelism. Final training: 64 H100 GPUs for approximately 12 days.

Stage 3: Autoregressive Block Training (for LongSANA). Starting from the pre-trained 5-second SANA-Video model, this stage teaches block-wise autoregressive generation using:

The monotonically increasing SNR sampler (described above)
Improved self-forcing (model-generated conditioning over long horizons, enabled by constant-memory KV cache)

This stage enables minute-long generation. The paper notes that this training is "continue training" — it builds on the already-trained short-video model rather than starting from scratch, leveraging the motion and quality learned in Stage 2.

Data Filtering Pipeline

The data pipeline (Section 3.5, Figure 5, Appendix D) processes raw videos into training-ready video-text pairs through five stages:

Scene Detection and Shot Cutting. Raw videos (from public and synthetic sources) are long and multi-scene. PySceneDetect [23] splits them into individual scenes, and FFmpeg [71] cuts each scene into 5-second clips at 16 FPS. The 5-second, 16 FPS target matches the pre-training setting and ensures each clip contains a single coherent scene.

Motion Filtering. Motion quality is assessed along two dimensions:

Optical flow magnitude using Unimatch [24]: frames are extracted every 0.5 seconds, resized to 320×576, and the average optical flow across selected frames is computed. Clips with too little motion (static scenes) or too much motion (rapid, jittery movement) are removed. The thresholds are set individually per data source — synthetic data might have different motion characteristics than real data — to ensure "moderate and clear motion."
Pixel difference using VMAF [25]: computed over consecutive frames and normalized. This catches temporal artifacts like frame drops, stuttering, or inconsistent frame timing that optical flow might miss.

The average optical flow value is also injected into the text prompt as Motion score: {unimatch value}, giving the model explicit motion controllability at inference (Figure 11 shows this works: higher motion scores produce visibly larger motion while maintaining consistency).

Aesthetic Filtering. A pre-trained video aesthetic model DOVER [26] scores each clip on three dimensions: aesthetic, technical, and overall. The paper uses the overall score to filter out low-quality videos — those with poor composition, awkward framing, or unappealing content. This follows a well-established principle from text-to-image work [53]: high-aesthetic training data improves generation quality because the model learns to associate prompts with visually pleasing outputs.

Saturation Filtering. Some data sources — especially synthetic data and HDR-to-SDR converted real data — exhibit unnaturally high color saturation. To prevent the model from learning these artifacts, frames are extracted every 0.5 seconds, converted from RGB to HSV color space, and the "S" (saturation) channel is averaged across pixels and frames to produce a saturation score. Clips exceeding a source-specific threshold are removed. The paper emphasizes this is particularly important for "synthetic data and real data converted from HDR to SDR" — these sources have systematic saturation biases that would otherwise pollute the training distribution.

Captioning with VLM. Following the finding in [3] that LLM-rewritten captions improve model performance, the paper uses Qwen-2.5-VL-7B [72] to caption each video clip. The captioning prompt (Figure 13) instructs the VLM to produce "80-100 words" describing the main subject's appearance, actions, expressions, environment, camera angles, and movement attributes using "simple and direct verbs." The example captions in Figure 13 are richly descriptive (e.g., "Photorealistic movie scene still, medium shot: A flamingo with iridescent pink feathers..."). This replaces the original captions from synthetic data sources (which may be misaligned with the actual video content) and enriches real data with detailed, consistent descriptions. The paper argues this makes the model "easier to learn" by providing consistent, distribution-matched text-video pairs.

SFT Data Curation. For the final supervised fine-tuning stage, approximately 5,000 human-preferred videos are selected with stringent criteria:

Motion: distinct object motion and/or camera motion with moderate magnitude, clearly focused action free from occlusions; camera movement must be "stable and smooth, without jittering, to maintain 3D consistency."
Aesthetics: balanced brightness, natural color, appealing composition and content.

The selected videos are then classified into four motion categories (human activities, animal activities, other objects, natural/urban scenes) and three aesthetic styles (realistic, cartoon, cinematic), and SFT data is sampled to maintain a balanced distribution across categories. The paper's Figure 14 shows the effect: fine-tuning on SFT data improves fine details (eyes in the first example) and motion realism (the pipe in the second example). This SFT stage is the final quality refinement — the model has already learned motion and semantics from the large-scale pre-training; SFT teaches it to prefer aesthetically pleasing, physically plausible outputs that align with human preferences.

Design Choices and Their Justifications: A Summary

ReLU linear attention over softmax attention: $O(N d^2)$ complexity vs. $O(N^2 d)$ — the efficiency gap widens with resolution and duration, exactly the regime video generation occupies. The choice of ReLU over other linear kernels (ELU, softmax approximations) follows SANA [9]'s empirical finding that ReLU provides sufficient expressivity with minimal computational overhead.
RoPE after ReLU, not before: preserves positional information (ReLU would erase negative components of RoPE-rotated vectors) and enables the numerator/denominator split that guarantees numerical stability (denominator stays positive without RoPE).
Spatial-temporal factorized Mix-FFN (3×3 spatial + 1D temporal) over full 3D attention: avoids $O(T^2 \times H^2 \times W^2)$ complexity while still modeling both spatial texture and temporal motion. The shortcut connection with zero-initialization enables seamless adaptation from image pre-trained weights.
DCAE-V over Wan 2.2-VAE for 720p: 32-channel latent matches pre-trained image model (faster convergence), smaller output dimension is easier for a 2B model to predict, and superior perturbation robustness (Table 7) reduces sensitivity to diffusion model approximation error.
Constant-memory linear KV cache over local attention windows: provides global context across arbitrarily long videos while maintaining fixed memory — no capacity-context trade-off. This is a capability advantage, not just an efficiency advantage.
Monotonically increasing SNR sampler over random timesteps: matches the inference causal structure (earlier blocks cleaner, later blocks noisier), reducing the training distribution's support and enabling faster convergence with better cross-block consistency.
Edit-distance-based difficulty estimation (in the prior paper analysis pattern): not applicable here — SANA-Video does not use difficulty estimation; it uses a uniform architecture for all videos. The "difficulty" analogue is resolution (480p vs. 720p) and duration (short vs. long), handled by the progressive training curriculum and the switch between Wan-VAE and DCAE-V.

4. Key Insights and Innovations

Innovation 1: Constant-Memory Global Attention as a Structural Capability, Not Just an Efficiency Gain

The paper's most distinctive intellectual contribution is the recognition that linear attention's cumulative-state property transforms long video generation from a memory-management problem into a solved architectural invariant. Prior work on long video generation — MAGI-1, SkyReel-V2, Self-Forcing, CausVid — all operates under the same assumption inherited from language modeling: autoregressive generation requires a growing KV cache, which must eventually be truncated to a local window to stay within GPU memory. This forces a hard trade-off between context length and memory, and every prior method accepts that trade-off: restrict the attention window to the most recent W tokens, use a sink token as a summary proxy, or limit self-generated conditioning to the pre-trained model's native context window.

Section 3.3.1's derivation (Equation 3) shows this entire framing is unnecessary for a specific class of attention mechanisms. Because linear attention decomposes into an accumulated state sum $\sum_j \varphi(K_j)^T V_j \in \mathbb{R}^{d \times d}$ and key sum $\sum_j \varphi(K_j)^T \in \mathbb{R}^d$ — both of fixed dimension independent of sequence length — the memory cost is genuinely $O(d^2)$ regardless of whether the video is 5 seconds or 5 minutes. This is not a clever compression scheme or a heuristic pruning method; it follows directly from the algebra of the ReLU kernel, which replaces the softmax normalization (which couples all tokens and prevents pre-computation) with a separable product form.

What makes this a fundamental rather than incremental advance: the paper isn't just reporting that linear attention is faster — that was already known from Katharopoulos et al. (2020) and SANA (2025). The contribution is the reconceptualization of the KV cache as a fixed-size state accumulator rather than a growing token buffer, and the demonstration that this reconceptualization unlocks capabilities (minute-long generation with true global attention, extended self-forcing training) that are structurally impossible under any full-attention or local-attention scheme, regardless of hardware budget.

The evidence for this as a capability claim rather than just an efficiency claim is Figure 1(c): causal full attention's VRAM grows from 7.2 GB at 1 second to 46 GB at 60 seconds to OOM at 65 seconds; block linear attention stays flat at 7.2 GB across the entire range. This is not a "4× cheaper" story — it's a binary capability boundary. Below 65 seconds, full attention works but is expensive; above 65 seconds, full attention stops working entirely on this hardware, while linear attention continues unchanged. The paper's LongSANA generating a 1-minute 480p video at 27 FPS real-time speed (Section 3.3.3) is a direct consequence of this property, not an incremental optimization.

This insight also resolves a tension in prior work that the paper surfaces in Section 3.3.2: Self-Forcing addresses exposure bias but limits itself to local attention because the full-attention KV cache grows too fast; LongLive extends streaming training to 1 minute but still uses local attention with a sink token. Both are trying to solve the train-test gap under a memory constraint that forces a context-quality trade-off. SANA-Video's constant-memory KV cache eliminates the constraint — not by finding a cleverer way to manage growing memory, but by using an attention mechanism where the memory doesn't grow. This is an architectural insight, not a systems optimization, and it reframes what's possible for autoregressive visual generation.

Innovation 2: The RoPE-Numerator/RoPE-Free-Denominator Split as a Stability Principle for Position-Aware Linear Attention

Prior to this work, integrating Rotary Position Embeddings into linear attention was known to be numerically unstable. The standard linear attention denominator $\varphi(Q_i)(\sum_j \varphi(K_j)^T)$ relies on the non-negativity of $\varphi(K_j)$ (guaranteed by ReLU) to ensure the sum is positive and the normalized output is well-behaved. Applying RoPE — which rotates vectors through pairs of dimensions, potentially producing negative components — to both Q and K in the denominator can cause the accumulated sum to approach zero or negative values, producing loss spikes and training collapse. The paper references this as a known issue [18], and prior attempts to combine positional encodings with linear attention (e.g., in the efficient attention literature) either avoided RoPE entirely or accepted degraded stability.

Section 3.2's solution — apply RoPE in the numerator (to both Q and K, providing position-aware attention scores) but remove RoPE from the denominator (keeping only ReLU-activated, non-negative K) — is a diagnostic insight that identifies why the instability occurs and isolates it to the normalization term specifically. The move is conceptually clean: positional encoding matters for the attention weights (numerator — which tokens get emphasized) but is unnecessary for the attention normalization (denominator — a scaling factor that should be stable and positive). By separating these concerns, the paper achieves both position-aware linear attention and guaranteed numerical stability without needing to engineer a new kernel or a new positional encoding scheme.

Figure 3(b) provides the critical evidence: the training loss curve when RoPE is included in the denominator diverges (presumably spiking), while the split formulation (numerator-only RoPE) tracks the stable training curve closely. Figure 3(a) closes the loop by showing that the split formulation actually works — it produces sparser, more localized attention maps compared to linear attention without any positional encoding, confirming that RoPE is providing meaningful positional signal even though it's absent from the denominator.

This is an incremental but practically important insight — the mechanism is a targeted modification of the linear attention formula rather than a new architectural paradigm, but it solves a genuine blocking issue for video generation where positional information is essential (temporal ordering of frames, spatial layout within frames). Without this fix, linear attention for video would either be position-agnostic (losing critical spatio-temporal structure) or numerically unstable (producing training crashes). The paper's framing in Section 3.2 — "this order is critical because it prevents the ReLU kernel from filtering out the positional information encoded by RoPE" — captures the subtlety: applying RoPE before ReLU causes the information loss; applying it after ReLU but in both numerator and denominator causes the instability. The split resolves both.

Innovation 3: Difficulty-Agnostic Architecture Adaptation Through Progressive Training and VAE Robustness Matching

SANA-Video does not use difficulty estimation or adaptive compute allocation (unlike the prior paper analysis pattern). Instead, it achieves efficiency through a difficulty-agnostic architectural design that makes every resolution and duration equally tractable — and then validates that the architecture's components are matched to the task through a robustness criterion that is not commonly articulated in video generation literature.

The key intellectual move appears in Appendix C.1 and Table 7: the selection between VAEs (Wan-VAE vs. DCAE-V) is not based solely on reconstruction quality at zero noise — Wan 2.2-VAE actually achieves higher PSNR (35.61) and SSIM (0.96) than DCAE-V (33.25, 0.94) on clean latents. Instead, the paper introduces perturbation robustness as the decisive criterion: how well does the decoder reconstruct from a noisy latent? The rationale — that the diffusion model's predicted $\hat{x}_0$ is never perfectly clean at inference — connects the VAE's properties to the diffusion model's training-test mismatch in a way that prior VAE comparisons for video generation typically do not. DCAE-V degrades by only 1.34 dB at $\epsilon = 0.1$ versus 5.5–5.8 dB for the Wan VAEs; at $\epsilon = 0.2$ , DCAE-V maintains 29.34 dB versus 24–26 dB. This is a 4–5 dB robustness advantage that directly translates to inference-time quality because the diffusion model's outputs live in this perturbed regime.

The paper frames this as matching the VAE to "a small diffusion model" — the 2B parameter DiT has less capacity to precisely fit the VAE's training distribution, so it needs a VAE whose decoder is forgiving of approximation error. This is a diagnostic principle that generalizes beyond the specific architecture: when deploying a resource-constrained generation model, the autoencoder should be selected not just for maximum clean-latent fidelity but for maximum robustness to the errors the generator will make in practice. Table 7 operationalizes this with a simple noise-addition experiment that any practitioner could replicate.

The progressive training strategy (Stage 1: VAE adaptation on images, 5-10k steps → Stage 2: coarse-to-fine video training → Stage 3: autoregressive block training) is individually not novel — progressive curricula are standard in video generation, and image-to-video weight transfer has been used since Make-A-Video and Tune-A-Video. What's intellectually distinctive is the combination of (a) architecture-level identity preservation (temporal components zero-initialized with skip connections so the model starts as a perfect image generator and learns motion without disrupting spatial quality) with (b) data-level progression (loose filters for motion learning, tight filters for aesthetic refinement). Many prior approaches add temporal layers as untrained modules that must learn from scratch, disrupting pre-trained spatial representations during early training. SANA-Video's design ensures the spatial quality is preserved at initialization and then refined — the temporal components can only improve on the baseline, never degrade it. Figure 6(b) shows the temporal convolution loss curve is lower than the baseline from the start (not crossing), confirming this property.

These are incremental contributions — they refine the training recipe rather than introducing a new paradigm — but they collectively explain how a 2B model trained for 12 days on 64 GPUs can match the quality of 14B models trained on far larger budgets. The VAE robustness matching and the identity-preserving temporal architecture are specific, transferable design principles that other practitioners can adopt, not proprietary hyperparameter settings.

Innovation 4: Monotonically Increasing SNR Sampling as a Causal Structure Prior for Block-Wise Diffusion Training

Section 3.3.2 introduces a training-time timestep schedule for block-wise autoregressive diffusion that is conceptually simple but represents a departure from standard diffusion training practice. In standard diffusion, each sample gets an independently sampled timestep. In block-wise autoregressive generation where multiple blocks are processed simultaneously, the natural extension would be to sample independent timesteps for each block — this is what the paper calls "random timestep sampling."

The proposed alternative — enforce $t_1 \leq t_2 \leq \dots \leq t_N$ by sampling one block's timestep from the SNR distribution and propagating to others — is a structural prior that encodes the inference-time causal relationship directly into the training distribution. The insight is that the training objective's role is not just to teach the model to denoise at each noise level, but to teach the model the dependency structure between blocks: earlier blocks provide cleaner conditioning for later blocks during inference, so the training distribution should reflect this asymmetry.

What distinguishes this from standard curriculum learning or noise scheduling is the propagation mechanism: rather than heuristically setting increasing noise levels, the paper samples one block's timestep from the optimal SNR distribution (preserving the benefits of SNR-weighted training) and propagates through a probability distribution that enforces monotonicity while maintaining coverage of the full timestep space. The paper argues this has "a much smaller sampling space than random timesteps, which results in faster convergence and better performance" — but the deeper point is that the smaller space is not arbitrary; it's the correct space that matches the inference causal structure. Random timesteps include configurations that never occur at inference (e.g., block 1 noisy, block 2 clean — which breaks the autoregressive dependency), wasting training compute on unrealizable scenarios.

Figure 6(d) provides the empirical grounding: monotonically increasing timesteps produce visually better quality and more consistency across blocks compared to random sampling. This is a modest but principled contribution — it doesn't change the architecture or the objective function, but it identifies that block-wise autoregressive diffusion has a different training distribution requirement than standard i.i.d. diffusion, and that matching this distribution through monotonicity constraints provides a free quality improvement without additional compute.

The connection to the broader autoregressive diffusion literature (Section 6.2) positions this as a refinement of ideas explored in diffusion forcing and AR-Diffusion — the paper builds on the concept that later tokens/blocks should have different noise levels than earlier ones, but contributes the specific finding that SNR-guided monotonic sampling (rather than fixed schedules or independent sampling) is the most effective instantiation for this particular architecture and task.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary evaluation uses VBench [31], a comprehensive benchmark for video generation models that evaluates multiple dimensions of quality. The paper evaluates both Text-to-Video (T2V) and Image-to-Video (I2V) generation on VBench's standard test prompts. For VAE evaluation (Appendix C.1), 1,000 samples from Panda-70M are used to measure reconstruction quality under perturbation. For autoregressive long video comparison (Table 5), VBench is again the evaluation framework.
Base model(s). The core model is SANA-Video-2B, a 2,056M-parameter video diffusion transformer (Table 6) initialized from the pre-trained SANA-1.6B text-to-image model [9]. The architecture uses a Linear DiT backbone with 2240 hidden dimension, 20 layers, 6720 FFN dimension, and 20 attention heads with head dimension 112. For FLOPs-matched comparisons and efficiency benchmarking, the paper compares against Wan 2.1-14B, Wan 2.1-1.3B, Wan 2.2-5B, CogVideoX-5B, MAGI-1 (4.5B), Step-Video (30B), SkyReels-V2 (1.3B), Open-Sora-2.0 (14B), HunyuanVideo-I2V (13B), and LTX-Video (2B). These span small (1.3B), medium (2–5B), and large (13–30B) model scales, with both full-attention and efficient-attention architectures.
Metrics. Performance is measured using VBench's composite scores: Total Score (overall), Quality Score (visual fidelity, temporal consistency, motion smoothness), and Semantic Score (text-video alignment for T2V; I2V Score for image-to-video). For VAE evaluation: PSNR (peak signal-to-noise ratio, higher is better), SSIM (structural similarity, higher is better), and LPIPS (learned perceptual similarity, lower is better). Efficiency is measured as generation latency in seconds on one H100 GPU with BF16 precision, batch size 1, at a specified resolution and frame count using each model's default inference steps. Speedup is reported relative to a reference model (typically Wan 2.1-14B or Open-Sora-2.0). For long video generation, VRAM usage in GB is measured as a function of video length (Figure 1(c)). The FLOPs-matched comparison is implicit in the latency and parameter-count trade-offs rather than explicitly computed as total FLOPs.
Baselines. The paper benchmarks against eight prior and concurrent video generation models in the T2V setting (Table 4): Wan 2.1-14B [3], Wan 2.1-1.3B [3], Wan 2.2-5B [3], CogVideoX1.5-5B [29], MAGI-1 (4.5B) [5], Step-Video (30B) [28], SkyReels-V2 (1.3B) [6], and Open-Sora-2.0 (14B) [25]. In the I2V setting: MAGI-1, Step-Video-TI2V, CogVideoX-5B-I2V, HunyuanVideo-I2V (13B) [30], and Wan 2.1-14B. For long video autoregressive generation (Table 5): CausVid [16], SkyReels-V2, and Self-Forcing [17]. For the 720p high-resolution comparison (Table 2): Wan 2.1-14B, Wan 2.1-1.3B, and Wan 2.2-5B. For efficiency-only comparisons (Figure 1), all models are measured at 480×832×81 resolution with their default inference steps.
Generation budget / compute accounting. Efficiency comparisons use generation latency (wall-clock seconds on one H100 GPU at BF16) as the primary compute metric, measured at matched resolution and frame count (480×832×81 for Table 4, 720×1280×81 for Table 2). This accounts for differences in model size, attention mechanism, VAE compression ratio, and inference steps simultaneously — rather than isolating FLOPs, it measures end-to-end practical cost. For the VAE comparison, the compression ratio is reported as a multiplicative factor (e.g., DCAE-V achieves 128× compression). Training cost is reported as GPU-days (64 H100 GPUs × 12 days). For the autoregressive block training (Figure 6(d)), generation quality is compared at matched compute between monotonically increasing and random timestep sampling. The paper does not standardize to a FLOPs-equivalent metric across models — the latency measurement incorporates all architectural differences (VAE compression, attention mechanism, model depth/width) into a single practical cost.
Cross-validation / statistical protocol. The paper does not report cross-validation or statistical significance testing. Performance is reported as point estimates on the VBench test set without confidence intervals. For the 720p comparison (Table 2), only four models are compared (Wan 2.1-14B, Wan 2.1-1.3B, Wan 2.2-5B, SANA-Video-2B). For the main VBench comparison (Table 4), 8 T2V and 6 I2V models are evaluated. The VAE perturbation experiment (Table 7) uses 1,000 Panda-70M samples with noise levels $\epsilon = 0, 0.1, 0.2$ . Ablation studies (Figure 6) report training loss curves and visual comparisons rather than statistical metrics on a validation set. The absence of error bars or statistical tests on VBench scores means the reported differences (e.g., SANA-Video's 84.05 vs. Wan 2.1-14B's 83.73 Total Score on 720p T2V) cannot be assessed for statistical reliability — a gap given the VBench test set size and the fact that the difference is only 0.32 percentage points.

Main Quantitative Results

Efficiency and Performance on 720p High-Resolution Video Generation (Table 2)

The headline result compares SANA-Video-2B against Wan 2.1-14B, Wan 2.1-1.3B, and Wan 2.2-5B on 720×1280×81 resolution videos. SANA-Video-2B achieves 36 seconds latency on an H100 GPU — this is 53× faster than Wan 2.1-14B (1,897 seconds), 11.1× faster than Wan 2.1-1.3B (400 seconds), and 3.2× faster than Wan 2.2-5B (116 seconds). Despite being the smallest or second-smallest model in the comparison, SANA-Video achieves the highest VBench Total Score of 84.05, marginally ahead of Wan 2.1-14B (83.73), Wan 2.1-1.3B (83.38), and Wan 2.2-5B (83.28).

Breaking down the VBench components: SANA-Video achieves the highest Semantic Score of 81.73, substantially ahead of Wan 2.1-14B (75.58), Wan 2.1-1.3B (74.22), and Wan 2.2-5B (76.28). This is the paper's strongest quality claim — the small linear-attention model achieves better text-video alignment than the much larger full-attention models, likely attributable to the efficient training pipeline and high-quality captioning (Section D). On Quality Score, SANA-Video achieves 84.63, which is lower than all three Wan models (85.77, 85.67, 85.03 respectively) — the paper acknowledges that quality (visual fidelity, temporal smoothness) is the area where linear attention slightly trails full attention's capacity for fine-grained detail modeling. However, the Total Score remains highest because the Semantic advantage (+6.15 over Wan 2.1-14B) outweighs the Quality deficit (-1.14).

An important caveat: the latency numbers are measured at each model's default inference steps. If Wan models use more denoising steps than SANA-Video, some of the speedup is attributable to step count rather than architectural efficiency. The paper does not report these step counts in the main comparison, though Figure 1(b) notes "50 denoising steps" as the default setting for the latency bar chart. If SANA-Video uses fewer steps than Wan (which the Rectified Flow framework enables — straight-line paths require fewer integration steps), the 53× speedup conflates architectural efficiency with sampling efficiency. This is a genuine practical advantage but should be understood as the combined effect of linear attention + Rectified Flow, not linear attention alone.

Comprehensive T2V and I2V Comparison on VBench (Table 4)

The main VBench evaluation (Table 4) compares SANA-Video against seven other models on 480×832×81 video generation, using each model's default inference settings. SANA-Video achieves 60 seconds latency — the fastest in the comparison, 1.7× faster than Wan 2.1-1.3B (103 seconds), 1.9× faster than CogVideoX1.5 (111 seconds), 2.2× faster than SkyReels-V2 (132 seconds), and 7.8× faster than Open-Sora-2.0 (465 seconds). The speedup relative to the slowest model (MAGI-1 at 435 seconds, which uses autoregressive generation) is 7.3×.

On T2V quality: SANA-Video achieves Total Score 83.71, which ranks 4th out of 8 and is comparable to Wan 2.1-1.3B (83.31) and Wan 2.1-14B (83.69), but behind Open-Sora-2.0 (84.34) and SkyReels-V2 (82.67 — technically lower, but SANA-Video's edge is marginal). The key differentiator is Semantic Score: 81.35, which is the highest among all T2V models — ahead of Open-Sora-2.0 (80.12), CogVideoX1.5 (79.76), Wan 2.1-14B (76.11), and substantially ahead of MAGI-1 (67.74) and Step-Video (71.28). On Quality Score, SANA-Video's 84.35 is mid-pack: behind SkyReels-V2 (84.70), Step-Video (84.46), Open-Sora-2.0 (85.4), and Wan models (85.23–85.59), but ahead of CogVideoX1.5 (82.78) and MAGI-1 (82.04). This pattern mirrors the 720p results: linear attention excels at semantic alignment but slightly trails full attention on pure visual quality.

On I2V: SANA-Video achieves Total Score 88.02, the second-highest behind MAGI-1 (89.28) and ahead of all other models including Wan 2.1-14B (86.86) and HunyuanVideo-I2V (86.82). The I2V Score of 96.40 is the highest among all models — ahead of MAGI-1 (96.12), Step-Video-TI2V (95.50), HunyuanVideo-I2V (95.10), CogVideoX-5B-I2V (94.79), and Wan 2.1-14B (92.90). This is notable: image-to-video generation requires strong conditioning on the first frame while generating plausible motion. SANA-Video outperforms much larger models on this metric (2B parameters achieving 96.40 vs. 14B achieving 92.90), suggesting the unified T2I/T2V/I2V training and the zero-noise first-frame conditioning scheme (Section 3.1) is particularly effective. Quality Score in I2V is 79.65 — competitive but not leading (MAGI-1 achieves 82.44).

A pattern emerges across both T2V and I2V: SANA-Video consistently leads on semantic/text-alignment metrics and is competitive-but-not-leading on visual quality metrics. This is consistent with the design philosophy: linear attention with RoPE and temporal convolutions captures spatio-temporal structure well enough for motion and semantics, but the ReLU kernel's denser attention patterns (Figure 3(a)) may not capture the fine-grained local texture details that softmax attention's sharper focus provides. The paper does not explicitly discuss this trade-off, but it is visible across all comparisons.

Long Video Autoregressive Generation (Table 5)

Comparing SANA-Video against other autoregressive video generation methods on VBench: SANA-Video achieves Total Score 83.70, comparable to Self-Forcing (84.31) and ahead of SkyReels-V2 (82.67) and CausVid (81.20). Quality Score is 84.43 (Self-Forcing: 85.07, SkyReels-V2: 84.70, CausVid: 84.05). Semantic Score is 80.78 — essentially tied with Self-Forcing's 81.28 and substantially ahead of SkyReels-V2 (74.53) and CausVid (69.80).

This is a critical comparison because it isolates the autoregressive mechanism's contribution: Self-Forcing and SANA-Video both use autoregressive generation with diffusion, but Self-Forcing uses full attention with a local window and restricted self-generation length (5 seconds), while SANA-Video uses linear attention with global context and extended self-forcing training (up to 1 minute). The results show comparable quality despite the architectural difference — Self-Forcing has a slight edge (+0.61 Total Score), but SANA-Video achieves this with constant memory and the capability to generate arbitrarily long videos, which Self-Forcing's full-attention architecture cannot match (Figure 1(c) shows full attention running OOM at 65 seconds while linear attention stays at 7.2 GB). The paper frames this as "comparable performance" while achieving a structural capability advantage — the near-parity in VBench scores with a method that uses quadratic attention and local windows is itself a strong result, because it means SANA-Video does not sacrifice short-video quality to enable long-video generation.

The comparison with SkyReels-V2 is similarly informative: SkyReels-V2 (1.3B parameters, 132 seconds latency) achieves 82.67 Total Score versus SANA-Video's 83.70 at 60 seconds. SANA-Video is both faster (2.2×) and higher quality, and unlike SkyReels-V2 (which restricts attention to a local window for long generation), SANA-Video maintains global context. This demonstrates that linear attention's constant-memory property translates to practical advantages on the quality-efficiency Pareto frontier, not just a theoretical complexity improvement.

VAE Performance and Robustness (Table 3, Table 7)

Table 3 compares VAE reconstruction quality on Panda-70M at 192p resolution. DCAE-V achieves 33.25 PSNR, 0.94 SSIM, and 0.03 LPIPS at 128× compression — lower than Wan 2.2-VAE (35.61 PSNR, 0.96 SSIM, 0.01 LPIPS at 21× compression) and comparable to Wan 2.1-VAE (34.41 PSNR, 0.95 SSIM, 0.01 LPIPS at 16× compression). In absolute reconstruction terms, DCAE-V is noticeably worse — which is expected given the 6–8× higher compression ratio. The key question is whether this reconstruction quality gap matters for generation, since the VAE's role in a diffusion pipeline is to provide a latent space that the DiT can learn to navigate, not to perfectly reconstruct videos independently.

Table 7 provides the more relevant comparison: robustness under Gaussian perturbation. At $\epsilon = 0.1$ (moderate noise added to the latent before decoding):

Wan 2.1-VAE: 28.61 PSNR, 0.89 SSIM, 0.06 LPIPS (degradation of 5.80 dB from clean)
Wan 2.2-VAE: 30.12 PSNR, 0.92 SSIM, 0.04 LPIPS (degradation of 5.49 dB)
DCAE-V: 31.91 PSNR, 0.93 SSIM, 0.04 LPIPS (degradation of only 1.34 dB)

At $\epsilon = 0.2$ (severe noise):

Wan 2.1-VAE: 24.25 PSNR, 0.78 SSIM, 0.16 LPIPS (degradation of 10.16 dB)
Wan 2.2-VAE: 25.94 PSNR, 0.84 SSIM, 0.10 LPIPS (degradation of 9.67 dB)
DCAE-V: 29.34 PSNR, 0.90 SSIM, 0.05 LPIPS (degradation of 3.91 dB)

DCAE-V is 3–4× more robust to latent perturbation than Wan VAEs when measured by PSNR degradation. This is the core evidence for the paper's claim that DCAE-V is better suited for small diffusion models: since a 2B DiT cannot perfectly predict clean latents, its outputs will have residual noise that falls in the $\epsilon = 0.1$ to $\epsilon = 0.2$ regime, where DCAE-V's reconstruction is substantially better than Wan VAEs despite having lower clean-latent fidelity. The LPIPS metric (perceptual similarity) is particularly striking: at $\epsilon = 0.2$ , DCAE-V achieves 0.05 versus 0.10–0.16 for Wan VAEs — the perceptual degradation is 2–3× smaller despite the 6–8× higher compression ratio.

The paper notes that this experiment tests "reconstruction generalization" and argues it justifies DCAE-V for the small model. This is a well-motivated ablative criterion — but it is not a generation-quality ablation, i.e., it does not compare generated videos using different VAEs on VBench. A direct comparison of generation quality with Wan-VAE vs. DCAE-V at 720p would more directly support the claim that the perturbation robustness advantage translates to better generation, but this experiment is not reported.

NVFP4 Quantization Results (Figure 7)

Deploying SANA-Video on an RTX 5090 GPU with NVFP4 quantization (via SVDQuant [32]) reduces generation latency from 71 seconds to 29 seconds for a 720×1280×81 video — a 2.4× speedup — and from 120 seconds to 50 seconds for a 480×832×81 video (also 2.4×). The quantization strategy selectively quantizes: the QKV and output projections in self-attention, the query and output projections in cross-attention, and the 1×1 convolutions in feed-forward layers. Normalization layers, temporal convolutions, and KV projections in cross-attention remain at higher precision.

The paper states this strategy "maintain[s] a quality indistinguishable from the BF16 baseline" — but no quantitative quality metrics (VBench scores, PSNR, LPIPS) are reported for the quantized model. This is a significant gap: the quality claim is based on visual inspection rather than measurement. Given that quantization can introduce subtle artifacts (temporal flickering, reduced detail, color shifts) that may not be visible in a few examples but would appear in benchmarking, the absence of quantitative quality evaluation limits the strength of the on-device deployment claim.

Ablation Studies and Robustness Checks

3D RoPE integration (Figures 3, 6(a)): Training loss curves with and without 3D RoPE show that including RoPE (applied after ReLU) produces "significantly lower training loss" (Figure 6(a)). The attention map visualization (Figure 3(a)) confirms the mechanism: without positional encoding, linear attention maps are dense and unfocused; with RoPE-after-ReLU, attention becomes "sparser, more localized" — concentrating on nearby spatial positions and proximate frames. The ablation where RoPE is applied before ReLU ( $\varphi(\text{RoPE}(x))$ ) is discussed but its loss curve is not plotted — only the "linear attention with no PE" and the proposed $\text{RoPE}(\varphi(x))$ are compared. A comparison with pre-ReLU RoPE would strengthen the claim that the ordering is critical.

Temporal convolution in Mix-FFN (Figure 6(b)): Adding the 1D temporal convolution to the Mix-FFN (with shortcut connection and zero initialization) yields a lower training loss curve compared to the baseline without temporal convolution. The paper claims this "significantly enhances performance." The loss improvement is visible but not dramatic — the curves are relatively close, and the paper does not report final VBench scores for this ablation to quantify the performance impact in terms of generation quality rather than training loss.

Linear vs. full attention latency scaling (Figure 6(c)): A latency comparison between SANA-Video's linear attention and an equivalent model using full attention at increasing resolutions shows that the relative speedup grows with resolution: at 480P, linear attention provides a 2× speedup; at 720P, the speedup increases to 4×. This directly validates the paper's $O(N)$ vs. $O(N^2)$ complexity argument — as the token count grows, the gap widens. This is the cleanest evidence that linear attention's advantage is not just a constant factor but a scaling property. However, the paper does not specify the full-attention baseline's architecture (same width/depth? same number of heads? same FFN?), making it unclear whether the comparison isolates the attention mechanism or conflates it with other architectural differences.

Monotonically increasing vs. random timestep sampling in autoregressive block training (Figure 6(d)): Visual comparison of generated video frames from two blocks (block 1 and block 3) shows that monotonically increasing SNR sampling produces "better quality and more consistency across blocks" compared to random timestep sampling. This is a qualitative ablation — no numerical metrics are reported. The effect is visible in the figure: random sampling shows more artifacts and less coherent motion across blocks. The paper argues this is because the monotonically increasing schedule matches the inference causal structure, but no intermediate schedules (e.g., loosely increasing but not strictly monotonic) are tested to characterize how strict the monotonicity constraint needs to be.

SFT data effect (Figure 14): A qualitative ablation showing generated videos with and without supervised fine-tuning on the 5,000 human-preferred samples. The SFT model shows improved fine details ("the eyes in the first example") and better motion realism ("the pipe of the second example"). No quantitative metrics are provided, and the examples shown are selected rather than random. While the effect is directionally convincing, the magnitude of SFT's contribution to VBench scores is not isolated.

Motion score controllability (Figure 11): A qualitative ablation on the I2V task showing that increasing the motion score appended to the text prompt produces videos with "larger but still consistent motion." This validates the motion controllability mechanism (injecting the average optical flow value into the prompt during training and inference) but is purely qualitative with only two examples at two motion levels. No evaluation of how motion score affects VBench Motion Smoothness or other metrics is provided.

Reconstruction generalization of VAEs (Table 7): As discussed in the main results, the perturbation experiment across three VAE models at three noise levels demonstrates DCAE-V's superior robustness. This is the most rigorous ablation in the paper — it uses a standardized test set (1,000 Panda-70M samples), quantitative metrics (PSNR, SSIM, LPIPS), and sweeps the noise parameter across three levels. It directly supports the architectural choice of DCAE-V for 720p generation with a small model. However, it evaluates VAE reconstruction quality, not end-to-end generation quality — the missing link is a generation benchmark comparing DCAE-V against Wan-VAE at 720p.

Autoregressive method comparison (Table 5): The comparison with CausVid, SkyReels-V2, and Self-Forcing on VBench isolates the long-video generation mechanism. All models use autoregressive block-wise generation; the differences are in the attention mechanism (linear vs. full vs. windowed), the KV cache management, and the training strategy. SANA-Video's comparable performance to Self-Forcing (which uses quadratic attention) while maintaining constant memory is the key finding, but the VBench evaluation is on standard-length videos, not on the long videos (1 minute) that the constant-memory KV cache enables. A long-video-specific benchmark (e.g., temporal consistency over 1-minute generations, scene coherence metrics) would more directly evaluate the claimed capability advantage.

Critical Assessment

Claim 1: SANA-Video is 16× faster while matching quality of state-of-the-art models

This claim, stated in the abstract and Figure 1(a), conflates different comparisons. The 16× figure appears to reference the latency advantage over Wan 2.1-1.3B (400 seconds vs. 36 seconds at 720p = 11.1×, Table 2) or over Open-Sora-2.0 (465 seconds vs. 60 seconds at 480p = 7.8×, Table 4). The actual 16× figure is not directly traceable to a single Table and appears to come from the comparison with SkyReels-V2 (132 seconds vs. 60 seconds = 2.2× — clearly not 16×) or an aggregate across models. The abstract number is not anchored to a specific paired comparison. The closest match is Figure 1(b): SANA-Video at ~36 seconds vs. Wan 2.1 at ~568 seconds on 720p = ~15.8×, which rounds to 16×. If so, the claim refers specifically to the 720p comparison against Wan 2.1-14B and does not generalize across all resolutions or all baselines. At 480p (Table 4), the speedup over the slowest model (MAGI-1) is 7.3×, and over the most comparable small model (Wan 2.1-1.3B) it is only 1.7×.

On quality matching: SANA-Video achieves comparable VBench Total Scores to Wan 2.1-1.3B and Wan 2.1-14B (Table 2, Table 4) and leads on Semantic Score. However, it trails on Quality Score across all comparisons (Table 2: 84.63 vs. 85.03–85.77; Table 4: 84.35 vs. 84.70–85.59). "Matching quality" is true for Total Score and Semantic Score but not for visual quality specifically — the paper's own decomposition shows that linear attention sacrifices some fine-grained visual fidelity for semantic alignment and speed. This is an honest trade-off but not fully conveyed by the phrase "matching quality."

The paper does not compare against Wan 2.2-5B in Table 4 (the main VBench table), which is notable because Wan 2.2 uses a higher-compression VAE and MoE architecture — making it the closest architectural analogue to SANA-Video's design philosophy (efficient attention, high compression). A direct head-to-head at 480p would be informative. Table 2 includes Wan 2.2-5B at 720p: SANA-Video is 3.2× faster and achieves higher Total Score (84.05 vs. 83.28), so the claim holds there, but the 480p omission is unexplained.

Claim 2: Constant-memory KV cache enables minute-long video generation at fixed memory cost

This claim is well-supported by the architecture analysis but weakly supported by generation experiments. The derivation in Section 3.3.1 (Equation 3, Table 1) establishes that causal linear attention has $O(D^2)$ memory independent of sequence length — this is a mathematical property, not an empirical claim, and it is correct. Figure 1(c) demonstrates it empirically: block linear attention VRAM stays flat at 7.2 GB from 1 to 65 seconds, while causal full attention grows from 7.2 GB to 46 GB to OOM. This is convincing evidence that the constant-memory property holds at the implementation level.

However, the quality of minute-long generation is demonstrated only through qualitative examples (Figure 12, a single 1-minute video of an Arctic fox). There is no quantitative evaluation of long-video quality — no benchmark for temporal consistency beyond 5 seconds, no human evaluation of long-video coherence, no comparison with other methods at 1-minute duration. The VBench evaluation in Table 5 uses standard-duration VBench prompts, not long videos. The claim that LongSANA "is able to generate motion consistent and semantically aligned long videos" is supported by exactly one cherry-picked example.

This matters because the constant-memory property is a necessary condition for long-video generation but not sufficient for quality. Degradation mechanisms — error accumulation across blocks, drift in content or style, catastrophic forgetting of early-block context despite global attention — could plausibly degrade quality at minute scale even if memory stays constant. The paper does not characterize these effects or demonstrate that the improved self-forcing training (Section 3.3.2) successfully mitigates them at 1-minute scale with measurable metrics.

The paper also does not compare long-video quality against Self-Forcing or SkyReels-V2 at matched long durations (e.g., 30-second or 1-minute generations). Self-Forcing uses local attention with a window — at 1 minute, it would have zero global context beyond the window, while SANA-Video would have full global context. A side-by-side long-video comparison would directly demonstrate the capability advantage claimed in Section 3.3.1.

Claim 3: DCAE-V is better suited for small diffusion models due to perturbation robustness

The perturbation experiment (Table 7) provides compelling evidence that DCAE-V's decoder is 3–4× more robust to latent noise than Wan VAEs. The logic connecting this to "better suited for small diffusion models" is sound: small DiTs predict less accurate latents, which land in the perturbed regime where DCAE-V excels.

However, the experiment has a gap: it tests VAE decoding from synthetically noised latents ( $x_t + \epsilon z$ ), but this does not match the actual error distribution of the diffusion model. The DiT's prediction error is not isotropic Gaussian — it has structured errors correlated with content, resolution, and timestep. Whether DCAE-V's robustness advantage generalizes from isotropic Gaussian noise to the structured errors of a 2B DiT is not tested. An experiment that passes actual DiT predictions (at various denoising step counts) through different VAEs and measures reconstruction would close this gap. The paper does not report this, though the overall generation quality comparison (Table 2) implicitly validates DCAE-V in the end-to-end pipeline — but without isolating the VAE's contribution from the DiT's.

The clean-latent reconstruction comparison (Table 3) also shows that DCAE-V is worse than Wan VAEs on PSNR (33.25 vs. 34.41–35.61) and SSIM (0.94 vs. 0.95–0.96). This means DCAE-V's higher compression ratio comes at a real reconstruction cost. The perturbation robustness does not negate this — it means DCAE-V is better when the latent is noisy, but Wan VAEs are better when the latent is clean. If a larger DiT could predict cleaner latents, Wan VAE might be the better choice. The paper's framing — DCAE-V is "the ideal choice for our small diffusion model" — is appropriately conditioned on model scale. The boundary conditions (at what model scale or prediction accuracy does Wan VAE become preferable?) are not explored.

Claim 4: Training cost is ~1% of MovieGen and 10% of Open-Sora

This claim is a cost comparison statement. SANA-Video trains on 64 H100 GPUs for 12 days. The paper cites MovieGen [7] and Open-Sora [8] costs but does not provide the comparison methodology — are they comparing GPU-hours, total FLOPs, or wall-clock time? MovieGen's training cost is not public (it's an industry model), and Open-Sora's cost depends on the specific training run. The 1% and 10% figures are stated without calculation details, making them difficult to verify. The paper would be stronger with a standardized FLOPs or GPU-hours comparison methodology, but this is acknowledged as an approximation.

Missing experiments that would strengthen the paper

VBench evaluation of NVFP4 quantized model: The on-device deployment claim (2.4× speedup, "quality indistinguishable") is entirely qualitative. A VBench score comparison between BF16 and NVFP4 at matched resolution would quantify the quality cost of quantization.
Long-video quantitative benchmark: A temporal consistency metric or human evaluation on 30-second and 1-minute generations, compared against Self-Forcing and SkyReels-V2 at those durations, would validate the constant-memory KV cache's capability advantage.
Isolation of linear attention vs. Rectified Flow contributions to speedup: The 53× speedup over Wan 2.1-14B at 720p conflates linear attention efficiency with the sampling efficiency of Rectified Flow. An ablation that runs both models at matched denoising steps would isolate the architectural contribution.
VAE ablation in the generation pipeline: End-to-end generation quality (VBench) with Wan-VAE vs. DCAE-V at 720p to validate that the perturbation robustness advantage translates to generation quality.
Scaling behavior of linear attention advantage: Figure 6(c) shows latency vs. resolution for linear and full attention, but only at two points (480p, 720p). A curve across multiple resolutions would characterize the scaling law and verify the $O(N)$ vs. $O(N^2)$ empirical scaling exponent.
Statistical significance on VBench comparisons: Several comparisons have margins of <1 point (e.g., 84.05 vs. 83.73 at 720p Total Score). Without confidence intervals, it is unclear whether these differences are reliable or within VBench's evaluation noise.

Cross-cutting concern: single evaluation benchmark

All quality evaluation uses VBench. While VBench is comprehensive for standard video generation metrics, it evaluates short videos at standard resolutions. SANA-Video's distinguishing contributions — high-resolution (720p) generation and long-video (1 minute) generation — are evaluated in VBench only at standard settings. The 720p comparison (Table 2) uses VBench but it is not clear whether VBench's prompts and evaluation pipeline are calibrated for 720p resolution. The long-video capability is evaluated only qualitatively. A more targeted evaluation suite — perhaps including temporal consistency metrics at long durations, resolution-specific quality metrics, and efficiency-quality trade-off curves across multiple resolutions and durations — would provide stronger evidence for the paper's specific contributions.

Summary assessment

The experimental section provides substantial evidence for efficiency claims (latency, memory, training cost) backed by quantitative measurements across multiple resolutions and model scales. The evidence for quality parity with larger models is mixed: Total Score and Semantic Score are competitive or leading, but Quality Score consistently trails full-attention models — a trade-off the paper acknowledges implicitly through the metric decomposition but does not discuss as a limitation. The evidence for long-video capability is the weakest part of the evaluation: the constant-memory property is mathematically and empirically validated, but the quality of long generations is demonstrated only through one qualitative example without benchmarks or comparisons. The paper would benefit from replacing some of the qualitative ablation figures (which show directionally correct but unquantified effects) with targeted quantitative evaluations of its most distinctive claims: long-video quality and quantized-model quality.

6. Limitations and Trade-offs

6.1 Single Benchmark, Single Model Family

The constraint. All quality evaluations use the VBench benchmark (Section 4.2, Tables 2, 4, 5) with a single model family (SANA-Video-2B, initialized from the SANA-1.6B T2I image model). The paper does not evaluate on other video generation benchmarks (e.g., UCF-101, Kinetics-400, MSR-VTT, EvalCrafter, or human evaluation studies) and does not demonstrate that the linear attention design generalizes across model scales or architectures. The paper states in Section 4 that the model is "representative of the capabilities of many contemporary LLMs" — but that claim is about the SANA base text encoder, not the video generation architecture itself.

The consequence. A practitioner cannot determine whether the efficiency-quality trade-off observed in VBench scores (leading on Semantic Score, trailing on Quality Score across all comparisons — Tables 2 and 4) is a property of linear attention in general or an artifact of this specific model size, training recipe, or VBench's evaluation criteria. VBench evaluates specific dimensions (motion smoothness, temporal consistency, aesthetic quality, text alignment) using automated metrics; these may favor or penalize linear attention's characteristics (denser attention maps, Figure 3(a)) in ways that do not generalize to other benchmarks or human judgment. The paper's strongest claim — that SANA-Video matches much larger models on Total Score while being 16× faster — depends entirely on VBench's weighting of sub-scores, and a different benchmark with different sub-score weighting could produce different conclusions about parity.

Additionally, the single-scale evaluation means we cannot assess whether linear attention's advantage scales with model size or saturates. A 14B linear attention model might close the Quality Score gap with full-attention models entirely, or it might show diminishing returns because the ReLU kernel's limited expressivity (compared to softmax) becomes the bottleneck. The paper provides no evidence either way.

What evidence exists. The Quality Score deficit is consistent across every comparison in the paper: Table 2 (720p, 84.63 vs. 85.03–85.77 for Wan models), Table 4 (T2V at 480p, 84.35 vs. 84.70–85.59 for competing models; I2V, 79.65 vs. 80.82–82.44). This is a systematic, 1–2 point gap that appears in every head-to-head and across both resolutions. The paper does not discuss this pattern or attribute it to a specific architectural cause. The single-benchmark limitation is not acknowledged explicitly in the main text; the paper treats VBench Total Score as the definitive quality measure throughout.

Mitigation status. Not addressed. The paper does not suggest evaluating on additional benchmarks, conduct human evaluations, or discuss the generalizability of VBench findings. Section 8 (Conclusion) does not mention this as a limitation or future work item. A replication on UCF-101 or MSR-VTT, or a human preference study, would substantially strengthen the quality parity claim.

6.2 Long Video Quality Is Evaluated Only Qualitatively

The constraint. The constant-memory KV cache (Section 3.3.1) is the paper's most distinctive architectural contribution and the enabler of minute-long video generation. However, the quality of long video generation is demonstrated through exactly one qualitative example (Figure 12, a 1-minute video of an Arctic fox) with no quantitative evaluation. The VBench comparison of autoregressive methods (Table 5) uses standard-length VBench videos — it evaluates short-video quality produced by the autoregressive mechanism, not actual long-video coherence, temporal consistency across blocks, or content drift over 30–60 seconds. The paper does not report any metric that specifically measures long-duration generation quality: no temporal consistency benchmark at 30+ seconds, no scene-coherence evaluation, no human study on long-video quality, and no comparison against Self-Forcing or SkyReels-V2 at matched long durations.

The consequence. The constant-memory KV cache is mathematically proven (Equation 3) and empirically validated for memory usage (Figure 1(c): 7.2 GB flat from 1 to 65 seconds), but the generation quality it enables at long durations is unverified. Several failure modes are plausible and unmeasured:

Error accumulation across blocks: Even with improved self-forcing training (Section 3.3.2), small artifacts in block 5 may compound through blocks 10, 20, and 60. The global attention mechanism provides access to all previous tokens, but it does not guarantee that the model uses this access correctly to maintain coherence over long horizons.
Content drift: A video of "a white Arctic fox runs through a forest" might gradually shift to a different animal, a different environment, or lose the motion pattern over 60 seconds. Global attention helps but is not immune to distributional drift in autoregressive generation.
Style degradation: Video quality (aesthetic score, motion smoothness) might degrade with block count as the model produces outputs further from its pre-training distribution.

The paper's claim that LongSANA "is able to generate motion consistent and semantically aligned long videos" (Appendix C.5) is supported by a single cherry-picked example — which is standard for a qualitative demonstration but insufficient to establish reliability.

What evidence exists. Figure 1(c) provides strong evidence that the memory property holds: linear attention VRAM is flat at 7.2 GB from 1 to 65 seconds while full attention grows to 46 GB and OOM. Section 3.3.3 states that "4-step LongSANA is able to generate 1-min and 16 FPS 480P video within 35 seconds on NVIDIA H100 GPU, achieving real-time, 27 FPS generation speed" — confirming the generation is fast, but not that it is good. Table 5 shows that SANA-Video's autoregressive mechanism achieves VBench Total Score 83.70 on standard-length videos, comparable to Self-Forcing's 84.31 — but this evaluates the mechanism's short-video quality, not its long-video scaling behavior.

Mitigation status. Not addressed quantitatively. The paper uses the standard VBench evaluation for autoregressive methods (Table 5) and provides one qualitative long-video example (Figure 12). Section 8 does not mention long-video quality evaluation as a limitation or future work. A longitudinal benchmark — measuring temporal consistency, semantic alignment, and aesthetic quality at 10-second intervals up to 60 seconds, compared against Self-Forcing (with its local attention window) — would directly test the capability claim that distinguishes SANA-Video from prior autoregressive methods.

6.3 NVFP4 Quantization Quality Is Not Evaluated Quantitatively

The constraint. The paper reports deploying SANA-Video on an RTX 5090 GPU with NVFP4 quantization (via SVDQuant [32]), achieving a 2.4× latency reduction (Figure 7: 71 seconds → 29 seconds for 720p 5-second video). This is positioned as a key practical contribution (Section 5, abstract: "deployable on RTX 5090 GPUs... accelerating the inference speed... from 71s to 29s"). However, the paper states quality is "indistinguishable from the BF16 baseline" without providing any quantitative quality measurement — no VBench scores, no PSNR/SSIM/LPIPS comparisons, no human evaluation, not even a side-by-side frame comparison between BF16 and NVFP4 outputs.

The consequence. Quantization at 4-bit precision (FP4) is aggressive for diffusion models, which are known to be sensitive to activation quantization error — especially in the later denoising steps where subtle details are refined. The selective quantization strategy (quantizing QKV/output projections in self-attention, query/output projections in cross-attention, and 1×1 convolutions, while keeping normalization layers, temporal convolutions, and cross-attention KV projections at higher precision) is a plausible heuristic, but its quality impact is unmeasured. Potential failure modes include:

Temporal flickering: Quantization noise that varies across frames can produce subtle jitter or flicker in otherwise smooth video regions — a common failure mode in quantized video models that is invisible in single-frame quality metrics but perceptually obvious in motion.
Reduced detail in high-frequency regions: Textures, fine patterns, and sharp edges are encoded in higher-precision activations and may be systematically degraded by 4-bit quantization.
Color shifts: Accumulated quantization error in cross-attention projections (which route text conditioning into visual features) can shift color distributions, especially in saturated or high-contrast regions.
Semantic drift: If the text conditioning pathway loses precision, the model's strong semantic alignment (its best metric across all comparisons) could degrade in quantization-specific ways.

The paper's "indistinguishable" claim is based on visual inspection of unspecified examples — which is insufficient to validate a deployment-critical quality claim.

What evidence exists. Figure 7 reports only latency numbers: BF16 (71s for 720p, 120s for 480p) vs. NVFP4 (29s for 720p, 50s for 480p). No quality metrics accompany these latencies. The paper specifies which layers are quantized (Section 5) but does not ablate this choice — e.g., quantifying how much quality degrades if temporal convolutions are also quantized, or whether the current selection is near-optimal. The SVDQuant reference [32] provides the quantization methodology, but the paper does not replicate SVDQuant's quality evaluation for the video generation task.

Mitigation status. Not addressed — explicitly unmeasured. The paper should report at minimum VBench Total/Quality/Semantic scores for the NVFP4 model at the resolutions tested in Figure 7, and ideally include a human preference study comparing BF16 vs. NVFP4 outputs. This is a critical gap because the on-device deployment narrative (abstract, Section 5) is one of the paper's three headline contributions, and its quality claim is unsupported.

6.4 The Comparison with Full-Attention Models Does Not Control for Denoising Steps

The constraint. The paper's efficiency comparisons measure generation latency (wall-clock seconds on H100, batch size 1) using each model's default inference settings (Tables 2, 4; Figure 1). SANA-Video uses Rectified Flow (Section 2.1), which enables straight-line transport paths requiring fewer denoising steps than the DDPM or score-matching objectives used by some competing models. The paper does not report how many denoising steps each model uses, nor does it conduct a step-matched comparison. The speedup numbers (53× over Wan 2.1-14B at 720p, 7.8× over Open-Sora-2.0 at 480p) therefore conflate architectural efficiency (linear vs. full attention) with sampling efficiency (Rectified Flow vs. other diffusion formulations).

The consequence. The headline 16× speedup claim is not a pure measure of linear attention's efficiency advantage. A practitioner choosing between SANA-Video and Wan 2.1 cannot determine how much of the speedup comes from the attention mechanism (which is the paper's core contribution) versus the diffusion formulation (which is inherited from prior work). If Wan 2.1 were adapted to use Rectified Flow with fewer sampling steps, the latency gap would narrow. Conversely, if SANA-Video was forced to use the same number of steps as Wan 2.1, its latency advantage would shrink.

This matters for two reasons:

Scientific attribution: The paper claims linear attention is the key enabler of efficiency, but the experiments do not isolate its contribution from the diffusion framework. A reader might attribute the speedup to linear attention when some (potentially substantial) fraction is due to Rectified Flow's sampling efficiency.
Practical decision-making: If a team has already invested in a full-attention DiT pipeline and is considering switching to linear attention, they need to know the attention-specific speedup at matched diffusion formulations and step counts. The paper does not provide this number.

The paper also does not report the quality of SANA-Video at increased step counts (e.g., 100 or 200 steps). If more denoising steps improve the Quality Score (where SANA-Video consistently trails), a practitioner might prefer a slower but higher-quality configuration. The latency-quality Pareto frontier across step counts is not explored.

What evidence exists. The paper mentions "50 denoising steps" in the Figure 1 caption as the default for latency measurement, but does not report step counts for competing models in Tables 2 or 4. The Rectified Flow objective (Equation 1) is defined in Section 2.1 with the note that it follows SANA [9] which uses RF — but the efficiency implications of RF vs. DDPM for the speedup comparison are not discussed. Figure 6(c) compares linear vs. full attention latency at matched resolution — but it does not specify whether the two are compared at matched step counts, and the full-attention baseline architecture is not described (same depth/width? same VAE?). If the full-attention model in Figure 6(c) uses the same number of steps as the linear attention model, then the 2× (480p) and 4× (720p) speedups are purely architectural — but this is not stated.

Mitigation status. Partially addressed indirectly: the paper's design philosophy is that linear attention + Rectified Flow are a jointly optimized system, and the practical latency advantage reflects this combination. This is a reasonable engineering position — deploy the fastest combination — but it weakens the scientific claim about linear attention specifically. A step-matched ablation (SANA-Video with and without linear attention, at matched RF step counts, same architecture otherwise) would isolate the attention mechanism's contribution. A step-count sweep for SANA-Video would characterize the quality-latency Pareto frontier.

6.5 The VAE Robustness Criterion Is Validated on Reconstruction, Not End-to-End Generation

The constraint. The choice of DCAE-V over Wan-VAEs for 720p generation is justified primarily through a perturbation robustness experiment (Appendix C.1, Table 7): Gaussian noise is added to encoded latents before decoding, and reconstruction quality is measured. DCAE-V degrades by only ~1.3 dB at $\epsilon = 0.1$ versus ~5.5–5.8 dB for Wan VAEs. The paper argues this robustness makes DCAE-V "the ideal choice for our small diffusion model" because the diffusion model's predicted $\hat{x}_0$ has residual noise of similar magnitude. However, this experiment evaluates VAE decoding in isolation — not the end-to-end generation pipeline where the DiT's prediction error distribution interacts with the VAE's decoder.

The consequence. The perturbation experiment proves that DCAE-V's decoder is robust to isotropic Gaussian noise added to clean latents. But the DiT's prediction error is not isotropic Gaussian — it is structured, content-dependent, and varies with timestep and resolution. The types of errors a 2B model makes when predicting $\hat{x}_0$ from $x_t$ (structured artifacts, systematic biases in high-frequency regions, error correlated with motion magnitude) may not resemble $\epsilon z \sim \mathcal{N}(0, I)$ at all. If the DiT's errors project onto latent dimensions that DCAE-V handles poorly (despite Gaussian robustness), or if Wan VAEs handle the DiT's specific error structure better than Gaussian noise, the perturbation experiment overstates (or understates) DCAE-V's practical advantage.

Additionally, Table 3 shows that DCAE-V has worse clean-latent reconstruction than Wan VAEs: 33.25 PSNR vs. 34.41–35.61, 0.94 SSIM vs. 0.95–0.96, 0.03 LPIPS vs. 0.01. This means that to the extent the DiT does produce accurate latents, DCAE-V will produce lower-fidelity video than Wan VAEs would. The net effect depends on the DiT's prediction accuracy distribution — which is not characterized. A large model that predicts cleaner latents might be better served by Wan VAE's superior clean-latent fidelity. The boundary condition (at what model size or prediction accuracy does the cross-over occur?) is unexplored.

What evidence exists. Table 7 provides strong evidence for DCAE-V's Gaussian perturbation robustness. Table 2 provides indirect end-to-end evidence: SANA-Video with DCAE-V achieves competitive 720p quality. But this does not isolate the VAE's contribution — the DiT architecture, the training data, and the progressive training strategy all contribute to the final quality. A direct end-to-end comparison of SANA-Video with DCAE-V vs. Wan-VAE at 720p (same DiT, same training, different VAE) would validate that the perturbation robustness advantage translates to generation quality. This experiment is not reported.

Mitigation status. Partially addressed by the overall generation results — the system works well with DCAE-V at 720p. But the specific claim that perturbation robustness is the reason DCAE-V is better is not causally validated. The paper could strengthen this by: (1) characterizing the DiT's actual prediction error distribution (comparing DiT-predicted $\hat{x}_0$ latents to ground-truth latents at various timesteps), (2) showing that this error distribution aligns with the perturbed regime where DCAE-V excels, and (3) conducting the end-to-end VAE ablation at 720p. The paper acknowledges the motivation (Section 3.4: "We hypothesize that a VAE with better reconstruction ability under perturbation will be a better fit") as a hypothesis, not a verified claim.

6.6 Quality Score Trails Full-Attention Models in Every Comparison — and the Source of the Gap Is Not Diagnosed

The constraint. Across every quantitative comparison in the paper — 720p T2V (Table 2), 480p T2V (Table 4), 480p I2V (Table 4), and autoregressive VBench (Table 5) — SANA-Video's Quality Score trails full-attention and competing models by 1–2 points. At 720p: 84.63 vs. 85.03–85.77 for Wan models. At 480p T2V: 84.35 vs. 84.70–85.59 for the top models. At 480p I2V: 79.65 vs. 80.82–82.44. This is systematic and consistent — not a one-off or resolution-specific artifact. The paper reports these numbers but does not analyze why linear attention produces lower Quality Scores, nor does it attempt to isolate which quality sub-dimensions (temporal consistency? motion smoothness? aesthetic quality? dynamic degree?) are driving the deficit.

The consequence. The paper's narrative emphasizes the Total Score and Semantic Score advantages (where SANA-Video leads), while the Quality Score gap is reported but unexplained. This matters because Quality Score captures visual fidelity dimensions that practitioners care about: temporal smoothness, frame-level aesthetic quality, motion realism. If the gap is inherent to linear attention (e.g., the ReLU kernel's denser attention maps cannot capture the fine-grained local detail that softmax attention's sparser, sharper focus provides — as suggested by Figure 3(a)), then it represents a fundamental trade-off: linear attention trades visual quality for speed and semantic alignment. If the gap is instead due to model scale (2B vs. 14B), training data volume, or the VAE compression ratio, then it might close with more parameters or different training. The paper does not distinguish between these hypotheses.

The absence of a Quality Score sub-dimension breakdown is particularly limiting. VBench's Quality Score composites multiple sub-metrics; if SANA-Video's deficit is concentrated in one sub-dimension (e.g., motion smoothness), a targeted architectural fix (e.g., a stronger temporal convolution, more temporal attention capacity) might close the gap. If the deficit is distributed across all sub-dimensions, the ReLU kernel's expressivity may be the root cause and require a different kernel.

What evidence exists. The Quality Score gap is visible in Tables 2, 4, and 5, with remarkable consistency (1–2 points in nearly every comparison). Figure 3(a) provides a potential mechanistic explanation: linear attention maps are "much denser and less focused on local details compared to softmax attention" — this diffuse attention pattern might miss the fine spatial correlations that produce sharp textures and smooth motion. But the paper does not explicitly connect the attention map observation to the Quality Score gap. The temporal convolution in Mix-FFN (Figure 6(b)) is proposed as a partial remedy for locality, but its effect on Quality Score specifically is not ablated.

Mitigation status. Not addressed. The paper reports the Quality Score numbers alongside Total and Semantic scores, allowing readers to observe the pattern, but does not discuss it as a limitation or trade-off. Section 8 (Conclusion) summarizes SANA-Video as achieving "competitive performance" without acknowledging that the competitiveness is primarily on Semantic metrics while Quality lags. Future work on improving linear attention's local detail capture — perhaps through learned kernel functions beyond ReLU, hybrid attention that mixes linear and sparse-local attention, or multi-scale attention — could directly target this gap, but the paper does not propose these directions.

7. Implications and Future Directions

How This Work Changes the Landscape

SANA-Video is not a paradigm shift in video generation — it does not introduce a new generative framework, a new training objective, or a new class of architectures. What it does is reframe the economics of video generation research and deployment by demonstrating that a specific, principled combination of existing building blocks (ReLU linear attention, RoPE with a stability-guaranteeing split, temporal convolutions, deep-compression VAEs, and block-wise autoregressive training) yields a system that matches the quality of models 7–14× larger while running 10–50× faster and training for 1% of the cost. This is an engineering reframing with scientific implications: it shifts the burden of proof onto the dominant full-attention paradigm by showing that a carefully designed linear-attention model can be competitive at a radically different point on the cost-quality Pareto frontier.

The paper resolves a tension that has been building in the video generation literature without being explicitly stated. On one side, the scaling hypothesis — exemplified by Wan 2.1-14B, Veo3, MovieGen — holds that video quality requires large models with full attention, trained on massive datasets at industrial scale. On the other side, a growing body of efficient-attention work (MAGI-1, SkyReel-V2, LinGen, sparse attention methods) claims efficiency gains but typically accepts a quality compromise or limits itself to local context windows. SANA-Video's contribution is to show that this trade-off is partially avoidable: with the right combination of attention mechanism (linear), positional encoding trick (RoPE with split numerator/denominator), locality injection (temporal conv in Mix-FFN), and VAE selection criterion (perturbation robustness rather than clean-latent fidelity), a small model can achieve comparable or better semantic alignment and competitive overall quality without the quadratic bottleneck.

The paper also shifts attention toward long-video generation as an architectural capability problem rather than a memory-management problem. Prior work on autoregressive video generation — MAGI-1, SkyReel-V2, Self-Forcing — treats the growing KV cache as an unavoidable cost that must be managed through local windows, sink tokens, or generation-length limits. SANA-Video demonstrates that linear attention's cumulative-state property makes this entire framing unnecessary: the KV cache is genuinely constant-memory with global context. This changes the conversation from "how do we manage growing memory for long videos?" to "how do we maintain generation quality over long horizons given that memory is solved?" — a qualitatively different research question.

Which research directions become more attractive and which become less so:

More attractive: Improving linear attention's visual quality (the consistent 1–2 point Quality Score gap), developing better kernel functions that capture local detail without sacrificing linear complexity, exploring hybrid linear-sparse attention that preserves the constant-memory property for global context while sharpening local focus, and designing long-video evaluation benchmarks that measure multi-block coherence rather than single-clip quality.
Less attractive: Scaling full-attention DiT models to higher resolutions without architectural changes — the paper shows that linear attention's relative advantage grows with token count (2× speedup at 480p, 4× at 720p, Figure 6(c)), meaning full attention becomes increasingly disadvantageous at the resolutions and durations that next-generation video applications demand. Pure scaling of full-attention models may still produce the highest absolute quality (the Quality Score gap suggests there is headroom), but the cost scaling is unsustainable, and linear attention provides a more scalable foundation for future work.

Follow-Up Research This Work Enables

Diagnosing and closing the Quality Score gap between linear and softmax attention. The paper's most consistent negative result — that SANA-Video trails full-attention models by 1–2 VBench Quality Score points in every comparison (Tables 2, 4, 5) — is reported but unexplained. A natural follow-up would decompose the Quality Score into its VBench sub-components (temporal consistency, motion smoothness, dynamic degree, aesthetic quality, imaging quality) for both SANA-Video and Wan 2.1-1.3B, identifying which specific dimensions drive the deficit. If the gap is concentrated in temporal consistency, it suggests the temporal convolution (Figure 6(b)) is insufficient and a stronger temporal modeling module is needed. If it is concentrated in aesthetic quality, it suggests the ReLU kernel's diffuse attention maps (Figure 3(a)) are failing to capture fine spatial textures. A mechanistic experiment — ablating the temporal convolution strength, the kernel function (ReLU vs. GeLU vs. learned kernel), and the RoPE frequency — against Quality sub-scores would directly inform whether the gap is architectural (fixable with better design) or fundamental (inherent to linear attention's reduced expressivity). The paper already provides the ablation infrastructure (Figure 6) and the VAE evaluation framework (Table 7); extending them to a detailed Quality Score decomposition is a low-risk, high-value follow-up.

Long-video evaluation benchmarks with multi-block coherence metrics. The paper's most significant capability claim — minute-long generation with global context at constant memory — is supported by exactly one qualitative example (Figure 12, the Arctic fox). This is the weakest link in an otherwise well-quantified paper. A follow-up study should define and measure long-duration generation quality: temporal consistency at 10-second intervals up to 60 seconds, semantic drift (does "a white Arctic fox in a forest" remain a white Arctic fox in a forest at second 30? second 60?), motion pattern preservation, and aesthetic quality stability across blocks. The natural baselines would be Self-Forcing (which uses local attention with a fixed window — at 60 seconds it has zero global context beyond the window, unlike SANA-Video) and SkyReel-V2. A comparison would directly test the paper's claim that constant-memory global attention provides a quality advantage for long videos, not just a memory advantage. The experiment is straightforward: generate 30-second and 60-second videos from all three models using the same prompts, measure the proposed metrics, and conduct a human preference study. The paper's existing VBench evaluation (Table 5) already validates that the autoregressive mechanism works well on standard-length videos; extending to long durations closes the evaluation gap between what the architecture enables and what the experiments demonstrate.

Learned kernel functions for linear attention that preserve local detail. The ReLU kernel $\varphi(x) = \max(0, x)$ is chosen for efficiency — it is cheap to compute and guarantees non-negativity for the denominator stability. But it is not learned, and its hard zeroing of negative components may discard features that are important for fine spatial and temporal detail. The attention map comparison (Figure 3(a)) shows that linear attention is "much denser and less focused on local details compared to softmax attention" — the kernel does not naturally produce the sharp, localized attention patterns that softmax provides. A natural extension would replace ReLU with a learned kernel function — a small MLP or a parameterized activation that maps queries and keys to non-negative feature vectors while learning to emphasize local interactions. The constraint: the kernel must output non-negative values (to keep the denominator positive) and should remain computationally cheap (to preserve the efficiency advantage). Candidate designs include: a learned PReLU with a small slope for negative values, a kernel that applies different activation functions at different layers (shallow layers use a broad kernel for global context, deep layers use a sharp kernel for local detail), or a hybrid attention block that applies linear attention globally but adds a small learned sparse-attention head for local refinement. The evaluation would measure whether a learned kernel closes any of the Quality Score gap against Wan models at matched model size and training budget, and whether the computational overhead of the learned kernel is small enough to preserve the 4× speedup at 720p.

Scaling laws for linear attention in video generation. The paper demonstrates efficiency and quality at exactly one model scale (2B parameters, 20 layers, 2240 width, 112 head dimension) and two resolutions (480p, 720p). A scaling study — training SANA-Video variants at 500M, 1B, 2B, 4B, and 8B parameters with matched training FLOPs, and evaluating latency, VBench scores, and the linear-vs-full-attention speedup ratio across resolutions (240p through 1080p) — would characterize the scaling behavior of linear attention for video. Key questions: Does the Quality Score gap between linear and softmax attention narrow with model scale (suggesting expressivity can be compensated by parameters) or remain constant (suggesting a fundamental kernel limitation)? Does the efficiency advantage (the $O(N)$ vs. $O(N^2)$ gap) continue to widen with resolution as predicted, and at what resolution does full attention become completely infeasible? What is the optimal allocation of parameters between the linear attention layers, the Mix-FFN, and the temporal convolution as model scale increases? The paper already provides the training infrastructure (64 H100 GPUs, 12-day budget for the 2B model) and the evaluation framework (VBench, latency profiling, VAE robustness testing); scaling to a family of models would establish linear attention as a scalable paradigm rather than a one-off efficient design. The Chinchilla-style framing — what is the compute-optimal architecture for video generation under an $O(N)$ attention constraint? — would connect this work to the broader efficient-scaling literature.

Combining linear attention with token compression for extreme-length video generation. The paper uses VAE compression (DCAE-V, 128×) to reduce token counts and linear attention to process the remaining tokens in $O(N)$ time. These are orthogonal mechanisms: VAE compression reduces $N$ , and linear attention reduces the per-token cost. A natural extension would push both axes further for extreme-length generation (5–10 minute videos). On the VAE side: train a DCAE-V variant with temporal compression factor $T = 8$ or $T = 16$ (vs. the current $T = 4$ ) — this would reduce token counts by another 2–4× at the cost of temporal resolution. On the attention side: introduce token merging or pruning between blocks — not as an approximation to linear attention (which already processes all tokens efficiently), but as a way to reduce the constant factor of the $O(N d^2)$ cost when $N$ becomes very large. The key question is whether the combination of aggressive VAE compression (trading temporal resolution for sequence length) and aggressive token reduction (trading per-token fidelity for total duration) can generate 5–10 minute videos with acceptable quality, and whether constant-memory global attention prevents the catastrophic coherence loss that would otherwise occur at those durations. The evaluation would need new metrics beyond VBench — 10-minute video generation is a different task than 5-second clip generation, and quality criteria (narrative coherence, scene-level consistency, long-range motion continuity) are not captured by standard benchmarks. This direction directly extends the paper's most distinctive capability claim into a regime where no prior method — including Self-Forcing — can operate due to growing KV cache memory.

Quantization-aware quality benchmarking for on-device video generation. The paper's NVFP4 deployment claim (2.4× speedup on RTX 5090, "quality indistinguishable from the BF16 baseline") is stated without quantitative quality evidence. A focused follow-up would evaluate the quantized model on VBench (or a subset of VBench prompts) at BF16 vs. NVFP4 precision, measuring Total, Quality, and Semantic Scores at both 480p and 720p resolutions. This directly addresses the paper's most significant evaluation gap and is actionable: the model is already quantized, the evaluation framework is already established, and the experiment is a straightforward comparison run. Beyond VBench, a human preference study comparing BF16 and NVFP4 outputs on 50–100 diverse prompts would provide the perceptual validation that "indistinguishable" claims require. If the quantized model shows a measurable quality drop on specific VBench dimensions (e.g., temporal consistency degrades more than semantic alignment), that would inform the selective quantization strategy — perhaps temporal convolutions should remain at higher precision if temporal artifacts dominate, or cross-attention KV projections should be quantized more aggressively if semantic alignment is robust. The paper's current quantization strategy (Section 5) is a heuristic based on layer type; a quality-informed refinement could improve the speed-quality Pareto frontier for the specific failure modes of quantized video diffusion.

Practical Applications and Downstream Use Cases

On-device video generation for creative tools and social media. SANA-Video's RTX 5090 deployment at 29 seconds for a 5-second 720p video (Figure 7) makes it the first competitive-quality video generator that runs on a single consumer GPU with sub-30-second latency. This unlocks integration into desktop creative applications (video editing software, motion graphics tools, game asset generators) where cloud API calls are undesirable due to latency, privacy, or cost. The 2.4× NVFP4 speedup means generation is fast enough for interactive use: a creator can iterate on prompts, seeing results in under 30 seconds rather than waiting 30+ minutes for a cloud model. The strong semantic alignment (81.73 Semantic Score at 720p, Table 2) means prompts are followed faithfully, reducing the "prompt debugging" cycle. The key deployment consideration: the Quality Score gap (1–2 points below full-attention models) means on-device SANA-Video will produce slightly less polished output than the best cloud models — but at a latency differential of 50–100×, many creative workflows will accept this trade-off for interactivity.

Real-time video generation for streaming and live content. Section 3.3.3 reports that 4-step LongSANA generates 1-minute 480p video at 27 FPS — faster than real-time playback. This enables applications where video must be generated on-the-fly: live streaming with AI-generated backgrounds or effects, real-time video-to-video translation (style transfer, animation), and interactive installations where the generated video responds to user input (motion, speech, text) with sub-second latency. The 27 FPS generation rate means the model can produce video as fast as it is consumed, enabling continuous, indefinite-duration generation if block-wise autoregressive quality holds. The key technical requirement for these applications is that the block-wise coherence remains stable over very long horizons (minutes to hours) — which the paper does not evaluate beyond a single 1-minute example. For practical streaming deployment, a quality stability benchmark at 5–10 minute durations would be essential. The constant-memory property (7.2 GB flat regardless of duration, Figure 1(c)) is the architectural prerequisite for this use case; no other competitive model can maintain global context at these durations without memory explosion.

Low-cost video data generation for training other AI systems. The paper mentions world model fine-tuning (Appendix E) for embodied AI, autonomous driving, and game generation. SANA-Video's training cost (12 days on 64 H100 GPUs) is low enough — and its inference speed fast enough (60 seconds per video at 480p, Table 4) — that generating large-scale synthetic video datasets becomes economically viable for research groups without industrial budgets. For example: fine-tuning SANA-Video on robot manipulation data (AgiBot, Figure 15) and generating 100,000 synthetic training videos of a specific manipulation task would cost approximately 1,700 GPU-hours at inference, or ~ $3,400 at$ 2/GPU-hour — orders of magnitude cheaper than collecting equivalent real-world data. The strong semantic alignment (leading I2V Score of 96.40, Table 4) means the generated videos accurately reflect the conditioning (first frame + text prompt), reducing the sim-to-real gap. The VAE perturbation robustness (Table 7) means the synthetic data's visual quality degrades gracefully even with imperfect generation, maintaining utility for downstream training. This application does not require the highest possible visual quality — it requires scale, diversity, and semantic accuracy, all of which SANA-Video provides at a cost point that makes large-scale synthetic data generation practical rather than aspirational.

Dependency tree

Papers this one builds on, walking backwards through the same domain and shared topics.

InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptation2025· architecture← in-domain predecessor
- SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse-Linear Attention2025· architecture← in-domain predecessor
FSVideo: Fast Speed Video Diffusion Model in a Highly-Compressed Latent Space2026· architecture← similar topic
- TurboDiffusion: Accelerating Video Diffusion Models by 100-200 Times2025· inference-optimization← LLM-judged similar
- Hybrid Linear Attention Done Right: Efficient Distillation and Effective Architectures for Extremely Long Contexts2026· architecture← in-domain predecessor
- Dynamic Large Concept Models: Latent Reasoning in an Adaptive Semantic Space2025· architecture← similar topic
- Multimodal Latent Language Modeling with Next-Token Diffusion2024· architecture← similar topic
Block Transformer: Global-to-Local Language Modeling for Fast Inference2024· architecture← similar topic
- Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality2024· architecture← in-domain predecessor
GoldFinch: High Performance RWKV/Transformer Hybrid with Linear Pre-Fill and Extreme KV-Cache Compression2024· architecture← similar topic
- Learning to (Learn at Test Time): RNNs with Expressive Hidden States2024· architecture← in-domain predecessor
- InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management2024· inference-optimization← topic thread
- Read More2023· architecture← similar topic

What it influenced

Later papers in the same chain. Direction reversed.

Fast-dLLM v2: Efficient Block-Diffusion LLM2025· llm-systems→ LLM-judged similar
- Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models2025· llm-systems→ in-domain predecessor
Less is More: Recursive Reasoning with Tiny Networks2025· architecture→ in-domain predecessor
- Artificial Hippocampus Networks for Efficient Long-Context Modeling2025· architecture→ in-domain predecessor
Block Transformer: Global-to-Local Language Modeling for Fast Inference2024· architecture→ similar topic
- Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs2025· scaling-laws→ LLM-judged similar
- Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling2024· architecture→ in-domain predecessor
GoldFinch: High Performance RWKV/Transformer Hybrid with Linear Pre-Fill and Extreme KV-Cache Compression2024· architecture→ similar topic
- The Pitfalls of KV Cache Compression2025· inference-optimization→ LLM-judged similar
- Achieving Human Level Competitive Robot Table Tennis2024· architecture→ in-domain predecessor
- ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference2024· serving→ topic thread
- Read More2023· architecture→ similar topic
- A Systematic Analysis of Hybrid Linear Attention2025· architecture→ similar topic
HunyuanCustom: A Multimodal-Driven Architecture for Customized Video Generation2025· architecture→ similar topic
- TurboDiffusion: Accelerating Video Diffusion Models by 100-200 Times2025· inference-optimization→ LLM-judged similar
- MiniMax-Speech: Intrinsic Zero-Shot Text-to-Speech with a Learnable Speaker Encoder2025· architecture→ in-domain predecessor
- Decoder-Hybrid-Decoder Architecture for Efficient Reasoning with Long Generation2025· architecture→ similar topic
- MoGA: Mixture-of-Groups Attention for End-to-End Long Video Generation2025· architecture→ similar topic