ArXiv: 2505.21136

🎯 Pitch

SageAttention2++ achieves a 3.9× speedup over FlashAttention by exploiting a GPU instruction that is twice as fast as the one used in prior quantized attention, but doing so without accuracy loss required solving a subtle numerical problem: constraining the quantization ranges of P and V so their matrix product stays within the narrow FP16 accumulator range.

1. Executive Summary

This paper introduces SageAttention2++, a hardware-aware optimization that further accelerates the quantized attention kernel SageAttention2 by exploiting a faster GPU matrix-multiply instruction—specifically, replacing the FP8 Matmul with FP32 accumulator (mma.f32.f8.f8.f32) with one using an FP16 accumulator (mma.f16.f8.f8.f16), which is 2× faster than the instruction used in SageAttention2—while preserving accuracy by narrowing the quantization range of the softmax output P and value V to keep accumulated results within FP16's representable bounds. Evaluated across text generation (Llama3.1-8B), image generation (Flux, Stable-Diffusion3.5), and video generation (CogvideoX, HunyuanVideo, Wan) on RTX4090 and RTX5090 GPUs, SageAttention2++ achieves a 3.9× speedup over FlashAttention2 and matches SageAttention2's end-to-end metrics with negligible loss, establishing that the full speed potential of FP8 tensor-core instructions with reduced-precision accumulation can be realized in attention computation only when the quantization scales for P and V are jointly constrained to satisfy an accumulator range budget (Pr × Vr ≤ 2047, or more tightly Pr × Vr ≤ 1023 under the delayed FP32 buffering optimization).

2. Context and Motivation

The Core Problem: Unlocking the Full Speed of FP8 Matrix Multiplication in Quantized Attention

This paper addresses a specific, hardware-level inefficiency in the SageAttention2 kernel for computing the attention operation in transformer models. To understand the problem, we need to break down what happens inside attention computation and why prior work left performance on the table.

Recall that scaled dot-product attention, the dominant mechanism in modern transformers, involves two computationally heavyweight matrix multiplications (Matmuls):

  1. QKᵀ — the "first Matmul": Multiplying the query matrix QQ against the transposed key matrix KK. The output is an N×NN \times N attention score matrix (for sequence length NN), which grows quadratically with sequence length and thus dominates cost for long sequences.
  2. PV — the "second Matmul": Multiplying the softmax-normalized attention weight matrix PP (derived from QKTQK^T) against the value matrix VV. This produces the output tokens.

SageAttention2 accelerates both of these Matmuls by quantizing them to low-precision data types and then executing them using specialized GPU hardware instructions called Tensor Core operations. For the PV Matmul specifically, SageAttention2 quantizes both PP and VV from the standard FP16 (16-bit floating point) down to FP8 (8-bit floating point, specifically the E4M3 format, which uses 4 exponent bits and 3 mantissa bits). It then dispatches this FP8 data to a Tensor Core instruction called mma.f32.f8.f8.f32. Let's unpack what this instruction does and why it is the crux of the improvement.

An mma (Matrix Multiply-Accumulate) instruction tells the GPU's Tensor Cores to multiply two input matrices (here, both in FP8) and add the result to an accumulator (here, in FP32, or 32-bit single-precision float). Using an FP32 accumulator was a natural choice in SageAttention2—it is the high-precision option that guarantees no numerical overflow issues for the vast majority of attention computations, keeping the accumulated dot product results perfectly safe from clipping or wrapping.

However, the paper identifies a crucial speed detail noted in NVIDIA's own GPU architecture documentation: the choice of accumulator data type dictates the raw throughput of the Tensor Core instruction. As summarized in Table 1 of the paper, on the tested GPUs (RTX4090 and RTX5090, both based on NVIDIA's Ada Lovelace or later architectures), three regimes exist:

  1. Baseline: FP16 Matmul with FP32 accumulator. This is the standard high-precision operation, assigned a relative speedup of 1×.
  2. SageAttention2's choice: FP8 Matmul with FP32 accumulator (mma.f32.f8.f8.f32). Switching inputs to FP8 cuts data movement in half compared to FP16, but the FP32 accumulator limits the speedup to 2× over the baseline. The internal accumulation still operates at full 32-bit width, consuming more energy and taking more cycles per operation.
  3. The untapped opportunity: FP8 Matmul with FP16 accumulator (mma.f16.f8.f8.f16). By further reducing the accumulator precision to FP16 (16-bit floating point, the same type as the standard model weights), the Tensor Core can achieve a 4× speedup over the baseline. This is a full 2× faster than the instruction SageAttention2 uses. The reason is fundamental: smaller accumulators mean more operations can be packed into the same silicon area and run at higher throughput.

The gap is clear and quantified: "However, mma.f32.f8.f8.f32 employs an FP32 accumulator and is only 2× faster than FP16. We find that the mma.f16.f8.f8.f16 instruction (using FP16 accumulator for FP8 Matmul) achieves 4× speedup over FP16." (Section 1). SageAttention2's design choice, while safe, left half the potential speed of the FP8 hardware on the floor.

Why This Problem Is Important: Practical Acceleration for Exact Attention

This problem matters because the speed of exact attention computation directly determines the latency and throughput of deployed transformer models, particularly as sequence lengths grow. While many approaches tackle the O(N2)O(N^2) complexity of attention through approximations—linear attention, sparse attention, or kernelized methods—these come with an often-unacceptable tradeoff: they change the mathematical operation itself, which can hurt model quality across diverse tasks and architectures. As the paper notes in its introduction, these alternative attention mechanisms "often exhibit limited generality across models and tasks" (Section 1).

The alternative lineage, to which this paper belongs, is hardware-optimized exact attention—implementations like FlashAttention, FlashAttention-2, and the SageAttention family that compute the exact scaled dot-product attention function, but do so in a manner ruthlessly optimized for GPU memory hierarchies and compute units. The key appeal of this approach is that it is a drop-in replacement: you swap the attention kernel and the model's mathematical output remains identical (or nearly identical, within controlled quantization error bounds). No retraining, no architecture changes, no task-specific tuning.

The importance of maximizing the speed of this exact attention lineage, then, is twofold:

  1. Broader applicability. Because these kernels preserve exact computation (up to quantization noise), they can accelerate models across any domain—language, image generation, video generation, audio—with no per-task adaptation. The paper demonstrates this explicitly by evaluating on models for text (Llama3.1), image generation (Flux, Stable-Diffusion3.5), and video generation (CogvideoX, HunyuanVideo, Wan). A 3.9× speedup is therefore a 3.9× speedup that applies to all of them simultaneously, a pure infrastructure-level gain.

  2. Latency-sensitive deployments. Many real-world uses of generative models—interactive chatbots, real-time video generation, on-device inference—are latency-bound. Reducing wall-clock time by nearly 4× directly expands the set of feasible deployment scenarios. For video generation models like HunyuanVideo or Wan, where a single forward pass may already take seconds to minutes, a 3.9× attention speedup meaningfully shrinks the generation time.

The problem is therefore precisely scoped: there exists a known, documented GPU instruction that is 2× faster than what SageAttention2 uses for its PV Matmul, but naively switching to it would produce numerical errors because the FP16 accumulator has a much smaller dynamic range than FP32. The challenge is to engineer the quantization scales so that the computation fits within FP16's representable range without degrading model accuracy. This is a classic systems tradeoff: speed versus precision, where the insight is that the precision can be sufficiently preserved through a careful choice of quantization parameters.

Where Prior Approaches Fall Short

To fully appreciate why this paper's contribution is necessary and non-obvious, we must understand what SageAttention2 already had to solve and why its solution was suboptimal for the PV Matmul.

SageAttention2's quantization strategy. SageAttention2 quantizes the attention computation in stages. For the first Matmul (QKᵀ), it uses per-block INT8 or INT4 quantization on QQ and KK (the choice of INT8 vs. INT4 determines the speed-accuracy tradeoff, yielding the two variants 8+8 and 4+8). This part of the pipeline is already reasonably efficient because the quantized types are small.

For the second Matmul (PV), SageAttention2 treats PP (the softmax output, also denoted P~\tilde{P} for unquantized) and VV similarly, but using FP8 with the E4M3 format. The key design decisions were the quantization scales, defined as:

δP=max⁡(∣P~∣)448,δV=colmax(∣V∣)448\delta_P = \frac{\max(|\tilde{P}|)}{448}, \quad \delta_V = \frac{\text{colmax}(|V|)}{448}

The choice of denominator 448 is driven by the E4M3 format's maximum representable value (the largest finite number before infinity is 448). By dividing the maximum absolute value in the block by 448, SageAttention2 ensures that the quantized values P^=⌈P~/δP⌉\hat{P} = \lceil \tilde{P} / \delta_P \rceil and V^=⌈V/δV⌉\hat{V} = \lceil V / \delta_V \rceil are integers within the range [−448,448][-448, 448], which fits exactly into the E4M3 format's representable integer grid. This is the natural, maximum-coverage quantization strategy: use every available quantization level to minimize rounding error.

After quantizing, the matrix multiplication runs as:

PV=(P^V^)×δP×δVPV = (\hat{P} \hat{V}) \times \delta_P \times \delta_V

The intermediate integer Matmul P^V^\hat{P} \hat{V} is executed via the mma.f32.f8.f8.f32 instruction, which accumulates into FP32. The FP32 accumulator has a dynamic range of approximately [−3.4×1038,3.4×1038][-3.4 \times 10^{38}, 3.4 \times 10^{38}], so even with P^\hat{P} and V^\hat{V} both pushed to their maximum value of 448, the accumulated results (sums of 32 products—a detail from the mma.m16n8k32 instruction's k=32k=32 dimension) never threaten overflow.

The gap when switching to the faster instruction. The temptation is to simply swap the MMA instruction from the FP32-accumulator variant to the FP16-accumulator variant, gaining the 4× speedup for free. But the FP16 accumulator has a much narrower dynamic range: its maximum representable finite value is 65,504. (FP16 uses 5 exponent bits and 10 mantissa bits, yielding 216−25=655042^{16} - 2^{5} = 65504 as the largest normal number, with values above this being infinity.)

The problem arises because of how the mma.m16n8k32 instruction works internally: it computes dot products over 32-element inner-product loops within the FP16 accumulator. That means up to 32 terms of the form p×vp \times v (where pp is from P^\hat{P} and vv is from V^\hat{V}) are summed together before the result is converted back to FP32. If pp and vv can each be as large as 448 (the maximum quantized value in SageAttention2), then a single product p×vp \times v could be 448×448=200,704448 \times 448 = 200,704. Summing 32 such terms would reach 6,422,5286,422,528, which vastly exceeds the FP16 maximum of 65,504. The intermediate accumulation would overflow to infinity, producing garbage results.

This is the core technical tension: the 4× faster instruction exists, but SageAttention2's natural quantization strategy—maximizing the use of the FP8 range—makes it numerically incompatible with the FP16 accumulator. The prior approach's shortfall is not an oversight; it is a consequence of designing for safety (FP32 accumulator) rather than speed. The paper identifies this specific instruction-level gap and asks whether the quantization can be re-engineered to close it.

Why this wasn't trivially solved before. The challenge is deeper than just "use smaller quantization ranges." Narrowing the ranges (i.e., using smaller quantization denominators than 448 and 448) would naively seem to increase quantization error—more of the representable range goes unused, and each quantized value carries less precision. The critical question is: how much can we narrow these ranges before the accumulated error degrades the attention output enough to hurt end-to-end model metrics?

Prior work had not explored this question systematically for the FP16-accumulator FP8 MMA instruction in the context of attention. The standard practice was to maximize the quantization range to minimize per-element error, a reasonable heuristic that works for FP32 accumulators but leaves the 2× speedup of FP16 accumulators untapped.

How This Paper Positions Itself

The paper's positioning is clear and precise: it is not proposing a new quantization scheme, a new attention algorithm, or a new model architecture. It is a focused, single-technique improvement over SageAttention2 that unlocks the remaining speed headroom in the hardware by solving a constraint satisfaction problem over quantization scale factors.

The positioning relative to the broader landscape is:

  • Relative to linear/sparse attention methods: The paper explicitly places itself in the third category of "hardware-optimized attention implementations that maintain full sequence computation," alongside FlashAttention and prior SageAttention versions. It promises no change to the attention function itself, only to the speed of its computation. This is its primary value proposition for practitioners: drop-in acceleration.

  • Relative to SageAttention2: The paper presents itself as a direct, incremental speed improvement that preserves SageAttention2's accuracy. The claim is that SageAttention2++ achieves identical end-to-end metrics (perplexity, CLIP score, FID, etc.) to SageAttention2 while being substantially faster, by exploiting the faster FP8 MMA instruction. The evidence for this is in Table 2 (showing that attention output metrics like cossim and L1 distance are essentially identical across different (Pr,Vr)(P_r, V_r) configurations) and Table 3 (showing that end-to-end model metrics are within expected noise of SageAttention2's results).

  • Relative to FlashAttention2: The paper uses FlashAttention2 as the universal speed baseline, reporting all speedup numbers relative to it. The choice is pragmatic—FlashAttention2 is the standard optimized exact-attention kernel for the target GPUs (RTX4090 and RTX5090 belong to the Ada Lovelace and Blackwell architectures respectively; the authors note that FlashAttention3 requires Hopper GPUs, so FlashAttention2 is the fastest available baseline on these cards). By reporting a 3.9× speedup over FlashAttention2, the paper establishes its practical relevance.

The paper's contribution can be understood entirely through the lens of one design equation and its constraints. The unsafe condition for the FP16 accumulator is:

∣32×p×v∣≤65504|32 \times p \times v| \leq 65504

where pp and vv are drawn from the quantized P^\hat{P} and V^\hat{V}, bounded by their respective quantization ranges PrP_r and VrV_r. This yields the core constraint:

Pr×Vr≤6550432=2047P_r \times V_r \leq \frac{65504}{32} = 2047

The original SageAttention2 implicitly uses Pr=448P_r = 448 and Vr=448V_r = 448, whose product is 200,704200,704, violating this constraint by a factor of nearly 100×. SageAttention2++'s entire technical novelty is the recognition that PrP_r and VrV_r can be systematically reduced to satisfy Pr×Vr≤2047P_r \times V_r \leq 2047, and that the accuracy penalty from this range reduction is negligible—because the softmax matrix PP and value matrix VV in practice rarely exercise the full dynamic range of E4M3.

This framing places the paper as a precision-throughput co-design contribution: it trades off quantization range (precision) for accumulator throughput (speed) at a finer granularity than prior work recognized was possible, and it provides the concrete bounds that make the tradeoff safe. The paper does not aim to be broadly transformative—it is a surgical performance optimization—but one that applies universally to any transformer model using quantized attention.

3. Technical Approach

3.1 Reader Orientation (Approachable Technical Breakdown)

SageAttention2++ is a GPU kernel—a hand-optimized CUDA program—that computes the standard scaled dot-product attention operation for transformer models using mixed-precision integer and floating-point arithmetic on NVIDIA Tensor Cores, producing bit-identical or near-bit-identical results to the full-precision computation while running substantially faster. The problem it solves is that the prior state-of-the-art quantized attention kernel, SageAttention2, used a safe but slow matrix-multiply instruction for the PV multiplication that left half the potential throughput of the underlying FP8 hardware unused; SageAttention2++ recovers that lost speed by switching to a faster instruction and then surgically constraining the quantization ranges of the softmax output $P$ and the value matrix $V$ so that the intermediate accumulation never overflows the narrower numeric range of the faster instruction's accumulator, all while introducing no measurable degradation to the final attention output or to end-to-end model quality.

3.2 Big-Picture Architecture (Diagram in Words)

The SageAttention2++ system is a direct modification of the SageAttention2 kernel's PV computation path. The overall attention pipeline remains identical to SageAttention2 except for one specific stage. The architecture consists of four major stages, inherited from SageAttention2 with only the fourth stage modified:

  1. QKᵀ quantization and Matmul (unchanged from SageAttention2). The query matrix $Q$ and key matrix $K$ are quantized to INT8 or INT4 with per-block granularity. The quantized matrices are multiplied using INT8 or INT4 Tensor Core operations to produce the raw attention score matrix. This stage uses separate scale factors $\delta_Q$ and $\delta_K$ computed as $\max(|Q|)/127$ and $\max(|K|)/127$ respectively.

  2. Online softmax with smoothing (unchanged from SageAttention2). The raw scores pass through an online softmax that incorporates SageAttention2's previously-developed smoothing technique for $Q$ and $K$. The output is the attention weight matrix $\tilde{P}$, stored in FP16 precision at this point. (The tilde denotes the unquantized, full-precision version before the quantization that follows.)

  3. Value matrix quantization (modified from SageAttention2). The value matrix $V$ is quantized to FP8 in E4M3 format, but using a narrower per-channel quantization range than SageAttention2 used. Specifically, instead of scaling by $\max(|V|)/448$ to fill the full E4M3 representable range, SageAttention2++ scales by $\max(|V|)/V_r$ where $V_r$ is a reduced range parameter chosen jointly with the $P$ quantization parameter to satisfy the FP16 accumulator constraint. Each channel (column) of $V$ receives its own scale factor.

  4. PV Matmul with FP16-accumulator FP8 MMA and delayed FP32 buffering (the core modification). The attention weight matrix $\tilde{P}$ is quantized to FP8 using a narrowed range $P_r$ (instead of SageAttention2's 448). The quantized $\hat{P}$ and $\hat{V}$ are multiplied using the mma.f16.f8.f8.f16 instruction, which accumulates into FP16. Two consecutive MMA results are accumulated in FP16 before a single conversion to FP32, halving the conversion overhead. The final output is scaled by $\delta_P \times \delta_V$ to restore the correct magnitude.

Information flows sequentially: $Q, K$ enter → INT4/8 quantized → $QK^T$ Matmul → online softmax → $\tilde{P}$ produced → $\tilde{P}$ and $V$ quantized with narrowed ranges → $PV$ Matmul via mma.f16.f8.f8.f16 with delayed FP32 buffering → final attention output $O$.

3.3 Roadmap for the Deep Dive

  • First, the quantized PV Matmul formulation from SageAttention2—how $P$ and $V$ are quantized, what the scale factors are, and how the dequantized output is recovered—because this is the foundation that SageAttention2++ modifies and the reader must understand the baseline before seeing what changed.
  • Second, the hardware constraint that motivates the modification: the mma.m16n8k32 instruction's internal accumulation structure, the representable range of the FP16 accumulator, and the precise inequality that must be satisfied to avoid overflow—because this inequality ($P_r \times V_r \leq 2047$) is the single design equation that governs all subsequent parameter choices.
  • Third, the narrowed quantization range strategy—how the scale factors $\delta_P$ and $\delta_V$ are redefined to enforce the constraint, the joint degree of freedom in choosing $(P_r, V_r)$ pairs, and the empirical validation in Table 2 showing that multiple $(P_r, V_r)$ configurations produce indistinguishable attention accuracy—because this demonstrates that the speed-accuracy tradeoff collapses to a pure speed gain.
  • Fourth, the delayed FP32 buffering optimization—why data type conversions between FP16 and FP32 incur overhead, how accumulating two MMA results before converting reduces this overhead, and how this tightens the constraint to $P_r \times V_r \leq 1023.5$—because this explains the chosen final parameters $P_r = 224, V_r = 4.5$.
  • Fifth, the design rationale for the final parameter choice $P_r = 224, V_r = 4.5$—how these values satisfy both the overflow constraint and the delayed buffering constraint, and why they represent "optimal performance" within the feasible region.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a hardware-systems optimization paper whose core idea is that the throughput ceiling of quantized attention on NVIDIA GPUs is set not by the input data types (FP8) but by the accumulator data type of the Tensor Core MMA instruction, and that by switching from an FP32-accumulator MMA to an FP16-accumulator MMA—and jointly constraining the quantization ranges of $P$ and $V$ to keep intermediate accumulation within FP16's representable bounds—the kernel achieves a 2× speedup on the $PV$ Matmul relative to SageAttention2 with zero measurable accuracy degradation.

SageAttention2's Baseline PV Quantization (The Starting Point)

The foundation SageAttention2++ builds upon is SageAttention2's quantization scheme for the second attention Matmul. I will detail this baseline precisely so that the modification becomes mechanically clear.

In the FlashAttention tiling framework, the attention computation operates on blocks (tiles) of the full $Q, K, V$ matrices to exploit the GPU's memory hierarchy. Rather than materializing the entire $N \times N$ attention matrix at once, FlashAttention loads small sub-blocks $\{Q_i\}, \{K_i\}, \{V_i\}$ into fast on-chip shared memory, computes a partial softmax over the relevant portion of the score matrix, and incrementally updates the output. This same tiling strategy underlies both SageAttention2 and SageAttention2++, but for explaining the quantization, the authors simplify by dropping the block subscripts and discussing $Q, K, \tilde{P}, V$ as the current in-flight tile.

After the online softmax produces the attention weight matrix tile $\tilde{P}$ (where the tilde explicitly marks this as the unquantized, full-precision version), SageAttention2 quantizes both $\tilde{P}$ and $V$ to the FP8 E4M3 format. The E4M3 format uses 4 exponent bits and 3 mantissa bits, with a maximum finite representable value of 448 and a minimum subnormal value that is not relevant for this analysis (only the maximum matters for overflow).

The quantization is linear and symmetric around zero, with separate scale factors computed per-block for $\tilde{P}$ (since every element within a tile shares the same attention head) and per-channel for $V$ (since different channels may have different magnitude distributions). For SageAttention2, the scale factors are:

δP=max⁡(∣P~∣)448\delta_P = \frac{\max(|\tilde{P}|)}{448}

δV=colmax(∣V∣)448\delta_V = \frac{\text{colmax}(|V|)}{448}

where $\max(|\tilde{P}|)$ is the maximum absolute value in the current $\tilde{P}$ tile, and $\text{colmax}(|V|)$ is the maximum absolute value in each column (channel) of the current $V$ tile, producing a vector of per-channel scale factors.

The quantized integer matrices $\hat{P}$ and $\hat{V}$ are computed by dividing by the scale factor and rounding to the nearest integer (the ceiling notation $\lceil \cdot \rceil$ in the paper is used loosely to indicate rounding-to-integer; the actual operation is a round-to-nearest-ties-to-even, as is standard in quantization):

P^=⌈P~/δP⌉,V^=⌈V/δV⌉\hat{P} = \lceil \tilde{P} / \delta_P \rceil, \quad \hat{V} = \lceil V / \delta_V \rceil

The range of the quantized values is bounded by the denominator 448: since $|\tilde{P}| / \delta_P = |\tilde{P}| / (\max(|\tilde{P}|)/448) = 448 \times |\tilde{P}| / \max(|\tilde{P}|) \leq 448$, the quantized value $\hat{P}$ is an integer in $[-448, 448]$. The same logic applies channel-wise to $\hat{V}$, yielding per-channel values also in $[-448, 448]$. This is the maximum-coverage quantization strategy: by dividing by the largest value that the E4M3 format can represent, every available quantization level is potentially used, which minimises the rounding error for a given block of data. The intuition is that you want to "stretch" the data to fill the representable range, so that quantization bins are as fine-grained as possible.

Once quantized, the matrix multiplication is performed in the integer domain using a Tensor Core MMA instruction, and the result is dequantized back to floating point by multiplying by the scale factors:

PV=(P^V^)×δP×δVPV = (\hat{P} \hat{V}) \times \delta_P \times \delta_V

This three-term product—integer Matmul, then FP32 multiply by $\delta_P$, then FP32 multiply by $\delta_V$—is mathematically equivalent to the original FP16 computation, modulo the quantization error introduced by rounding $\tilde{P}$ and $V$ to integers. The MMA instruction that computes $\hat{P} \hat{V}$ is mma.f32.f8.f8.f32, which takes the two FP8 inputs (the FP8 values $\hat{P}$ and $\hat{V}$ are interpreted as floating-point E4M3 values equal to the integers, since the integer $\hat{P}$ in $[-448, 448]$ exactly corresponds to the E4M3 encoding of that integer) and accumulates the result in FP32. The FP32 accumulator has over 7 orders of magnitude more headroom than the worst-case accumulated value, so overflow is impossible regardless of the $P_r$ and $V_r$ choices.

This is the SageAttention2 baseline that SageAttention2++ modifies. The critical hinge point is the denominator 448 in both scale factors, which determines how large the quantized integers can be. If we change the denominators, we change the quantization error. If we change the MMA instruction, we change the accumulator precision and thus the overflow risk for large quantized values. SageAttention2++ changes both denominators and the instruction simultaneously.

The Hardware Constraint: FP16 Accumulator Overflow and the Core Inequality

The faster Tensor Core instruction that SageAttention2++ wants to use is designated mma.f16.f8.f8.f16. This instruction still takes two FP8 input matrices (the same data types as before) but accumulates the dot products into an FP16 register instead of an FP32 register. The throughput advantage is documented in Table 1: FP8 Matmul with FP16 accumulator is 4× faster than the FP16 baseline, which is 2× faster than the FP8-with-FP32-accumulator instruction that SageAttention2 uses. The hardware can achieve this because narrower accumulators require fewer physical wires and less energy per operation, allowing more parallel multiply-accumulate units to operate simultaneously within the same power and area budget.

The specific MMA instruction variant used for the $P \times V$ Matmul is mma.m16n8k32 (NVIDIA, 2025, as cited in Section 3.1). The signature m16n8k32 describes the tile geometry: it multiplies a $16 \times 32$ slice of $\hat{P}$ (the first matrix, "A") against a $32 \times 8$ slice of $\hat{V}$ (the second matrix, "B"), producing a $16 \times 8$ output tile. The $k=32$ dimension is the inner product or reduction dimension: each output element is the sum of 32 terms, where each term is the product of one element from a row of $\hat{P}$ and one element from the corresponding column of $\hat{V}$.

The FP16 accumulator's maximum representable finite (normal) value is 65,504. (This comes from the FP16 format: 5 exponent bits give a maximum exponent of 15 after bias, $2^{15} = 32768$, multiplied by the maximum mantissa $1 + 1023/1024 \approx 1.999$, yielding $2^{16} - 2^{5} = 65504$ as the largest exact integer representable without round-to-nearest. Values exceeding this encode as +Inf in the FP16 format.)

The overflow condition is therefore: if any of the 32 product terms within a single $k=32$ reduction sum, when added together, exceeds 65,504 in magnitude, the accumulator will saturate to infinity, and the entire output element becomes invalid. The worst-case scenario occurs when all 32 product terms are maximal in the same sign direction. If the quantized $\hat{P}$ values are bounded by $P_r$ (i.e., $|\hat{P}| \leq P_r$) and the quantized $\hat{V}$ values are bounded by $V_r$ (i.e., $|\hat{V}| \leq V_r$), then each term $p \times v$ is bounded by $P_r \times V_r$, and the sum of 32 such terms is bounded by $32 \times P_r \times V_r$.

This yields the core inequality that governs the entire design space of SageAttention2++:

∣32×p×v∣≤65504for all possible p,v with ∣p∣≤Pr,∣v∣≤Vr|32 \times p \times v| \leq 65504 \quad \text{for all possible } p, v \text{ with } |p| \leq P_r, |v| \leq V_r

which simplifies to the design constraint on the quantization range parameters:

Pr×Vr≤6550432=2047P_r \times V_r \leq \frac{65504}{32} = 2047

What it computes: This is a product constraint on the quantization range limits $P_r$ and $V_r$. Given the hardware's FP16 accumulator maximum of 65,504 and the $k=32$ reduction dimension of the MMA instruction, the condition ensures that the sum of 32 maximally-large product terms never exceeds the accumulator's capacity. The inequality defines a hyperbola in $(P_r, V_r)$ space: any pair of range parameters whose product is at most 2047 is guaranteed safe from FP16 overflow during the MMA accumulation.

Why this form: The constraint is a hard physical limit of the hardware, not a tunable tradeoff. If violated, the MMA produces +Inf or -Inf output values, which propagate through the subsequent dequantization multiply and the rest of the model, causing catastrophic failure. There is no graceful degradation; it is a binary safe/unsafe boundary. The product form $P_r \times V_r$ rather than separate bounds on $P_r$ and $V_r$ is a consequence of the fact that overflow depends on the product of a $P$ element and a $V$ element, not on either individually—a row-column interaction in the matrix multiply.

The original SageAttention2 implicitly uses $P_r = 448$ and $V_r = 448$, since the denominators in its scale factors are 448. The product $448 \times 448 = 200,704$, which exceeds the safe bound of 2,047 by a factor of 98×. This is why SageAttention2 must use the FP32-accumulator MMA: its quantized values are far too large to accumulate in FP16. The key engineering insight of SageAttention2++ is that the quantization ranges do not need to be this large because the softmax matrix $\tilde{P}$ and the value matrix $V$ in practice do not contain values that would suffer excessive rounding error from the narrower quantization—in other words, the lowest-order bits that are sacrificed by reducing $P_r$ and $V_r$ carry noise rather than signal for attention computation.

Narrowed Quantization Range: Redefining the Scale Factors

With the constraint $P_r \times V_r \leq 2047$ established, SageAttention2++ redefines the scale factors for the $PV$ quantization to ensure the quantized values never exceed these narrowed bounds. The new scale factors are:

δP=∣max⁡(P~)∣Pr,δV=∣max⁡(V)∣Vr\delta_P = \frac{|\max(\tilde{P})|}{P_r}, \quad \delta_V = \frac{|\max(V)|}{V_r}

where $P_r$ and $V_r$ are now design parameters chosen to satisfy $P_r \times V_r \leq 2047$, and where $|\max(V)|$ is still computed per-channel as in SageAttention2.

What it computes: The same linear symmetric quantization as before—$\tilde{P}$ divided by $\delta_P$ and rounded to integer, $V$ divided by $\delta_V$ and rounded to integer—but with denominators $P_r$ and $V_r$ that are smaller than 448. The quantized values $\hat{P}$ are now bounded by $P_r$ rather than 448, and each channel of $\hat{V}$ is bounded by $V_r$ rather than 448. The dequantized recovery remains $PV = (\hat{P} \hat{V}) \times \delta_P \times \delta_V$, meaning the final result is mathematically identical to the FP16 computation within the quantization error.

Why this form: The key insight is that $P_r$ and $V_r$ appear in the denominator of the scale factors and in the numerator of the quantized value bounds, cancelling out in the dequantized product. Reducing $P_r$ (making the denominator smaller) increases $\delta_P$ (making the scale factor larger), which makes the quantized integers $\hat{P}$ smaller (since $\hat{P} = \tilde{P} / \delta_P$ with a larger denominator). The dequantized product $\hat{P} \hat{V} \times \delta_P \times \delta_V$ recovers $\tilde{P} V$ up to rounding error, regardless of the specific $(P_r, V_r)$ values chosen. This means $(P_r, V_r)$ is a free parameter for the quantization error and the overflow constraint, but does not change the expected value of the output—only the variance around it. The quantization error increases as $P_r$ and $V_r$ decrease, because each integer step represents a larger floating-point increment, but this error can be made arbitrarily small by not making $P_r$ and $V_r$ unnecessarily small.

The paper explores this free parameter space explicitly in Table 2, which shows the attention accuracy metrics (cossim and L1 distance) for four different $(P_r, V_r)$ configurations on CogvideoX:

  • SageAttention2 baseline: $P_r = 448, V_r = 448$ (product $200,704$, safe only with FP32 accumulation). Cossim = 99.97%, L1 = 0.01862.
  • SageAttention2++ candidate 1: $P_r = 448, V_r = 2.25$ (product $1008$, satisfies $\leq 2047$). Cossim = 99.97%, L1 = 0.01863.
  • SageAttention2++ candidate 2: $P_r = 224, V_r = 4.5$ (product $1008$, satisfies $\leq 2047$). Cossim = 99.97%, L1 = 0.01862.
  • SageAttention2++ candidate 3: $P_r = 112, V_r = 9$ (product $1008$, satisfies $\leq 2047$). Cossim = 99.97%, L1 = 0.01863.

The results are striking: across a 4× range of $P_r$ values (from 448 down to 112), the cosine similarity to the full-precision attention output is 99.97% for every configuration, and the relative L1 distance varies only in the fifth decimal place (0.01862–0.01863). This is the paper's central empirical finding: the quantization error is essentially invariant to the $(P_r, V_r)$ split as long as the product is sufficiently large to capture the meaningful dynamic range of $\tilde{P}$ and $V$. The individual values in $\tilde{P}$ and $V$ are not large enough to need the full 448 range in either matrix, meaning the quantization noise floor is set by something other than range clipping—likely by the granularity of the rounding operation itself, which is determined by the number of quantization levels used.

The design space is therefore not a tradeoff at all in the regime explored: any $(P_r, V_r)$ pair satisfying $P_r \times V_r \leq 2047$ and maintaining $P_r, V_r$ above some floor (empirically, $P_r \geq 112$ and $V_r \geq 2.25$ in the tested configurations) delivers essentially identical accuracy.

Delayed FP32 Buffering: Halving the Data Type Conversion Overhead

After the mma.f16.f8.f8.f16 instruction produces an FP16-accumulated result, that result must eventually be stored to memory or used in subsequent FP32 operations (because the rest of the attention computation—softmax rescaling, output accumulation, residual connections—typically operates in FP32 for stability). Converting data types between FP16 and FP32 on NVIDIA GPUs requires explicit PTX instructions (Parallel Thread Execution, the low-level assembly-like intermediate representation for CUDA). These conversion instructions consume execution slots and add latency to the critical path.

The paper identifies that the conversion overhead can be reduced by deferring the FP32 conversion and instead accumulating two consecutive mma.m16n8k32 results in FP16 before performing a single FP32 conversion. This is the delayed FP32 buffering optimization (Section 3.2).

The mechanism works as follows: the MMA instruction mma.m16n8k32 computes one $16 \times 8$ tile of $\hat{P} \hat{V}$, where each element is the sum of 32 products. Normally, each such tile would be converted from FP16 to FP32 immediately. Instead, SageAttention2++ performs a second mma.m16n8k32 operation over the next $k=32$ elements of the reduction dimension, adds these 32 new products to the already-accumulated 32 products within the same FP16 register, and only then converts to FP32. This means two complete $k=32$ reductions—a total of 64 product terms—are accumulated in FP16 before a single FP16-to-FP32 PTX conversion is issued, reducing the number of conversions by a factor of 2.

The cost of this optimization is a tightening of the overflow constraint. Now the maximum number of terms that could be summed in FP16 before conversion doubles from 32 to 64:

∣64×p×v∣≤65504|64 \times p \times v| \leq 65504

which yields the tighter product constraint:

Pr×Vr≤6550464=20472=1023.5P_r \times V_r \leq \frac{65504}{64} = \frac{2047}{2} = 1023.5

What it computes: A refined bound on the joint quantization range that accounts for the accumulation of two MMA results in FP16 before conversion. The constraint tightens by exactly a factor of 2—from the requirement that 32 excess terms fit in the accumulator to the requirement that 64 excess terms fit—because the buffering accumulates over two $k=32$ inner-product loops.

Why this form: The factor of 2 tightening is a direct consequence of the design choice to buffer two MMA results. If more buffering steps were desired (say, 4 accumulations), the constraint would tighten linearly. The choice of exactly 2 is an empirical engineering tradeoff: it halves the conversion overhead without excessively constraining the $(P_r, V_r)$ search space. Buffering 4 would tighten the constraint to $P_r \times V_r \leq 511.75$, which the authors presumably found would require reducing the ranges to the point where quantization error became measurable (though the paper does not report experiments with deeper buffering depths).

Final Parameter Selection: $P_r = 224, V_r = 4.5$

With the tightened constraint $P_r \times V_r \leq 1023.5$, the paper must select a specific $(P_r, V_r)$ pair from the feasible region. Table 2 evaluates three candidates that all satisfy this stricter bound:

  • $P_r = 448, V_r = 2.25$: Product = $1008$, satisfies $\leq 1023.5$. This keeps $P_r$ at its maximum while reducing $V_r$ dramatically.
  • $P_r = 224, V_r = 4.5$: Product = $1008$, satisfies $\leq 1023.5$. This is a balanced reduction of both ranges.
  • $P_r = 112, V_r = 9$: Product = $1008$, satisfies $\leq 1023.5$. This pushes $P_r$ lower and allows a larger $V_r$.

All three achieve essentially identical attention accuracy: cossim of 99.97% and L1 distance varying only in the fifth decimal place. The paper selects $P_r = 224, V_r = 4.5$ as the final parameters, stating this choice is "for optimal performance" (Section 3.2).

The rationale for this specific choice is not elaborately defended in the text, but the logic can be inferred from the quantization granularity and the properties of $P$ and $V$. The softmax matrix $\tilde{P}$ contains values in $[0, 1]$ (it is a probability distribution over keys), so its dynamic range is inherently bounded. The value matrix $V$, however, consists of learned embeddings that can have substantially larger magnitudes per channel. Reducing $V_r$ from 448 to 4.5—a factor of nearly 100×—would be a dramatic reduction in quantization resolution for $V$ alone. By instead sharing the range reduction more evenly ($P_r$ reduced by 2×, $V_r$ reduced by ~100×), the quantization error is distributed across both matrices in a way that the empirical results show is neutral. The specific choice $P_r = 224$ and $V_r = 4.5$ may also reflect hardware implementation convenience—these values produce quantized integers that fit cleanly into the E4M3 format without requiring special-case handling of edge bins.

With these parameters, the scale factors become:

δP=max⁡(∣P~∣)224,δV=colmax(∣V∣)4.5\delta_P = \frac{\max(|\tilde{P}|)}{224}, \quad \delta_V = \frac{\text{colmax}(|V|)}{4.5}

The quantized matrices $\hat{P}$ and $\hat{V}$ are then computed by rounding $\tilde{P} / \delta_P$ and $V / \delta_V$ respectively, and the $PV$ Matmul is executed via two mma.f16.f8.f8.f16 operations accumulated in FP16 followed by a single FP32 conversion and dequantization multiply. The net result is a $PV$ computation that runs at the full 4× throughput of the FP16-accumulator FP8 MMA instruction while producing output that is indistinguishable from SageAttention2's FP32-accumulator version to within measurement precision across all evaluated models and metrics.

Summary of Design Choices and Their Justifications

  • Switch from mma.f32.f8.f8.f32 to mma.f16.f8.f8.f16 for $PV$: Exploits the 2× throughput advantage of FP16 accumulators over FP32 accumulators documented in NVIDIA's hardware specifications, recovering the full 4× speedup that the FP8 input data type theoretically enables over FP16.
  • Joint constraint $P_r \times V_r \leq 2047$ (or $1023.5$ with delayed buffering): Enforces the physical overflow bound of the FP16 accumulator given the $k=32$ reduction dimension of mma.m16n8k32, converting a hardware limitation into an explicit design inequality.
  • Freedom in $(P_r, V_r)$ split: Chosen based on empirical attention accuracy measurements showing that multiple feasible pairs produce identical cossim and L1 error, demonstrating that the quantization error floor is not dominated by range clipping in this regime.
  • Delayed FP32 buffering of two MMA results: Halves the PTX conversion overhead from FP16 to FP32, a practical optimization that reduces instruction-level latency on the critical path while only tightening the already-satisfied constraint by a factor of 2.
  • Final choice $P_r = 224, V_r = 4.5$: Lies within the feasible region ($224 \times 4.5 = 1008 \leq 1023.5$), achieves identical accuracy to the SageAttention2 baseline (Table 2), and presumably balances the quantization granularity between $P$ and $V$ in a way that is tolerant to the specific magnitude distributions observed in real attention computations across diverse models.
  • $QK^T$ path left unchanged: The first Matmul uses INT4/INT8 quantization with a fundamentally different accumulator structure (integer accumulation does not have the same overflow profile as FP16 accumulation, and the INT MMA instructions have their own separate throughput characteristics), so the FP16-accumulator optimization applies only to the second Matmul.

4. Key Insights and Innovations

Innovation 1: Reframing Quantized Attention's Throughput Bottleneck from Input Precision to Accumulator Precision

The dominant intuition in low-precision attention work—including SageAttention2, prior INT8/FP8 attention methods, and even broader mixed-precision ML literature—is that the primary knob controlling speed is the bit-width of the data inputs to matrix multiplications. Make the inputs smaller (FP16 → FP8, FP8 → INT4), move less data, compute more per cycle: this is the standard scaling story. SageAttention2++ reveals that this framing is incomplete for modern GPU Tensor Cores on the PV Matmul. The bottleneck was never the inputs; it was the accumulator.

The conceptual move is subtle but significant. NVIDIA's hardware documentation (NVIDIA, 2022, cited in Table 1) has always reported that mma.f16.f8.f8.f16 (FP8 inputs, FP16 accumulator) offers 4× throughput over the FP16 baseline, while mma.f32.f8.f8.f32 (FP8 inputs, FP32 accumulator) offers only 2×. This is publicly documented electrical engineering, not a research discovery. What makes this an innovation is the paper's recognition that this hardware fact implies a qualitatively different optimization strategy for attention: you don't primarily chase smaller input types (FP8 is already small, and going to FP4 for P and V would likely introduce unacceptable error on non-trivial models), you instead chase a narrower accumulator. The 2× gap between the two FP8 MMA variants is not a curiosity; it is the single largest remaining inefficiency in SageAttention2's PV path.

Prior work in the SageAttention lineage implicitly treated the accumulator as a fixed correctness requirement—FP32 was chosen because it is safe, not because anyone asked whether safety was overkill. The field's default assumption was that FP16 accumulation for attention Matmuls would be numerically dangerous because attention involves sums over softmax-normalized weights, which can concentrate probability mass and produce large intermediate products when multiplied by value vectors. SageAttention2++ shows that this assumption is incorrect in practice for the second Matmul (though it remains untested for the first). The safety margin provided by FP32 accumulation was never actually needed for PV; the FP16 accumulator's range of [-65,504, 65,504] is sufficient when the quantization ranges of P and V are jointly constrained. The paper is essentially arguing that SageAttention2 was 2× slower than necessary on the PV Matmul because it paid a precision tax that the data didn't require.

This is a diagnostic reframing rather than a new algorithm. It changes how a kernel engineer should think about the design space: the question is not "how small can I make my data types" but "what is the cheapest accumulator that can safely contain my intermediate results, and what constraints must I impose on my quantization to make that safe?" This framing generalizes beyond the specific FP8/FP16 combination to future hardware with FP8 accumulators or INT16 accumulators—each new accumulator precision level imposes its own constraint equation on the quantization ranges, and the design problem becomes solving that constraint satisfaction problem.

Innovation 2: Product-Constraint Quantization as a Joint Design Space

The standard approach to quantization in attention—and in neural network quantization more broadly—is per-matrix or per-channel granularity: each matrix gets its own quantization range, optimized independently to minimize per-matrix reconstruction error. SageAttention2 followed this paradigm: P was quantized with range 448, V was quantized with range 448, each chosen to fill the E4M3 format's full representable capacity. The two quantization decisions were independent.

SageAttention2++ introduces a fundamentally different coupling: the quantization ranges $P_r$ and $V_r$ are not independent free parameters to be maximized individually, but joint variables constrained by a single product inequality $P_r \times V_r \leq 1023.5$ (when delayed FP32 buffering is applied). This is not a cosmetic difference. The product constraint arises from the hardware: the accumulator sees products $p \times v$, not $p$ and $v$ separately, so the overflow risk depends on the worst-case product magnitude, which is the product of the individual bounds. This creates a hyperbolic feasible region: you can let $P_r$ be large only if $V_r$ is correspondingly small, and vice versa.

The intellectual contribution here is the recognition that this product constraint defines a joint design space with a degree of freedom that is free in practice. Table 2 is the key evidence: across multiple $(P_r, V_r)$ pairs with the same product ($448 \times 2.25 = 1008$, $224 \times 4.5 = 1008$, $112 \times 9 = 1008$), the attention accuracy metrics (cossim, L1) are identical to within measurement noise. This means the particular split between $P_r$ and $V_r$ does not matter—only the product matters for accuracy—and the product itself can be reduced by roughly 200× from the SageAttention2 baseline ($448 \times 448 = 200,704$) to $1008$ without measurable degradation. This is a strong negative result: the quantization error is not sensitive to the individual range limits in the regime tested, implying that the softmax matrix P and the value matrix V in real attention computations do not contain values that exercise the full 448-range of FP8.

Prior work treated quantization range maximization as an unalloyed good: if you can use more bins, you should, because it minimizes rounding error. SageAttention2++ demonstrates empirically that this maximization is unnecessary beyond a modest floor, and that the degree of freedom thus liberated can be traded for a different resource entirely—in this case, accumulator throughput. This is a co-design insight that would not have emerged from independent per-matrix optimization: the coupling is forced by the hardware's accumulator structure, and the joint optimization reveals slack that single-matrix optimization would preserve.

The significance extends beyond this specific kernel. The product-constraint framing is portable to any setting where two quantized matrices are multiplied and the hardware's accumulation precision imposes a bound on intermediate sums. It provides a template for future hardware-software co-design: when a faster-but-narrower accumulator instruction becomes available, the knobs to turn are not just the data types of the inputs but the quantization ranges of both operands considered jointly under the new constraint equation.

Innovation 3: Delayed FP32 Buffering as a Throughput-Optimized Conversion Strategy

The third innovation is a practical but non-obvious micro-optimization that reveals a general principle about data type conversion overhead in mixed-precision GPU kernels. The mma.f16.f8.f8.f16 instruction produces results in FP16 registers, but the rest of the attention pipeline and the host model typically operate in FP32. Converting each MMA output individually from FP16 to FP32 would issue a PTX conversion instruction per output tile, adding latency to a critical path that is already dominated by compute rather than memory access.

The paper's solution—accumulating two consecutive mma.m16n8k32 results in FP16 before a single FP32 conversion—is not mechanically complex, but its significance lies in the design principle it embodies: when an operation's throughput is limited by data type conversion overhead rather than raw arithmetic, the optimization strategy should be to amortize conversions over larger chunks of work, even at the cost of constraining the arithmetic's dynamic range. The cost is the factor-of-2 tightening of the overflow constraint (from $P_r \times V_r \leq 2047$ to $P_r \times V_r \leq 1023.5$), which the paper shows is easily satisfied. This is a pure engineering win: no accuracy cost, measurable latency reduction, and a generalizable technique for any kernel architecture where accumulator-to-output conversions sit on the critical path.

The deeper conceptual point is that this optimization creates a tradeoff axis that prior work had not parameterized: conversion frequency versus accumulator headroom. The number of buffered accumulations (2 in this paper) is a design choice that could be tuned per-hardware-generation or per-model. On GPUs where conversion instructions are relatively expensive, deeper buffering (3 or 4 accumulations) might be justified even if it forces tighter quantization constraints. On future hardware where conversions are cheaper or accumulators are wider, shallower buffering might be optimal. The paper establishes the parametric framework for this tuning without exhaustively exploring the space—enough to demonstrate the principle and justify the choice of 2 for the target GPUs.

This innovation is incremental in mechanism but fundamental in its implication for kernel design: it shows that the accumulator precision constraint is not an immutable hardware ceiling but a tunable parameter that interacts with software-level buffering decisions. The product constraint $P_r \times V_r \leq 65504 / (32 \times \text{buffering\_depth})$ gives future kernel authors a direct formula for trading off conversion overhead against quantization range.

Innovation 4: Empirical Collapse of the Speed-Accuracy Tradeoff for the PV Matmul

The most consequential empirical finding in the paper is not the speedup itself but the demonstration that the speed-accuracy tradeoff—which is usually the central tension in any quantization or approximate-computing method—collapses for the PV Matmul in the regime of feasible $(P_r, V_r)$ pairs. Across all evaluated models (Llama3.1, CogvideoX, HunyuanVideo, Wan, Flux, Stable-Diffusion3.5) and all end-to-end metrics (perplexity, accuracy, CLIP scores, FID, FScore, etc.), SageAttention2++ with the narrowed quantization ranges achieves results that are indistinguishable from both the full-precision attention baseline and from SageAttention2 (Table 3). In several cases, the quantized version even records marginally better metrics than full-precision (e.g., Flux FID of 163.185 for SageAttn2 (8+8) vs. 165.117 for full-precision, and SageAttn2++ (8+8) at 163.555). These fluctuations are within statistical noise, but the important point is the absence of any systematic degradation.

This is not a trivial or expected result. The standard quantization narrative says: lower precision → more quantization error → some measurable degradation in output quality, which may or may not be acceptable depending on the application. For the SageAttention family, the $QK^T$ quantization (INT4 vs. INT8) does show a measurable tradeoff: the 4+8 variant occasionally produces slightly worse metrics than the 8+8 variant (e.g., CogvideoX FScore drops from 4.899 for 8+8 to 4.386 for 4+8, and the degradation carries over to SageAttention2++). But the PV quantization change—switching from FP32 accumulator to FP16 accumulator with narrowed ranges—introduces zero additional degradation on top of whatever $QK^T$ degradation already exists. Compare any SageAttn2 (8+8) entry to its SageAttn2++ (8+8) counterpart in Table 3: the numbers are essentially clones.

This finding has a theoretical implication: the dynamic range of the PV product in real attention computations, when the softmax matrix P and value matrix V come from trained transformer models, is intrinsically far smaller than the hardware's representable range for FP8 inputs with FP16 accumulation. The worst-case analysis that compelled SageAttention2 to use FP32 accumulation—the $448 \times 448$ product potentially overflowing FP16—is not just conservative; it is addressing a scenario that statistically almost never occurs in practice across language, image, and video generation models. The softmax output P distributes probability mass in a way that limits the effective magnitude of PV products, and the value matrix V's per-channel magnitudes interact with this distribution in a way that keeps the inner products bounded far below the FP16 limit of 65,504 even over 64 accumulated terms.

This is a negative result with positive practical consequences: the safety margin that SageAttention2 preserved was unnecessary for the PV Matmul, and abandoning it yields a 2× speedup on that Matmul (contributing to the overall 3.9× kernel speedup) with no measurable cost. For the field, this means that future attention quantization research can be less conservative about accumulator precision for the second Matmul and can focus its accuracy-preservation efforts on the $QK^T$ path, where the precision-accuracy tradeoff is real and constraining.

Evidence: Table 3 is the core demonstration, spanning three modalities (language, image, video), six models, and 14 total evaluation metrics (perplexity, accuracy, CLIPSIM, CLIP-T, VQA-a, VQA-t, FScore, FID, sFID, CLIP, IR). The SageAttn2++ variants match SageAttn2 variants within the precision of the reported digits for essentially every metric. This is a remarkably comprehensive null result for accuracy degradation.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The paper evaluates on a multi-modal benchmark suite: language models use WikiText (Merity et al., 2022) for perplexity, LAMBADA (Paperno et al., 2016) for contextual understanding accuracy, and the Needle-in-A-Haystack (NIAH) task (Kamradt, 2023) for long-context retrieval accuracy; video generation models use the Open-Sora prompt set (Zheng et al., 2024); and image generation models use COCO annotations (Lin et al., 2014). These datasets are standard in their respective domains and are used for inference-time evaluation only—no training or fine-tuning is performed.

  • Base model(s). The paper evaluates six models spanning three modalities: Llama3.1 (8B) for text-to-text, CogvideoX (2B), HunyuanVideo, and Wan for text-to-video, and Flux (schnell) and Stable-Diffusion3.5 (turbo) for text-to-image. These models are chosen as representative state-of-the-art architectures in their respective domains at scales that fit on consumer GPUs (RTX4090/5090), and they exercise attention across diverse sequence lengths, head dimensions, and attention patterns (causal, bidirectional, spatial-temporal). The diversity is deliberate: demonstrating that a drop-in attention kernel works across language, image, and video generation without per-model tuning is central to the paper's "plug-and-play" claim.

  • Metrics. End-to-end model quality is measured with modality-specific metrics. For language: perplexity on WikiText, accuracy on LAMBADA and NIAH. For video: CLIPSIM and CLIP-Temp (text-video alignment), VQA-a and VQA-t (aesthetic and technical quality), and Flow-score (FScore) for temporal consistency. For images: FID and sFID (fidelity), CLIP score (text-image alignment), and ImageReward (IR) (human preference). Attention-level accuracy uses three metrics defined in Appendix A.2: cosine similarity (Cossim = ∑ OO' / √∑O² √∑O'²), relative L1 distance (L1 = ∑|O - O'| / ∑|O|), and RMSE = √(1/n ∑(O - O')²), all computed between the quantized attention output O′O' and the full-precision output OO flattened into vectors.

  • Baselines. Four baselines are compared: (1) FlashAttention2 (Dao, 2024), the standard optimized exact-attention kernel serving as the universal speed baseline—the paper notes that FlashAttention3 requires Hopper GPUs and is therefore unavailable on the RTX4090/5090 test hardware; (2) SageAttention (Zhang et al., 2025c), the original 8-bit quantized attention; (3) SageAttention2 (Zhang et al., 2025a), the immediate predecessor with quantized Q,KQ, K (INT4 or INT8) and P,VP, V (FP8 with FP32 accumulation); and (4) full-precision attention, the unquantized FP16 attention computation for each model as an accuracy ceiling. SageAttention2++ itself comes in two variants: 8+8 (INT8 for Q,KQ, K, FP8 with FP16 accumulation for P,VP, V) and 4+8 (INT4 for Q,KQ, K, FP8 with FP16 accumulation for P,VP, V), matching SageAttention2's variant naming convention.

  • Generation budget / compute accounting. For kernel-level speed measurements (Figures 1–4), the metric is raw kernel execution time in milliseconds across varying sequence lengths (64 to 16,384 tokens) with head dimensions of 64 and 128, both with and without causal masking. Speedup factors are reported as ratios of FlashAttention2's execution time to the measured kernel's execution time at the same configuration. For end-to-end model evaluation, there is no adjustable generation budget—the kernel replaces all attention operations in the model, and the full model is run once to generate outputs for evaluation. The speedup claim of 3.9× refers to the attention kernel alone, not end-to-end model latency.

  • Cross-validation / statistical protocol. None reported. End-to-end metrics are single-run evaluations on standard test sets. No multiple seeds, error bars, or statistical significance tests are reported for any metric. The paper relies on the granularity of the reported digits (3 decimal places for most metrics) to argue that differences between SageAttention2 and SageAttention2++ are negligible—but without reported variance, it is impossible to confirm that a difference of 0.001 in L1 or 0.1 in FID is smaller than run-to-run variability.

Main Quantitative Results

Kernel-Level Speed

The headline speedup claim is that SageAttn2++ (4+8) achieves a 3.9× speedup over FlashAttention2 on RTX4090 at headdim=128, while SageAttn2++ (8+8) achieves approximately 3.0×. These numbers are extracted from Figure 1, which plots kernel time in milliseconds against sequence length for all methods.

Figure 1 (RTX4090, headdim=128) shows the following pattern:

  • FlashAttention2 (the 1× baseline) scales roughly linearly with sequence length, as expected for an O(N2)O(N^2) operation where NN is the sequence length.
  • SageAttention is approximately 2.0–2.5× faster than FlashAttention2 across sequence lengths, consistent with its previously reported performance.
  • SageAttention2 (4+8) and SageAttention2 (8+8) achieve roughly 2.8× and 2.4× respectively—already representing the gains from quantized QKTQK^T and PVPV with FP32 accumulation.
  • SageAttention2++ (4+8) achieves the highest speedup at approximately 3.9× across most sequence lengths (the gap is slightly narrower at very short sequences, where kernel launch overhead dominates).
  • SageAttention2++ (8+8) achieves approximately 3.0×, reflecting the additional cost of INT8 QKTQK^T versus INT4.

The comparison between SageAttention2 variants and their SageAttention2++ counterparts is the critical within-family speedup: SageAttn2 (4+8) at ~2.8× versus SageAttn2++ (4+8) at ~3.9×, representing an additional ~1.4× speedup from the FP16 accumulator switch alone (noting that the absolute ratio 3.9/2.8 ≈ 1.39, close to the expected ~1.4–1.5× for a 2× speedup on the PV Matmul which constitutes roughly half the attention computation).

Figures 2 (RTX4090, headdim=64), 3 (RTX5090, headdim=128), and 4 (RTX5090, headdim=64) show similar relative orderings. The absolute speedups on RTX5090 (Figures 3, 4) differ slightly from RTX4090 but the qualitative pattern holds: SageAttn2++ (4+8) is the fastest, followed by SageAttn2++ (8+8), then SageAttn2 variants, then SageAttention, with FlashAttention2 as the slowest. Exact speedup factors are not reported in the text for these configurations, only visible in the figures.

A notable non-result: the paper does not break down what fraction of the total speedup comes from the QKTQK^T quantization versus the PVPV quantization. The reader cannot determine from the reported data whether the 3.9× over FlashAttention2 is, for example, 2.0× from QKTQK^T INT4 and 1.95× from PVPV FP8-with-FP16-accumulation, or some other split. This matters because it obscures how much headroom remains—if QKTQK^T is already near-optimal and PVPV is now fully exploited, the total speedup is near a ceiling; if not, further gains may be possible.

Attention-Level Accuracy (Table 2)

Table 2 is the linchpin of the paper's accuracy argument. It reports the average attention accuracy across all attention layers of CogvideoX for four configurations:

MethodPrP_rVrV_rCossim ↑L1 ↓
SageAttn244844899.97%0.01862
SageAttn2++4482.2599.97%0.01863
SageAttn2++2244.599.97%0.01862
SageAttn2++112999.97%0.01863

The finding is unambiguous: cosine similarity is unchanged at 99.97% across all configurations, and the relative L1 distance varies only in the fifth decimal place (0.01862 vs. 0.01863). The product Pr×VrP_r \times V_r is 1008 for all three SageAttn2++ configurations (satisfying the delayed buffering constraint Pr×Vr≤1023.5P_r \times V_r \leq 1023.5), and the SageAttn2 baseline product is 200,704 (only safe with FP32 accumulation). The paper's conclusion—that narrowing the quantization range introduces negligible error—rests entirely on this table.

This is strong evidence, but with two important caveats:

  1. This is one model (CogvideoX). The paper does not report analogous Table 2 results for Llama3.1, HunyuanVideo, Wan, Flux, or Stable-Diffusion3.5. The reader must assume that CogvideoX's attention statistics are representative of all target models, which the end-to-end results in Table 3 implicitly support (since those models show no degradation either), but the direct per-layer attention accuracy comparison is not shown for other models. If any model had attention weight distributions that exercise the FP16 accumulator's range more aggressively, it would surface here—and we have no evidence it does not exist.

  2. Only two metrics are reported. Cosine similarity and L1 distance aggregate error across the entire attention output tensor. They might mask structured errors in specific attention heads or specific sequence positions that could affect end-to-end behavior. The paper does not report per-head breakdowns, per-layer breakdowns, or worst-case element-wise errors. The RMSE metric defined in Appendix A.2 is not reported in Table 2, which is an omission given that RMSE penalizes large individual errors more heavily than L1.

End-to-End Model Quality (Table 3)

Table 3 is the comprehensive end-to-end evaluation, spanning three blocks:

Language (Llama3.1 8B): Three methods compared—Full-Precision, SageAttn2 (8+8), and SageAttn2++ (8+8). The 4+8 variants are not evaluated for language. Results:

  • WikiText perplexity: Full-Precision = 6.013, SageAttn2 (8+8) = 6.019, SageAttn2++ (8+8) = 6.020. The differences (0.006–0.007) represent a ~0.1% relative change and are smaller than typical perplexity variance on WikiText.
  • LAMBADA accuracy: Full-Precision = 0.815, SageAttn2 = 0.811, SageAttn2++ = 0.813. SageAttn2++ is 0.002 above SageAttn2 and 0.002 below Full-Precision—noise-level differences.
  • NIAH accuracy: Full-Precision = 0.906, SageAttn2 = 0.903, SageAttn2++ = 0.901. SageAttn2++ drops 0.005 from Full-Precision and 0.002 from SageAttn2. This is small but directionally consistent—possibly random, but without error bars, no conclusion can be drawn.

Text-to-Video (CogvideoX 2B, HunyuanVideo, Wan): All five methods evaluated—Full-Precision, SageAttn2 (4+8), SageAttn2 (8+8), SageAttn2++ (4+8), and SageAttn2++ (8+8). The key comparison for the paper's claim is SageAttn2 (8+8) versus SageAttn2++ (8+8), and SageAttn2 (4+8) versus SageAttn2++ (4+8). Results for CogvideoX:

  • SageAttn2++ (8+8) vs. SageAttn2 (8+8): CLIPSIM: 0.179 vs. 0.178 (SageAttn2++ higher); CLIP-T: 0.997 vs. 0.997 (tied); VQA-a: 76.309 vs. 74.322 (SageAttn2++ notably higher); VQA-t: 73.165 vs. 74.447 (SageAttn2++ lower); FScore: 4.386 vs. 4.899 (SageAttn2++ lower). The VQA-a and VQA-t differences are in opposite directions and not trivial (2–3 points), but without variance estimates, these could be within run-to-run noise for video generation metrics. The FScore drop of 0.513 is more substantial relative to the absolute values (~10% relative), which warrants attention.

  • SageAttn2++ (4+8) vs. SageAttn2 (4+8): Nearly identical to the reported precision for all metrics. CLIPSIM: 0.179 vs. 0.179; CLIP-T: 0.997 vs. 0.997; VQA-a: 74.387 vs. 76.309 (wait—SageAttn2 (4+8) shows 76.309 at VQA-a, not 74.387; let me read the table carefully). From Table 3: SageAttn2 (4+8) at VQA-a = 76.309, SageAttn2++ (4+8) at VQA-a = 74.387. That's a 1.922 point drop for SageAttn2++, which contradicts the "matching" claim. However, looking at the pattern across columns, there may be a row alignment or reporting issue: the VQA-a value for SageAttn2 (8+8) is 74.322, which is closer to SageAttn2++ (4+8)'s 74.387, while SageAttn2 (4+8)'s 76.309 matches SageAttn2++ (8+8)'s 76.309. This suggests a possible copy error in the table (rows possibly misaligned during composition) or genuine metric instability that the paper does not discuss.

For HunyuanVideo and Wan, similar patterns: SageAttn2++ variants track SageAttn2 variants closely, with occasional fluctuations of 1–3 points on VQA and FScore metrics. The paper interprets these as negligible, and in the context of these metrics' typical variance, this is a defensible interpretation—but the absence of error bars makes the defense weaker than it could be.

Text-to-Image (Flux, Stable-Diffusion3.5): For Flux:

  • SageAttn2++ (8+8) vs. SageAttn2 (8+8): FID: 163.555 vs. 163.185 (SageAttn2++ slightly worse); sFID: 146.036 vs. 146.101 (SageAttn2++ slightly better); CLIP: 31.445 vs. 31.453 (near-identical); IR: 0.902 vs. 0.905 (SageAttn2++ slightly worse). Differences are in the third decimal place or single digits for FID, well within typical run variance for these metrics.
  • SageAttn2++ (4+8) vs. SageAttn2 (4+8): Identical to three decimal places across all four metrics (FID: 164.170, sFID: 147.185, CLIP: 31.358, IR: 0.910 for both). This is the cleanest match in Table 3.

For Stable-Diffusion3.5, SageAttn2++ (8+8) shows FID = 165.842 vs. SageAttn2 (8+8) at 164.971 (worse by 0.9), and IR = 0.929 vs. 0.931 (near-identical). SageAttn2++ (4+8) is identical to SageAttn2 (4+8) across all metrics.

Qualitative results: Figures 5, 6, and 7 show side-by-side visual examples for image and video generation with full-precision attention versus SageAttention2++. The differences are visually imperceptible—the generated images and video frames appear identical—supporting the claim that the quantization error does not produce visible artifacts.

Summary of Main Results

The central claim—that SageAttention2++ matches SageAttention2's accuracy while being faster—is supported by the preponderance of endpoint metrics in Table 3, but with two qualifications: (1) a few metric cells show differences of 1–3 points on VQA or ~0.5 on FScore that are not trivially "within noise" without variance reporting, and (2) the possibility of a copy error in the CogvideoX VQA-a column for SageAttn2 (4+8) prevents a fully clean "identical" conclusion. The kernel speedup claim (3.9× over FlashAttention2) is directly measured and well-supported by Figures 1–4.

Ablation Studies and Robustness Checks

$(P_r, V_r)$ parameter sweep (Table 2): The paper tests four $(P_r, V_r)$ configurations on CogvideoX attention accuracy: the original SageAttention2 baseline (448, 448) and three SageAttention2++ configurations with product Pr×Vr=1008P_r \times V_r = 1008—(448, 2.25), (224, 4.5), and (112, 9). All three SageAttention2++ configurations produce identical cosine similarity (99.97%) and near-identical L1 distance (0.01862–0.01863). This demonstrates that the particular split between PrP_r and VrV_r does not affect accuracy and that the product constraint being satisfied is the sufficient condition for safety. The paper does not test configurations with product values other than 1008 (e.g., a more aggressive 500 or a more conservative 1500), so the lower bound on the safe product is not established—1008 works, but the reader does not know whether 500 would also work or whether 1500 would provide even more margin.

$P_r$ and $V_r$ final selection rationale: The choice of Pr=224,Vr=4.5P_r = 224, V_r = 4.5 over the alternatives is stated as "for optimal performance" without elaboration. No ablation demonstrates that this specific choice yields faster kernel execution than (448, 2.25) or (112, 9). Since all three satisfy the delayed buffering constraint, they should all be able to use the same mma.f16.f8.f8.f16 instruction and the same delayed buffering depth, so the performance should be identical—if so, the choice is arbitrary and the "optimal performance" phrasing is misleading. If there is a subtle performance difference (e.g., from the quantization rounding implementation handling certain values more efficiently), it is not documented.

Delayed FP32 buffering (Section 3.2): The paper describes the delayed buffering optimization that halves conversion overhead and tightens the constraint from Pr×Vr≤2047P_r \times V_r \leq 2047 to Pr×Vr≤1023.5P_r \times V_r \leq 1023.5, but it does not report any ablation comparing performance with and without delayed buffering. The reader cannot determine what fraction of the speedup comes from the MMA instruction swap alone versus the combined effect of instruction swap plus delayed buffering. A kernel-time comparison at fixed $(P_r, V_r)$ with delayed buffering enabled vs. disabled would quantify this contribution.

$QK^T$ quantization levels (8+8 vs. 4+8): The two variants are evaluated across all models in Table 3. The 4+8 variant (INT4 for Q,KQ, K) consistently shows slight metric degradation relative to 8+8 on some models (e.g., CogvideoX FScore drops from 4.899 to 4.386 for SageAttn2, and from 4.386 to 4.333 for SageAttn2++). This is not an ablation of SageAttention2++ per se—the QKTQK^T quantization is inherited unchanged from SageAttention2—but it confirms that the accuracy-critical path remains the first Matmul, not the second. The fact that SageAttn2++ (4+8) and SageAttn2 (4+8) produce identical metrics (within reporting precision) reinforces the paper's core claim: the PVPV accumulator change introduces no additional error beyond what the QKTQK^T quantization already introduces.

Causal vs. non-causal masking (Figures 1–4): The kernel speed benchmarks in Figures 1–4 report results with and without causal masking. The paper does not comment on whether the relative speedups differ between these two regimes, but the figures suggest they are broadly consistent—the ordering of methods is preserved with and without causal masking, indicating that the optimization is not specific to bidirectional attention and applies equally to autoregressive models.

Hardware portability (RTX4090 vs. RTX5090, Figures 1–4): The speed benchmarks on both GPU generations show similar relative orderings, suggesting the optimization is robust across Lovelace (RTX4090) and Blackwell (RTX5090) architectures. The absolute speedups may differ slightly (the paper does not quote separate speedup factors per GPU), but the qualitative advantage of SageAttn2++ over SageAttn2 and FlashAttention2 persists. No H100 or A100 results are reported, which would test whether the FP16 accumulator advantage generalizes to datacenter GPUs or is specific to consumer architectures.

No per-layer or per-head breakdown of attention error: A meaningful ablation that is absent from the paper is a per-layer analysis of attention accuracy. Modern transformers often have wild variations in attention patterns across layers—early layers may have diffuse attention, later layers may have highly peaked attention—which could interact differently with the narrowed quantization ranges. The paper reports only "average attention accuracy across all attention layers" for CogvideoX (Table 2). If some layers experience significantly larger attention errors (even if the average is fine), those layers could bottleneck end-to-end quality on specific inputs, especially for long sequences or out-of-distribution prompts. No such breakdown is provided.

Missing $QK^T$ accumulator analysis: The paper focuses exclusively on the PVPV Matmul and explicitly leaves the QKTQK^T Matmul unchanged from SageAttention2. It does not investigate whether the FP16-accumulator optimization could also apply to the QKTQK^T Matmul with analogous quantization range constraints. This is a natural question given the paper's contribution framework, and its absence is an explicit scope limitation rather than an ablation.

Critical Assessment

Claim 1: "SageAttention2++ achieves a 3.9× speedup over FlashAttention"

This claim is well-supported by the kernel microbenchmarks in Figures 1–4. The 3.9× number is specifically for SageAttn2++ (4+8) on RTX4090 with headdim=128 (Figure 1). The evidence is direct: wall-clock kernel execution time measured across sequence lengths shows SageAttn2++ (4+8) approximately 3.9× faster than FlashAttention2 at the same configuration.

However, the claim is narrower than it initially appears. The 3.9× refers to the attention kernel in isolation, not end-to-end model inference. For a typical transformer, attention is one component among many (MLP layers, normalization, embedding lookups, etc.), and Amdahl's law applies: the end-to-end speedup is bounded by the fraction of time spent in attention. The paper reports no end-to-end latency measurements, no throughput numbers, and no wall-clock generation time comparisons for any of the evaluated models. A practitioner reading "3.9× speedup over FlashAttention" might reasonably expect this to translate to a near-4× reduction in generation latency; in practice, the end-to-end speedup will be substantially smaller (perhaps 1.5–2× for models where attention is 50–70% of total compute, less for models with shorter sequences where attention is a smaller fraction). This is a standard limitation of kernel-level benchmarking, but the paper does not discuss it, which is a notable omission.

Additionally, the 3.9× is the best-case variant (4+8). The (8+8) variant achieves ~3.0×, and the (8+8) variant is the one that achieves "almost no metrics loss" across all models (the paper explicitly states "SageAttn2++(4+8) brings a little metrics loss"). So the variant with the highest accuracy preservation is ~30% slower than the headline number. This nuance—that you trade speed for accuracy across the 4+8/8+8 dimension, even if not across the FP32/FP16 accumulator dimension—is important for practical adoption but is not highlighted in the abstract or conclusion.

Claim 2: "SageAttention2++ maintains the same attention accuracy as SageAttention2"

The evidence for this claim is mixed. Table 2 provides direct support at the attention-output level for CogvideoX: cosine similarity of 99.97% and L1 distance of ~0.0186 are identical between SageAttention2 and all SageAttention2++ configurations. This is a clean, well-measured result.

Table 3 provides broader but less controlled support. For most models and most metrics, SageAttn2++ and SageAttn2 produce values within reporting precision. However, there are exceptions:

  • CogvideoX: SageAttn2 (4+8) VQA-a = 76.309, SageAttn2++ (4+8) VQA-a = 74.387 (difference: 1.922). This is a non-trivial difference, and the direction is inconsistent with the claim of "matching"—SageAttn2++ is worse. However, the SageAttn2 (8+8) VQA-a is 74.322, and the SageAttn2++ (8+8) is 76.309. These values appear swapped relative to expectations, raising the possibility of a copy error in the table rather than a genuine accuracy difference. The paper does not comment on this anomaly.

  • CogvideoX: SageAttn2 (8+8) FScore = 4.899, SageAttn2++ (8+8) FScore = 4.386 (difference: 0.513, or 10.5% relative). This is a meaningful drop. The SageAttn2++ (8+8) FScore matches SageAttn2 (4+8)—which is the lower-accuracy variant—suggesting the SageAttn2++ (8+8) is performing at the degraded level on this metric.

  • HunyuanVideo VQA-t: SageAttn2 (8+8) = 54.878, SageAttn2++ (8+8) = 51.080 (difference: 3.798). This is a substantial drop.

Without error bars, standard deviations, or multiple runs, the reader cannot determine whether these discrepancies are noise, systematic degradation, or typographical errors in the table. The most charitable interpretation is that they represent run-to-run variance in video generation metrics, which are known to be high-variance. But for a paper whose central claim is "same accuracy," reporting zero variance estimates is a significant weakness. A simple 3-run average with standard deviation for each metric would have made the "matching" claim falsifiable and far more credible.

The categorical statement that "SageAttn2++(8+8) incurs almost no metrics loss across various models" (Section 4.3) is too strong given the data presented. A more accurate characterization would be: "Most metrics are within the expected run-to-run variance of generative model evaluation; a few metrics show differences of 1–5 points whose significance cannot be assessed without variance estimates."

Claim 3: "SageAttention2++ effectively accelerates various models, including those for language, image, and video generation, with negligible end-to-end metrics loss"

The "effectively accelerates" portion is not directly demonstrated. The paper shows kernel-level speedup (Figures 1–4) and end-to-end accuracy (Table 3), but end-to-end latency or throughput speedup is never measured. This is the most significant gap between what the paper claims and what it demonstrates. The kernel speedup of 3.9× is a necessary condition for end-to-end acceleration, but it is not sufficient—the actual user-perceived speedup depends on the attention-to-total-compute ratio in each model, which varies with sequence length, model architecture, and batch size. For short-sequence language generation (e.g., Llama3.1 with 2048-token context), attention might be 20–40% of total compute, yielding perhaps a 20–40% end-to-end speedup despite the 3.9× kernel speedup. For long-sequence video generation with high-resolution spatial attention, the fraction could be much higher. The paper provides no end-to-end wall-clock numbers, no throughput comparisons, and no guidance on what overall speedup a practitioner should expect when dropping SageAttention2++ into their pipeline.

This is a standard limitation of kernel papers—many FlashAttention papers similarly report kernel speedups without end-to-end benchmarks—but it limits the practical interpretability of the "3.9×" figure. A reader deploying a video generation model should not expect 3.9× faster generation; they should expect some smaller, model-specific factor that the paper does not help them estimate.

Claim 4: The design constraint Pr×Vr≤2047P_r \times V_r \leq 2047 (or 1023.51023.5 with delayed buffering) ensures safe FP16 accumulation

This is a mathematical guarantee, not an empirical claim, and it follows directly from the hardware specification of the FP16 format and the mma.m16n8k32 instruction. It requires no experimental support to be true. However, the paper does not empirically validate that violating the constraint produces failures—no experiment shows kernel outputs going to infinity or producing NaN when Pr×Vr>2047P_r \times V_r > 2047. Such a "negative ablation" would strengthen the paper by making the failure mode concrete, but it is not essential to the argument.

Missing Experiments That Would Strengthen the Paper

  1. End-to-end latency benchmarks. Wall-clock time for a full forward pass (or generation of a fixed number of tokens/frames) on each model with each attention kernel. This is the single most important missing experiment for practitioners evaluating whether to adopt SageAttention2++.

  2. Variance estimates for end-to-end metrics. At minimum, 3-run averages with standard deviations for a subset of models (e.g., Llama3.1 and one video model) to contextualize the metric differences in Table 3.

  3. Sensitivity analysis for the Pr×VrP_r \times V_r product. Testing product values of 2000, 1500, 1008 (already tested), 750, and 500 to establish the empirical lower bound where attention accuracy degrades. This would give practitioners guidance on how much additional headroom exists for future hardware with even narrower accumulators.

  4. Per-layer attention error analysis. Reporting cossim and L1 per layer for one model to verify that no individual layer experiences disproportionate degradation that is masked by averaging.

  5. Comparison against FlashAttention3 on supported hardware. The paper notes that FlashAttention3 requires Hopper GPUs and therefore cannot run on RTX4090/5090, but a single data point on an H100 would contextualize how SageAttention2++'s approach compares to the latest exact-attention implementation on datacenter hardware.

  6. Ablation of delayed FP32 buffering. Kernel time with and without the buffering optimization to quantify its contribution to the total speedup.

  7. End-to-end speedup with FlashAttention2 as the attention backend versus SageAttention2++ as the backend. This is the comparison implied by the abstract's "3.9× speedup over FlashAttention" but is never directly measured for a complete model.

6. Limitations and Trade-offs

The Headline 3.9× Speedup Is a Kernel-Level Microbenchmark, Not an End-to-End Throughput Measurement

The assumption or constraint. The paper's central speedup claim—"SageAttention2++ achieves a 3.9× speedup over FlashAttention" (Abstract, Section 4.2)—is measured exclusively on the attention kernel in isolation, not on complete model inference. The speed benchmarks in Figures 1–4 time only the attention operation itself, excluding all other model components: MLP layers, layer normalization, embedding lookups, softmax, residual additions, and any host-to-device or device-to-host data transfers. The paper acknowledges this implicitly by referring to "the speed of the attention kernel" and "kernel speed" throughout Section 4.2, but the abstract and conclusion use unqualified language ("3.9× speedup over FlashAttention," "effectively accelerates various models") that does not distinguish kernel-level from end-to-end speedup.

The consequence. A practitioner deploying SageAttention2++ in a production inference pipeline should not expect a 3.9× reduction in generation latency or a 3.9× increase in throughput. By Amdahl's law, the end-to-end speedup is capped by the fraction of total compute time spent in attention. For a typical transformer at moderate sequence lengths (e.g., Llama3.1-8B at 2048 tokens), the attention sub-component might account for 30–50% of total forward-pass FLOPs, meaning the maximum achievable end-to-end speedup from a 3.9× faster attention kernel is approximately 1.5–2.0×—a substantially smaller gain than the headline number implies. For very long sequences where attention dominates (e.g., 16K+ tokens), the end-to-end speedup would approach the kernel speedup more closely, but the paper does not provide the attention-fraction data that would let practitioners estimate the speedup for their specific workload. For video generation models where spatial and temporal attention constitute a larger fraction of compute, the end-to-end benefit may be larger; for short-sequence language tasks, much smaller. The absence of end-to-end latency numbers makes it impossible to calibrate expectations.

What evidence exists in the paper. None. The paper reports zero end-to-end latency or throughput measurements for any model. Table 3 reports end-to-end accuracy metrics, but no corresponding timing measurements. Figures 1–4 are strictly kernel microbenchmarks. Section 4.3 ("End-to-end Performance") discusses only metrics loss, not wall-clock speed. The paper never measures how long a complete forward pass or a complete generation takes with FlashAttention2 vs. SageAttention2++ for Llama3.1, CogvideoX, or any other model.

Mitigation status. The paper does not acknowledge this limitation. The abstract and conclusion use unqualified language that could mislead a reader into expecting a 3.9× system-level speedup. This is a standard practice in attention kernel papers (FlashAttention papers similarly report kernel speedups without always providing end-to-end benchmarks), but the omission is consequential for practitioners making deployment decisions. Future work would need to report end-to-end generation latency or throughput for the specific models and sequence lengths targeted in Section 4.3.

The "Matching Accuracy" Claim Is Weakened by Absent Variance Estimates and Unexplained Outliers

The assumption or constraint. The paper's core accuracy claim is that SageAttention2++ "matches SageAttention2's end-to-end metrics" and "incurs almost no metrics loss across various models" (Section 4.3, Section 5). This claim is supported by Table 3, which reports single-run metric values to three or four decimal places for each model, method, and metric combination. The paper provides no standard deviations, no confidence intervals, no multiple-run averages, and no statistical tests for any end-to-end metric. Differences between SageAttn2 and SageAttn2++ are interpreted as negligible based solely on the reported digits, without any estimate of the metric's run-to-run variability.

The consequence. Several cells in Table 3 show differences between SageAttn2 and SageAttn2++ that are large enough to be meaningful if stable, but their significance cannot be assessed without variance estimates:

  • CogvideoX, VQA-a, (4+8) variants: SageAttn2 (4+8) = 76.309, SageAttn2++ (4+8) = 74.387 (Δ = 1.922). Curiously, SageAttn2 (8+8) = 74.322 and SageAttn2++ (8+8) = 76.309—the (4+8) and (8+8) values appear crossed between the two methods, raising the possibility of a copy error in the table rather than a genuine accuracy difference. If the 76.309 values both belong to the same method, the difference between SageAttn2++ and SageAttn2 would be even larger than reported.

  • CogvideoX, FScore, (8+8) variants: SageAttn2 (8+8) = 4.899, SageAttn2++ (8+8) = 4.386 (Δ = 0.513, a 10.5% relative drop). The SageAttn2++ (8+8) value matches SageAttn2 (4+8)—the lower-accuracy variant—suggesting the FP16 accumulator change may be degrading the (8+8) variant to (4+8) quality on this metric.

  • HunyuanVideo, VQA-t, (8+8) variants: SageAttn2 (8+8) = 54.878, SageAttn2++ (8+8) = 51.080 (Δ = 3.798 points).

  • HunyuanVideo, VQA-a, (8+8) variants: SageAttn2 (8+8) = 78.145, SageAttn2++ (8+8) = 78.569 (SageAttn2++ higher by 0.424, inconsistent direction with other VQA differences).

Without variance estimates, the reader cannot determine whether these fluctuations represent (a) genuine systematic degradation from the narrower quantization, (b) run-to-run noise inherent to video generation metrics (VQA scores are known to have high variance across generation seeds), or (c) typographical errors in the table. The claim that SageAttn2++ "matches" SageAttn2 is therefore an overstatement relative to the evidence presented—more accurately, "most metrics are within the expected range of the reported precision, and the few metrics showing larger differences cannot be assessed for significance."

What evidence exists in the paper. Table 3 is the sole end-to-end accuracy evidence. Table 2 provides per-layer attention-level accuracy for CogvideoX only and shows clean matching at that granularity, but per-layer attention accuracy does not guarantee end-to-end metric stability—small per-layer errors can compound or interact with model-specific sensitivity. Appendix A.2 defines the accuracy metrics but provides no variance quantification protocol.

Mitigation status. The paper does not acknowledge the absence of variance estimates as a limitation. The strong "matching" language in the abstract and conclusion implies a precision of comparison that the single-run experimental design cannot support for the metrics that show discrepancies. A minimal mitigation would be 3-run averages with standard deviations for a subset of models and metrics.

The PV Quantization Range Safety Is Validated Only on CogvideoX Attention Accuracy, Not on the Full Model Suite

The assumption or constraint. The paper's entire safety argument for the narrowed quantization ranges—that Pr×Vr≤1023.5P_r \times V_r \leq 1023.5 with delayed buffering is sufficient to prevent FP16 accumulator overflow while preserving attention accuracy—rests on Table 2, which reports average attention accuracy metrics (cossim and L1) across all attention layers of CogvideoX for four (Pr,Vr)(P_r, V_r) configurations. Table 2 demonstrates that three different (Pr,Vr)(P_r, V_r) pairs with product 1008 produce identical attention-level accuracy to the SageAttention2 baseline with product 200,704. The paper uses this evidence to select Pr=224,Vr=4.5P_r = 224, V_r = 4.5 as the final parameters for all models and all evaluations. No analogous per-layer attention accuracy analysis is provided for Llama3.1, HunyuanVideo, Wan, Flux, or Stable-Diffusion3.5.

The consequence. The assumption that CogvideoX's attention weight statistics are representative of all evaluated models—and that attention accuracy at the per-layer level translates to end-to-end quality—is untested. Attention patterns vary substantially across model architectures: language models (Llama3.1) use causal masking and have attention distributions that differ qualitatively from the bidirectional spatial-temporal attention in video diffusion models (CogvideoX, HunyuanVideo, Wan) and the bidirectional spatial attention in image diffusion models (Flux, Stable-Diffusion3.5). The softmax output PP in a causally-masked language model concentrates probability mass differently than in a bidirectional video model, and the value matrix VV in different model families may have different per-channel magnitude distributions due to different training recipes, initialization schemes, and normalization placements. If a particular model has attention heads where PP or VV exhibit values that push closer to the FP16 accumulator boundary, the narrowed quantization could produce larger per-layer errors that the end-to-end metrics in Table 3 might not capture—or might capture in ways masked by the absence of variance estimates.

The paper also does not explore the lower bound of the product constraint. The fact that Pr×Vr=1008P_r \times V_r = 1008 works does not establish whether Pr×Vr=500P_r \times V_r = 500 or Pr×Vr=2047P_r \times V_r = 2047 (without delayed buffering) would also work, or where the empirical accuracy cliff lies. Practitioners targeting future hardware with even narrower accumulators (e.g., FP8 accumulation on next-generation Tensor Cores) would need this sensitivity information to assess portability.

What evidence exists in the paper. Table 2 covers CogvideoX only. Table 3 provides indirect evidence that the parameters generalize—end-to-end metrics for the other models are mostly stable—but these are single-run measurements without per-layer breakdowns that could reveal compensating errors or specific vulnerable layers. The paper does not report, for any model other than CogvideoX, the cossim or L1 between quantized and full-precision attention outputs.

Mitigation status. The paper does not discuss this as a limitation. It implicitly treats CogvideoX attention accuracy as generalizable. A minimal mitigation would be a single-figure per-layer cossim plot for one additional model (e.g., Llama3.1) showing that attention accuracy remains at the same 99.97% level, or a statement that per-layer attention accuracy was verified on all models and matched the CogvideoX results.

The QKᵀ Matmul Accumulator Is Not Addressed, Limiting the Total Speedup Ceiling

The assumption or constraint. The paper focuses exclusively on accelerating the second Matmul (PVPV), leaving the first Matmul (QKTQK^T) unchanged from SageAttention2. The QKTQK^T path uses INT4 or INT8 quantization with MMA instructions whose accumulator precision the paper does not discuss or attempt to optimize. The speedup from QKTQK^T quantization (INT4 or INT8 vs. FP16) is the primary source of the difference between the (4+8) and (8+8) variants, but the paper provides no breakdown of what fraction of the total attention time is spent in QKTQK^T versus PVPV, making it impossible to determine how much additional headroom exists in the first Matmul.

The consequence. The FP16-accumulator optimization might also apply to the QKTQK^T Matmul if the quantization ranges of QQ and KK could be similarly constrained, but the paper does not investigate this. The QKTQK^T Matmul has a fundamentally different accumulation structure: it produces an N×NN \times N score matrix (quadratic in sequence length) rather than an N×dN \times d output matrix, and the softmax that follows QKTQK^T is applied after the Matmul, not before. The accumulator overflow analysis would need to consider the magnitudes of quantized QQ and KK values rather than PP and VV, and the acceptable error tolerance might differ because the downstream softmax can amplify or attenuate errors nonlinearly. If the QKTQK^T path could achieve a similar 2× speedup from accumulator narrowing, the total attention speedup could push substantially beyond 3.9×. Conversely, if QKTQK^T is fundamentally limited by the INT8/INT4 MMA instructions available, then the 3.9× represents a hard speedup ceiling for exact quantized attention on current hardware, and further gains will require architectural changes to transformers themselves.

The paper also does not discuss whether the choice of QKTQK^T quantization (INT8 vs. INT4) interacts with the PVPV accumulator choice. The (8+8) variant shows smaller speedup (~3.0×) than (4+8) (~3.9×), and the (8+8) variant is the one the paper recommends for "almost no metrics loss" (since (4+8) "brings a little metrics loss," Section 4.3). This creates an implicit speed-vs-accuracy tradeoff that is orthogonal to the paper's contribution: a practitioner who needs maximal accuracy must accept the (8+8) variant's lower speedup, even though the PVPV optimization itself introduces no additional accuracy degradation. The paper does not disentangle these two tradeoff dimensions or provide guidance on model-specific optimal variant selection.

What evidence exists in the paper. Figures 1–4 show the speed of all variants together but do not break down the QKTQK^T vs. PVPV time fractions. The paper mentions that the QKTQK^T quantization is unchanged from SageAttention2 (Section 3 introduction) and does not discuss applying FP16 accumulators to this path. No sensitivity analysis of QQ and KK quantization ranges analogous to Table 2 is provided.

Mitigation status. The paper implicitly scopes its contribution to the PVPV Matmul and does not claim to optimize QKTQK^T. This is a legitimate scope limitation for a short paper targeting a specific kernel improvement, but the absence of any discussion about whether the same technique could apply to QKTQK^T—or why it cannot—leaves an obvious question unanswered. Future work could investigate whether INT8/INT4 QKTQK^T Matmuls can use narrower accumulators with analogous quantization range constraints, potentially pushing the total speedup beyond 4×.

The Difficulty Estimation Overhead (Generating 2048 Samples per Question) Is Not Accounted for in the Reported Speedup

Wait—this limitation template item from the prior sections was about a different paper (the compute-optimal test-time scaling paper). SageAttention2++ does not use difficulty estimation or sample generation. I need to identify limitations specific to this kernel paper. Let me re-identify the remaining consequential limitation.

The Delayed FP32 Buffering Optimization's Contribution to the Speedup Is Not Isolated or Ablated

The assumption or constraint. Section 3.2 introduces the delayed FP32 buffering optimization—accumulating two consecutive mma.m16n8k32 results in FP16 before a single FP32 conversion to halve the PTX conversion overhead—and notes that this tightens the overflow constraint from Pr×Vr≤2047P_r \times V_r \leq 2047 to Pr×Vr≤1023.5P_r \times V_r \leq 1023.5. The paper presents this as part of the complete SageAttention2++ design and selects all (Pr,Vr)(P_r, V_r) parameters to satisfy the tighter constraint. However, the paper never isolates the performance contribution of delayed buffering: no kernel timing comparison is reported between a version with the mma.f16.f8.f8.f16 instruction swap but without delayed buffering (using Pr=448,Vr=4.5P_r = 448, V_r = 4.5 to satisfy Pr×Vr≤2047P_r \times V_r \leq 2047) and the full SageAttention2++ with delayed buffering enabled.

The consequence. The reader cannot determine what fraction of the SageAttention2++ speedup over SageAttention2 comes from the MMA instruction switch alone versus the combined effect of instruction switch plus delayed buffering. The paper reports speedups relative to FlashAttention2 (Figures 1–4), not relative to a SageAttention2++ variant with instruction swap only and no buffering. If the delayed buffering optimization provides, for example, a 10% additional speedup on top of the MMA instruction swap, the choice of buffering depth (2 accumulations) represents a design tradeoff with measurable performance consequences that the paper does not quantify. Conversely, if delayed buffering provides negligible speedup in practice (because the conversion overhead is amortized or hidden by other pipeline stages), the tightened constraint to Pr×Vr≤1023.5P_r \times V_r \leq 1023.5 was unnecessary and the paper could have used the looser Pr×Vr≤2047P_r \times V_r \leq 2047 bound, giving more headroom for future models with larger value matrix magnitudes.

The lack of buffering-depth analysis also means the paper does not explore whether deeper buffering (3 or 4 accumulations before conversion) would yield further speedups that justify the even tighter quantization range constraints. The choice of depth 2 is stated without empirical justification.

What evidence exists in the paper. None. The paper describes delayed FP32 buffering as a design component (Section 3.2) and uses it in the final kernel whose performance is reported in Figures 1–4, but provides no ablation comparing buffering-depths or buffering-enabled vs. buffering-disabled variants. Table 2 evaluates (Pr,Vr)(P_r, V_r) pairs that all satisfy the tighter buffering constraint (Pr×Vr≤1023.5P_r \times V_r \leq 1023.5), so even the attention accuracy analysis does not compare against a design that satisfies only the looser constraint.

Mitigation status. The paper does not acknowledge the absence of this ablation or discuss the buffering depth as a tunable parameter. A single additional curve in Figure 1 showing the performance of SageAttention2++ without delayed buffering (or with buffering depths 1, 2, and 3) would resolve this and provide practical guidance for kernel engineers adapting the technique to other hardware.

End-to-End Metrics Are Reported for Only One Inference Run per Configuration, Masking Seed Sensitivity in Generative Models

The assumption or constraint. All end-to-end metrics in Table 3 are single-run evaluations: one inference pass per model, per attention method, per metric. Generative models—particularly video diffusion models like CogvideoX, HunyuanVideo, and Wan—produce outputs with substantial variation across random seeds (initial noise, sampling stochasticity). Metrics like VQA-a, VQA-t, and FScore are computed on generated videos and are therefore subject to seed-dependent variance. Image generation metrics like FID are typically computed over thousands of samples to achieve stable estimates; the paper does not report how many samples were used per metric, but the single-run values suggest small sample sizes that may not be stable.

The consequence. Fluctuations of 1–5 points in video quality metrics (VQA-a, VQA-t, FScore) between SageAttn2 and SageAttn2++ could reflect seed variance rather than quantization-induced degradation. The CogvideoX VQA-a anomaly—where SageAttn2 (4+8) and SageAttn2++ (8+8) both score 76.309 while SageAttn2 (8+8) and SageAttn2++ (4+8) score 74.322 and 74.387 respectively—is consistent with two independent generation runs with different seeds producing scores that cluster around two values, potentially due to a single "good" or "bad" seed rather than a method difference. Without multiple runs, this hypothesis is untestable, and the paper cannot distinguish systematic accuracy loss from sampling noise.

The image generation metrics (FID, sFID, CLIP, IR) are less variable per-run if computed over large sample sets, but the paper does not specify the number of generated images used for each metric. Standard practice for FID on COCO is 30K generated images versus 30K reference images, but the paper may not follow this convention given the computational expense of evaluating multiple attention methods across multiple models.

What evidence exists in the paper. Table 3 provides single-run values. Appendix A.2 describes the datasets (COCO annotations, Open-Sora prompts) and metrics but does not specify the number of samples generated per evaluation or the seed protocol. Figures 5, 6, and 7 show individual qualitative examples but are inherently anecdotal.

Mitigation status. The paper does not discuss seed sensitivity, reporting protocols, or variance. This is a common weakness in generative model evaluation papers, but it is particularly consequential here because the central claim is that accuracy is preserved—a claim that requires demonstrating that any observed differences are smaller than run-to-run noise. At minimum, for a subset of models and metrics, reporting mean and standard deviation over 3–5 independent runs with different seeds (or different prompt subsets) would make the "matching accuracy" claim falsifiable and credible.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new attention algorithm, a new quantization scheme, or a new model architecture. It makes a single, surgical modification to an existing quantized attention kernel—swapping one Tensor Core MMA instruction for a faster one and adjusting two quantization scale factors to keep intermediate accumulation within the narrower numeric range—and delivers a ~1.4× speedup on the second attention Matmul with zero measurable accuracy degradation across language, image, and video generation models. The contribution is therefore best characterized as an incremental refinement with outsized practical impact: the improvement is small in conceptual novelty but applies universally to every transformer model that uses quantized attention, making it a pure infrastructure-level gain.

The conceptual shift the paper triggers, however, is larger than the technical mechanism might suggest. Prior work in the SageAttention lineage—and in low-precision attention more broadly—implicitly treated the accumulator data type as a fixed correctness requirement: FP32 accumulation was chosen because it is safe, and the question of whether safety was overkill for the PV Matmul was never seriously examined. The paper reframes the accumulator not as a correctness invariant but as a tunable resource: a narrower accumulator (FP16) provides higher throughput at the cost of a tighter dynamic range budget, and the quantization ranges of the two input matrices can be jointly constrained to fit within that budget. This reframing generalizes beyond the specific FP8/FP16 combination. Every future hardware generation that introduces a faster-but-narrower accumulator instruction—FP8 accumulation, INT16 accumulation, block floating-point accumulation—will impose its own constraint equation of the form $P_r \times V_r \leq \text{accumulator\_max} / (k \times \text{buffering\_depth})$, and the design problem becomes solving that constraint satisfaction problem while verifying that the narrowed quantization ranges do not degrade accuracy. The paper provides the template for this verification: measure per-layer attention accuracy (cossim, L1) across the feasible parameter space and confirm that end-to-end metrics are invariant. This is a portable engineering methodology, not a one-off kernel patch.

The paper also resolves a latent contradiction in the low-precision attention literature. NVIDIA's hardware documentation has long stated that mma.f16.f8.f8.f16 offers 4× throughput over the FP16 baseline while mma.f32.f8.f8.f32 offers only 2×, yet attention kernels consistently used the slower FP32-accumulator variant—presumably because the FP16 accumulator's 65,504 maximum seemed dangerously close to worst-case dot product magnitudes in attention. SageAttention2++ demonstrates that this conservatism was unnecessary for the PV Matmul: the effective dynamic range of $P \times V$ products in real trained transformers is far smaller than the worst-case analysis would suggest, and the FP16 accumulator's budget of 65,504 over 64 accumulated terms ($P_r \times V_r \times 64 \leq 65504$) is comfortably large enough when the quantization ranges are chosen to satisfy the product constraint. This finding lowers the barrier for future kernels to adopt narrower accumulators by establishing an empirical precedent that the safety margin provided by FP32 was not needed—at least for the second Matmul. The $QK^T$ Matmul remains an open question, but for PV, the speed-accuracy tradeoff collapses: you get the full speed of the FP16-accumulator instruction and you give up nothing.

For the broader field, this paper shifts attention (no pun intended) from data-type compression (making inputs smaller) to accumulator-precision compression as an under-explored source of throughput gains. The dominant narrative in efficient attention has been about reducing the bit-width of $Q, K, V$—INT8, FP8, INT4, FP4—with the accumulator treated as an implementation detail. SageAttention2++ shows that for one of the two attention Matmuls, the accumulator is the bottleneck, not the inputs. This redirects research attention: for the PV path, further input compression (e.g., FP4 for $P$ and $V$) would likely introduce unacceptable error, but optimizing the accumulator—and jointly constraining the quantization ranges to make it safe—is a relatively untapped design dimension. Future work on attention quantization should include accumulator precision as a first-class design variable alongside input precision.

The paper also makes a negative result practically valuable: the finding that accuracy is invariant to the $(P_r, V_r)$ split as long as the product satisfies the constraint (Table 2: $448 \times 2.25$, $224 \times 4.5$, and $112 \times 9$ all produce identical cossim and L1) means that the joint quantization range space has a degree of freedom that is free in practice. Kernel engineers can choose the split based on implementation convenience—numerical properties of the rounding operation, register allocation, instruction scheduling—without worrying about accuracy. This collapses a two-dimensional hyperparameter search into a one-dimensional product constraint that is trivial to satisfy, making the optimization practical to deploy across diverse models without per-model tuning.

Follow-Up Research This Work Enables

Applying the FP16 accumulator to the $QK^T$ Matmul with analogous quantization range constraints. The paper explicitly scopes its contribution to the PV Matmul and leaves the $QK^T$ path unchanged from SageAttention2, which uses INT4 or INT8 quantization with MMA instructions whose accumulator precision is not discussed. The natural follow-up question is whether the same FP16-accumulator optimization can apply to the first Matmul. The $QK^T$ Matmul has a fundamentally different structure: the quantized $Q$ and $K$ values are integers in $[-127, 127]$ (INT8) rather than FP8 values in $[-P_r, P_r]$, so the overflow analysis would need to account for the larger magnitude of INT8 dot products over the $k$-dimension reduction. Concretely, with INT8 $Q$ and $K$ bounded by 127, a 32-element dot product could reach $32 \times 127^2 = 516,128$, which exceeds the FP16 maximum of 65,504 by nearly 8×. Reducing the $Q$ and $K$ quantization ranges to satisfy $Q_r \times K_r \leq 2047$ would mean $Q_r = K_r \approx 45$—roughly 3× smaller than the natural INT8 range of 127, which would discard over half the quantization levels and likely introduce substantial accuracy degradation on the softmax input, where precision is known to matter more than in the PV path. A strong follow-up would systematically sweep $(Q_r, K_r)$ pairs analogous to Table 2, measure cossim and L1 of the $QK^T$ output, and determine whether any feasible $(Q_r, K_r)$ pair preserves attention accuracy on language and video models. A negative result—that no safe $(Q_r, K_r)$ pair preserves accuracy—would establish a clean boundary: the FP16-accumulator trick works for PV but not for $QK^T$, meaning the total attention speedup from accumulator narrowing is bounded by what the second Matmul can contribute, and further gains require architectural changes to how attention computes $QK^T$.

Establishing the empirical lower bound of the $P_r \times V_r$ product constraint through a sensitivity sweep. The paper tests three $(P_r, V_r)$ pairs all with product 1008, demonstrating that this product is sufficient for accuracy preservation, but does not establish how much lower the product could go before accuracy degrades—the necessary bound. This matters for two reasons. First, future hardware may have even narrower accumulators: if a next-generation Tensor Core supports FP8 accumulation (maximum ~448 per element, ~14,336 for 32-term dot product), the safe product constraint would tighten to $P_r \times V_r \leq 448 / 32 \approx 14$, two orders of magnitude tighter than the current $1023.5$. Knowing whether real attention computations tolerate products of 500, 200, or 50 would determine whether the technique ports forward. Second, even on current hardware, a lower product bound might enable deeper delayed buffering (4 or 8 accumulations before FP32 conversion), further reducing conversion overhead. A direct experiment would sweep $P_r \times V_r$ products from 2047 down to 50 in logarithmic steps, measuring per-layer cossim and L1 on CogvideoX (as in Table 2) and Llama3.1, and identifying the product value at which cossim drops below, say, 99.9% or L1 increases measurably. The Llama3.1 component is important because causal attention produces different $P$ distributions than bidirectional attention, and the lower bound may differ across architectures. This experiment would produce a portability curve mapping accumulator width to feasible product bounds, which future hardware-aware kernel authors could consult directly.

End-to-end latency benchmarking to bridge the gap between kernel microbenchmarks and deployment impact. The paper's headline 3.9× speedup is a kernel-level measurement, and the paper reports zero end-to-end latency or throughput numbers for any model. This is the single most important missing experiment for practitioners. A direct follow-up would measure wall-clock forward-pass time and token-generation latency for Llama3.1-8B at sequence lengths of 1024, 2048, 4096, and 8192 tokens, comparing FlashAttention2, SageAttention2, and SageAttention2++ as the attention backend while holding all other model components identical. The experiment would quantify: (a) the actual end-to-end speedup from the kernel improvement, which by Amdahl's law will be smaller than 3.9× and depends on the attention-to-total-compute ratio at each sequence length; (b) whether the speedup grows with sequence length (as attention becomes a larger fraction of total compute) or plateaus (if other components become bottlenecks); and (c) whether the (8+8) and (4+8) variants differ meaningfully in end-to-end latency in addition to the accuracy differences already reported. For video generation models (CogvideoX, HunyuanVideo), measuring total generation time for a fixed number of frames at a fixed resolution would give practitioners concrete expectations for deployment speedups in the most compute-intensive attention regime. This experiment requires no new methods—only measurement infrastructure—and would dramatically increase the paper's practical utility.

Per-layer attention error analysis to rule out compensating errors or vulnerable layers. The paper reports only average attention accuracy across all layers of CogvideoX (Table 2), which could mask a scenario where a few layers experience large attention errors that are offset by near-zero errors in other layers. While the end-to-end metrics in Table 3 suggest no catastrophic degradation, per-layer breakdowns would reveal whether certain layer types (e.g., early layers with diffuse attention vs. late layers with peaked attention) are more sensitive to the narrowed quantization ranges. A simple experiment would plot cossim and L1 error versus layer index for Llama3.1 (32 layers) and CogvideoX with the final $(P_r = 224, V_r = 4.5)$ parameters, overlaid with the SageAttention2 baseline. If all layers show cossim > 99.9% and L1 < 0.02, the average-based safety argument is robust. If specific layers (e.g., the first and last layers) show systematically higher error, that would identify an accuracy bottleneck that could be addressed with per-layer $(P_r, V_r)$ tuning—using more conservative ranges on sensitive layers and more aggressive ranges on tolerant layers—potentially recovering additional speedup without sacrificing overall accuracy. This experiment also addresses the concern that CogvideoX's attention statistics are not representative of all models (Limitation discussion in Section 6).

Exploring delayed buffering depth as a tunable parameter with a cost-benefit ablation. The paper chooses a buffering depth of 2 (two MMA results accumulated in FP16 before FP32 conversion) and states this halves conversion overhead, but provides no empirical measurement of the overhead reduction or any comparison with depth 1 (no buffering) or depth 4 (four accumulations before conversion). The choice of depth 2 appears motivated by the constraint tightening to $P_r \times V_r \leq 1023.5$, which Table 2 shows is easily satisfied—but if depth 4 with $P_r \times V_r \leq 511.75$ also preserves accuracy, it could further reduce conversion overhead. The experiment would implement SageAttention2++ variants with buffering depths of 1, 2, 3, and 4, measure kernel time at headdim=128 across sequence lengths (as in Figure 1), and report per-layer attention accuracy for each depth on CogvideoX (as in Table 2). If depth 4 preserves accuracy and yields measurable speedup over depth 2, the optimal depth shifts upward. If depth 4 degrades accuracy (because $P_r \times V_r = 512$ is too constraining), the experiment identifies the accuracy cliff and establishes depth 2 or 3 as the Pareto-optimal choice. This directly informs future kernel implementations and provides the buffering-depth sensitivity analysis that the paper currently lacks.

Cross-architecture validation on datacenter GPUs (H100/H200) to test generality beyond consumer hardware. The paper evaluates exclusively on RTX4090 (Ada Lovelace) and RTX5090 (Blackwell), both consumer GPUs. The mma.f16.f8.f8.f16 instruction's 4× throughput advantage is documented in NVIDIA's architecture whitepapers for these GPUs, but the relative speedup may differ on datacenter GPUs (H100, H200) where memory bandwidth, SM count, and Tensor Core organization differ substantially. The paper also notes that FlashAttention3 is available on Hopper GPUs, making it a stronger baseline than FlashAttention2. A direct comparison on H100 would measure: (a) whether SageAttention2++'s speedup over FlashAttention2 persists (expected) and how it compares to FlashAttention3 (uncertain—FlashAttention3 already uses low-precision and asynchronous execution that may close some of the gap); (b) whether the FP16-accumulator advantage is larger, smaller, or identical on H100 versus RTX4090; and (c) whether the end-to-end metrics remain stable on H100 hardware, which has different numerical behavior (e.g., fused multiply-add rounding modes). This would determine whether the technique is broadly applicable across NVIDIA's product stack or specific to the consumer GPU architectures tested.

Practical Applications and Downstream Use Cases

Latency-sensitive interactive video generation. Video diffusion models like HunyuanVideo and Wan are among the most computationally expensive generative models to run, with inference times measured in minutes for even short clips. The paper demonstrates that SageAttention2++ preserves end-to-end video quality metrics (CLIPSIM, VQA, FScore) on these models while providing a kernel-level attention speedup of ~3.9× over FlashAttention2. For a video generation pipeline where spatial and temporal attention constitute a large fraction of total compute—potentially 60–80% for high-resolution, multi-frame generation—the end-to-end speedup could approach 2.5–3.0×, reducing generation latency from, say, 3 minutes to ~1 minute. This directly expands the set of feasible deployment scenarios: real-time or near-real-time video generation for interactive applications, faster iteration cycles for video content creators, and reduced per-query cost for video generation APIs. The key enabling property is that SageAttention2++ requires no model retraining, architecture changes, or per-model tuning—it is a drop-in replacement for the attention backend, making adoption a matter of swapping a CUDA kernel and re-measuring quality metrics (which Table 3 already validates for these models).

On-device inference for large language models on consumer GPUs. The paper benchmarks on RTX4090 and RTX5090, placing its practical deployment scope squarely in the consumer GPU segment. For Llama3.1-8B running on a single RTX4090 with FP16 weights, attention at moderate sequence lengths (2048–4096 tokens) constitutes roughly 30–50% of total forward-pass FLOPs. A 3.0–3.9× kernel-level attention speedup translates to an estimated 1.5–2.0× end-to-end throughput improvement, which is meaningful for interactive applications (chatbots, coding assistants, local RAG systems) where per-token latency directly determines user experience. The fact that SageAttn2++ (8+8) incurs "almost no metrics loss" (Section 4.3, Table 3: WikiText perplexity 6.020 vs. 6.013 full-precision, LAMBADA accuracy 0.813 vs. 0.815) means practitioners can adopt it without the accuracy anxiety that often accompanies model quantization. For on-device deployments where a larger model is infeasible, this speedup effectively makes the existing model feel faster without hardware upgrades.

Batch inference cost reduction for image generation APIs. Text-to-image models like Flux and Stable-Diffusion3.5 are deployed at scale in commercial APIs where per-image generation cost is dominated by GPU time. The paper shows that SageAttention2++ preserves FID, CLIP score, and ImageReward for these models (Table 3: Flux FID 163.555 for SageAttn2++ (8+8) vs. 165.117 for full-precision, a negligible difference within run variance). For high-throughput batch generation—producing thousands of images from COCO-style prompts—every percentage point of attention speedup directly reduces operating cost. The PV Matmul optimization specifically benefits the cross-attention layers where $P$ comes from text-image attention weights and $V$ comes from image features, making the speedup relevant even for models with relatively short sequence lengths (where $QK^T$ does not dominate). The drop-in nature of the kernel means an API provider could upgrade their attention backend with a CUDA library swap, run a regression test suite to confirm the quality metrics in Table 3 hold on their specific prompt distribution, and realize cost savings without any model retraining or pipeline changes.

Enabling longer-sequence generation on memory-constrained hardware. While not directly a memory optimization, attention speedup indirectly enables longer sequences by making the time cost of long-sequence attention more tolerable. On an RTX4090 with 24GB of VRAM, models like Llama3.1-8B can already fit long sequences in memory (up to ~16K tokens with FP16 KV-cache), but the latency of computing attention at 16K tokens can be prohibitive for interactive use. A 3.9× kernel speedup reduces the wall-clock time spent in the attention operation for long sequences, making 8K–16K context windows more practical on consumer hardware. For video models, where spatial-temporal attention over high-resolution frames is both memory- and compute-intensive, the attention speedup makes higher frame counts or higher spatial resolutions feasible within the same latency budget. The paper does not report memory usage or KV-cache statistics, so the precise memory-constrained scenarios require further characterization, but the principle is clear: faster attention kernels expand the envelope of feasible sequence lengths at interactive latencies.

When to Prefer This Method

The paper does not articulate an explicit tradeoff between SageAttention2++ and named alternatives beyond the accuracy-preserving speedup it provides over FlashAttention2 and SageAttention2. The positioning is straightforward: SageAttention2++ is a direct replacement for SageAttention2 in all scenarios, offering strictly higher speed at identical accuracy. The choice between SageAttn2++ (8+8) and SageAttn2++ (4+8) mirrors the same tradeoff inherited from SageAttention2—(8+8) for maximal accuracy preservation across all models, (4+8) for maximal speed with minor accuracy impact on some metrics (e.g., CogvideoX FScore drops from 4.899 to 4.386). The decision between SageAttention2++ and FlashAttention2 is the standard quantized-vs-exact attention tradeoff: SageAttention2++ is ~3–4× faster but introduces a small, empirically quantified quantization error; FlashAttention2 computes exact attention with no quantization error but runs slower. The paper's end-to-end results in Table 3 provide the data practitioners need to make that tradeoff for their specific model and metric sensitivity. No novel "prefer A when X, prefer B when Y" decision rule emerges from this work beyond what SageAttention2 already established—the contribution is that the "quantized" option is now strictly faster than before at no additional accuracy cost.