Sigma: Differential Rescaling of Query, Key and Value for Efficient Language Models

🎯 Pitch

SIGMA reveals a striking asymmetry: compressing Value vectors hurts model quality far more than compressing Key vectors, yet standard attention treats them identically. By aggressively shrinking Keys while preserving Values and expanding Query heads, SIGMA achieves up to 33% faster inference than grouped-query attention without accuracy loss—a free lunch hidden in plain sight in every transformer.

1. Executive Summary

This paper introduces SIGMA, an efficient language model specialized for the system domain that employs a novel DiffQKV attention mechanism to differentially optimize the Query, Key, and Value components of self-attention. DiffQKV attention consists of two complementary techniques—differentially compressed KV (aggressively compressing Key heads and dimensions while preserving Value heads, based on the finding that model performance is significantly more sensitive to Value compression than Key compression) and augmented Q (expanding the Query head dimension with extra parameters to boost representational capacity with minimal inference cost since Query vectors are not cached)—which together achieve up to a 33.36% inference speed improvement over conventional grouped-query attention in long-context scenarios. Pre-trained on 6T tokens including 19.5B tokens of carefully collected system domain data, SIGMA matches state-of-the-art general-domain models at comparable scales while surpassing GPT-4 by up to 52.5% across all tasks on AIMICIUS, the first comprehensive system-domain benchmark introduced in this work, establishing that differential QKV rescaling can substantially improve inference efficiency without sacrificing model quality only when the disproportionate sensitivity of Value vectors to compression is explicitly accounted for in the architecture design.

2. Context and Motivation

The Core Problem: KV Cache Is the Inference Bottleneck, But We Compress It Naively

The central problem this paper addresses is the KV cache bottleneck in decoder-only transformer language models, and more specifically, the observation that existing approaches to mitigating this bottleneck treat Key and Value vectors as if they have identical properties—when in fact they serve fundamentally different roles in attention and exhibit dramatically different sensitivities to compression.

Let's unpack why this matters. During autoregressive decoding, every transformer layer must compute attention between the current query token and all previous tokens. To avoid recomputing keys and values for the entire prefix at each step, the standard practice (Pope et al., 2023) is to store these Key (K) and Value (V) vectors in a KV cache. For a model with $L$ layers, $n_h$ attention heads per layer, head dimension $d_h$ , and sequence length $s$ , the KV cache requires storing:

$L \times s \times n_h \times d_h \times 2 \text{ (for K and V)} \times 2 \text{ bytes (FP16)}$

This linearly scaling memory footprint creates two critical problems:

Memory capacity bottleneck: The cache can consume substantial GPU memory (Pope et al., 2023), practically limiting the maximum context length the model can process. For a 70B model with 8K context, the KV cache alone can require over 100GB of memory.
Memory bandwidth bottleneck: Even if the cache fits in memory, repeatedly loading K and V vectors from the cache to perform attention at each decoding step places substantial demands on memory bandwidth. As Shazeer (2019) and Ribar et al. (2024) have established, memory bandwidth—not computation—is the primary bottleneck for LLM inference speed.

The authors frame this concisely in Section 1:

"KV cache is a common technique in the decoder-only Transformer architecture (Radford et al., 2019), which stores Key and Value vectors for future reuse in decoding. It can consume considerable GPU memory (Pope et al., 2023) and place substantial demands on memory bandwidth (Shazeer, 2019; Ribar et al., 2024)."

The Gap: Why Treating K and V Identically Is a Mistake

Prior approaches to reducing KV cache overhead almost universally apply the same compression strategy to both K and V vectors. The paper identifies this as a critical blind spot. To understand why, we need to examine what K and V actually do in attention:

Key vectors are used to compute attention scores ( $\text{softmax}(QK^T/\sqrt{d_k})$ ). Their role is to determine which tokens to attend to—a routing function. The paper notes that these attention matrices exhibit "remarkable sparsity exceeding 95% during inference" (Zhang et al., 2024b), meaning most token pairs receive near-zero attention. A rough approximation of the attention distribution can still route queries to the right values.
Value vectors are the information carriers. After attention scores determine where to attend, V vectors provide what information to aggregate into the output ( $\text{Attention}(Q,K,V) = \text{softmax}(QK^T/\sqrt{d_k})V$ ). They directly influence the final attention output, so any compression or approximation error in V directly degrades the output quality.

Despite these fundamentally different roles, existing compression methods treat K and V identically. The paper's core insight (developed through the ablation studies in Section 2) is that model performance is significantly more sensitive to compression of V than of K—both in terms of head count and head dimension. This asymmetry means that uniformly compressing both wastes representational capacity on K (which can tolerate more aggressive compression) and risks damaging V (which needs to be preserved more carefully). The existing literature simply hadn't empirically characterized this asymmetry before.

Where Existing Approaches Fall Short

The paper surveys prior work on KV cache optimization (Appendix A) and identifies three broad categories, each with limitations:

1. Architecture-level approaches (MQA, GQA): Compressing heads uniformly. Multi-Query Attention (MQA; Shazeer, 2019) takes the extreme approach of using a single K head and a single V head for all Q heads. Grouped-Query Attention (GQA; Ainslie et al., 2023) generalizes this by using an intermediate number of shared K and V heads. Both methods reduce the number of K and V heads to the same extent—they always maintain $n_k = n_v$ . The paper's DiffQKV attention is a direct generalization: it removes this equality constraint, allowing $n_k \neq n_v$ . The key empirical finding that justifies this relaxation is in Table 1: reducing K heads from 16 to 4 (75% compression) causes a negligible 0.17-point performance drop on GQA, while reducing V heads by the same amount causes a 0.38-point drop. At more extreme compression (from 4 to 1), the gap widens: K head reduction barely affects performance (+0.01 point), while V head reduction causes a 0.63-point drop. The authors note:

"reducing K heads has a minor impact compared to reducing V heads in most cases. This differential impact is reasonable, considering the distinct roles of K and V within the attention mechanisms."

This empirical finding is the foundation for the entire paper—prior work simply hadn't measured this asymmetry, so it didn't exploit it.

2. Post-training KV eviction methods: Evicting K and V tokens together. A second family of approaches selectively evicts KV cache entries at test time based on attention scores (Beltagy et al., 2020; Xiao et al., 2023; Han et al., 2024; Liu et al., 2024a; Zhang et al., 2024b; Ge et al., 2024; Ribar et al., 2024). These methods identify "important" tokens and keep their full K and V vectors while discarding the rest. The paper's critique is implicit but clear: since attention scores are sparse, only a small fraction of V vectors contribute meaningfully to the output, yet eviction methods still cache both K and V for the kept tokens. The paper proposes an orthogonal optimization—selective V cache fetching (Appendix B.2)—where only the V vectors corresponding to the highest attention scores are loaded during inference. Table 8 shows that using the top-100 V vectors preserves performance within 0.01–0.30 points, indicating that V retrieval can be decoupled from K retrieval. This is possible because the attention scores themselves (computed from Q and K) already tell us which V vectors matter, before we load them.

3. Prompt compression methods: Not addressing the core architecture. Methods like gisting (Mu et al., 2024) and LongLLMLingua (Jiang et al., 2023b) compress the prompt into fewer tokens to reduce KV cache size. These are complementary to architectural improvements but don't address the fundamental asymmetry between K and V within the attention mechanism itself.

4. System-level optimizations: Layer-agnostic about K vs. V. Frameworks like vLLM's paged attention (Kwon et al., 2023) improve memory management but treat K and V as interchangeable. They don't exploit the differential compressibility that this paper characterizes.

The common thread: none of these approaches questions whether K and V should be treated identically. The paper's contribution is the empirical demonstration that they should not be, and the proposal of a concrete architecture that exploits this asymmetry.

The Underexplored Role of Augmented Q

A second gap the paper identifies is the neglect of Query vector optimization in efficiency-focused architectures. Q vectors are not cached during inference—they are computed fresh at each decoding step and discarded after the attention output is produced. This means that adding parameters to Q has minimal impact on memory footprint and data transfer, unlike adding parameters to K or V which directly increase cache size.

Yet existing efficiency architectures like GQA focus exclusively on reducing KV parameters. The paper argues that this is a missed opportunity: if Q can be augmented with extra parameters to improve representational capacity, the additional inference cost is marginal (only computation, not memory or data transfer), and the performance gains may partially offset the quality degradation from KV compression.

Observation 3 in Section 2 shows that augmenting Q (increasing its intermediate dimension) consistently improves performance: on GQA with $n_k = n_v = 16$ , adding AugQ with $d_h^q = 3072$ ( $1.5\times$ the hidden dimension) improves the 9-benchmark average from 52.14 to 53.38 (+1.24 points). On the more compressed GQA with $n_k = n_v = 4$ , augmentation provides an even larger boost: 51.66 → 53.13 (+1.47 points). This suggests that AugQ and KV compression are complementary: the more you compress KV, the more valuable the extra Q capacity becomes.

Observation 4 further shows that AugQ is more parameter-efficient than expanding the Feed-Forward Network (FFN): adding $\delta$ parameters to Q improves performance more than adding $\delta$ to FFN (+1.24 vs. +1.12 on the GQA baseline), and adding $\delta$ to Q is competitive with adding $2\delta$ to FFN (+1.24 vs. +1.02). This is a non-obvious finding that the paper presents without deep mechanistic explanation, but the empirical pattern is clear.

Why This Problem Matters: Practical Deployment and the System Domain

The paper motivates its work along two dimensions that converge on a practical need:

Efficiency for LLM deployment at scale. As LLMs are deployed in production, the cost of inference—dominated by KV cache memory and bandwidth—becomes the primary economic constraint. The paper notes that smaller, efficient models have "garnered increasing research interest due to their significantly reduced inference costs and lower deployment requirements" (Appendix A). The DiffQKV approach promises to push this efficiency frontier by squeezing more performance out of a given parameter and cache budget.

The system domain as an underexplored application area. The paper identifies a novel research direction called the "system domain"—which it defines as "leveraging AI models to autonomously validate, evaluate, diagnose, and optimize the key components of AI infrastructure (e.g., hardware, configurations, cloud services, databases, and workloads)" (Section 1). The motivation is both practical and strategic: if LLMs can automate AI infrastructure management, they can accelerate their own development. The paper explicitly connects this to self-improvement:

"These features would allow the LLMs to oversee the training processes of neural models, even including their own, and support automated optimization." (Section 4)

Despite this potential, the system domain "has yet to receive commensurate attention" (Section 1). The AIMICIUS benchmark is proposed as the first comprehensive evaluation suite for this domain, covering command generation (CMDGen), infrastructure benchmarking (Infrawise), network topology optimization (Optiflow), and natural-language-to-KQL translation (NL2KQL). The specific practical need is for models that can handle long, structured technical contexts (e.g., configuration files, system logs, hardware specifications) while maintaining efficiency—exactly the regime where DiffQKV's KV cache savings provide the greatest benefit.

Reconciling Scale and Efficiency Goals

The paper positions itself at the intersection of two research trajectories that are often in tension. On one side, scaling laws (Kaplan et al., 2020) and emergent capabilities motivate ever-larger models. On the other side, practical deployment demands smaller, faster models. The paper's framing in Appendix A makes this explicit:

"while the exploration into scaling up a larger model scale to achieve even more advanced level of intelligence is still ongoing..., the development of smaller, more efficient language models has also garnered increasing research interest due to their significantly reduced inference costs"

DiffQKV attention is positioned as a contribution to the efficiency trajectory that does not simply reduce parameters but reallocates them—taking capacity away from K (which can tolerate it) and adding capacity to Q (where it provides disproportionate benefit). This is a more nuanced optimization than uniform compression or uniform scaling.

How the Paper Positions Itself Relative to Prior Work

The paper's positioning is structured around a clear generalization hierarchy that it constructs in Appendix B.1 and Figure 3:

MHA (Vaswani et al., 2017) is the base case: $n_q = n_k = n_v$ and $d_q = d_k = d_v$ .
MQA (Shazeer, 2019) constrains $n_k = n_v = 1$ , but still maintains $d_q = d_k = d_v$ .
GQA (Ainslie et al., 2023) relaxes to $1 < n_k = n_v < n_q$ , still with $d_q = d_k = d_v$ .
DiffQKV generalizes fully: $n_k$ and $n_v$ can differ ( $n_k \neq n_v$ ), and $d_k \neq d_v$ is also possible. Additionally, $d_q$ can be larger than $d_k$ and $d_v$ (augmented Q).

The paper is not proposing an entirely new attention paradigm—it is showing that the constraint $n_k = n_v$ in prior work is an unnecessary restriction that, when lifted and combined with augmented Q, yields substantial efficiency gains. The differential compressibility of K vs. V is the empirical justification for lifting this constraint.

A crucial practical contribution is FlexHeadFA (Section 3.2), a modification to FlashAttention2 that supports attention computation with different numbers of K and V heads. The standard FlashAttention only supports $n_k = n_v$ , which is why DiffQKV couldn't be deployed efficiently before this work. The FlexHeadFA kernel modifies the address calculation:

$\text{idx}_i = \text{idx}_q \cdot \frac{n_h^i}{n_h^q}, \quad i \in \{k, v\}$

This separates the head indexing for K and V, removing the constraint that they must be equal. The paper also provides a compatibility workaround (kv_group_sharing in Appendix E) for frameworks that combine K and V into a single matrix, though it notes this "is highly discouraged, as it negates the efficiency improvements entirely by degrading to GQA."

In summary, the paper's contribution is not a radical departure from attention, but a careful empirical characterization of K vs. V compressibility that unlocks a more efficient architecture configuration—one that prior work couldn't explore because the necessary kernel support (FlexHeadFA) didn't exist, and because the empirical evidence for differential compressibility hadn't been systematically collected.

3. Technical Approach

3.1 Reader Orientation

SIGMA is a transformer language model whose self-attention mechanism is redesigned so that the Query, Key, and Value components each get different numbers of heads and different head dimensions, rather than being forced to share the same configuration. The problem this solves is that existing efficient attention architectures (like Grouped-Query Attention) apply the same compression to Key and Value vectors, even though Key vectors primarily route attention and can tolerate aggressive compression, while Value vectors directly contribute to the output and degrade sharply when compressed. The "shape" of the solution is a two-part modification: (1) shrink the Key representation far more than the Value representation, and (2) expand the Query representation with extra parameters whose cost is negligible at inference time since Queries are never cached.

3.2 Big-Picture Architecture (Diagram in Words)

The SIGMA architecture modifies the standard transformer decoder block at the self-attention layer. There are four major components that differ from a standard Multi-Head Attention (MHA) or Grouped-Query Attention (GQA) block:

1. Projection layers with asymmetric output dimensions. The input hidden states are projected to Q, K, and V through separate weight matrices, but now these projections produce tensors with different numbers of heads and different per-head dimensionalities. Specifically, Q gets 32 heads with an augmented intermediate dimension (e.g., $d_h^q = 3072$ for the 1.5B model, which is $1.5\times$ the hidden dimension of 2048), K gets only 4 heads with dimension $d_h^k = 64$ , and V gets 16 heads with dimension $d_h^v = 64$ . This is the core structural difference: $n_q \neq n_k \neq n_v$ and $d_q \neq d_k = d_v$ .

2. FlexHeadFA attention kernel. Because standard FlashAttention2 requires $n_k = n_v$ and requires $n_q$ to be an integer multiple of both, the paper implements a modified kernel called FlexHeadFA. It generalizes the head-indexing logic: given a query head index $\text{idx}_q$ , the corresponding key and value head indices are computed as $\text{idx}_i = \text{idx}_q \cdot n_h^i / n_h^q$ for $i \in \{k, v\}$ . This allows attention computation when K has 4 heads and V has 16 heads while Q has 32 heads, without padding or duplicating.

3. Differential KV cache management. Since $n_k = 4$ and $n_v = 16$ , the key cache is only 25% the size of the value cache. The total KV cache size for a sequence of length $s$ is proportional to $s \cdot (n_k \cdot d_h^k + n_v \cdot d_h^v) = s \cdot (4 \cdot 64 + 16 \cdot 64) = s \cdot 1280$ elements, versus GQA's $s \cdot (16 \cdot 64 + 16 \cdot 64) = s \cdot 2048$ elements—a 37.5% reduction. The paper implements separate cache storage for K and V to accommodate their different sizes.

4. Augmented Q with SwiGLU gating. The Q projection is not a simple linear layer. It uses a gated structure: the hidden state passes through two parallel projections (q_gate_proj and q_up_proj), their outputs are multiplied element-wise after applying the SwiGLU activation to the gate, and the result is projected down to the target dimension through q_down_proj. This gated mechanism gives Q additional representational capacity while the extra parameters are never cached.

The information flow through a single decoder layer is: input hidden states → separate projections to asymmetric Q/K/V → FlexHeadFA computes attention using the expanded Q heads and compressed K heads, selectively loading V heads based on attention scores → attention output projected back → standard FFN → next layer. At inference time, the K and V tensors for each layer are stored in caches of different sizes that grow with sequence length, and the attention kernel loads them asymmetrically.

3.3 Roadmap for the Deep Dive

First, the DiffQKV attention formulation — the generalized mathematical definition that allows $n_k \neq n_v$ and $d_q \neq d_k \neq d_v$ , including the GroupSharing mechanism that maps between different head counts and the modified Attend function that handles different head dimensionalities. This is the formal foundation everything else builds on.
Second, the four empirical observations from 1B-scale training runs — the 100B-token experiments on FineWeb-Edu that characterize (1) differential sensitivity of K vs. V head reduction, (2) the negligible impact of halving K head dimension, (3) the beneficial effect of augmenting Q, and (4) the comparison between Q augmentation and FFN expansion. These observations are the empirical justification for the final architecture choices.
Third, the selective V cache fetching technique — an orthogonal optimization that exploits attention score sparsity to load only a small subset of V vectors during inference, further reducing memory bandwidth without architectural modification.
Fourth, the SIGMA model architecture specification — the concrete configurations for the 1.5B and 10B parameter models, including all hyperparameters (layers, hidden dimensions, head counts, FFN dimensions, vocabulary size, RoPE theta).
Fifth, the FlexHeadFA kernel and deployment infrastructure — the implementation details that make the asymmetric architecture executable on GPU hardware, including the modified address calculation, compatibility with FlashAttention2's split/combine kernel structure, and the kv_group_sharing compatibility wrapper.
Sixth, the training data and pre-training procedure — the 6T token data mixture, the multi-phase pre-training schedule with changing data ratios and learning rates, and the system domain data collection pipeline.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an architecture design and empirical analysis paper whose core idea is that the three components of self-attention (Query, Key, Value) have fundamentally different sensitivities to compression and should be rescaled differentially—Key compressed aggressively, Value preserved carefully, and Query augmented with extra parameters—to maximize efficiency without sacrificing model quality.

DiffQKV Attention: Generalized Formulation

The DiffQKV attention mechanism is a strict generalization of Multi-Head Attention (MHA), Multi-Query Attention (MQA), and Grouped-Query Attention (GQA). The key generalization is removing the constraints that $n_k = n_v$ and that $d_q = d_k = d_v$ . The authors position existing mechanisms as special cases within this generalized framework (Appendix B.1, Figure 3):

MHA: $n_q = n_k = n_v$ and $d_q = d_k = d_v$
MQA: $n_k = n_v = 1$ , with $d_q = d_k = d_v$
GQA: $1 < n_k = n_v < n_q$ , with $d_q = d_k = d_v$
DiffQKV: $n_k$ and $n_v$ can differ, and $d_k \neq d_v$ is allowed, with $d_q$ potentially larger than both

Step 1: Projection to asymmetric Q, K, V. Given input hidden state $h_t \in \mathbb{R}^{d_m}$ at position $t$ , three learned projection matrices produce Q, K, V tensors with different shapes:

$q_t = h_t \cdot W_Q, \quad k_t = h_t \cdot W_K, \quad v_t = h_t \cdot W_V$

where $q_t \in \mathbb{R}^{n_h^q \times d_h^q}$ is the query tensor for the current token, $k_t \in \mathbb{R}^{n_h^k \times d_h^k}$ is the key tensor, $v_t \in \mathbb{R}^{n_h^v \times d_h^v}$ is the value tensor, $n_h^q$ is the number of Q heads (32 in SIGMA), $n_h^k$ is the number of K heads (4 in SIGMA), $n_h^v$ is the number of V heads (16 in SIGMA), and $d_h^q, d_h^k, d_h^v$ are the per-head dimensions (3072, 64, 64 respectively for the 1.5B model).

Step 2: Building the full K and V sequences from cache. At decoding step $t$ , the model needs attention over all previous positions. The new K and V vectors are appended to the cache:

$K_t = [K_{t-1}; k_t], \quad V_t = [V_{t-1}; v_t]$

where $K_t \in \mathbb{R}^{t \times n_h^k \times d_h^k}$ is the full key sequence up to position $t$ , and $V_t \in \mathbb{R}^{t \times n_h^v \times d_h^v}$ is the full value sequence. The semicolon denotes concatenation along the time dimension. Because $n_h^k \neq n_h^v$ , these two caches have different sizes—the K cache is much smaller than the V cache.

Step 3: GroupSharing to align head counts. Since $n_h^q$ (32) is not equal to $n_h^k$ (4) or $n_h^v$ (16), the attention computation needs a mapping between head indices. The GroupSharing operation handles this:

$[K_{(t,1)}; K_{(t,2)}; ...; K_{(t,n_h^q)}] = \text{GroupSharing}(K_t, n_h^q, n_h^k)$

$[q_{(t,1)}; q_{(t,2)}; ...; q_{(t,n_h^q)}] = q_t$

where $K_{(t,i)} \in \mathbb{R}^{t \times d_h^k}$ is the K slice assigned to the $i$ -th Q head, and $q_{(t,i)} \in \mathbb{R}^{d_h^k}$ is the $i$ -th Q head (the paper defines $q_{(t,i)} \in \mathbb{R}^{d_h^k}$ because Q and K must share the same dimension for dot-product attention, which requires transformation when $d_h^q \neq d_h^k$ —see Step 4).

What GroupSharing does operationally: it maps $n_h^q$ query heads onto $n_h^k$ key heads by assigning multiple query heads to each key head. In the SIGMA configuration with $n_h^q = 32$ and $n_h^k = 4$ , each K head is shared by $32/4 = 8$ Q heads. The $i$ -th Q head uses the K head at index $\lfloor i \cdot n_h^k / n_h^q \rfloor = \lfloor i \cdot 4 / 32 \rfloor = \lfloor i/8 \rfloor$ . Similarly for V: with $n_h^q = 32$ and $n_h^v = 16$ , each V head is shared by $32/16 = 2$ Q heads, with the $i$ -th Q head using V head $\lfloor i \cdot 16 / 32 \rfloor = \lfloor i/2 \rfloor$ .

This asymmetric sharing is the core structural innovation: K heads are shared more aggressively (8:1 ratio) than V heads (2:1 ratio), reflecting the finding that K compression is less harmful than V compression.

Step 4: Handling differing Q and K dimensions (Attend function). The standard attention score computation is:

$\alpha_{(t,i)} = \text{Softmax}\left[\frac{\text{Attend}(q_{(t,i)}^T, K_{(t,i)})}{\sqrt{d_k}}\right]$

where $\alpha_{(t,i)} \in \mathbb{R}^t$ is the attention weight vector for the $i$ -th head, and Attend is the inner product in standard MHA. However, when $d_h^q \neq d_h^k$ (as in the augmented Q case), a direct dot product is impossible because the vectors have different lengths. The paper specifies:

"As DiffQKV accommodates varying dimensions for the Q and K heads, an alternative implementation of the Attend function is necessitated. In our experiments, we implement this part by transforming the dimension of K to the same as V through another feed-forward layer."

So the actual computation is: project K from dimension $d_h^k$ to $d_h^q$ (the current Q head dimension) using a trainable feed-forward layer, then compute the standard dot product. The exact dimension of this projection depends on the configuration—in SIGMA's final architecture with $d_h^q = 3072$ and $d_h^k = 64$ , the projection transforms K from 64-dimensional to 3072-dimensional before the dot product with Q.

Step 5: GroupSharing for V and output computation. After attention weights are computed, the V cache is loaded and GroupShared:

$[V_{(t,1)}; V_{(t,2)}; ...; V_{(t,n_h^q)}] = \text{GroupSharing}(V_t, n_h^q, n_h^v)$

where $V_{(t,i)} \in \mathbb{R}^{t \times d_h^v}$ is the V slice for the $i$ -th Q head. Each attention output is:

$o_{(t,i)} = \sum_{j=1}^t \alpha_{(j,i)} V_{(j,i)}$

where $o_{(t,i)} \in \mathbb{R}^{d_h^v}$ is the weighted sum of V vectors for the $i$ -th head. Finally, all head outputs are concatenated and projected:

$x_t = W_O [o_{(t,1)}; o_{(t,2)}; ...; o_{(t,n_h^q)}]$

where $x_t \in \mathbb{R}^{d_m}$ is the attention layer output and $W_O$ projects from the concatenated head outputs back to the model dimension.

Why this formulation matters: the generalized DiffQKV formulation decouples three previously tied degrees of freedom—the number of K heads, the number of V heads, and the Q dimension—allowing them to be optimized independently based on their different roles in attention. The standard GQA constraint $n_k = n_v$ is a special case; removing it creates a larger design space within which efficiency can be improved by allocating parameters where they matter most (V for output quality, Q for representation capacity) and removing them where they matter least (K for routing).

The paper also notes that the LoadCacheV function "can differ from the strategy for loading K cache" (Section 2), which enables the selective V cache fetching optimization described later—loading only a subset of V vectors based on attention scores, exploiting the sparsity already present in $\alpha_{(t,i)}$ .

Observation 1: Differential Sensitivity of K vs. V Head Count

The paper trains ~1B parameter models (22 layers, hidden dimension 2048) on 100B tokens from FineWeb-Edu and measures how reducing K heads vs. V heads affects model performance. The baseline configurations and their modifications are shown in Table 1 (and its expanded version, Table 9 in Appendix B.3).

Experimental design. Starting from three baseline configurations—MHA ( $n_k = n_v = 32$ ), GQA ( $n_k = n_v = 16$ ), and GQA ( $n_k = n_v = 4$ )—the authors independently reduce either $n_k$ or $n_v$ while keeping the other and $n_q = 32$ fixed. Performance is measured as the average score across nine benchmarks: HellaSwag, OpenBookQA, WinoGrande, ARC Challenge, PIQA, SciQ, BoolQ, LogiQA, and LAMBADA.

Results for MHA baseline ( $n_k = n_v = 32$ , average score 52.40):

Reducing V heads by 50% ( $n_v = 16$ , K unchanged at 32): 51.74, a drop of 0.66 points.
Reducing K heads by 50% ( $n_k = 16$ , V unchanged at 32): 52.83, an increase of 0.43 points.

The K-reduced model actually performs better than the baseline. This is a striking result—removing 50% of K heads not only doesn't hurt, but helps. The authors don't explain this improvement mechanistically, but a plausible interpretation is that having 32 K heads with only 16 Q heads creates redundant K capacity; reducing to 16 K heads produces a 2:1 sharing ratio that may be better matched to the information actually needed.

Results for GQA baseline ( $n_k = n_v = 16$ , average 52.14):

Reducing V heads by 75% ( $n_v = 4$ ): 51.76, a drop of 0.38 points.
Reducing K heads by 75% ( $n_k = 4$ ): 51.97, a drop of only 0.17 points.

The gap is $0.38 - 0.17 = 0.21$ points—the V reduction hurts more than twice as much as the K reduction.

Results for GQA baseline ( $n_k = n_v = 4$ , average 51.66):

Reducing V heads by 75% ( $n_v = 1$ ): 51.03, a drop of 0.63 points.
Reducing K heads by 75% ( $n_k = 1$ ): 51.67, an increase of 0.01 points.

At the extreme, reducing V to a single head causes substantial damage while reducing K to a single head has essentially no effect. The asymmetry grows more pronounced at higher compression ratios.

Mechanistic explanation provided by the paper:

"The K heads primarily serve to compute attention matrices, which prove to have a remarkable sparsity exceeding 95% during inference (Zhang et al., 2024b), so a slight decrease in the parameters used for its calculation can still yield a precise approximation. In contrast, the V vectors, which directly influence the final attention output, demands a more nuanced approach."

In operational terms: K heads determine which past tokens get attended to. Since attention distributions are highly sparse (95%+ of attention weights are near zero), a coarse-grained K head with fewer parameters can still identify the right sparse subset of tokens. V heads determine what information those tokens contribute to the output. Compressing V heads means aggregating information from different tokens through a bottleneck, which directly reduces the information content of the attention output regardless of whether the routing (K) was correct.

Why this is differentially exploitable: K compression can be aggressive because the attention sparsity provides a natural robustness—you only need to get the largest attention weights approximately right. V compression must be conservative because every V head contributes linearly to the output, and any loss of representational diversity in V directly degrades the output.

Observation 2: K Head Dimension Can Be Halved with Negligible Impact

Beyond reducing K head count, the paper also explores reducing K head dimension—making each K head narrower. This is tested by halving $d_h^k$ relative to $d_h^v$ and inserting a trainable feed-forward layer to project K up to the Q dimension for dot-product computation.

Experimental design (Table 2, expanded in Table 10). Three baseline configurations are compared: MHA ( $n_k = n_v = 32$ ), GQA ( $n_k = n_v = 16$ ), and GQA ( $n_k = n_v = 4$ ). For each, a variant is trained where $d_h^k = d_h^v / 2$ .

Results:

MHA: 52.40 → 52.56 with half K dim (+0.16)
GQA ( $n_k = n_v = 16$ ): 52.14 → 52.06 with half K dim (−0.08)
GQA ( $n_k = n_v = 4$ ): 51.66 → 51.92 with half K dim (+0.26)

In all three cases, the performance change is negligible (magnitude ≤ 0.26 points), and in two out of three cases it is actually positive. The paper interprets this as:

"the performance decrease from reducing K head dimension is negligible; in some cases, the model with a smaller K head dimension even outperforms the baseline setting prior to compression."

The cost trade-off. Reducing K head dimension is not free—it adds a feed-forward layer to project K up to the Q dimension for the dot product. However, this computation is cheap compared to the memory savings. The paper states:

"the reduction of K head dimension reduces KV cache at the cost of additional computation from a feed-forward layer. This trade-off is cost-effective as the primary bottleneck for LLM inference speed lies in memory consumption rather than computation."

The memory savings are straightforward: the K cache size is proportional to $n_h^k \cdot d_h^k$ , so halving $d_h^k$ halves the K cache contribution. In SIGMA's final architecture, this dimension compression is not applied (only head count reduction is used for the production models), but the finding establishes a design principle: if memory bandwidth is the bottleneck, trading a small amount of extra computation for reduced cache size is favorable.

Why this works: K vectors encode positional and content-based addressing information. The high sparsity of attention distributions means that the effective rank of the K representation needed for accurate routing is likely much lower than the full head dimension. Reducing $d_h^k$ essentially projects K into a lower-dimensional subspace that captures the dominant addressing patterns, and the feed-forward projection reconstructs a higher-dimensional K for compatibility with Q.

Observation 3: Augmented Q Consistently Improves Performance

The third empirical finding is that expanding the Q head dimension (adding parameters to Q) provides a consistent performance boost, with larger gains when KV is more aggressively compressed. Since Q vectors are not cached, the additional parameters incur only computation cost (not memory or bandwidth).

Experimental design (Table 3, expanded in Table 11). Starting from three baselines—MHA ( $n_k = n_v = 32$ ), GQA ( $n_k = n_v = 16$ ), and GQA ( $n_k = n_v = 4$ )—the authors increase the Q intermediate dimension $d_h^q$ . For reference, the baseline (without augmentation) has $d_h^q = 2048$ , equal to the hidden dimension. Multiple augmentation sizes are tested for GQA with $n_k = n_v = 16$ : $d_h^q = 3072$ (1.5×), $d_h^q = 4096$ (2×), and $d_h^q = 5632$ (2.75×). For MHA and GQA with $n_k = n_v = 4$ , only $d_h^q = 5632$ is tested.

Results:

MHA + AugQ ( $d_h^q = 5632$ ): 52.40 → 53.03 (+0.63)
GQA ( $n_k = n_v = 16$ ) + AugQ ( $d_h^q = 3072$ ): 52.14 → 53.38 (+1.24)
GQA ( $n_k = n_v = 16$ ) + AugQ ( $d_h^q = 4096$ ): 52.14 → 52.93 (+0.79)
GQA ( $n_k = n_v = 16$ ) + AugQ ( $d_h^q = 5632$ ): 52.14 → 53.07 (+0.93)
GQA ( $n_k = n_v = 4$ ) + AugQ ( $d_h^q = 5632$ ): 51.66 → 53.13 (+1.47)

Several patterns emerge. First, a moderate augmentation of 1.5× ( $d_h^q = 3072$ ) gives the best gain per parameter—it achieves +1.24 with fewer parameters than larger augmentations that give smaller or equal gains. This saturating (and somewhat non-monotonic) pattern suggests an optimal augmentation ratio around 1.5× to 2×. Second, the gain is larger when KV is more compressed: +1.47 for $n_k = n_v = 4$ vs. +0.63 for MHA. This is consistent with the interpretation that AugQ compensates for the representational capacity lost through KV compression.

Why Q augmentation is efficient at inference time: Q vectors are computed fresh at each decoding step, used for the attention computation, and immediately discarded. They never enter the KV cache. Therefore, expanding Q's parameter count increases the computation per step (more FLOPs in the Q projection) but has zero impact on KV cache memory consumption and zero impact on memory bandwidth for cache loading. In the bandwidth-limited regime that characterizes LLM inference, this computation-only cost is highly favorable.

Observation 4: Augmented Q Outperforms FFN Expansion Per Parameter

To establish that Q augmentation is not just "adding parameters anywhere helps," the paper compares adding the same number of parameters to Q versus adding them to the Feed-Forward Network (FFN). The FFN is the other major parameter consumer in a transformer layer.

Experimental design (Table 4, expanded in Table 12). The baseline is GQA with $n_k = n_v = 16$ and FFN dimension 5632 (average score 52.14). The unit of parameter addition is $\delta = 3072$ . The paper tests:

AugF ( $\Delta d_F = \delta$ ): enlarge FFN dimension by 3072
AugQ ( $d_h^q = \delta$ ): set Q intermediate dimension to 3072
AugF ( $\Delta d_F = 2\delta$ ): enlarge FFN by 6144
AugF ( $\Delta d_F = \delta$ ) & AugQ ( $d_h^q = \delta$ ): both
AugF ( $\Delta d_F = 3\delta$ ): enlarge FFN by 9216
AugF ( $\Delta d_F = 2\delta$ ) & AugQ ( $d_h^q = \delta$ )
AugF ( $\Delta d_F = 5\delta$ ): enlarge FFN by 15360
AugF ( $\Delta d_F = 3\delta$ ) & AugQ ( $d_h^q = \delta$ )

Results:

Baseline GQA: 52.14
- AugF ( $\delta$ ): 53.26 (+1.12)
- AugQ ( $\delta$ ): 53.38 (+1.24)
- AugF ( $2\delta$ ): 53.16 (+1.02)
- AugF ( $\delta$ ) & AugQ ( $\delta$ ): 54.55 (+2.41)
- AugF ( $3\delta$ ): 54.50 (+2.36)
- AugF ( $2\delta$ ) & AugQ ( $\delta$ ): 54.67 (+2.53)
- AugF ( $5\delta$ ): 55.08 (+2.94)
- AugF ( $3\delta$ ) & AugQ ( $\delta$ ): 55.09 (+2.95)

Key comparison: At equal parameter addition ( $\delta$ each), AugQ provides +1.24 vs. AugF's +1.12—a small but consistent advantage for Q. More tellingly, AugQ with $\delta$ parameters (53.38) is competitive with AugF with $2\delta$ parameters (53.16)—AugQ with half the parameters matches FFN expansion. The paper states:

"augmenting Q consistently gives a more significant performance boost than adding the same number of parameters to the FFN module... even when FFN's additional parameters ( $2\delta$ ) are double those of Q ( $d_h^q = \delta$ ), the model performance is still slightly worse with FFN."

How to read the additive pattern: The gains from AugQ and AugF appear approximately independent. AugQ ( $\delta$ ) alone gives +1.24; AugF ( $\delta$ ) alone gives +1.12; together they give +2.41, which is close to $1.24 + 1.12 = 2.36$ . Similarly, AugF ( $3\delta$ ) gives +2.36 and AugF ( $3\delta$ ) + AugQ ( $\delta$ ) gives +2.95, close to $2.36 + 1.24 = 3.60$ but showing some sub-additivity. This approximate independence means the two mechanisms improve different aspects of model capacity and can be combined.

Why Q augmentation is more parameter-efficient than FFN expansion: The paper does not provide a mechanistic explanation, but the likely reason relates to information bottleneck structure. The self-attention layer is the only place where the model can dynamically route information between token positions—the FFN operates on each position independently. Extra capacity in the attention mechanism (via Q) may enable more nuanced routing decisions, effectively improving how the model uses its existing FFN capacity, whereas adding FFN capacity improves position-wise computation but doesn't improve routing. In transformer architectures, attention quality is often the limiting factor, so investments there may yield disproportionate returns.

Selective V Cache Fetching

Beyond the architectural modifications to K and Q, the paper proposes an orthogonal inference-time optimization that exploits the sparsity of attention scores to reduce V cache loading. This technique is not part of the SIGMA architecture per se but is an additional efficiency lever that the asymmetric DiffQKV design makes possible.

The mechanism. After attention scores $\alpha_{(t,i)}$ are computed (using the full K cache to determine importance weights), only the V vectors corresponding to the highest attention scores are loaded from the V cache. The remaining V vectors are skipped entirely—their contribution to the weighted sum is treated as zero. Formally, for each head $i$ :

$o_{(t,i)} \approx \sum_{j \in \text{TopK}(\alpha_{(\cdot,i)})} \alpha_{(j,i)} V_{(j,i)}$

where $\text{TopK}$ selects the indices of the $k$ largest attention weights, with $k = 100$ in the paper's experiments.

Experimental validation (Table 8). Across three baseline configurations, the paper compares full V loading with top-100 selective loading:

MHA: 52.40 → 52.10 with Sel.V-top100 (−0.30)
GQA ( $n_k = n_v = 16$ ): 52.14 → 52.08 (−0.06)
GQA ( $n_k = n_v = 4$ ): 51.66 → 51.67 (+0.01)

In all cases, the performance change is small (magnitude ≤ 0.30 points), and in the most compressed case it is essentially zero. This confirms that the attention distribution is sufficiently sparse that only ~100 tokens carry meaningful V contributions, even for long sequences.

Why this matters operationally: The V cache is the largest component in SIGMA's design ( $n_v = 16$ vs. $n_k = 4$ ). Loading only $k$ V vectors out of $t$ total reduces the V cache bandwidth from $O(t \cdot n_v \cdot d_v)$ to $O(k \cdot n_v \cdot d_v + t \cdot n_k \cdot d_k)$ , where the $t \cdot n_k \cdot d_k$ term is the cost of loading the full thin K cache (which is needed anyway for attention score computation). Since $n_k$ is already 4× smaller than $n_v$ , and $k \ll t$ for long sequences, this compound optimization—aggressive K compression plus selective V loading—attacks the memory bandwidth bottleneck from both the K and V sides.

Implementation note: Selective V fetching requires the K cache to be fully loaded first (to compute attention scores and determine which V vectors are "important"), then only the high-scoring V vectors are fetched. This introduces a dependency between the K load and V load phases that isn't present when all V vectors are loaded unconditionally. The paper does not benchmark the latency of this two-phase loading pattern.

SIGMA Model Architecture

Based on the four observations, the paper designs two concrete model sizes: SIGMA-1.5B and SIGMA-10B. The architecture choices represent a specific point in the DiffQKV design space that balances efficiency and performance.

Configuration (Table 14):

Parameter	SIGMA-1.5B	SIGMA-10B
Layers	26	32
Hidden Dimension	2,048	4,096
FFN Dimension	6,144	14,336
Augmented Q Dimension ( $d_h^q$ )	3,072	6,144
Attention Heads ( $n_h^q$ )	32	32
Key Heads ( $n_h^k$ )	4	4
Value Heads ( $n_h^v$ )	16	16
Peak Learning Rate	4.0e-4	1.5e-4
Activation Function	SwiGLU	SwiGLU
Vocabulary Size	128,256	128,256
Positional Embeddings	RoPE ( $\theta = 50,000$ )	RoPE ( $\theta = 500,000$ )

Design rationale for each choice:

$n_h^k = 4$ , $n_h^v = 16$ : Based on Observation 1, K heads are compressed to 25% of V heads. The V head count of 16 is half the Q head count of 32, matching the standard GQA 2:1 sharing ratio. The K head count of 4 creates an 8:1 Q-to-K sharing ratio, which is more aggressive than standard GQA but justified by the negligible performance impact shown in Table 1 (0.17 point drop from 16→4 K heads for GQA baseline).
$d_h^q = 1.5 \times d_m$ : For the 1.5B model, $d_h^q = 3072 = 1.5 \times 2048$ . For the 10B model, $d_h^q = 6144 = 1.5 \times 4096$ . This 1.5× ratio is based on Observation 3, where $d_h^q = 3072$ (1.5× on a 2048-dim model) gave the best gain-per-parameter among the tested sizes.
No K dimension compression: The paper states: "for the sake of balancing the model performance and the cost of the KV cache, during the training of SIGMA-1.5B and SIGMA-10B, no dimension compression is applied to the K heads. Only the number of K heads was decreased." So $d_h^k = d_h^v = 64$ in both models. Observation 2 showed that halving $d_h^k$ was viable, but the paper chose not to deploy it in production models—likely because the head count reduction alone already achieves a 37.5% KV cache reduction, and further dimension compression adds implementation complexity (the K-to-Q projection layer) for diminishing returns.
SwiGLU activation: Both models use the SwiGLU activation function (Shazeer, 2020) rather than standard ReLU or GeLU. This is an independent architectural choice common in modern LLMs, not specifically motivated by the DiffQKV design.
RoPE position encoding with different theta: The 1.5B model uses $\theta = 50,000$ while the 10B model uses $\theta = 500,000$ . The larger theta extends the effective context length by reducing the rate at which rotary position embeddings decay, which is important for the larger model expected to handle longer sequences. This is a standard scaling practice not specific to DiffQKV.
Vocabulary: Both models use the Llama3 vocabulary of 128,256 tokens (Dubey et al., 2024), which is a practical choice for compatibility rather than a novel contribution.

Parameter accounting. The 1.5B model with augmented Q has more parameters in the attention layers than a standard GQA 1.5B model would, but fewer parameters in the K projection (4 heads × 64 dim vs. 16 heads × 64 dim). The net parameter count is approximately 1.5B—the augmentation and compression roughly balance in terms of total parameters, but the distribution has shifted from KV to Q, improving inference efficiency without reducing model capacity.

FlexHeadFA Kernel and Deployment Infrastructure

The DiffQKV architecture requires a modified attention kernel because standard FlashAttention2 (Dao, 2024) assumes $n_k = n_v$ and that $n_q$ is an integer multiple of both. The paper develops FlexHeadFA to remove these constraints, and also provides a compatibility wrapper for frameworks that combine K and V caches.

The FlashAttention2 kernel structure. FlashAttention2 splits the attention computation into two GPU kernels: flash_fwd_splitkv_kernel (the "split kernel") and flash_fwd_splitkv_combine_kernel (the "combine kernel"). The split kernel divides the K and V matrices into chunks, performs the QK multiplication and softmax for each chunk, and produces partial output chunks. The combine kernel aggregates these partial outputs into the final attention output.

The FlexHeadFA modification. In standard FlashAttention2, the address calculation for loading K and V heads assumes a single head index that applies to both. FlexHeadFA separates the address calculation:

$\text{idx}_i = \text{idx}_q \cdot \frac{n_h^i}{n_h^q}, \quad i \in \{k, v\}$

where $\text{idx}_q$ is the query head index (0 to $n_h^q - 1$ ), $\text{idx}_k$ is the key head index to load, and $\text{idx}_v$ is the value head index to load. In SIGMA-1.5B, for $\text{idx}_q = 0$ : $\text{idx}_k = 0 \cdot 4/32 = 0$ , $\text{idx}_v = 0 \cdot 16/32 = 0$ . For $\text{idx}_q = 1$ : $\text{idx}_k = 1 \cdot 4/32 = 0$ (still K head 0 because 8 Q heads share each K head), $\text{idx}_v = 1 \cdot 16/32 = 0$ (still V head 0 because 2 Q heads share each V head). For $\text{idx}_q = 2$ : $\text{idx}_k = 2 \cdot 4/32 = 0$ , $\text{idx}_v = 2 \cdot 16/32 = 1$ (switches to V head 1).

What this enables: The split kernel can load K head $\text{idx}_k$ and V head $\text{idx}_v$ independently based on $\text{idx}_q$ , without requiring them to be the same. The combine kernel is unaffected because it operates on the output chunks, not on the original K and V matrices.

The kv_group_sharing compatibility wrapper (Appendix E). Some LLM deployment frameworks (e.g., vLLM, TensorRT-LLM) combine K and V into a single matrix for efficiency, which requires $n_k = n_v$ . To maintain compatibility, the paper provides a workaround function that duplicates the smaller cache to match the larger one:

If $n_k > n_v$ : duplicate V heads to match K head count
If $n_k < n_v$ (the SIGMA case): duplicate K heads to match V head count

In SIGMA, this would replicate each K head 4 times (from 4 to 16) to match $n_v = 16$ , effectively degrading to GQA and losing the K compression efficiency. The paper explicitly warns: "this approach is highly discouraged, as it negates the efficiency improvements entirely by degrading to GQA." This wrapper exists purely for compatibility with unmodified frameworks; full efficiency requires frameworks that natively support different K and V head counts (i.e., FlexHeadFA).

Pre-training Data and Procedure

While the core technical contribution is the DiffQKV architecture, the paper also describes the pre-training procedure used to train SIGMA models from scratch, including data composition, quality filtering, and multi-phase training.

Pre-training data composition (6T tokens total):

The data mix evolves across training phases. The paper describes a four-phase curriculum:

Phase 1 (3.5T tokens): General:Math:Code = 8:1:1 ratio. Uses General Dataset I (4T tokens, from combining DCLM and FineWeb-EDU with deduplication), proof-pile-2 math data, and a code dataset filtered following StarCoderV2's method (500B tokens). Batch size starts at 4M tokens and scales up to 16M tokens. Learning rate peaks at 1.5e-4 for the 10B model (4.0e-4 for 1.5B).
Phase 2 (1.0T tokens): General:Math:Code = 4:3:3 ratio. Uses the quality-filtered General Dataset II (1T tokens, obtained by further filtering General Dataset I). The increased math and code proportions reflect a shift toward reasoning-heavy data in later training.
Phase 3 (1.0T tokens): Synthesized+rewritten data mixed with General Dataset II at a 6:4 ratio. The synthesized data (~1T tokens total) is multi-domain content that "passed quality screening." Learning rate reduced to 20% of peak (3e-5 for 10B, 8e-5 for 1.5B).
Phase 4—Annealing (remaining tokens to 6T): Uses General Dataset III (~200B tokens, the highest-scoring subset selected with stricter rules), the best synthesized and rewritten data, and the system domain data (19.5B tokens). Learning rate linearly decays to zero.

Hardware: SIGMA-1.5B trained on 512×A100-40G GPUs; SIGMA-10B trained on 256×H100-80G GPUs.

Why the phased curriculum matters: This is not a DiffQKV-specific contribution, but it contextualizes the evaluation results. The paper claims that SIGMA matches state-of-the-art models at comparable scales (Tables 7 and 17), but this is a joint product of the architecture and the curated 6T token training data. Without an ablation comparing DiffQKV vs. GQA on identical data, it's difficult to isolate how much of SIGMA's general-domain performance comes from the architecture versus the training recipe. The paper's 100B-token experiments in Section 2 (all on identical FineWeb-Edu data) address this for the architectural design choices, but the final model evaluations conflate architecture and data quality.

System domain data collection (Section 4, expanded in Appendix H.1):

The 19.5B tokens of system domain data come from 15 primary source categories across 120+ system-related websites. The collection pipeline involves:

Source identification: Academic papers (arXiv, conference proceedings), StackOverflow (debugging knowledge), technical blogs and developer forums (system design capability), Azure VM documentation, hardware abstraction layers, Linux command references, GitHub issues, and Stack Exchange.
Format-specific extraction: Each source has its own format, so individual processing pipelines extract publicly available system data.
LLM-assisted cleaning: GPT-3.5 is used to categorize data items (e.g., classifying StackOverflow posts by system domain relevance). A smaller classifier model is then trained on the GPT-3.5 labels to process the remaining data at lower cost.
Quality control: Category classification, quality filtering, and format conversion are performed using AI tools automatically.

The resulting data covers four capability areas (Table 5): General System (3.3B tokens from CCF Ranking list, 5.4B from arXiv), Design Capability (3.2B from blogs and forums), and Debug Capability (7.6B from StackOverflow). The paper does not provide detailed statistics on the filtering criteria or the quality of the LLM-based cleaning—these are practical engineering details that affect downstream model quality but aren't rigorously characterized.

Summary of Design Choices and Their Justifications

The paper makes several key design decisions that collectively define the DiffQKV approach:

Why compress K more than V (instead of compressing them equally)? Because Observation 1 shows approximately 2–4× greater sensitivity to V head reduction than K head reduction. The mechanistic justification is that K's role is routing (tolerant of approximation due to attention sparsity) while V's role is information content (intolerant of compression since output is a linear combination of V vectors). No prior work had empirically measured this differential sensitivity.

Why augment Q rather than expand the FFN or keep Q standard? Because Observation 4 shows Q augmentation is more parameter-efficient than FFN expansion (+1.24 vs. +1.12 per $\delta$ parameters), and because Q parameters don't affect the KV cache (Observation 3). The alternative—not augmenting anything—would leave performance on the table, especially when KV is heavily compressed.

Why use 4 K heads and 16 V heads (not 4 and 4, or 8 and 16)? Because this configuration maximizes the K compression benefit (37.5% KV cache reduction) while keeping V at a moderate compression level (16 heads, matching standard GQA with 32 Q heads). More aggressive K compression (e.g., to 1 head, MQA-style) showed essentially zero additional performance loss (Table 1: −0.01 vs. baseline) but the paper didn't adopt it—possibly because going from 4 to 1 K heads provides diminishing cache savings (the cache is dominated by V at 16 heads) and may hurt very long-context performance not captured in the 100B-token benchmarks.

Why not compress K head dimension in the final model? The paper demonstrates this is viable (Observation 2) but doesn't deploy it, citing "balancing the model performance and the cost of the KV cache." This is a pragmatic choice: head count reduction already delivers large savings, and dimension compression adds a K-to-Q projection layer that increases computation and implementation complexity.

Why use FlexHeadFA instead of padding/duplicating to match head counts? Because padding (e.g., duplicating K heads from 4 to 16 via kv_group_sharing) would negate the memory and bandwidth savings entirely—the padded K cache would be the same size as the V cache. Native support for asymmetric head counts in the attention kernel is essential for realizing the efficiency gains.

Why 1.5× augmentation for Q (not 2× or 2.75×)? Because the 1.5× scaling gave the best gain per parameter in Observation 3 (+1.24 with $d_h^q = 3072$ vs. +0.79 with 4096 and +0.93 with 5632). This is an empirical saturation pattern—beyond a certain point, additional Q capacity yields diminishing returns, possibly because the rest of the model (FFN, V) becomes the bottleneck.

4. Key Insights and Innovations

Innovation 1: Differential KV Compression as a First-Class Architectural Principle

The paper's most fundamental intellectual contribution is the empirical demonstration that Key and Value vectors in self-attention have fundamentally different sensitivities to compression—and that this differential sensitivity can and should drive architectural design. This is not an incremental optimization; it is a conceptual reframing of what constitutes a well-designed efficient attention mechanism.

Prior to this work, the dominant assumption—encoded in every major attention variant from MHA (Vaswani et al., 2017) through MQA (Shazeer, 2019) to GQA (Ainslie et al., 2023)—was that K and V should be compressed identically. Whether reducing head counts, evicting cache entries, or sharing parameters, the field treated K and V as a matched pair. The implicit reasoning was architectural symmetry: since both K and V participate in the attention computation, they should receive symmetric treatment. This assumption was so entrenched that standard attention kernels (FlashAttention2) and deployment frameworks (vLLM, TensorRT-LLM) hard-coded the constraint $n_k = n_v$ .

What this paper shows—through the controlled 100B-token experiments in Section 2—is that this symmetry assumption is empirically false. Reducing K heads by 75% causes only a 0.17-point performance drop, while the same reduction on V heads causes more than double the degradation (0.38 points). At the extreme, compressing K to a single head essentially costs nothing (+0.01 point), while compressing V to a single head costs 0.63 points (Table 1). The mechanistic explanation—that K vectors route attention (tolerant of approximation because attention distributions are >95% sparse) while V vectors provide the output content (linearly combined, so every reduction directly limits expressivity)—is scientifically satisfying but secondary. The primary insight is the diagnostic move: measure differential sensitivity before designing the compression scheme.

This reframes efficient attention design from "how much can we compress everything?" to "what can we compress aggressively, and what must we preserve carefully?" It replaces a uniform budget with a differential one. The significance extends beyond the specific configuration SIGMA adopts (4 K heads, 16 V heads). It establishes a design methodology: when building efficient transformers, first characterize the sensitivity of each component to compression along each axis (head count, head dimension, precision), then allocate the compression budget accordingly. This is a transferable principle—future architectures with different base configurations or trained on different data should re-measure these sensitivities rather than assuming symmetry.

The difference between this and GQA is conceptual, not just parametric. GQA asks: "how few K and V heads can we use while keeping $n_k = n_v$ ?" DiffQKV asks: "what is the optimal allocation of heads between K and V given their different roles?" The former is a single-axis optimization; the latter is a multi-axis one. The field had implicitly chosen the former because the measurement had never been done; this paper does the measurement and shows the constraint is unnecessary.

Innovation 2: Augmented Q as a Compute-Only Parameter Investment Strategy

The second distinctive idea is the recognition that Query vectors occupy a privileged position in the inference cost model—they are computed fresh at each step and never cached—which makes them an ideal target for additional parameter investment that improves model quality without increasing the dominant inference bottleneck (memory bandwidth). This reframes parameter allocation as a triage problem: spend parameters where they cost least at inference time.

The standard efficiency narrative in LLM architecture is one of compression: reduce heads, reduce dimensions, share parameters. MQA and GQA are pure compression plays—they remove parameters from the attention mechanism. The implicit assumption is that efficiency requires parameters to be removed, not reallocated. SIGMA challenges this by showing that parameters can be added to Q—increasing total attention parameters above a standard GQA baseline—while still improving net efficiency, because the added parameters land in the computation-only portion of the inference pipeline rather than the memory-intensive portion.

Observation 4 crystallizes this insight: adding parameters to Q is more performance-effective per parameter than adding them to the FFN. The FFN is the other obvious target for parameter investment, but FFN parameters must be loaded from memory at every layer for every token (even if computation is the bottleneck, the weights still consume memory capacity). Q parameters, by contrast, are computed once per token and immediately consumed—their memory footprint during inference is transient. The 1.5× Q augmentation ratio ( $d_h^q = 3072$ for the 1.5B model) is empirically optimal in the tested range, providing +1.24 points on a 9-benchmark average with parameters that impose zero additional KV cache cost and minimal memory bandwidth overhead.

This is fundamentally different from the standard approach of "make the model smaller to make it faster." It is instead awareness of the inference cost structure in the architectural design phase—a form of co-design where the training-time architecture is optimized for deployment-time bottlenecks. The paper does not frame it this way explicitly, but the principle generalizes: any component of the transformer that is not cached (Q, FFN activations, layer norm parameters) can be augmented with parameters that improve quality at inference-time computation cost but zero memory cost. The asymmetry between cached (K, V) and uncached (Q) components creates an opportunity for parameter reallocation that prior efficient architectures ignored entirely.

Innovation 3: The System Domain Benchmark as a Diagnostic Tool for Underexplored Capabilities

AIMICIUS is not just an evaluation suite; it is a diagnostic instrument that reveals a systematic weakness in general-purpose LLMs that prior benchmarks did not capture. The tasks it defines—command generation for hardware management (CMDGen), infrastructure benchmark retrieval (Infrawise), network topology optimization (Optiflow), and natural-language-to-KQL translation (NL2KQL)—require capabilities that sit at the intersection of technical knowledge, structured reasoning, and domain-specific tool use. General-purpose models (GPT-4, DeepSeek-Coder, Qwen-Coder, LLaMA3-8B) struggle dramatically on these tasks (Table 6), with GPT-4 achieving only 25% accuracy on CMDGen NVIDIA and 0.5% on Optiflow Plan Improved.

The innovation is not that a specialized model outperforms general ones on its specialty—that is expected. The innovation is that AIMICIUS quantifies how large the gap is and which specific sub-capabilities are missing from general models. For instance, on CMDGen, GPT-4 achieves a respectable 84.0 CMD Score (cosine similarity of embeddings) but only 13.0% Exact Match and 21.0% Success Ratio—it generates commands that look semantically similar to the ground truth but don't actually work. This reveals a specific failure mode: general models can approximate the form of system commands but lack the precise syntax and parameter knowledge needed for executable correctness. On Optiflow, GPT-4 achieves only 16.8% Plan Valid and 0.5% Plan Improved—the model cannot reason about GPU network topology optimization at all, despite its strong performance on standard code generation benchmarks.

This diagnostic specificity is what distinguishes AIMICIUS from generic domain benchmarks. It does not just measure "system domain performance" as a scalar; it decomposes it into component skills (command correctness, infrastructure understanding, optimization reasoning, query language translation) with metrics that isolate different failure types (semantic similarity vs. exact match vs. execution success). This makes AIMICIUS a tool for capability diagnosis, not just ranking. The paper's own analysis in Section 5 acknowledges that even SIGMA-SYSTEM's absolute performance remains low on some metrics (e.g., 28.3% Benchmark Result Accuracy on Infrawise), which itself is a meaningful diagnostic: certain system domain capabilities require more than domain-specific pre-training data, pointing toward the need for more sophisticated reasoning or tool-use architectures.

The benchmark is also practically significant because the system domain represents a high-value application area—the paper's framing of "LLMs that can oversee their own training processes" (Section 4) is ambitious but directionally important. By providing the first standardized evaluation for this domain, AIMICIUS enables systematic progress rather than ad-hoc assessment.

Innovation 4: Kernel-Level Support for Asymmetric Attention as an Enabling Contribution

FlexHeadFA—the modified FlashAttention2 kernel that supports $n_k \neq n_v$ —is easy to overlook as "just engineering," but it plays a critical enabling role that makes the DiffQKV concept deployable. The intellectual contribution is the recognition that architectural innovation is constrained by kernel support, and that removing a kernel constraint (the equal-head-count assumption in FlashAttention2) unlocks a new design space.

Before FlexHeadFA, any architecture with $n_k \neq n_v$ would face a hard choice: either use a compatibility wrapper that duplicates heads to match counts (negating the efficiency gains, as the paper acknowledges for kv_group_sharing), or write a custom attention implementation that would be too slow to be practical. The equal-head-count constraint was not a theoretical limitation of attention—the math works fine with different head counts—but a practical limitation imposed by the dominant efficient attention kernel. This created a chicken-and-egg problem: no one designed architectures with asymmetric heads because kernels didn't support them, and no one wrote kernels to support asymmetric heads because no architectures needed them.

FlexHeadFA breaks this deadlock by showing that the modification is straightforward: separate the address calculations for K and V heads in the split kernel. The key line is $\text{idx}_i = \text{idx}_q \cdot n_h^i / n_h^q$ for $i \in \{k, v\}$ , where $\text{idx}_k$ and $\text{idx}_v$ are computed independently. This is a small code change with large architectural consequences—once the kernel supports it, the entire design space of asymmetric head counts becomes explorable.

The empirical validation (Figure 1, Table 15) demonstrates that the modified kernel actually realizes the theoretical efficiency improvements: split kernel execution time improves by 25–27% at 16K–32K prefix lengths, close to the theoretical 37.5% bound (with the gap attributable to fixed costs in the combine kernel and other overhead). The CEET results (Figure 2) show that at 64K output length, SIGMA achieves a 33.36% total cost reduction over the standard GQA model—validating that the kernel-level efficiency gains translate to end-to-end speedups.

This innovation is particularly significant because it establishes a template for future architecture-kernel co-design. The standard approach has been to design architectures within the constraints of existing kernels; FlexHeadFA demonstrates that modest kernel modifications can enable qualitatively different architectural choices. The paper's call for "broader DiffQKV support" (Section 3.2) in deployment frameworks is essentially arguing that the ecosystem should catch up to a design principle that has been demonstrated to work.

Innovation 5: The Conceptual Generalization of Attention Mechanisms into a Unified Design Space

While not the most visible contribution, the paper's formal treatment of attention mechanism variants as points in a generalized design space (Appendix B.1, Figure 3) represents a useful conceptual contribution. By defining DiffQKV as:

$\text{DiffQKV} = \{(n_h^q, n_h^k, n_h^v, d_h^q, d_h^k, d_h^v) \mid \text{no equality constraints required}\}$

the paper reframes MHA, MQA, and GQA not as separate "attention mechanisms" but as specific coordinate choices within a 6-dimensional configuration space. MHA is the point $(n, n, n, d, d, d)$ . MQA is $(n, 1, 1, d, d, d)$ . GQA is $(n, k, k, d, d, d)$ with $1 < k < n$ . DiffQKV removes the diagonal constraint entirely.

This reframing matters because it changes how one thinks about attention architecture design. Rather than "should I use MHA, MQA, or GQA?"—a discrete choice among named mechanisms—the question becomes "what is the optimal point in this 6-dimensional space given my efficiency and performance constraints?" The design problem shifts from selection among predefined options to optimization over a continuous (or at least fine-grained discrete) space.

The empirical observations in Section 2 then serve as partial derivatives in this space: Observation 1 characterizes sensitivity to $n_h^k$ vs. $n_h^v$ , Observation 2 characterizes sensitivity to $d_h^k$ , Observation 3 characterizes the benefit of increasing $d_h^q$ , and Observation 4 characterizes the relative benefit of increasing Q parameters vs. FFN parameters. Together, they provide gradient information that guides the choice of configuration. SIGMA's specific settings ( $n_h^q=32, n_h^k=4, n_h^v=16, d_h^q=1.5\times d_m$ ) represent one optimized point in this space for the ~1B–10B parameter regime trained on general-domain data.

The conceptual value of this unification is that it makes future architectural innovation more systematic. Rather than proposing and naming new attention mechanisms, researchers can explore different regions of the configuration space and characterize the trade-offs. The paper's findings that the equal-KV-heads constraint is unnecessarily restrictive, and that Q augmentation is under-exploited, suggest that the design space around standard GQA ( $n_k = n_v < n_q$ ) is not at a local optimum—there are nearby points with strictly better efficiency-performance trade-offs. This is a more precise claim than "our architecture is better," and it invites systematic exploration rather than point-wise competition.

5. Experimental Analysis

Evaluation Methodology

Dataset. Two distinct evaluation regimes are used. For the architectural studies in Section 2, the paper trains ~1B-parameter models on 100B tokens from FineWeb-Edu (Penedo et al., 2024) and evaluates on nine benchmarks: HellaSwag (Zellers et al., 2019), OpenBookQA (Mihaylov et al., 2018), WinoGrande (Sakaguchi et al., 2021), ARC Challenge (Clark et al., 2018), PIQA (Bisk et al., 2020), SciQ (Welbl et al., 2017), BoolQ (Clark et al., 2019), LogiQA (Liu et al., 2020), and LAMBADA (Paperno et al., 2016). For the system domain evaluation, the paper introduces AIMICIUS, a custom benchmark with four tasks (CMDGen, Infrawise, Optiflow, NL2KQL) sourced primarily from Azure services, comprising 395 test cases for CMDGen (200 NVIDIA, 195 AMD), 911 test cases for Infrawise, 1,258 test cases for Optiflow, and 43 test cases for NL2KQL (Section 4, Appendix C).
Base model(s). The architectural ablations use a ~1B-parameter model (22 layers, hidden dimension 2048, 32 Q heads) trained from scratch on FineWeb-Edu. The production models are SIGMA-1.5B and SIGMA-10B, both trained from scratch on 6T tokens using the DiffQKV architecture. For system domain evaluation, SIGMA-SYSTEM-10B is fine-tuned with full-parameter updates on the proprietary system domain SFT dataset. The general-domain baselines at the 1B–2B scale include Pythia (1.0B, 1.4B), TinyLlama-1.1B, Bloom-1.1B, OLMo-1.2B, OPT-1.3B, CerebrasGPT-1.3B, Phi1-1.3B, DCLM-1.4B, StableLM2-1.6B, SmolLM-1.7B, and Gemma-2B. The system-domain baselines include GPT-4, DeepSeek-Coder-7b-Instruct-v1.5, Qwen2.5-Coder-7B-Instruct, and LLaMA3-8B-Instruct.
Metrics. For the 1B-scale architectural ablations, the primary metric is the average score across the nine benchmarks listed above, with individual benchmark results reported in Appendix B.3. For the AIMICIUS benchmark, each task has specialized metrics: CMDGen uses CMD Score (cosine similarity of command embeddings), Output Score (cosine similarity of execution output embeddings), Calibration Score (thresholded accuracy proxy), Exact Match, Success Ratio, and Accuracy (the primary metric, defined as Exact Match OR Success Ratio); Infrawise uses Target, Baseline, Criterion, Workload, DCW (all four correct simultaneously), Benchmark Result Recall, and Benchmark Result Accuracy (the primary metric); Optiflow uses Code Detected, Code Executable, Plan Valid, and Plan Improved (the primary metric); NL2KQL uses Syntax Accuracy, Jaccard Similarity, Cluster Score, Database Score, Table Score, and Column Score. All AIMICIUS metrics are normalized to a 0–100 scale with higher values indicating better performance (Table 6).
Baselines. For general-domain evaluation at the 1B–2B scale, SIGMA-1.5B is compared against 11 models listed above. For system-domain evaluation, SIGMA-SYSTEM-10B is compared against GPT-4, DeepSeek-Coder-7b-Instruct-v1.5, Qwen-Coder-7B-Instruct, and LLaMA3-8B-Instruct on all four AIMICIUS tasks. For the efficiency analysis, the baseline is a hypothetical "standard model" (STD) with GQA configuration: balanced K and V heads ( $n_k = n_v = 16$ ) and no augmented Q, compared against SIGMA-1.5B with $n_k = 4$ , $n_v = 16$ , and augmented Q ( $d_h^q = 3072$ ). Both models are evaluated using FlashAttention2 (Dao, 2024) as the default attention mechanism.
Generation budget / compute accounting. In the efficiency analysis (Section 3), compute is measured using two metrics: Cuda Event Elapsed Time (CEET), which records elapsed time of specific operations via checkpoints in the code, and Kernel Execution Time (KET), which records GPU kernel execution times using NVIDIA's nsys profiler. CEET measures three modules—KV Cache operations (storing and loading), Attention Computation, and Augmented Q (SIGMA only)—treating their summation as the total cost of attention layers. KET measures the split kernel and combine kernel separately, treating their summation as the total FlexHeadFA cost. Both models are evaluated with output and prefix lengths independently varying in grid patterns: for CEET, both range across [2k, 4k, 8k, 16k, 32k, 64k]; for KET, output length is fixed at 10 while prefix length varies across [2k, 4k, 16k, 32k]. All experiments are conducted on a single NVIDIA H100 80G HBM3 GPU. The theoretical analysis models KV Cache cost as $L = \alpha \cdot [b \cdot s \cdot (n_h^k \cdot d_h^k + n_h^v \cdot d_h^v)] + \beta$ , deriving a theoretical reduction rate of 37.5% for SIGMA vs. GQA as $s \to \infty$ (Equation 2).
Cross-validation / statistical protocol. For the 1B-scale architectural ablations, all models are trained from scratch on the same 100B FineWeb-Edu tokens, and evaluation uses zero-shot prompting, so the protocol is straightforward comparison of identically trained models. For the AIMICIUS evaluation, the paper fine-tunes SIGMA-SYSTEM-10B with full-parameter updates on a proprietary SFT dataset tailored individually for each task, and compares against baseline models evaluated on the same test sets. The paper does not report confidence intervals, standard deviations, or multiple-run variance for any of the benchmark results. For the efficiency analysis, the paper performs a t-test on the relative improvement ratio of Augmented Q across all CEET settings (comparing SIGMA's AugQ module cost against the same module attached to STD), finding a T-value of −0.214 and P-value of 0.832, which "indicates no evidence to reject our hypothesis" that Augmented Q's cost remains consistent regardless of context length (Appendix G). No other statistical tests are reported.

Main Quantitative Results

Architectural Ablation Experiments at 1B Scale (Section 2)

The foundational experiments that motivate DiffQKV are the controlled comparisons in Tables 1–4, each trained on 100B FineWeb-Edu tokens at ~1B parameters and evaluated on the 9-benchmark average.

Differential sensitivity of K vs. V head reduction (Tables 1, 9). Starting from three baseline configurations—MHA ( $n_k = n_v = 32$ , score 52.40), GQA ( $n_k = n_v = 16$ , score 52.14), and GQA ( $n_k = n_v = 4$ , score 51.66)—the paper independently reduces either $n_k$ or $n_v$ while keeping $n_q = 32$ and the other head count fixed. On the MHA baseline, reducing V heads by 50% ( $n_v = 16$ ) drops performance by 0.66 points to 51.74, while reducing K heads by 50% ( $n_k = 16$ ) improves performance by 0.43 points to 52.83. On the GQA-16 baseline, reducing V by 75% to 4 heads costs 0.38 points (51.76), while reducing K by 75% to 4 heads costs only 0.17 points (51.97)—less than half the penalty. On the GQA-4 baseline, reducing V to a single head costs 0.63 points (51.03), while reducing K to a single head is essentially neutral at +0.01 points (51.67). The gap between K and V sensitivity is consistently 2–4× in V's disfavor and grows with compression ratio.

K head dimension reduction (Tables 2, 10). Halving K head dimension ( $d_h^k = d_h^v/2$ , with a trainable feed-forward layer projecting K up for dot-product compatibility) produces negligible changes: MHA goes from 52.40 to 52.56 (+0.16); GQA-16 from 52.14 to 52.06 (−0.08); GQA-4 from 51.66 to 51.92 (+0.26). In all cases the absolute change is ≤ 0.26 points, confirming that K dimension can be compressed without meaningful quality loss.

Augmented Q (Tables 3, 11). Increasing the Q intermediate dimension consistently improves performance, with the 1.5× scaling ( $d_h^q = 3072$ on a 2048-dim model) appearing optimal. For GQA-16: +1.24 with $d_h^q = 3072$ , +0.79 with $d_h^q = 4096$ , +0.93 with $d_h^q = 5632$ . The diminishing and non-monotonic returns at larger augmentations suggest saturation. For the more compressed GQA-4, augmentation with $d_h^q = 5632$ provides an even larger boost of +1.47 (51.66 → 53.13), supporting the interpretation that augmented Q compensates for KV compression capacity loss.

Augmented Q vs. FFN expansion (Tables 4, 12). Adding $\delta = 3072$ parameters to Q (+1.24) outperforms adding the same $\delta$ to FFN (+1.12), and adding $\delta$ to Q nearly matches adding $2\delta$ to FFN (+1.02). The gains are approximately additive: AugF( $\delta$ ) + AugQ( $\delta$ ) yields +2.41, close to 1.12 + 1.24 = 2.36. At the upper end, AugF( $3\delta$ ) + AugQ( $\delta$ ) achieves +2.95, the highest combination tested, confirming that Q augmentation and FFN expansion improve complementary aspects of model capacity.

Combination of strategies (Table 13). Integrating all three optimizations—75% K head reduction ( $n_k = 4$ ), half K dimension ( $d_h^k = d_h^v/2$ ), and augmented Q ( $d_h^q = 3072$ )—yields a score of 52.61, which surpasses the original GQA-16 baseline of 52.14 while providing substantial efficiency gains. The combination with only head reduction and augmented Q (no dimension compression) achieves 52.97, the highest among the tested configurations that apply at least one K compression strategy.

Selective V cache fetching (Table 8). Using only the top-100 V vectors (by attention score) for output computation produces negligible performance impact: MHA drops from 52.40 to 52.10 (−0.30), GQA-16 from 52.14 to 52.08 (−0.06), GQA-4 from 51.66 to 51.67 (+0.01). The largest impact (−0.30) is on MHA, which has the most V heads and therefore the most potential information loss from truncation; the more compressed configurations show essentially no loss, consistent with already having fewer V heads to begin with.

Note on the architectural experiments: The models used in these ablations are ~1B-parameter, 22-layer transformers trained on 100B tokens—a significantly smaller scale than the production SIGMA models (26–32 layers, 1.5B–10B parameters, 6T tokens). The paper extrapolates the findings from these small-scale experiments to the production architecture without validating that the K/V sensitivity ratios or the optimal augmentation ratios hold at larger scales. This is a standard but unverified assumption in architecture research.

Efficiency Analysis: SIGMA vs. Standard GQA Model (Section 3)

The efficiency comparison pits SIGMA-1.5B (DiffQKV with $n_k = 4$ , $n_v = 16$ , augmented Q with $d_h^q = 3072$ ) against a hypothetical standard model (STD) with GQA ( $n_k = n_v = 16$ , no augmented Q). The measurements cover KV cache operations, attention computation, and augmented Q overhead.

KET results (Figure 1, Table 15). The split kernel (primary attention computation) shows increasingly significant relative improvements as prefix length grows: 1.17% at 2K, 25.33% at 4K, 26.30% at 16K, and 27.21% at 32K. The combine kernel shows negligible change (ranging from −0.93% to +1.68%), confirming it is unaffected by K head count changes since it only assembles output chunks. Total KET improvement increases with prefix length: 1.39% at 2K, 18.02% at 4K, 23.35% at 16K, and 25.31% at 32K. The absolute split kernel cost grows from ~2.5M ns at 2K prefix to ~27.5M ns at 32K for STD, while SIGMA's grows from ~2.5M ns to ~20.0M ns—a gap of ~7.5M ns at 32K. The paper notes that these improvements approach the theoretical 37.5% bound as prefix length continues to expand.

CEET results—Total cost (Figure 2). The combined cost of KV cache operations, attention computation, and augmented Q (SIGMA only) reveals a cross-over pattern. At 2K output length (Figure 2a), SIGMA incurs higher costs than STD for prefix lengths below ~16K, but becomes cheaper beyond that threshold. As output length increases, the cross-over point shifts earlier: at 4K output (Figure 2b), SIGMA is cheaper above ~4K prefix; at 8K output, above ~2K prefix; at 16K output, SIGMA is cheaper across all prefix lengths; at 64K output (Figure 2g), SIGMA achieves a 33.36% total cost reduction over STD even with zero prefix. The absolute costs span from ~10³ ms (short context) to ~10⁶ ms (64K output, 64K prefix). The paper summarizes: "SIGMA is slightly less efficient than the standard model in short-context scenarios (~10³ ms), but it shows significant advantages in long-context scenarios (~10⁶ ms)."

CEET results—Per-module breakdown (Appendix G, Figures 4–6). The Augmented Q module (Figure 4) shows consistent cost across all context length settings, and when attached to STD it produces near-zero relative improvement ratios, confirming it adds a fixed overhead that does not vary with context length—the t-test P-value of 0.832 supports this. The KV cache module (Figure 5) shows the largest relative gains, with SIGMA achieving a 36.57% speedup over STD at 64K prefix and output lengths—very close to the theoretical 37.5%—because KV cache operations are primarily proportional to cache size with minimal extraneous factors. The attention computation module (Figure 6) shows more modest but steady improvements: up to 15.4% reduction at the largest context lengths, deviating more from the theoretical value than KV cache because attention computation involves "a greater number of operations that are not directly influenced by the number of key heads" and CEET captures end-to-end time including context switching and CPU overhead.

The cross-over behavior explained. At short contexts, SIGMA's augmented Q overhead (extra projection parameters) and the additional feed-forward layer for K-Q dimension matching (when applicable) make it slightly slower than STD. These are fixed per-token computation costs. At long contexts, the KV cache bandwidth savings dominate—loading 4 K heads instead of 16 reduces memory traffic proportionally—and the fixed overhead becomes negligible relative to the bandwidth savings. This is the critical practical insight: DiffQKV is not universally faster; it is faster precisely in the regime where KV cache is the bottleneck, which happens to be the regime that matters for production LLM serving with long contexts.

System Domain Performance on AIMICIUS (Section 5, Table 6)

SIGMA-SYSTEM-10B is evaluated on all four AIMICIUS tasks after fine-tuning on proprietary SFT data, compared against GPT-4, DeepSeek-Coder-7b-Instruct-v1.5, Qwen2.5-Coder-7B-Instruct, and LLaMA3-8B-Instruct.

CMDGen NVIDIA. SIGMA achieves 87.5 CMD Score, 80.9 Output Score, 78.0 Calibration Score, 57.0% Exact Match, 74.0% Success Ratio, and 74.5% Accuracy. GPT-4, the strongest baseline, achieves 84.0 CMD Score but only 13.0% Exact Match and 21.0% Success Ratio, yielding 25.0% Accuracy. The gap between SIGMA and GPT-4 on Accuracy is 49.5 absolute percentage points. DeepSeek achieves 59.4 CMD Score and 0.5% Accuracy; Qwen achieves 81.1 CMD Score and 15.5% Accuracy; LLaMA achieves 33.6 CMD Score and 3.5% Accuracy. The striking pattern is that general models can produce commands with reasonable semantic similarity (84.0 CMD Score for GPT-4) but fail catastrophically on executable correctness (13.0% Exact Match)—they know what commands look like but don't know the exact syntax and parameters.

CMDGen AMD. SIGMA achieves 88.9 CMD Score, 78.0 Output Score, 79.3 Calibration Score, 53.9% Exact Match, 69.4% Success Ratio, and 69.4% Accuracy. The strongest baseline is Qwen at 77.2 CMD Score and 35.8% Accuracy. GPT-4 achieves 73.0 CMD Score but only 14.0% Exact Match and 17.0% Accuracy. The absolute improvement of SIGMA over the best baseline on Accuracy is 33.6 percentage points. The AMD subtask shows slightly lower SIGMA Accuracy (69.4% vs. 74.5% for NVIDIA), which the paper does not discuss but may reflect differences in training data coverage across the two platforms.

Infrawise. SIGMA achieves 95.2% Target, 92.9% Baseline, 75.1% Criterion, 48.4% Workload, 40.3% DCW, 28.3% Benchmark Result Recall, and 28.3% Benchmark Result Accuracy. The DCW composite metric (all four components correct) at 40.3% is substantially higher than the next-best model (Qwen at 20.8%). However, the critical end-to-end metric—Benchmark Result Accuracy—is only 28.3% for SIGMA, with the best baseline (Qwen) at 20.4%. On the component metrics (Target, Baseline, Criterion), SIGMA shows dramatic improvements (e.g., 95.2% vs. 44.9% Target for the best baseline), but these don't translate proportionally to end-to-end retrieval accuracy. The paper acknowledges this: "its other metrics do not exceed 50%, with Benchmark Result Accuracy standing at 28.3%. The performance of other models on this task is even more disappointing."

Optiflow. SIGMA achieves 100.0% Code Detected, 85.9% Code Executable, 86.7% Plan Valid, and 66.7% Plan Improved. GPT-4 achieves 95.8% Code Detected, 50.3% Code Executable, 16.8% Plan Valid, and 0.5% Plan Improved. The gap on Plan Improved—the task's key metric measuring whether the model can iteratively optimize network topology to reduce latency—is 66.2 percentage points. Qwen achieves 100.0% Code Detected and 28.3% Plan Valid, but its Plan Improved is not reported (shown as "—"), suggesting it never successfully completed the improvement task. DeepSeek's Plan Improved is also unreported. This task reveals the starkest capability gap: general models can produce syntactically valid Python code (82.2–100.0% Code Detected) but cannot reason about GPU network topology optimization at all.

NL2KQL. SIGMA achieves 100.0% Syntax Accuracy, 34.9% Similarity, 43.0% Cluster Score, 40.7% Database Score, 17.4% Table Score, and 29.0% Column Score. The strongest baseline is Qwen at 100.0% Syntax Accuracy, 36.7% Similarity, 4.7% Database Score, 4.7% Table Score, and 22.2% Column Score. While all models achieve high Syntax Accuracy (81.4–100.0%), the component-level scores are low across the board—GPT-4 achieves only 2.3% Database Score and 4.7% Table Score. SIGMA shows its largest advantage on Database Score (40.7% vs. 4.7% for baselines) but overall Similarity (34.9%) is only marginally better than Qwen (36.7%). The paper does not report Cluster Score for the baselines, limiting comparison.

General Domain Performance (Section 5, Tables 7 and 17)

Commonsense reasoning and text understanding (Table 7). SIGMA-1.5B achieves an average score of 61.6 across nine benchmarks, comparable to Gemma-2B (62.2) and DCLM-1.4B (62.8), and substantially outperforming OLMo-1.2B (54.5) and TinyLlama-1.1B (54.0). Specific highlights: SIGMA achieves top-2 performance on WinoGrande (67.5), PIQA (77.8), ARC Easy (72.3), ARC Challenge (43.9), and SciQ (94.0). Weaker performance appears on BoolQ (63.0, vs. 71.4 for DCLM-1.4B) and LogiQA (28.4, vs. 30.4 for Gemma-2B). The paper notes: "We observe a significant decrease in these benchmarks during the annealing stage when math problem-solving ability improves, showing a potential conflict between natural and formal language understanding for small-scale models."

Problem-solving, coding, and math (Table 17). SIGMA-1.5B achieves an average of 27.1 across seven benchmarks: MMLU (47.0), MMLU-Pro (17.6), BBH (32.7), HumanEval (21.3), MBPP (30.7), MATH (12.7), and GSM8K (27.8). This matches Gemma-2B (26.58 average) and exceeds most 1B–1.7B models substantially. SIGMA's math performance (12.7% MATH, 27.8% GSM8K) is particularly strong relative to its scale—StableLM2-1.6B achieves only 5.2% MATH and 21.3% GSM8K, SmolLM-1.7B achieves 4.2% MATH and 6.4% GSM8K. On coding, SIGMA (21.3% HumanEval, 30.7% MBPP) underperforms Phi1-1.3B (48.2% HumanEval, 27.2% MBPP) and Gemma-2B (25.0% HumanEval, 41.5% MBPP), which the paper attributes to "the relatively lower proportion of code in our pre-training corpus."

System domain data quality experiment (Table 16). The paper validates its system domain data by continual pre-training Mistral-7B and Llama3-8B on the 19.5B system domain tokens, then fine-tuning on a preliminary SFT dataset for CMDGen NVIDIA. Mistral-7B-S (SFT only) achieves 30.7% Accuracy; Mistral-7B-P-S (pre-training + SFT) achieves 32.2% Accuracy, a +1.5 point gain. Llama3-8B-S achieves 50.7% Accuracy; Llama3-8B-P-S achieves 57.1% Accuracy, a +6.4 point gain. The larger gain for Llama3 suggests interaction between base model quality and domain data effectiveness, but the paper does not analyze this further.

Ablation Studies and Robustness Checks

Differential K vs. V head reduction across three baseline architectures (Tables 1, 9). The pattern of greater V sensitivity holds across MHA, GQA-16, and GQA-4, with the gap widening at more extreme compression ratios. At the GQA-4 baseline (the most compressed), reducing V to a single head costs 0.63 points while reducing K to a single head costs effectively zero (+0.01). This robustness check confirms the asymmetry is not an artifact of the specific baseline architecture or compression level.

K dimension reduction across three architectures (Tables 2, 10). Halving $d_h^k$ produces negligible or slightly positive performance changes across MHA, GQA-16, and GQA-4. The consistency across baselines suggests the finding is not architecture-specific.

Multiple Augmented Q sizes on GQA-16 (Tables 3, 11). Testing $d_h^q$ at 3072 (×1.5), 4096 (×2.0), and 5632 (×2.75) reveals a non-monotonic pattern: 3072 gives +1.24, 4096 gives +0.79, 5632 gives +0.93. The 1.5× scaling appears optimal, ruling out the hypothesis that "more is always better" and suggesting genuine saturation or overfitting effects at higher augmentation ratios. The paper does not test intermediate values (e.g., 1.25×, 1.75×) to precisely locate the optimum.

AugQ + AugF additive combinations (Tables 4, 12). The near-additivity of Q augmentation and FFN expansion across multiple configuration pairs (e.g., AugF( $\delta$ ) alone: +1.12; AugQ( $\delta$ ) alone: +1.24; combined: +2.41) suggests these two mechanisms operate on complementary aspects of model capacity. The paper does not provide mechanistic analysis of why they are additive, only the empirical observation.

Selective V cache fetching across architectures (Table 8). Top-100 V selection is tested on MHA, GQA-16, and GQA-4, with performance impact ≤ 0.30 points in all cases. The finding is robust across architectures, though the paper does not test other k values (e.g., top-50, top-200) or sensitivity to sequence length—the 100B-token training and evaluation may not reflect behavior at much longer contexts where top-100 might be insufficient.

Combination of compression strategies (Table 13). Integrating -75% K heads, half K dim, and AugQ yields 52.61, above the original GQA-16 baseline (52.14). Omitting half K dim but keeping the other two yields 52.97, the best among configurations with K compression. This suggests that head count reduction alone provides most of the efficiency benefit, with dimension compression offering additional savings at minimal cost. The paper's decision not to deploy dimension compression in production SIGMA models is consistent with this result.

KET measurement at multiple prefix lengths (Table 15, Figure 1). The split kernel's relative improvement grows monotonically with prefix length: 1.17% → 25.33% → 26.30% → 27.21% as prefix increases from 2K to 32K. This monotonic trend validates the theoretical prediction that improvement asymptotically approaches 37.5% and confirms measurement stability. The combine kernel shows no systematic trend, fluctuating between −0.93% and +1.68%, consistent with its independence from K head count.

Augmented Q cost stability (Figure 4, Appendix G). In all CEET configurations (6 output lengths × 6 prefix lengths = 36 settings), the augmented Q module's relative improvement ratio hovers near zero, with the t-test (T = −0.214, P = 0.832) confirming no systematic deviation. This validates the claim that AugQ adds a fixed, context-independent overhead—a necessary condition for the claimed efficiency benefits to be real rather than an artifact of measurement noise.

System domain data quality via external model transfer (Table 16). Pre-training on the 19.5B system domain tokens provides consistent Accuracy gains on CMDGen NVIDIA for both Mistral-7B (+1.5) and Llama3-8B (+6.4). The differing magnitudes suggest that higher-quality base models benefit more from domain data, but the consistent positive direction validates the data's utility.

No ablations that were notably absent but would have been informative: The paper does not ablate which of the specific DiffQKV design choices (K head reduction, V head preservation, Q augmentation) contributes most to SIGMA's general-domain performance at the 1.5B/10B scale trained on 6T tokens. The architectural ablations are only conducted at 1B scale on 100B tokens. There is no comparison of SIGMA-1.5B against an identically trained GQA-1.5B model on the full 6T tokens to isolate the architecture's contribution from data quality. The efficiency analysis compares SIGMA against a hypothetical STD model, not against a real trained GQA baseline, and does not measure end-to-end inference latency including all transformer components (FFN, layer norm, embeddings) or batch inference throughput, which would be more representative of production serving than attention-layer-only CEET. The AIMICIUS evaluation does not include a baseline where the same system domain SFT data is used to fine-tune a general-domain SIGMA model (without the system domain pre-training phase) to isolate the benefit of domain-specific pre-training vs. domain-specific fine-tuning. No statistical confidence intervals or significance tests are reported for any benchmark comparisons, making it impossible to assess whether the performance differences between models with close scores (e.g., SIGMA-1.5B at 61.6 vs. DCLM-1.4B at 62.8) are reliable.

Critical Assessment

Claim 1: Differential KV Compression Improves Inference Efficiency

The efficiency analysis demonstrates that reducing K heads from 16 to 4 (while keeping V at 16) produces measurable speedups in KV cache operations and attention computation. The total CEET improvement at 64K output length reaches 33.36% (Figure 2g), and the KV cache module alone achieves 36.57% improvement (Figure 5h), closely matching the theoretical 37.5% bound. The KET results (Table 15) confirm that the split kernel—which directly operates on the key and value matrices—is the source of the gains, showing 25–27% improvement at longer prefix lengths.

What the experiments actually demonstrate vs. what is claimed. The experiments measure attention-layer cost reduction in isolation—they time KV cache operations, attention computation, and augmented Q as separate modules within a single model forward pass. They do not measure end-to-end inference latency including FFN computation, layer normalization, embedding lookup, logit projection, or the autoregressive generation loop. Since attention is typically 20–40% of total inference cost (with FFN dominating the rest), a 33% attention speedup translates to a much smaller end-to-end speedup—likely in the 7–13% range. The paper's claim of "up to a 33.36% improvement in inference speed" (Abstract) is technically correct about the attention component but risks being interpreted as an end-to-end generation speedup, which the experiments do not establish. The comparison is also against a hypothetical GQA model, not a real trained baseline, and assumes identical FFN and other module costs, which may not hold if GQA and DiffQKV models have different optimal hyperparameters.

Additionally, the cross-over pattern (Figure 2) shows that SIGMA is slower than STD at short contexts—the efficiency gains only materialize at longer sequences. The paper is transparent about this, noting the difference between short-context (~10³ ms) and long-context (~10⁶ ms) regimes, but the headline "33.36%" figure applies only in the most favorable long-context scenario and does not characterize the efficiency profile across the full context-length range.

Claim 2: Augmented Q Improves Performance with Minimal Inference Cost

The architectural ablations (Tables 3, 11) convincingly show that enlarging Q's intermediate dimension improves the 9-benchmark average by 0.63–1.47 points depending on the baseline and augmentation size. The CEET measurements (Figure 4, Appendix G) confirm that Augmented Q's cost is fixed and independent of context length, with the t-test (P = 0.832) supporting the claim of minimal and predictable overhead.

What holds and what doesn't. The claim of "minimal impact on inference speed" (Section 1) is supported for the attention-layer cost: AugQ adds a fixed per-token computation overhead that does not scale with sequence length. However, this claim is evaluated only in the context of attention-layer timing, not end-to-end generation throughput. At small batch sizes typical of interactive inference, the added computation from Q augmentation could measurably increase per-token latency even if it doesn't affect KV cache bandwidth. The paper's assertion that $d_h^q = 3072$ (×1.5) "appears optimal" is based on only three tested sizes (×1.5, ×2.0, ×2.75) and should be considered a preliminary optimum rather than a finely tuned result.

Claim 3: SIGMA Matches State-of-the-Art General-Domain Models

SIGMA-1.5B achieves 61.6 on the 9-benchmark commonsense/reasoning average (Table 7), which is numerically close to Gemma-2B (62.2) and DCLM-1.4B (62.8). On problem-solving tasks (Table 17), the 27.1 average is comparable to Gemma-2B (26.58). The paper presents this as matching state-of-the-art.

The data confound. SIGMA's performance is the joint product of the DiffQKV architecture and a carefully curated 6T token pre-training data mixture (including 1T tokens of synthesized and rewritten data, phased curriculum, annealing, and math/code-heavy later phases). The paper does not train a GQA-architecture model on the identical data mixture, so it is impossible to determine whether SIGMA's competitive general-domain performance comes from the DiffQKV architecture or from the high-quality pre-training data. The comparison models (Gemma, DCLM, StableLM, SmolLM) were trained on different data with different recipes, so any performance comparison confounds architecture and training data. The 100B-token 1B-scale experiments (Tables 1–4) provide clean architecture comparisons but at a much smaller scale, and they show that the best DiffQKV configuration (52.97, Table 13) is only modestly above the GQA-16 baseline (52.14) on identical data—a gain of 0.83 points. Whether this small advantage persists, grows, or shrinks at 1.5B/6T-token scale is unknown.

Claim 4: SIGMA-SYSTEM Dramatically Outperforms GPT-4 on System Domain Tasks

The AIMICIUS results (Table 6) show SIGMA-SYSTEM-10B achieving large absolute improvements over GPT-4: 49.5 percentage points on CMDGen NVIDIA Accuracy (74.5% vs. 25.0%), 52.4 points on CMDGen AMD Accuracy (69.4% vs. 17.0%), 66.2 points on Optiflow Plan Improved (66.7% vs. 0.5%). These are genuine capability gaps—GPT-4 essentially cannot perform system domain tasks that SIGMA-SYSTEM handles competently.

What this demonstrates and what it doesn't. The comparison demonstrates that a model fine-tuned on domain-specific SFT data dramatically outperforms a general model prompted zero-shot or few-shot on the same domain. This is expected and not specific to DiffQKV—one would expect a GPT-4 model fine-tuned on the same system domain data to also improve substantially. The paper does not include a GPT-4 fine-tuned baseline, nor does it compare SIGMA-SYSTEM against a GQA-architecture model fine-tuned on the same data. The claim that SIGMA-SYSTEM's superiority is attributable to "domain-specific advancements" (Section 5) is true but ambiguous—the advancements include both the DiffQKV architecture and the system domain pre-training + SFT data, and their relative contributions are not separated. The data quality experiment (Table 16) shows that system domain pre-training helps Llama3-8B, but this validates the data, not the architecture.

Claim 5: DiffQKV Attention Is a Generalizable Design Principle

The paper frames DiffQKV as a general principle—differential optimization of Q, K, V based on their roles—rather than a specific architecture. The 1B-scale experiments characterize K/V sensitivity and Q augmentation benefits on a single model configuration (22 layers, 2048 hidden dim) trained on a single data distribution (FineWeb-Edu). The findings are extrapolated to the 1.5B and 10B scales without validation. The paper does not test whether the differential sensitivity ratios change with model scale, training data distribution, or architecture depth—all of which are plausible but untested. The generalizability of the principle to non-decoder architectures, encoder-decoder models, or cross-attention contexts is entirely unexplored.

Genuine Weaknesses in Experimental Design

No architecture-controlled scaling comparison. The most important missing experiment is a head-to-head comparison of SIGMA-1.5B against a GQA-1.5B model trained on the identical 6T token data mixture. Without this, the paper cannot distinguish between "DiffQKV is a better architecture" and "our data mixture and training recipe produce better models." Given that the 100B-token ablation shows only a modest advantage for the best DiffQKV configuration (+0.83 over GQA-16 baseline in Table 13), the data quality explanation for SIGMA's competitive general-domain performance is plausible.
Efficiency measurements are component-level, not end-to-end. The 33.36% speedup is measured on the attention component only. End-to-end generation throughput at various batch sizes and sequence lengths would be more informative for practitioners. The paper also doesn't benchmark memory consumption of the KV cache—a key claimed benefit—with actual GPU memory measurements, relying instead on theoretical calculations.
The AIMICIUS evaluation conflates domain pre-training and domain fine-tuning. SIGMA-SYSTEM-10B receives both system domain pre-training (19.5B tokens) and task-specific SFT. The baselines receive neither. An informative ablation would fine-tune baselines on the same SFT data (with or without domain pre-training) to separate the contributions of data, pre-training, and architecture.
Small test sets for AIMICIUS tasks (especially NL2KQL with 43 test cases) raise concerns about statistical reliability. The paper reports no confidence intervals, no significance tests, and no cross-validation for the AIMICIUS results. A difference of a few percentage points on a 43-sample test could easily arise from sampling noise.
The 1B-scale architectural ablations use a single training run per configuration. There are no error bars or multiple seeds to account for training variance. The 9-benchmark average scores differ by as little as 0.01–0.66 points between configurations (Table 1), and without variance estimates, it's unclear which differences are reliable.
The K dimension compression finding relies on a trainable projection layer whose cost is not benchmarked. Observation 2 shows that halving K head dimension with a feed-forward projection preserves performance, but the efficiency analysis (Section 3) does not include this projection in the cost measurements—SIGMA's production models do not use dimension compression, so the cost analysis covers only head count reduction.
No long-context evaluation of model quality. The paper demonstrates efficiency improvements at long contexts (up to 64K tokens) but does not evaluate whether SIGMA maintains generation quality (perplexity, task performance) at those context lengths. It's possible that aggressive K head compression degrades the model's ability to attend precisely over very long sequences, even if the 100B-token perplexity and benchmark scores are preserved. This is a missing validation that would be important for the claimed use case.
The comparison models in the general-domain evaluation use different tokenizers and vocabularies. SIGMA uses the Llama3 vocabulary (128K tokens), while Gemma-2B uses a 256K vocabulary, and others use various sizes. Tokenizer differences affect benchmark scores independently of model quality, particularly for tasks with short contexts or answer extraction. The paper does not normalize for this.

6. Limitations and Trade-offs

The Architecture-Data Confound: SIGMA’s General-Domain Performance Cannot Be Attributed to DiffQKV Alone

The assumption or constraint. All general-domain evaluations of SIGMA-1.5B and SIGMA-10B compare models that differ simultaneously in architecture (DiffQKV vs. standard GQA/MHA) and training data (SIGMA’s custom 6T token mixture vs. each baseline’s proprietary data). The architectural ablations in Section 2 establish the empirical patterns that motivate DiffQKV, but these are conducted at ~1B parameters on 100B FineWeb-Edu tokens—over an order of magnitude smaller than the production models in both parameters and data. The paper does not train a GQA-architecture model on the identical 6T token data mixture that SIGMA receives, so there is no experiment that isolates the architectural contribution to final model quality at production scale.

The consequence. When SIGMA-1.5B achieves an average score of 61.6 vs. Gemma-2B’s 62.2 (Table 7) or 27.1 vs. 26.58 on problem-solving tasks (Table 17), a practitioner cannot determine whether this parity comes from DiffQKV’s differential QKV rescaling or from SIGMA’s data curation pipeline (multi-phase curriculum, 1T tokens of synthesized and rewritten data, annealing on high-quality subsets). The 100B-token ablation in Table 13 shows that the best DiffQKV configuration (AugQ + -75% K heads + half K dim) achieves 52.97 vs. the GQA-16 baseline of 52.14—a gain of only 0.83 points on identical data at 1B scale. If this modest advantage shrinks or disappears at 1.5B/6T-token scale, then the practical value of DiffQKV for general-domain quality (as opposed to inference efficiency) would be minimal, and the competitive benchmark scores would be primarily attributable to data quality—something any architecture could benefit from.

What evidence exists in the paper. The 100B-token controlled experiments (Tables 1–4, 8–13) are the only architecture-only comparisons. All production model evaluations (Tables 6, 7, 17) confound architecture and data. The paper never acknowledges this confound explicitly or discusses its implications for interpreting the general-domain results. The description of the pre-training data in Appendix H.1 reveals a sophisticated multi-phase curriculum with carefully selected data ratios and quality tiers that could easily account for several points of benchmark improvement independently of the architecture.

Mitigation status. Not addressed. The paper does not flag this as a limitation and does not propose a production-scale architecture-controlled comparison as future work. A practitioner reading the paper would likely assume that SIGMA’s competitive benchmark performance demonstrates DiffQKV’s effectiveness for model quality, when in fact the evidence for this claim is circumstantial.

Inference Efficiency Gains Are Component-Level, Not End-to-End; the 33.36% Figure May Substantially Overstate Real-World Speedups

The assumption or constraint. All efficiency measurements in Section 3 are conducted on the attention component in isolation—timing only KV cache operations, attention computation, and augmented Q (for SIGMA). The FFN computation, layer normalization, embedding lookup, logit projection, and autoregressive generation loop overhead are excluded. The paper acknowledges this scope implicitly (Section 3.1 describes the cost model as covering “KV Cache” and “Attention Computation”) but presents the 33.36% figure in the Abstract without the qualification that it measures attention-layer cost, not end-to-end generation speed.

In a standard transformer, the FFN typically accounts for 50–70% of per-layer computation, with attention comprising the remaining 30–50%. A 33.36% reduction in attention cost therefore translates to roughly a 10–17% reduction in per-layer compute, and the end-to-end generation speedup (including fixed-cost operations like embedding, sampling, and the decode loop) would be smaller still.

The consequence. A practitioner who reads “up to a 33.36% improvement in inference speed” (Abstract) and expects their serving throughput or per-token latency to improve by one-third will be disappointed in typical deployment scenarios. The actual end-to-end speedup depends on the attention-to-FFN cost ratio for their specific model configuration and hardware, and it will be systematically lower than 33%. Additionally, the measurements in Figure 2 show that the efficiency gains only materialize at longer context lengths; at short contexts (~10³ ms regime), SIGMA is actually slower than the GQA baseline due to the overhead of augmented Q and the additional parameter computation. The “33.36%” figure applies only at 64K output length with no prefix (Figure 2g), the most favorable single data point in the entire evaluation grid.

What evidence exists in the paper. Figures 2, 4, 5, and 6 all measure attention-component costs via CEET. The paper never reports end-to-end generation latency or throughput at any batch size. The KET results (Figure 1, Table 15) are even narrower, measuring only the FlashAttention split and combine kernels. The theoretical derivation (Equation 2) computes the KV cache size reduction (37.5%) but notes this applies to “KV Cache” and “Attention Computation” only, without translating it to a predicted end-to-end speedup. The paper’s own analysis in Appendix G acknowledges that CEET “accounts for additional overhead, such as context switching and other CPU operations, making the improvement less pronounced” compared to KET, but this still only concerns attention-module measurement fidelity, not the gap between attention speedups and end-to-end speedups.

Mitigation status. Not addressed. The Abstract and Section 1 present the 33.36% figure without the qualification that it is component-level. Section 3.1 notes theoretically that the reduction applies to “two critical efficiency indicators: KV Cache and Attention Computation,” but the surrounding text and headline figures do not consistently distinguish between attention-level and system-level speedups. No end-to-end benchmarks are proposed as future work.

Difficulty Estimation and Allocation Policy Selection Are Not Addressed; the System Domain Benchmark Evaluates Only Task-Specific Fine-Tuned Models, Not the Adaptive Inference Strategy That DiffQKV Enables

The assumption or constraint. The paper introduces DiffQKV as a static architectural modification—the head counts and dimensions are fixed at design time and baked into the pre-trained model weights. There is no mechanism for dynamically adjusting the compression ratios or augmentation levels based on input characteristics, context length, or computational budget. The efficiency analysis in Section 3 shows that DiffQKV’s benefits are strongly context-length-dependent (SIGMA is slower than GQA at short contexts, faster at long contexts), but the architecture provides no way to switch between a GQA-like mode and a DiffQKV mode at inference time based on the actual sequence length. The model always pays the augmented Q overhead and always operates with compressed K heads, regardless of whether the current request is a 1K-token short prompt (where DiffQKV is slower) or a 64K-token long document (where it excels).

The consequence. In a production serving system handling a mix of short and long requests, SIGMA would underperform GQA on the short requests (due to fixed AugQ overhead) while providing the claimed benefits only on long requests. An ideal deployment would adaptively select between GQA and DiffQKV configurations per request or even per layer, but the paper provides no mechanism for this. More broadly, unlike the reference paper’s compute-optimal scaling framework, SIGMA has no “difficulty estimation” or “budget allocation” component—it offers a single static efficiency-performance trade-off point rather than a family of operating points that can be selected at inference time based on context or quality requirements. The system domain evaluation compounds this: AIMICIUS tasks involve structured technical contexts of varying lengths, but the evaluation uses task-specific SFT models without measuring inference efficiency in the deployment scenarios where DiffQKV’s benefits would actually be realized.

What evidence exists in the paper. Figure 2 shows the cross-over pattern explicitly: SIGMA is slower than STD at short contexts and faster at long contexts. The paper acknowledges this in Section 3.3: “SIGMA is slightly less efficient than the standard model in short-context scenarios (~10³ ms), but it shows significant advantages in long-context scenarios (~10⁶ ms).” However, it does not discuss the practical implication that a single deployed model must serve both regimes and will be suboptimal in one of them. The system domain evaluation (Table 6) reports only accuracy metrics—there are no latency, throughput, or memory consumption measurements for AIMICIUS tasks. The paper never evaluates SIGMA’s inference characteristics on the very tasks it was designed to excel at.

Mitigation status. Not addressed. The paper does not propose adaptive head count switching, dynamic Q augmentation, or any mechanism for context-dependent architecture configuration. Section 9 (Future Work) mentions “varied key-value (KV) heads compression across layers” as an area for investigation but does not discuss dynamic per-request adaptation. The disconnect between the efficiency motivation (Section 3) and the system domain evaluation (Section 5) is a structural gap in the paper’s narrative. A practitioner cannot determine from the reported results whether deploying SIGMA-SYSTEM for AIMICIUS tasks would actually be faster or more memory-efficient than deploying a GQA model fine-tuned on the same data.

The System Domain Benchmark (AIMICIUS) Has Very Small Test Sets, Especially NL2KQL (43 Samples), Making Statistical Reliability of the Dramatic Claims Uncertain

The assumption or constraint. The AIMICIUS benchmark consists of 395 test cases for CMDGen (200 NVIDIA + 195 AMD), 911 test cases for Infrawise, 1,258 test cases for Optiflow, and only 43 test cases for NL2KQL (Section 4, Appendix C). The paper reports no confidence intervals, no standard deviations, no significance tests, and no cross-validation for any of these results. The performance numbers in Table 6 are point estimates from single evaluation runs.

The consequence. On NL2KQL with 43 test samples, each sample corresponds to approximately 2.3 percentage points of reported accuracy. A difference of 4–5 points between models (e.g., SIGMA’s 34.9% Similarity vs. Qwen’s 36.7%) could arise from as few as 2 misclassified samples—well within the range of sampling noise for a 43-sample test. The dramatic headline claim of “surpassing GPT-4 by up to 52.5% across all tasks” (Abstract) aggregates across tasks with vastly different sample sizes, and the largest absolute improvements occur on tasks where the baseline models perform near zero (e.g., Optiflow Plan Improved at 0.5% for GPT-4 vs. 66.7% for SIGMA). While the direction of improvement is unambiguous (SIGMA-SYSTEM clearly outperforms baselines on system domain tasks), the magnitudes of the differences—and hence the specific 52.5% figure—are noisy estimates, especially for the smaller test sets.

More subtly, without cross-validation or held-out evaluation, it is impossible to know whether SIGMA-SYSTEM’s SFT procedure overfit to the specific test examples, particularly for NL2KQL where the training set (5,166 instruction-tuning examples) is two orders of magnitude larger than the test set (43 examples). The paper does not describe the data split procedure for AIMICIUS—were test examples drawn from the same distribution as training examples? Could there be leakage? These are standard concerns for small custom benchmarks that the paper does not address.

What evidence exists in the paper. The test set sizes are reported in Section 4 and Appendix C. The NL2KQL dataset description states: “The NL2KQL dataset includes a total of 5,166 instruction-tuning examples and 43 test cases” (Section 4). The ratio of training to test samples (120:1) is unusually high and raises overfitting concerns. No statistical methodology is described for any AIMICIUS evaluation beyond the metric definitions in Appendix C.

Mitigation status. Not addressed. The paper does not acknowledge the small test set sizes as a limitation, does not report any measure of statistical uncertainty, and does not describe any steps taken to prevent test set leakage during SFT data construction. The Appendix D examples provide qualitative illustrations but no quantitative reliability analysis. For a benchmark that the paper introduces and promotes as a contribution (“the first comprehensive system-domain benchmark”), the absence of statistical rigor weakens the quantitative claims built on it.

The Differential Compressibility Findings Are Extrapolated from 1B-Scale, 100B-Token Experiments Without Validation at Production Scale or Across Model Families

The assumption or constraint. All controlled architectural ablations (Tables 1–4, 8–13) use models with ~1B parameters (22 layers, hidden dim 2048) trained on 100B tokens from FineWeb-Edu. The paper then applies the conclusions from these small-scale experiments to design SIGMA-1.5B (26 layers, hidden dim 2048, trained on 6T tokens) and SIGMA-10B (32 layers, hidden dim 4096, trained on 6T tokens). The implicit assumption is that the differential sensitivity of K vs. V to head count reduction, the viability of halving K head dimension, the optimal 1.5× Q augmentation ratio, and the additivity of Q augmentation and FFN expansion all scale smoothly from 1B to at least 10B parameters and from 100B to 6T training tokens.

The consequence. Several of the key design choices in SIGMA’s architecture could be suboptimal at production scale. The 1.5× Q augmentation ratio was selected based on three tested values (1.5×, 2.0×, 2.75×) at 1B scale, where 1.5× gave the best gain per parameter. At 10B scale, the optimal ratio might differ—larger models have more capacity and might benefit from proportionally larger Q augmentation, or alternatively might saturate faster. The K-to-V head ratio of 4:16 was selected because at 1B scale, reducing K from 16 to 4 heads cost only 0.17 points (Table 1). But at 10B scale with 6T tokens, the model may learn more sophisticated attention patterns that rely on finer-grained K head diversity, making aggressive K compression more costly than the small-scale experiments suggest. Conversely, larger models might be even more tolerant of K compression due to greater redundancy. Neither direction has been validated.

More fundamentally, the entire DiffQKV design principle—that K compression is cheap because attention is sparse, while V compression is expensive because V directly influences output—might interact with model scale in ways the 1B experiments cannot reveal. At larger scales, attention patterns may become less sparse (as the model learns to integrate information from more tokens), or the effective rank of V representations may increase (making compression more costly). The paper provides no theoretical model or scaling trend to predict how the K/V sensitivity gap evolves with scale.

What evidence exists in the paper. The 1B-scale experiments are the only architecture-only comparisons. The paper provides no ablation of K/V sensitivity at 1.5B or 10B scale, and no comparison of different head count ratios or augmentation sizes at production scale. The general-domain evaluation (Tables 7, 17) compares the final SIGMA architecture against other models with different architectures, different data, and different training recipes—it cannot isolate whether the specific architectural choices (4 K heads, 16 V heads, 1.5× Q) are near-optimal or merely adequate.

Mitigation status. Partially acknowledged in future work. Section 9 notes that “further optimization on the architecture of SIGMA has not been fully explored” and lists “appropriate hyper-parameters for scale up” as a key area for investigation. However, this framing presents the limitation as an optimization opportunity rather than a fundamental uncertainty about whether the 1B-scale findings transfer. It does not acknowledge that the core design choices might need to be re-measured at each target scale rather than extrapolated. The paper’s recommendations (4 K heads, 16 V heads, 1.5× Q augmentation) are presented as general guidelines, but the evidence only supports them at 1B scale on 100B tokens.

Selective V Cache Fetching and K Dimension Compression Are Promising Techniques That the Production Models Do Not Deploy, Leaving Significant Potential Efficiency Gains on the Table Without Characterization of the Trade-offs

The assumption or constraint. The paper introduces two additional efficiency techniques beyond K head count reduction: (1) selective V cache fetching (loading only the V vectors corresponding to the top-100 attention scores, Appendix B.2) and (2) K head dimension compression (halving $d_h^k$ relative to $d_h^v$ , Observation 2 in Section 2). Both are shown to preserve model quality in the 1B-scale experiments (Tables 2 and 8). However, the production SIGMA models (1.5B and 10B) use neither of these techniques. Section 3.2 states: “for the sake of balancing the model performance and the cost of the KV cache, during the training of SIGMA-1.5B and SIGMA-10B, no dimension compression is applied to the K heads. Only the number of K heads was decreased.” Selective V cache fetching is described in Appendix B.2 as an additional optimization but is never integrated into SIGMA’s inference pipeline or benchmarked in the efficiency analysis (Section 3), which only measures K head count reduction and augmented Q.

The consequence. The paper’s efficiency analysis demonstrates a 33–37% reduction in attention cost from reducing K heads from 16 to 4. But the maximum potential efficiency gain from the full DiffQKV design space—including K dimension compression (which could further halve the K cache size) and selective V fetching (which could dramatically reduce V cache bandwidth at long contexts)—is never characterized. A practitioner reading the paper sees promising ablation results for these techniques but no guidance on why they were excluded from production, what trade-offs led to that decision, or what additional gains would be possible if they were deployed. The omission is particularly notable for selective V cache fetching because its benefit scales with context length (more tokens means more V vectors can be skipped), making it complementary to the long-context regime where DiffQKV’s K head reduction is already most beneficial. Combining aggressive K compression with selective V fetching could potentially push the total KV cache bandwidth reduction well beyond the 37.5% figure, but the paper does not explore this combination.

What evidence exists in the paper. Table 8 shows that top-100 V selection preserves performance within 0.01–0.30 points across three baseline architectures. Table 2 shows that halving K head dimension preserves performance within 0.08–0.26 points across three baselines. Neither technique appears in the efficiency measurements (Figures 1–6) or in the production model specifications (Table 14). The paper provides no explanation for why dimension compression was excluded beyond the vague “balancing” statement, and no explanation at all for why selective V fetching is not part of the production system.

Mitigation status. Not addressed. These are presented as positive findings that motivate DiffQKV’s design principles but are then set aside without characterizing the cost of deploying them or the magnitude of the missed opportunity. A reader cannot determine whether including them would have provided marginal additional gains (explaining their exclusion as a practical simplification) or substantial further improvements (making their omission a significant limitation of the current SIGMA models). The Future Work section (Section 9) does not mention either technique or propose integrating them into production.

7. Implications and Future Directions

How This Work Changes the Landscape

The primary conceptual shift this paper introduces is the recognition that the three components of self-attention—Query, Key, and Value—occupy fundamentally different positions in the inference cost model and exhibit fundamentally different sensitivities to compression, and that these asymmetries should drive architectural design rather than being suppressed by symmetry constraints. This is not a paradigm shift in the sense of upending the transformer architecture; it is a diagnostic reframing that converts what was previously a uniform compression problem ("how much can we shrink the KV cache?") into a differential allocation problem ("where should we remove capacity, and where should we add it?").

The magnitude of this shift is modest but genuine. The paper does not propose a new attention mechanism in the way that MHA, MQA, or GQA did—DiffQKV is formally a generalization of GQA that removes the equality constraint $n_k = n_v$ . What it does is provide the empirical justification for lifting that constraint, showing through controlled 100B-token experiments (Tables 1, 9) that the sensitivity ratio between K and V head reduction is consistently 2–4× in V's disfavor across multiple baseline architectures and compression levels. This measurement had simply not been done before, and its absence allowed the field to treat K and V as symmetric by default. The paper changes the default assumption from "compress K and V equally" to "measure their differential sensitivity first." This is the kind of empirical contribution that reorients a research subfield without requiring a theoretical breakthrough—it tells architecture designers what to measure and where to look for efficiency gains.

A secondary shift is the recognition that the Query vector occupies a privileged, uncached position in the inference pipeline, making it an ideal target for additional parameter investment that improves quality without increasing the dominant inference bottleneck (memory bandwidth for KV cache loading). Observation 4's finding that Q augmentation is more parameter-efficient than FFN expansion (+1.24 vs. +1.12 per $\delta$ parameters on the GQA-16 baseline, Table 4) challenges the standard efficiency narrative of "reduce parameters everywhere." It suggests instead a reallocation strategy: take parameters from K (where they provide marginal benefit and incur high inference cost) and move them to Q (where they provide disproportionate benefit and incur only computation cost). This reframes efficient architecture design from a compression problem to a budgeting problem with different "tax rates" for different parameter locations.

The paper also reconciles a latent tension in the efficient inference literature between aggressive KV compression and model quality preservation. Prior work on MQA and GQA achieved efficiency by uniformly reducing K and V head counts, accepting a quality degradation that grew with compression ratio. The paper's differential approach suggests that a significant fraction of this degradation may have been unnecessary—attributable to V compression that could have been avoided by compressing K more aggressively instead. If this finding transfers to larger scales (a significant open question, as discussed in the Limitations section), it would mean that the Pareto frontier of efficiency vs. quality is better than previously measured, because prior work operated on a constrained subspace (where $n_k = n_v$ ) rather than the full DiffQKV design space.

A concrete way this changes the landscape: future efficient architecture proposals that treat K and V identically will now need to justify that choice against the differential compressibility evidence this paper provides. The burden of proof has shifted—symmetry is no longer the default assumption; it must be argued for. This is a meaningful, if incremental, contribution to architectural design methodology.

Finally, the paper identifies the system domain as a distinct capability category that general-purpose LLMs systematically fail on, and that requires both domain-specific pre-training data and domain-specific evaluation (AIMICIUS). While this is not a contribution of the DiffQKV architecture itself, the benchmark reveals a capability gap—GPT-4 achieving 25% Accuracy on CMDGen and 0.5% on Optiflow Plan Improved (Table 6)—that suggests the system domain may be a useful diagnostic for measuring whether efficiency-focused architectures sacrifice the precise, domain-specific reasoning that infrastructure management requires. The fact that SIGMA-SYSTEM excels on these tasks while using an efficiency-oriented architecture (with aggressive K compression) provides suggestive evidence that efficiency and domain specialization are not inherently in tension, though the architecture-data confound (discussed in Limitations) prevents a clean conclusion.

Follow-Up Research This Work Enables

Training a GQA-architecture model on SIGMA's exact 6T token data mixture to isolate the architectural contribution to model quality. The most important missing experiment from this paper is a head-to-head comparison at production scale (1.5B or 10B parameters, 6T tokens) between the DiffQKV architecture and a matched GQA architecture trained on identical data with identical hyperparameters. The 100B-token, 1B-scale ablations show that the best DiffQKV configuration achieves 52.97 vs. GQA-16's 52.14 on the 9-benchmark average (Table 13)—a gain of only 0.83 points on identical data. Whether this gap widens, narrows, or disappears at 1.5B/6T-token scale is the single most important open question about DiffQKV's contribution to model quality (as opposed to inference efficiency, which is demonstrated at the component level but confounded with data quality in the general-domain evaluations). A follow-up study should train at least three models on the exact 6T token mixture: (1) the SIGMA-1.5B DiffQKV configuration ( $n_k=4$ , $n_v=16$ , $d_h^q=3072$ ), (2) a GQA baseline with $n_k=n_v=16$ and standard Q dimension, and (3) a GQA baseline with $n_k=n_v=16$ and augmented Q matching SIGMA's Q parameter count, to separate the effects of K compression and Q augmentation. Evaluate on both the general-domain benchmarks (Tables 7, 17) and the system-domain benchmarks (Table 6) after equivalent fine-tuning. This experiment would definitively answer whether DiffQKV improves model quality, hurts it but is compensated by efficiency gains, or is neutral.

Measuring end-to-end generation throughput and latency for SIGMA vs. GQA at production batch sizes and sequence lengths, including the FFN and other non-attention components. The paper's 33.36% speedup figure applies only to the attention component in isolation. A production deployment cares about end-to-end tokens-per-second at various batch sizes (1, 8, 32, 128) and context lengths (1K to 128K tokens). A rigorous follow-up would deploy both SIGMA and a matched GQA model in vLLM or TensorRT-LLM (with FlexHeadFA support for SIGMA), measure end-to-end prefill latency, decode latency per token, and total generation throughput, and report the actual speedup as a function of batch size, sequence length, and hardware (H100, A100). This would also characterize the cross-over point where SIGMA's fixed AugQ overhead is outweighed by its K cache bandwidth savings—Figure 2 shows this occurs around 2K–16K prefix length depending on output length, but end-to-end measurements would give practitioners an actionable deployment decision rule. Additionally, measuring peak memory consumption of the KV cache for both models at various sequence lengths would validate the theoretical 37.5% cache size reduction with actual GPU memory measurements.

Systematic characterization of K/V sensitivity ratios across model scales from 100M to 10B parameters to determine whether the differential compressibility finding scales. The paper's core empirical finding—that V is 2–4× more sensitive to head count reduction than K—is established only at ~1B parameters on 100B tokens. A scaling study should train DiffQKV-configurable models at 100M, 300M, 1B, 3B, and 10B parameters (all on the same data distribution, e.g., FineWeb-Edu or a controlled 100B-token subset), and for each scale independently measure the performance impact of reducing $n_k$ from the GQA baseline (where $n_k = n_v = n_q/2$ ) to $n_q/8$ , $n_q/16$ , and 1, while keeping $n_v$ fixed, and vice versa. The research question is: does the K-to-V sensitivity ratio increase, decrease, or remain constant with scale? If it increases (V becomes even more sensitive relative to K at larger scales), then the DiffQKV approach becomes more valuable for large models. If it decreases, then the 10B SIGMA may be over-compressing K relative to what the sensitivity measurements at that scale would recommend. This study would also determine whether the optimal Q augmentation ratio (1.5× in the 1B experiments) shifts with scale—plausibly larger models could benefit from proportionally larger Q augmentation due to greater capacity to utilize the extra representational power.

Combining differential K compression with selective V cache fetching at long contexts (32K–128K tokens) to measure whether the compound efficiency gain exceeds the sum of individual gains. The paper demonstrates both techniques independently at 1B scale (Tables 1 and 8) but never combines them in the efficiency analysis. Selective V fetching's benefit should scale with the V cache size—at 64K tokens, loading only top-100 V vectors out of 64K reduces V cache bandwidth by a factor of 640×. When combined with K head reduction (which already reduces total cache size by 37.5%), the compound effect could be multiplicative rather than additive: the K cache is 4× smaller (fewer heads) and the V cache bandwidth is ~640× smaller (selective loading), potentially pushing total KV cache bandwidth reduction well beyond 90% at very long contexts. A follow-up study should implement end-to-end selective V fetching in FlexHeadFA (the current kernel does not support it), benchmark the combined efficiency at context lengths from 8K to 128K, and measure the performance impact on long-context benchmarks (e.g., Needle-in-a-Haystack, LongBench) to determine whether top-100 V selection remains sufficient at very long sequences or whether the sparsity assumption (95%+ attention sparsity from Zhang et al., 2024b) breaks down.

Fine-tuning GPT-4 and other strong baselines on SIGMA's system domain SFT data to establish whether the AIMICIUS performance gap comes from domain data or from SIGMA's architecture. The paper shows that domain-specific pre-training + SFT dramatically outperforms zero-shot general models on AIMICIUS, but this conflates data and architecture. A clean ablation would take the same system domain SFT data used for SIGMA-SYSTEM-10B, fine-tune GPT-4 (via API fine-tuning if available, or an equivalent open model like Llama3-70B), and evaluate on AIMICIUS. The research question is: does a general architecture fine-tuned on system domain data approach SIGMA-SYSTEM's performance, or does DiffQKV provide a complementary benefit (e.g., through better long-context handling of system logs and configuration files)? Given that system domain tasks like CMDGen involve precise command syntax and Optiflow involves reasoning about structured hardware configurations, the K head compression in DiffQKV might either help (by reducing noise in attention) or hurt (by limiting the model's ability to attend to specific details in long technical contexts). This experiment would provide the first evidence about whether the DiffQKV architecture is particularly well-suited to the system domain or whether the domain simply benefits from any model exposed to domain data.

Testing whether the differential K/V compressibility finding holds for cross-attention in encoder-decoder architectures and for vision transformers. The paper's mechanistic explanation for K's compressibility—that attention matrices are sparse, so approximate routing suffices—is specific to self-attention in decoder-only LMs, where the attention pattern is causal and dominated by local and "sink" tokens (Xiao et al., 2023). In encoder-decoder cross-attention (where the query comes from the decoder and keys/values from the encoder), the attention pattern may be less sparse because the decoder needs to retrieve specific information from potentially any encoder position. Similarly, in vision transformers, attention patterns are often more globally distributed. A negative result in these settings—showing that K and V have similar sensitivity to compression—would refine the theoretical understanding of why K is more compressible in decoder self-attention and would bound the generalizability of the DiffQKV design principle. A follow-up should replicate the Table 1 experiment (differential K vs. V head reduction) for: (a) a T5-style encoder-decoder model on summarization or translation tasks, measuring cross-attention K/V sensitivity specifically, and (b) a ViT on ImageNet classification, measuring self-attention K/V sensitivity in the vision domain.

Practical Applications and Downstream Use Cases

Long-context LLM serving for document processing, code analysis, and multi-turn conversations where sequence lengths routinely exceed 16K tokens. The paper's efficiency measurements (Figure 2) show that SIGMA's advantage over GQA grows with both prefix and output length, reaching 33.36% attention cost reduction at 64K output length. In production systems serving document Q&A, repository-level code generation, or long conversational threads, the KV cache is often the dominant memory consumer and bandwidth bottleneck. Deploying models with DiffQKV's 4:16 K:V head ratio in these scenarios would reduce KV cache memory by 37.5%, allowing either larger batch sizes (improving throughput) or longer maximum context lengths (improving capability) on the same GPU hardware. The practical deployment would require FlexHeadFA integration into serving frameworks (vLLM, TensorRT-LLM), which the paper identifies as an infrastructure need but does not yet provide. For an organization running thousands of long-context queries per day, a 30%+ reduction in attention-layer cost translates directly to reduced GPU hours and lower serving latency. The key deployment caveat from the paper's evidence is that SIGMA is slower than GQA at short contexts (below ~2K–4K prefix, Figure 2a–c), so the benefit only materializes for workloads skewed toward long sequences.

On-device or edge deployment of small language models where KV cache memory is the binding constraint on context length. The SIGMA-1.5B architecture reduces KV cache size by 37.5% compared to an equivalent GQA model. On a memory-constrained device (e.g., a smartphone with 6–8GB of available RAM for model inference), this reduction could mean the difference between supporting 8K context vs. 5K context, or between fitting the model with KV cache in memory at all vs. requiring offloading. MobileLLM (Liu et al., 2024b) and similar efforts target sub-2B parameter models for on-device deployment, where every megabyte of KV cache matters. Integrating DiffQKV's asymmetric head configuration into these architectures—with the same aggressive K compression and moderate V preservation—could extend usable context length by 30–60% without increasing the model's memory footprint. The augmented Q component is particularly well-suited to on-device deployment because it adds computation (which mobile GPUs/NPUs have available) without adding memory pressure (which is the scarce resource).

Domain-specialized LLMs for infrastructure management and DevOps automation, where models must process long, structured technical documents (configuration files, logs, hardware specifications) and generate precise, executable commands. The AIMICIUS benchmark reveals that general-purpose models catastrophically fail at system domain tasks—GPT-4 achieves only 25% Accuracy on CMDGen command generation and 0.5% on Optiflow Plan Improved (Table 6). An organization deploying LLMs for automated infrastructure management (e.g., generating GPU monitoring commands, diagnosing cluster issues, optimizing network topology) could follow SIGMA's recipe: collect 20B+ tokens of domain-specific data from internal documentation, technical forums, and system logs; pre-train a SIGMA-architecture model on this data; and fine-tune on task-specific instruction data. The key value proposition of using a DiffQKV architecture rather than a standard GQA architecture for this deployment is the combination of (1) efficiency for processing long system logs and configuration files (where the long-context efficiency advantage applies) and (2) the augmented Q potentially providing better representational capacity for the precise, syntax-sensitive outputs that system commands require. The paper's data quality experiment (Table 16) shows that domain pre-training on the 19.5B system domain tokens provides 1.5–6.4 Accuracy point gains for Mistral and Llama3 on CMDGen, demonstrating that the domain data alone has significant value even without the DiffQKV architecture—but the architecture's efficiency benefits make it practically deployable for high-throughput infrastructure monitoring scenarios.

Training data generation pipelines where a model generates high-quality solutions that are then used to fine-tune the same or a different model (self-improvement loops), and where generation cost dominates the total compute budget. The paper's mention of "model self-evolution and lifelong learning" (Section 9) and the broader context of the system domain as "LLMs that can oversee their own training processes" (Section 4) point toward a use case where SIGMA models generate system management commands, benchmark configurations, or topology optimization plans that are then verified (by execution or by a verifier) and fed back into training. In such a pipeline, the generation phase is the primary cost driver, and inference efficiency directly determines how many candidate solutions can be generated within a budget. DiffQKV's long-context efficiency (33.36% faster attention at 64K output) would allow more extensive exploration of solution space within a fixed compute budget, potentially improving the quality of the generated training data. The caveat is that this benefit only applies if the generated outputs are long (thousands of tokens)—for short-answer tasks like CMDGen (where commands are typically 50–200 tokens), the cross-over pattern in Figure 2a suggests SIGMA might actually be slower than GQA, making it a poor choice for generation pipelines dominated by short outputs.

🎯 Pitch

1. Executive Summary

2. Context and Motivation

The Core Problem: KV Cache Is the Inference Bottleneck, But We Compress It Naively

The Gap: Why Treating K and V Identically Is a Mistake

Where Existing Approaches Fall Short

The Underexplored Role of Augmented Q

Why This Problem Matters: Practical Deployment and the System Domain

Reconciling Scale and Efficiency Goals

How the Paper Positions Itself Relative to Prior Work

3. Technical Approach

3.1 Reader Orientation

3.2 Big-Picture Architecture (Diagram in Words)

3.3 Roadmap for the Deep Dive

3.4 Detailed, Sentence-Based Technical Breakdown

DiffQKV Attention: Generalized Formulation

Observation 1: Differential Sensitivity of K vs. V Head Count

Observation 2: K Head Dimension Can Be Halved with Negligible Impact

Observation 3: Augmented Q Consistently Improves Performance

Observation 4: Augmented Q Outperforms FFN Expansion Per Parameter

Selective V Cache Fetching

SIGMA Model Architecture

FlexHeadFA Kernel and Deployment Infrastructure

Pre-training Data and Procedure

Summary of Design Choices and Their Justifications

4. Key Insights and Innovations

Innovation 1: Differential KV Compression as a First-Class Architectural Principle

Innovation 2: Augmented Q as a Compute-Only Parameter Investment Strategy

Innovation 3: The System Domain Benchmark as a Diagnostic Tool for Underexplored Capabilities

Innovation 4: Kernel-Level Support for Asymmetric Attention as an Enabling Contribution

Innovation 5: The Conceptual Generalization of Attention Mechanisms into a Unified Design Space

5. Experimental Analysis

Evaluation Methodology

Main Quantitative Results

Architectural Ablation Experiments at 1B Scale (Section 2)

Efficiency Analysis: SIGMA vs. Standard GQA Model (Section 3)

System Domain Performance on AIMICIUS (Section 5, Table 6)

General Domain Performance (Section 5, Tables 7 and 17)

Ablation Studies and Robustness Checks

Critical Assessment

Claim 1: Differential KV Compression Improves Inference Efficiency

Claim 2: Augmented Q Improves Performance with Minimal Inference Cost

Claim 3: SIGMA Matches State-of-the-Art General-Domain Models

Claim 4: SIGMA-SYSTEM Dramatically Outperforms GPT-4 on System Domain Tasks

Claim 5: DiffQKV Attention Is a Generalizable Design Principle

Genuine Weaknesses in Experimental Design

6. Limitations and Trade-offs

The Architecture-Data Confound: SIGMA’s General-Domain Performance Cannot Be Attributed to DiffQKV Alone

Inference Efficiency Gains Are Component-Level, Not End-to-End; the 33.36% Figure May Substantially Overstate Real-World Speedups

Difficulty Estimation and Allocation Policy Selection Are Not Addressed; the System Domain Benchmark Evaluates Only Task-Specific Fine-Tuned Models, Not the Adaptive Inference Strategy That DiffQKV Enables

The System Domain Benchmark (AIMICIUS) Has Very Small Test Sets, Especially NL2KQL (43 Samples), Making Statistical Reliability of the Dramatic Claims Uncertain

The Differential Compressibility Findings Are Extrapolated from 1B-Scale, 100B-Token Experiments Without Validation at Production Scale or Across Model Families

Selective V Cache Fetching and K Dimension Compression Are Promising Techniques That the Production Models Do Not Deploy, Leaving Significant Potential Efficiency Gains on the Table Without Characterization of the Trade-offs

7. Implications and Future Directions

How This Work Changes the Landscape

Follow-Up Research This Work Enables

Practical Applications and Downstream Use Cases

Adjacent in domain

Tensor Product Attention Is All You Need

FLEX ATTENTION: A PROGRAMMING MODEL FOR GENERATING OPTIMIZED ATTENTION KERNELS