SwiftKV: Fast Prefill-Optimized Inference with Knowledge-Preserving Model Transformation

🎯 Pitch

By rewiring later transformer layers to derive their KV cache from an earlier layer’s output, SwiftKV allows prompt tokens to skip up to half the model during prefill with barely any accuracy drop. This transforms compute-bound enterprise deployments—boosting aggregate throughput by 2× and cutting per-output-token time by 60%.

1. Executive Summary

SwiftKV introduces a model transformation and lightweight distillation procedure that reduces the prefill computation of large language models by rewiring later transformer layers to derive their KV cache from an earlier layer's output — enabling prompt tokens to skip those later layers entirely — while optionally merging the KV cache of consecutive skipped layers (AcrossKV) for additional memory savings. Evaluated across Llama-3.1, Mistral, Qwen2.5, and Deepseek-V2 model families on standard benchmarks (ARC-Challenge, Winogrande, HellaSwag, TruthfulQA, MMLU, and GSM8K), SwiftKV reduces prefill computation by 25–50% with minimal quality degradation (<1–2% average accuracy loss) while achieving up to 2× higher aggregate throughput and 60% lower time-per-output-token in end-to-end serving, culminating in a normalized 560 TFlops/GPU for Llama-3.1-70B-Instruct. The approach establishes that knowledge-preserving model transformation can substantially accelerate inference for prefill-dominant workloads only when the hidden states of deeper layers exhibit sufficient similarity — a condition the paper verifies empirically by showing that average input similarity between a layer and all subsequent layers exceeds 0.5 at roughly half the model depth, with performance degrading sharply when SwiftKV attempts to skip beyond 50% of layers.

2. Context and Motivation

The Core Problem: Enterprise LLM Workloads Are Prefill-Dominated

This paper addresses a specific mismatch between how modern LLM inference systems are optimized and how enterprise applications actually use them. The central observation — grounded in production telemetry at Snowflake — is that typical enterprise workloads consume far more input tokens than output tokens. Tasks like code completion, text-to-SQL generation, document summarization, and retrieval augmented generation (RAG) all share a common pattern: users submit long, context-rich prompts but receive relatively concise responses. The paper reports observing an average input-to-output token ratio of roughly 10:1, with mean prompt lengths between 500 and 1000 tokens in production deployments.

This matters because the prefill phase — where the model processes all input tokens in parallel before generating any output — dominates the total computational cost of serving in these scenarios. During prefill, every input token must flow through every transformer layer, computing query, key, value, attention, and MLP operations. For a prompt of 1000 tokens and a model with 32 layers, that translates to 32,000 layer-forward computations just to process the input context before a single output token is generated. If the model only produces 100 output tokens in response, roughly 91% of the floating-point operations are spent on prefill (1000 tokens × 32 layers for prefill versus 100 tokens × 32 layers for decode, in simplified accounting).

The practical consequences are threefold:

Throughput bottlenecks. Inference throughput — measured in total tokens processed per second — is gated by how quickly the system can chew through prefill computation. When a GPU is saturated processing large prompt batches, decode operations queue behind them, reducing aggregate throughput. The CPU-bound prefill phase becomes the rate-limiting step.

Latency degradation for interactive applications. In chatbots, coding assistants, and copilot-style interactions, users perceive responsiveness through two metrics: time-to-first-token (TTFT), which measures how long they must wait before any output appears, and time-per-output-token (TPOT), which determines how smoothly tokens stream once generation begins. Prefill computation directly inflates TTFT, and because most production serving systems (e.g., vLLM, SGLang) interleave prefill and decode on the same GPUs, heavy prefill also indirectly worsens TPOT by starving decode operations of GPU time.

Cost inefficiency at scale. For organizations serving millions of LLM requests daily, the FLOPs consumed during prefill translate directly to GPU-hours and cloud costs. Reducing prefill computation by 50% would halve the compute cost per request for prefill-dominant workloads — a direct operational savings with no change to the model architecture or pretraining budget.

The paper frames this as a workload-characteristic-driven optimization problem: the inference system should be designed to exploit the fact that enterprise prompts are long and outputs are short, rather than treating prefill and decode as symmetric operations requiring identical compute per token.

Why Prior Approaches Fall Short for This Specific Problem

The paper identifies three broad categories of existing inference acceleration techniques and explains why each fails to address prefill-dominant enterprise workloads effectively.

1. Model Pruning and Layer Skipping

Prior pruning methods (Ma et al., 2023; Sreenivas et al., 2024; Xia et al., 2024) remove weights or entire structural components from the model, followed by extensive post-training on 10–100 billion tokens to recover accuracy. For example, Nemotron-51B — derived from Llama-3.1-70B — uses neural architecture search guided pruning with 40 billion tokens of distillation to achieve a 28% reduction in model size. DarwinLM-8.4B (Tang et al., 2025) prunes Qwen2.5-14B to 8.4 billion parameters using 10 billion tokens of training data.

These approaches have two fundamental limitations for the prefill-dominant scenario:

They reduce compute for both prefill AND decode equally. Since pruning permanently removes capacity, every token — whether it appears in a 2000-token prompt or a single generated token — sees the same reduced model. In prefill-dominant workloads, this means much of the compute savings happens during decode, where it provides minimal benefit since decode is already a small fraction of total FLOPs. The compute that dominates — prefill — receives the same proportional reduction as everything else.

The recovery cost is enormous. Training on 10–100 billion tokens for hours or days on large GPU clusters is necessary because pruning removes model capacity wholesale, requiring the remaining parameters to relearn capabilities that the pruned components previously handled. This makes pruning expensive to apply to new models and impractical for rapid iteration.

Training-free layer-skipping methods (Jaiswal et al., 2024; Men et al., 2024; Yang et al., 2024; Ashkboos et al., 2024) avoid the expensive retraining but introduce different problems. FFN-SkipLLM (Jaiswal et al., 2024) adaptively skips feed-forward network (MLP) layers based on input similarity heuristics, without any fine-tuning. The paper's experiments show this approach works only for Llama-family models and even then only at modest skip rates (under 20% of layers). When applied to Mistral, Deepseek, or Qwen models, FFN-SkipLLM causes catastrophic accuracy degradation — for example, dropping from 78.23% to 45.71% average accuracy on Mistral-Small-Instruct, and from 64.12% to 24.83% on Deepseek-V2-Lite-Chat (Table 2). The paper attributes this brittleness to the heuristic nature of the approach: similarity-based skipping decisions are sensitive to architectural differences between model families and cannot be tuned without labeled data.

Crucially, both pruning and layer-skipping treat all tokens identically. SwiftKV's key insight is that prefill tokens and decode tokens have different computational requirements — the KV cache needed for future attention computation can be derived more cheaply for prefill tokens, while decode tokens still need full layer processing to produce accurate next-token predictions.

2. KV Cache Compression

A large body of work — including multi-query attention (MQA; Shazeer, 2019), grouped-query attention (GQA; Ainslie et al., 2023b), low-rank KV approximations (Chang et al., 2024), and KV quantization (Hooper et al., 2024) — focuses on reducing the memory footprint of the KV cache. These techniques address a real bottleneck: the KV cache grows linearly with sequence length, and for very long contexts (>100K tokens), it can consume more GPU memory than the model weights themselves.

However, the paper argues that KV cache compression addresses a memory bottleneck that is often not the binding constraint in production deployments. On modern datacenter GPUs like the NVIDIA H100 with 80GB of memory, the primary constraint for typical enterprise workloads (prompts of 500–2000 tokens, batch sizes of tens to hundreds) is compute throughput, not memory capacity. The FLOPs required to process all those tokens through all those layers dominate wall-clock time.

KV cache compression reduces memory usage but does not reduce the number of floating-point operations performed — in fact, techniques like quantization add overhead (conversion operations, dequantization kernels) that can slightly increase compute. The paper explicitly demonstrates this by constructing a thought-experiment baseline called Merge-all-Layers, which merges the KV cache of every layer into a single shared representation, eliminating virtually all KV cache memory. Table 3 shows that Merge-all-Layers achieves only a 10% throughput improvement over the unmodified baseline at 80GB memory — because memory was already sufficient, and the compute load remains unchanged. In contrast, SwiftKV without KV compression achieves a 35% improvement from reducing prefill FLOPs alone.

This is not to say KV cache compression is useless — it becomes critical in memory-constrained settings (edge devices, multi-tenant deployments with small per-request memory budgets). Rather, the paper positions KV cache compression as complementary to, not a substitute for, compute reduction. SwiftKV's AcrossKV feature provides memory savings on top of compute savings, and the paper shows that combining both is particularly effective in moderate-memory scenarios (Table 3, 20GB setting where 50% SwiftKV + 4× AcrossKV + FP8 quantization achieves near-optimal throughput).

3. Sparse Attention

Systems like ALISA (Zhao et al., 2024) and MInference (Jiang et al., 2024) exploit naturally occurring sparsity patterns in attention matrices — for example, the observation that most attention heads attend strongly to only a small subset of tokens — to reduce the quadratic cost of the attention operation. These methods can dramatically reduce attention FLOPs for very long sequences (>100K tokens) where attention dominates the total compute.

The paper identifies two limitations of sparse attention for the targeted enterprise workloads:

Attention is not the dominant operation at moderate sequence lengths. Table 1 provides the breakdown for Llama-3.1-70B: at standard sequence lengths, the attention operation accounts for approximately 160 GFlops per prefill token out of a total of 302 GFlops per prefill token — roughly 53%. The MLP operations consume 113 GFlops (37%), with the remaining 10% split between vocabulary projections and K,V projections. For the sub-100K token prompts typical of enterprise applications, attention is important but not overwhelmingly dominant. Reducing only attention leaves substantial compute on the table.

Sparse attention methods require custom kernels. Implementing dynamic sparsity patterns efficiently on GPU hardware often requires specialized CUDA kernels that are difficult to integrate into production serving frameworks (vLLM, SGLang) and may not compose well with other optimizations like tensor parallelism or chunked prefill. SwiftKV's design philosophy is explicitly to make "minimal changes to the model architecture" (Section 3.5) so that it can plug into existing serving infrastructure without new kernel development.

More fundamentally, SwiftKV skips all operations in later layers for prefill tokens — not just attention, but also query/output projections and MLPs. This means it reduces the 302 GFlops/pretill token by 25–50% across all operation types, not just the attention fraction.

4. Early Exit and Decoder-Head Methods

Prior work on early exit (Elhoushi et al., 2024; Schuster et al., 2022) adds auxiliary prediction heads at intermediate layers, allowing the model to produce output tokens without processing all layers. During decode, if the model is confident enough at layer 16 of a 32-layer model, it can emit the token immediately and skip layers 17–32.

While this reduces decode compute, it has a subtle interaction with KV caching that limits its applicability to prefill: if a token exits early, the KV cache for the skipped layers is never computed, which means future tokens on the autoregressive chain cannot attend to those layers' representations of that token. The paper notes that addressing this requires either (a) computing the KV cache for all layers anyway (defeating the purpose of early exit), (b) training additional mechanisms to reconstruct the KV cache later, or (c) accepting that future tokens will operate with incomplete attention — all of which introduce complexity or accuracy loss.

SwiftKV inverts this approach: rather than trying to exit early during decode, it pre-computes the KV cache for all layers using an early layer's output, so that decode tokens can still attend to complete KV representations while prefill tokens avoid the compute. The KV cache is fully populated; only the prefill forward pass is truncated.

Reconciling a Contradiction in the Literature

The paper's motivation is also rooted in a tension it identifies in prior research on hidden state similarity. Several works (Liu et al., 2024b; Gromov et al., 2024) observe that the hidden states of deep transformer layers become progressively more similar — meaning the representations computed at layer 20 look a lot like those at layer 25, which look a lot like those at layer 30. Gromov et al. (2024) provocatively titled their paper "The Unreasonable Ineffectiveness of the Deeper Layers," suggesting that later layers contribute marginal value.

If this is true, one might ask: why not simply remove those later layers entirely? The answer, as the paper's experiments confirm, is that skipping layers without adaptation destroys model quality. Simply rewiring the model to compute KV cache from an earlier layer with no fine-tuning (the "W/o Distill" row in Table 4a) produces a 2.64-point accuracy degradation compared to the distilled version. The layers are similar but not identical — the KV projection weights are tuned to expect specific input distributions, and feeding them a different layer's output produces misaligned representations.

This is where the paper's contribution becomes clear: the similarity of hidden states creates the potential for compute savings, but realizing that potential requires targeted knowledge recovery through lightweight distillation. The distillation described in Section 3.4 trains only the Q, K, and V projection matrices of the affected layers (<10% of total parameters) on <1 billion tokens, making it orders of magnitude cheaper than the prune-and-distill approaches that train all remaining parameters on 10–100 billion tokens. The paper positions this as discovering a sweet spot: the hidden states are similar enough that minimal adaptation can realign the projections, but different enough that naive rewiring fails.

Positioning: A New Design Point in the Inference Optimization Landscape

The paper explicitly positions SwiftKV as occupying an underexplored region in the space of inference optimization methods. The taxonomy implied by Section 2 can be reconstructed as follows:

Approach Category	Reduces Prefill Compute?	Reduces KV Cache Memory?	Requires Extensive Retraining?	Architecture-Agnostic?
Pruning + Distillation	Yes (but symmetric)	Yes (fewer layers)	Yes (10–100B tokens)	Partially
Training-Free Layer Skip	Yes (but symmetric)	No	No	No (brittle across families)
KV Cache Compression	No	Yes	No	Yes
Sparse Attention	Yes (attention only)	No	No	Requires custom kernels
SwiftKV	Yes (prefill-specific)	Yes (AcrossKV)	No (<1B tokens)	Yes (tested on 5 families)

The paper claims SwiftKV is the first method to achieve all four desirable properties simultaneously: prefill-specific compute reduction (exploiting workload asymmetry), memory reduction via KV cache sharing, low-cost adaptation (under 3 hours on 8 H100s for Llama-3.1-8B), and broad architecture compatibility (tested on dense, MoE, and latent-attention architectures spanning 3B to 405B parameters).

The positioning is not that SwiftKV fundamentally invents new techniques — layer skipping, knowledge distillation, and KV cache sharing all have precedents — but rather that it combines them in a way specifically optimized for prefill-dominant workloads, with an emphasis on practical deployability in production serving frameworks. The paper's integration with vLLM and SGLang (Section 3.5) and its open-source release of both training and inference code reinforce this engineering-first positioning.

A critical boundary condition the paper establishes: SwiftKV works because prefill tokens have different computational requirements than decode tokens. For prefill, the only output needed from later layers is the KV cache (to enable future attention), while for decode, the full forward pass through all layers is necessary to compute the next-token logits accurately. This asymmetry is what SwiftKV exploits, and it is the reason why prior symmetric methods (pruning, uniform layer-skipping) leave performance on the table for enterprise workloads.

3. Technical Approach

3.1 Reader Orientation

SwiftKV is a model transformation and lightweight distillation procedure that rewires a pretrained large language model so that during the inference prefill phase — when the model processes input prompt tokens — a fraction of the later transformer layers can be entirely skipped, with their key-value (KV) cache derived directly from an earlier layer's output instead of being computed through the normal forward pass. The system solves the problem of wasted prefill computation by exploiting the empirical observation that hidden state representations become increasingly similar across deeper layers, allowing a single early-layer representation to serve as a reasonable proxy for computing the KV cache of multiple subsequent layers, with only minimal fine-tuning needed to realign the affected projection weights to preserve quality.

3.2 Big-Picture Architecture (Diagram in Words)

The system consists of four major components working together:

Base pre-trained LLM (any standard transformer architecture — Llama, Mistral, Qwen, Deepseek-V2) — this is the starting point, a fully trained model with no structural modifications. All original weights are preserved.
SwiftKV rewiring logic — a structural modification that changes how the KV cache is populated for layers beyond a chosen cutoff index $l$ . Instead of each layer $j > l$ computing its own keys and values from its own input hidden state $x_j$ through its own $W_{KV}^j$ projection, all layers $j > l$ derive their KV cache from the output of layer $l$ (the last unskipped layer), using the formula $KV_j = W_{KV}^j \cdot x_{l+1}$ . This means prompt tokens can exit the forward pass at layer $l$ — the KV cache for all deeper layers is pre-computed in one shot from $x_{l+1}$ , and the query, attention, and MLP operations of those deeper layers are skipped for prefill tokens. Decode tokens continue to propagate through all layers normally.
Knowledge-preserving distillation procedure — since feeding $x_{l+1}$ into projection weights that were trained to expect $x_j$ from their own layer creates a distribution mismatch, the model's accuracy degrades unless the affected projection matrices are fine-tuned. The distillation procedure trains only the $W_{QKV}$ matrices (query, key, and value projections) of the skipped layers ( $j > l$ ) using a teacher-student setup where the original unmodified model serves as the teacher and the rewired model as the student. All other parameters — including MLP weights, attention output projections, layer norms, and the entire earlier layers — remain frozen. The loss is a standard distillation loss (Kullback-Leibler divergence between teacher and student output logits) rather than the standard language modeling loss, which the paper shows is critical for quality recovery.
AcrossKV KV cache compression (optional) — an additional mechanism that reduces KV cache memory by sharing a single layer's KV cache across multiple consecutive skipped layers, rather than computing separate caches for each. For example, with 4-way AcrossKV, the 16 skipped layers (in a 50% reduction configuration on a 32-layer model) are grouped into 4 groups of 4, with only one KV cache computed per group, yielding a 37.5% reduction in KV cache memory on top of the compute savings.

The information flow during inference is: (1) prompt tokens enter the model and propagate normally through layers 1 through $l$ , computing full attention and MLP operations at each layer; (2) at the boundary after layer $l$ , the output hidden state $x_{l+1}$ is used to compute the KV cache for all remaining layers $j > l$ in a single operation (or a small number of operations if AcrossKV is active); (3) the prompt tokens exit — no further computation is performed on them; (4) for decode tokens, the normal forward pass resumes, with each generated token flowing through all layers 1 through $L$ , attending to the pre-computed KV cache using standard attention. The first generated token's output logits come from the full final layer $L$ , just as in the unmodified model.

3.3 Roadmap for the Deep Dive

First: The core observation — hidden state similarity across depth. We will examine how the paper quantifies the phenomenon that deeper layers have increasingly similar representations, what this similarity score actually measures, and what the empirical values look like across different model families and sizes. This is the scientific foundation that motivates the entire approach — without this similarity, SwiftKV would not work.
Second: The SwiftKV rewiring mechanism. We will walk through the precise mathematical formulation (Equation 2) that defines how KV cache is projected from an early layer's output, what operations are skipped and what operations remain, and a concrete FLOPs breakdown (Table 1) that shows exactly where the savings come from for a representative model (Llama-3.1-70B). This establishes the compute model — the quantitative relationship between which layer is chosen as the cutoff and how much prefill FLOPs are eliminated.
Third: AcrossKV — cross-layer KV cache sharing. We will examine how the rewired architecture naturally enables merging KV caches across consecutive layers, the compression ratios this achieves, how it differs from prior cross-layer merging approaches, and the design tradeoffs between inter-layer sharing (AcrossKV) and intra-layer sharing (GQA/MQA).
Fourth: The knowledge recovery distillation procedure. This is the most methodologically novel component. We will walk through the teacher-student setup, the choice to train only $W_{QKV}$ (not MLP or other parameters), why distillation loss outperforms standard LM loss, the training hyperparameters (learning rate, dataset composition, sequence length, distillation temperature, optimizer settings), and the computational cost of this distillation relative to prior methods (prune-and-distill, full model fine-tuning).
Fifth: The inference implementation. We will cover how SwiftKV integrates into production serving frameworks (vLLM and SGLang), how it interacts with chunked prefill and PagedAttention, and the key implementation detail — that after layer $l$ completes, the KV cache for layers $>l$ is computed immediately, and only decode tokens propagate through the remaining layers — that makes the transformation compatible with existing serving infrastructure without custom kernels.
Sixth: Design choices and their justifications. We will synthesize the design decisions: why train only $W_{QKV}$ and not all parameters, why distillation instead of LM loss, why prefill-specific skipping instead of symmetric layer removal, why the cutoff layer is a hyperparameter rather than learned, and what alternative designs were considered or ablated.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and methodology paper whose core idea is that for prefill-dominant workloads, the computational requirements of prompt tokens can be substantially reduced by exploiting the similarity of hidden states across deep transformer layers, and that this reduction can be achieved through a simple architectural rewiring followed by lightweight distillation that trains only a small fraction of the model's parameters.

Foundation: Quantifying Hidden State Similarity Across Layers

The central empirical claim that motivates SwiftKV is that the hidden state representations of deep transformer layers become progressively more similar as depth increases. The paper formalizes this claim with a quantitative similarity metric and measures it across multiple model families and scales.

For a given layer $l$ , the input hidden state is denoted $x_l$ , and the $i$ -th token of that layer's input is $x_l^{(i)}$ . The paper defines a layer similarity score $\text{SimScore}(x_l)$ as the average cosine similarity between layer $l$ 's input and the inputs of all subsequent layers:

$\text{SimScore}(x_l) = \frac{\sum_{j=l+1}^{L} \text{Similarity}(x_l, x_j)}{L - l}$

where $L$ is the total number of transformer layers in the model, and $\text{Similarity}(x_l, x_j)$ is the average cosine similarity taken over all token positions $i$ between the hidden state vectors $x_l^{(i)}$ and $x_j^{(i)}$ .

What this computes: for each pair of layers $(l, j)$ where $j > l$ , the function computes the cosine similarity between every token's representation at layer $l$ and the same token's representation at layer $j$ , then averages over all tokens in the sequence. This per-pair score is a number in $[-1, 1]$ where 1.0 indicates the two layers produce identical representations (up to scaling) and 0.0 indicates orthogonal representations. The function then averages these per-pair scores across all subsequent layers $j > l$ to produce a single number characterizing how well layer $l$ 's representation serves as a proxy for all deeper layers' representations.

Why this form: cosine similarity is scale-invariant — it measures angular alignment independent of the magnitude of the hidden state vectors. This is critical because layer normalization in transformers controls vector magnitudes, meaning two layers could produce vectors of different lengths but identical directions, and a magnitude-sensitive metric like Euclidean distance would misleadingly report them as different. Cosine similarity isolates the directional information, which is what matters for downstream projection operations (like the KV projections), since these projections are applied after layer normalization and are thus sensitive to angular alignment. Averaging across all tokens and all subsequent layers produces a single summary statistic that captures the overall substitutability of an early layer's output for all later layers' inputs — exactly the property SwiftKV needs to exploit.

Empirical findings (Figure 2, left and middle panels). The paper presents SimScore curves for six models: three in the ~7-8B parameter range (Llama-3.1-8B-Instruct, Mistral-7B-Instruct-v0.1, Qwen2.5-7B-Instruct) and three in the ~70B parameter range (Llama-3.1-70B-Instruct, Mistral-Large-Instruct-2407, Qwen2.5-72B-Instruct). The key patterns:

All models show monotonically increasing SimScore with depth: layers at positions 25–30 have substantially higher similarity to subsequent layers than layers at positions 5–10. This is consistent with the hypothesis that later layers make progressively smaller modifications to the representation.
At roughly 50% of the model depth, average similarity exceeds 0.5 for all models. For Llama-3.1-8B-Instruct, the crossover occurs around layer 16 of 32. For the larger models (70B scale), the crossover is slightly later relative to depth. This threshold of 0.5 is the paper's heuristic justification for why 50% reduction is feasible — the representations at the halfway point are sufficiently aligned with all deeper layers that a single set of KV projections can, with modest fine-tuning, produce useful KV caches for the entire second half of the model.
Qwen models show lower similarity at 50–75% depth than Llama or Mistral. For Qwen2.5-7B-Instruct, SimScore at layer 16 (50% depth) is around 0.45, compared to ~0.55 for Llama-3.1-8B. This lower similarity correlates with SwiftKV's worse accuracy on Qwen models at 50% reduction (7.4% degradation vs. 1–2% for Llama, Table 2), suggesting that the similarity score is indeed predictive of how much prefill can be skipped before accuracy degrades.
Larger models show slightly higher similarity overall. The 70B-scale models have SimScore curves that sit noticeably above their 7B-scale counterparts, particularly in the early layers. For example, Llama-3.1-70B at layer 20 (of 80) shows SimScore around 0.5, while Llama-3.1-8B only reaches that level around layer 12 (of 32). The paper does not explicitly theorize why this occurs, but it is consistent with the hypothesis that larger models exhibit more representational redundancy — with more parameters per layer, each layer can learn a more complete and stable representation, reducing the need for subsequent layers to make dramatic modifications.

This similarity analysis is not used during inference — the cutoff layer $l$ is chosen as a fixed hyperparameter before deployment. The analysis serves as pre-deployment validation: if a model family shows low similarity at the desired reduction level, SwiftKV is unlikely to work well on that model without more extensive adaptation.

The SwiftKV Rewiring Mechanism

The core architectural modification is expressed in a single equation. For a chosen cutoff layer index $l$ (0-indexed, so the last unmodified layer is layer $l$ ), the KV cache for all subsequent layers $j > l$ is computed as:

$KV_j = W_{KV}^j \cdot x_{l+1}, \quad \text{for all } j > l$

where $x_{l+1}$ is the output hidden state of layer $l$ (equivalently, the input to layer $l+1$ ), $W_{KV}^j$ is the combined key-value projection weight matrix for layer $j$ , and $KV_j$ is the resulting key-value cache for layer $j$ .

What this computes: for each skipped layer $j$ , instead of that layer receiving its own input $x_j$ (computed from the previous layer's attention and MLP operations) and projecting it through $W_{KV}^j$ , the layer takes $x_{l+1}$ — the output from the last unskipped layer — and projects it through $W_{KV}^j$ . This produces a KV cache tensor that encodes what that layer "would have attended to" had it processed the tokens normally, but at the cost of only the matrix multiplication $W_{KV}^j \cdot x_{l+1}$ rather than the full attention and MLP forward pass. Importantly, $W_{KV}^j$ is the original projection weight of layer $j$ — it is preserved from the pretrained model, not newly learned or replaced. The weights are later fine-tuned during distillation, but the architecture of the projection remains the same.

Why this form: the key design choice is using a single source ( $x_{l+1}$ ) for all subsequent layers' KV caches rather than a cascaded approach where each layer computes its KV cache from the previous layer's (skipped) output. A cascaded approach — where layer $l+1$ receives $x_{l+1}$ , computes its outputs (but only the hidden state, not the full forward pass), and passes that to layer $l+2$ , etc. — would require that the skipped layers still perform their attention and MLP operations, defeating the compute savings. The single-source approach means only one hidden state vector per token needs to be broadcast to all subsequent KV projections, reducing the compute to essentially $(L - l)$ matrix multiplications of size $d \times 2d_k$ per token, where $d$ is the model dimension and $d_k$ is the per-head key/value dimension. Since these matrix multiplications are small relative to the MLP and attention operations they replace, the net FLOPs reduction is approximately proportional to $l / L$ — if half the layers are skipped, roughly half the prefill FLOPs are eliminated.

What operations are skipped. For each prompt token, in each layer $j > l$ , the following operations are not performed:

Query projection: $Q_j = W_Q^j \cdot x_j$ . Since the prompt token is not being used to produce an output at these layers (its only purpose is to populate the KV cache for future decode tokens), the query — which determines what the token attends to — is irrelevant. The token has no need to attend to anything at layers it is skipping.
Key and value projections from the layer's own input: $K_j = W_K^j \cdot x_j$ , $V_j = W_V^j \cdot x_j$ . These are replaced by the single computation $KV_j = W_{KV}^j \cdot x_{l+1}$ .
Attention computation: $\text{Attention}(Q_j, K_{1:j}, V_{1:j})$ . This is the quadratic operation that computes attention scores between the query and all previous keys. Since there is no query for prompt tokens at skipped layers, there is no attention computation.
Output projection: $O_j = W_O^j \cdot \text{Attention output}$ . This linear projection follows the attention operation.
MLP (feed-forward) computation: the two-layer feed-forward network $\text{MLP}(x) = W_2 \cdot \sigma(W_1 \cdot x)$ where $\sigma$ is an activation function (typically SiLU or GELU). For Llama-3.1-70B, the MLP intermediate dimension is typically 8/3 times the model dimension, making this the single most expensive operation per layer.

What is performed: only $W_{KV}^j \cdot x_{l+1}$ for each $j > l$ — a single matrix multiply per skipped layer to generate the KV cache entries.

Concrete FLOPs breakdown (Table 1). The paper provides a detailed operation-level FLOPs accounting for Llama-3.1-70B-Instruct with an 80-layer architecture. For a single prefill token in the baseline model, the total is 302 GFlops, broken down as:

Vocabulary projection: 4.3 GFlops (the embedding lookup and the final LM head projection, though the latter is only relevant for the last token)
K,V projections: 2.6 GFlops (the per-layer key and value projection matrices, applied 80 times)
Q,O projections: 22 GFlops (the per-layer query and attention output projection matrices, applied 80 times)
MLP: 113 GFlops (the per-layer feed-forward networks, applied 80 times)
Attention: 160 GFlops (the quadratic attention score computation, applied 80 times)

With 25% SwiftKV (skipping the last 20 of 80 layers), the total drops to 228 GFlops — a 24.5% reduction. The savings come from eliminating the Q,O projections (22 → 16 GFlops), MLP (113 → 85 GFlops), and attention (160 → 120 GFlops) for the skipped layers. The K,V projection cost remains unchanged (2.6 GFlops) because those projections still execute — they just take a different input.

With 50% SwiftKV (skipping the last 40 of 80 layers), the total drops to 154 GFlops — a 49.0% reduction. Here Q,O projections are halved (22 → 11 GFlops), MLP is halved (113 → 56 GFlops), and attention is halved (160 → 80 GFlops).

With 50% SwiftKV + 4× AcrossKV, the total drops slightly further to 153 GFlops, with K,V projections reducing from 2.6 to 1.7 GFlops (because only one set of K,V projections is executed per group of 4 layers rather than one per layer).

A critical detail visible in this breakdown: the attention operation accounts for only 53% of baseline FLOPs (160 out of 302). Even if attention were entirely eliminated (via sparse attention or other methods), at least 47% of the compute would remain from MLP, Q/O projections, and vocabulary projections. SwiftKV's advantage is that it reduces all these components proportionally — it does not target attention alone.

What the first generated token does differently. The very last input token (which immediately precedes the first generated token) must complete the full forward pass through all layers — it cannot skip layers $j > l$ . This is because that token's hidden state at the final layer $L$ is used to produce the logits for the next token prediction. However, the benefit remains because for a prompt of $N$ tokens, only 1 token pays this full cost while $N-1$ tokens skip the later layers. At the typical 10:1 input-to-output ratio, the overhead is negligible: for 1000 input tokens, 999 benefit from the reduction while 1 does not.

Right panel of Figure 2 — wall-clock validation. The paper provides a direct measurement of the forward pass time for Llama-3.1-8B-Instruct across batch sizes from 128 to 2048 tokens (the right panel of Figure 2). The baseline forward pass scales from approximately 100ms at batch size 128 to approximately 350ms at batch size 2048. With 50% SwiftKV ( $l = L/2$ ), the forward pass scales from approximately 60ms to 200ms — a consistent 40–45% reduction in wall-clock time. With 75% SwiftKV ( $l = L/4$ , the most aggressive setting), it scales from approximately 40ms to 125ms — roughly 65% reduction. This confirms that the theoretical FLOPs reduction translates linearly to wall-clock speedup, which is expected since the operations being eliminated (primarily matrix multiplications) are GPU-friendly and well-optimized, meaning there are no hidden bottlenecks from memory bandwidth or kernel launch overhead.

AcrossKV extends the SwiftKV rewiring to also reduce KV cache memory by having multiple consecutive skipped layers share a single KV cache. Without AcrossKV, each skipped layer $j > l$ computes and stores its own $KV_j$ using its own projection weights $W_{KV}^j$ . With AcrossKV, the skipped layers are partitioned into groups, and within each group, only one layer's projection is used to compute a shared KV cache that is reused by all layers in the group.

Mechanism. For a group size $g$ (the "way" of AcrossKV — e.g., 2-way, 4-way), the $L - l$ skipped layers are divided into $(L - l) / g$ groups. Within each group, one layer is designated as the representative, and its $W_{KV}^{\text{rep}}$ is used to compute the shared KV cache from $x_{l+1}$ :

$KV_{\text{group}} = W_{KV}^{\text{rep}} \cdot x_{l+1}$

All layers in the group use this same $KV_{\text{group}}$ as their KV cache, rather than computing separate caches.

Memory reduction. In a standard GQA model (grouped-query attention), the KV cache per layer consumes memory proportional to $2 \cdot d_k \cdot \text{num\_kv\_heads}$ per token. With $L$ layers, the total KV cache memory is:

$M_{\text{baseline}} = L \cdot 2 \cdot d_k \cdot h_{kv} \cdot S$

where $S$ is the sequence length and $h_{kv}$ is the number of key-value heads (fewer than query heads in GQA). With $g$ -way AcrossKV, the skipped portion uses only $(L - l) / g$ sets of KV caches rather than $L - l$ sets, while the unskipped layers ( $1$ through $l$ ) retain their full KV caches. The total becomes:

$M_{\text{AcrossKV}} = \left(l + \frac{L - l}{g}\right) \cdot 2 \cdot d_k \cdot h_{kv} \cdot S$

For a 50% SwiftKV configuration on Llama-3.1-8B-Instruct (32 layers, $l = 16$ ):

2-way AcrossKV ( $g = 2$ ): the 16 skipped layers use 8 KV caches, total KV cache layers = 24, 25% reduction in KV cache memory.
4-way AcrossKV ( $g = 4$ ): the 16 skipped layers use 4 KV caches, total KV cache layers = 20, 37.5% reduction in KV cache memory.
8-way AcrossKV ( $g = 8$ ): the 16 skipped layers use 2 KV caches, total KV cache layers = 18, 43.75% reduction in KV cache memory.
16-way AcrossKV ( $g = 16$ ): all 16 skipped layers share a single KV cache, total KV cache layers = 17, 46.875% reduction in KV cache memory.

Relationship to prior cross-layer sharing. Prior work by Liu et al. (2024a) (MiniCache) showed that KV caches can be merged for certain pairs of adjacent layers. AcrossKV differs in two ways: (1) it can merge more than two layers into a single shared cache, enabling higher compression ratios; (2) it simplifies implementation because the sharing is predetermined by the SwiftKV rewiring rather than requiring similarity analysis between specific layer pairs. During training, the AcrossKV layers are treated as additional trainable parameters (their $W_{KV}$ projections are among the $W_{QKV}$ weights that are fine-tuned), with the rest of the model remaining frozen.

Interaction with GQA. The paper notes an interesting design choice explored in Appendix B.2 (Table B.2): when AcrossKV provides $g$ -way compression, this compression can be allocated either between layers (AcrossKV) or within layers (by reducing the number of key-value heads, i.e., converting GQA to MQA). For a fixed 37.5% KV cache reduction, three configurations were tested on Llama-3.1-8B with 50% SwiftKV:

Pure MQA: all attention heads share a single KV head per layer, with no cross-layer sharing. Accuracy: 54.13% average — a catastrophic 17-point drop from the AcrossKV-GQA variant.
AcrossKV with MHA (multi-head attention): full query and key-value heads per layer, but 4-way cross-layer sharing to achieve the compression. Accuracy: 69.76% average.
AcrossKV with GQA (the paper's default): GQA's existing head compression plus 4-way cross-layer sharing. Accuracy: 71.49% average — the best of the three.

This ablation demonstrates that cross-layer sharing (AcrossKV) is more parameter-efficient than intra-layer head reduction (MQA) for achieving KV cache compression. The paper hypothesizes that this is because cross-layer sharing preserves more distinct attention patterns — each layer still has multiple key-value heads, allowing it to capture different types of attention relationships — whereas MQA forces all attention heads to share a single key-value representation, collapsing the diversity of attention patterns.

Accuracy impact of increasing AcrossKV compression (Table 2). For Llama-3.1-8B-Instruct with 50% SwiftKV, accuracy degrades gradually as the compression ratio increases: 72.70% (no AcrossKV) → 71.82% (2-way) → 71.49% (4-way) → 70.50% (8-way) → 70.22% (16-way). The total degradation from base (72.70%) to 16-way (70.22%) is only 2.48 percentage points, while KV cache memory has been nearly halved. This is a remarkably flat scaling curve — the model retains substantial capability even when all 16 skipped layers share a single KV cache. However, for Deepseek-V2-Lite-Chat, the degradation is steeper: 63.51% (no AcrossKV) → 63.07% (2-way) → 59.32% (4-way) — a 4.19-point drop at 4-way compression. The paper attributes this to Deepseek-V2's novel latent attention mechanism, which may involve more layer-specific KV representations that are less amenable to sharing.

Complementarity with KV quantization (Appendix B.1, Table B.1). AcrossKV is demonstrated to combine orthogonally with per-token FP8 KV cache quantization. For Llama-3.1-8B-Instruct:

50% SwiftKV + 4-way AcrossKV: 71.49% average accuracy, 37.5% KV cache memory reduction
50% SwiftKV + 4-way AcrossKV + FP8 quantization: 71.35% average accuracy, 68.75% KV cache memory reduction

The combined approach achieves nearly 2/3 reduction in KV cache memory with only an additional 0.14-point accuracy loss beyond what AcrossKV alone incurs. Crucially, this quantization is applied post-training with no quantization-aware fine-tuning, demonstrating that the SwiftKV-distilled model's KV cache representations are robust to aggressive compression.

Knowledge Recovery Through Lightweight Distillation

The SwiftKV rewiring changes the input distribution to the KV projection matrices for layers $j > l$ . Originally, layer $j$ 's projections receive $x_j$ — the output of layer $j-1$ 's attention and MLP operations. After rewiring, they receive $x_{l+1}$ — the output of a much earlier layer. Since the projections were trained to map from $x_j$ (with its specific statistical properties — mean, variance, directional correlations) to useful key-value representations, feeding them $x_{l+1}$ (which has different statistics) produces degraded KV caches and, consequently, degraded model outputs.

The paper shows (Table 4a, "W/o Distill") that without any fine-tuning, 50% SwiftKV on Llama-3.1-8B-Instruct produces an average accuracy of 70.06% versus 72.70% with distillation — a 2.64-point gap. The degradation is particularly severe on generative tasks: MMLU-CoT drops from 69.73% to 65.60% (4.13 points) and GSM-8K drops from 79.45% to 72.71% (6.74 points). These tasks require coherent multi-step reasoning, suggesting that the distribution mismatch most severely impacts the model's ability to maintain reasoning chains across the truncated layers.

The distillation procedure is designed to be as lightweight as possible while recovering this lost quality. It has three key design elements:

The Teacher-Student Architecture Setup

The distillation uses a dual-mode architecture where the original model and the SwiftKV-rewired model share the same underlying parameters but operate in different computational graphs:

$y_{\text{teacher}} = M(x, \text{SwiftKV} = \text{False})$ $y_{\text{student}} = M(x, \text{SwiftKV} = \text{True})$

where $M$ is the model, $x$ is the input, and $y_{\text{teacher}}$ and $y_{\text{student}}$ are the output logits (unnormalized prediction scores over the vocabulary). The $\text{SwiftKV}$ flag toggles whether layers $j > l$ receive their own computed inputs ( $\text{False}$ — the original architecture where each layer processes $x_j$ ) or the boundary output $x_{l+1}$ ( $\text{True}$ — the SwiftKV-rewired architecture).

What this computes: both the teacher and student logits are $d_{\text{vocab}}$ -dimensional vectors where each entry represents the model's unnormalized score for a particular token. The teacher logits come from the original, unmodified architecture — this is the "correct" output distribution that the student should learn to match. The student logits come from the rewired architecture — this is the distribution that will be used at inference time and that needs to be calibrated to match the teacher.

Why this dual-mode setup: it is a memory-efficient implementation trick. Since the teacher and student share all parameters except the $W_{QKV}$ weights for layers $j > l$ (which are duplicated — one frozen copy for the teacher, one trainable copy for the student), only one full copy of the model needs to reside in GPU memory. The trainable $W_{QKV}$ parameters are typically less than 10% of total model parameters for popular GQA architectures like Llama, Mistral, and Qwen. This is because these models allocate most parameters to the MLP layers (feed-forward networks) and the attention output projections, while the query, key, and value projections are relatively small — in Llama-3.1-8B, $d = 4096$ , $d_k = 128$ , and $h_{kv} = 8$ , so $W_{KV} \in \mathbb{R}^{4096 \times 1024}$ (4.2M parameters) and $W_Q \in \mathbb{R}^{4096 \times 4096}$ (16.8M parameters), totaling about 21M parameters per layer versus the MLP's ~67M parameters. For half the layers of an 8B model, the trainable portion is ~340M parameters out of ~8B total — roughly 4.25%.

The paper reports concrete training costs: Llama-3.1-8B-Instruct is distilled on 680M tokens in 3 hours using 8 H100 GPUs; Llama-3.1-70B-Instruct is distilled in 5 hours using 32 H100 GPUs across 4 nodes. These are remarkably short training times compared to prune-and-distill methods that require 10-100B tokens and days of training on large clusters.

The Distillation Loss

The training objective is the standard knowledge distillation loss, which minimizes the Kullback-Leibler (KL) divergence between the temperature-scaled teacher and student output distributions:

$\mathcal{L}_{\text{distill}} = \sum_{i} \text{KL}\left(\text{softmax}\left(\frac{y_{\text{teacher}}}{\tau}\right) \;\Big\|\; \text{softmax}\left(\frac{y_{\text{student}}}{\tau}\right)\right)$

where $\tau$ is the distillation temperature, set to 2.0 in the paper's experiments.

What this computes: the softmax operation with temperature $\tau$ converts the logits into probability distributions over the vocabulary. At $\tau = 1$ , the distribution is the standard softmax; at $\tau > 1$ , the distribution is "softened" — probability mass is spread more evenly across tokens, reducing the peakiness of the distribution and exposing the relative ordering of lower-probability tokens. The KL divergence measures how different the student's softened distribution is from the teacher's, penalizing the student when it assigns probability mass differently across the vocabulary. Summing over all token positions $i$ in the training sequence gives the total loss.

Why this form and temperature: the temperature parameter is critical. At $\tau = 1$ , the teacher distribution is often extremely peaked — the correct next token might have probability 0.99, with 0.01 distributed across the remaining 100K+ vocabulary. In this regime, the student learns only to match the single most-likely token, missing the subtler signal about which tokens are "second-best," "third-best," etc. The softened distribution at $\tau = 2.0$ reveals the teacher's full ranking over the vocabulary — which tokens the teacher considers plausible alternatives, which it definitively rules out — giving the student a richer training signal. This is standard in knowledge distillation (Hinton et al., 2015) and is particularly important for SwiftKV because the distribution mismatch between $x_j$ and $x_{l+1}$ means the student's logits will initially be poorly calibrated; the softened teacher signal provides more informative gradients for correcting this miscalibration across the full vocabulary rather than just at the top-1 prediction.

The paper explicitly compares distillation loss against standard language modeling loss (Table 4a) and finds that distillation yields a 2.64-point higher average accuracy (72.70% vs. 70.06%) with particularly large gains on generative tasks — 4.13 points on MMLU-CoT and 6.74 points on GSM-8K. The paper hypothesizes that the language modeling loss (cross-entropy with the ground-truth next token) provides too sparse a signal — it only corrects the student when the top-1 prediction is wrong, but does not guide the student toward the correct distribution over alternatives. Since SwiftKV's rewiring primarily causes distribution shift rather than catastrophic top-1 errors on most tokens, the denser distillation signal is more effective at realigning the student's internal representations.

Partial Model Training: Only $W_{QKV}$ , Not MLP or Other Parameters

A critical design choice is that only the $W_{QKV}$ parameters of the skipped layers are trained — the MLP weights, attention output projections ( $W_O$ ), layer normalization parameters, embedding layer, and LM head all remain frozen at their pretrained values. This is not an arbitrary restriction; it is motivated by prior research and validated through ablation.

Why train only $W_{QKV}$ : the paper cites three lines of prior work (Meng et al., 2024; Geva et al., 2021; Elhage et al., 2021) suggesting that MLP layers are the primary storage of factual knowledge in transformer LLMs. These works show that (1) factual associations can be localized to specific MLP weight matrices in mid-to-late layers, (2) the MLP acts as a key-value memory where the first layer encodes input patterns and the second layer retrieves associated outputs, and (3) interventions that modify MLP weights can edit specific facts while preserving general capabilities. The implication for SwiftKV: since the model's knowledge is stored in its MLPs, freezing the MLPs preserves what the model knows, while fine-tuning the attention projections ( $W_{QKV}$ ) adjusts only how the model accesses and combines that knowledge — a much less invasive change.

The $W_{QKV}$ projections control what information the attention mechanism can retrieve from the KV cache. When the KV cache is computed from $x_{l+1}$ instead of $x_j$ , the projections need to learn to extract the same semantic content from a different input representation. But the downstream processing — how that retrieved information is transformed through the MLP and combined to form the next layer's representation — remains identical because the MLP weights are unchanged. This decomposition — knowledge in MLP, access patterns in attention projections — is consistent with the mechanistic interpretability literature and provides a principled justification for partial training.

Ablation validation (Table 4b). The paper compared partial model training ( $W_{QKV}$ only) against full model training (all parameters in the later 50% of layers) for Llama-3.1-8B-Instruct with 50% SwiftKV. The results strongly favor partial training: 72.70% average accuracy for partial vs. 68.23% for full — a 4.47-point advantage for the restricted training. On MMLU, the gap is 4.36 points; on MMLU-CoT, 5.53 points; on GSM-8K, 10.08 points. The paper interprets this as evidence that training the MLPs on the SwiftKV-modified architecture causes the model to "unlearn" some of its stored knowledge — the gradient updates that help the MLPs work better with the modified attention patterns also overwrite factual associations learned during pretraining. By freezing the MLPs, the distillation can focus exclusively on teaching the attention projections to extract equivalent information from the earlier-layer representations, without disturbing the stored knowledge.

Additional practical benefit: training fewer parameters reduces GPU memory requirements and speeds up training. For Llama-3.1-8B, training only $W_{QKV}$ for half the layers means optimizing roughly 340M parameters instead of ~4B — a 12× reduction in optimizer state memory (which typically requires 2–3× the parameter count for AdamW moments and variance estimates). This is what enables the 3-hour training time on 8 H100s.

Training Hyperparameters and Dataset

The paper provides the following training configuration (Section 3.4 and Appendix A.1):

Optimizer: AdamW with learning rate $3 \times 10^{-4}$ , weight decay $0.05$ , and warmup ratio $5\%$ (the learning rate linearly increases from 0 to $3 \times 10^{-4}$ over the first 5% of training steps, then follows a cosine decay schedule).
Training epochs: 2 epochs over the dataset.
Maximum sequence length: 8192 tokens, with attention-separated sequence packing — multiple training examples are concatenated into sequences of up to 8192 tokens, with attention masks preventing cross-example attention.
Distillation temperature: $\tau = 2.0$ .

Dataset composition: a mixture of three publicly available instruction-tuning datasets totaling roughly 680 million Llama-3.1 tokens:

UltraChat (Ding et al., 2023): HuggingFaceH4/ultrachat_200k — approximately 200K multi-turn conversations covering a broad range of topics, designed to simulate natural human-AI interactions.
OpenHermes-2.5 (Teknium, 2023): teknium/OpenHermes-2.5 — a synthetic dataset of diverse instruction-following examples generated by GPT-4, covering coding, reasoning, creative writing, and general knowledge tasks.
SlimOrca (Lian et al., 2023): Open-Orca/SlimOrca — a filtered and deduplicated subset of the OpenOrca dataset, containing FLAN-style reasoning traces augmented with GPT-4 completions.

The paper is transparent that no effort was made to optimize the data recipe (Section B.3). They chose these datasets for their "popular adoption and broad domain and task coverage." To quantify the quality ceiling, they fine-tuned a base Llama-3.1-8B model directly on these datasets (without SwiftKV) and compared it to Meta's official Llama-3.1-8B-Instruct (Table B.3a). The result: the dataset-trained model achieves 65.77% average accuracy vs. 73.71% for Llama-3.1-8B-Instruct — a 7.94-point gap. This indicates that the training data is substantially worse than whatever Meta used for instruction tuning, meaning the SwiftKV distillation is starting from a weaker data foundation and could potentially achieve better results with higher-quality data. Indeed, when the paper added just 83K additional math and code training examples (16M tokens, a ~2.4% increase in dataset size), GSM-8K accuracy improved by 0.53 points (Table B.3b), confirming that data quality improvements can further close the gap to the original model.

The key takeaway from the data analysis: the reported SwiftKV accuracy numbers are lower bounds. Better training data — matching the quality of what Meta used for Llama-3.1-Instruct — would likely reduce the quality degradation below the already-small 1–2% figures reported in Table 2.

Inference Implementation in Production Serving Frameworks

The paper emphasizes that SwiftKV is designed for practical deployability — it makes "minimal changes to the model architecture" (Section 3.5) so that it can be integrated into existing production serving systems without requiring new CUDA kernels, custom attention implementations, or novel inference scheduling procedures.

Integration with vLLM and SGLang. The SwiftKV inference procedure is implemented in both vLLM (Kwon et al., 2023) and SGLang (Zheng et al., 2024), two of the most widely used open-source LLM serving frameworks. Both frameworks use PagedAttention (vLLM) or RadixAttention (SGLang) for efficient KV cache memory management and support chunked prefill (also called SplitFuse; Holmes et al., 2024; Agrawal et al., 2024), where long prefill sequences are split into chunks and interleaved with decode steps to reduce pipeline bubbles.

The key implementation logic, described in Section 3.5, is:

"During each forward pass, after completing layer $l$ , the KV-cache for the remaining layers ( $>l$ ) are immediately computed, and only the decode tokens are propagated through the rest of the model layers."

In concrete terms, when the serving system schedules a batch containing a mix of prefill chunks and decode tokens:

All tokens in the batch (both prefill and decode) propagate through layers 1 through $l$ normally, with full attention and MLP operations at each layer. The KV cache for these early layers is populated normally — each token's $K$ and $V$ at each layer is computed from that layer's own input and stored.
At the boundary after layer $l$ , the hidden state $x_{l+1}$ is extracted for all tokens. For the prefill tokens (the chunks of input prompts being processed), $x_{l+1}$ is immediately used to compute the KV cache for all remaining layers $j > l$ using $W_{KV}^j \cdot x_{l+1}$ . These KV cache entries are stored in the PagedAttention-managed memory blocks, associated with their respective sequence positions and layers. Once this computation is complete, the prefill tokens exit the forward pass — they receive no further processing in layers $l+1$ through $L$ .
For the decode tokens (the single token being generated for each active request), the forward pass continues through layers $l+1$ through $L$ normally. These tokens compute their own queries at each layer and attend to the KV cache — which now contains both the normally-computed early-layer entries and the SwiftKV-computed late-layer entries. The decode token produces the next-token logits from the final layer $L$ exactly as in the unmodified model.
When the decode token eventually completes its generation and the next token needs to be produced, that token will be treated as a decode token (not a prefill token) and will propagate through all layers. However, it will attend to the SwiftKV-computed KV cache from the original prefill tokens — and because those KV cache entries were computed from $x_{l+1}$ rather than from each layer's own processing, the attention patterns may differ slightly from what the original model would have produced.

Compatibility with chunked prefill. The chunked prefill optimization partitions long prompts into smaller chunks to avoid long prefill operations blocking decode. SwiftKV is compatible with this because the $l$ -layer boundary is respected within each chunk: within a chunk of prefill tokens, after layer $l$ completes, the KV cache for layers $>l$ is computed from that chunk's $x_{l+1}$ , and the chunk's tokens exit. The decode tokens in the same batch continue through all layers. No special handling is needed for chunk boundaries — the KV cache management is handled by PagedAttention as usual.

Why no custom kernels are needed. The SwiftKV rewiring reuses standard transformer operations: $W_{KV}^j \cdot x_{l+1}$ is a standard matrix multiply, identical to what each layer already does with its own input $x_j$ . The attention operation for decode tokens is unchanged — it still computes $\text{softmax}(QK^T/\sqrt{d_k})V$ with the same dimensionality. The only difference is when and from what input the KV entries are computed, not how they are computed or used. This means existing highly-optimized CUDA kernels for matrix multiplication, attention, and KV cache management can be used without modification — the changes are purely at the Python orchestration level in the model's forward method.

Batch inference performance (Figure 3). The end-to-end results validate that the theoretical FLOPs reduction translates to real throughput improvements. For Llama-3.1-8B-Instruct on a single H100:

With 2000-token inputs, 50% SwiftKV achieves 1.2–1.3× higher combined throughput (input + output tokens per second) than the baseline.
With 128K-token inputs, the improvement grows to 1.8–1.9×. This is expected because longer prompts make prefill an even larger fraction of total compute, magnifying the benefit of SwiftKV's prefill-specific optimization.
Normalized throughput reaches 30K tokens/sec/GPU at 8K input length, corresponding to 480 TFLOPS/GPU of BF16 compute.

For Llama-3.1-70B-Instruct on 4 H100s:

With 2000-token inputs, 50% SwiftKV achieves 1.4–1.5× higher combined throughput.
With 128K-token inputs, it achieves 1.8–2.0× higher throughput.
At 8K input, throughput exceeds 16K tokens/sec over 4 GPUs, corresponding to 560 TFLOPS/GPU when normalized to the baseline model's FLOPs count. This represents a 56.6% Model FLOPs Utilization (MFU) — meaning over half of the GPU's theoretical peak BF16 throughput is being used for useful computation. The paper calls this "an unprecedented throughput for BF16 inference workloads."

Interactive inference performance (Figure 4, Figure A.1). For latency-sensitive applications:

TTFT reduction: at low request arrival rates (where the system is not backlogged), SwiftKV reduces time-to-first-token by up to 50% for workloads with longer input lengths. This is a direct consequence of reducing prefill FLOPs — the prompt gets processed faster, so the first output token appears sooner.
TTFT under load: SwiftKV can sustain 1.5–2.0× higher request arrival rates before TTFT explodes (the point where queued requests accumulate faster than they can be processed). This is because each prefill takes less time, allowing the system to process more requests per second before saturating.
TPOT reduction: SwiftKV reduces time-per-output-token by up to 60% in some settings. At first glance this is surprising since SwiftKV does not reduce decode FLOPs — each generated token still processes all layers. The explanation, per the paper, is that in production serving systems, prefill and decode contend for the same GPU compute. Reducing prefill time frees up GPU cycles that would otherwise be spent processing prompts, allowing those cycles to be allocated to decode. Even though decode operations themselves are not accelerated, they experience less queuing delay behind prefill operations, reducing the per-token latency. This effect is most pronounced at moderate-to-high request rates where the GPU is well-utilized and prefill-decode contention is the primary bottleneck.

Real-world workload validation (Appendix A.5, Table A.3). To verify that the synthetic benchmarks transfer to real workloads, the paper evaluates on the ShareGPT dataset (ShareGPT Team, 2023) — real conversations between users and ChatGPT collected in the wild. The throughput improvements (1.25–1.7× for Llama-3.1-8B, 1.25–1.8× for Llama-3.1-70B) are consistent with the synthetic benchmarks when matched on input-to-output length ratio, confirming that SwiftKV's benefits are not an artifact of the benchmark design.

Design Choices and Their Justifications

Why prefill-specific reduction rather than symmetric layer removal: the motivation comes from the workload asymmetry observation (10:1 input-to-output ratio). Removing layers entirely (via pruning) would reduce both prefill and decode compute, but since decode is a small fraction of total compute, the additional savings from reducing decode would be marginal. More importantly, removing layers entirely degrades the model's ability to process generated tokens — each decode token would have fewer layers available to refine its representation, potentially reducing generation quality. SwiftKV preserves full depth for decode while saving compute on prefill, aligning the architectural asymmetry with the workload asymmetry.

Why train only $W_{QKV}$ : justified by the mechanistic interpretability literature (MLPs store knowledge) and validated by the ablation showing 4.5-point advantage over full model training. This design enables the extremely lightweight distillation (<1B tokens, <10% parameters) that makes SwiftKV practical to apply to new models.

Why distillation loss rather than LM loss: the distillation loss provides a denser training signal — it corrects not just top-1 errors but the full distribution over the vocabulary, which is important because the rewiring primarily causes distribution shift (miscalibration) rather than catastrophic classification errors. The 2.64-point advantage over LM loss validates this choice.

Why a fixed cutoff layer rather than a learned or adaptive mechanism: simplicity and predictability. A fixed $l$ means the prefill compute reduction is deterministic and can be planned for in advance. The paper explores multiple values of $l$ (25%, 50%, 62.5% of layers) and finds that 50% is the sweet spot for most models — beyond that, the similarity between $x_{l+1}$ and $x_j$ becomes too low for lightweight distillation to compensate. An adaptive mechanism that selects $l$ per-input or per-token could theoretically achieve better accuracy-efficiency tradeoffs, but would introduce runtime decision overhead and complicate the inference implementation.

Why use a single source $x_{l+1}$ for all skipped layers rather than a cascaded approach: the cascaded approach would require propagating through the skipped layers (even if only partially), defeating the compute savings. The single-source approach is the most aggressive reduction — all later-layer KV caches are derived from a single early representation — and its viability depends entirely on the hidden state similarity documented in Figure 2. The fact that it works (with distillation) validates the similarity hypothesis.

Why the cutoff is at a layer boundary rather than within a layer: practical implementation simplicity. The boundary after layer $l$ is a natural point where the hidden state $x_{l+1}$ is available, having passed through all of layer $l$ 's operations (attention, MLP, residual connection, layer norm). Attempting to extract an intermediate representation from within a layer would require architectural modifications to expose internal activations, complicating the implementation.

Why AcrossKV uses grouped sharing rather than learning which layers to merge: simplicity and deterministic compression. The groups are contiguous blocks of layers (e.g., layers 17–20 share one KV cache, layers 21–24 share another), which makes the implementation straightforward — the sharing pattern is hardcoded in the model configuration. The alternative of learning which layers to merge (e.g., via similarity analysis or optimization) could potentially achieve better compression rates for the same accuracy, but would add complexity and model-specific tuning. The paper shows that even the simple contiguous grouping works well across diverse architectures, suggesting that the gains from learned merging would be incremental.

Why the first generated token processes all layers: this token produces the logits for the second generated token, and its representation at the final layer is critical for accurate prediction. Allowing it to skip layers would mean the output logits come from an intermediate layer (effectively an early exit), which would require auxiliary training to add a prediction head at layer $l$ . The paper explores this direction in Appendix B.4 but finds it substantially more difficult — early exit logits align with final logits only 66% of the time even after training, and using a simple threshold-based heuristic (exit early if confidence > 0.95) still requires careful tuning. The conservative choice — one token pays the full cost — keeps the approach simple and reliable while still delivering the vast majority of the compute savings (since one token out of hundreds or thousands is negligible).

Why the training dataset is only 680M tokens: the paper explicitly chose not to optimize the data recipe, using only popular public datasets. The speed of distillation (3 hours for 8B model) was prioritized over squeezing out the last fraction of a point in accuracy. The paper's analysis (Table B.3a) shows that better data exists — the Meta-internal instruction tuning recipe produced a much stronger model than the same base model trained on the public datasets — so there is headroom. This design choice reflects a philosophy of "good enough" practical engineering rather than exhaustive optimization: SwiftKV achieves near-original accuracy with minimal cost and can be improved further with better data, but the current results are already sufficient for production deployment.

4. Key Insights and Innovations

Innovation 1: Prefill-Specific Compute Reduction as a New Design Axis in Inference Optimization

The paper's most fundamental conceptual contribution is the recognition that prefill and decode tokens have categorically different computational requirements, and that this asymmetry can be exploited through architecture-level rewiring rather than generic compression. Prior work in LLM inference optimization — whether pruning (Sreenivas et al., 2024; Tang et al., 2025), layer-skipping (Jaiswal et al., 2024; Men et al., 2024), KV cache compression (Hooper et al., 2024; Ainslie et al., 2023b), or sparse attention (Zhao et al., 2024; Jiang et al., 2024) — treated all tokens identically. A pruned model processes every token through its reduced layer count; a quantized KV cache stores every token's entries at lower precision; sparse attention prunes connections for every query regardless of whether it belongs to the prompt or the generation. This symmetry is natural if you view inference as a uniform computational process, but it is suboptimal when the workload is systematically asymmetric.

SwiftKV introduces the idea that prompt tokens are fundamentally KV-cache producers while decode tokens are both KV-cache producers AND logit consumers. A prompt token's only lasting contribution to the inference process is the keys and values it deposits into the KV cache for future decode tokens to attend to. Unlike a decode token, a prompt token does not need to produce a next-token prediction at every layer — the model's output logits are only needed from the final token of the prompt (and even that token principally matters because it conditions the first generated token). This means the query, attention, and MLP operations at later layers — which exist to transform representations toward the final prediction — are, for purely prefill purposes, unnecessary. The prompt token's job is done once its KV cache is populated.

This reframing is significant because it changes the optimization problem from "how do we make the model smaller?" or "how do we make attention cheaper?" to "what is the minimum computation needed to produce useful KV cache entries for prompt tokens?" The answer — demonstrated by SwiftKV's results — is that this minimum is substantially less than the full forward pass, provided the hidden states are similar enough across depth. This is a genuinely new design axis: prefill-specific compute reduction is different in kind from model compression (which reduces capacity for all tokens) or attention sparsification (which reduces only one operation type). It is a workload-aware optimization that matches the architectural modification to the specific asymmetry of the deployment scenario.

The evidence that this axis matters independently of others appears in Table 3, where the paper constructs a "Merge-all-Layers" baseline that achieves extreme KV cache compression (eliminating nearly all KV cache memory) but provides only a 10% throughput improvement at full GPU memory — because the memory reduction doesn't address the compute bottleneck. SwiftKV without any KV cache compression achieves a 35% throughput improvement on the same hardware by reducing prefill FLOPs. The comparison isolates compute reduction as the primary lever for prefill-dominant workloads on memory-sufficient GPUs, establishing that prior work's focus on memory efficiency addressed a constraint that is often not binding in production.

This contribution is fundamental rather than incremental: it defines a new region in the design space that prior work did not explicitly target, and it demonstrates that this region yields practical gains orthogonal to existing techniques.

Innovation 2: The Sufficiency of Hidden State Similarity as a Scientific Basis for Layer Substitution

The paper's second distinctive contribution is establishing — through quantitative measurement, architectural design, and experimental validation — that the empirically-observed similarity of hidden states across deep transformer layers is sufficient to enable compute reuse, provided a minimal amount of targeted adaptation is applied. This is more than a restatement of prior observations about representational similarity (Liu et al., 2024b; Gromov et al., 2024); it is a demonstration that this similarity has actionable engineering consequences and specific boundary conditions.

Prior work had documented that deeper layers produce increasingly similar representations. Gromov et al. (2024) showed that removing deeper layers entirely sometimes caused surprisingly small accuracy degradation, leading to their provocative title "The Unreasonable Ineffectiveness of the Deeper Layers." But these observations were primarily descriptive — they characterized a property of trained transformers without providing a principled method for exploiting it. The natural engineering response to "deeper layers are similar" would be "remove them," but as the paper's ablation in Table 4a ("W/o Distill") shows, naive removal degrades accuracy substantially (2.64 points on average, up to 6.74 points on GSM-8K). The similarity is real but not perfect — the representations are not interchangeable without adaptation.

What SwiftKV contributes is a precise characterization of how much adaptation is needed and where it must be applied. By training only the W_QKV projections of the affected layers (less than 10% of parameters) for less than 3 hours on 8 GPUs, the paper shows that the similarity is sufficient to recover near-original accuracy. The critical design choice — freezing MLP weights to preserve stored knowledge while retraining attention projections to accommodate the shifted input distribution — operationalizes a hypothesis about where the representation similarity is "close enough" (the MLP transformations are largely invariant to small shifts in input representation) and where adaptation is needed (the attention projections, which must extract specific information from the shifted hidden state).

The SimScore metric introduced in Equation 1 and visualized in Figure 2 serves as a diagnostic tool that predicts where SwiftKV will work. Models with higher similarity at the 50% depth mark (Llama, Mistral) tolerate 50% prefill reduction with minimal accuracy loss; models with lower similarity (Qwen) degrade more. This transforms the similarity observation from a post-hoc explanation into a pre-deployment feasibility check — practitioners can measure SimScore on their target model and estimate the viable reduction level before investing in distillation.

This contribution is fundamental for methodology: it provides a scientific basis for architectural decisions that were previously made through trial-and-error (how many layers to prune, where to add early-exit heads, etc.). The SimScore measurement costs nothing but a single forward pass and predicts the difficulty of the adaptation task. The paper's finding that similarity exceeds 0.5 at roughly half the model depth across all tested architectures suggests this may be a general property of transformer training dynamics, though the paper appropriately stops short of claiming universality given the sample size of six models.

Innovation 3: Lightweight Distillation as an Alternative to Full Retraining for Architecture-Preserving Model Transformation

The paper introduces a novel point in the cost-quality tradeoff space for adapting pretrained models to modified architectures. Prior approaches to architectural modification fall into two categories: training-free methods that apply heuristics to skip or compress components without any fine-tuning (FFN-SkipLLM, ShortGPT, SliceGPT), and prune-and-distill methods that remove components and then retrain all remaining parameters on massive datasets (10–100 billion tokens for Nemotron-51B at 40B tokens, DarwinLM-8.4B at 10B tokens). Training-free methods are cheap but brittle — the paper shows FFN-SkipLLM works only on Llama and fails catastrophically on Mistral (45.71% vs. 78.23% baseline) and Deepseek (24.83% vs. 64.12%). Prune-and-distill methods are robust but expensive — they require large-scale training infrastructure and days of compute.

SwiftKV's distillation procedure — training only W_QKV parameters on <1 billion tokens with a teacher-student KL divergence objective — occupies a previously empty region of this tradeoff space: it achieves robustness comparable to full retraining methods at a cost comparable to training-free methods. The 3-hour training time on 8 H100s for an 8B model makes the approach practical to apply to new models as they are released, without requiring the compute budgets that only large organizations can access. The open-source release of both training and inference code reinforces this practical orientation.

The intellectual contribution is not the distillation technique itself (KL divergence distillation with temperature scaling is standard since Hinton et al., 2015), but rather the recognition that architectural rewiring which preserves model parameters can be corrected through extremely targeted adaptation — and that the adaptation should be limited to the components that interface with the changed data flow (the attention projections), not the components that store knowledge (the MLPs). The paper validates this decomposition through the ablation showing that full model training yields 4.47 points worse accuracy than partial training (Table 4b), a counterintuitive result that supports the "MLPs store knowledge" hypothesis from mechanistic interpretability (Meng et al., 2024; Geva et al., 2021).

The finding that distillation loss outperforms standard language modeling loss by 2.64 points (Table 4a) is also instructive: it demonstrates that the degradation from rewiring is primarily distributional (the student model's logits are miscalibrated across the vocabulary) rather than categorical (the student model makes wrong top-1 predictions). The language modeling loss only corrects categorical errors; distillation corrects distributional miscalibration. This diagnostic distinction — between categorical and distributional degradation — is a useful concept for thinking about what goes wrong when architectures are modified, and it suggests that different adaptation strategies may be appropriate depending on which type of degradation dominates.

This contribution is incremental in technique but fundamental in its implications for the economics of model customization: it establishes that architecture-level modifications need not require the massive retraining budgets previously assumed, opening the door to more rapid experimentation with inference-optimized model variants.

The paper's AcrossKV mechanism introduces a simple but previously underexplored idea: use the same architectural rewiring that enables prefill compute reduction to also enable KV cache memory reduction by sharing caches across groups of consecutive layers. Prior cross-layer KV cache sharing work (Liu et al., 2024a, MiniCache) merged caches for specific pairs of adjacent layers based on similarity analysis. AcrossKV generalizes this to arbitrary group sizes — 2-way, 4-way, 8-way, up to 16-way sharing — where all layers in a group share a single KV cache computed from the boundary hidden state x_{l+1}.

The intellectual contribution is twofold. First, AcrossKV demonstrates that cross-layer sharing (inter-layer) is more parameter-efficient than head reduction (intra-layer) for achieving KV cache compression. The ablation in Appendix B.2 (Table B.2) shows that for a fixed 37.5% compression target, AcrossKV + GQA achieves 71.49% average accuracy while pure MQA (which achieves the same compression by reducing KV heads within each layer) achieves only 54.13% — a 17-point gap. This indicates that layers benefit more from maintaining diverse attention patterns within each layer (multiple KV heads) than from maintaining distinct KV caches across layers, suggesting that cross-layer KV cache redundancy is genuinely high and exploitable.

Second, AcrossKV establishes that KV cache compression can be achieved as a byproduct of compute reduction rather than as a separate optimization target. Since the SwiftKV rewiring already computes KV caches from a shared input x_{l+1}, merging those caches across layers is architecturally natural — it simply means using one projection matrix to serve multiple layers rather than one per layer. This unified approach to compute and memory reduction is architecturally cleaner than deploying separate techniques for each (e.g., pruning for compute + quantization for memory) and may explain why the combined approach degrades so gracefully: the distillation procedure jointly optimizes the shared projections for both compute reduction and memory compression, rather than layering independently optimized compression on top of compute reduction.

The scaling behavior of AcrossKV is noteworthy: on Llama-3.1-8B, accuracy drops only from 72.70% (no AcrossKV) to 70.22% at 16-way compression — a 2.48-point degradation for a 46.875% KV cache memory reduction. This remarkably flat scaling suggests that for Llama-family models, the KV cache representations in the deeper layers are highly redundant, and a single set of KV projections trained on the shared input can serve as an adequate approximation for all of them. The fact that Deepseek-V2-Lite-Chat degrades more steeply (63.51% → 59.32% at only 4-way, a 4.19-point drop) provides an interesting architectural diagnostic: Deepseek-V2's latent attention mechanism may produce more layer-specific KV representations that are less amenable to sharing, highlighting that the viability of AcrossKV-style compression is architecture-dependent in ways that merit further investigation.

This contribution is incremental as a technique (cross-layer sharing extends prior pairwise merging) but fundamental in its demonstration of how much redundancy exists in deep-layer KV caches and in its integration of memory reduction into a unified transformation framework that addresses both compute and memory bottlenecks simultaneously.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary evaluation benchmark is a set of seven standard tasks: ARC-Challenge (Clark et al., 2018), Winogrande (Sakaguchi et al., 2019), HellaSwag (Zellers et al., 2019), TruthfulQA (Lin et al., 2022), MMLU (Hendrycks et al., 2021), MMLU-CoT, and GSM8K-CoT (Cobbe et al., 2021). The evaluation harness follows the configuration from neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8, which implements chat-templated evaluations critical for the Llama-3.1/3.2 instruction-tuned models. The exact number of shots and metrics per task are specified in Table A.1: ARC-Challenge uses 0-shot exact_match/multi_choice, Winogrande uses 5-shot accuracy, HellaSwag uses 10-shot accuracy_normalized, TruthfulQA uses 0-shot truthfulqa_mc2 (accuracy), MMLU uses 5-shot exact_match/multi_choice, MMLU-CoT uses 0-shot exact_match/strict-match, and GSM8K-CoT uses 8-shot exact_match/strict-match. This is an accuracy evaluation only — the paper does not evaluate on generation quality metrics like ROUGE or BLEU, which limits the assessment to tasks with objectively verifiable correct answers.
Base model(s). The paper evaluates on five model families spanning diverse architectures and scales: Llama-3.1-Instruct (3B, 8B, 70B, and 405B variants), Mistral-Small-Instruct-2409, Qwen2.5-14B-Instruct, and Deepseek-V2-Lite-Chat. The Llama models span two orders of magnitude in parameter count (3B to 405B), with the 405B variant using FP8 (W8A16) quantization. Deepseek-V2-Lite-Chat is a mixture-of-experts model implementing a novel latent attention mechanism (DeepSeek-AI et al., 2024), making it architecturally distinct from the dense transformer variants. The selection is deliberate: it tests whether SwiftKV generalizes across model scales, across architectures (dense, MoE, latent-attention), and across training regimes (the models come from different organizations with different pretraining and instruction-tuning recipes). The paper argues these models are "representative of the capabilities of many contemporary LLMs" (Section 4), though all evaluations are on instruction-tuned chat variants — the behavior of base (non-instruction-tuned) models is not assessed.
Metrics. The primary metric is average task accuracy across the seven benchmarks, computed as the unweighted mean of the individual task scores (each task's score is computed according to its standard metric as specified in Table A.1). The paper reports per-task accuracy in all result tables, with the "Avg." column providing the summary statistic. For the inference performance evaluation, the metrics are combined throughput (total input + output tokens processed per second), time-to-first-token (TTFT, the latency from request submission to the first output token), time-per-output-token (TPOT, the average latency between consecutive output tokens), and normalized TFLOPS/GPU (the floating-point operations performed per second, normalized to the baseline model's FLOPs count to enable fair comparison when SwiftKV reduces the operations performed). The throughput metric is measured as combined input+output tokens per second because the prefill/decode split varies across workloads, and a pure-output-token metric would obscure the prefill processing cost that SwiftKV targets.
Baselines. The paper compares against three baselines from prior work. (1) FFN-SkipLLM (Jaiswal et al., 2024): a training-free method that adaptively skips feed-forward network (MLP) layers during inference based on hidden state similarity heuristics, with no fine-tuning. The paper sets candidate layers to be skipped from 35–80% depth in each model, reflecting the settings in the original paper. Note that FFN-SkipLLM skips only MLP layers (not attention), and its skip rate varies between models and tasks since it is adaptively determined during inference — the prefill reduction percentages reported in Table 2 represent only the fraction of MLP layers skipped, not total FLOPs reduction. (2) Llama-3.1-Nemotron-51B-Instruct (Sreenivas et al., 2024): a model pruned and distilled from Llama-3.1-70B-Instruct using neural architecture search guided pruning with distillation on 40 billion tokens, achieving 28% prefill reduction and 50% KV cache reduction. (3) DarwinLM-8.4B (Tang et al., 2025): a model pruned and distilled from Qwen2.5-14B-Instruct using 10 billion tokens of training data, achieving 40% parameter reduction. Additionally, the paper evaluates against the original unmodified model for each family, which serves as the accuracy upper bound. For the inference performance experiments, the baseline is the unmodified model running on the same hardware with the same serving configuration.
Generation budget / compute accounting. For the model quality evaluation, there is no explicit generation budget — SwiftKV models are evaluated on standard benchmark tasks using the same evaluation protocol as the baseline models. The "budget" is instead measured in terms of the prefill compute reduction percentage, defined as (L - l) / L where L is the total number of layers and l is the index of the last unskipped layer. This percentage approximates the fraction of prefill FLOPs eliminated (Section 3.2). For example, "50% SwiftKV" means l = L/2, approximately halving prefill FLOPs. For the KV cache reduction via AcrossKV, the exact memory reduction is reported as a percentage of the original KV cache memory. For inference performance evaluation (Section 4.2), the compute budget is implicit in the hardware configuration: Llama-3.1-8B-Instruct runs on 1 NVIDIA H100 GPU with 80GB memory, and Llama-3.1-70B-Instruct runs on 4 NVIDIA H100 GPUs with 4-way tensor parallelism. Each experiment submits roughly 15 million tokens worth of requests, with each request generating 256 output tokens.
Cross-validation / statistical protocol. There is no cross-validation protocol for the model quality evaluation. Each SwiftKV configuration is distilled once and evaluated once on the fixed set of seven benchmarks. The paper does not report error bars, confidence intervals, or standard deviations for any accuracy measurements. Results are reported as single-point estimates. For the inference performance experiments, each benchmark configuration is run once under controlled conditions (fixed hardware, fixed request patterns), with the system allowed to reach steady state — the paper does not describe running multiple trials or reporting variance. This absence of statistical rigor is a weakness: for a test set of only seven tasks (each with its own metric and scale), the "Avg." column can be sensitive to how tasks are weighted and to variance in individual task scores. The lack of error bars makes it difficult to assess whether differences of 0.1–0.5 percentage points in average accuracy (which appear frequently in Table 2) are statistically meaningful or within measurement noise.

Main Quantitative Results

Model Quality Across Architectures, Scales, and Reduction Levels

The central quality results are presented in Table 2, which reports per-task and average accuracy for every model, SwiftKV configuration, AcrossKV configuration, and baseline. The paper evaluates SwiftKV at reduction levels of 25%, 40%, 45%, 50%, and 62.5%, depending on the model, with AcrossKV configurations ranging from 2-way to 16-way sharing.

Llama-3.1-8B-Instruct. The baseline achieves 73.71% average accuracy. At 25% SwiftKV (skipping 8 of 32 layers), average accuracy is 73.59% — a 0.12-point degradation, well within what could be measurement noise. At 50% SwiftKV (skipping 16 layers), accuracy drops to 72.70% — a 1.01-point degradation. At 62.5% SwiftKV (skipping 20 layers), accuracy falls to 66.09% — a 7.62-point drop, representing the cliff beyond which the approach fails. Looking at individual tasks at 50% reduction: ARC-Challenge drops from 82.00% to 80.38% (−1.62), GSM8K-CoT drops from 82.56% to 79.45% (−3.11), MMLU-CoT drops from 70.63% to 69.73% (−0.90), while Winogrande actually improves from 77.90% to 78.22% (+0.32) and TruthfulQA is essentially unchanged (54.56% → 54.54%). The GSM8K-CoT degradation is notably larger than other tasks — a pattern the paper attributes to insufficient math data in the distillation dataset (Section B.3).

Adding AcrossKV to the 50% reduction configuration shows graceful degradation: 2-way (71.82%, −0.88 from no-AcrossKV 50%), 4-way (71.49%, −1.21), 8-way (70.50%, −2.20), 16-way (70.22%, −2.48). The degradation from 50% SwiftKV with no AcrossKV to 16-way AcrossKV is only 2.48 points while achieving 46.875% KV cache memory reduction.

Llama-3.1-70B-Instruct. The baseline achieves 84.31% average. At 25% reduction: 83.83% (−0.48). At 50% reduction: 82.98% (−1.33). The pattern is similar to the 8B model — degradation is modest at 50% and fairly uniform across tasks, with GSM8K again showing a larger drop (95.15% → 93.56%, −1.59) than knowledge-recall tasks like MMLU (83.97% → 82.51%, −1.46). With 2-way AcrossKV at 50%: 82.63% (−1.68). With 4-way AcrossKV: 82.96% (−1.35). Critically, the 4-way AcrossKV configuration actually performs better than the 2-way configuration (82.96% vs. 82.63%), which might reflect noise or a small regularization benefit from the shared projections — the paper does not comment on this inversion.

Llama-3.1-405B-Instruct (FP8). The baseline achieves 86.6% average accuracy (note: this is an FP8-quantized model, so the baseline is lower than what FP16 would achieve). At 50% SwiftKV: 85.9% — a 0.7-point degradation. The paper only reports one SwiftKV configuration for this model (50%, no AcrossKV), presumably due to the computational cost of experimenting with the 405B model. The result demonstrates that SwiftKV scales to very large models, though the FP8 quantization means the baseline is already operating with some degradation.

Llama-3.2-3B-Instruct. The baseline achieves 66.47% average. At 25% reduction: 66.55% — essentially unchanged (−0.08). At 40% reduction: 65.55% (−0.92). At 50% reduction: 64.09% (−2.38). This model shows faster degradation than the 8B and 70B variants — at 50%, the 2.38-point drop is roughly double the 8B model's 1.01-point drop at the same reduction level. The paper does not explicitly discuss this, but it is consistent with the hypothesis that smaller models have less representational redundancy — fewer parameters per layer means each layer's output carries more unique information, making substitution from an earlier layer's output more lossy. With 2-way and 4-way AcrossKV at 40% reduction: 65.13% and 65.19% respectively — essentially flat relative to the no-AcrossKV 40% configuration.

Mistral-Small-Instruct-2409. The baseline achieves 78.23% average. At 25% reduction: 77.80% (−0.43). At 50% reduction: 77.24% (−0.99). The degradation is comparable to Llama-3.1-8B, demonstrating SwiftKV's applicability beyond the Llama architecture. With 2-way AcrossKV at 50%: 77.21% (−1.02). With 4-way AcrossKV: 76.66% (−1.57). The GSM8K degradation at 50% is 86.50% → 84.30% (−2.20), smaller than the Llama models' GSM8K drops.

Deepseek-V2-Lite-Chat. The baseline achieves 64.12% average (note: this is substantially lower than the dense models, reflecting that this is a smaller MoE model and that the baseline accuracy ceiling is lower). At 25% reduction: 64.19% — essentially unchanged (+0.07). At 45% reduction: 63.51% (−0.61). With 2-way AcrossKV at 45%: 63.07% (−1.05). With 4-way AcrossKV at 45%: 59.32% (−4.80). This is the steepest AcrossKV degradation observed — a 4.19-point drop from 45% no-AcrossKV to 45% 4-way AcrossKV. The paper attributes this to Deepseek-V2's novel latent attention mechanism, which may produce more layer-specific KV representations that are less amenable to cross-layer sharing. The result is important as a boundary condition: AcrossKV is not universally benign and can cause significant degradation on architecturally distinct models.

Qwen2.5-14B-Instruct. The baseline achieves 77.38% average. At 25% reduction: 76.23% (−1.15). At 50% reduction: 69.93% (−7.45). This is dramatically worse than Llama, Mistral, or Deepseek — a 7.45-point drop at 50% versus 1-point drops for Llama-8B and Mistral. This is the paper's clearest negative result and the strongest validation of the SimScore diagnostic: Figure 2 shows that Qwen models have lower SimScore at 50–75% depth than Llama or Mistral, and the paper explicitly states that Qwen suffers larger degradations "which may be due to Qwen models having lower similarity between layers at 50–75% depth." With 2-way AcrossKV at 25%: 76.43% (−0.95 from baseline, +0.20 from 25% no-AcrossKV — an improvement, likely noise). With 4-way AcrossKV at 25%: 75.85% (−1.53 from baseline, −0.38 from 25% no-AcrossKV). The paper does not report AcrossKV at 50% for Qwen, likely because the 50% no-AcrossKV degradation is already too large to be practically useful.

Comparison with baselines (all from Table 2).

FFN-SkipLLM achieves dramatically worse results than SwiftKV across nearly all models. For Llama-3.1-8B: 70.62% vs. SwiftKV 50%'s 72.70% — SwiftKV is 2.08 points better while skipping both MLP and attention (FFN-SkipLLM skips only 12–19% of MLPs). For Llama-3.1-70B: FFN-SkipLLM is not reported (the row is absent from Table 2, suggestively because the method was not evaluated or produced results so poor they were omitted). For Mistral-Small: 45.71% — a catastrophic 32.52-point drop from the baseline (78.23% → 45.71%), compared to SwiftKV 50%'s 77.24%. For Deepseek-V2-Lite: 24.83% — a 39.29-point drop (64.12% → 24.83%), compared to SwiftKV 45%'s 63.51%. For Qwen2.5-14B: 62.53% — a 14.85-point drop (77.38% → 62.53%), compared to SwiftKV 25%'s 76.23%. The FFN-SkipLLM results validate the paper's claim that training-free methods are brittle across architectures — they work acceptably only on Llama-family models and even then underperform SwiftKV while skipping fewer operations.

Nemotron-51B achieves 82.78% average accuracy — nearly identical to SwiftKV 50% on Llama-3.1-70B (82.98%). Both achieve similar accuracy, but Nemotron-51B reduces prefill compute by 28% (vs. SwiftKV's 50%) and was distilled on 40B tokens (vs. SwiftKV's <1B tokens on <10% of parameters). The comparison highlights SwiftKV's efficiency advantage: comparable accuracy with more compute reduction and orders-of-magnitude cheaper adaptation.

DarwinLM-8.4B achieves 35.94% average accuracy — a 41.44-point drop from Qwen2.5-14B's 77.38%. This is a massive degradation, far worse than SwiftKV 50%'s 69.93% on the same architecture family. The paper notes that DarwinLM-8.4B uses 10B tokens of distillation data, yet achieves far worse quality than SwiftKV with <1B tokens. However, this is not an apples-to-apples comparison: DarwinLM-8.4B is a 40% parameter reduction (the model is structurally pruned to 8.4B from 14B), while SwiftKV preserves all parameters and only changes the data flow for prefill. The tasks where DarwinLM-8.4B collapses are particularly striking: MMLU drops to 12.46% (from 76.58%), MMLU-CoT to 0.00%, and GSM8K-CoT to 1.90% — suggesting the pruning destroyed the model's reasoning capabilities entirely.

Inference Performance (Section 4.2, Figures 3–4, Table 3)

The inference experiments validate that SwiftKV's theoretical FLOPs reduction translates to end-to-end throughput and latency improvements in production serving systems.

Batch inference throughput (Figure 3). For Llama-3.1-8B-Instruct on 1 H100:

At 2000-token inputs: baseline achieves ~22–24K tokens/s combined throughput; 50% SwiftKV achieves ~27–30K (1.2–1.3×); 50% SwiftKV + 4× AcrossKV achieves ~29–32K.
At 8000-token inputs: baseline ~22K; SwiftKV ~29–31K (1.3–1.4×); with AcrossKV ~31–34K.
At 32K-token inputs: baseline ~17K; SwiftKV ~26–28K (1.5–1.6×); with AcrossKV ~27–29K.
At 128K-token inputs: baseline ~8K; SwiftKV ~14–15K (1.8–1.9×); with AcrossKV ~15–16K.

The improvement grows with input length because longer prompts make prefill a larger fraction of total compute, magnifying the benefit of SwiftKV's prefill-specific optimization. The AcrossKV configurations consistently provide an additional ~10–15% throughput improvement over SwiftKV-only, attributable to larger batch sizes enabled by reduced KV cache memory (more requests can be processed concurrently when each consumes less memory).

For Llama-3.1-70B-Instruct on 4 H100s:

At 2000-token inputs: baseline ~10K tokens/s; SwiftKV ~14–15K (1.4–1.5×); with AcrossKV ~15–16K.
At 8000-token inputs: baseline ~10K; SwiftKV ~15–16K (1.5–1.6×); with AcrossKV ~16–17K.
At 32K-token inputs: baseline ~8K; SwiftKV ~13–14K (1.6–1.75×); with AcrossKV ~14–15K.
At 128K-token inputs: baseline ~4.5K; SwiftKV ~8–9K (1.8–2.0×); with AcrossKV ~8.5–9K.

The paper highlights that at 8K input length, Llama-3.1-70B with 50% SwiftKV achieves over 16K tokens/s combined throughput across 4 GPUs, corresponding to 560 TFLOPS/GPU when normalized to the baseline model's FLOPs count — representing a 56.6% Model FLOPs Utilization (MFU). This is a meaningful metric because it indicates not just absolute speed but how efficiently the GPU hardware is being utilized: an MFU of 56.6% on H100 GPUs for BF16 inference is significantly higher than typical serving system utilization (often 30–40% due to memory bandwidth bottlenecks and kernel launch overhead). The paper attributes this high utilization to the fact that SwiftKV eliminates computation without introducing irregular memory access patterns or control flow divergences that would reduce GPU efficiency.

Interactive inference latency (Figure 4). For Llama-3.1-70B-Instruct:

TTFT at low arrival rates: SwiftKV reduces TTFT by up to 50% for workloads with longer input lengths. At 2000-token inputs with low arrival rate, baseline TTFT is ~0.5–0.7s; SwiftKV achieves ~0.25–0.35s. At 32K-token inputs, baseline is ~5–6s; SwiftKV achieves ~2.5–3s.
TTFT under load: SwiftKV sustains 1.5–2.0× higher request arrival rates before TTFT explosion. For 8000-token inputs, the baseline TTFT begins increasing sharply at ~4–5 requests/s; SwiftKV handles ~7–8 requests/s before the same inflection. This has direct operational implications: a service using SwiftKV can handle roughly double the user load before latency degrades.
TPOT reduction: at moderate-to-high arrival rates, SwiftKV reduces TPOT by up to 60%. At 2000-token inputs and 4 requests/s, baseline TPOT is ~20–25ms; SwiftKV achieves ~10–12ms. At 32K-token inputs and 0.5 requests/s, baseline is ~50ms; SwiftKV achieves ~20ms. At very low arrival rates (where the GPU is underutilized and there is no prefill-decode contention), the TPOT reduction is minimal — consistent with the explanation that TPOT improvement comes from reduced prefill-decode contention rather than faster decode operations.

For Llama-3.1-8B-Instruct (Figure A.1), the relative improvements are similar but the absolute latencies are lower due to the smaller model size. At low arrival rates with 32K inputs, baseline TTFT is ~600–800ms; SwiftKV achieves ~300–400ms. TPOT improvement follows the same pattern as the 70B model — substantial at moderate load, minimal at very low load.

Compute vs. memory reduction tradeoff (Table 3). The paper constructs the Merge-all-Layers baseline — an idealized KV cache compression scheme where all layers' KV caches are merged into a single layer, eliminating virtually all KV cache memory but retaining all computation. At 80GB memory (full GPU), Merge-all-Layers achieves 25.1K tokens/s vs. baseline 22.9K — only a 9.6% improvement. SwiftKV 50% without AcrossKV achieves 31.0K — a 35.4% improvement. This quantifies the paper's central claim that compute reduction matters more than memory reduction in memory-sufficient scenarios. As memory is constrained: at 40GB, Merge-all-Layers achieves 25.2K vs. baseline 20.6K (22.3% improvement — better but still below SwiftKV 50% at 27.3K). At 20GB, Merge-all-Layers achieves 25.2K vs. baseline 10.8K (133% improvement — now outperforming SwiftKV 50% without AcrossKV at 12.2K, because memory is so constrained that the baseline and SwiftKV-only struggle to fit reasonable batch sizes). SwiftKV 50% + 4× AcrossKV achieves 18.0K at 20GB — better than SwiftKV-only (12.2K) but below Merge-all-Layers (25.2K). Adding FP8 quantization closes the gap: SwiftKV 50% + 4× AcrossKV + FP8 achieves 23.2K at 20GB — approaching Merge-all-Layers performance while also reducing compute. At 16GB (barely enough for model weights), the baseline and SwiftKV-only run out of memory (OOM); Merge-all-Layers achieves 24.8K; SwiftKV + 4× AcrossKV achieves only 4.22K; SwiftKV + 4× AcrossKV + FP8 achieves 7.28K. The pattern demonstrates that compute reduction (SwiftKV) provides the largest gains when memory is sufficient, memory reduction (Merge-all-Layers) becomes critical when memory is scarce, and the combination (SwiftKV + AcrossKV + FP8) provides the best overall performance across memory regimes.

ShareGPT real-world validation (Table A.3). For Llama-3.1-8B on the original ShareGPT dataset (average input/output ratio of 1.5): baseline 23.7K tokens/s, SwiftKV 50% 27.6K (1.16×), with 4× AcrossKV 29.4K (1.24×). Filtering for higher input/output ratios: at ratio 3.4 (0.2 min ratio filter), improvement is 1.24×; at ratio 10 (2.0 min ratio filter), improvement is 1.43×; at ratio 40 (20 min ratio filter), improvement is 1.53×; at ratio 150 (100 min ratio filter), improvement is 1.65×. The throughput and improvement ratios scale with the input/output length ratio exactly as the paper's workload-asymmetry hypothesis predicts — workloads with longer prompts relative to outputs benefit more. For Llama-3.1-70B, the pattern is identical: from 1.15× at the unfiltered ratio of 1.5 to 1.70× at ratio 150.

SGLang results (Table A.2). The paper replicates the throughput experiments in SGLang and finds "similar relative improvements over the baseline" — for Llama-3.1-8B, 1.4–1.8× throughput; for Llama-3.1-70B, 1.5–1.8×. No per-configuration numbers are provided in the main text, but Table A.2 reports representative figures: at 2000 input/256 output, baseline 27.4K tokens/s (8B) and 11.6K (70B); 50% SwiftKV achieves 36.2K (8B, 1.32×) and 15.7K (70B, 1.35×). The consistency across serving frameworks validates that SwiftKV's benefits are not implementation-specific.

Ablation Studies and Robustness Checks

Distillation vs. no distillation (Table 4a, "The effect of distillation"): For Llama-3.1-8B-Instruct with 50% SwiftKV, training with distillation loss achieves 72.70% average accuracy vs. 70.06% without distillation — a 2.64-point advantage. The largest gaps appear on MMLU-CoT (69.73% vs. 65.60%, +4.13) and GSM8K (79.45% vs. 72.71%, +6.74), indicating that generative reasoning tasks benefit most from the distillation signal. The paper attributes this to distillation providing a denser training signal that corrects distributional miscalibration across the vocabulary, which matters more for tasks requiring coherent multi-step generation than for discriminative tasks like multiple-choice QA.

Distillation vs. standard LM loss (Table 4a): The "W/o Distill" row uses standard language modeling loss (cross-entropy with ground-truth next tokens) rather than distillation from the teacher model. The 2.64-point gap demonstrates that distillation is meaningfully better than standard fine-tuning for recovering quality after the SwiftKV rewiring. The paper's explanation — that the rewiring primarily causes distribution shift rather than catastrophic top-1 errors, and distillation's softened distribution provides richer gradients — is plausible but not directly tested (there is no experiment varying the distillation temperature or comparing with alternative distribution-matching objectives).

Partial model training vs. full model training (Table 4b): Training only the W_QKV parameters of the skipped layers achieves 72.70% average, while training all parameters in the later 50% of layers achieves only 68.23% — a 4.47-point advantage for partial training. The gap is particularly large on GSM8K (79.45% vs. 69.37%, +10.08) and MMLU-CoT (69.73% vs. 64.20%, +5.53). This is a counterintuitive result — training more parameters produces worse results — and supports the paper's hypothesis that MLP layers store factual knowledge that gets corrupted when trained on the shifted input distribution. However, the paper does not ablate which parameters to freeze: it's possible that freezing a subset of MLP layers while training only the earliest skipped MLP layers would yield benefits, or that different learning rates for MLP vs. attention parameters could close the gap.

Dataset composition and quality (Table B.3a, Table B.3b): The paper explicitly acknowledges that the distillation dataset (UltraChat + OpenHermes-2.5 + SlimOrca, 680M tokens) is not optimal. When the paper trained a base Llama-3.1-8B model directly on this dataset (without SwiftKV), it achieved only 65.77% average vs. Llama-3.1-8B-Instruct's 73.71% — a 7.94-point gap, indicating the dataset quality is substantially below Meta's internal instruction-tuning data. This means the SwiftKV distillation is starting from a weaker data baseline, and the reported accuracy numbers are lower bounds: better data could further reduce the quality gap. Adding 83K math and code examples (16M tokens, ~2.4% increase in dataset size) improved GSM8K by 0.53 points and average accuracy by 0.23 points (Table B.3b), confirming that data quality improvements translate to accuracy improvements. However, this is a small-scale experiment — the paper does not systematically explore data scaling, data mixing ratios, or domain-specific data augmentation.

KV cache quantization compatibility (Table B.1): For Llama-3.1-8B-Instruct with 50% SwiftKV, adding per-token FP8 KV cache quantization (post-training, no quantization-aware fine-tuning) causes minimal additional degradation: 72.70% → 72.30% (no AcrossKV), 71.82% → 71.69% (2-way AcrossKV, combined 62.5% memory reduction), 71.49% → 71.35% (4-way AcrossKV, combined 68.75% memory reduction). The total accuracy loss from baseline (73.71%) to the most compressed configuration (4-way AcrossKV + FP8, 71.35%) is 2.36 points while achieving 68.75% KV cache memory reduction and 50% prefill compute reduction. This demonstrates that AcrossKV and quantization are orthogonal and composable — the SwiftKV-distilled model's KV cache representations are robust to aggressive post-training compression.

Inter-layer vs. intra-layer KV cache sharing (Table B.2): For a fixed 37.5% KV cache memory reduction on Llama-3.1-8B with 50% SwiftKV, the paper compares three approaches: (a) pure MQA (all attention heads share one KV head per layer): 54.13% average — catastrophic; (b) AcrossKV with MHA (full heads per layer, 4-way cross-layer sharing): 69.76%; (c) AcrossKV with GQA (the default, GQA head compression plus 4-way cross-layer sharing): 71.49%. The 17.36-point gap between MQA and AcrossKV-GQA demonstrates that cross-layer sharing preserves much more representational capacity than within-layer head reduction for the same memory savings. The gap between AcrossKV-MHA (69.76%) and AcrossKV-GQA (71.49%) of 1.73 points is smaller, suggesting that GQA's head compression is largely complementary to cross-layer sharing.

Early exit for decode tokens (Appendix B.4, Figure B.1): The paper explores adding an auxiliary LM head at the boundary layer l to enable early exit during decode. After training on 160M tokens, the early exit logits align with the final logits approximately 66% of the time when the early exit's maximum softmax probability exceeds 0.95 (Figure B.1). Using a simple threshold heuristic (exit if max probability > 0.95), the system can skip some decode layers with reasonable accuracy — examples in Appendix B.4.1 show coherent outputs for straightforward questions (e.g., "What is the capital of France?") but the paper acknowledges that "how to use early exit is always an interesting direction and research topic" and does not integrate this into the main SwiftKV pipeline. This is a negative result in the sense that naive early exit is not reliable enough for production use — the 66% alignment rate means roughly one-third of early exits would produce different tokens than the full model, which is unacceptable for most applications.

Model-specific AcrossKV degradation: The Deepseek-V2-Lite-Chat result at 4-way AcrossKV (59.32%, a 4.80-point drop from baseline at 45% reduction) and the steeper degradation for Qwen2.5-14B at 50% reduction (69.93%, a 7.45-point drop) serve as negative results that define the boundaries of SwiftKV's applicability. The paper does not ablate why these models degrade more — whether it is the architectural differences (latent attention for Deepseek, training recipe for Qwen) or the lower SimScore values — but the correlation with SimScore (Figure 2) is suggestive.

Critical Assessment

The experiments in this paper collectively support the central claim that SwiftKV can reduce prefill computation by 25–50% with minimal quality degradation across diverse model families. However, several important qualifications are necessary when interpreting the reported numbers.

The accuracy evaluation has significant methodological limitations. First, the test set consists of only seven benchmarks. While these are standard in the LLM evaluation literature, they represent a narrow slice of model capabilities — predominantly English-language, multiple-choice or short-answer, and focused on factual knowledge and reasoning. The paper does not evaluate on code generation benchmarks (HumanEval, MBPP), long-form generation tasks (summarization quality, dialogue coherence), multilingual capabilities, or safety/alignment metrics. This matters because SwiftKV's mechanism — substituting KV cache inputs from an earlier layer — could have differential impacts on capabilities that depend on precise long-range attention patterns, such as code generation where variable references must be tracked across long contexts, or multilingual generation where the model must maintain language-specific representations. The claim of "minimal quality degradation" is currently only validated on a specific set of academic benchmarks that may not reflect production use cases.

Second, all evaluations are on instruction-tuned chat models. The behavior of base (pretraining-only) models under SwiftKV is not assessed. Given that instruction tuning is known to significantly alter model representations and capabilities, it is possible that base models — which typically have more diffuse probability distributions and less well-calibrated confidence — would behave differently. The SimScore curves in Figure 2 are shown for instruction-tuned models only; base model similarity might differ, affecting the viable reduction level.

Third, the adaptation is evaluated only in the zero-to-few-shot setting. The paper does not evaluate whether SwiftKV models retain their few-shot learning capabilities or whether the quality degradation compounds across multiple turns of a conversation. For chatbot applications — exactly the use case the paper targets — multi-turn consistency could be more fragile than single-turn accuracy if the KV cache substitutions accumulate errors across conversation turns.

Fourth, there are no error bars or statistical significance tests. The "Avg." column in Table 2 is an unweighted mean of seven tasks with very different scales and variances. A 0.5-point drop in this average could be driven entirely by one task with high variance while other tasks are unchanged — without standard deviations, there is no way to distinguish signal from noise. The paper's headline claim of "<1–2% degradation" is based on point estimates that could easily overlap within measurement uncertainty.

The inference performance evaluation is thorough but has a narrow hardware scope. All experiments use NVIDIA H100 GPUs (80GB) in a single configuration. The paper acknowledges this — the instance is an AWS p5.48xlarge — but does not test on A100 GPUs, lower-memory configurations, or different GPU architectures. The claim that "SwiftKV increases throughput by up to 2×" is conditioned on H100 hardware with 80GB memory. On hardware with different compute/memory ratios (e.g., A100 40GB, H200 with 141GB, or inference-optimized hardware like TPUs or Inferentia), the balance between compute reduction and memory reduction would shift, potentially changing the optimal configuration and the magnitude of improvements.

Additionally, all inference experiments use a fixed 256 output tokens per request and vary only the input length. Real workloads have joint distributions of input and output lengths — for example, summarization tasks have long inputs and long outputs, while classification tasks have long inputs and short outputs. The paper's experimental design cleanly isolates the prefill-to-decode ratio effect, but does not evaluate the more complex scenario where output length also varies and the decode phase becomes a larger fraction of total compute.

The comparison with baselines is informative but not fully balanced. The FFN-SkipLLM comparison demonstrates that training-free methods are brittle, but FFN-SkipLLM was not designed for the same use case — it skips MLP layers for both prefill and decode tokens, which is a different optimization target. The Nemotron-51B comparison is more directly relevant (both aim to reduce inference cost while preserving accuracy), but Nemotron-51B was designed as a general-purpose smaller model, not specifically for prefill reduction. DarwinLM-8.4B is compared against Qwen2.5-14B, but the architectures differ so dramatically (DarwinLM is structurally pruned to 60% of the original size) that the comparison primarily demonstrates that aggressive pruning can destroy model quality — not that SwiftKV is superior to all pruning methods.

Critically, the paper does not compare against a simple layer-dropping baseline with equivalent distillation budget. If one simply removed the later 50% of layers from Llama-3.1-8B and fine-tuned the remaining layers on the same 680M tokens, what accuracy would result? This would isolate whether SwiftKV's benefit comes from the specific rewiring (keeping KV projections from the skipped layers) or simply from preserving more parameters. The absence of this baseline makes it difficult to attribute the gains specifically to the SwiftKV mechanism rather than to the conservative strategy of keeping all weights and only changing data flow.

The distillation cost comparison is somewhat overstated. The paper emphasizes that SwiftKV distillation uses <1B tokens while Nemotron-51B uses 40B and DarwinLM-8.4B uses 10B. This is a valid efficiency advantage, but the paper does not control for model scale, distillation objective, or target accuracy. Nemotron-51B was targeting a specific parameter budget (51B, a 27% reduction from 70B) and was trained to convergence on 40B tokens — it is possible that similar accuracy could have been achieved with fewer tokens but the authors of that work did not explore the minimum viable distillation budget. The claim that SwiftKV's distillation is "orders of magnitude" cheaper conflates what was done in prior work with what is necessary for prior approaches.

The SimScore diagnostic is predictive but not mechanistically validated. The correlation between SimScore (Figure 2) and SwiftKV degradation (Table 2) is consistent — Qwen has lower SimScore at 50% depth and degrades more; Llama has higher SimScore and degrades less. But the paper does not establish that SimScore is the causal factor. Alternative explanations — differences in training data composition, instruction-tuning procedures, or architecture-specific properties of the KV projections — could explain the cross-model variation. A stronger validation would manipulate SimScore (e.g., by training models with different depth-to-width ratios or different initialization schemes) and show that the manipulation produces the predicted change in SwiftKV degradation. Without this, SimScore remains a correlational observation rather than a validated design rule.

The "up to 2× throughput" claim aggregates across very different workloads. The 1.8–2.0× improvements appear primarily at 128K input lengths — the longest context evaluated. At the more typical 2000-token input length observed in Snowflake's production, the improvement is 1.2–1.5×. The "up to 2×" framing is technically accurate but could mislead readers who skim for the headline number. The paper's own production data (Section 1, "average prompt length between 500 and 1000") suggests that the 1.2–1.5× improvement at 2000 tokens is more representative of typical enterprise workloads than the 2× improvement at extreme context lengths.

The TPOT improvement claim raises questions about the serving system configuration. The paper attributes TPOT reduction to reduced prefill-decode contention (prefill operations complete faster, freeing GPU time for decode). This explanation is plausible, but it implies that the baseline system was not optimally configured — if prefill and decode are contending for GPU time, this suggests the chunked prefill scheduling or batch composition could be tuned differently. The paper uses vLLM's default configuration (max_num_batched_tokens set to 2048); different settings might reduce contention in the baseline, narrowing the apparent TPOT improvement from SwiftKV. The paper does not explore how the TPOT benefit varies with scheduling parameters.

Missing experiments that would strengthen the paper.

Long-form generation evaluation: benchmarks like AlpacaEval, MT-Bench, or human evaluation of generated summaries would test whether SwiftKV preserves generation quality, not just multiple-choice or short-answer accuracy.
Multi-turn conversation evaluation: testing whether SwiftKV models maintain coherence and consistency across extended dialogues, where KV cache substitutions might accumulate errors.
Scaling the distillation data: training on 100M, 680M, 2B, and 10B tokens to establish the data scaling behavior of the distillation procedure — does accuracy saturate at 680M or continue improving?
Ablation on which layers to skip: the paper skips later layers (layers l+1 through L). What happens if the skipped layers are in the middle, or in the early layers? Is it specifically the later layers' similarity that enables this approach, or could early layers be substituted if SimScore is high enough?
Comparison with a same-budget layer-dropping baseline: remove the later layers entirely and fine-tune the remaining layers on 680M tokens to isolate the contribution of preserving the skipped layers' KV projections.
Evaluations on a broader range of hardware: A100, H200, and lower-memory configurations to validate the compute-vs-memory tradeoff analysis beyond the single H100 80GB setting.
User study or production A/B test: while the paper cites Snowflake production data for the workload characterization, it does not report production deployment results for SwiftKV itself — the inference performance is measured in a controlled benchmark environment. Actual production latency and throughput under real traffic patterns (with variable request sizes, arrival patterns, and system load) could differ from the synthetic benchmark results.

Bottom line. The experiments convincingly demonstrate that SwiftKV achieves its claimed prefill compute reduction with modest accuracy degradation on standard benchmarks, and that this translates to throughput and latency improvements in serving systems. The paper's central contributions — the prefill-specific reduction approach, the lightweight distillation procedure, and the integration of KV cache compression — are well-supported within the scope of the evaluation. However, the evaluation scope is narrower than the paper's claims suggest: "minimal quality degradation" is validated only on seven short-answer/multiple-choice benchmarks for instruction-tuned models; "up to 2× throughput" is achieved primarily at context lengths far exceeding typical enterprise usage; and the approach's generalizability to production deployment with real traffic patterns, to base models, and to generation-heavy tasks remains untested. The paper is best understood as demonstrating the viability and promise of prefill-specific compute reduction rather than as providing a comprehensive characterization suitable for unconditional production adoption across all model families and use cases.

6. Limitations and Trade-offs

Limitation 1: Quality Evaluation Scope Is Too Narrow to Support "Minimal Quality Degradation" Claims

The assumption or constraint. The paper validates SwiftKV model quality exclusively on seven standard academic benchmarks: ARC-Challenge, Winogrande, HellaSwag, TruthfulQA, MMLU, MMLU-CoT, and GSM8K-CoT (Table 2, Appendix A.1). These are almost entirely multiple-choice or short-answer tasks with objectively verifiable correct answers. The paper does not evaluate on code generation (HumanEval, MBPP), long-form generation quality (summarization coherence, dialogue quality), multilingual capabilities, retrieval-augmented generation accuracy, or any open-ended generation metric. The paper acknowledges dataset limitations only indirectly — in Appendix B.3, it notes that the distillation dataset was chosen for "popular adoption and broad domain and task coverage" rather than optimized for quality, but it does not acknowledge that the evaluation benchmarks themselves might fail to capture degradation modes relevant to production use.

The consequence. The paper's headline conclusion — "minimal quality degradation (<1–2%) averaged across a wide range of tasks" (Section 4.1) — may substantially underestimate degradation on capabilities that are critical for the enterprise applications the paper targets. This is not a hypothetical concern: the paper's own data shows that within the seven-task suite, degradation is highly task-dependent. At 50% SwiftKV on Llama-3.1-8B-Instruct, average degradation is 1.01 points, but GSM8K drops by 3.11 points while Winogrande improves by 0.32 points and TruthfulQA is essentially unchanged. This variance means the "average" masks significant disparity — and the seven benchmarks likely under-sample the capability dimensions where SwiftKV's mechanism could cause problems. Specifically, SwiftKV substitutes KV cache inputs from an earlier layer for all later layers' attention operations. This could degrade performance on tasks requiring precise long-range attention to specific tokens — for example, code generation where variable names must be tracked across hundreds of tokens, multi-hop reasoning where earlier context must be precisely retrieved, or instruction-following where specific constraints from the prompt must be respected. None of these capability dimensions are tested by the seven benchmarks. A practitioner deploying SwiftKV on a code completion service, for instance, has no evidence from this paper about whether the model will correctly track variable types and function signatures across a long prompt.

What evidence exists in the paper. The paper's GSM8K-CoT results (Table 2) provide the clearest internal evidence that task type matters. Across all models and reduction levels, GSM8K degrades more than knowledge-recall tasks like MMLU or TruthfulQA. For Llama-3.1-8B at 50%: GSM8K drops 3.11 points vs. 0.60 for MMLU; at 62.5% (the breaking point), GSM8K drops 13.64 points vs. 6.35 for MMLU. This pattern — reasoning tasks degrading more than factual recall — is consistent with the hypothesis that KV cache substitution disrupts the attention patterns needed for multi-step inference while leaving factual knowledge (stored in MLP weights, which are frozen) largely intact. The paper's analysis in Section B.3 confirming that adding math-specific data to the distillation dataset improves GSM8K (by 0.53 points) further supports this interpretation — the degradation is real and task-specific, but can be partially mitigated with better data. However, the evaluation suite contains no code tasks, no long-form generation tasks, and no tasks testing instruction-following fidelity — precisely the capabilities likely to be sensitive to KV cache perturbation.

Mitigation status. The paper does not address this limitation directly. Section B.3 shows that adding domain-specific data helps (GSM8K improves with math data), which is encouraging but is tested on only one task with a modest improvement of 0.53 points. The authors do not frame the narrow evaluation as a limitation or propose expanding the benchmark suite. A practitioner would need to conduct their own evaluation on domain-specific tasks before deploying SwiftKV in production.

Limitation 2: Difficulty Estimation and Cutoff Layer Selection Require Per-Model Empirical Tuning

The assumption or constraint. The paper presents the SimScore metric (Equation 1, Figure 2) as a diagnostic that correlates with SwiftKV's viability — models with higher similarity at 50% depth (Llama, Mistral) degrade less than models with lower similarity (Qwen). However, the paper does not provide a decision rule or threshold for selecting the cutoff layer $l$ . Each model family is tested at multiple reduction levels (25%, 40%, 45%, 50%, 62.5%), and the paper retroactively observes that 50% works well for Llama and Mistral but not for Qwen, while 25% works for Qwen but is unnecessarily conservative for Llama. The choice of $l$ is presented as a hyperparameter to be swept rather than something derivable from model properties.

The paper states: "Beyond 50% SwiftKV, model quality drops quickly. For example, Llama-3.1-8B-Instruct incurs a 7% accuracy gap at 62.5% SwiftKV" (Section 4.1). But no method is provided for predicting where this cliff occurs for a new model without running the full distillation and evaluation pipeline at multiple reduction levels.

The consequence. Applying SwiftKV to a new model architecture requires a non-trival empirical search: distilling multiple variants (at least 2–3 different cutoff layers), evaluating each on a held-out benchmark suite, and selecting the best tradeoff. The paper's distillation is lightweight (3 hours on 8 H100s for 8B models), but running this 3 times plus evaluation still represents a meaningful engineering cost — especially for very large models where even a single distillation run requires 32 GPUs and 5 hours. Moreover, the selection process requires a representative evaluation benchmark to judge quality — but as Limitation 1 notes, the paper's own benchmark suite may not capture all relevant degradation modes. A practitioner would need to decide: trust the paper's SimScore heuristic and hope 50% works, or invest in multi-configuration distillation and domain-specific evaluation to find the optimal $l$ . The paper provides no guidance on how to make this decision.

The Qwen2.5-14B result illustrates the risk of getting this wrong. At 50% reduction, Qwen degrades by 7.45 points — a catastrophic loss that would make the model unusable for most applications. A practitioner who applied 50% SwiftKV to Qwen assuming it would behave like Llama (based on architectural similarity — both are dense transformer models with GQA) would discover the degradation only after completing distillation and evaluation. The paper's post-hoc explanation — lower SimScore at 50–75% depth — is only identifiable after measuring similarity, and the paper does not establish a quantitative relationship between SimScore and degradation that could be used for prediction.

What evidence exists in the paper. The cross-model variation in Table 2 is the primary evidence: Llama-3.1-8B at 50% degrades by 1.01 points, Mistral-Small at 50% degrades by 0.99 points, Deepseek-V2-Lite at 45% degrades by 0.61 points, Qwen2.5-14B at 50% degrades by 7.45 points, and Llama-3.2-3B at 50% degrades by 2.38 points. The SimScore curves in Figure 2 show that Qwen has lower similarity than Llama or Mistral at 50–75% depth, and the 3B model has lower similarity than the 8B model. The correlation is visually apparent but not quantified — there is no scatter plot of SimScore vs. degradation, no correlation coefficient, and no predictive model. The paper's claim that Qwen's degradation "may be due to Qwen models having lower similarity between layers at 50–75% depth" is stated as a hypothesis, not as a validated causal relationship.

Mitigation status. The paper provides the SimScore measurement methodology as a diagnostic tool, which is useful but incomplete. A practitioner can measure SimScore on their model before distillation, observe whether similarity at the desired reduction level exceeds the ~0.5 threshold observed for successful models, and make an informed guess about viability. But this is a heuristic, not a guarantee — the paper does not establish that SimScore above 0.5 is sufficient for low degradation, only that models with SimScore above 0.5 at 50% depth happened to work well. The authors do not suggest automating this selection or training a predictor. Future work on "predicting difficulty of a question" is mentioned in a different context (Section 8 of the original paper's discussion, not directly addressing cutoff layer selection), but no specific direction for this limitation is proposed.

Limitation 3: No Evaluation of Multi-Turn Conversation or Accumulated KV Cache Distortion

The assumption or constraint. All quality evaluations in the paper (Table 2, Table 4, Table B.1–B.3) are single-turn: a prompt is provided, the model generates a response, and the response is scored. The inference performance experiments (Section 4.2, Figures 3–4) also use single-turn requests (each request is an independent prompt-generate cycle). The paper does not evaluate multi-turn conversations where the KV cache from earlier turns persists and is attended to by later turns.

This matters because SwiftKV fundamentally changes what is stored in the KV cache for the later layers. Instead of each layer $j > l$ storing keys and values computed from its own processed representation of each token, it stores keys and values computed from $x_{l+1}$ — the output of an earlier layer. In a single-turn setting, this substitution is applied once and then the generated tokens attend to the substituted KV cache. In a multi-turn setting, the model's own generated tokens from turn 1 (which were not SwiftKV-compressed — decode tokens propagate through all layers) are in the KV cache alongside the SwiftKV-compressed prompt tokens from turn 1. In turn 2, the model must attend to a mixture of compressed and uncompressed KV entries. In turn 3, it must attend to compressed turn-1 prompt tokens, uncompressed turn-1 output tokens, compressed turn-2 prompt tokens, and uncompressed turn-2 output tokens. Over many turns, the KV cache contains a mixture of representations computed at different effective "depths" — some from layer $l+1$ of the original prompt, others from the full-depth processing of generated tokens.

The consequence. There is a plausible failure mode: the model's attention mechanism may struggle to integrate information from representations computed at different effective depths. The attention softmax $\text{softmax}(QK^T/\sqrt{d_k})$ assumes that all keys in the cache live in a comparable representational space — that the similarity between a query vector and a key vector has consistent semantic meaning across different positions and layers. If some keys were computed from an earlier hidden state (lower effective depth, potentially less refined representations) while others were computed from the full-depth processing, the attention scores may be systematically biased: keys from deeper-processed tokens might dominate attention because their representations are more similar to the query (which is computed at full depth), or conversely, keys from shallower-processed tokens might be systematically ignored. Over many conversation turns, this could cause the model to gradually "forget" earlier context that was SwiftKV-compressed while over-attending to recent context — a form of context corruption that would not appear in single-turn evaluations.

A separate but related concern: the accumulated KL divergence between the teacher's and student's output distributions across multiple turns could compound. Even if per-turn degradation is small (the 1–2% average accuracy loss observed in single-turn benchmarks), the error in the KV cache representation at turn 1 might cause the model to generate slightly different tokens in turn 1's response. Those tokens, stored in the KV cache, then influence attention in turn 2, potentially causing a larger deviation in turn 2's response, and so on. This is essentially an autoregressive error accumulation problem — the sort of distribution shift that causes autoregressive models to drift over long sequences — but applied across conversation turns rather than within a single generation.

What evidence exists in the paper. None. The paper does not report any multi-turn evaluation, does not discuss the mixed-depth KV cache problem, and does not measure whether turn-10 responses from a SwiftKV model systematically differ from turn-10 responses from the original model. All evidence is single-turn. This is a significant gap for a method targeted at "chatbots, copilots" and "interactive-inference" scenarios (Section 4.2) — precisely the applications where multi-turn conversations are the norm.

Mitigation status. Not addressed. The paper does not mention multi-turn evaluation as a limitation or as future work. The inference implementation description (Section 3.5) focuses exclusively on single-request processing: "During each forward pass, after completing layer $l$ , the KV-cache for the remaining layers ( $>l$ ) are immediately computed, and only the decode tokens are propagated through the rest of the model layers." This description is silent on how the KV cache is managed across multiple requests in a conversation session — whether compressed entries from previous turns persist, whether they are recomputed when the model processes a new turn, or whether the system needs special handling for multi-turn state. A practitioner deploying SwiftKV in a chatbot service would need to design and evaluate this multi-turn handling themselves, with no guidance from the paper.

Limitation 4: Distillation Data Quality Is a Confounding Variable — Reported Accuracy Represents a Lower Bound But the Ceiling Is Unknown

The assumption or constraint. The paper distills all SwiftKV models using a fixed dataset mixture (UltraChat + OpenHermes-2.5 + SlimOrca, ~680M tokens) chosen for popularity rather than quality optimization. The paper is transparent about this: "Note that in Sec. 4, we did not try to maximize the performance of SwiftKV from the data recipe perspective since the search space is very large and outside the scope of our paper" (Appendix B.3). The paper explicitly measures the quality ceiling of this dataset: a base Llama-3.1-8B model fine-tuned directly on this dataset (without SwiftKV rewiring) achieves 65.77% average accuracy, versus 73.71% for Meta's official Llama-3.1-8B-Instruct — a 7.94-point gap (Table B.3a). This means the distillation data is substantially worse than whatever Meta used for instruction tuning.

The consequence. The reported SwiftKV accuracy numbers should be interpreted as lower bounds on achievable quality, not as estimates of the inherent quality cost of SwiftKV. The paper cannot distinguish between degradation caused by the SwiftKV rewiring (which must be compensated by distillation) and degradation caused by the low-quality distillation data (which could be improved independently). It is possible — and the paper's analysis suggests it is likely — that with a distillation dataset matching Meta's internal instruction-tuning quality, the accuracy gap between SwiftKV and the original model would be smaller than the 1–2% reported. It is also possible that some of the reported degradation is entirely attributable to the data and would disappear with better data.

This confounding has practical consequences for practitioners. If an organization deploys SwiftKV using the paper's exact recipe (public datasets, 680M tokens), they should expect accuracy close to the reported numbers. But if they invest in higher-quality distillation data — using proprietary instruction-tuning datasets, generating synthetic data from stronger models, or carefully curating domain-specific examples — they might achieve substantially better accuracy than the paper reports. The paper provides no guidance on the data quality scaling behavior: does accuracy improve linearly with data quality, or does it saturate? Are there diminishing returns? Would 10× more data of the same quality help, or is quality the binding constraint? The small-scale experiment in Table B.3b (adding 83K math+code examples, improving GSM8K by 0.53 points) is too limited to answer these questions.

A related practical problem: the paper's distillation recipe is validated on instruction-tuned chat models, where high-quality instruction-following data is abundant. For base models (which many organizations fine-tune for custom applications), the appropriate distillation data would be the pretraining corpus — but training on pretraining data with a distillation objective (KL divergence from the original model) is a different setup than the instruction-tuning distillation described in the paper. The paper provides no results for base model distillation.

What evidence exists in the paper. Table B.3a provides the key evidence: the data quality gap is 7.94 points for Llama-3.1-8B. The fact that SwiftKV 50% on this model achieves 72.70% — only 1.01 points below the original 73.71% — means either that the distillation process is remarkably effective at closing the data quality gap, or that the architecture's preservation of MLP weights (which store knowledge) means the model is naturally robust to low-quality distillation data. The paper does not tease apart these explanations. Table B.3b provides limited evidence that adding domain-specific data helps — the 0.53-point GSM8K improvement from 83K additional examples (increasing the dataset by 2.4%) suggests that data improvements translate to accuracy improvements, but the effect size is small and is measured on only one task.

Mitigation status. The paper acknowledges the data limitation and frames it as an opportunity: "This study indicates that improvements in distillation data is potentially an important direction for future work, particularly domain-specific datasets to reduce the quality gap compared to the original model when using SwiftKV" (Appendix B.3). The authors are transparent that they did not optimize the data recipe, which is appropriate for a paper introducing a new method. However, the lack of a data scaling study (varying dataset size and quality systematically) means a practitioner cannot estimate how much investment in better data would improve their specific deployment. The mitigation is partial: the paper identifies the issue but provides minimal actionable guidance beyond "better data would probably help."

Limitation 5: All Inference Performance Measured on a Single GPU Architecture; Memory-Constrained and Heterogeneous Hardware Regimes Untested

The assumption or constraint. All inference performance experiments (Section 4.2, Figures 3–4, Table 3) are conducted on NVIDIA H100 GPUs with 80GB of memory, using either a single GPU (Llama-3.1-8B) or 4 GPUs with tensor parallelism (Llama-3.1-70B). The paper acknowledges the specific hardware: "We ran all inference speedup experiments on a AWS p5.48xlarge instance, with 8 NVIDIA H100 GPUs, 192 vCPUs, and 2TB memory" (Appendix A.2). No experiments are reported on A100 GPUs (the previous-generation datacenter standard), on H200 GPUs (with 141GB memory), on consumer GPUs with limited memory, on inference-optimized hardware (TPUs, Inferentia, Groq), or on CPU-based inference.

The consequence. The paper's key performance claims — "up to 2× higher aggregate throughput," "60% lower time per output token," "560 TFlops/GPU normalized inference throughput" — are all conditioned on H100 hardware. The H100 has specific architectural properties that affect how SwiftKV's compute reduction translates to speedup: high memory bandwidth (3.35 TB/s), large L2 cache (50 MB), and the Transformer Engine with FP8 support. On different hardware, the compute-to-memory-bandwidth ratio changes, potentially shifting the bottleneck.

On A100 GPUs (2.0 TB/s bandwidth, 40GB or 80GB memory), the baseline inference throughput is lower to begin with, and the relative improvement from SwiftKV might differ because the balance of compute-bound vs. memory-bound operations changes. The paper's analysis in Table 3 shows that at 80GB (the full H100 memory), compute reduction matters more than memory reduction — but this is specifically because the H100 is sufficiently memory-rich that the baseline can already run with large batch sizes. On an A100 40GB, the baseline model might be severely memory-constrained even at moderate batch sizes, making the KV cache memory reduction from AcrossKV proportionally more valuable and the compute reduction proportionally less impactful. The paper's "compute vs. memory reduction" analysis (Table 3) begins to explore this tradeoff by artificially limiting memory to 20GB, 40GB, and 16GB on the H100 — but this is a simulated constraint, not actual different hardware. GPU memory bandwidth, cache hierarchy, and tensor core throughput cannot be simulated by simply limiting available memory on an H100.

On inference-optimized hardware with different computational characteristics — e.g., TPUs with their systolic array architecture and high-bandwidth inter-chip interconnects, or Groq's deterministic processor architecture — the relationship between FLOPs reduction and throughput improvement could be entirely different. The paper provides no evidence about whether SwiftKV's benefits transfer across hardware platforms.

What evidence exists in the paper. The paper provides a single hardware datapoint (H100) with thorough experimentation within that datapoint — multiple input lengths, arrival rates, batch sizes, and memory constraints. The throughput scaling with input length (Figure 3) is internally consistent and aligns with the theoretical FLOPs reduction. The controlled memory experiments (Table 3) demonstrate graceful degradation in memory-constrained scenarios, suggesting the approach would work on lower-memory hardware. But these are simulations, not measurements on actual different GPUs. The paper reports vLLM and SGLang results that are consistent with each other (Section 4.2 vs. Appendix A.4), which validates that the benefits are not framework-specific, but both frameworks run on the same H100 hardware.

Mitigation status. Minimal. The paper does not discuss hardware generality as a limitation, does not report results on any non-H100 GPU, and does not provide guidance for practitioners deploying on different hardware. The memory-constraint experiments in Table 3 partially address the question of lower-memory hardware but cannot capture differences in memory bandwidth, cache size, or tensor core throughput that distinguish real GPU architectures. A practitioner deploying on A100 or consumer GPUs would need to benchmark SwiftKV themselves — the paper's throughput improvement ratios should be treated as H100-specific until validated on target hardware.

Limitation 6: The Fundamental Tradeoff Between Prefill Reduction and Decode Quality Degradation Is Not Characterized

The assumption or constraint. SwiftKV reduces prefill computation by substituting the KV cache for later layers with KV cache computed from an earlier layer's output. Decode tokens continue to process all layers normally — but they attend to the substituted KV cache entries during their attention computations. The paper's quality evaluation (Table 2) measures overall model accuracy, which reflects both the quality of the KV cache (fixed by SwiftKV for prompt tokens) and the model's ability to use that cache during decode (unchanged architecture for generated tokens). The paper never isolates whether the observed degradation comes from the substituted KV cache (prefill-side) or from the interaction between full-depth decode tokens and shallower KV cache entries (decode-side).

The consequence. There is no way to determine whether SwiftKV's degradation is prompt-length-dependent. If the degradation comes primarily from the KV cache substitution being lossy for long-range attention patterns, then longer prompts — where the model must attend to tokens far back in the sequence — should degrade more than shorter prompts. But the paper's evaluation benchmarks use fixed prompt lengths (often fitting within 2048 tokens), and the accuracy numbers aggregate across all test examples regardless of prompt length. If a practitioner's workload involves very long prompts (e.g., 32K–128K tokens as tested in the inference benchmarks), the quality degradation might be substantially worse than the 1–2% reported on the evaluation benchmarks, because the substitute KV cache must support retrieval of information across much longer distances.

The paper's inference performance results (Figure 3) show that at 128K input lengths, throughput improves by 1.8–2.0×, suggesting SwiftKV is particularly effective for long-context workloads in terms of speed. But the paper provides no quality evaluation at these long context lengths. The benchmarks used for quality evaluation (MMLU, GSM8K, etc.) have typical prompt lengths well under 8K tokens. A practitioner evaluating SwiftKV for long-context RAG or document summarization applications would see impressive throughput improvements (Figure 3, 128K results) but would have zero evidence about whether the model's accuracy on those long-context tasks degrades more than the 1–2% observed on short-context benchmarks.

Conversely, if the degradation comes primarily from the decode side — from the attention mechanism receiving keys from a shallower effective depth — then the degradation should be roughly constant regardless of prompt length, because every decode token attends to the same quality of KV cache. But the paper cannot distinguish these hypotheses.

What evidence exists in the paper. The inference experiments (Figures 3–4) show throughput scaling with input length but no quality measurements. The quality evaluation (Table 2) uses standard benchmarks with standard (relatively short) prompt lengths. There is a complete disconnect between the context lengths at which throughput benefits are measured (up to 128K) and the context lengths at which quality is validated (well under 8K). The paper does not report any long-context evaluation benchmark (e.g., LongBench, L-Eval, Zero-SCROLLS, or needle-in-a-haystack retrieval tests). This is a critical omission for a method that specifically targets prefill-dominant workloads, which often involve long prompts.

Mitigation status. Not addressed. The paper does not discuss prompt-length-dependent degradation as a concern, does not evaluate on any long-context benchmark, and does not provide an analysis of how accuracy varies with prompt length in the standard benchmarks. A practitioner deploying SwiftKV for long-context applications must either assume (without evidence) that the 1–2% degradation holds at their target context length, or conduct their own long-context evaluation — which is likely to be expensive given the cost of long-context inference. The absence of this analysis is a significant gap in the paper's validation of its central claim that SwiftKV is suitable for "enterprise workloads" that "process more input tokens than output tokens."

7. Implications and Future Directions

How This Work Changes the Landscape

SwiftKV introduces a conceptual shift in how the field thinks about inference optimization: from architecture-agnostic compression (treating all tokens identically) to workload-aware architectural rewiring (exploiting the asymmetry between prefill and decode). This is not a paradigm shift on the scale of the transformer architecture itself or the discovery of scaling laws — rather, it is a reframing that opens a previously underexplored design axis. Prior work asked "how can we make the model smaller or attention cheaper for all tokens?" SwiftKV asks "what is the minimum computation needed specifically for prompt tokens given that their only lasting contribution is the KV cache?" The answer — substantially less than the full forward pass, provided the hidden states are similar enough — defines a new optimization target that is orthogonal to model compression, quantization, and sparse attention.

The magnitude of this shift is incremental but practically significant. The paper demonstrates that prefill-specific reduction can achieve 25–50% FLOPs reduction with less than 2% average accuracy loss across multiple model families (Table 2), and that this translates to 1.2–2.0× throughput improvements in production serving systems (Figure 3). These are not speculative gains — they are measured on real hardware running vLLM and SGLang with open-sourced code. The 560 TFLOPS/GPU normalized throughput achieved for Llama-3.1-70B (Section 4.2) represents a concrete new efficiency frontier for BF16 inference.

The paper resolves a subtle tension in prior work. Gromov et al. (2024) observed that deeper layers are "unreasonably ineffective" — their representations are highly similar to earlier layers, suggesting they contribute marginal value. Yet naive layer removal degrades accuracy (the paper's "W/o Distill" result in Table 4a: 2.64-point drop). This created a puzzle: if deeper layers are so similar, why can't we just remove them? SwiftKV's resolution is that the similarity is sufficient for KV cache construction but insufficient for next-token prediction. The later layers' MLP and attention operations refine the representation for the final logit computation — a function that remains necessary for decode tokens but is unnecessary for prompt tokens that only need to deposit keys and values. By separating these two functions — KV cache generation (which can be done cheaply from an earlier representation) and logit refinement (which requires full depth) — SwiftKV extracts most of the efficiency that the similarity observation promises while preserving most of the quality that naive removal destroys.

This reframing makes several research directions newly attractive:

Workload-specific model architectures become more compelling. If prefill and decode have different computational requirements, why should they use identical architectures? Future models could be designed from scratch with asymmetric depth — more layers for decode, fewer for prefill — rather than retrofitting asymmetry onto symmetric models via distillation. The paper's success with post-hoc rewiring suggests that purpose-built asymmetric architectures could achieve even better efficiency.
Lightweight architectural adaptation as a deployment strategy becomes validated. The paper's 3-hour distillation on 8 GPUs demonstrates that architecture-level modifications can be corrected with far less compute than previously assumed (compare: Nemotron-51B used 40B tokens, DarwinLM used 10B). This lowers the barrier for practitioners to create inference-optimized variants of existing models, enabling a model customization workflow where a base model is rapidly adapted to the specific workload characteristics (prefill-to-decode ratio, typical prompt length, memory constraints) of the deployment environment.
The "preserve MLPs, retrain attention projections" decomposition is empirically validated as a knowledge-preserving adaptation strategy. The 4.47-point advantage of partial training over full model training (Table 4b) provides concrete evidence for the mechanistic interpretability hypothesis that MLPs store factual knowledge while attention projections control access patterns. This decomposition could inform other adaptation scenarios beyond inference optimization — for example, domain adaptation where only attention weights are retrained to preserve general knowledge while specializing retrieval patterns.

Conversely, some research directions become less attractive based on the paper's negative results:

Training-free layer-skipping methods (like FFN-SkipLLM) are shown to be brittle across architectures — catastrophic 32-point drops on Mistral and 39-point drops on Deepseek (Table 2). The paper's evidence suggests that without at least some adaptation, similarity-based heuristics are too fragile for production deployment across diverse model families. Researchers pursuing training-free methods should focus on understanding why some architectures are more amenable than others, rather than assuming universality.
Pure memory compression without compute reduction is shown to be insufficient for modern datacenter GPUs. The Merge-all-Layers experiment (Table 3) demonstrates that even extreme KV cache compression (32× for 8B, 80× for 70B) yields only a 10% throughput improvement when memory is sufficient — because the bottleneck is compute, not memory capacity. This redirects attention toward methods that reduce FLOPs (like SwiftKV) rather than methods that only reduce memory footprint (like KV cache quantization alone), at least for deployments on memory-rich hardware.

Follow-Up Research This Work Enables

Long-context quality evaluation of SwiftKV models. The paper evaluates quality exclusively on standard benchmarks with prompt lengths well under 8K tokens (Table 2, Appendix A.1), while reporting throughput improvements at up to 128K input lengths (Figure 3). This creates a critical evidence gap: do SwiftKV models maintain their accuracy at the long context lengths where their throughput benefits are most dramatic? A strong follow-up would evaluate SwiftKV models on long-context benchmarks (LongBench, L-Eval, Zero-SCROLLS, RULER, or needle-in-a-haystack retrieval tasks at 32K, 64K, 128K tokens) and compare degradation to the original model as a function of context length. The experiment would test whether SwiftKV's KV cache substitution causes cumulative attention degradation over long distances — a plausible failure mode given that keys and values for later layers are computed from an earlier hidden state that may lack the representational refinement needed for precise long-range retrieval. A finding that degradation scales with context length would establish a practical upper bound on SwiftKV's applicability, while a finding that degradation is constant would validate the approach for the long-context workloads where its throughput benefits are largest.

Scaling the distillation data quantity and quality to map the accuracy ceiling. The paper acknowledges (Appendix B.3) that its distillation dataset is substantially lower quality than Meta's internal instruction-tuning data — a 7.94-point gap between a base model trained on the paper's data and the official Llama-3.1-8B-Instruct (Table B.3a). The small experiment adding 83K math+code examples (Table B.3b) improves GSM8K by 0.53 points, suggesting data quality matters. A systematic study would distill SwiftKV at multiple dataset scales (100M, 680M, 2B, 5B, 10B tokens) using both the public data mixture and higher-quality alternatives (e.g., synthetic data generated from a stronger teacher model, curated domain-specific datasets), measuring accuracy across all seven benchmarks plus code generation and long-form tasks. The goal is to determine: (a) whether the 1–2% degradation gap can be closed entirely with better data, (b) whether data quality or quantity is the binding constraint at 680M tokens, and (c) whether different tasks benefit differentially from data scaling (the paper's evidence suggests GSM8K is more data-responsive than factual recall tasks). This would establish the true accuracy ceiling of SwiftKV and provide practitioners with a data investment roadmap.

Combining SwiftKV prefill reduction with early exit for decode tokens. The paper explores this direction in Appendix B.4 as a preliminary experiment, finding that an auxiliary LM head at the boundary layer achieves only 66% alignment with the final logits (Figure B.1). This is insufficient for reliable deployment, but the experiment was conducted with only 160M tokens of training and a simple threshold heuristic. A thorough follow-up would: (a) train the early exit head on the full 680M-token distillation dataset (32× more data), (b) explore more sophisticated exit criteria beyond the max-probability threshold (e.g., entropy-based, learned exit classifiers, or comparing the early exit distribution to the teacher's final distribution via KL divergence), (c) measure the fraction of decode tokens that can be safely exited early at different accuracy targets, and (d) report the additional throughput gains on top of SwiftKV's prefill reduction. The paper's architecture makes this particularly natural — the KV cache is already fully computed at the boundary layer, so early-exited decode tokens would still leave complete KV entries for future tokens to attend to. If even 30–40% of decode tokens could exit early with negligible accuracy loss, the combined system would reduce both prefill and decode compute, approaching a symmetric speedup without the quality degradation of symmetric layer removal.

Stress-testing SwiftKV on architecturally distinct models to map the boundary conditions. The paper tests five model families and finds that SwiftKV works well on Llama, Mistral, and Deepseek but degrades sharply on Qwen at 50% reduction (7.45-point drop, Table 2) — correlating with Qwen's lower SimScore at 50–75% depth (Figure 2). This is a single negative datapoint. A systematic study would evaluate SwiftKV on a broader range of architectures: (a) non-GQA models (original multi-head attention) to test whether GQA's head compression contributes to the redundancy that SwiftKV exploits, (b) models with different normalization schemes (pre-norm vs. post-norm, RMSNorm vs. LayerNorm) to test whether the similarity patterns depend on normalization, (c) models with different activation functions (SiLU, GELU, ReLU) in the MLP, (d) vision-language models where the KV cache must represent cross-modal information, and (e) models trained with different objectives (base vs. chat, pretraining-only vs. RLHF-tuned). For each architecture, the study would measure SimScore curves and SwiftKV degradation at multiple reduction levels, attempting to establish whether SimScore above a specific threshold is both necessary and sufficient for <2% degradation. A finding that certain architectural properties (e.g., pre-norm, GQA, deep MLPs) reliably produce high SimScore and low SwiftKV degradation would provide design principles for future models, while finding architectures where SimScore is high but degradation is nonetheless severe would reveal that similarity is not sufficient — that other factors (e.g., the linearity of the representation manifold, the rank of the hidden state covariance) mediate SwiftKV's viability.

Joint optimization of the cutoff layer and AcrossKV grouping using learned layer similarity. The paper selects the cutoff layer l as a fixed hyperparameter based on a coarse sweep (25%, 50%, 62.5%) and groups layers contiguously for AcrossKV (e.g., layers 17–20, 21–24, etc.). Both choices are made without optimizing for the specific similarity structure of each model. A more sophisticated approach would: (a) compute a full pairwise similarity matrix between all layers' hidden states (not just the average SimScore which collapses subsequent layers into a single number), (b) use this matrix to identify natural "similarity clusters" — groups of consecutive layers whose representations are particularly interchangeable, and (c) set both the cutoff layer (the boundary where similarity drops below a threshold) and the AcrossKV groups (matching the cluster structure) based on this analysis. This would be tested against the paper's uniform contiguous grouping: does cluster-aware AcrossKV grouping reduce degradation compared to uniform grouping at the same compression ratio? Does it enable higher compression (e.g., 6-way AcrossKV for a particularly homogeneous cluster while keeping 2-way for more diverse layers)? The experiment would replace the paper's one-size-fits-all grouping with a model-specific optimization, potentially improving the accuracy-efficiency tradeoff — particularly for models like Qwen where the uniform approach degrades more than expected based on average similarity.

Measuring and mitigating multi-turn conversation quality degradation. As discussed in Limitation 3, the paper provides no multi-turn evaluation. A critical stress-test would measure whether SwiftKV models maintain conversation quality over extended multi-turn interactions. The experiment would: (a) run 10-turn conversations through both the original and SwiftKV models on a multi-turn benchmark (MT-Bench, WildChat, or a custom RAG-based dialogue task where earlier turns contain facts that later turns must reference), (b) measure turn-by-turn accuracy (for tasks with verifiable answers) or consistency (for open-ended dialogue), and (c) test whether degradation compounds over turns (does turn-10 accuracy drop more than turn-2 accuracy?). If degradation compounds, the experiment would test countermeasures: (i) "refreshing" the prompt tokens' KV cache by reprocessing them through all layers at turn boundaries (accepting the compute cost at turn transitions but maintaining quality), (ii) adding a consistency loss during distillation that penalizes deviations between teacher and student across multi-turn rollouts, or (iii) mixing full-depth and SwiftKV-compressed representations in the KV cache to reduce the effective depth disparity between prompt and generated tokens. This experiment would determine whether SwiftKV is suitable for chatbot applications (its stated target) or is limited to single-turn workloads like batch summarization.

Practical Applications and Downstream Use Cases

Cost-efficient batch inference for document processing pipelines. Organizations that process large document collections through LLMs — for summarization, entity extraction, classification, or RAG indexing — submit long documents as prompts and receive short structured outputs. For example, a legal document processing pipeline might submit 5000-token contracts and receive 200-token summaries, a 25:1 ratio. At this ratio, prefill dominates total compute by over 95%. Deploying SwiftKV at 50% reduction on Llama-3.1-70B would approximately halve the inference cost per document (since prefill FLOPs are roughly halved and decode is negligible), while the paper's evidence suggests summarization-quality degradation would likely fall within the 1–2% average observed on the benchmark suite — though this specific task was not evaluated. On the throughput side, Figure 3 shows 1.4–1.5× higher combined throughput at 2000–8000 token inputs for 70B, meaning a fixed GPU allocation could process roughly 50% more documents per hour. For a pipeline processing millions of documents monthly, this translates directly to reduced GPU-hours and cloud costs.

Latency reduction for interactive coding assistants. Code completion tools like GitHub Copilot submit context-rich prompts (the code file up to the cursor, potentially thousands of tokens of surrounding context) and generate short completions (tens of tokens). At a typical 50:1 ratio, prefill dominates latency. Figure 4 shows that SwiftKV reduces time-to-first-token (TTFT) by up to 50% for Llama-3.1-70B at moderate input lengths — and interactive coding is highly TTFT-sensitive because developers perceive any delay between typing and seeing suggestions. At 2000-token inputs (a reasonable code context window), the baseline TTFT of ~0.5–0.7s drops to ~0.25–0.35s with SwiftKV at low arrival rates. This moves the interaction from "noticeable pause" to "near-instantaneous" in terms of human perception thresholds (~200–300ms). Additionally, the time-per-output-token reduction of up to 60% (Figure 4, moderate-to-high arrival rates) means that once the first token appears, the completion streams faster — further reducing the perceived latency. The paper's GSM8K results (a math reasoning task, not code generation) provide imperfect but suggestive evidence that reasoning quality is preserved — a 3.11-point drop at 50% SwiftKV on Llama-3.1-8B, partially recoverable with additional math+code distillation data (Table B.3b). A deployment would require code-specific evaluation (HumanEval, MBPP) not provided in the paper.

High-throughput RAG serving for enterprise search applications. Retrieval-augmented generation systems, increasingly common in enterprise knowledge management, submit prompts containing the user query plus multiple retrieved document chunks — easily reaching 4000–8000 tokens of context — and generate concise answers (100–300 tokens). At an 8000:200 (40:1) ratio, prefill dominates. Figure 3 shows that at 8000-token inputs on Llama-3.1-70B, 50% SwiftKV achieves 1.5–1.6× higher combined throughput than the baseline, with 4× AcrossKV pushing to 1.6–1.7×. For a service handling hundreds of queries per second, this means handling peak load with ~40% fewer GPUs, or serving ~60% more queries with the same GPU allocation. The paper's MMLU results (a factual knowledge benchmark, which approximates the kind of knowledge retrieval RAG requires) show minimal degradation: at 50% SwiftKV on Llama-3.1-70B, MMLU drops from 83.97% to 82.51% — a 1.46-point change, with 4× AcrossKV bringing it to 82.60%. This suggests that RAG answer quality would be largely preserved. The critical caveat is that the paper does not evaluate RAG-specific metrics (retrieval accuracy, answer faithfulness, groundedness), and the interaction between SwiftKV's substituted KV cache and the model's ability to attend to specific retrieved passages has not been characterized. A deployment should include A/B testing on RAG-specific quality metrics before full rollout.

On-device or edge deployment with memory-constrained hardware. While the paper's primary results target datacenter GPUs, the combination of SwiftKV's compute reduction and AcrossKV's memory reduction directly addresses the constraints of edge deployment. Table 3 shows that at 20GB of GPU memory (simulating a consumer GPU or edge accelerator), the baseline Llama-3.1-8B achieves only 10.8K tokens/s — and SwiftKV 50% without AcrossKV achieves 12.2K (a modest 13% improvement), because the system is memory-bound (KV cache consumes available memory, limiting batch size). Adding 4× AcrossKV raises throughput to 18.0K (67% improvement over baseline) by freeing KV cache memory for larger batches. Adding FP8 quantization further raises it to 23.2K (115% improvement). This combination — 50% SwiftKV + 4× AcrossKV + FP8 — achieves 68.75% KV cache memory reduction and 50% prefill compute reduction while degrading average accuracy by only 2.36 points from the baseline (Table B.1: 73.71% → 71.35%). For an edge deployment where GPU memory is the primary constraint and throughput directly impacts the number of concurrent users that can be served, SwiftKV + AcrossKV + quantization provides a path to running capable models (8B parameters) on hardware that would otherwise be limited to smaller models or severely restricted batch sizes. The paper provides a concrete configuration recipe: 50% SwiftKV, 4-way AcrossKV, FP8 KV cache quantization, distilled on the public dataset mixture.