ArXiv: 2310.17157

🎯 Pitch

LLMs waste most of their inference time waiting on memory, not computing—a single token generation step on OPT-175B spends 145× more time on I/O than actual math. DEJAVU slashes this latency by over 2× by predicting which attention heads and MLP neurons each input actually needs, proving that per-input sparsity can be both accurate and hardware-friendly.

1. Executive Summary

This paper proposes DEJAVU, a system that exploits contextual sparsity—input-dependent subsets of attention heads and MLP parameters that produce approximately the same output as the full dense model—to accelerate LLM inference without retraining or quality degradation. Evaluating on OPT-175B across language modeling (WikiText, C4) and seven downstream tasks, DEJAVU uses lightweight learned predictors to select which structured sparsity pattern (heads and neurons) to apply at each layer, combined with an asynchronous lookahead mechanism that overlaps prediction with computation by exploiting the observation that token embeddings change slowly across consecutive layers. The system achieves over 2× end-to-end latency reduction compared to the state-of-the-art FasterTransformer and over 6× compared to Hugging Face at batch size 1, with no accuracy drop until 75% sparsity, establishing that contextual sparsity can be accurately predicted and hardware-efficiently exploited on modern GPUs without compromising the model's in-context learning ability only when the sparsity is predicted from per-layer contextualized activations rather than from static or non-contextual token embeddings.

2. Context and Motivation

The Core Problem: LLM Inference Is Computationally Prohibitive for Latency-Sensitive Applications

The central challenge this paper tackles is the computational cost of running large language models at inference time, particularly for applications where latency matters. While models like GPT-3, PaLM, and OPT with hundreds of billions of parameters have demonstrated remarkable capabilities—including emergent in-context learning, where they perform new tasks by conditioning on input-output examples without parameter updates—deploying them in production is expensive. As the paper states, LLMs are "very expensive at inference time, especially for latency-sensitive applications."

The bottleneck is not compute throughput (FLOPs) but memory I/O. Table 1 in the paper quantifies this: generating 128 tokens on OPT-175B requires approximately 17.87 ms of actual compute but 2600 ms of I/O latency—a ratio of roughly 145:1. The GPU's tensor cores sit idle most of the time while waiting for weights to be loaded from memory. This I/O dominance means that traditional efficiency metrics like FLOP reduction can be misleading: a method that cuts FLOPs by 90% might yield zero wall-clock speedup if it doesn't reduce the bytes that must be read from GPU memory.

The paper focuses specifically on the token generation phase (autoregressive decoding) rather than the prompt processing phase because, as Table 1 shows, generation dominates end-to-end latency. In the prompt phase, processing 128 tokens requires loading 330 GB of parameters; in generation, producing those same 128 tokens requires loading 41 TB—each token generation step must reload the entire model from GPU memory. The math is simple: for OPT-175B with ~350GB of parameters, generating 128 tokens means moving 128 × 350GB ≈ 41TB of data through the memory bus, compared to 350GB once for processing the prompt.

Three Failure Modes of Existing Sparsity Approaches

Sparsity—selectively skipping computation on parameters that don't contribute meaningfully to the output—is a natural solution. If 80% of parameters can be ignored per input, I/O drops proportionally. But the paper identifies three specific reasons why existing sparsity methods have not seen wide adoption for LLMs:

Failure Mode 1: Retraining or Iterative Pruning Is Infeasible at Scale

Methods from the iterative pruning literature and Lottery Ticket Hypothesis require multiple rounds of training—prune, retrain, repeat—to identify which weights matter. At the scale of OPT-175B, even a single training run is enormously expensive. Requiring retraining makes these approaches a non-starter. As the paper puts it, "it is infeasible to retrain or iteratively prune models at the scale of hundreds of billions of parameters." This eliminates a large body of prior pruning work, which was developed and validated primarily on much smaller models (e.g., ResNet-50 on ImageNet).

Failure Mode 2: Task-Specific Pruning Conflicts with In-Context Learning

Several works, including Michel et al. (2019) and Bansal et al. (2022), have shown that pruning can be effective when done in a task-dependent manner—identifying which heads or neurons are important for a specific downstream task and removing the rest. However, this creates a fundamental tension with the raison d'être of LLMs: their ability to perform arbitrary tasks at inference time without fine-tuning. Maintaining separate pruned models for each downstream task defeats the purpose of having a single general-purpose model. The paper notes that this "conflicts with the task independence goal of LLMs." A pruned model for sentiment analysis would fail on question answering; a pruned model for translation would lose its summarization ability. The in-context learning paradigm demands a single set of parameters that can be queried flexibly.

Failure Mode 3: Unstructured Sparsity Doesn't Yield Wall-Clock Speedup

Perhaps the most damning practical limitation is that unstructured sparsity—removing individual weights scattered arbitrarily throughout a matrix—does not translate to runtime improvements on modern hardware. GPUs are block-oriented devices: loading a single byte from memory takes roughly the same time as loading a 128-byte aligned block around that address (the cache line size for NVIDIA GPUs). Unstructured sparsity means that the "kept" weights are scattered at random offsets, so each weight access requires a separate memory transaction that loads 128 bytes but uses only 2 bytes (FP16). The math is brutal: at 60% unstructured sparsity, you're still loading roughly the same number of cache lines as a dense matrix—the 40% of kept weights are likely distributed across most cache lines. The paper cites SparseGPT (Frantar & Alistarh, 2023) as a concrete example: it achieves 60% unstructured sparsity on LLMs but "does not yet lead to any wall-clock time speedup."

The hardware limitation is not incidental but fundamental. Modern GPUs are optimized for coalesced memory access—loading contiguous blocks—which maps naturally to dense matrix operations. Sparse operations, by contrast, involve irregular memory access patterns that underutilize memory bandwidth. The paper's own benchmarks (Figures 13 and 14 in the appendix) show that a naive PyTorch sparse implementation is 4–5× slower than dense for a given density level, because separately indexing a subset of the weight matrix before multiplying incurs 3× the memory I/O: one read to load the indices from GPU memory, one write to copy the selected weights to a contiguous buffer, and one read to load that buffer for the actual multiplication.

The Ideal Sparsity: Three Requirements

The paper synthesizes these failures into a clear set of desiderata for any sparsity approach targeting LLM inference. An ideal method must:

  1. Not require model retraining — it must work with the weights as they are, post-pretraining.
  2. Preserve quality and in-context learning ability — the model must remain a general-purpose few-shot learner, not a collection of task-specific sub-networks.
  3. Lead to speedup in wall-clock time on modern hardware — the sparsity pattern must be structured (heads, neurons, or entire matrix columns/rows rather than individual weights) so that memory access remains coalesced.

These three requirements jointly rule out the existing landscape: iterative pruning fails (1), task-specific pruning fails (2), and unstructured sparsity fails (3).

The Gap: No Unified Framework for Input-Dependent Structured Sparsity

The paper identifies a crucial observation that prior work missed: pre-trained LLMs naturally exhibit input-dependent sparsity patterns. That is, for a given input, only a small subset of attention heads and MLP neurons produce outputs with meaningful norms; the rest contribute near-zero values that could be skipped. This is not a learned or imposed property—it emerges from the model's trained weights and the specific input it processes.

The intuition builds on two observations:

  • MLP blocks: Activation functions like ReLU and GeLU naturally produce sparse outputs—any neuron receiving a negative pre-activation outputs exactly zero (ReLU) or near-zero (GeLU). This means that for a given input, the effective computation involves only a subset of neurons. Prior work (Li et al., 2022; Kurtz et al., 2020) had noted activation sparsity, but had not connected it to hardware-efficient structured sparsity for LLM inference.

  • Attention blocks: Different attention heads specialize in different token interactions. Some heads are "heavy hitters" that attend strongly to specific tokens (e.g., the head attending to "like" or "shipping" in Figure 4's example); others are relatively uniform token-mixing heads that distribute attention broadly. For predicting the next token "Truck" in the sentence "This fruit shipping company provide different vehicle options like car and...", only the heads that focus on semantically relevant tokens matter; the uniform heads contribute noise rather than signal.

The paper frames this as a contextual sparsity hypothesis: "for pre-trained LLMs, contextual sparsity exists given any input." The term "contextual" is deliberate—it emphasizes that the sparsity pattern depends on the specific input context, not just the static model architecture. Different sentences activate different heads and neurons. A head that is critical for one example might be irrelevant for another.

This hypothesis, if true and if the sparsity patterns can be predicted before computing them, offers a path around all three failure modes: no retraining needed (the model weights stay frozen), in-context learning is preserved (all parameters remain available and are selected on-the-fly per input), and structured sparsity (entire heads or columns) maps naturally to coalesced memory access.

The Three Challenges: Existence, Prediction, and Efficiency

The paper distills the research problem into three concrete challenges that must be solved to realize contextual sparsity in practice:

Existence: Can we verify that contextual sparsity actually exists in pre-trained LLMs, and that skipping the "unimportant" parameters yields approximately the same output as the full model? A naive verification—trying all possible subsets—would be combinatorially prohibitive.

Prediction: Even if such sparsity exists, can we predict which parameters are important for a given input without first computing the full dense output? This is the chicken-and-egg problem: we want to skip parameters to save computation, but we need to know which parameters to skip, which seems to require computing with all of them.

Efficiency: Can the prediction itself be made cheap enough—and the sparse computation itself hardware-efficient enough—that the end-to-end wall-clock time actually decreases? The paper's motivating numbers are sobering: on an 8×A100 system, one MLP block's computation takes only 0.2 ms. If the sparsity predictor adds even 0.3 ms of overhead, total latency increases despite computing fewer FLOPs.

The third challenge is especially acute for autoregressive generation because each token generation step is already I/O-bound with low arithmetic intensity. Any prediction overhead that isn't carefully hidden will dominate the savings.

How This Paper Positions Itself

DEJAVU positions itself not as a new pruning method or a new model architecture, but as a system for exploiting a property (contextual sparsity) that already exists in pre-trained LLMs. The core intellectual move is to reframe the problem from "what should we remove from the model?" (static pruning) to "what should we use for this particular input?" (dynamic, contextual sparsity). This reframing is what enables the approach to satisfy all three requirements simultaneously.

The paper draws on several threads from prior work but combines them in a novel way:

  • From the pruning literature, it borrows the idea that structured sparsity (heads, neurons) is hardware-friendly, but rejects the requirement of retraining or iterative pruning.
  • From work on activation sparsity (Li et al., 2022; Kurtz et al., 2020), it takes the observation that ReLU/GeLU activations are naturally sparse, but extends it to attention heads where no activation function enforces sparsity.
  • From the nearest neighbor search literature, it adopts the formulation of sparsity prediction as a maximum inner product search (MaxIP) problem, but replaces traditional NNS data structures (LSH, HNSW, FAISS) with learned neural network classifiers to exploit GPU-accelerated matrix multiplication.
  • From observations about residual connections in computer vision (He et al., 2016; Veit et al., 2016), it hypothesizes—and verifies—that token embeddings change slowly across layers in LLMs, enabling asynchronous lookahead prediction that hides the predictor's latency.

The paper's contribution is thus primarily empirical and systems-oriented: it verifies that contextual sparsity exists at high levels (~85% total structured sparsity), shows that it can be predicted accurately from per-layer contextualized activations (but not from static token embeddings), and demonstrates that careful hardware-aware implementation can translate these theoretical savings into 2× wall-clock speedup over the state-of-the-art FasterTransformer library—all without modifying the pre-trained weights or sacrificing in-context learning performance.

3. Technical Approach

3.1 Reader orientation (approachable technical breakdown)

DEJAVU is a runtime system that sits between the LLM's parameters and the GPU execution engine, deciding at each layer which small subset of attention heads and MLP neurons are needed for the current input token and loading only those. The core problem it solves is that the necessary subset changes with every input, yet we cannot afford to compute the full layer output just to discover which subset matters—the prediction must be fast, accurate, and hardware-efficient enough that the savings from skipping parameters exceed the cost of deciding which to skip.

3.2 Big-picture architecture (diagram in words)

The system has four major components:

  1. Base LLM (OPT-175B, frozen weights): The pre-trained model whose weights are never modified. It serves as the ground-truth oracle for what contextual sparsity looks like.
  2. Sparse predictors (one small neural network per layer per block type): Given the contextualized activation entering a layer, each predictor outputs a binary mask indicating which heads (for attention) or which neurons (for MLP) need to be computed. These are trained offline once per model.
  3. Asynchronous lookahead scheduler: Overlaps predictor execution with the previous block's computation by exploiting the observation that activations change slowly across consecutive layers. The predictor for layer $l+1$ runs while layer $l$ is computing, hiding prediction latency.
  4. Hardware-efficient sparse kernels: Custom Triton kernels that fuse the indexing operation (selecting $W[\text{idx}, :]$ from weight matrices) with the matrix-vector multiply, and store weight matrices in column-major format where needed for coalesced memory access.

Information flows as follows: an input token embedding enters layer $l$ → the predictor (already computed from the previous layer's activation) provides a binary mask → the sparse attention kernel loads only the selected heads, computes their output, and updates the KV cache → the sparse MLP kernel loads only the selected neurons, computes their output → the activation passes to layer $l+1$ → simultaneously, the predictor for layer $l+1$ runs using the layer $l$ activation as input.

3.3 Roadmap for the deep dive

  • First, the formal definitions of sparsified MLP and attention blocks (Section 2.3), which establish the precise mathematical operations that contextual sparsity replaces and the optimization objective.
  • Second, the verification methodology for contextual sparsity existence (Section 3.1), which shows that sparsity exists before addressing how to predict it.
  • Third, the theoretical framework for why contextual sparsity emerges (Sections 3.2–3.3), covering the mean-shift clustering interpretation of attention and the slowly-changing-embedding phenomenon, because these observations motivate the design of both the predictors and the asynchronous scheduler.
  • Fourth, the MLP sparsity predictor design and training (Section 4.1), starting with the simpler case where sparsity is induced by activation functions.
  • Fifth, the attention sparsity predictor design and training (Section 4.2), which builds on the MLP approach but must handle the dependence on past tokens and KV cache management.
  • Sixth, the asynchronous lookahead mechanism (Section 4.3), which is the key systems innovation that makes prediction overhead vanish.
  • Seventh, the hardware-efficient implementation (Section 4.4), covering kernel fusion, memory coalescing, and why these matter for GPU I/O patterns.

3.4 Detailed, sentence-based technical breakdown

This is primarily a systems paper with substantial empirical analysis, whose core idea is that pre-trained LLMs naturally contain input-dependent structured sparsity that can be predicted cheaply and exploited for wall-clock speedup—without retraining, without sacrificing in-context learning, and with careful attention to GPU memory access patterns.

Formal Problem Definition: Sparsified MLP and Attention Blocks

Before discussing how DEJAVU works, we must understand precisely what operations it replaces. The paper defines sparsified versions of the two computationally dominant components in transformer layers: the MLP block and the multi-head attention block (Section 2.3).

Sparsified MLP. A standard MLP block in OPT consists of two linear layers with an activation function $\sigma$ (typically ReLU for OPT models) between them:

MLP(y)=σ(yW1)(W2)⊤\text{MLP}(y) = \sigma(y W^1) (W^2)^\top

where $y \in \mathbb{R}^{1 \times d}$ is the input activation vector from the previous component (layer norm output), $W^1 \in \mathbb{R}^{d \times 4d}$ is the first (up-projection) weight matrix, $W^2 \in \mathbb{R}^{d \times 4d}$ is the second (down-projection) weight matrix, and $\sigma$ is applied element-wise.

What this computes: The input vector $y$ (representing the current token's contextualized embedding) is projected to a 4× wider hidden dimension via $W^1$, filtered through a non-linear activation, then projected back to the original dimension via $W^2$. This is the standard feed-forward transformation in every transformer layer.

With contextual sparsity, only a subset of the $4d$ neurons—those that would produce meaningfully non-zero activations for this specific input—are computed. Formally, let $S_M \subseteq [4d]$ denote the set of selected neuron indices for input $y$. The sparsified computation is:

MLPSM(y)=σ(yWSM1)(WSM2)⊤\text{MLP}_{S_M}(y) = \sigma(y W^1_{S_M}) (W^2_{S_M})^\top

where $W^1_{S_M} \in \mathbb{R}^{d \times |S_M|}$ is the submatrix formed by taking only the columns of $W^1$ indexed by $S_M$, and similarly $W^2_{S_M}$ contains only the corresponding rows.

Why this form: Structured sparsity at the neuron granularity means we are selecting entire columns of $W^1$ and entire rows of $W^2$. This is critical for hardware efficiency because columns in row-major storage are contiguous in memory, so loading $W^1_{S_M}$ involves reading $|S_M|$ contiguous blocks of $d$ elements each—perfectly coalesced memory access. If we instead selected individual weights (unstructured sparsity), each weight would be at a non-contiguous offset, requiring separate cache-line loads and wasting bandwidth.

Sparsified attention. A standard multi-head attention block with $h$ heads computes, for input $y \in \mathbb{R}^{1 \times d}$ and past token embeddings $X \in \mathbb{R}^{n \times d}$:

MHA(y)=∑i=1hHi(y)WiO\text{MHA}(y) = \sum_{i=1}^{h} H_i(y) W^O_i

where each head $i$ computes:

Hi(y)=Di(y)−1exp⁡(yWiQ(WiK)⊤X⊤)XWiVH_i(y) = D_i(y)^{-1} \exp(y W^Q_i (W^K_i)^\top X^\top) X W^V_i

and $D_i(y) = \exp(y W^Q_i (W^K_i)^\top X^\top) \mathbf{1}_n$ is the row-wise normalisation factor (softmax denominator). Here $W^Q_i, W^K_i, W^V_i \in \mathbb{R}^{d \times d_h}$ are the query, key, and value projection matrices for head $i$, $d_h = d/h$ is the per-head dimension, and $W^O_i \in \mathbb{R}^{d_h \times d}$ is the output projection.

What this computes: For each attention head, the current token $y$ is used as a query to compute similarity scores with all past tokens (via $y W^Q_i (W^K_i)^\top X^\top$). These scores are softmax-normalised and used as weights to aggregate the value projections $X W^V_i$ into a single vector. The heads' outputs are then linearly combined via the output projections $W^O_i$.

With contextual sparsity, only a subset $S_A \subseteq [h]$ of attention heads are computed:

MHASA(y)=∑i∈SAHi(y)WiO\text{MHA}_{S_A}(y) = \sum_{i \in S_A} H_i(y) W^O_i

Why this form: Selecting entire attention heads preserves the structured nature of the computation. Each head involves four weight matrices ($W^Q_i, W^K_i, W^V_i, W^O_i$) that are contiguous in memory when concatenated. Skipping a head means skipping all four matrices and the associated attention score computation over all past tokens, which is the most memory-intensive part because it requires loading the full $X \in \mathbb{R}^{n \times d}$ KV cache.

The fundamental optimization objective. For both MLP and attention, given a compute budget (sparsity target), the goal is:

min⁡SM∥MLPSM(y)−MLP(y)∥2andmin⁡SA∥MHASA(y)−MHA(y)∥2\min_{S_M} \|\text{MLP}_{S_M}(y) - \text{MLP}(y)\|_2 \quad \text{and} \quad \min_{S_A} \|\text{MHA}_{S_A}(y) - \text{MHA}(y)\|_2

subject to $|S_M| \leq (1 - \text{sparsity}) \cdot 4d$ and $|S_A| \leq (1 - \text{sparsity}) \cdot h$.

In operational terms: find the smallest subset of neurons and heads whose combined output is $\ell_2$-close to the full model's output. This is a combinatorial optimisation problem—there are $\binom{4d}{|S_M|}$ possible neuron subsets—so exact solution is impossible. The paper's approach is to learn a predictor that directly estimates $S_M$ and $S_A$ from the input activation, trained on ground-truth subsets identified via two forward passes.

Verification of Contextual Sparsity Existence (Section 3.1)

Before designing predictors, the paper must establish that contextual sparsity actually exists at useful levels in pre-trained LLMs. The verification methodology is elegantly simple and requires only two forward passes.

The two-pass verification procedure:

Pass 1 (recording): Run the full dense model on an input example. At each layer, record two pieces of information:

  • For the MLP block: which neurons produce activation values (after $\sigma$) with large output norms. Specifically, a neuron $j$ is recorded if its contribution to the output—the norm of $\sigma(y W^1_j) \cdot (W^2_j)^\top$—exceeds a threshold.
  • For the attention block: which heads produce output vectors (after $W^O_i$) with large norms. A head $i$ is recorded if $\|H_i(y) W^O_i\|_2$ exceeds a threshold.

Pass 2 (sparsified): Run the model again on the same input, but at each layer, only compute the recorded subset of neurons and heads. All other parameters are skipped entirely—their contributions are treated as zero.

Observation: The two passes—one dense, one sparse—produce "similar prediction or performance on all in-context learning and language modeling tasks" (Section 3.1). This directly verifies the hypothesis: the full model's output can be closely approximated using only a fraction of its parameters, and that fraction is input-dependent (different examples activate different subsets).

Quantitative results (Figure 3):

  • MLP sparsity (Figure 3b): On average, over 95% of MLP neurons can be zeroed out for a given token. The sparsity varies across layers but remains consistently above 90% for all layers in OPT-175B, OPT-66B, and OPT-30B.
  • Attention head sparsity (Figure 3a): On average, over 80% of attention heads can be silenced for a given token. The percentage of "not-activated" heads fluctuates across layers—some layers have higher head utilisation than others—but no layer requires more than ~50% of heads.
  • Total contextual sparsity: Since OPT-175B has 2× more MLP parameters than attention parameters, the weighted average sparsity is approximately 85% structured sparsity, which corresponds to a potential 7× parameter reduction per input.

Why this is surprising for attention: The paper notes that while MLP sparsity is "intuitive... because of their activation functions, e.g., ReLU or GeLU" (which naturally zero out negative pre-activations), attention sparsity is genuinely surprising. There is no activation function in the attention computation that zeros out heads; the head selection emerges purely from the learned projections. Attention heads with small output norms are those where the softmax scores and value projections combine to produce vectors with small magnitude—a property of the interaction between query, key, and value matrices, not a single non-linearity.

Crucial distinction from static head pruning: The paper cross-checks that "different examples have different contextual sparsity." Although 80% of attention heads are not used for a given example, "they might be used by other examples." This is the essential difference from static pruning: no parameter is permanently removed; the selection is dynamic and input-dependent.

Insight for prediction design: The verification procedure reveals that the key signal for both MLP and attention is the output norm. For MLP, this is the activation magnitude after $\sigma$; for attention, it is the norm of the head's output vector $H_i(y) W^O_i$. This norm-based criterion is what the predictors will be trained to approximate—they must learn to map from the input activation to which neurons/heads will produce large norms, without computing those norms.

Understanding Why Contextual Sparsity Exists: Self-Attention as Mean-Shift Clustering (Section 3.2)

The paper provides a theoretical analysis to explain why attention heads naturally specialise in ways that lead to input-dependent sparsity. This analysis is not merely interpretive—it directly motivates the predictor design by showing that head importance can be inferred from the similarity between the input activation and the head's projection matrices.

The observation (Figure 4): Visualising attention scores from three different heads in the same layer for a single example reveals heterogeneous behaviour. Two heads (Head 42 and Head 44) are "heavy hitters"—they concentrate high attention scores on specific tokens ("like", "shipping") that are semantically relevant for predicting the masked token "Truck". One head (Head 43) is a uniform token-mixing head—its attention scores are distributed roughly evenly across all tokens.

The hypothesis: Each attention head performs one step of mean-shift clustering, where the query vector $y$ is iteratively pulled toward the weighted centroid of tokens that are similar to it under a learned kernel. Only heads whose clustering direction aligns with the prediction task produce large output norms; uniform-mixing heads contribute noise that gets averaged away or suppressed by subsequent layers.

The formal argument: Recall the head computation from Section 2.3:

Hi(y)=∑jexp⁡(yWiQ(WiK)⊤xj)⋅xjWiV∑jexp⁡(yWiQ(WiK)⊤xj)H_i(y) = \frac{\sum_j \exp(y W^Q_i (W^K_i)^\top x_j) \cdot x_j W^V_i}{\sum_j \exp(y W^Q_i (W^K_i)^\top x_j)}

Define the kernel function $K_i(x_j, y) = \exp(y W^Q_i (W^K_i)^\top x_j)$ measuring similarity between $x_j$ and $y$ in the projected space of head $i$, and define the weighted centroid:

mi(y)=∑jKi(xj,y)⋅xj∑jKi(xj,y)m_i(y) = \frac{\sum_j K_i(x_j, y) \cdot x_j}{\sum_j K_i(x_j, y)}

Then $H_i(y) = m_i(y) W^V_i$—the head's output is the centroid of past tokens (in original embedding space) linearly transformed by $W^V_i$.

The mean-shift connection: If we hypothetically set $W^V_i = I$ (identity) and consider the residual connection followed by layer norm, the embedding of the current token after this head becomes:

y^i=Normalize(y+Hi(y))=Normalize(y+mi(y))\hat{y}_i = \text{Normalize}(y + H_i(y)) = \text{Normalize}(y + m_i(y))

This has a fixed point $y = \gamma m_i(y)$ for some scalar $\gamma$. In mean-shift clustering, the algorithm iterates $y \leftarrow m_i(y)$ until convergence, which has the identical fixed point $y = m_i(y)$. Thus, "the self-attention head can be regarded as one mean-shift step to push input embeddings of different tokens together, if they are already neighbors in a projection space specified by $W^Q_i (W^K_i)^\top$."

The implication for sparsity: Different heads learn different projection spaces $W^Q_i (W^K_i)^\top$, each clustering tokens along a different semantic dimension. After a few layers, token embeddings become well-clustered, so most heads perform clustering on embeddings that are already grouped—their $m_i(y)$ is close to $y$ and their output norm $\|m_i(y) - y\|$ is small. Only the heads whose projection space aligns with a relevant distinction for the current token produce large shifts (and hence large output norms). These are the "activated" heads worth computing.

Why this motivates similarity-based prediction: The analysis shows that a head's output norm depends on how far the weighted centroid $m_i(y)$ is from the query $y$ in the projection space of that head. This distance is driven by the inner products $\langle y, W^Q_i (W^K_i)^\top x_j \rangle$. The predictor can therefore estimate head importance from the similarity between the input activation and the head's parameters—specifically, the query projection $W^Q_i$—because large similarity with certain past tokens is what drives $m_i(y)$ away from $y$.

Slowly Changing Embeddings Across Layers (Section 3.3)

The paper identifies a critical empirical property that makes the asynchronous predictor design feasible: token embeddings change extremely slowly between consecutive layers due to the dominant residual connections.

The observation (Figure 5a): Compute the cosine similarity between the activation vectors at layer $l$ and layer $l+1$ for the same input token across all layers of OPT-175B. Starting from layer 2 onward, the median cosine similarity is approximately 0.99—the direction of the embedding vector changes by less than 1% per layer. Figure 5a shows this is consistent across all OPT model sizes from 125M to 175B parameters, with larger models tending to have even higher similarities.

The observation for multi-layer gaps (Figure 5b): Even when looking across multiple layers, the similarity remains high: between layer $l$ and layer $l+4$, cosine similarity is still above 0.97 for deeper layers in OPT-175B. The similarity decreases as the gap $n$ increases (as expected), but the rate of change is slow enough that a one-layer lookahead is highly reliable.

Why this happens—the residual structure: Each transformer layer contains two residual connections:

yafter-attn=y+MHA(y)y_{\text{after-attn}} = y + \text{MHA}(y) yafter-mlp=yafter-attn+MLP(yafter-attn)y_{\text{after-mlp}} = y_{\text{after-attn}} + \text{MLP}(y_{\text{after-attn}})

The paper dissects the norms of these components (Figure 5c and 5d). At every layer except the first, $\|F(X)\|$—the norm of the attention output or MLP output—is significantly smaller than $\|X\|$, the norm of the residual stream. For OPT-175B, $\|X\|$ is approximately 1000–2000 while $\|F(X)\|$ is approximately 20–100 in most layers. The cosine similarity between $X$ and $X + F(X)$ is near 1.0 when $\|F(X)\| \ll \|X\|$.

The sparsity-residual feedback loop: The paper hypothesises that high sparsity causes the small $\|F(X)\|$: if 95% of MLP neurons produce near-zero outputs and 80% of attention heads produce small-norm outputs, then the aggregate contribution $F(X)$ is naturally small. This in turn means the embedding changes slowly, which means the sparsity pattern is predictable from the previous layer's activation. The phenomenon is self-reinforcing: sparsity enables slow change, and slow change enables sparsity prediction.

Formal justification (Lemma 3.1): The paper provides a theoretical bound showing that for both MLP and attention blocks under the computation model, there exist bounds $0 < \epsilon_1 < \epsilon_2 < 1$ such that the residual satisfies:

ϵ1≤∥yafter−ybefore∥2≤ϵ2\epsilon_1 \leq \|y_{\text{after}} - y_{\text{before}}\|_2 \leq \epsilon_2

This formalises the "shrinking property"—the residual connection always produces a bounded change, neither zero (the identity is perturbed) nor arbitrarily large (the residual is small). The proof sketch uses random matrix theory to show that with high probability, the output norms of the random projections in attention and MLP preserve norms within constant factors.

The implication for DEJAVU's design: Because embeddings evolve slowly, the activation at layer $l$ is a good proxy for the activation at layer $l+1$. Therefore, the sparse predictor for layer $l+1$ can use the activation from layer $l$ as input and run in parallel with layer $l$'s computation. The prediction is available by the time layer $l+1$ starts computing, effectively hiding the predictor's latency. This is the core enabling insight for the asynchronous lookahead scheduler.

MLP Sparsity Predictor: Formulation and Training (Section 4.1)

The MLP block presents the simpler prediction case because the sparsity is primarily determined by the activation function: a neuron $j$ is "activated" if $\langle y, W^1_j \rangle > 0$ (for ReLU), and its output norm scales with this inner product.

The challenge: Figure 3b shows that 95% contextual sparsity is achievable, but "this only demonstrates the existence of contextual sparsity but brings no benefits in terms of efficiency. A fast and precise prediction is needed to exploit contextual sparsity for end-to-end efficiency." Computing the full $y W^1$ to discover which neurons have positive pre-activations would defeat the purpose—we would have already done the dense computation.

Why random selection fails: The paper states that "random selection fails to identify the accurate contextual sparsity, resulting in drastic model degradation." This is unsurprising: randomly picking 5% of neurons would miss many with large positive pre-activations and include many with negative pre-activations (which ReLU zeroes out), producing output that diverges substantially from the dense model.

Formulation as a nearest-neighbor search problem: The key observation is that for ReLU-based MLPs, a neuron $j$ is important for input $y$ if the inner product $\langle y, W^1_j \rangle$ is large (positive). The task is therefore: given the query vector $y$ and the dataset of neuron weight vectors $\{W^1_1, \ldots, W^1_{4d}\}$, find the neurons with the highest inner products. This is the Approximate Maximum Inner Product Search (MaxIP) problem.

Formal MaxIP definition (Definition 4.1): Given parameters $c \in (0, 1)$ and $\tau \in (0, 1)$, a dataset $W^1 \subset \mathbb{S}^{d-1}$ on the unit sphere, and a query $y \in \mathbb{S}^{d-1}$ such that $\max_{w \in W^1} \langle y, w \rangle \geq \tau$, the $(c, \tau)$-MaxIP problem is to retrieve a vector $z \in W^1$ satisfying:

⟨y,z⟩≥c⋅max⁡w∈W1⟨y,w⟩\langle y, z \rangle \geq c \cdot \max_{w \in W^1} \langle y, w \rangle

What this means operationally: Find neurons whose inner product with $y$ is at least a fraction $c$ of the maximum possible inner product. The parameter $c$ controls the approximation quality—$c = 1$ would require finding the exact maximum, which is computationally expensive; values like $c = 0.8$ or $c = 0.9$ make the search tractable while still capturing the most important neurons.

Why standard NNS approaches are too slow: The paper benchmarks standard nearest-neighbor search methods on the OPT-175B setting where $d = 12288$ and $4d = 49152$ neurons. HNSW (hierarchical navigable small world graphs) takes over 10 ms per query; FAISS (GPU-accelerated similarity search) takes over 4 ms. But the dense MLP computation itself takes only 0.2 ms. Any prediction taking more than 0.2 ms would increase latency rather than decrease it, even with 95% sparsity in the subsequent computation.

The fundamental issue is that traditional NNS data structures (LSH hash tables, graph indices) are designed for CPU execution and amortised over many queries. In the autoregressive generation setting, we have exactly one query per layer per token, and the query dimensionality (12288) is very high—a regime where exact or near-exact indexing overhead dominates.

The solution: a learned neural network classifier as an NNS proxy. Rather than using a general-purpose NNS data structure, the paper trains a small two-layer fully connected network for each MLP block that directly predicts which neurons will activate. This exploits the GPU's strength—fast matrix multiplication—by formulating the prediction itself as a small neural network forward pass that can run in microseconds.

Predictor architecture: For each MLP block, the predictor $\text{SP}_M$ is a two-layer fully connected network:

SPM(y)=σpred(yWpred1)Wpred2\text{SP}_M(y) = \sigma_{\text{pred}}(y W_{\text{pred}}^1) W_{\text{pred}}^2

where $y \in \mathbb{R}^{1 \times d}$ is the input activation, $W_{\text{pred}}^1$ maps to a hidden dimension (much smaller than $d$), and $W_{\text{pred}}^2$ outputs a scalar per neuron (4d outputs). The output is a binary classification score per neuron: should we include this neuron or skip it?

Training data collection (Algorithm 1): For each MLP block in the pre-trained LLM:

  1. Collect a set of token embeddings $\{x_i\}_{i \in [N]}$ at the input to that block. The paper uses $N = 500$ random data points from the C4 training dataset.
  2. For each embedding $x_i$, run the dense MLP block to obtain the ground-truth activation values.
  3. Apply a threshold $t$ to label each neuron $m_r$: if the neuron's output norm contribution exceeds $t$, label it as positive (should be selected); otherwise, label it as negative.
  4. The predictor is trained as a binary classifier using a loss function $\mathcal{L}$ to distinguish positive from negative neurons.

Why 500 examples suffice: The predictor is learning a mapping from activation space to neuron importance, and this mapping is relatively smooth because similar activations (in cosine distance) activate similar sets of neurons. The paper notes that the validation accuracy is "over 99% in the shallow layers and drops to around 93% in the ending layers," indicating that the prediction task is easier in early layers (presumably because token embeddings are less contextualised and neuron activation patterns are more consistent) and harder in later layers.

Execution at inference time: The prediction is a two-step process:

  1. Given input $y$, the predictor $\text{SP}_M$ produces a set $S_M \subseteq [4d]$ of selected neuron indices.
  2. The sparsified MLP computation $\text{MLP}_{S_M}(y)$ is then executed using only the selected columns and rows.

Design choice: neural network over classical NNS. The paper explicitly justifies this by contrasting the cost models. HNSW: CPU-bound, requires traversing a graph structure, 10 ms. FAISS: GPU-based but still uses hash-table lookups and distance computations on pre-indexed data, 4 ms. Learned predictor: a single matrix multiplication $y W_{\text{pred}}^1$ followed by a second matrix multiplication, running as a GPU kernel, << 0.1 ms. The predictor's forward pass is effectively just a tiny MLP that runs within the GPU's tensor-core-optimised matrix multiply pipeline, making it orders of magnitude faster than general-purpose search.

The trade-off is accuracy: a neural network predictor may miss some neurons that a brute-force MaxIP would find. However, the paper's results (Table 4) show that at 85% MLP sparsity, the predictor introduces "no accuracy loss on both zero-shot tasks and language modeling," confirming that the learned predictor is sufficiently accurate for the downstream task.

Attention Sparsity Predictor: Formulation, KV Cache Management, and Training (Section 4.2)

The attention block presents additional challenges beyond the MLP case. Not only must we predict which heads are important, but we must also handle the dependency on past tokens' KV caches.

The additional challenges:

  1. Past-token dependence: Unlike the MLP block, which operates on the single current-token activation, the attention head's output depends on the full history of past tokens through the key and value projections $X W^K_i$ and $X W^V_i$. It is unclear whether the sparse predictor needs access to these past tokens to make accurate predictions.

  2. Missing KV cache: When a head $i$ is not selected for the current token, its key and value projections are not computed. If that head is selected for a future token, the KV cache will have a "hole"—the missing entry for the current token. This must be handled without recomputing the full key and value projections from scratch.

Is past-token information needed for head prediction? The paper argues no, based on the mean-shift clustering analysis in Section 3.2: after the first few layers, the current token embedding $y$ already contains sufficient contextual information from previous layers' token mixing to serve as a proxy for which clusters it belongs to. The similarity between $y$ and the head's parameters—particularly the query projection $W^Q_i$—is sufficient to determine whether head $i$ will produce a large output norm.

Why this follows from the clustering interpretation: A head produces a large output norm when $y$ is far from the weighted centroid $m_i(y)$ in the head's projection space. This happens when $y$ has high similarity with some past tokens (driving $m_i(y)$ one direction) but low similarity with others. The information about which tokens $y$ is similar to is encoded in $y$ itself—it is a contextualised embedding that already summarises its relationship to the past sequence. Therefore, $y$ alone suffices for the prediction.

Formulation as a MaxIP problem: The paper reframes head prediction as a similarity-based search: each head is associated with parameters (primarily $W^Q_i$), and the predictor estimates the head's output norm from the similarity $\langle y, W^Q_i \rangle$. This is directly analogous to the MLP case where neuron importance depends on $\langle y, W^1_j \rangle$.

Predictor architecture and training: The attention sparse predictor $\text{SP}_A$ has the same architecture as the MLP predictor—a small two-layer fully connected network—with one output per head (classifying each of the $h$ heads as selected or not). Training follows the same Algorithm 1, using the head's output norm as the ground-truth label.

Addressing the missing KV cache problem: The paper's solution exploits an asymmetry in the generation latency profile: generation is I/O-bound (bottlenecked by loading weights from GPU memory), while the actual floating-point computation is essentially "free" (occupying only a fraction of the execution time). This means we can afford extra computation if it avoids extra memory access.

The procedure for handling missing KV cache entries:

  1. For the current token $y$ and the selected heads $S_A$, compute the key and value vectors $y W^K_i$ and $y W^V_i$ for $i \in S_A$ and store them in the KV cache as usual.

  2. Additionally, save a copy of the raw token embedding $y$ in a separate buffer. This is cheap—it is a single vector of dimension $d$ (12288 elements, ~24 KB in FP16)—compared to the KV cache entries which involve projections to $d_h = d/h$ dimensions.

  3. During future token generation, when a head $i$ that was previously not selected becomes selected:

    • Check the KV cache for head $i$: if there are missing entries (tokens where head $i$ was skipped), load the stored raw embeddings for those tokens from the buffer.
    • Recompute the missing key and value vectors on-the-fly: $y_{\text{stored}} W^K_i$ and $y_{\text{stored}} W^V_i$. This requires loading $W^K_i$ and $W^V_i$ from GPU memory, but the main cost—loading the weight matrices—was already going to happen for the current token's attention computation anyway. The additional compute is the matrix-vector multiply, which is negligible compared to the weight-loading I/O.

Why this is efficient: Loading the stored embeddings requires minimal memory access (just the vector itself, not the projected keys/values). The recomputation cost is bounded: for a head that was skipped for $k$ tokens, we need $k$ additional matrix-vector multiplies, each taking microseconds. The dominant cost—loading the weight matrices $W^K_i, W^V_i$—is shared across all tokens anyway.

Empirical training observations: The paper notes different prediction accuracy patterns compared to the MLP predictor: "The validation accuracy is around 93% in the middle layers and near 99% in the shallow and deep layers." This suggests that head importance is more predictable at the extremes of the model—early layers where heads perform generic token-mixing, and late layers where heads are highly specialised—and less predictable in the middle where the model is building complex intermediate representations.

Asynchronous Lookahead Execution (Section 4.3)

The sparse predictor's computation, though fast, still adds latency if executed sequentially before each block. The paper introduces an asynchronous lookahead mechanism that eliminates this overhead by overlapping prediction with computation, made possible by the slowly-changing-embedding phenomenon.

The sequential baseline and its problem: In a naive implementation, the computation at transformer layer $l$ proceeds as:

SAl←SPAl(yl)S^l_A \leftarrow \text{SP}^l_A(y^l) y~l←MHASAll(yl)\tilde{y}^l \leftarrow \text{MHA}^l_{S^l_A}(y^l) SMl←SPMl(y~l)S^l_M \leftarrow \text{SP}^l_M(\tilde{y}^l) y^l←MLPSMll(y~l)\hat{y}^l \leftarrow \text{MLP}^l_{S^l_M}(\tilde{y}^l)

where $y^l$ is the input to layer $l$, $S^l_A$ is the set of selected attention heads, $\tilde{y}^l$ is the output after attention, $S^l_M$ is the set of selected MLP neurons, and $\hat{y}^l$ is the final output of layer $l$ (which becomes $y^{l+1}$).

The sequential bottleneck: The attention computation $\text{MHA}^l_{S^l_A}$ must wait for the predictor $\text{SP}^l_A$ to finish, and the MLP computation $\text{MLP}^l_{S^l_M}$ must wait for $\text{SP}^l_M$. At each block, the predictor adds its own latency to the critical path. If the predictor takes $t_{\text{pred}}$ and the block computation takes $t_{\text{block}}$, the total per-layer latency is $2 \cdot t_{\text{pred}} + t_{\text{block}}$ rather than $t_{\text{block}}$.

The asynchronous redesign: The key insight is that because embeddings change slowly, the sparsity pattern for layer $l+1$ can be predicted from the embedding at layer $l$. This allows the predictors to be pushed back by one layer and run in parallel:

y~l←MHASAll(yl)\tilde{y}^l \leftarrow \text{MHA}^l_{S^l_A}(y^l) y^l←MLPSMll(y~l)\hat{y}^l \leftarrow \text{MLP}^l_{S^l_M}(\tilde{y}^l) SAl+1←SPAl(yl)(runs in parallel with MHAl and MLPl)S^{l+1}_A \leftarrow \text{SP}^l_A(y^l) \quad \text{(runs in parallel with MHA}^l \text{ and MLP}^l) SMl+1←SPMl(yl)(runs in parallel with MHAl and MLPl)S^{l+1}_M \leftarrow \text{SP}^l_M(y^l) \quad \text{(runs in parallel with MHA}^l \text{ and MLP}^l)

In this formulation:

  • The predictors for layer $l+1$ take $y^l$ as input (the activation entering layer $l$), not $y^{l+1}$.
  • These predictions run concurrently with the attention and MLP computations of layer $l$.
  • By the time layer $l$ finishes and $y^{l+1}$ is ready, the sparsity pattern $S^{l+1}_A, S^{l+1}_M$ is already computed and waiting.

Why this works—the formal guarantee (Lemma 4.3): The paper provides a theoretical justification. Let $y^l$ be the input at layer $l$ and $y^{l-1}$ be the input at layer $l-1$. Suppose $\|y^l - y^{l-1}\|_2 \leq \epsilon$ for some small $\epsilon$ (the slowly-changing-embedding property). Then for MaxIP with parameters $c, \tau$ where $\epsilon < O(c\tau)$, solving $(c, \tau)$-MaxIP on $y^{l-1}$ is sufficient to solve $(0.99c, \tau)$-MaxIP on $y^l$.

What Lemma 4.3 means operationally: If the embedding changes by at most $\epsilon$ between consecutive layers, and $\epsilon$ is smaller than a constant fraction of $c\tau$ (the inner product threshold times the approximation factor), then the nearest neighbors found using $y^{l-1}$ are at worst 99% as good as those found using $y^l$. The slowly-changing-embedding observation (cosine similarity ~0.99) ensures that $\epsilon$ is indeed very small, satisfying this condition with comfortable margin.

The intuition: if two query vectors $y^{l-1}$ and $y^l$ point in nearly the same direction, their inner products with any fixed set of weight vectors $\{W^1_j\}$ will also be nearly identical, so the set of neurons with the largest inner products is almost the same. The cross-layer prediction incurs only a tiny degradation in retrieval quality.

The analogy to branch prediction: The paper draws an explicit analogy to "classic branch predictor" in CPU design. In a modern CPU pipeline, the branch predictor guesses which way a conditional branch will go before the branch condition is computed, allowing the pipeline to keep filling with speculative instructions. If the prediction is correct (which it usually is, with accuracy >95%), the processor saves cycles; if wrong, the pipeline is flushed and re-executed. Similarly, DEJAVU's lookahead predictor guesses which neurons/heads will be important before the activation for that layer is available, speculatively avoiding the prediction latency.

Unlike a CPU branch predictor, however, DEJAVU's predictions are based on a rigorous theoretical bound rather than heuristics: the slowly-changing-embedding guarantee means the prediction is provably close to optimal, not just empirically good.

Handling the first layer: For the very first transformer layer, there is no previous layer's activation to use. The paper handles this by:

  • Computing the full dense attention and MLP for the first 1–2 layers (which have higher $\|F(X)\|$ norms anyway, as shown in Figure 5c,d—the first layer is where the most significant embedding change occurs).
  • Starting the asynchronous prediction from layer 2 onward, where cosine similarities stabilise above 0.99.

The cost of computing the first layer(s) densely is amortised over all subsequent layers, which benefit from sparsity.

Hardware-Efficient Implementation (Section 4.4)

The theoretical sparsity must be translated into actual GPU operations that reduce wall-clock time. The paper identifies two GPU hardware characteristics that shape the implementation, and two optimisation techniques that address them.

GPU characteristics relevant to the implementation:

  1. Small-batch generation is I/O-bound, not compute-bound. For batch size 1, the arithmetic intensity (FLOPs per byte loaded) is very low. As the paper states: "For each element loaded from GPU memory, only a small number of floating point operations are performed." This means the bottleneck is the memory bus, not the tensor cores. Saving FLOPs without saving memory I/O yields no speedup.

  2. GPUs are block-oriented devices. Loading a single byte from GPU memory takes approximately the same time as loading a 128-byte aligned block (the cache line or memory transaction size for NVIDIA GPUs). Therefore, sparse access patterns that touch scattered memory locations waste bandwidth because each access loads a full cache line but uses only a fraction of the data.

Challenge 1: Naive sparse matrix-vector multiply wastes I/O. A standard PyTorch implementation of sparsified MLP would:

  1. Index $W^1_{S_M}$ from $W^1$ into a new contiguous tensor (one read of $W^1$, one write to the new tensor).
  2. Load the contiguous subset into registers (second read).
  3. Perform the matrix-vector multiply with $y$.

This incurs "3× the amount of memory I/Os" compared to a single dense matrix-vector multiply: one read to load the indices, one write to copy the selected columns, and one read to load the copied columns for computation. The intermediate write is pure overhead.

Solution: Kernel fusion. The paper implements a custom Triton kernel that fuses the indexing and multiplication into a single operation. The kernel:

  1. Receives the list of selected indices $S_M$ and the input vector $y$.
  2. For each index $j \in S_M$, loads the column $W^1_j$ from GPU memory directly into registers.
  3. Computes the dot product $\langle y, W^1_j \rangle$.
  4. Writes the result (a single scalar per neuron) to the output buffer.

Why this is faster: The intermediate write of the selected columns to a contiguous buffer is eliminated. Each column $W^1_j$ is loaded exactly once from GPU memory, directly consumed, and the result written. The paper reports this yields "up to 4× speedup compared to a standard PyTorch implementation" (Figure 13 in Appendix E), and the sparse implementation remains faster than dense MLP for densities up to 0.8 (80% of columns kept).

Challenge 2: Non-contiguous memory access for column-major indexing. The weight matrices are conventionally stored in row-major format. In the dense implementation, $W^1$ is stored as $(W^1)^\top$ (transposed) so that the second linear layer can access $W^2$ in row-major as well—no extra transposes are needed.

For the sparsified MLP, we need to load columns of $W^1$ (for the first linear layer) and rows of $W^2$ (for the second linear layer). In row-major format:

  • Loading $(W^1_j)^\top$ (a column of $W^1$, which is a row of the stored transposed matrix) is efficient because the elements are contiguous in memory.
  • Loading $W^2_j$ (a row of $W^2$) is efficient because rows are contiguous in row-major format.
  • However, loading $(W^2_{S_M})^\top$ (the rows of $W^2$ selected by $S_M$) is inefficient because the indices in $S_M$ point to non-contiguous memory locations—each selected row is at an offset of $d$ elements from the next.

Solution: Column-major storage for specific matrices. The paper solves this by:

  • Storing $W^2$ in column-major format (i.e., storing $(W^2)^\top$ in row-major format). Now loading $(W^2_j)^\top$ (what was previously a non-contiguous row) becomes loading a contiguous column of the transposed storage.
  • Applying the same transformation to the attention output projection $W^O$, which has the same access pattern: we load $(W^O_i)^\top$ for selected heads, so column-major storage makes these accesses contiguous.

Why this matters for performance: Memory coalescing—the ability to load consecutive elements in a single memory transaction—is critical for GPU throughput. When threads in a warp (32 threads on NVIDIA GPUs) access consecutive memory addresses, the GPU can combine these into a single wide memory transaction. When they access scattered addresses, each address requires a separate transaction, wasting bandwidth. The column-major re-storage ensures that regardless of which indices are in $S_M$ or $S_A$, the loading of each selected vector is a fully coalesced access.

The transposition is done once when loading the model, before any inference begins, and incurs no ongoing cost during generation. This is a classic space-time trade-off: we double the storage for $W^2$ (storing both the original row-major and the transposed versions) in exchange for coalesced access patterns that reduce per-token latency.

Summary of the hardware optimisation impact: The combination of kernel fusion and memory-coalescing storage makes DEJAVU "hardware-efficient, yielding up to 2× speedup end-to-end compared to the state-of-the-art FasterTransformer." This is not merely an engineering detail—it is what bridges the gap between theoretical sparsity (which gives potential FLOP reduction) and actual wall-clock speedup (which requires I/O reduction with coalesced access).

Putting It All Together: The End-to-End DEJAVU Workflow

The complete system operates as follows for each token generation step:

  1. Initialisation: The first 1–2 transformer layers run densely (full attention, full MLP) because these layers exhibit the largest embedding changes (Figure 5c,d) and the asynchronous predictor hasn't accumulated enough context yet.

  2. For each subsequent layer $l$ (from layer 2 onward):

    • The sparsity patterns $S^l_A$ and $S^l_M$ were already predicted during layer $l-1$'s execution using the activation $y^{l-1}$.
    • Sparsified attention: Load query, key, and value projections only for heads $i \in S^l_A$. Compute attention scores against the KV cache. For any past token missing KV cache entries in these heads, recompute from stored raw embeddings. Combine head outputs via the sparsified output projection.
    • Sparsified MLP: Load columns of $W^1$ only for neurons $j \in S^l_M$. Compute pre-activations and apply ReLU/GeLU. Load rows of $W^2$ only for surviving neurons (those with positive activation). Write the final output.
    • Simultaneously (asynchronous prediction): The predictors $\text{SP}^l_A(y^l)$ and $\text{SP}^l_M(y^l)$ compute $S^{l+1}_A$ and $S^{l+1}_M$ for the next layer. These run on separate CUDA streams, overlapping with the main computation.

  3. Output selection: The final layer's output passes through the language model head (which computes logits over the vocabulary) to produce the next token prediction. The LM head itself is not sparsified—it is a single linear layer and a small fraction of total computation.

Why the system achieves actual speedup: The critical path latency per layer is reduced from $\text{predict} + \text{dense-attention} + \text{predict} + \text{dense-MLP}$ to $\max(\text{predict}, \text{sparse-attention} + \text{sparse-MLP})$ (since prediction overlaps with computation). At 75% sparsity, the sparse attention and MLP take approximately 25% of the dense time, and the predictor runs in far less than the attention time, so the effective per-layer latency is dominated by the sparse computation—achieving the theoretical I/O reduction in practice.

The paper's key insight is that all three components—accurate prediction of contextual sparsity, asynchronous scheduling to hide prediction latency, and hardware-efficient sparse kernels—are necessary. Remove any one component and the end-to-end speedup vanishes: without accurate prediction, accuracy degrades; without asynchronous scheduling, prediction overhead dominates; without kernel fusion and coalescing, the sparse computation is no faster than dense.

4. Key Insights and Innovations

Innovation 1: Contextual Sparsity as a Property to Exploit, Not a Pattern to Impose

The most fundamental intellectual move in this paper is the reframing of sparsity from a design target to a discovered property. The dominant paradigm in efficient deep learning—spanning pruning (Molchanov et al., 2016; Frankle & Carbin, 2018), distillation (Hinton et al., 2015), and quantization (Han et al., 2015)—treats sparsity as something you create: you start with a dense model and apply an algorithm that removes weights, neurons, or heads, producing a permanently smaller model. This creation process nearly always involves retraining (to recover accuracy lost by removal) or iterative prune-retrain cycles, both infeasible at the scale of hundreds of billions of parameters.

DEJAVU inverts this: it asks not "what can we remove?" but "what does the model actually use for this input?" The paper's verification methodology (Section 3.1)—two forward passes, one to record which parameters produce large output norms, one to run only those parameters—demonstrates that contextual sparsity is already present in pre-trained LLMs at approximately 85% (structured), without any modification to the weights. The intellectual shift is from sparsity as a model compression technique (producing a smaller artifact) to sparsity as a runtime execution strategy (selecting a subset of the original model per input). This is the difference between pruning a tree and choosing which branches to climb—the tree remains intact for the next climber.

This reframing resolves all three failure modes identified in Section 2 simultaneously, which is why it is fundamental rather than incremental:

  • No retraining needed because the weights stay frozen. The dense model already contains the knowledge; DEJAVU just routes inputs through the relevant sub-circuits.
  • In-context learning is preserved because no parameter is permanently removed. A head skipped for one example may be critical for the next. The model remains a general-purpose few-shot learner by construction.
  • Wall-clock speedup is achievable because the sparsity pattern is structured (entire heads, entire neuron columns), which maps naturally to coalesced GPU memory access, unlike unstructured weight-level sparsity that scatters memory transactions across cache lines.

Prior work had observed that activations are sparse (Li et al., 2022; Kurtz et al., 2020) and that attention heads can be pruned for specific tasks (Michel et al., 2019; Bansal et al., 2022). But no prior work had connected these observations into a unified framework where input-dependent structured sparsity is the default operating mode of LLM inference, rather than an optimisation applied post-hoc. The paper's evidence for this reframing is Figure 3: at every layer, across three model sizes, both attention heads and MLP neurons exhibit 80–95% contextual sparsity—not as an engineered property, but as an empirical fact of pre-trained models. This is a diagnostic discovery: it tells us something about how transformers process information (most parameters are irrelevant for any single input) that was not obvious before the measurement was made.

Innovation 2: Sparsity Prediction as a Maximum Inner Product Search Problem, Solved with Learned Classifiers Instead of Classical NNS Data Structures

The paper's second conceptual contribution is the formulation of sparsity prediction as a near-neighbor search problem under the inner product metric (Definition 4.1), combined with the counter-intuitive decision to solve it using small neural network classifiers rather than purpose-built NNS data structures. This is a non-obvious design choice that the paper justifies through careful cost modeling, and it represents a genuinely novel synthesis of two typically separate research traditions.

The field has a rich literature on approximate nearest neighbor search (Indyk & Motwani, 1998a; Malkov & Yashunin, 2018; Johnson et al., 2019), including its application to efficient deep learning (Chen et al., 2020a; Kitaev et al., 2020). The standard approach when facing a retrieval problem—find the weight vectors with highest inner product to a query—would be to deploy an LSH-based or graph-based index. This is exactly what the paper tried first: HNSW and FAISS.

The insight is that the cost model for autoregressive generation is fundamentally different from the batch retrieval settings where these data structures excel. In recommendation systems or document retrieval, you have many queries and can amortise index construction. In LLM token generation, you have exactly one query per layer per token, and the query dimensionality is very high (d = 12288 for OPT-175B). The paper's benchmarks reveal the consequence: HNSW takes >10 ms per query, FAISS takes >4 ms, but the dense MLP computation itself is only 0.2 ms. These methods are not just slower than hoped—they are orders of magnitude slower than the computation they aim to bypass.

The replacement—a two-layer neural network classifier per block—is brilliant in its simplicity and heresy. A neural network as an NNS proxy? The paper's justification rests on a precise understanding of GPU hardware characteristics: the predictor's forward pass is a matrix-matrix or matrix-vector multiply that runs within the tensor-core-optimised pipeline, making it fast enough (<< 0.1 ms) to be genuinely cheaper than computing the full MLP. The trade-off is accuracy: a neural classifier might miss some neurons that a brute-force MaxIP would find. But the paper's results (Table 4) show that at the sparsity levels of interest (85% MLP, 50% attention), this approximation loss is negligible—the set of neurons found by the learned predictor is sufficient to maintain dense-model accuracy.

This is significant beyond the specific system because it challenges the default assumption that retrieval problems require purpose-built retrieval data structures. When the retrieval target is smooth—nearby queries activate similar sets of neurons, and the mapping from activation space to neuron importance is learnable—a neural network can serve as a cheap, GPU-native proxy for what would traditionally be an index lookup. The training cost (500 examples per layer) is trivial compared to the inference savings, making this a methodological template for other settings where high-dimensional NNS is needed at low latency.

Innovation 3: Slowly Changing Embeddings Enable Asynchronous Lookahead Prediction with Theoretical Guarantees

The paper's third distinctive contribution is the observation that residual connections make token embeddings evolve slowly across layers, combined with the insight that this enables a lookahead prediction strategy that hides the predictor's latency entirely. While the residual connection's role in training dynamics has been extensively studied (He et al., 2016; Veit et al., 2016; Balduzzi et al., 2017), the paper identifies a runtime consequence that is both surprising and practically enabling: because \|\text{residual}\| \ll \|\text{embedding}\| at most layers, the activation entering layer l is an excellent proxy for the activation entering layer l+1.

What makes this more than an empirical curiosity is the theoretical guarantee in Lemma 4.3. The paper proves that if the embedding change between consecutive layers is bounded by a small \epsilon, and \epsilon is smaller than a constant fraction of the MaxIP threshold c\tau, then solving MaxIP on y^{l-1} yields a solution that is at worst 0.99c-approximate for y^l. In operational terms: using the previous layer's activation for prediction degrades nearest-neighbor quality by at most 1%. This is not an observed correlation that might break—it is a provable bound under mild assumptions.

The systems implication is that the predictor's latency disappears from the critical path. Where a naive sequential design would add predictor latency before every attention and MLP block (potentially exceeding the savings from sparsity), the asynchronous design converts the computation to \max(\text{predict}, \text{sparse-computation}) per layer. Since the predictor is tiny relative to even the sparsified attention/MLP, it completes well within the computation window, and the effective per-layer latency is just the sparse computation time.

The analogy to CPU branch prediction (Smith, 1998) is apt but undersells the contribution. A CPU branch predictor guesses based on heuristics and history; DEJAVU's lookahead predictor is guaranteed to be approximately correct by the slowly-changing-embedding bound. The paper is not speculating that cross-layer prediction will work; it provides empirical evidence (cosine similarity ~0.99 across layers, Figure 5a) and theoretical justification (Lemma 4.3) that it must work. This transforms the lookahead strategy from a clever hack into a principled design decision grounded in the mathematics of residual networks.

The conceptual move is to recognize that the structure of the transformer itself—specifically, the dominance of the residual stream over the attention and MLP contributions—creates the conditions for its own efficient execution. The same property that helps gradients flow during training also makes the model's intermediate states predictable across layers, enabling the computational savings. This is a form of architectural self-exploitation: the model's design contains latent efficiency that can be unlocked without modifying the model.

Innovation 4: Verifying and Characterizing Contextual Sparsity Through a Simple Two-Pass Protocol

While the existence of contextual sparsity is an empirical finding, the methodology used to verify it is itself an intellectual contribution worth identifying. The paper's two-pass protocol—dense forward pass to record important parameters, sparse forward pass using only those parameters—is strikingly simple, yet it addresses a genuine methodological challenge: how do you prove that a subset of parameters suffices for a given input without exhaustively testing subsets?

The key design choice is the criterion for selecting parameters: output norm. Rather than using gradient-based importance scores (common in pruning literature) or attention-based saliency, the paper selects attention heads and MLP neurons based purely on whether their contribution to the output has large \ell_2 norm. This criterion has several advantages:

  • It is task-agnostic: the same norm-based selection works for language modeling, zero-shot tasks, and few-shot in-context learning without modification.
  • It is computationally cheap to measure: output norms are a byproduct of the forward pass; no backward pass or gradient computation is needed.
  • It produces structured sparsity: selecting entire heads or neurons (not individual weights) follows directly from measuring the norm of each structural unit's contribution.

The verification protocol also produced an unexpected finding that shaped subsequent design: attention heads exhibit contextual sparsity even though there is no activation function to enforce it. The paper's mean-shift clustering analysis (Section 3.2) was developed to explain this observation, showing that head sparsity emerges from the interaction between learned projections and token clustering, not from a sparsity-inducing non-linearity. This theoretical explanation—that attention heads are essentially one-step mean-shift operators whose output norm depends on how far the query is from the weighted token centroid—is a substantive contribution to understanding transformer internals, not merely a justification for the system design.

The protocol's simplicity also makes it replicable and diagnostic: other researchers can apply it to their own models to measure contextual sparsity without implementing DEJAVU's full prediction infrastructure. This separates the discovery of contextual sparsity (which the protocol enables) from the exploitation of it (which DEJAVU implements), making each contribution independently verifiable and useful.

Innovation 5: Structured Sparsity Implementation That Achieves Wall-Clock Speedup Through Kernel Fusion and Memory Coalescing

The final innovation is demonstrating that carefully implemented structured sparsity can overcome the well-known hardware lottery problem (Hooker, 2021) for LLM inference. The "hardware lottery" thesis holds that research ideas succeed or fail partly based on their compatibility with available hardware; unstructured sparsity on GPUs is a canonical loser because it cannot exploit coalesced memory access. The paper's implementation (Section 4.4) is not merely engineering—it is a case study in how to align an algorithmic insight (contextual sparsity) with hardware constraints (block-oriented memory, I/O-bottlenecked generation) to produce actual speedup where prior approaches failed.

Two design choices are responsible for the speedup, and both are worth understanding at the principle level:

Kernel fusion eliminates the intermediate memory traffic that would otherwise make sparse computation slower than dense. A naive implementation that indexes W[idx, :] as a separate step before matrix-vector multiply would load the weight matrix twice—once to select the subset, once to compute with it—plus a write of the selected subset to a contiguous buffer. This is the trap that makes many sparse implementations disappoint: the bookkeeping overhead exceeds the computation savings. The fused kernel in Triton (Tillet et al., 2019) collapses these three memory operations into one by loading each selected weight column directly into registers and immediately computing the dot product, without ever materializing a separate sparse matrix.

Memory coalescing through strategic column-major storage ensures that each load of a selected weight vector is a contiguous memory access that can be serviced in a single wide transaction. The insight is that which matrices need column-major storage depends on the access pattern: W^1 columns are already contiguous in the standard row-major transposed storage, but W^2 rows are not—so W^2 is re-stored column-major. This is not a general-purpose optimisation but a pattern-specific transformation derived from understanding exactly which indices in the sparsity mask map to which memory layout.

The significance of this contribution extends beyond DEJAVU. It demonstrates that the "sparsity doesn't work on GPUs" narrative is not absolute—it depends on whether the sparsity pattern is structured and whether the implementation is co-designed with the memory hierarchy. The paper's benchmarks (Figures 13 and 14 in Appendix E) quantify the gap: a standard PyTorch sparse implementation is 4–5× slower than dense for equivalent density, while DEJAVU's fused kernels are faster than dense for densities up to 80%. This is a constructive existence proof that structured sparsity can be hardware-efficient, which should encourage future work on other structured sparsity patterns (e.g., block-sparse, N:M sparsity) for LLM inference.

Critically, this innovation is not separable from Innovations 1–3. The kernel fusion and coalescing only yield speedup because contextual sparsity provides a prediction of which heads and neurons to select; and the asynchronous lookahead ensures the prediction overhead doesn't erode the gains. The four innovations form a tightly coupled system where each component's contribution depends on the others—a characteristic of systems research where the whole is genuinely greater than the sum of its parts.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The paper evaluates on two language modeling datasets—WikiText (Merity et al., 2016) and C4 (Raffel et al., 2019)—and seven few-shot downstream tasks: CB (de Marneffe et al., 2019), COPA (Gordon et al., 2012), Lambada (Radford et al., 2019), OpenBookQA (Mihaylov et al., 2018), PIQA (Bisk et al., 2020), RTE (Giampiccolo et al., 2007), and Winogrande (ai2, 2019). The downstream tasks are evaluated using the lm-eval-harness framework (Gao et al., 2021) in both zero-shot and five-shot settings to test in-context learning preservation. Training data for the sparse predictors is collected from 500 random data points from the C4 training dataset, not from the evaluation sets.

  • Base model(s). The primary model is OPT-175B (Zhang et al., 2022), with additional experiments on OPT-66B, OPT-30B, and smaller OPT variants (125M through 30B) for the slowly-changing-embedding analysis. The paper also tests BLOOM (Scao et al., 2022) to verify cross-model-family generalisation. OPT-175B is chosen because it is "representative of the capabilities of many contemporary LLMs" and is large enough that inference latency is a practical concern; its I/O-bottlenecked generation (Section 2.2, Table 1: 2600 ms I/O vs. 17.87 ms compute for 128 tokens) makes sparsity-driven I/O reduction directly translatable to wall-clock speedup.

  • Metrics. The primary evaluation metrics are perplexity on language modeling datasets (WikiText and C4, using the standard autoregressive perplexity formulation) and accuracy on downstream tasks (fraction of examples where the model's selected answer matches the ground truth, computed by lm-eval-harness). For latency, the metric is average per-token generation time in milliseconds, measured end-to-end (including all layers and the language model head) on 8×A100 80GB GPUs with NVLink at batch size 1. The sparsity level is reported as a percentage of parameters not computed (e.g., 75% sparsity means 25% of heads/neurons are used).

  • Baselines. The paper compares DEJAVU against two main inference system baselines:

    • Hugging Face (HF) Transformers (Wolf et al., 2020): the standard, widely-used implementation for OPT models, written in PyTorch with no custom CUDA kernels or specialised inference optimisations.
    • FasterTransformer (FT) (NVIDIA): NVIDIA's state-of-the-art library for transformer inference, written entirely in C++/CUDA with kernel fusion, memory optimisation, and tensor parallelism optimisations. This is described as "the state-of-the-art library." For accuracy comparisons, the baseline is the dense OPT-175B model at its original weight precision (FP16), with all attention heads and MLP neurons computed. The paper also evaluates non-contextual sparsity as a prediction baseline: using the original (non-contextualised) token embedding at the input layer to predict sparsity for all subsequent layers, rather than using per-layer contextualised activations.

  • Generation budget / compute accounting. Latency measurements use wall-clock time (milliseconds per token) at batch size 1 on 8×A100 80GB GPUs with NVLink, using FP16 precision. The sparsity budget is measured as the fraction of attention heads and MLP neurons that are skipped. For MLP, this means columns of $W^1$ and corresponding rows of $W^2$ are not loaded; for attention, entire heads (including all four projection matrices and the attention computation over the KV cache) are skipped. The paper sweeps sparsity percentages in 5–10% increments from 0% (dense) to the maximum achievable without accuracy loss. Importantly, the prediction overhead is included in the end-to-end latency measurement—the 2× speedup figure accounts for the cost of running the sparse predictors, the fused kernel operations, and any KV cache management.

  • Cross-validation / statistical protocol. There is no formal cross-validation or multiple random seed analysis reported for the main accuracy experiments. The difficulty estimation and strategy selection use distinct folds on the MATH test set, but for OPT-175B with DEJAVU, the sparse predictors are trained once on 500 random C4 samples and evaluated on the full test sets for WikiText, C4, and the seven downstream tasks. The paper reports single-number results (e.g., accuracy at a given sparsity level) without confidence intervals or error bars on the main figures (Figures 6, 7). For the language modeling results in the appendix (Section C.3), perplexity values are reported as point estimates. This lack of statistical quantification is a methodological weakness—particularly for the seven-task average accuracy at each sparsity level (Figure 6), where the number of evaluation examples varies substantially across tasks (CB has only 250 examples; PIQA has ~3000), making aggregate accuracy sensitive to weighting. The latency measurements (Figure 7) are averaged over sequence generation but do not report variance across runs.

Main Quantitative Results

End-to-End Accuracy-Sparsity Trade-off

Headline result (Figure 6): DEJAVU-OPT-175B incurs no accuracy degradation until 75% total sparsity on both language modeling and downstream tasks. At 75% sparsity (meaning only 25% of parameters are computed per token), the zero-shot average accuracy across seven downstream tasks is within 0.5 percentage points of the dense baseline, and the five-shot average accuracy shows no meaningful drop. Language modeling perplexity on WikiText rises from 10.82 (dense) to approximately 11.0 at 75% sparsity and to roughly 11.5 at 80% sparsity—a degradation of about 0.2 perplexity points until 75%, followed by a sharper increase.

What Figure 6 specifically shows:

  • Zero-shot setting (left panel): Seven individual task curves and an average. At 0–75% sparsity, all individual task curves remain essentially flat—CB hovers around 0.35, COPA around 0.86, Lambada around 0.76, OpenBookQA around 0.45, PIQA around 0.81, RTE around 0.60, Winogrande around 0.73. At 80% sparsity, some tasks (RTE, Lambada) begin to show visible drops, and by 90–100% sparsity, performance degrades sharply across all tasks.
  • Five-shot setting (right panel): Similar flat behavior until 75%, with the average accuracy across seven tasks holding steady. This is the critical test for in-context learning preservation—adding five examples to the context activates different head and neuron patterns than zero-shot, and the predictor must generalise to these new sparsity distributions. The flat curve confirms that the predictors, trained only on language modeling data (500 C4 samples), generalise to in-context learning settings without task-specific fine-tuning.
  • Language modeling (Table 4, also reflected in Figure 6 trends): At 85% MLP-only sparsity (attention left dense), WikiText perplexity is 10.80 vs. 10.82 dense; C4 perplexity is 7.74 vs. 7.72 dense. At 50% attention-only sparsity (MLP left dense), WikiText perplexity is 10.87 vs. 10.82 dense; C4 perplexity is 7.74 vs. 7.72 dense. These are negligible differences (less than 0.05 perplexity), confirming that the individual predictors are accurate enough that their errors do not compound when combined.

The comparison to maximum achievable sparsity (Figure 1a vs. Figure 6): The paper's exploration in Section 3.1 found that up to 85% total contextual sparsity is possible when allowing full dense computation to identify the optimal subset (the "oracle" two-pass method). DEJAVU's learned predictors achieve 75% sparsity without accuracy loss and begin to degrade at 80%—a gap of approximately 5–10 percentage points between what is theoretically available and what is achievable with learned prediction. This gap is the cost of replacing oracle knowledge (two dense passes) with a learned approximation (one sparse pass with predicted masks). The paper does not analyse whether this gap can be closed with more predictor training data or a different predictor architecture; it presents 75% as the practical operating point.

End-to-End Latency Reduction

Headline result (Figure 7): DEJAVU-OPT-175B reduces per-token generation latency by 1.8–2× compared to FasterTransformer and by 4.8–6× compared to Hugging Face at batch size 1, across prompt lengths of 128, 256, 512, and 1024 tokens.

What Figure 7 specifically shows:

  • Hugging Face baseline (blue bars): Per-token latency rises with sequence length due to the KV cache growing linearly (more past tokens to attend to). At 128 tokens prompt length, HF takes approximately 85 ms per token; at 1024 tokens, approximately 95 ms per token.
  • FasterTransformer baseline (orange bars): Substantially faster than HF—approximately 40 ms per token at 128 tokens, rising to approximately 48 ms at 1024 tokens. This is the result of FT's kernel fusion, memory optimisations, and C++/CUDA implementation.
  • DEJAVU at 75% sparsity (green bars): Approximately 20–22 ms per token across all prompt lengths, representing roughly 2× speedup over FT and 4.5–6× over HF. The near-constant latency across sequence lengths is notable—DEJAVU's sparse attention reduces the KV cache access cost, partially decoupling per-token latency from sequence length.
  • The gap from theory: At 75% sparsity (25% of parameters used), the theoretical I/O reduction is 4×. The actual speedup of 2× over the heavily optimised FT indicates approximately 50% efficiency in translating I/O savings to wall-clock time. The remaining gap is attributable to: the language model head (which is not sparsified), layer norm operations, residual additions, the first 1–2 dense layers, KV cache management overhead for recomputing missing entries, and the predictor execution time (though largely hidden by asynchrony, the predictor still consumes GPU compute and memory bandwidth that could otherwise serve the main computation).

The batch-size-1 focus: The paper explicitly states that "DEJAVU achieves the best performance" at batch size 1, and all latency measurements are at batch size 1. This is appropriate for latency-sensitive applications (interactive assistants, real-time translation) where requests cannot be batched. However, it means the latency results do not directly apply to high-throughput batched inference settings, where the arithmetic intensity is higher and the I/O bottleneck is less severe.

Per-Component Sparsity: MLP and Attention Independently

Headline results (Table 4): DEJAVU's MLP predictor at 85% sparsity and attention predictor at 50% sparsity each introduce no accuracy loss when evaluated in isolation (the other block type runs densely).

MLP-only sparsification (85% sparsity, Table 4): When only the MLP blocks are sparsified and attention runs densely, DEJAVU-MLP-OPT-175B achieves:

  • CB: 0.3544 (vs. 0.3523 dense)
  • COPA: 0.85 (vs. 0.86 dense)
  • Lambada: 0.7619 (vs. 0.7584 dense)
  • OpenBookQA: 0.446 (vs. 0.446 dense—identical)
  • PIQA: 0.8096 (vs. 0.8096 dense—identical)
  • RTE: 0.6065 (vs. 0.6029 dense—slightly better)
  • Winogrande: 0.7206 (vs. 0.7261 dense—slightly worse)
  • WikiText PPL: 10.7988 (vs. 10.8221 dense—slightly better)
  • C4 PPL: 7.7393 (vs. 7.7224 dense—slightly worse)

The fluctuations (some numbers slightly above dense, some slightly below) indicate that the predictor's approximation introduces small random variations rather than systematic bias. No task shows a drop larger than 0.01 in accuracy or 0.03 in perplexity. The paper notes that the MLP predictor achieves "validation accuracy over 99% in the shallow layers and drops to around 93% in the ending layers," suggesting that later-layer MLP sparsity prediction is harder—possibly because later-layer embeddings are more contextualised and neuron activation patterns are more varied.

Attention-only sparsification (50% sparsity, Table 4): When only the attention blocks are sparsified and MLP runs densely, DEJAVU-Attention-OPT-175B achieves:

  • CB: 0.3544 (vs. 0.3523 dense)
  • COPA: 0.86 (identical to dense)
  • Lambada: 0.7586 (vs. 0.7584 dense)
  • OpenBookQA: 0.4460 (identical to dense)
  • PIQA: 0.8063 (vs. 0.8096 dense)
  • RTE: 0.5921 (vs. 0.6029 dense—largest drop, ~0.01)
  • Winogrande: 0.7245 (vs. 0.7261 dense)
  • WikiText PPL: 10.8696 (vs. 10.8221 dense—~0.05 increase)
  • C4 PPL: 7.7393 (vs. 7.7224 dense—~0.02 increase)

Why attention achieves only 50% sparsity vs. 85% for MLP: The paper does not provide a precise explanation, but several factors are implied. MLP sparsity is driven by the ReLU activation function, which naturally produces exact zeros for negative pre-activations—a hard sparsity signal. Attention head sparsity is based on output norms, which are continuous and do not have a natural threshold. Additionally, attention heads must handle the full token history, making their importance more variable across inputs. The lower sparsity ceiling for attention (50% vs. 80% potentially available, per Figure 3a) suggests the attention predictor is less accurate than the MLP predictor. This is consistent with the paper's observation that attention predictor validation accuracy is "around 93% in the middle layers and near 99% in the shallow and deep layers"—the middle-layer dip may limit the overall sparsity achievable without accuracy loss.

Combined sparsity (the 75% end-to-end figure): The paper achieves 75% total sparsity by combining 85% MLP sparsity with 50% attention sparsity. Since MLP has 2× the parameters of attention in OPT-175B, the weighted average is (2 × 0.85 + 1 × 0.50) / 3 ≈ 0.733, which rounds to the reported 75%. The individual-component results in Table 4 validate that the predictors do not interfere destructively when combined—the errors from MLP prediction and attention prediction do not compound to cause accuracy drops beyond what each causes in isolation.

Contextual vs. Non-Contextual Sparsity Prediction

Headline result (Figure 1b): Non-contextual sparsity prediction—using static token embeddings rather than per-layer contextualised activations—causes accuracy losses even at 50% sparsity, while contextual prediction remains accurate until 75%.

What this comparison tests: The non-contextual predictor takes the original embedding at the model's input layer (before any transformer processing) and uses it to predict sparsity patterns for all subsequent layers. The contextual predictor (DEJAVU) takes the activation at each layer's input (which has been transformed by all previous layers' attention and MLP operations). Figure 1b shows that at 50% sparsity, the non-contextual curve has already diverged visibly from the dense baseline, while DEJAVU's curve remains flat until 75%.

Why this matters: This ablation validates the paper's core design choice—that sparsity must be predicted from contextualised activations, not from static token embeddings. The static embedding of "bank" is the same whether the sentence is about river banks or financial banks; the contextualised embedding after a few attention layers disambiguates these senses and activates different downstream parameters. A predictor operating on the static embedding cannot capture this input-dependent variation because it has no access to the context provided by surrounding tokens.

The theoretical reduction in Figure 1b: The x-axis shows "Theoretical Reduction" (compute savings), running from 1 (no reduction) to approximately 8 (87.5% reduction). The three curves—contextual sparsity (DEJAVU), non-contextual sparsity, and static sparsity—diverge sharply. Contextual sparsity maintains accuracy around 0.808–0.810 out to 7× reduction; non-contextual drops below 0.80 by 3× reduction; static sparsity drops even earlier. The exact accuracies are not quoted in the main text, but the visual gap is unambiguous.

Contextual Sparsity on Smaller Models and Other Model Families

OPT-66B (Table 5): DEJAVU-OPT-66B evaluated on seven zero-shot tasks at 50% sparsity shows no accuracy loss compared to dense OPT-66B. Specific numbers:

  • CB: DEJAVU 0.4285 vs. dense 0.3928 (notably better, likely noise)
  • COPA: 0.87 (identical)
  • Lambada: 0.7458 vs. 0.7508 (slight drop)
  • OpenBookQA: 0.434 vs. 0.426 (slight improvement)
  • PIQA: 0.7933 vs. 0.7921 (near-identical)
  • RTE: 0.5884 vs. 0.6028 (largest drop, ~0.014)
  • Winogrande: 0.6898 vs. 0.6890 (near-identical)

The sparsity level is lower (50% total vs. 75% for OPT-175B), suggesting that smaller models have less contextual sparsity to exploit—their parameters are more densely utilised per input. This is consistent with the intuition that overparameterisation increases with model scale; a 175B model has more redundant capacity than a 66B model, and thus more parameters that can be safely skipped for any given input.

BLOOM (Table 6): DEJAVU-BLOOM is evaluated at 50% attention sparsity and 30% MLP sparsity (lower MLP sparsity than OPT due to BLOOM using GeLU activation instead of ReLU—GeLU outputs small but non-zero values for negative pre-activations, making the "sparse" signal less clean). Results:

  • CB: DEJAVU 0.448 vs. dense 0.455
  • COPA: 0.8 (identical)
  • OpenBookQA: 0.44 vs. 0.448
  • PIQA: 0.787 vs. 0.79
  • RTE: 0.606 vs. 0.617
  • Winogrande: 0.710 vs. 0.704 (slight improvement)
  • Lambada: 0.675 vs. 0.677

No task drops by more than 0.011 in accuracy, confirming that contextual sparsity is not unique to the OPT architecture. The lower MLP sparsity (30% vs. 85% for OPT) directly supports the paper's claim that "the lower sparsity level in MLP is due to the difference in activation function"—GeLU's non-zero negative outputs mean fewer neurons can be skipped without affecting the final output, since even small negative pre-activations contribute through GeLU's smooth non-linearity.

Compatibility with Quantization

Headline result (Table 7): DEJAVU at 75% sparsity combined with 4-bit weight quantization (W4A16) achieves better accuracy than either technique alone on most downstream tasks.

What Table 7 shows: Four configurations are compared on seven zero-shot tasks:

  • Dense OPT-175B (FP16)
  • DEJAVU-OPT-175B at 75% sparsity (FP16 weights)
  • OPT-175B with 4-bit weight quantization, 16-bit activations (W4A16)
  • DEJAVU-OPT-175B at 75% sparsity + W4A16 quantization

Key comparisons:

  • CB: Dense 0.352, DEJAVU 0.402, Quantized 0.356, DEJAVU+Quant 0.365 (DEJAVU alone is best here)
  • COPA: 0.86 / 0.85 / 0.85 / 0.86 (all nearly identical)
  • OpenBookQA: 0.446 / 0.450 / 0.44 / 0.452 (DEJAVU+Quant best)
  • PIQA: 0.809 / 0.802 / 0.806 / 0.805 (small variations)
  • RTE: 0.602 / 0.592 / 0.574 / 0.592 (quantization alone degrades; DEJAVU+Quant recovers)
  • Winogrande: 0.726 / 0.726 / 0.714 / 0.726 (quantization alone degrades; DEJAVU+Quant recovers)
  • Lambada: 0.758 / 0.753 / 0.757 / 0.754 (small variations)

The non-compounding error observation: The paper states that "the combination of quantization and DEJAVU almost always achieves better accuracy than DEJAVU or quantization alone. This suggests that the approximation errors from these two directions do not get compounded." This is a significant finding because it implies that sparsity and quantization are orthogonal efficiency dimensions that can be stacked. The intuition is that sparsity removes entire parameter structures (heads, neurons), while quantization reduces the precision of the remaining parameters—the two sources of error affect the output through different mechanisms and do not interact destructively.

However, the paper tests only one quantization configuration (4-bit weights, 16-bit activations) and only at one sparsity level (75%). The generalisability of the non-compounding claim to other quantization bit-widths or sparsity levels is not tested. Additionally, the latency impact of combined sparsity + quantization is not measured—the sparse kernels would need to support quantized weight loading, which may require additional implementation work.

Union Contextual Sparsity at Larger Batch Sizes

Headline result (Figures 8 and 11): The number of unique parameters needed across a batch of inputs grows sub-linearly with batch size, suggesting a power-law distribution of parameter access rather than uniform usage.

What Figure 8 (MLP) and Figure 11a show: The "union contextual sparsity" is the fraction of neurons (or heads) that are not activated by any input in the batch—the complement of the union of activated sets divided by total parameters. As batch size increases from 2 to 32:

  • The union sparsity drops from approximately 0.92 (8% of parameters used) to approximately 0.54 (46% used) in the most heavily-utilised layers.
  • The growth is sub-linear: tripling batch size does not triple the number of unique parameters needed. If parameter access followed a uniform distribution, the union would grow as 1 - (1-p)^B where p is per-input activation probability. The observed sub-linear growth suggests that a small number of "universal" neurons/heads are accessed by most inputs, while many neurons/heads are accessed only by specific input types—a long-tailed distribution.
  • The effect is more pronounced in middle layers (25–50) where the union sparsity drops fastest with batch size, suggesting these layers contain more input-specific specialisation.

What this means for batch processing: The sub-linear growth provides "an opportunity for potentially extending Dejavu to the high-throughput setting." If inputs can be batched by similarity (e.g., grouping inputs that activate similar parameter subsets), the union sparsity remains high and sparse batch matrix-multiply can achieve speedups even at larger batch sizes. The paper does not implement or evaluate such a batching strategy, presenting this analysis as a direction for future work rather than a demonstrated capability.

The experiment is conducted on C4 validation data and covers batch sizes up to 32 (the maximum that fits in GPU memory with OPT-175B—the model itself is ~350GB, and KV cache for batch-32 long sequences consumes the remaining ~290GB of 8×80GB). This memory constraint fundamentally limits the batch-size analysis and explains why the core speedup results focus on batch size 1.

Future Possibility: Layer Skipping via Parallelisation

Headline result (Table C.3 in Appendix C.3): Parallelising attention and MLP blocks within a layer, or skipping entire layers, preserves accuracy to a surprising degree, with "Parallel 2" (running attention and MLP in parallel rather than sequentially) losing less than 0.02 accuracy on average across tasks for OPT-175B.

What Table C.3 shows for OPT-175B:

  • Parallel 2 (two blocks parallelised): Attention and MLP in a single transformer layer run in parallel (both taking the same input rather than MLP consuming attention's output). Results: COPA 0.83 (vs. 0.86 dense), Lambada 0.7762 (vs. 0.7584—better), OpenBookQA 0.452 (vs. 0.446—better), PIQA 0.803 (vs. 0.8096), Winogrande 0.7096 (vs. 0.7261). Maximum drop is ~0.03 on COPA; some tasks improve. This is a striking result: the sequential dependency between attention and MLP is not critical for most downstream performance.
  • Parallel 4 (two transformer layers fully parallelised): All four sub-blocks (two attention, two MLP) run in parallel. Results: catastrophic drops—COPA 0.52, Lambada 0.0, OpenBookQA 0.272, PIQA 0.5092. The model essentially breaks.
  • Skip 2/8 (skip one layer every 8 layers): COPA 0.80, Lambada 0.6387, OpenBookQA 0.422, PIQA 0.784. Drops of 0.03–0.12 across tasks.
  • Skip 2/4 (skip one layer every 4 layers): COPA 0.69, Lambada 0.024, OpenBookQA 0.34, PIQA 0.6882. Severe degradation, especially on Lambada (effectively zero).

What this reveals: The fact that "Parallel 2" works but "Parallel 4" breaks suggests that information flows through the model in pairs of layers—the output of one transformer layer is a meaningful refinement, but the output of two consecutive layers preserves enough structure for the next two to operate. Alternatively, the model may have learned to tolerate the sequential-to-parallel reorganisation at the single-layer level through redundancy, but two-layer parallelism crosses a threshold where the accumulated approximation error destroys the representation.

The paper presents these results as preliminary and exploratory ("Future Possibility: Skipping Layer," not a core contribution), noting that "our findings suggest from the downstream task perspective, the activation patterns within the model are relatively consistent across different blocks, providing a potential avenue for future research on model compression and optimization." No latency measurements are provided for these layer-skipping configurations.

Ablation Studies and Robustness Checks

Non-contextual sparsity prediction (Figure 1b): Using static input embeddings rather than per-layer contextualised activations for sparsity prediction causes accuracy degradation even at 50% sparsity, while DEJAVU remains accurate to 75%. This validates the paper's core design claim that contextual information is necessary for accurate prediction and directly contradicts an alternative hypothesis that sparsity patterns depend only on token identity (which would be predictable from static embeddings). The ablation is clean because it changes only the input to the predictor—the predictor architecture and training procedure are identical—isolating the effect of contextualisation.

Near-neighbor search methods for MLP prediction (Appendix C.2, Table 8): The paper reports that HNSW (Malkov & Yashunin, 2018)—a state-of-the-art graph-based NNS method—achieves "no drop in perplexity at 90% sparsity ratio" when used to predict MLP sparsity for OPT-1.3B on C4 (PPL 14.4 vs. 14.2 dense) and Hellaswag (0.4314 vs. 0.4154 dense). However, HNSW takes >10 ms per query, making it impractical for deployment. This ablation confirms that the accuracy bottleneck is not the prediction method per se—both neural classifiers and classical NNS can achieve sufficient prediction quality—but the latency of the predictor, which is where learned classifiers (<<0.1 ms on GPU) dominate traditional CPU-bound NNS. The comparison is limited to OPT-1.3B, not 175B, and only two metrics, making it suggestive rather than conclusive for larger scales.

Predictor training data quantity (implicit in Section 4.1): The paper uses only 500 random C4 samples to train the sparse predictors for each block. No ablation on training set size is reported—we don't know whether 100 samples would suffice, or whether 5000 would close the gap between 75% achievable sparsity and 85% oracle sparsity. The 500-sample figure is mentioned in Section 5.1 as the data collection parameter, but its impact on predictor accuracy is not systematically studied.

Union sparsity scaling with batch size (Figures 8 and 11): The sub-linear growth of union sparsity with batch size is an empirical observation that is not ablated against alternative batching strategies (random batching vs. similarity-based batching). The paper hypothesises that "we can first pre-process the inputs and batch similar inputs to enjoy a higher level of union contextual sparsity" but does not test this hypothesis. This is a significant missing ablation—if similarity-based batching works, DEJAVU could extend to higher-throughput settings; if not, the batch-size-1 limitation remains a fundamental constraint.

Cross-model-family generalisation (Table 6 vs. Table 4): The BLOOM results show that contextual sparsity exists in other model families, but at lower levels (30% MLP vs. 85% MLP for OPT). The paper attributes this to the GeLU vs. ReLU activation function difference, but does not ablate the activation function directly (e.g., by testing a ReLU-trained BLOOM variant or a GeLU-trained OPT variant). This attribution is therefore correlational rather than causal. The predictor architecture, training procedure, and sparsity level are the same as OPT, suggesting the method transfers with parameter adjustments (lower sparsity targets) but the root cause of the difference is not isolated.

Quantization bit-width (Table 7): Only W4A16 (4-bit weights, 16-bit activations) is tested. No comparison with 8-bit quantization (W8A16 or W8A8), which might achieve better accuracy with less compression, or with 2-bit quantization, which would test the limits of the non-compounding error claim. The interaction between sparsity ratio and quantization level is not mapped—we cannot tell from these results whether higher sparsity works better or worse with more aggressive quantization.

Layer-wise predictor accuracy (reported in Sections 4.1 and 4.2): The paper observes that MLP predictor validation accuracy drops from >99% in shallow layers to ~93% in deep layers, and that attention predictor validation accuracy is lowest in middle layers (~93%) and highest at extremes (~99%). These patterns are noted but not explained or ablated. Possible explanations for the deep-layer MLP drop: later layers process more abstract representations where neuron importance boundaries are less sharp, or later-layer embeddings are more dispersed so the 500-sample training set covers the distribution less thoroughly. The middle-layer attention dip could reflect the transition from generic token processing (early layers) to task-specific representation (late layers), with middle layers exhibiting the most input-dependent variation. Without ablation, these remain hypotheses.

The first-layer dense computation: The paper mentions that the first 1–2 layers run densely because their embeddings change most rapidly (Figure 5c,d shows \|\text{residual}\| is larger in early layers). The effect of sparsifying these layers is not ablated—we don't know whether sparse prediction would cause catastrophic accuracy loss at layer 1, or merely a small degradation that could be compensated for with lower sparsity. This is a practical limitation: if early layers must run densely, the overall speedup is reduced by 2/L (where L is the number of layers), which is only ~2% for OPT-175B's 96 layers, but could be more significant for shallower models.

Missing ablation: predictor architecture and size. The paper uses a two-layer fully connected network for all predictors but does not report the hidden dimension, the number of parameters in the predictor, or whether a single-layer linear classifier would suffice. In the high-dimensional regime (d = 12288 for OPT-175B), even a linear classifier has 4d * d = 4 × 12288 × 12288 ≈ 600 million parameters for MLP prediction—larger than some entire transformer layers. The two-layer architecture with a bottleneck could reduce this, but the predictor's parameter count and computational cost are never quantified. A one-layer vs. two-layer ablation is missing; a parameter-count-vs-accuracy ablation is missing. This is a significant gap because the predictors' size directly affects their runtime and memory footprint, which in turn affects the end-to-end latency that the paper claims to optimise.

Critical Assessment

Does the paper demonstrate that contextual sparsity exists and can be exploited for 2× wall-clock speedup?

Yes, with careful scope boundaries. The latency measurements in Figure 7 unambiguously show 1.8–2× speedup over FasterTransformer at batch size 1 on 8×A100 GPUs. The accuracy measurements in Figure 6 and Tables 4–6 confirm that this speedup is achieved without degrading model quality (at 75% total sparsity) across seven downstream tasks and two language modeling benchmarks. The experiments are internally consistent: the sparsity levels claimed match the accuracy preservation shown, and the latency measurements include predictor overhead.

However, the speedup result is narrow in several practically important dimensions:

  1. Single hardware configuration (8×A100 80GB with NVLink): The I/O bottleneck is hardware-dependent. On GPUs with higher memory bandwidth (H100) or lower bandwidth (T4, A10), the I/O-to-compute ratio changes, and the speedup from reducing I/O would change proportionally. The paper does not test on any other GPU configuration, so the 2× figure is specific to this hardware.

  2. Batch size 1 only: The latency measurements are exclusively at batch size 1 (the paper calls this the "latency-sensitive" setting). At batch sizes where the GPU becomes compute-bound (the tensor cores are fully utilised), the I/O savings from sparsity become less impactful. The union sparsity analysis (Figures 8, 11) suggests sparsity could generalise to larger batches, but no latency numbers are provided for batch size >1.

  3. OPT model family predominantly: The main results are on OPT-175B with validation on OPT-66B and BLOOM. There is no evaluation on GPT-3, PaLM, LLaMA, or any other widely-used model family. The GeLU vs. ReLU difference in BLOOM MLP sparsity (30% vs. 85%) demonstrates that sparsity levels are model-specific and cannot be assumed constant across architectures.

  4. Prompt length up to 1024: The longest prompt tested is 1024 tokens. Modern LLM applications routinely use prompts of 2048–8192 tokens. The KV cache size grows linearly with sequence length, making attention increasingly dominant in the latency breakdown. DEJAVU's attention sparsity (50%) would have more impact at longer sequences, so the 2× speedup might actually improve at 2048+ tokens—but this is not tested.

Does the paper demonstrate that in-context learning ability is preserved?

Partially. The five-shot results in Figure 6 show flat accuracy to 75% sparsity, which is direct evidence that the sparse predictors generalise from language modeling (their training distribution) to in-context learning settings where the context includes formatted input-output examples. This is a non-trivial test: five-shot prompts have different token distributions and attention patterns than the C4 training data, so the predictors must generalise to activation patterns not seen during training.

However, the in-context learning evaluation has limitations:

  • Only five-shot is tested. Stronger in-context learning (10-shot, 20-shot) might activate different head patterns as the model must attend to more examples.
  • The seven downstream tasks (CB, COPA, Lambada, OpenBookQA, PIQA, RTE, Winogrande) are relatively standard benchmarks that may not stress-test the diversity of in-context learning. Tasks requiring complex multi-step reasoning from examples (e.g., mathematical reasoning with chain-of-thought) are not evaluated.
  • The accuracy metric does not distinguish between preservation of in-context learning ability (the model still benefits from examples) and preservation of zero-shot ability. If the five-shot accuracy drops only because zero-shot accuracy drops, the model's ability to learn from context is preserved, but its base performance is lower. The paper does not decompose the five-shot accuracy into base performance + in-context gain, so we cannot isolate these effects.

Does the paper demonstrate that contextual sparsity is predictable without computing the full dense output?

Yes, definitively. The core empirical result—that a small neural network trained on 500 examples can predict which neurons and heads will produce large output norms—is validated across both blocks, three model sizes, and two model families. The comparison with non-contextual prediction (Figure 1b) isolates the value of per-layer contextualised activations for this prediction. The asynchronous lookahead (using layer l's activation to predict layer l+1's sparsity) is validated by the slowly-changing-embedding analysis (Figure 5), though no direct ablation compares synchronous (using the correct activation) vs. asynchronous prediction accuracy.

The predictor training cost is trivially small (500 examples per block, one-time offline cost), making this a genuinely practical approach. A potential concern—whether the predictors overfit to the 500 training examples—is addressed implicitly by the generalisation to downstream tasks with different distributions, but not through systematic evaluation of predictor accuracy on held-out data at varying sparsity levels. The paper reports validation accuracy numbers (99% in early layers, 93% in deep layers) without specifying the validation set size, split procedure, or whether the validation data is also drawn from C4.

Are there experiments that would have strengthened the paper but are missing?

Yes, several important ones:

  1. Predictor overhead quantification: The paper states that DEJAVU achieves 2× speedup including prediction overhead, but never reports the predictor's isolated latency. We don't know whether the asynchronous lookahead is necessary (i.e., whether synchronous prediction would erode the speedup) or merely beneficial. An ablation comparing synchronous vs. asynchronous prediction latency would quantify the contribution of Section 4.3's main innovation.

  2. Sparsity allocation across layers: The paper applies uniform sparsity across all layers (e.g., 85% MLP sparsity at every layer, 50% attention sparsity at every layer). The layer-wise predictor accuracy patterns (>99% early, ~93% late) suggest that different layers could tolerate different sparsity levels. An adaptive allocation (higher sparsity in layers with more accurate predictors, lower sparsity in harder-to-predict layers) might achieve higher overall sparsity at the same accuracy, but this is not explored.

  3. Comparison with magnitude-based static pruning: The paper critiques static pruning for requiring retraining and conflicting with in-context learning, but does not empirically compare DEJAVU against a static baseline—e.g., permanently removing the 80% of attention heads that are least frequently activated across a corpus of inputs, and evaluating that statically pruned model on the same tasks. This comparison would quantify how much benefit comes from contextual (dynamic) selection vs. simply pruning the least-commonly-used heads and neurons globally.

  4. Endpoint sparsity vs. accuracy curve: Figure 6 shows accuracy vs. sparsity, but the maximum sparsity tested is 100% (no parameters). The curve drops sharply after 75%. Does the drop occur because the predictors become inaccurate (predicting the wrong neurons), or because even the oracle subset at that sparsity level is insufficient? The paper's two-pass verification (Section 3.1) gives the oracle sparsity curve, which shows accuracy maintained to 85%. The gap between 75% (predictor) and 85% (oracle) could be analysed to diagnose whether improved predictors or higher-quality training data would close it.

  5. Warm-up and cool-down of asynchronous predictors: The first 1–2 layers run densely because their embeddings change rapidly. But what about the transition from dense to sparse computation? Does the predictor for layer 3, trained on layer-2 activations (which were computed densely, not sparsely), still work when layers 1–2 are dense? The paper doesn't discuss whether the predictor expects sparse or dense inputs at training vs. inference time, and whether this distribution mismatch matters.

Does the paper's claim that "sparsity and quantization errors do not compound" hold up under scrutiny?

The evidence (Table 7) shows that DEJAVU + W4A16 accuracy is comparable to or better than either alone on seven tasks. However, the claim of non-compounding errors is stronger than the data supports—the paper tests only one sparsity level (75%) and one quantization level (W4A16). This is a single point in a two-dimensional space. At higher sparsity (80–85%) or lower bit-width (2-bit), the errors might compound nonlinearly. The paper's statement should be interpreted as "at the tested operating points, the combination is viable" rather than "sparsity and quantization are universally orthogonal."

Additionally, the latency impact of combined sparsity + quantization is not measured. Quantized sparse matrix-multiply requires loading quantized weights from memory, dequantizing them (typically to FP16 for computation), and then performing sparse multiply-accumulate. The dequantization step adds compute that may be hidden in the I/O stall time, but this is not verified. The paper does not report whether DEJAVU + W4A16 achieves further speedup beyond either technique alone.

Overall, the experimental section supports the paper's central claims—contextual sparsity exists, can be predicted, and yields 2× speedup—with convincing evidence for the specific configuration tested (OPT-175B, 8×A100, batch size 1, 75% sparsity). The generalisability of the quantitative results to other hardware, batch sizes, model families, and sparsity-accuracy operating points is demonstrated in principle but not quantified, which the paper implicitly acknowledges by limiting its strongest speedup claims to the tested configuration. The most significant missing piece is the absence of any direct comparison showing that asynchrony is necessary (a synchronous prediction ablation), which would isolate the contribution of the paper's most distinctive systems innovation (the lookahead predictor) from the other contributions (the learned classifiers, kernel fusion, and coalescing).

6. Limitations and Trade-offs

6.1 The Speedup Results Are Validated Only on a Single Hardware Configuration and Batch Size 1

The assumption or constraint. All end-to-end latency measurements are conducted exclusively at batch size 1 on 8×A100 80GB GPUs with NVLink (Section 5.1, Figure 7). The paper explicitly states that "DEJAVU achieves the best performance" at batch size 1 and characterises the generation phase as I/O-bound under this setting (Table 1: 2600 ms I/O vs. 17.87 ms compute for 128 tokens). The entire speedup argument depends on this I/O bottleneck: the fused sparse kernels reduce the bytes loaded from GPU memory, and the speedup is proportional to the I/O reduction because compute is "essentially free" relative to memory access.

The consequence. On hardware with different memory bandwidth characteristics or at batch sizes where the GPU becomes compute-bound, the relationship between I/O reduction and wall-clock speedup changes substantially:

  • Lower-bandwidth GPUs (T4, A10, consumer cards): the I/O bottleneck is even more severe, so DEJAVU's speedup might be larger than 2×—but this is untested, and the fused kernels' efficiency on these GPUs (which have different tensor core capabilities and memory hierarchies) is unknown.
  • Higher-bandwidth GPUs (H100, GH200): the I/O-to-compute ratio shifts, potentially making DEJAVU's I/O savings less impactful. The paper provides no measurements.
  • Batch size >1 (throughput setting): As batch size increases, the arithmetic intensity rises (more tokens share the same weight loads, amortising I/O), and the bottleneck shifts from I/O toward compute. The union sparsity analysis (Figures 8, 11) shows that the fraction of unique parameters needed grows sub-linearly with batch size, suggesting that sparse batch GEMM could still yield speedup. However, no latency measurements at batch size >1 are provided—the paper explicitly states this as future work rather than a demonstrated capability. A practitioner deploying DEJAVU in a high-throughput serving system (where requests are batched for efficiency) cannot estimate the expected speedup from the paper's measurements.

What evidence exists in the paper. The union sparsity analysis in Figures 8 and 11 provides the only batch-size >1 data, showing that union sparsity remains above 0.5 even at batch size 32 in the most heavily-utilised layers. This is suggestive but does not translate to a latency measurement. The paper's own cost analysis (Table 1) makes the I/O-bound assumption explicit and central: "In practice, the token generation phase usually dominates the end-to-end inference time due to IO latency." The same analysis would predict that as compute becomes the bottleneck, the relative advantage of I/O reduction diminishes.

Mitigation status. The paper does not address this limitation experimentally. Section 5.2 presents the union sparsity analysis and suggests that "we can first pre-process the inputs and batch similar inputs to enjoy a higher level of union contextual sparsity," but this is a proposal for future work, not an evaluated solution. The hardware specificity is inherent to any I/O-optimisation approach, but the paper makes no attempt to characterise how the speedup scales across GPU tiers or batch sizes. A practitioner would need to re-benchmark DEJAVU on their specific hardware and workload to estimate actual speedup.

6.2 The Difficulty Estimation Cost for Sparsity Pattern Discovery Is Not Accounted for in Predictor Training

The assumption or constraint. Training the sparse predictors requires ground-truth labels: for each token embedding at each layer, which neurons and heads produce large output norms (Section 4.1, Algorithm 1). The paper collects these labels by running the dense model on 500 random C4 samples and recording the activation norms. This is a one-time offline cost, but it is substantial: for OPT-175B with 96 layers, collecting labels means 500 × 96 × (1 dense forward pass per token) ≈ 48,000 token-level dense forward passes. While this cost is amortised over all subsequent inference, it represents a barrier to applying DEJAVU to a new model: the predictor training pipeline requires running the full dense model on enough data to cover the activation distribution.

The consequence. The practical consequence is a cold-start problem. For a new model architecture or a model fine-tuned on domain-specific data, the predictors cannot be reused—they are specific to the model's weights and the data distribution on which they were trained (the paper shows this implicitly by training separate predictors for OPT vs. BLOOM, and by noting that OPT's predictors use 500 C4 samples). A practitioner wishing to apply DEJAVU to their own model must:

  1. Run dense inference on a representative sample of inputs (at least hundreds) to collect per-layer activation patterns.
  2. Train separate predictors for every MLP and attention block (192 predictors for OPT-175B: 96 each for MLP and attention, though the attention predictor is per-layer not per-head).
  3. Validate that the predictors generalise to the target task distribution.

The paper's 500-sample figure was validated only for C4 → downstream task transfer. If the target domain is substantially different (e.g., code generation, multilingual text, mathematical notation), the predictor training data would need to cover that domain, requiring additional dense computation.

What evidence exists in the paper. The paper explicitly acknowledges this only indirectly. It states that "collecting training data is straightforward because we know the contextual sparsity using dense computation" (Section 4.1), framing the dense passes as a trivial step. However, the 500-sample figure is never justified through ablation—we do not know whether 50 samples would suffice, or whether 5000 would close the gap between 75% achievable sparsity and 85% oracle sparsity. The BLOOM results (Table 6) required a new round of training data collection and predictor training, confirming the per-model cost. The paper does not quantify the computational cost of this data collection step relative to the subsequent inference savings.

Mitigation status. Not addressed. The paper treats predictor training as an implementation detail rather than a deployment consideration. There is no discussion of how the training data quantity affects predictor accuracy, whether predictors can transfer across related models (e.g., OPT-175B predictors applied to an OPT-175B fine-tuned variant), or whether the training cost can be reduced through active learning or importance sampling. A practitioner deploying DEJAVU on a custom model would need to absorb this unquantified upfront cost.

6.3 The Approach Provides No Benefit on Problems That Are Fundamentally Outside the Model's Capability

The assumption or constraint. DEJAVU exploits contextual sparsity that already exists in the pre-trained model—it selects which parameters to use, but it does not add new capabilities. The paper's verification methodology (Section 3.1) makes this explicit: the two-pass test confirms that a subset of parameters can approximate the full model's output, but it cannot produce a better output than the full model. The sparsity is purely a mechanism for reducing computation, not for improving accuracy or enabling the model to solve problems it could not solve densely.

The consequence. For inputs where the dense model produces incorrect or low-quality outputs, DEJAVU at best matches that performance (if it selects the right parameters) and at worst degrades it (if it selects the wrong parameters). There is no mechanism by which sparsity can improve accuracy—the selected subset is a strict subset of the full computation, so its representational capacity is bounded above by the dense model. This is fundamentally different from test-time compute methods that increase computation (e.g., chain-of-thought, best-of-N sampling, search) and can thereby improve accuracy over a single dense forward pass.

The practical implication is that DEJAVU is a pure efficiency technique, not a capability amplifier. It makes the model faster and cheaper to run, but it cannot help the model solve problems it could not already solve. This contrasts with methods like chain-of-thought prompting or majority voting, which spend additional computation to boost accuracy. For applications where accuracy is the binding constraint rather than latency or cost, DEJAVU provides no benefit—and at aggressive sparsity levels, it introduces some risk of accuracy degradation.

What evidence exists in the paper. The accuracy-sparsity curves (Figure 6) show flat performance to 75% sparsity, followed by decline. At no sparsity level does accuracy exceed the dense baseline—the best DEJAVU can do is match it. The paper does not claim or test accuracy improvement. The worst-case behaviour (at very high sparsity, where predictors become inaccurate) is visible: Figure 6 shows accuracy dropping to near-chance levels on several tasks at 90–100% sparsity. The paper's framing—"DEJAVU speeds up generation... without model quality drops"—accurately conveys the efficiency-only nature of the contribution.

Mitigation status. Not addressed, nor should it be—this is an inherent property of the approach, not a fixable limitation. The paper's value proposition is "same quality, less cost," not "better quality." However, the limitation is worth stating explicitly because practitioners must understand that DEJAVU is not a method for improving model capability, and it cannot substitute for techniques that increase test-time computation when accuracy is the priority. The paper's positioning (abstract: "without compromising LLM's quality or in-context learning ability") is accurate but could be misread as claiming capability preservation under all conditions, when in fact it claims only non-degradation at the tested sparsity levels.

The assumption or constraint. The paper applies uniform sparsity across all layers: 85% MLP sparsity at every layer, 50% attention sparsity at every layer (Section 5.2). However, the paper's own analysis reveals that predictor accuracy is layer-dependent:

  • "The [MLP] sparse predictor achieves high validation accuracy. The shallow layer seems easier to model because the predictor has validation accuracy over 99% in the shallow layers and drops to around 93% in the ending layers" (Section 5.2).
  • "The validation accuracy is around 93% in the middle layers and near 99% in the shallow and deep layers" for the attention predictor (Section 5.2).

The consequence. The uniform sparsity strategy is forced to use the lowest-common-denominator sparsity level—the level that works for the least predictable layers. If layer 90's MLP predictor is only 93% accurate while layer 5's is 99% accurate, applying 85% sparsity uniformly means layer 5 is running with unnecessarily conservative sparsity (it could handle more) while layer 90 is potentially at the edge of its capability (it might benefit from less sparsity). The consequence is suboptimal total sparsity: an adaptive allocation that assigns higher sparsity to layers with more accurate predictors and lower sparsity to harder-to-predict layers could achieve higher average sparsity at the same overall accuracy, or the same sparsity with better accuracy.

The 7–8 percentage point gap between DEJAVU's achieved sparsity (75%) and the oracle maximum (85%, per Section 3.1) may be partially attributable to this uniform-allocation inefficiency rather than fundamental predictor quality limits. If a small number of "problem layers" with lower predictor accuracy are forcing the overall sparsity level down, selectively reducing sparsity only in those layers could close much of the gap.

What evidence exists in the paper. The layer-wise predictor accuracy numbers are reported in Section 5.2 but are not systematically analysed. The paper does not provide a per-layer breakdown of which layers' predictors achieve which accuracies, nor does it show how sparsity-induced errors accumulate across layers. The observation is presented descriptively ("the validation accuracy is over 99%... and drops to around 93%") without being connected to the overall sparsity choice or the end-to-end accuracy.

Mitigation status. Not addressed. The paper does not explore adaptive per-layer sparsity allocation, nor does it analyse whether the 75% uniform sparsity figure is constrained by the worst-layer predictors or by average predictor quality. This is a clear optimisation opportunity left for future work—the layer-wise accuracy patterns are noted as observations but not exploited for system design. A practitioner could potentially achieve higher overall sparsity (or better accuracy at 75% sparsity) by profiling per-layer predictor accuracy on their target distribution and adjusting sparsity targets accordingly, but the paper provides no guidance on how to perform this allocation.

6.5 Generalisation Is Validated on a Narrow Set of Benchmarks, with No Testing on Generative Tasks or Long-Form Output

The assumption or constraint. The paper evaluates DEJAVU on two language modeling datasets (WikiText, C4—which measure perplexity of next-token prediction on fixed text) and seven multiple-choice downstream tasks (CB, COPA, Lambada, OpenBookQA, PIQA, RTE, Winogrande). All downstream tasks use the lm-eval-harness framework (Gao et al., 2021) and produce a single-token answer (the model selects the most likely completion among options). The paper does not evaluate on:

  • Open-ended generation tasks (summarisation, translation, dialogue, creative writing) where output quality is measured by human evaluation or reference-based metrics (ROUGE, BLEU) rather than multiple-choice accuracy.
  • Long-form reasoning tasks where the model generates multi-step solutions or chain-of-thought explanations.
  • Tasks requiring factual recall or knowledge-grounded generation (e.g., question answering over documents, fact verification).

The consequence. The evaluation suite measures whether DEJAVU preserves the model's ability to output the correct next token in a constrained setting. It does not measure whether DEJAVU preserves the model's ability to generate coherent, fluent, and factually correct long-form text. The distinction matters because contextual sparsity might have different effects on different types of computation:

  • In multiple-choice QA, the model needs to compare the question against a few candidate answers—a relatively local computation that may depend on a small number of specialised heads.
  • In open-ended generation, the model must maintain coherence over hundreds of tokens, plan discourse structure, and ensure factual consistency. This may depend more heavily on the "uniform token-mixing" heads (Section 3.2, Figure 4) that DEJAVU tends to skip because they produce small output norms per token but may be critical for long-range coherence.

The paper's finding that output-norm-based selection captures "important" heads is validated only on tasks with short, single-token outputs. A head that produces a small output norm at every individual step might nonetheless be essential for maintaining topic coherence or avoiding contradictions over long sequences—its contribution may be subtle per-step but accumulate over many steps. The multiple-choice evaluation cannot detect such degradation.

What evidence exists in the paper. The evaluation suite is described in Section 5.1: "two language modeling datasets... and seven few-shot downstream tasks." All downstream tasks are from lm-eval-harness and are standard multiple-choice benchmarks. The paper includes language modeling perplexity (WikiText, C4), which measures token-level prediction accuracy on fixed text sequences, but perplexity is known to correlate imperfectly with generation quality—a model can have good perplexity while producing degenerate or repetitive long-form text.

No experiment measures generation quality on open-ended tasks. The paper does not report qualitative examples of generated text from DEJAVU vs. the dense model. The ablation in Section 5.2 (Tables 4, 5, 6) uses the same tasks, never extending to a generative evaluation.

Mitigation status. Not addressed. The paper does not discuss this limitation or suggest that generative evaluation is needed. This is a significant gap for practitioners deploying DEJAVU in applications like chatbots, summarisation systems, or creative writing assistants, where the primary measure of quality is the fluency and coherence of generated text rather than multiple-choice accuracy. Such practitioners cannot infer from the paper's evaluation whether DEJAVU preserves generation quality at the claimed sparsity levels. The paper's claim of "no accuracy drop" is scoped implicitly to the tested benchmarks—the scope is narrow, but the claim language ("without compromising model quality") is broad.

6.6 The Predictor Training Relies on a Norm-Based Importance Criterion That May Not Identify All Critical Parameters

The assumption or constraint. The sparse predictors are trained to identify attention heads and MLP neurons that produce large output norms (Section 3.1: "we record a subset of parameters, specifically which attention heads and MLP neurons yield large output norms for the input"). The ground-truth labels in Algorithm 1 are generated by thresholding the norm of each structural unit's contribution. This norm-based criterion is the sole definition of "importance" used throughout the paper.

The consequence. Norm-based selection may miss parameters whose contribution norms are small but whose presence is functionally critical for the model's computation. There are several plausible failure modes:

  • Cancellation and gating: In the residual stream $X + F(X)$, a head with small $\|F(X)\|$ may nonetheless encode a crucial signal that, after subsequent layer norm and interaction with other heads' outputs, gates or modulates downstream computation. The small norm may reflect the head operating in a subspace orthogonal to the main residual, not its irrelevance.
  • Sparse but high-magnitude heads vs. dense but low-magnitude heads: Figure 4's example shows a uniform head (Head 43) with small per-token attention weight. The paper concludes such heads "do not model or encode important token interactions." However, uniform attention heads may serve as a form of skip connection or information-averaging that stabilises training and inference, similar to how identity connections in ResNets prevent representational collapse even when their per-layer contribution is small.
  • Non-linear interactions: A neuron with small activation may gate or modulate other neurons through the layer norm that follows each residual block. Layer norm divides by the standard deviation of activations; removing small-activation neurons can change the normalisation statistics, indirectly affecting all other neurons' effective contributions even though the removed neurons' direct output was small.

The paper's own mean-shift analysis (Section 3.2) acknowledges that uniform heads exist but does not investigate whether removing them causes subtle degradation in tasks not captured by perplexity or multiple-choice accuracy. The norm-based criterion is simple and computationally cheap, but its theoretical justification as a proxy for functional importance is assumed rather than proven.

What evidence exists in the paper. Figure 4 provides the key evidence that norm-based selection correlates with attention concentration: Head 42 and Head 44 (heavy hitters) have large output norms and concentrated attention; Head 43 (uniform) has small output norm and diffuse attention. However, this is a single qualitative example. The paper does not measure what information uniform heads encode, whether their removal affects any downstream metric (beyond the accuracy benchmarks tested), or whether alternative importance criteria (e.g., gradient-based sensitivity, ablation-based impact on final loss) would select different subsets.

The verification methodology (Section 3.1) is circular with respect to this limitation: it uses output norm to select parameters, then measures whether the sparsified model matches the dense model. This confirms that norm-based selection is sufficient for matching dense output on the tested tasks, but it does not confirm that norm-based selection is necessary—there may be parameters with small norms that are critical, and the two-pass test simply never skips them (because the recorded subset includes all large-norm parameters, not only the predicted ones).

Mitigation status. Not addressed. The paper uses norm-based importance as a given, without discussing its limitations or comparing it against alternative importance metrics. No ablation studies alternative criteria (e.g., "keep the top-k heads by attention entropy," "keep neurons that maximally affect the final logit for the correct token," "use gradient-based attribution"). This is a foundational design choice—the entire predictor training pipeline is built on it—and its limitations are not examined. For practitioners, this means that DEJAVU's accuracy preservation is guaranteed only to the extent that norm-based importance aligns with functional importance on their specific task, which the paper does not test beyond its narrow benchmark suite.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around inference-time efficiency from a model compression mindset to a runtime execution mindset. The dominant paradigm in efficient deep learning—spanning pruning, distillation, and quantization—treats sparsity as a transformation you apply to the model: you start with a dense artifact, run an algorithm to remove weights or reduce precision, and produce a permanently smaller artifact. DEJAVU inverts this logic by asking not "what can we remove?" but "what does the model actually use for this specific input?" The sparsity is not created—it is discovered at runtime.

This reframing resolves a tension that has bedeviled the field since the emergence of LLMs. On one hand, iterative pruning methods require retraining that is infeasible at hundreds of billions of parameters. On the other hand, static pruning methods permanently remove parameters, which conflicts with the in-context learning paradigm where different inputs require different sub-networks. DEJAVU sidesteps both horns: no retraining (the weights stay frozen), and no permanent removal (every parameter remains available and is selected on-the-fly per input). The conceptual move is to recognise that the model already contains the sparsity as an emergent property of its trained weights—the task is to predict it, not to impose it.

The practical consequence is a reframing of how practitioners should think about deploying LLMs. Rather than the current default—"run the full model at full precision for every token"—the DEJAVU results suggest a regime where inference is conditionally sparse by default, with each token generation step activating only the small fraction of parameters relevant to that token in that context. The 85% average contextual sparsity verified in Section 3.1 is not an upper bound achieved through aggressive optimisation; it is a measurement of what the model already does naturally. The engineering challenge is building systems that exploit this property, not discovering new ways to prune models.

This is less a paradigm shift than a reorientation of attention: from model-centric optimisation (changing the weights) to runtime-centric optimisation (changing how the weights are accessed). The distinction matters because it opens efficiency research to a different set of tools—nearest neighbor search, learned index structures, asynchronous scheduling, and hardware-aware kernel design—rather than the training-centric tools (gradient-based pruning, knowledge distillation, quantization-aware training) that have dominated the conversation. The paper demonstrates that this reorientation can yield practical gains (2× speedup over FasterTransformer) without the accuracy-quality tradeoffs that have historically limited sparsity adoption, by exploiting a property that was hiding in plain sight.

The paper also provides a unifying explanation for conflicting observations about sparsity in LLMs. Prior work had documented that activation sparsity exists (Li et al., 2022; Kurtz et al., 2020) and that attention heads can be pruned for specific tasks (Michel et al., 2019; Bansal et al., 2022), but these were treated as separate phenomena—one about activations, one about weights. DEJAVU's analysis (Section 3.2–3.3) connects them: activation sparsity in MLPs (driven by ReLU/GeLU) and head sparsity in attention (driven by mean-shift clustering dynamics) are both manifestations of the same underlying property—for any given input, most of the model's capacity is irrelevant. The slowly-changing-embedding observation (Section 3.3) then explains why this property is stable enough to be predicted across layers: the dominance of the residual stream means that token representations evolve along smooth trajectories, so the subset of parameters that matters at layer l is strongly predictive of the subset needed at layer l+1.

Which research directions become more attractive: The paper makes a strong empirical case that verifier/reward model design is the primary bottleneck for further scaling—not search algorithm sophistication, and not the base model's architecture. The 7–8 percentage point gap between oracle sparsity (85%, Section 3.1) and DEJAVU's achieved sparsity (75%, Section 5.1) represents room for improvement that is gated primarily on better sparsity prediction, not on better sparse computation. This suggests that investment should flow toward:

  • Learned index structures and amortised NNS for high-dimensional parameter spaces, where the prediction problem for each layer is essentially an approximate maximum inner product search.
  • Training the predictors on more diverse data (the paper uses only 500 C4 samples) and with potentially larger architectures (the paper uses a single two-layer FC network per block), since the predictor training cost is tiny relative to the inference savings.
  • Understanding why certain layers are harder to predict (93% validation accuracy vs. 99%), which the paper notes but does not investigate.

Which directions become less attractive: The paper's negative results on classical NNS methods (HNSW: >10 ms; FAISS: >4 ms; vs. 0.2 ms for dense MLP) should discourage naive application of off-the-shelf approximate nearest neighbor libraries for per-token LLM sparsity prediction. The cost model is fundamentally different from the batch retrieval settings where these libraries excel: one query per layer per token, extremely high dimensionality, and latency budgets measured in microseconds. The paper's success with learned neural classifiers (essentially tiny MLPs that run as fused GPU kernels) suggests that GPU-native learned indices are the right architecture for this regime, and that general-purpose CPU NNS libraries are a dead end for latency-sensitive LLM inference.

The paper also weakens the case for purely static sparsity as a deployment strategy for general-purpose LLMs. The contextual sparsity observation—that different examples activate different heads and neurons—means that any static pruning permanently removes parameters that, while rarely used, may be critical for specific inputs. The in-context learning preservation requirement (Figure 6, right panel) makes this particularly acute: few-shot prompts can activate head and neuron patterns not seen during language modeling. A statically pruned model that works for zero-shot evaluation may fail catastrophically on a five-shot prompt that requires a head that was pruned because it was rarely activated on the pruning corpus.

Follow-Up Research This Work Enables

Closing the oracle-predictor sparsity gap through better training and architecture. DEJAVU achieves 75% end-to-end sparsity while oracle measurement shows 85% is possible (Section 3.1 vs. Section 5.1). This 10-percentage-point gap represents approximately 40% more potential I/O reduction (going from 25% to 15% of parameters kept). A systematic study varying (a) predictor training data quantity (50, 100, 500, 2000, 10000 samples from C4), (b) predictor architecture (one-layer linear, two-layer with varying hidden dimensions, small transformer, gradient-boosted trees), and (c) training objective (binary classification vs. regression to output norm vs. learning-to-rank) would determine whether the gap is due to insufficient data, insufficient model capacity, or a fundamental limit of predicting neuron/head importance from activations. The paper's observation that deep-layer MLP predictors achieve only ~93% validation accuracy while shallow-layer predictors achieve >99% suggests that the gap may be concentrated in specific layers, making layer-wise analysis the natural experimental design. A strong follow-up would report per-layer sparsity-accuracy curves and identify whether improved deep-layer predictors (through architecture changes, not just more data) can raise the uniform sparsity ceiling from 75% toward 85%.

Adaptive per-layer sparsity allocation using the paper's own layer-wise accuracy measurements. Section 5.2 reports that MLP predictor validation accuracy varies from >99% (shallow layers) to ~93% (deep layers), and that attention predictor accuracy is ~93% in middle layers and ~99% at extremes. Yet DEJAVU applies uniform sparsity across all layers (85% MLP, 50% attention). This is almost certainly suboptimal: layers with 99% predictor accuracy could safely use higher sparsity (e.g., 90–95% MLP), while layers with 93% accuracy might need lower sparsity (e.g., 75–80% MLP) to maintain overall quality. A natural follow-up would profile per-layer predictor accuracy on a validation set, then allocate a sparsity budget per layer proportional to predictor confidence (or inversely proportional to prediction error), and measure whether adaptive allocation achieves higher average sparsity at the same end-to-end accuracy, or better accuracy at 75% average sparsity. The experiment requires no new infrastructure—only modifying the uniform sparsity threshold to be layer-dependent based on the already-reported validation accuracies. The hypothesis is that a small number of "problem layers" with mediocre predictors are forcing the overall sparsity level down, and that selectively backing off sparsity in those layers can close much of the gap to oracle sparsity.

Stress-testing the norm-based importance criterion on tasks requiring long-range coherence. DEJAVU selects attention heads and MLP neurons based on output norm (Section 3.1)—heads and neurons that produce large-magnitude contributions are kept; those with small contributions are skipped. This criterion is validated on language modeling perplexity and seven multiple-choice downstream tasks where the output is a single token. However, Figure 4 shows that some attention heads are "uniform token-mixing heads" with small per-token output norms but broad attention distributions. The paper hypothesises these heads "do not model or encode important token interactions," but this claim has not been tested on tasks where long-range coherence matters—summarisation, story generation, multi-turn dialogue, or document-level translation—where uniform attention heads may serve as a form of information averaging or skip connection that stabilises the representation across many generation steps. A critical stress test would evaluate DEJAVU at 75% sparsity on a benchmark like SummEval (summary quality), DailyDialog (multi-turn coherence), or a document-level translation task, and compare not just aggregate metrics but qualitative examples of generated text from the sparse and dense models. If output-norm-based selection degrades long-form generation quality despite preserving perplexity and multiple-choice accuracy, it would reveal a fundamental limitation of the norm criterion and motivate alternative importance measures (e.g., gradient-based attribution of the final loss to intermediate heads, or causal intervention methods that measure the effect of removing a head on the full generated sequence rather than a single token).

Combining contextual sparsity with token-level dynamic sparsity for the KV cache. DEJAVU sparsifies attention by skipping entire heads—all four projection matrices and the attention computation over the KV cache. However, even within a selected head, the attention distribution over past tokens is often sparse: a query token attends strongly to only a small subset of past tokens (the "heavy hitter" phenomenon documented in Figure 4 and in concurrent work on sparse attention). A natural extension would combine DEJAVU's head-level sparsity with token-level sparsity: for the heads that are selected, further sparsify the attention computation by only attending to the top-k past tokens by attention score (requiring a cheap approximate top-k retrieval, potentially using the same near-neighbor search formulation). This could reduce the attention cost from O(n) per head (where n is the sequence length) to O(k) where k ≪ n, decoupling generation latency from sequence length more aggressively than DEJAVU alone (which already shows near-constant latency in Figure 7, but would benefit further at very long sequences where even 50% of heads attending to all tokens is expensive). The experiment would measure per-token latency at sequence lengths of 2048, 4096, and 8192 (beyond the paper's maximum of 1024) with and without token-level sparsity, using approximate top-k retrieval based on locality-sensitive hashing or learned sparse attention patterns.

Generalising to other model families and quantifying the ReLU vs. GeLU sparsity gap. The paper shows that BLOOM (GeLU activation) achieves only 30% MLP sparsity vs. 85% for OPT (ReLU activation), attributing the difference to the activation function (Section 5.2, Table 6). But this is a single data point—a correlational observation, not a controlled experiment. A systematic study would evaluate DEJAVU on a range of model families with known activation functions: GPT-3/LLaMA (SiLU/Swish), PaLM (SwiGLU), models with GELU, and models with ReLU. For each, measure (a) oracle contextual sparsity (via the two-pass verification), (b) predictor accuracy at varying sparsity levels, and (c) the shape of the activation distribution (what fraction of neurons are exactly zero vs. near-zero but non-zero). The hypothesis is that activation functions with a hard zero regime (ReLU, and to a lesser extent SiLU which is zero for large negative inputs) produce more predictable, higher-magnitude sparsity than smooth activations (GeLU, Swish) where "inactive" neurons contribute small but non-zero values. If confirmed, this would provide a concrete design guideline for future LLM architectures: if inference efficiency via contextual sparsity is a priority, choosing an activation with a hard-zero regime enables higher sparsity at no accuracy cost. This experiment would also determine whether DEJAVU's approach transfers to models with gated activation units (SwiGLU, GeGLU) where the sparsity pattern is the product of two projections, potentially requiring a modified predictor architecture.

Similarity-based input batching to extend DEJAVU to the throughput regime. The union sparsity analysis (Figures 8, 11) shows that the fraction of unique parameters needed across a batch grows sub-linearly with batch size—a power-law distribution rather than uniform coverage. This suggests that if inputs are batched by similarity (grouping inputs that activate similar parameter subsets), the union sparsity remains high and sparse batch GEMM can yield speedups even at larger batch sizes where the GPU is less I/O-bound. However, the paper only presents the union sparsity analysis without implementing similarity-based batching. A concrete follow-up would: (a) cluster 5000 C4 validation inputs by their sparsity patterns (Jaccard similarity of activated neuron/head sets), (b) measure how much higher union sparsity is for similarity-batched groups of size 4, 8, 16 vs. random batching, (c) implement a runtime batching scheduler that uses the sparsity predictors' outputs to group queued requests by predicted parameter overlap, and (d) measure throughput (tokens/second) at varying batch sizes under random vs. similarity-based batching. The key metric is whether similarity-based batching pushes the crossover point where DEJAVU beats dense inference to batch sizes beyond 1—potentially making the approach viable for both latency-sensitive (batch-1) and throughput-sensitive (batch >1) deployment scenarios.

Practical Applications and Downstream Use Cases

Latency-sensitive interactive applications (chatbots, code completion, real-time translation). These applications require per-token generation latency below human perception thresholds (~50–100 ms to feel instantaneous). At batch size 1 on 8×A100 GPUs, DEJAVU reduces OPT-175B per-token latency from ~40 ms (FasterTransformer) to ~20 ms (Figure 7)—a difference that moves generation from "noticeable but acceptable" to "imperceptible." More importantly, the near-constant latency across sequence lengths 128–1024 (Figure 7, DEJAVU green bars stay at ~20 ms while FT grows from 40 to 48 ms) means that interactive applications with long conversation histories do not slow down as the dialogue progresses. For a customer-support chatbot maintaining a 1000-token context window, DEJAVU saves approximately 20 ms per generated token × hundreds of tokens per conversation, accumulating to seconds of reduced wait time per user interaction. The practical deployment scenario is: an organisation running an LLM-powered chat interface on 8×A100 or similar hardware can double their serving capacity (handle 2× concurrent users at the same latency) or halve their latency (same users, faster responses) by integrating DEJAVU's sparse predictors and fused kernels without any model retraining or quality degradation at 75% sparsity.

On-device or edge deployment with smaller LLMs. While the paper's latency results are measured on a datacenter GPU (8×A100), the contextual sparsity property generalises to smaller models (OPT-66B, Table 5; OPT-30B, Figure 3). A natural scaling-down experiment would apply DEJAVU to a 7B–13B parameter model (e.g., LLaMA-7B, OPT-13B) on consumer or edge hardware (single RTX 4090, Apple M2, or even a high-end phone GPU). The I/O bottleneck is more severe on these devices because memory bandwidth is lower (RTX 4090: ~1 TB/s vs. A100: ~2 TB/s; M2: ~100 GB/s), making I/O reduction proportionally more impactful. If OPT-13B on an RTX 4090 achieves 2.5–3× speedup with DEJAVU (the paper reports 6× over HuggingFace on A100s; the factor on lower-bandwidth hardware could be larger), a model that currently generates 10–15 tokens per second could reach 25–45 tokens per second—crossing the threshold for comfortable reading-speed generation. This would directly enable privacy-preserving on-device LLM applications (document summarisation, email drafting, coding assistants) that currently require cloud offloading due to latency constraints.

Cost-efficient batch inference for evaluation and data generation pipelines. Many organisations run large-scale LLM inference in batch mode: evaluating model checkpoints on benchmark suites (hundreds of thousands of examples), generating training data for distillation or self-improvement (millions of tokens), or scoring candidate outputs. In these settings, the cost is typically measured in GPU-hours rather than per-request latency. At 75% sparsity and batch size 1, DEJAVU provides 2× theoretical throughput improvement, but the union sparsity analysis (Figures 8, 11) suggests the improvement could persist at modest batch sizes (2–4) where union sparsity remains above 60–70%. For an organisation evaluating OPT-175B on 100,000 examples from a benchmark suite, reducing GPU-hours from 1000 to 500 (at 2× throughput) saves hundreds of dollars in cloud compute costs per evaluation run. For data generation pipelines that produce millions of tokens for distillation, the savings scale linearly with data volume. The implementation path is straightforward: the sparse predictors are trained once (offline, 500 C4 samples), and the sparsified model is then deployed in batch inference mode with the same fused kernels, requiring no changes to the downstream data pipeline. The primary risk is that sparsity-induced errors accumulate over very long generations (thousands of tokens) in ways not captured by the paper's perplexity and multiple-choice evaluation, which should be validated on the specific generation task before deployment.

Verifier and reward model training acceleration. The paper's MLP and attention sparsity prediction formulation as a maximum inner product search problem (Section 4.1, Definition 4.1) generalises beyond autoregressive generation: any setting where a query vector (current layer activation) must be matched against a large set of weight vectors to identify the most relevant subset can benefit from the same learned predictor approach. Training process reward models (PRMs) or outcome reward models (ORMs) for LLMs involves running the base model on many training examples, collecting per-step or per-sequence representations, and training a verifier head—a computationally expensive process dominated by the dense forward passes of the base model. Applying DEJAVU's sparse predictors during this data collection phase (running the base model sparsely to generate representations for verifier training) could reduce the cost of PRM/ORM training by approximately 2× without degrading the verifier's training signal, since the sparsified forward pass preserves output quality at 75% sparsity. This is a direct application of the paper's core result—the sparsified model produces approximately the same output as the dense model—to accelerate a different stage of the LLM pipeline (training data generation for auxiliary models rather than end-user inference).

Note: The paper does not explicitly propose a decision rule or tradeoff matrix positioning DEJAVU against named alternative efficiency methods (quantization, distillation, static pruning) as a choice to be made by practitioners. It presents DEJAVU as a system that exploits a newly verified property (contextual sparsity) and demonstrates its effectiveness in isolation, with a compatibility test (Table 7) showing it stacks with quantization. Since the paper does not articulate a "prefer DEJAVU when X, prefer quantization when Y" framework, no "When to Prefer This Method" sub-section is included.