IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse

ArXiv: 2603.12201

🎯 Pitch

DeepSeek's Sparse Attention (DSA) reduces core attention cost but leaves a costly indexer that must scan every token at every layer—and this paper shows you can simply reuse the same top-k indices across 75% of layers with no quality loss. The surprising result is that even a naive uniform interleaving pattern works, but only if you explicitly train the surviving indexers to serve multiple layers via a simple distillation loss.

---

1. Executive Summary

This paper introduces IndexCache, a method that accelerates DeepSeek Sparse Attention (DSA) by exploiting the cross-layer redundancy of the lightning indexer's top-k token selections, partitioning layers into a small set of Full layers that compute fresh indices and a majority of Shared layers that reuse cached indices from the nearest preceding Full layer (adding only one conditional branch at inference). Evaluated on a 30B DSA model across nine long-context and reasoning benchmarks, IndexCache removes 75% of indexer computations with negligible quality degradation, yielding up to 1.82× prefill speedup and 1.48× decode speedup at 200K context length, with preliminary results on the 744B GLM-5 confirming scalability at over 1.3× end-to-end speedup. The paper further provides two complementary configuration methods—a training-free greedy search that selects which layers retain indexers by minimizing language modeling loss on a calibration set, and a training-aware multi-layer distillation loss that trains each retained indexer against the averaged attention distributions of all layers it serves—establishing that even simple uniform interleaving patterns can match full-indexer accuracy only when the model is explicitly trained for cross-layer sharing.

2. Context and Motivation

The Core Problem: The Indexer Becomes the Bottleneck in Sparse Attention

This paper addresses a specific, measurable inefficiency in DeepSeek Sparse Attention (DSA): the lightning indexer module, which was designed to reduce attention cost, has itself become a computational bottleneck as context lengths grow. To understand why this matters, we need to trace the logic chain that leads from standard attention to DSA to IndexCache.

Standard transformer self-attention has quadratic complexity $O(L^2)$ in sequence length $L$. For a 200K-token context — common in long chain-of-thought reasoning, multi-step agentic workflows, and retrieval-augmented generation over web-scale sources — computing full attention at every layer is prohibitively expensive. DSA addresses this by splitting each attention layer into two stages: (1) a lightning indexer that scores all preceding tokens and selects the top-$k$ most relevant ones (where $k = 2048 \ll L$), and (2) a sparse core attention that computes the actual attention only over this small subset. This reduces the core attention cost from $O(L^2)$ to $O(Lk)$ per layer.

However — and this is the gap the paper identifies — the indexer itself still operates at $O(L^2)$. Every indexer at every layer must independently score all preceding tokens to determine its own top-$k$ set. Across $N$ layers, the total indexer cost is $O(NL^2)$. The paper provides a concrete profiling analysis that makes this bottleneck starkly visible:

"profiling a 30B DSA model reveals that the indexer's share of total latency rises sharply with context length, particularly during the prefill stage"

The numbers are striking: at 10K tokens, the indexer accounts for 27% of total attention time during prefill and 27% during decode. At 200K tokens, this jumps to 81% during prefill and 41% during decode. The indexer is designed to be lightweight — it uses fewer heads, low-rank projections, and FP8 arithmetic, making it an order of magnitude cheaper per-FLOP than the main Multi-head Latent Attention (MLA) computation — but because its cost scales quadratically while the core attention cost scales linearly in $k$, the indexer's relative cost grows with context length until it dominates the total inference budget.

This is a subtle but important diagnosis: DSA successfully made core attention cheap, but in doing so it merely shifted the bottleneck from core attention to the indexer. The problem is not that DSA is inefficient — it is that DSA's efficiency gains create a new frontier where further improvement requires attacking the indexer itself.

Why This Matters: The Real-World Stakes

The paper situates its work within a clear deployment trend: LLMs are increasingly used in settings that demand extended contexts. The authors cite long chain-of-thought reasoning, multi-step agentic workflows, and retrieval-augmented generation over web-scale sources as defining use cases. Each of these pushes context lengths to 100K tokens and beyond, at which point the indexer bottleneck becomes acute.

The practical impact is measured in two dimensions:

Prefill latency (time-to-first-token). During prefill, the model processes the entire input context in parallel. At 200K tokens, the 30B DSA model requires 19.5 seconds just for this prefill phase. The indexer consumes 81% of that time. For interactive applications — chatbots, coding assistants, real-time agents — this delay directly degrades user experience. Reducing indexer cost translates almost 1:1 into faster time-to-first-token.

Decode throughput (tokens per second). During autoregressive generation, each new token requires the indexer to re-score the entire accumulated context. At 200K tokens under single concurrency, the 30B model produces only 58 tokens per second. This limits the responsiveness of streaming applications and the throughput of batch inference pipelines. The paper reports that when the KV cache is fully saturated (~800K tokens per GPU), total decode throughput drops to just 197 tokens per second — a severe constraint for high-volume serving.

The paper also makes a broader industry observation: sparse attention is becoming the default for frontier LLMs, citing DeepSeek-V3.2 and GLM-5 as production models that already use DSA. This means the indexer bottleneck is not a niche academic concern — it affects the inference economics of some of the largest deployed language models. Any method that reduces indexer cost without degrading quality has immediate practical value for production serving infrastructure.

Prior Approaches and Where They Fall Short

To understand IndexCache's contribution, we need to examine the two lines of prior work it builds on — and why neither directly solves the DSA indexer bottleneck.

Cross-Layer Token Selection Stability

A well-established empirical finding in full-attention transformers is that the set of important tokens is remarkably stable across consecutive layers. Deshmukh et al. (2025) and Gao et al. (2026) both observe that adjacent layers share the vast majority of their top-$k$ attention mass. This observation has been exploited by several methods:

• TidalDecode (Yang et al., 2025a), LessIsMore (Yang et al., 2025b), OmniKV (Hao et al., 2025), and DELTA (Zarch et al., 2025) reuse top-$k$ indices from periodic anchor layers for sparse decoding.
• Kascade (Deshmukh et al., 2025) formalizes anchor layer selection via dynamic programming over a cross-layer similarity matrix, identifying head-aware remapping as critical for maintaining accuracy.
• HySparse (Gao et al., 2026) unifies both index reuse and KV cache sharing, interleaving full attention layers with sparse layers that inherit both top-$k$ block indices and KV caches.

However, all of these methods share a critical dependency: they require full attention layers as the oracle. The anchor layers compute complete $O(L^2)$ attention to identify the truly important tokens, and the intermediate sparse layers simply reuse those indices. This works because full attention provides a "ground truth" signal about which tokens matter — the anchor layer actually computes the full attention distribution, so its top-$k$ selection is exact.

In DSA, this assumption breaks down completely. DSA has eliminated full attention entirely — there is no layer that computes $O(L^2)$ attention. Every layer uses the lightweight indexer for token selection, and every layer computes only sparse $O(Lk)$ core attention. There is no "oracle" full-attention layer to serve as an anchor. The paper states this gap explicitly:

"Crucially, both approaches depend on full attention as the oracle for identifying important tokens. In DSA, full attention has been eliminated entirely—replaced by the lightweight indexer. This raises a question that, to our knowledge, has not been addressed: does the indexer's output also exhibit cross-layer stability? If so, we can apply the same sharing principle to eliminate redundant indexer computations without requiring any full attention oracle at all."

This is IndexCache's foundational insight: the indexer's output is itself redundant across layers, and the sharing principle that prior work applied to full attention can be ported to sparse attention — but with the crucial advantage that what we are sharing (the indexer's output) is cheaper to compute than full attention, so the retained oracle layers are themselves lightweight.

Efficient Attention Methods

The paper situates DSA within the broader landscape of efficient attention methods, distinguishing between training-free and trainable approaches:

Training-free sparse methods introduce sparsity at inference through fixed patterns (sliding windows, sink tokens), heuristic eviction strategies (H2O, streaming attention sinks), or lightweight importance estimation (Quest, SparQ). These methods avoid retraining but suffer from a fundamental limitation: "the resulting training-inference mismatch can cause error accumulation in long-context settings" (citing Hu et al., 2026). The sparsity pattern at inference differs from what the model saw during training, leading to distributional shift that compounds across layers.

Trainable sparse methods incorporate sparsity directly into training, avoiding the training-inference mismatch. This includes learned gating mechanisms (SeerAttention), end-to-end sparse pre-training (NSA), block-level mixture routing (MoBA, InfLLM-v2), and full-to-sparse distillation (DSA, SSA). DSA specifically distills a lightweight indexer from full attention during a continued pre-training phase, then jointly optimizes the model with sparse attention. This approach preserves model quality better than post-hoc sparsity but introduces a new cost: the indexer itself.

The paper's key observation is that DSA's design creates an asymmetry: core attention was successfully made sparse and cheap, but the mechanism that enables that sparsity — the indexer — remains dense and quadratic. This is not a failure of DSA but rather an opportunity: the indexer's quadratic cost was accepted as necessary overhead, but if that overhead can be reduced through cross-layer sharing, the total cost of sparse attention drops substantially without requiring any architectural change to the core attention mechanism.

Cross-Layer KV Cache Sharing

A separate line of work reduces memory by letting multiple layers share key-value caches across depth: MiniCache (Liu et al., 2024b), SwiftKV (Qiao et al., 2025), and others. IndexCache is orthogonal to this direction — it reduces computation (indexer FLOPs) rather than memory (KV cache size) — but the conceptual parallel is informative. Just as KV cache sharing exploits the observation that key-value representations are similar across layers, IndexCache exploits the observation that token selection indices are similar across layers.

How IndexCache Positions Itself

The paper positions IndexCache as filling a specific, previously unaddressed gap: cross-layer index reuse for sparse attention where no full attention oracle exists. The contribution is not the general principle of cross-layer sharing (which was established by prior work) but rather:

1. Empirical verification that the indexer's top-$k$ selections exhibit cross-layer stability in DSA, with adjacent layers sharing 70–100% of their selected tokens (Appendix A, Figure 4). This extends the known cross-layer stability phenomenon from full attention to indexer outputs.

2. A method for exploiting this redundancy without requiring any full attention computation. Unlike prior anchor-layer approaches that need expensive full-attention oracles, IndexCache's retained Full layers are themselves just DSA layers running their own lightweight indexers. The oracle is cheaper because the indexer is cheaper than full attention.

3. Two systematic techniques for optimizing the sharing configuration, going beyond the hand-tuned or simple uniform patterns used in prior work. The training-free greedy search (Section 3.1) provides a principled way to select which layers retain indexers for any off-the-shelf DSA model. The training-aware multi-layer distillation (Section 3.2) goes further, showing that if you can retrain, you can eliminate the layer-specific sensitivity that makes pattern design difficult in the training-free setting.

4. A demonstration that extreme sharing ratios are achievable: removing 75% of indexers with negligible quality loss. Prior cross-layer sharing methods for full attention typically work at lower ratios; IndexCache shows that the indexer's output is more redundant than full attention's output, enabling more aggressive reuse.

The paper explicitly connects IndexCache to the broader trajectory of LLM inference optimization:

"As sparse attention becomes the default for frontier LLMs (DeepSeek-V3.2, GLM-5), we expect cross-layer index reuse to become a standard component of efficient inference pipelines."

This framing positions IndexCache not as a one-off technique for DSA but as a general principle applicable to any sparse attention method that uses dynamic (non-fixed-pattern) token selection — including block-level selection methods like MoBA and NSA. The core insight is that whatever mechanism selects which tokens to attend to — whether it is DSA's per-token indexer or MoBA's block-level router — its output is likely redundant across layers and can be shared.

The Two-Pronged Approach: Why Training-Free and Training-Aware?

A key design choice the paper makes — providing both training-free and training-aware variants — reflects a practical understanding of real-world deployment constraints:

Training-free IndexCache targets the common scenario where practitioners have a pre-trained DSA model and want to accelerate inference without modifying weights. This is important because retraining a 30B or 744B model is expensive, and in many production settings the model is a fixed asset. The challenge is that without retraining, each layer's core attention has been optimized to work with its own indexer's top-$k$ selection. Reusing a different layer's indices introduces distributional shift — the S layer receives a slightly different set of tokens than it was trained to expect. The greedy search addresses this by finding which layers are least sensitive to receiving cached indices, effectively routing around the layers where the distributional shift would cause cascading errors.

Training-aware IndexCache targets the scenario where training from scratch or continued pre-training is feasible. By explicitly training the retained indexers to serve multiple layers simultaneously (via multi-layer distillation), and training the S layers to expect inherited indices, the distributional shift problem is eliminated entirely. This is what enables uniform interleaving — a fixed pattern like FSSS FSSS... — to match full-indexer quality even at 1/4 retention, where in the training-free setting uniform patterns cause significant degradation (Table 3 vs. Table 2).

The two approaches are complementary rather than competing: training-free IndexCache provides immediate acceleration for existing models, while training-aware IndexCache provides a path to even more robust sharing for future models trained with this objective from the start.

3. Technical Approach

3.1 Reader Orientation

IndexCache is a lightweight modification to the inference loop of any DeepSeek Sparse Attention (DSA) model that eliminates redundant indexer computations by partitioning transformer layers into a small set of Full layers that compute fresh top-k indices via their own indexers and a majority of Shared layers that skip their indexers entirely and reuse the most recent Full layer's cached indices, adding exactly one conditional branch per layer. The problem it solves is the $O(NL^2)$ quadratic indexer cost that dominates inference at long context lengths (81% of prefill time at 200K tokens); the solution's shape is a simple binary pattern over layers that exploits the empirical finding that adjacent layers' top-k token selections overlap by 70–100%, meaning most indexer computations are redundant.

3.2 Big-Picture Architecture (Diagram in Words)

IndexCache has four major components, arranged as a modification to the existing DSA inference loop rather than a new architecture:

1. Layer Role Partitioning — A binary pattern string $c = c_1 c_2 \cdots c_N$ (where $c_\ell \in \{F, S\}$) that assigns each of the $N$ transformer layers to be either a Full layer (retains its indexer) or a Shared layer (skips its indexer). The first layer is always $F$ to seed the initial indices.

2. Index Cache — A temporary buffer (T_cache) that stores the top-k index tensor produced by the most recent Full layer. At each Full layer, this buffer is overwritten with fresh indices; at each Shared layer, it is read to provide the indices for sparse core attention. No additional GPU memory beyond what standard DSA allocates is required.

3. Configuration Method — One of two approaches for determining the pattern $c$: (a) a training-free greedy search that uses language modeling loss on a calibration set to select which layers should retain indexers, for use with any off-the-shelf DSA model; or (b) a training-aware multi-layer distillation loss that retrains the retained indexers to produce indices jointly useful for all layers they serve, enabling even simple uniform patterns to match full-indexer accuracy.

4. DSA Inference Backbone — The existing sparse attention mechanism: lightning indexers (at Full layers only), top-k selection, sparse core attention with $k=2048$, and feed-forward networks. IndexCache changes nothing about the core attention computation or the model's parameters (in the training-free case).

Information flows as follows: input embeddings enter layer 1 (always $F$) → the indexer scores all preceding tokens and selects the top-$k$ indices → these indices are stored in T_cache and used for sparse core attention → the output passes to the next layer → if the next layer is $F$, it runs its own indexer from scratch and overwrites the cache; if it is $S$, it reads the cached indices and proceeds directly to sparse core attention → this repeats through all $N$ layers → the final layer's output is projected to vocabulary logits as in standard decoding.

3.3 Roadmap for the Deep Dive

The explanation must build from the concrete motivation (why the indexer is both essential and expensive) through the empirical observation that makes sharing possible, to the two complementary methods for determining which layers share:

• First, we revisit DSA's indexer mechanism in detail — what it computes, why it costs $O(L^2)$, and where that cost appears in the latency profile — because the entire method rests on understanding what computation IndexCache eliminates.

• Second, we establish the empirical foundation for sharing by examining the cross-layer top-k index overlap analysis (Appendix A, Figure 4), which quantifies how much redundancy exists and reveals the block structure that motivates non-uniform sharing patterns.

• Third, we walk through the core inference modification — the single conditional branch that distinguishes Full from Shared layers — and explain why the design choice to use a simple cached tensor rather than a learned aggregation is both sufficient and efficient.

• Fourth, we detail the training-free greedy search algorithm (Algorithm 1) including the calibration set construction, the loss-based evaluation metric, the greedy selection procedure, and the pipeline-parallelism acceleration strategy.

• Fifth, we explain the training-aware multi-layer distillation loss (Equation 1), prove its equivalence to distillation against an averaged target (Proposition 1), and discuss the two-stage training pipeline that optimizes indexers for cross-layer sharing.

• Sixth, we cover the practical considerations: pattern search complexity, the similarity-based approach that failed (Appendix C, as a cautionary tale), and the greedy pattern properties that make it a valid proxy for downstream quality.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems-and-optimization paper whose core idea is that the lightning indexer's top-k token selections in DSA are highly redundant across consecutive layers, and that this redundancy can be exploited by having most layers skip their indexers and reuse cached indices from a small set of retained Full layers — with the specific pattern of which layers retain indexers determined either by greedy loss-based search on a frozen model or by multi-layer distillation during training.

---

The DeepSeek Sparse Attention Indexer: What Gets Eliminated

To understand what IndexCache removes, we must first understand precisely what the DSA indexer computes and why its cost grows quadratically with sequence length.

DSA decomposes each attention layer into two sequential stages. In the selection stage, a lightning indexer module takes the current query representation (the hidden state at position $t$ that will be used for attention) and scores it against all preceding token positions $1, 2, \ldots, t-1$. These scores are produced by a multi-head ReLU-gated dot product: the query is projected through a small number of heads (fewer than the main attention), each head computes dot-product similarity with key projections of all preceding tokens, and a ReLU gating mechanism filters the scores. The result is a score vector $I^{(\ell)}_t \in \mathbb{R}^L$ (where $L$ is the total sequence length, and positions beyond $t$ are masked). This vector is then passed through a top-k operation to select the $k = 2048$ highest-scoring positions: $T^{(\ell)}_t = \text{Top-}k(I^{(\ell)}_t)$.

In the computation stage, sparse core attention is computed only over these $k$ selected positions using the full Multi-head Latent Attention (MLA) mechanism. This reduces the core attention cost from $O(L^2)$ (standard full attention) to $O(Lk)$ per layer. Since $k = 2048 \ll L$ at long contexts, this is a dramatic reduction.

The indexer achieves its efficiency through several design choices: it uses fewer attention heads than the core MLA computation, applies low-rank projections to reduce the dimensionality of query and key representations, and operates in FP8 precision (8-bit floating point) rather than the higher precision typically used in core attention. The paper states it is "an order of magnitude cheaper per-FLOP than the main Multi-head Latent Attention." However, this per-FLOP efficiency does not change the asymptotic complexity: the indexer still computes dot products between the query and every preceding token position, meaning its cost scales as $O(L)$ for a single query position and $O(L^2)$ when processing all positions in parallel during prefill.

For a single layer $\ell$, the indexer cost during prefill (processing a sequence of length $L$) is proportional to $L^2$ because it scores all pairs $(t, s)$ where $s < t$. During autoregressive decode, generating one new token requires the indexer at each layer to score the new query against all $L$ preceding tokens, incurring $O(L)$ cost per token per layer. Across $N$ layers, the total indexer cost becomes:

• Prefill: $O(NL^2)$ — every layer independently scores all pairs.
• Decode per token: $O(NL)$ — every layer scores the new query against the full context.

The paper's profiling analysis (shown in the bar charts of the introduction) quantifies this concretely for a 30B DSA model with $N = 47$ layers: at 10K context length, the indexer accounts for 27% of total attention time during prefill and 27% during decode. At 200K context length, these numbers rise to 81% during prefill and 41% during decode. The core attention cost grows only modestly because it scales linearly in $k$, while the indexer cost scales quadratically in $L$. This is the bottleneck that IndexCache targets.

The key structural property that enables IndexCache is that this indexer computation is performed independently at every layer. Each layer's indexer has its own learned parameters (projection matrices for queries and keys, ReLU gating weights) and produces its own score vector based on that layer's specific hidden state representations. If we can show that these per-layer score vectors — and the resulting top-k index sets — are highly similar across layers, then most of these independent computations are redundant.

---

The Empirical Foundation: Cross-Layer Top-k Index Overlap

IndexCache's feasibility rests on a single empirical claim: the set of top-$k$ tokens selected by the indexer is highly similar across consecutive layers. The paper validates this claim through a systematic pairwise overlap analysis described in Appendix A.

The procedure is straightforward. For every pair of layers $(i, j)$ in the 47-layer 30B DSA model, the authors compute the overlap ratio between their top-k index sets:

$\text{Overlap}(i, j) = \frac{|T^{(i)} \cap T^{(j)}|}{k}$

where $T^{(i)}$ is the set of $k = 2048$ indices selected by layer $i$'s indexer, $T^{(j)}$ is the set selected by layer $j$'s indexer, and the intersection counts positions selected by both layers. This ratio is averaged over 768 samples from a calibration set, each with a context length of 200K tokens.

The results are visualized as a $47 \times 47$ heatmap in Figure 4. Several patterns emerge that directly inform IndexCache's design:

Adjacent layers share 70–100% of their selected tokens. The diagonal band of the heatmap is bright (high overlap), confirming that consecutive layers overwhelmingly attend to the same token positions. This is the fundamental justification for index reuse: if layer $\ell$ and layer $\ell+1$ agree on 90% of their top-k tokens, then layer $\ell+1$ computing its own indexer from scratch provides minimal additional information over simply reusing layer $\ell$'s indices.

The overlap exhibits block structure. The heatmap reveals distinct clusters of layers with mutually high overlap — for instance, layers 17–30 form a visible block, as do layers 31–36. Within these blocks, even non-adjacent layers show substantial overlap. This suggests the model organizes into functional groups where token selection preferences are internally consistent, and that sharing within a block should be particularly effective.

Overlap decays unevenly across block boundaries. The transition between blocks is sharper than the gradual decay within a block. This means that a few "transition" layers shift the attention focus substantially — the token subsets that matter change qualitatively at these boundaries — while within a block the changes are gradual. This has direct implications for pattern design: F layers should be placed near block boundaries to provide fresh indices when the attention focus shifts.

Early and late layers attend to fundamentally different token subsets. The bottom-left and top-right corners of the heatmap (comparing early layers to late layers) show overlap ratios below 0.4, confirming that the model's attention focus drifts substantially from input-processing layers to output-preparation layers. This means a single F layer at the beginning cannot serve the entire model — F layers must be distributed throughout the network to re-anchor the attention focus as it shifts.

The paper notes an important mismatch between the natural overlap clusters and the greedy-searched sharing blocks (marked as red boxes in Figure 4). The two partitions do not fully coincide because overlap is an aggregate metric — it counts how many tokens are shared but not which ones differ. In the training-free setting where weights are frozen, even a small set of mismatched critical tokens can perturb a layer's hidden state in ways that cascade through all downstream layers. Early layers are especially vulnerable because their perturbations traverse the longest propagation path. This explains why a simple similarity-based pattern selection (Appendix C) fails as a proxy for downstream quality, and why an end-to-end loss-based search is necessary.

---

The Core Inference Modification: Full and Shared Layers

IndexCache modifies the DSA inference loop by introducing exactly one conditional branch at each layer. The modification is presented as pseudocode in Figure 2(b) and is conceptually minimal: the only change to standard DSA is that before the indexer runs, the layer checks its role in the pre-determined pattern $c$.

The modified inference loop proceeds layer by layer. For each layer $\ell = 1, 2, \ldots, N$:

If $c_\ell = F$ (Full layer): The layer's lightning indexer is executed exactly as in standard DSA. It takes the current hidden state $X$ (the input to this layer), computes score vectors for all query positions against all preceding token positions, applies the top-k operation to select $k = 2048$ indices per query position, and produces an index tensor $T^{(\ell)}$. This tensor is immediately used for sparse core attention at this layer, and it is also written to a temporary buffer T_cache, overwriting whatever was stored there previously. The sparse core attention and feed-forward network then execute as normal.

If $c_\ell = S$ (Shared layer): The layer's indexer is not executed. Instead, the layer reads the cached index tensor from T_cache — which was written by the most recent Full layer — and uses these inherited indices directly for sparse core attention. The layer's own indexer parameters are never invoked; the layer simply trusts that the nearest preceding Full layer's indices are sufficiently similar to what its own indexer would have produced.

The first layer is always $F$ ($c_1 = F$) to ensure that T_cache is initialized before any Shared layer attempts to read it. This is a hard constraint because there is no preceding Full layer to inherit from otherwise.

The T_cache buffer is a temporary tensor that holds only the current index set. It is overwritten at each Full layer and requires no additional GPU memory beyond what standard DSA already allocates for index tensors. During standard DSA inference, every layer already allocates memory for its own index tensor; IndexCache simply repurposes a subset of these allocations as the cache buffer. There is no persistent storage of multiple index sets, no learned aggregation of indices from multiple layers, and no increase in memory footprint.

Why this simple design works. The key insight is that the indexer's output — a set of $k$ positions — is a discrete selection rather than a continuous representation. When a Full layer runs its indexer, it produces exactly which positions $1, 2, \ldots, L$ will be attended to. If a Shared layer's indexer would have selected a substantially similar set (which the overlap analysis shows is typically true), then the Shared layer can use the cached set and produce nearly identical attention outputs. There is no need for interpolation, weighted averaging, or learned combination of multiple index sets, because the goal is not to approximate a "better" set than any single indexer would produce — it is simply to avoid recomputing what would overwhelmingly likely be the same set.

The alternative — having Shared layers produce some weighted combination of indices from multiple preceding Full layers — would introduce complexity (which layers to combine, with what weights) without addressing the fundamental observation that a single preceding Full layer's indices are already highly similar to what the Shared layer would compute. The simplicity of the single-nearest-Full-layer design is a deliberate choice justified by the empirical overlap data.

A subtle but important detail: the Shared layer does not simply copy the attention weights from the Full layer — it copies the index set (the positions to attend to) and recomputes its own attention weights over those positions using its own query, key, and value projections. This means the Shared layer's attention computation is still personalized to its own hidden state; it is only the selection of which tokens to consider that is inherited. This preserves much of the layer-specific modeling capacity while eliminating the expensive selection step.

---

Training-Free IndexCache: Greedy Loss-Based Pattern Search

The training-free variant targets the common deployment scenario: a practitioner has a pre-trained DSA model and wants to accelerate inference without modifying any weights. The challenge is to determine which layers can safely skip their indexers — that is, which layers' attention outputs are least perturbed by receiving cached indices from a preceding Full layer. The paper first establishes why the most obvious approach fails, then presents the greedy search solution.

Why Uniform Interleaving Is Suboptimal

The simplest strategy would be a uniform interleaving pattern: retain every $r$-th layer's indexer and skip the rest (e.g., with $r=4$, the pattern would be F S S S F S S S ...). This treats all layers as equally sensitive to index removal. The paper demonstrates empirically that this assumption is false, with two pieces of evidence:

Quantitative degradation. Table 2 reports that uniform interleaving at 1/4 retention drops the Long Avg benchmark score by 7.2 points (from 50.2 to 43.0), while the greedy-searched pattern at the same retention ratio preserves Long Avg at 49.9 — essentially identical to the original DSA's 50.2. This means uniform interleaving's degradation is not inherent to the retention ratio but rather to which specific layers lose their indexers.

Layer-specific sensitivity. The paper observes that certain layers — "particularly those in the early and transitional regions of the network" — are far more sensitive to indexer removal than others. The greedy search loss curve (shown in the small inline figure in Section 3.1.2) reveals a clear separation: the first ~20 layers converted to S incur minimal loss increase (the "easy" regime), while the last ~10 layers before the target retention ratio show sharply increasing loss (the "critical" regime). A uniform pattern cannot distinguish between these two regimes — it may remove a critical indexer while retaining a redundant one, leading to noticeable quality degradation.

This motivates a data-driven approach: let the model itself tell us, through its language modeling loss, which indexers are expendable.

Calibration Set Construction

The greedy search uses a small calibration set of $B$ mini-batches cached from the training data. The paper specifies that the calibration set uses SFT (supervised fine-tuning) data with a batch size of 768 and a context length of 200K tokens for the 30B model experiments. All candidate patterns are evaluated on exactly the same batches, ensuring that loss differences reflect only the effect of the pattern change, not data variance. This is a critical design choice: if different patterns were evaluated on different random batches, the noise from data sampling could swamp the signal from pattern quality, making it impossible to identify the genuinely best layers to convert.

The loss metric used is the standard per-token language modeling loss (cross-entropy between the model's predicted next-token distribution and the ground-truth next token). This is computed via a full forward pass of the model with the given pattern $c$ applied. The function is denoted:

$\text{EVALLOSS}(M, D, c)$

where $M$ is the frozen DSA model, $D$ is the calibration batch, and $c$ is the binary pattern string. The output is a single scalar: the average negative log-likelihood per token.

Greedy Search Procedure (Algorithm 1)

The search proceeds incrementally, starting from the all-Full baseline and greedily converting one Full layer to Shared at a time until the target number of Shared layers is reached.

Initialization. All layers start as Full: $c = \text{F}^N$ (a string of $N$ F characters). The set of candidate layers $R$ is initialized to $\{2, 3, \ldots, N\}$ — all layers except layer 1, which is fixed as F because there must always be at least one Full layer to seed the initial indices.

Iterative conversion. For $K$ steps, where $K$ is the target number of Shared layers (e.g., $K = 3N/4$ to retain only 1/4 of indexers):

1. For each currently-Full layer $\ell \in R$, tentatively flip $c_\ell$ to S and evaluate the resulting LM loss: $\text{EVALLOSS}(M, D, c \text{ with } c_\ell \leftarrow S)$.

2. Select the layer whose tentative conversion produces the lowest loss: $\ell^* = \arg\min_{\ell \in R} \text{EVALLOSS}(M, D, c|_{c_\ell \rightarrow S})$.

3. Permanently convert that layer: $c_{\ell^*} \leftarrow S$, and remove it from the candidate set: $R \leftarrow R \setminus \{\ell^*\}$.

After $K$ steps, the pattern $c$ has exactly $N - K$ Full layers and $K$ Shared layers. The order in which layers are converted encodes an implicit importance ranking: layers converted early (with minimal loss increase) are the least sensitive to index removal; layers converted late or never converted are the most critical to retain as Full.

What the loss evaluates. Crucially, the loss evaluation at each step is an end-to-end measure — it captures not just how the converted layer's own attention output changes, but how that perturbation propagates through all subsequent layers to affect the final token prediction. This is what distinguishes the greedy search from the failed similarity-based approach (Appendix C). A local metric (cosine similarity of attention outputs at a single layer) cannot predict whether a small perturbation at layer 5 will be amplified or dampened by layers 6–47. The LM loss implicitly captures this propagation behavior because it measures the model's final output, which is the product of all layers' computations.

Pipeline Parallelism Acceleration

The paper notes that a naive implementation of the greedy search requires $N(N-1)/2$ forward passes — one for each tentative flip at each step, summed over all steps. For $N=47$, this is approximately 1,081 forward passes per calibration batch, which could be prohibitively expensive for large models.

To accelerate the search when the model is partitioned into $P$ pipeline stages, the authors split the layers into $P$ blocks (with each block's first layer fixed as F to maintain an index source for that block). Within each search step, the blocks are searched sequentially: the best flip in block 1 is identified and committed before block 2 is searched, and so on. This allows up to $P$ layers to be converted per step (one per block), reducing the total number of forward passes by roughly a factor of $P$. The paper does not specify the exact value of $P$ used in their experiments, but typical pipeline parallelism configurations for 30B models would involve 4–8 stages.

Properties of the Greedy Solution

The paper reports three consistent empirical properties of the greedy search results that validate the approach:

The searched pattern outperforms uniform interleaving. This is demonstrated quantitatively in Table 2: at 1/4 retention, the searched pattern achieves Long Avg 49.9 vs. 43.0 for uniform, and at 1/8 retention, 46.1 vs. 35.3. The gap widens at more aggressive retention ratios, suggesting that the importance of which layers are retained grows as fewer indexers remain.

The per-step loss curve reveals a natural importance ordering. The inline figure in Section 3.1.2 shows a plot of LM validation loss versus the number of S layers converted (with markers at 1/2, 1/4, and 1/8 retention ratios). The curve is initially flat (the first ~20 conversions cause negligible loss increase), then gradually rises, and finally steepens sharply after about 35 conversions. This "elbow" in the curve represents the transition from redundant to critical indexers — layers that genuinely need their own indexer to avoid cascading errors. The paper interprets this as evidence of a "natural ordering of indexer importance" that is an intrinsic property of the model.

Results are stable across different calibration sets. The importance ranking (which layers get converted early vs. late) is consistent across different random draws of calibration data, indicating that it reflects a structural property of the model rather than overfitting to a particular data sample. This is important for practical deployment: the pattern can be determined once on a representative calibration set and then applied to all future inference without re-tuning.

LM loss serves as a valid proxy for downstream task performance. The paper states that "lower LM loss is positively correlated with better task performance," which is the justification for using LM loss rather than directly optimizing downstream benchmark scores during the search. This is a standard assumption in language model optimization (reducing perplexity tends to improve downstream metrics), but it is important that it holds here because the alternative — evaluating each candidate pattern on the full benchmark suite — would be computationally infeasible.

Complexity and Practicality

The full greedy search from all-F to all-S performs $O(N^2)$ forward passes. With the pipeline parallelism acceleration, this is reduced to approximately $O(N^2 / P)$ passes. For the 47-layer 30B model with a handful of pipeline stages, this is computationally tractable as a one-time offline optimization. The paper does not report the wall-clock time required for their pattern search, but notes that the calibration set consists of only $B$ mini-batches (a small fraction of the full training data), which limits the per-pass cost.

---

Training-Aware IndexCache: Multi-Layer Distillation

The training-free approach works by avoiding sensitive layers — the greedy search finds which layers happen to be robust to receiving cached indices. The training-aware approach takes the opposite strategy: it eliminates layer sensitivity by explicitly training each retained indexer to serve multiple layers, and training each Shared layer to expect inherited indices. This is feasible only when training from scratch or via continued pre-training, but it produces a more robust solution where even simple uniform patterns can match full-indexer quality.

The Core Training Objective: Multi-Layer Distillation Loss

In standard DSA training, each indexer at layer $\ell$ is trained via a KL-divergence distillation loss against its own layer's aggregated full attention distribution. The standard loss is:

$L_I = \sum_t D_{\text{KL}}\left(p^{(\ell)}_t \| q^{(\ell)}_t\right)$

where $p^{(\ell)}_t$ is the aggregated attention distribution at layer $\ell$ for query position $t$ (obtained by averaging softmax attention weights across all heads), and $q^{(\ell)}_t = \text{Softmax}(I^{(\ell)}_t)$ is the indexer's output distribution (the indexer's raw scores passed through a softmax to produce a probability distribution over all preceding token positions). The KL divergence $D_{\text{KL}}(p \| q) = \sum_s p(s) \log(p(s)/q(s))$ measures how much information is lost when using $q$ to approximate $p$; training minimizes this divergence so the indexer learns to produce scores that, when softmaxed, match the true attention distribution as closely as possible.

IndexCache generalizes this to a multi-layer objective. Consider a retained Full layer $\ell$ and $m$ subsequent Shared layers $\ell+1, \ell+2, \ldots, \ell+m$ that will reuse its index set $T^{(\ell)}$. During training, the indexer at layer $\ell$ is now responsible for producing a top-k selection that is useful not only for layer $\ell$ itself but also for all $m$ layers that will inherit its indices. The multi-layer distillation loss encourages this by summing the KL divergences against all served layers' attention distributions:

$L_I^{\text{multi}} = \sum_{j=0}^{m} \frac{1}{m+1} \sum_t D_{\text{KL}}\left(p^{(\ell+j)}_t \| q^{(\ell)}_t\right)$

where $j$ indexes over the served layers (from $\ell$ to $\ell+m$), $p^{(\ell+j)}_t$ is the target attention distribution at the served layer $\ell+j$, and $q^{(\ell)}_t$ is the Full layer $\ell$'s indexer output distribution. The $1/(m+1)$ factor equally weights each served layer.

What this loss achieves. By training against the attention distributions of all served layers simultaneously, the indexer at layer $\ell$ learns to predict a top-k set that is a consensus — it must cover the important tokens for layer $\ell$'s own attention, layer $\ell+1$'s attention, layer $\ell+2$'s attention, and so on. If different served layers attend to slightly different token subsets, the indexer learns to include the union of the most important tokens across all layers. This is fundamentally different from the standard single-layer distillation, where the indexer can specialize to its own layer's idiosyncratic preferences.

Gradient equivalence to averaged-target distillation. A potential concern is whether summing multiple KL terms introduces complex interactions or conflicting gradient signals. The paper proves that the multi-layer loss is mathematically equivalent to a simpler objective: distilling against the averaged attention distribution of all served layers.

Define the averaged target distribution:

$\bar{p}_t = \sum_{j=0}^{m} \frac{1}{m+1} p^{(\ell+j)}_t$

and the corresponding single-target distillation loss:

$L_I^{\text{avg}} = \sum_t D_{\text{KL}}\left(\bar{p}_t \| q^{(\ell)}_t\right)$

Proposition 1 (Gradient equivalence). The paper proves that $\nabla_\theta L_I^{\text{multi}} = \nabla_\theta L_I^{\text{avg}}$, meaning training with the multi-layer loss produces exactly the same parameter updates as training with the averaged-target loss.

The proof (provided in the main text) relies on the linearity of the gradient operator and the fact that the KL divergence's gradient with respect to the model parameters $\theta$ (which only appear in $q^{(\ell)}_t$) is:

$\nabla_\theta D_{\text{KL}}(p \| q^{(\ell)}_t) = -\nabla_\theta \sum_s p(s) \log q^{(\ell)}_t(s)$

because the entropy term $-\sum_s p(s) \log p(s)$ does not depend on $\theta$. Summing over $j$ and factoring out the summation over $s$ yields the gradient of the averaged-target loss exactly. The proof is a direct algebraic manipulation; the key insight is that the KL divergence is linear in its first argument ($p$) when differentiating with respect to the second argument's parameters.

Why this equivalence matters. It provides a clean interpretation: the multi-layer distillation objective is not an ad-hoc regularizer but is exactly equivalent to training the indexer to predict the centroid (pointwise average) of the served layers' attention distributions. The indexer learns a consensus distribution that jointly covers the important tokens across all layers it serves. This interpretation also explains why the training-aware approach eliminates layer-specific sensitivity: each retained indexer is now optimized for a shared target rather than a layer-specific one, so the Shared layers inheriting its indices see an attention distribution that was explicitly designed to be useful for them.

Implementation Choice: Multi-Layer vs. Averaged-Target Loss

Despite the gradient equivalence, the paper uses $L_I^{\text{multi}}$ in practice rather than $L_I^{\text{avg}}$. The reason is implementation efficiency:

• In the multi-layer formulation, for each served layer $j$, the model only needs to pass the current layer $\ell$'s predicted distribution $q^{(\ell)}$ to the loss computation. The target distribution $p^{(\ell+j)}$ is already available at layer $\ell+j$ from the full attention computation used during training.

• In the averaged-target formulation, computing $\bar{p}_t$ requires averaging the attention distributions from all $m+1$ served layers, which means passing $p^{(\ell)}, p^{(\ell+1)}, \ldots, p^{(\ell+m)}$ to where $q^{(\ell)}$ is computed, introducing additional communication overhead and memory usage.

The multi-layer formulation avoids this by computing each KL term locally (at each served layer, comparing that layer's target against the shared $q^{(\ell)}$) and summing the losses. This is more memory-efficient and requires no additional cross-layer communication during training.

Two-Stage Training Pipeline

Training-aware IndexCache follows the same two-stage DSA training procedure used for standard DSA models, with the multi-layer distillation loss substituted for the single-layer loss:

Stage 1: Dense warm-up (1,000 steps). Only the indexer parameters are trained; all other model parameters (core attention, feed-forward networks, embeddings, etc.) are frozen. The objective is the multi-layer distillation loss $L_I^{\text{multi}}$, which trains each retained indexer against the averaged attention distributions of all layers it will serve. During this phase, top-k selection is not applied — the indexer is trained using dense (full) attention targets to learn good scoring before the discretization of top-k selection is introduced. This warm-up phase is critical because training an indexer from scratch with top-k selection active would create a pathological feedback loop: poor initial indexer scores → poor top-k selections → poor attention → poor gradients → continued poor indexer scores.

Stage 2: Sparse training (4,000 steps). Top-k selection is activated: the indexer at each Full layer selects the top $k = 2048$ highest-scoring positions, and core attention is computed only over this sparse subset. The indexer continues to receive distillation gradients via $L_I^{\text{multi}}$, but now the KL divergence is computed only over the selected top-k tokens (rather than over the full distribution). All model parameters (indexer, core attention, FFN, embeddings) are jointly optimized during this phase. The training also includes the standard language modeling loss (next-token prediction cross-entropy) so the full model learns to produce correct outputs with sparse attention.

The paper initializes training from the base GLM-4.7-Flash model (rather than from scratch) to save computational resources, and uses a shorter pipeline (1,000 + 4,000 = 5,000 total steps) compared to full DSA training. They report that this shortened pipeline "closely matches the performance of full DSA training and suffices for evaluating IndexCache's training-aware component."

Key hyperparameters: The paper specifies a context length of 200K tokens during training, matching the longest evaluation context. The SFT data used for both training stages is the same data used for the calibration set in the training-free variant. The 1,000-step warm-up and 4,000-step sparse phase are specific to the 30B model; training-aware IndexCache for the 744B GLM-5 is left to future work.

Why Training-Aware IndexCache Eliminates Pattern Sensitivity

The most striking result of the training-aware approach (Table 3) is that uniform interleaving at 1/2 retention matches or exceeds the greedy-searched pattern, and even the full-indexer baseline. This is in sharp contrast to the training-free results (Table 2), where uniform interleaving causes significant degradation and the greedy search is essential.

The paper explains this through the mechanism of joint adaptation. During training-aware IndexCache training:

For retained Full layers: The multi-layer distillation loss forces each indexer to predict a top-k set that covers important tokens for all layers it serves, not just its own. This means the indexer learns to produce a broader selection that is robust to the variations in attention focus across its served block. In the training-free case, each indexer was trained only for its own layer and may have specialized to that layer's specific preferences, making its indices less suitable as a proxy for other layers.

For Shared layers: The sparse training phase trains these layers' core attention to work with inherited indices (since during training, the top-k selection at these layers is drawn from the Full layer's indexer output). In the training-free case, the Shared layers' attention weights were trained to work with their own indexers' selections; reusing another layer's indices introduces a distributional shift that was never seen during training. Training-aware IndexCache eliminates this shift by design: the Shared layers are trained from the start to expect inherited indices.

Joint adaptation: The combination of these two effects means that the retained indexers and the Shared layers co-adapt during training. The indexer learns to produce indices that are useful for its served block, and the block's layers learn to attend effectively given those indices. This co-adaptation makes the system robust to the specific sharing pattern — whether uniform or searched, the model has learned to function with cross-layer index reuse, so the exact boundary positions between blocks matter less than they do in the training-free case.

The paper also reports an ablation confirming the importance of the cross-layer distillation loss: removing it (training each indexer only against its own layer's attention, even though it will be shared at inference) drops Long Avg from 51.6 to 49.8, with AA-LCR falling from 49.8 to 44.0. This directly quantifies the benefit of the multi-layer objective over simply training a standard DSA model and then applying a sharing pattern.

---

The Similarity-Based Approach: A Negative Result That Informs the Final Design

The paper includes Appendix C as a "negative result" — an approach that was explored, found insufficient, but is reported for completeness and to provide insight into why the loss-based search is necessary. This transparency is valuable for understanding the design space and avoiding dead ends.

Similarity Matrix Construction

The idea behind the similarity-based approach is natural: instead of evaluating LM loss for each candidate pattern (which requires a full forward pass), build a matrix that measures how well each layer's index can serve as a proxy for each other layer's index. The paper constructs an $N \times N$ lower-triangular similarity matrix $S$, where $S_{i,j}$ (for $i > j$) quantifies how well layer $j$'s index set can replace layer $i$'s own index set.

The procedure requires $N$ separate forward passes over a calibration set. For each pass, one layer $j$ is treated as the "anchor" (its indexer runs and produces $T^{(j)}$), and all subsequent layers $i > j$ compute their core attention twice: once using their own indexer's selection $T^{(i)}$ (the original model behavior), and once using the cached selection $T^{(j)}$ (as if layer $i$ were a Shared layer inheriting from layer $j$). The cosine similarity between the two resulting attention outputs at layer $i$ is recorded as $S_{i,j}$.

A value $S_{i,j} \approx 1$ indicates that layer $i$'s attention output is nearly identical whether it uses its own indexer or layer $j$'s indexer, suggesting that layer $i$ can safely inherit from layer $j$ with minimal distortion. A value $S_{i,j} \ll 1$ indicates a substantial difference, suggesting that layer $i$ needs its own indexer or a closer Full layer.

Dynamic Programming for Pattern Selection

Given the similarity matrix $S$ and a target number of Full layers $M$, the goal is to find the pattern $c^*$ that maximizes the total similarity across all Shared layers:

$c^* = \arg\max_{c: |\{i: c_i = F\}| = M} \sum_{\ell: c_\ell = S} S_{\ell, \text{src}(\ell)}$

where $\text{src}(\ell) = \max\{j < \ell : c_j = F\}$ is the nearest preceding Full layer from which layer $\ell$ inherits its indices, and the sum is over all Shared layers, each weighted by how well its attention output matches when using the inherited indices.

This optimization can be solved exactly via dynamic programming. Let $\text{dp}[i][k]$ denote the maximum cumulative similarity achievable for layers $1, \ldots, i$ using exactly $k$ Full layers, with layer $i$ itself being a Full layer. The recurrence is:

$\text{dp}[i][k] = \max_{j < i, c_j = F} \left( \text{dp}[j][k-1] + \sum_{m=j+1}^{i-1} S_{m, j} \right)$

where the summation accounts for all Shared layers between the previous Full layer $j$ and the current Full layer $i$, each reusing layer $j$'s indices with similarity $S_{m,j}$.

Why Similarity-Based Search Fails

Despite its theoretical appeal and computational efficiency (requiring only $N$ forward passes to build the matrix, versus $O(N^2)$ passes for the greedy search), the similarity-based approach produces patterns that perform no better than uniform interleaving on downstream benchmarks (Table 5). At 1/2 retention, the similarity-searched pattern achieves Long Avg 49.8 — essentially identical to uniform interleaving's 50.7 and significantly below the greedy-searched pattern's 50.3.

The paper identifies the root cause: per-layer output similarity is a local metric that cannot capture cascading error propagation. When layer $i$ reuses layer $j$'s indices instead of computing its own, its attention output changes slightly. This change feeds into layer $i+1$'s hidden state, which changes layer $i+1$'s attention output (even if layer $i+1$ is a Full layer computing its own indices), which feeds into layer $i+2$'s hidden state, and so on. The cosine similarity $S_{i,j}$ measures only the immediate effect at layer $i$ — it cannot predict whether a small perturbation at layer $i$ will be amplified, dampened, or qualitatively transformed by the subsequent layers.

A concrete example: suppose the index mismatch causes layer $i$ to miss a specific named entity token that is critical for later co-reference resolution. At layer $i$, the attention output might be 99% similar to the original (high $S_{i,j}$). At layer $i+10$, where the model needs to link a pronoun back to that entity, the missing information causes a qualitatively different attention pattern. The similarity metric at layer $i$ sees a 1% difference; the downstream effect is a completely wrong co-reference resolution. The greedy loss-based search avoids this by evaluating the end-to-end LM loss, which inherently captures all downstream propagation effects.

The paper also notes that "two layers may have nearly identical attention outputs yet differ in subtle ways that matter for downstream quality: for instance, the reused index may miss a small number of critical tokens whose importance only becomes apparent in later layers' reasoning steps." This is the fundamental limitation of any local similarity metric for this problem — the tokens that matter for a given layer's attention are not necessarily the tokens whose absence causes cascading failures across the full network depth.

---

Practical Considerations and Implementation Details

Pattern Search Complexity Comparison

The paper explores three approaches to pattern selection, with escalating computational cost and escalating pattern quality:

• Uniform interleaving: Zero search cost, but unacceptable quality degradation at aggressive retention ratios (e.g., 7.2-point Long Avg drop at 1/4 retention in the training-free setting).

• Similarity-based DP: $O(N)$ forward passes to build the matrix plus $O(N^2 M)$ DP time (where $M$ is the target number of F layers). Fast but quality is comparable to uniform interleaving — the search doesn't help because the similarity metric is a poor proxy for downstream quality.

• Greedy loss-based search: $O(N^2/P)$ forward passes, where $P$ is the number of pipeline stages. More expensive but produces the highest-quality patterns, recovering full-indexer performance at 1/4 retention.

The paper's recommendation is clear: if you cannot afford the greedy search (because the model is too large or calibration data is unavailable), uniform interleaving at modest ratios like 1/2 may be acceptable, but for aggressive retention ratios (1/4 or lower), the greedy search is necessary in the training-free setting. If retraining is possible, training-aware IndexCache with uniform interleaving is the most practical approach, as it eliminates the need for any pattern search entirely while matching full-indexer quality.

The First Layer Constraint

The paper enforces that layer 1 is always Full ($c_1 = F$). This is a hard constraint because there is no preceding layer to provide cached indices. Without this constraint, a Shared layer 1 would have nothing to read from T_cache, resulting in undefined behavior. This constraint is satisfied by all three pattern selection methods: uniform interleaving naturally starts with F, the DP search explicitly requires it, and the greedy search initializes all layers as F and only considers converting layers 2 through $N$.

Memory Overhead of the Index Cache

A practical concern might be whether T_cache requires additional GPU memory beyond what standard DSA allocates. The paper clarifies that standard DSA already allocates memory for each layer's index tensor (since every layer runs its own indexer and stores its $k$ selected indices). IndexCache simply repurposes the memory that would have been allocated for Shared layers' index tensors — because those layers no longer run their indexers, they don't need to store their own indices, and the memory can be shared with the cache buffer. The total GPU memory footprint is therefore unchanged from standard DSA.

Pattern Stability Across Data Distributions

The paper reports that the greedy search results are "stable across different calibration sets," meaning the specific layers selected as Full vs. Shared are consistent regardless of which random subset of training data is used for the search. This is important for practical deployment: the pattern can be determined once on a representative data sample and then applied to all future inference, without needing to re-tune for different downstream tasks or data distributions. The implication is that the indexer importance ordering is an intrinsic property of the model architecture and its trained weights, not a function of the particular calibration data.

Extension Beyond DSA

The paper claims that IndexCache's core principle extends to any sparse attention method that uses dynamic token selection rather than a fixed sparse pattern. Specifically, it mentions that "the block-level selection in MoBA (Lu et al., 2025) and NSA (Yuan et al., 2025) could similarly benefit from cross-layer reuse." The general insight is that whatever mechanism selects which tokens to attend to — whether it is DSA's per-token indexer, MoBA's block-level router, or another dynamic selection mechanism — its output is likely redundant across layers and can be shared. The specific implementation (which layers to designate as Full, how to handle the cached indices) would need to be adapted to each method's selection granularity and interface, but the principle is transferable.

4. Key Insights and Innovations

Innovation 1: Reframing the Bottleneck — From "Core Attention Is Expensive" to "The Indexer Is Expensive"

The dominant narrative in sparse attention research has been that the core attention computation is the enemy. The entire field — from training-free heuristic eviction methods (Zhang et al., 2023; Xiao et al., 2024) through trainable sparse architectures (Yuan et al., 2025; Lu et al., 2025) to production systems like DSA (Liu et al., 2025) — has been organized around a single premise: reduce the $O(L^2)$ attention cost to $O(Lk)$, and inference becomes fast. DSA was the culmination of this logic: it successfully reduced core attention to $O(Lk)$ while preserving model quality through continued pre-training and distillation.

IndexCache's first conceptual contribution is to point out that this success story has a hidden chapter. The mechanism that enables sparse attention — the lightning indexer — was treated as overhead, as the small price you pay for making the expensive thing cheap. But the paper's profiling analysis (Section 1, intro figure) shows that this "overhead" grows with context length until it becomes the dominant cost: 27% of prefill time at 10K tokens, 81% at 200K. The indexer was designed to be lightweight (FP8 arithmetic, low-rank projections, few heads), but lightweight-times-quadratic still beats heavyweight-times-linear at sufficient scale. The paper's diagnosis is that DSA didn't eliminate the quadratic bottleneck — it moved it from core attention to the indexer, and as context lengths grow, the indexer becomes the new bottleneck.

This reframing is significant because it changes what problem the community should be solving. Before IndexCache, the research agenda was "make sparse attention sparser" or "design better indexers." After IndexCache, the agenda becomes "the indexer itself is quadratic and redundant — how do we eliminate most of it?" This is not an incremental improvement to DSA but a fundamental reconceptualization of what limits sparse attention inference at scale. The paper makes this explicit by profiling the indexer's cost share across context lengths (Table 1, Figure 3), showing that the asymptotic behavior is what matters, not the per-FLOP efficiency of any individual component.

Innovation 2: Extending Cross-Layer Sharing Beyond Full Attention Anchors

Cross-layer token selection stability was established by prior work (Deshmukh et al., 2025; Gao et al., 2026) as a property of full-attention transformers: adjacent layers attend to largely the same tokens. Methods like TidalDecode, Kascade, and HySparse exploited this by designating a few anchor layers that compute full $O(L^2)$ attention and letting intermediate layers reuse their top-k indices. This worked because the anchor layer's full attention provides a "ground truth" signal — the anchors actually compute the complete attention distribution, so their top-k selections are exact.

IndexCache's conceptual move is to demonstrate that this sharing principle survives when full attention is removed entirely. The paper shows that DSA's lightweight indexer outputs — which are learned approximations to full attention, not exact computations — exhibit the same cross-layer stability (Appendix A, Figure 4): adjacent layers share 70–100% of their top-2048 tokens, the overlap matrix shows clear block structure, and the redundancy is sufficient to support aggressive sharing ratios (up to 75% of indexers removed with negligible quality loss).

This is not an obvious extension. One could reasonably have expected that indexer outputs would be less stable across layers than full attention, because each indexer is a learned approximation that might specialize to its own layer's specific attention patterns. The fact that the stability is strong enough to support 4× indexer reduction without retraining (Table 2: 1/4 retention with searched pattern achieves Long Avg 49.9 vs. 50.2 for full DSA) is an empirical finding with practical consequences. But the deeper contribution is conceptual: it establishes that the selection mechanism — whatever form it takes — is the redundant component, not the attention computation itself. This decouples the sharing principle from the specific implementation and opens the door to applying it to any dynamic sparse attention method (the paper explicitly mentions MoBA and NSA as candidates).

The significance beyond performance is that this finding changes how we should think about designing sparse attention systems. If token selection is inherently redundant across layers, then future architectures should not waste parameters or compute on per-layer selection mechanisms. The paper's split into Full and Shared layers is one way to exploit this, but the principle suggests more radical designs: perhaps selection should be a cross-layer shared module from the start, rather than a per-layer component that we retroactively share. This is a design philosophy implication that goes beyond the specific IndexCache implementation.

Innovation 3: The Greedy Loss-Based Search as a Global Diagnostic for Layer Sensitivity

The most common approach to determining sharing patterns in prior cross-layer methods has been either uniform interleaving (Yang et al., 2025a,b; Hao et al., 2025) or similarity-based selection (Deshmukh et al., 2025, using dynamic programming over a cross-layer similarity matrix). IndexCache's training-free variant introduces a fundamentally different approach: use the model's own end-to-end language modeling loss as the selection criterion, and greedily accumulate Shared layers in order of increasing loss impact.

What makes this distinctive is not the greedy algorithm itself (which is standard) but the choice of what to optimize — and the negative result in Appendix C that shows why the obvious alternative fails. The similarity-based approach (measuring per-layer attention output similarity when indices are reused) seems theoretically natural: if layer $i$'s attention output is 99% similar whether it uses its own indexer or layer $j$'s, then $i$ should be safe to share from $j$. The dynamic programming formulation over this similarity matrix is elegant and computationally cheap ($O(N)$ forward passes versus $O(N^2)$ for greedy search). But it produces patterns that perform no better than uniform interleaving on downstream benchmarks (Table 5).

The paper's diagnosis of this failure — that per-layer similarity is a local metric that cannot capture cascading error propagation across subsequent layers — is a conceptual contribution in its own right. It tells us that when we remove an indexer, we are not just changing one layer's attention output; we are perturbing the hidden state that feeds into all subsequent layers, and those perturbations can be amplified, dampened, or qualitatively transformed in ways that a local cosine similarity cannot predict. A 1% difference at layer 5 might cause a 20% difference at layer 30 if the missing information was critical for a reasoning step that only becomes relevant later.

The greedy loss-based search succeeds precisely because it is global: the LM loss measures the model's final output distribution, which is the product of all layers' computations. It implicitly captures the full propagation chain. The per-step loss curve (shown in the inline figure in Section 3.1.2) is itself a diagnostic tool — it reveals a natural "elbow" separating easily-shared layers from critical ones, providing insight into the model's internal organization that a similarity matrix cannot.

This innovation matters beyond IndexCache because it establishes a general methodology for evaluating which components in a deep network are sensitive to approximation. Any time we consider removing or simplifying internal computations at specific layers, a local input-output similarity metric may be misleading, and an end-to-end loss-based search may be necessary. The paper provides both the positive result (greedy search works) and the cautionary tale (similarity-based search doesn't), which together define the methodological standard for this class of problems.

Innovation 4: The Training-Aware Approach as a Demonstration That Sharing-Induced Distribution Shift Can Be Eliminated Rather Than Avoided

The training-free variant of IndexCache works by avoiding sensitive layers — the greedy search finds which layers happen to tolerate inherited indices with minimal quality impact. This is a mitigation strategy: it routes around the problem rather than solving it. The training-aware variant takes the opposite approach: it eliminates the sensitivity entirely by retraining the model to expect cross-layer sharing from the start.

The conceptual contribution here is not the multi-layer distillation loss itself (distillation against averaged targets is a known technique) but rather the demonstration that the entire pattern sensitivity problem vanishes under retraining. In the training-free setting (Table 2), uniform interleaving at 1/2 retention causes a 2.8-point Long Avg drop compared to full DSA; at 1/4 retention, the drop is 7.2 points. The greedy search is essential for recovering quality — the searched pattern at 1/4 retention achieves 49.9, nearly matching the 50.2 baseline. This tells us that in a frozen model, layer-specific indexer coupling is real and must be respected.

In the training-aware setting (Table 3), the story inverts. Uniform interleaving at 1/2 retention achieves Long Avg 51.6, surpassing the full-indexer baseline of 51.0, and the greedy-searched pattern achieves only 50.6. The pattern no longer matters — or rather, the model has been trained to make any reasonable pattern work. This is not a performance gain (the difference between 51.0 and 51.6 is within noise) but a robustness gain: the system is now insensitive to the specific sharing configuration, which means practitioners can use simple uniform patterns without any pattern search, without any calibration data, and without any worry about hitting a sensitive layer.

The mechanism that enables this — joint adaptation of the Full layer's indexer (trained to produce indices useful for all served layers) and the Shared layers' core attention (trained to expect inherited indices) — is described in Section 3. But the conceptual takeaway is broader: it suggests that many "sensitivity" problems in deep networks that appear when we post-hoc modify inference behavior (pruning, quantization, sparsification, index sharing) are artifacts of the training objective being mismatched to the inference objective. When the training objective is aligned with the inference modification — when the model is trained to expect index reuse rather than having it imposed retroactively — the sensitivity disappears. This is a specific instance of a general principle that has implications beyond sparse attention: if you want to modify inference behavior, train for it explicitly rather than trying to find configurations that happen to work on a frozen model. The ablation removing the cross-layer distillation loss (Table 3, "w/o cross-layer loss": Long Avg drops from 51.6 to 49.8) confirms that it is specifically the multi-layer training objective, not just any retraining, that produces this robustness. Training each indexer only against its own layer and then applying sharing at inference doesn't work — it reproduces the training-free problem. The indexers must be explicitly trained for the sharing pattern that will be used.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All main experiments are evaluated across nine benchmarks: five long-context benchmarks — MRCR v2 (OpenAI, 2025b), GraphWalks (OpenAI, 2025a), LongBench v2 (Bai et al., 2025), RULER (Hsieh et al., 2024), and AA-LCR (Artificial Analysis, 2025) — and four general and reasoning benchmarks — AIME 2025 (Mathematical Association of America, 2025), GPQA-Diamond (Rein et al., 2024), LiveCodeBench v6 (Jain et al., 2024), and IFBench (Pyatkin et al., 2025). The calibration set for training-free pattern search uses SFT (supervised fine-tuning) data with a batch size of 768 and a context length of 200K tokens, drawn from the same training data used for model development (Section 4.1).

• Base model(s). The primary model is a 30B-A3B Mixture-of-Experts model with 47 layers using Multi-head Latent Attention (MLA), obtained via two-stage DSA training starting from the base model of GLM-4.7-Flash (Zeng et al., 2026). The paper states that its "evaluation performance is comparable to that of the original GLM-4.7-Flash" (Section 4.1). For scaling experiments, the paper uses GLM-5, a 744B-parameter (40B active) production model that uses DSA by default (Section 4.5).

• Metrics. The paper reports benchmark-specific accuracy scores as defined by each benchmark's standard evaluation protocol: MRCR v2 reports average score across 2-, 4-, and 8-needle settings; GraphWalks reports average over Parent-type and BFS-type problems; RULER reports scores on all instances with context lengths from 4K to 128K; LongBench v2, AA-LCR, and the general/reasoning benchmarks each report their standard single-number accuracy metrics (Appendix D). These individual scores are then aggregated into Long Avg (the arithmetic mean of the five long-context benchmarks) and G&R Avg (the arithmetic mean of the four general and reasoning benchmarks) to provide summary statistics (Tables 2–4). For inference speed measurements, the paper reports prefill latency in seconds, decode throughput in tokens per second (both per-request under single concurrency and total throughput when the KV cache is fully saturated at ~800K tokens per GPU), and relative speedup as a percentage over the DSA baseline (Table 1, Figure 3).

• Baselines. The primary baseline is standard DSA — the same model with all indexers running at every layer (Full pattern for all 47 layers). For training-free IndexCache (Table 2), the paper compares three retention ratios (1/2, 1/4, 1/8) each in two configurations: uniform interleaving (e.g., F S S S F S S S ... for 1/4 retention) and a greedy-searched pattern from Algorithm 1. For training-aware IndexCache (Table 3), the baselines include: the same original DSA baseline (but trained under the shortened DSA pipeline used in this section), uniform interleaving at 1/2 and 1/4 retention, and ablations that use the greedy-searched pattern or remove the cross-layer distillation loss (training each indexer only against its own layer's attention). The uniform interleaving baseline serves as the critical comparison point because it represents the simplest possible pattern with no search cost.

• Generation budget / compute accounting. IndexCache's cost reduction is measured in terms of fraction of indexer computations eliminated, not in FLOPs directly. Removing 75% of indexers corresponds to eliminating 75% of the $O(NL^2)$ total indexer cost. The paper uses retention ratio as the independent variable: 1/2 retention means half of layers retain their indexers, 1/4 means a quarter retain, and 1/8 means an eighth retain. For end-to-end speedup measurements (Table 1, Figure 3), the paper reports actual wall-clock latency and throughput on identical hardware (NVIDIA H100 node, 8-way data parallelism with dp attention enabled in SGLang) across four context lengths: 10K, 60K, 120K, and 200K tokens. The speedup numbers directly measure the reduction in inference time from eliminating indexer computations, with no additional overhead from T_cache management (Figure 1 mentions ~1.2× end-to-end speedup for GLM-5 at 1/2 retention; Table 1 and Figure 3 provide the detailed breakdown for the 30B model).

• Cross-validation / statistical protocol. The greedy pattern search (Algorithm 1) evaluates all candidate patterns on exactly the same calibration batches to ensure loss differences reflect only pattern changes, not data variance (Section 3.1.2). The paper reports that search results are "stable across different calibration sets" (Section 3.1.2), indicating that the greedy solution is robust to which specific data samples are used. For benchmark evaluation (Appendix D), all generations use temperature 1.0, top-p = 0.95, and top-k = 40. Long-context tasks use a 200K-token context window with 32K reserved for output; general and reasoning tasks allow a maximum output length of 64K tokens. For benchmarks where the input exceeds the effective budget of 168K tokens (200K − 32K output reservation), middle truncation is applied (Appendix D).

Main Quantitative Results

End-to-End Inference Speedup

Table 1 and Figure 3 present the core efficiency claim: IndexCache delivers substantial speedups that grow with context length and retention ratio. The headline numbers at 200K context length are:

"At 200K tokens, IndexCache (1/4) reduces prefill latency from 19.5s to 10.7s, achieving a 1.82× speedup"

and for decode throughput:

"At 200K, DSA's decode speed is 58 tok/s, while IndexCache (1/4) achieves 86 tok/s, a 1.48× speedup."

Prefill latency scaling (Figure 3a, Table 1). The prefill speedup exhibits a clear dependence on both context length and retention ratio. At 10K context, where the indexer accounts for only 27% of total prefill time, the speedup is modest: 1.21× at 1/2 retention, 1.27× at 1/4 retention. By 200K context, where the indexer dominates (81% of prefill time), the speedup grows to 1.42× at 1/2 retention (19.5s → 13.7s) and 1.82× at 1/4 retention (19.5s → 10.7s). The paper notes that "extrapolating to longer contexts (>200K), IndexCache is expected to deliver even greater speedups" because the indexer's cost share continues to grow with $L$. The 1/4 configuration at 60K already achieves 1.31× speedup, exceeding the 1.21× at 10K for 1/2 retention, showing that aggressive retention at moderate context lengths can outpace conservative retention at short contexts.

Per-request decode throughput (Figure 3b, Table 1). During autoregressive generation under single concurrency (one request per GPU), the decode speedup also grows with context length. At 10K, DSA achieves 73.5 tok/s, IndexCache (1/4) achieves 91.0 tok/s (1.24× speedup). At 200K, DSA drops to 58.0 tok/s while IndexCache (1/4) maintains 86.0 tok/s (1.48× speedup). The absolute gap widens from 17.5 tok/s at 10K to 28.0 tok/s at 200K, confirming that the indexer bottleneck during decode becomes more severe at longer sequences. The paper explains: "the decode phase in DSA involves a per-token indexer pass over the full context, which becomes the bottleneck at long sequences; IndexCache directly reduces this bottleneck."

Full-KV-cache decode throughput (Figure 3c, Table 1). When the KV cache is saturated (~800K tokens per GPU), total decode throughput also improves substantially. At 200K, DSA delivers 197 tok/s total; IndexCache (1/4) achieves 297 tok/s (1.51× increase). The improvement is consistent across context lengths: 1.14× at 10K (2700 → 3310 tok/s), 1.22× at 60K (613 → 840 tok/s), 1.26× at 120K (341 → 498 tok/s), and 1.51× at 200K. The paper observes: "the largest gains at longer contexts" for this metric as well.

GLM-5 scaling results (Section 4.5, Figure 1). Preliminary experiments on the 744B GLM-5 model with IndexCache at 1/4 retention yield "at least 1.3× improvement in both prefill latency and decode throughput at context lengths beyond 100K." Figure 1 shows a broader evaluation across the Artificial Analysis Index: GLM-5 + IndexCache at 1/2 retention delivers "~1.2× end-to-end speedup" while maintaining "nearly identical" performance to the original GLM-5 across both long-context and reasoning benchmarks. The bar chart in Figure 1 visualizes this parity — the two bars are visually indistinguishable for most benchmarks — confirming that the speedup claims scale to production model sizes.

Training-Free IndexCache Benchmark Results

Table 2 reports the central quality preservation claim: at 1/4 retention with a searched pattern, the 30B DSA model with IndexCache achieves Long Avg 49.9 vs. 50.2 for original DSA (a 0.3-point difference), and G&R Avg 74.9 vs. 74.6 (a 0.3-point improvement). The specific numbers per benchmark follow a consistent pattern.

Searched patterns recover long-context quality. The degradation from uniform interleaving is severe and grows with retention ratio:

• At 1/2 retention, uniform interleaving drops Long Avg by 2.8 points (50.2 → 47.4), with the largest drops on MRCR v2 (24.5 → 22.0, −2.5 points), RULER (87.9 → 83.6, −4.3 points), and AA-LCR (43.6 → 38.6, −5.0 points). The searched pattern recovers all of this: Long Avg returns to 50.3, with MRCR v2 at 24.7 (0.2 points above baseline) and RULER at 87.8 (0.1 point below).
• At 1/4 retention, uniform interleaving drops Long Avg by 7.2 points (50.2 → 43.0). The degradation is concentrated in MRCR v2 (24.5 → 17.7, −6.8 points) and GraphWalks (49.6 → 37.2, −12.4 points). The searched pattern achieves Long Avg 49.9, with MRCR v2 at 25.1 (0.6 points above baseline) and GraphWalks at 47.4 (2.2 points below but substantially improved from 37.2).
• At 1/8 retention, uniform interleaving collapses Long Avg to 35.3 (−14.9 points vs. baseline), with catastrophic drops on MRCR v2 (24.5 → 12.9, −11.6 points) and GraphWalks (49.6 → 33.1, −16.5 points). The searched pattern recovers to 46.1 Long Avg — still 4.1 points below baseline but 10.8 points above uniform interleaving. The paper acknowledges this as a "non-negligible" degradation, indicating that 1/8 retention pushes beyond the feasible sharing limit for the training-free approach.

General and reasoning capabilities remain intact. Across all configurations except uniform interleaving at 1/8, G&R Avg stays within 1 point of the 74.6 baseline:

• At 1/2 with searched pattern: 74.4 vs. 74.6 baseline (−0.2).
• At 1/4 with searched pattern: 74.9 vs. 74.6 baseline (+0.3).
• At 1/8 with searched pattern: 73.7 vs. 74.6 baseline (−0.9).

The paper highlights that the 1/4 searched pattern improves over DSA on AIME 2025 (92.6 vs. 91.0, +1.6 points) and GPQA-Diamond (78.6 vs. 77.6, +1.0 points), suggesting "that removing redundant indexer computation may act as a mild regularizer during inference." This is a subtle but important finding: it is not merely that IndexCache avoids hurting reasoning — it sometimes helps, which means that the per-layer indexer computations in standard DSA may occasionally introduce noise or overfitting that is eliminated through index reuse. This regularization effect is not claimed as a general property but is an observed empirical pattern in these specific benchmarks.

Benchmark-level patterns. Examining Table 2 at the individual benchmark level reveals heterogeneous sensitivity:

• MRCR v2 (needle-in-haystack retrieval) shows the largest susceptibility to uniform interleaving at 1/8 (12.9 vs. 24.5 baseline) but the searched pattern recovers to 21.7 — a significant improvement but still 2.8 points below baseline. This suggests that MRCR v2's needle-retrieval task is highly sensitive to which specific tokens the attention mechanism selects, and even the searched pattern at 1/8 misses some critical positions.
• RULER (context-length stress test across 4K–128K) drops from 87.9 to 68.8 with uniform interleaving at 1/8, and recovers to 82.0 with the searched pattern — again improved but below baseline. This task's varying context lengths may expose IndexCache to length-dependent selection shifts that the fixed pattern cannot fully accommodate.
• LongBench v2 (realistic long-context reasoning tasks) is more robust: from 45.5 baseline, uniform interleaving at 1/4 retains 43.1 (−2.4 points), and the searched pattern at 1/4 achieves 45.7 (+0.2). Even at 1/8 with the searched pattern, LongBench v2 stays at 42.3 (−3.2 points). This suggests that many LongBench v2 tasks do not require the precise token-level selection fidelity that MRCR v2 and GraphWalks demand.
• GraphWalks (graph traversal reasoning) shows the most severe degradation with uniform interleaving: from 49.6 baseline to 37.2 at 1/4 (−12.4 points). The searched pattern recovers to 47.4 at 1/4, but at 1/8 uniform drops to 33.1 and the search only recovers to 43.8. GraphWalks may require maintaining a consistent set of node-tracking tokens across layers, making it acutely sensitive to indexer mismatch.
• LiveCodeBench v6 (code generation) drops substantially under aggressive sharing: from 71.4 baseline to 58.7 with uniform interleaving at 1/8 (−12.7 points). The searched pattern recovers to 69.6 at 1/8 — still 1.8 points below baseline. Code generation may require attention to syntactically important tokens (variable definitions, control flow structures) that are sensitive to indexer precision.
• IFBench (instruction following) is notably robust: from 58.4 baseline, the range across all configurations is only 58.0–59.0, with the 1/4 uniform interleaving achieving 58.9 (+0.5). This benchmark appears to measure capabilities that are insensitive to token selection granularity.

Training-Aware IndexCache Benchmark Results

Table 3 demonstrates that uniform interleaving matches or exceeds the full-indexer baseline when the model is retrained with the multi-layer distillation loss. At 1/2 retention with uniform interleaving, Long Avg is 51.6 vs. 51.0 for the DSA baseline (trained under the same shortened pipeline), and G&R Avg is 74.5 vs. 74.2. At 1/4 retention with uniform interleaving, Long Avg is 50.6 vs. 51.0 (a 0.4-point difference) and G&R Avg is 74.1 vs. 74.2 (a 0.1-point difference). The paper states these results confirm that "DSA can be trained to adapt to the sharing pattern."

Uniform interleaving outperforms the searched pattern under training. The most striking comparison in Table 3 is between uniform interleaving and the greedy-searched pattern at 1/2 retention:

• Uniform: Long Avg 51.6, G&R Avg 74.5
• Searched: Long Avg 50.6, G&R Avg 73.6

This is the inverse of the training-free results (Table 2), where the searched pattern was substantially better than uniform. The paper explains: "when the model is retrained with a sharing-aware objective, the S layers learn to adapt their attention to inherited indices, and the retained indexers simultaneously learn to produce selections that generalize across their served layers. This joint adaptation eliminates the layer-specific sensitivity entirely, allowing even a simple uniform pattern to match the full-indexer baseline." The fact that the searched pattern actually performs worse than uniform under training-aware training suggests that the greedy search — which was optimized for a frozen model — selects a pattern that is suboptimal when the model is co-adapted to a different (uniform) sharing structure. The uniform pattern, by providing regular spacing of Full layers, may be a more natural fit for the learned consensus distributions.

The cross-layer distillation loss provides a measurable benefit. The ablation removing the multi-layer loss ("w/o cross-layer loss" row, Table 3) drops Long Avg from 51.6 to 49.8 (−1.8 points). The largest individual benchmark decline is AA-LCR: 49.8 → 44.0 (−5.8 points). Other benchmarks show smaller but consistent drops: LongBench v2 from 47.2 to 45.0, MRCR v2 from 23.8 to 24.6 (a slight increase, likely noise), GraphWalks from 50.2 to 48.3. This confirms that the multi-layer distillation objective is not merely a heuristic — it provides a practically significant improvement over training each indexer solely for its own layer and then applying sharing at inference. The paper's interpretation: without the cross-layer loss, the indexers overfit to their own layers' attention distributions, and the distributional shift from sharing is not fully compensated by the downstream layers' adaptation during sparse training. The cross-layer loss forces each indexer to learn a consensus distribution from the start, eliminating this residual mismatch.

Comparison of training-aware vs. training-free at fixed retention. Comparing Table 3 (training-aware) to Table 2 (training-free) at 1/4 retention:

• Training-free, uniform: Long Avg 43.0 (degraded by 7.2 points)
• Training-aware, uniform: Long Avg 50.6 (degraded by 0.4 points vs. its own DSA baseline of 51.0)
• Training-free, searched pattern: Long Avg 49.9 (degraded by 0.3 points vs. its own DSA baseline of 50.2)
• Training-aware, searched pattern: Long Avg 50.6 (degraded by 0.4 points)

The training-aware approach matches or exceeds the quality of the training-free approach without requiring any pattern search. This is the practical value proposition: if retraining is feasible, you can use a simple uniform pattern, avoid the calibration data collection and $O(N^2)$ forward passes of greedy search, and achieve equivalent or better quality. If retraining is not feasible, the greedy search on a frozen model can achieve comparable quality but requires the one-time search cost.

Training pipeline note. The paper acknowledges that the DSA baselines in Table 2 and Table 3 are not directly comparable because the training-aware experiments use a shortened DSA training pipeline (1,000-step warm-up + 4,000-step sparse training) rather than the full training procedure used for the model in Table 2. This results in a small performance difference: the DSA baseline in Table 3 is 51.0 Long Avg and 74.2 G&R Avg, while in Table 2 it is 50.2 Long Avg and 74.6 G&R Avg. The paper says this shortened pipeline "closely matches the performance of full DSA training and suffices for evaluating IndexCache's training-aware component." The absolute numbers differ slightly but the relative comparisons within each table are valid and consistent.

GLM-5 Scaling Results

Table 4 presents training-free IndexCache applied to the 744B GLM-5 model, the largest-scale validation. The patterns mirror the 30B findings with some scale-specific nuances.

Uniform interleaving at 1/2 retention happens to preserve quality. Long Avg is 78.1 vs. 78.4 baseline (−0.3 points). However, the paper explicitly cautions against interpreting this as evidence that uniform interleaving works at scale:

"uniform interleaving at 1/2 retention happens to preserve Long Avg... but this is likely coincidental, where the fixed alternating pattern simply avoids skipping the most critical indexer layers by chance."

This is an important methodological honesty: a single positive result with uniform interleaving could mislead practitioners into thinking pattern search is unnecessary. The paper argues it is an artifact of the specific 1/2 pattern for this specific model, not a general property.

The searched pattern provides consistent stability. At 1/2 retention, the searched pattern achieves Long Avg 78.7, slightly exceeding the 78.4 baseline. At 1/4 retention, the searched pattern achieves Long Avg 78.0, just 0.4 points below baseline. In contrast, uniform interleaving at 1/4 drops Long Avg to 72.7 (−5.7 points), with catastrophic degradation on GraphWalks (92.7 → 74.9, −17.8 points) and MRCR v2 (71.1 → 65.8, −5.3 points). The searched pattern recovers GraphWalks to 90.3 (−2.4 points vs. baseline) and MRCR v2 to 70.8 (−0.3 points).

Individual benchmark sensitivity at scale:

• RULER is remarkably robust: 97.7 baseline → 97.3 at 1/2 searched → 97.6 at 1/4 searched. The 1/4 uniform interleaving still achieves 96.2, only a 1.5-point drop. This suggests RULER's multi-length stress test may be less sensitive to token selection fidelity in larger models.
• LongBench v2 shows modest degradation: 64.5 baseline → 66.0 at 1/2 searched (+1.5) → 63.7 at 1/4 searched (−0.8). The 1/4 uniform drop is 62.2 (−2.3 points).
• GraphWalks and MRCR v2 remain the most sensitive to sharing ratio, as in the 30B model, with uniform interleaving at 1/4 causing the largest drops.
• AA-LCR shows an interesting pattern: 66.2 baseline → 67.2 at 1/2 searched (+1.0) → 67.6 at 1/4 searched (+1.4). This benchmark improves with more aggressive sharing, reminiscent of the mild regularization effect observed on AIME and GPQA in the 30B model. The paper does not speculate on the mechanism but this is consistent across both model scales.

Figure 1 (all-round evaluation). The bar chart in Figure 1 shows GLM-5 + IndexCache at 1/2 retention across the full Artificial Analysis Index. The performance is described as "nearly identical" to the original GLM-5 model, with the two bars visually overlapping for most benchmarks. This is the highest-level validation in the paper: a production-scale 744B model with IndexCache matches its full-indexer counterpart on a comprehensive evaluation suite while delivering ~1.2× end-to-end speedup.

Ablation Studies and Robustness Checks

• Cross-layer top-k index overlap quantification (Appendix A, Figure 4): The pairwise overlap heatmap for the 47-layer 30B DSA model confirms that adjacent layers share 70–100% of their top-2048 selected tokens, averaged over 768 samples of 200K-token length. The heatmap reveals block structure (e.g., layers 17–30, 31–36 form internally coherent clusters) and an early-late distinction (overlap ≤ 0.4 between early and late layers). The paper notes that the greedy-searched sharing blocks (marked as red boxes) "do not fully coincide" with the natural overlap clusters, because overlap counts which tokens are shared but not how critical they are. This mismatch is the empirical justification for why end-to-end loss-based search is necessary: local similarity metrics miss the cascading effects of missing critical tokens on downstream layers.

• Similarity-based pattern search failure (Appendix C, Table 5): A dynamic programming approach that maximizes per-layer attention output cosine similarity when inheriting indices from a Full layer produces patterns that perform no better than uniform interleaving. At 1/2 retention, the similarity-searched pattern achieves Long Avg 49.8 vs. 50.7 for uniform interleaving and 50.3 for the loss-based greedy pattern (Table 2 comparison). The paper diagnoses the failure as a limitation of local metrics: "even a small set of mismatched critical tokens can perturb a layer's hidden state in ways that cascade through all downstream layers," and "two layers may have nearly identical attention outputs yet differ in subtle ways that matter for downstream quality." This negative result is explicitly positioned as a cautionary tale that justifies the more expensive loss-based search.

• Cross-layer distillation loss ablation (Table 3, "w/o cross-layer loss" row): Training IndexCache with each retained indexer distilled only against its own layer's attention (rather than the multi-layer averaged target of all layers it serves) drops Long Avg from 51.6 to 49.8 (−1.8 points) at 1/2 retention. The largest individual drop is AA-LCR: 49.8 → 44.0 (−5.8 points). This directly quantifies the benefit of the multi-layer distillation loss and confirms Proposition 1's practical significance: distilling against the centroid of served layers' distributions (which the multi-layer loss implicitly does) produces indexers that generalize better to the sharing setting than layer-specific distillation. The fact that even without the cross-layer loss, the training-aware model still outperforms the training-free uniform baseline at 1/4 (49.8 vs. 43.0) suggests that the sparse training phase alone provides partial adaptation — the Shared layers learn to some extent to work with inherited indices — but the multi-layer distillation provides substantial additional benefit.

• Retention ratio sweep and loss curve shape (Section 3.1.2 inline figure, Table 2): The per-step LM validation loss curve during greedy search reveals a "clear separation between 'easy' layers (the first 20 steps) and 'critical' layers (after 35 steps)." This elbow at approximately 20–35 converted layers (out of 47) corresponds to retention ratios between ~0.57 (27/47 Full) and ~0.26 (12/47 Full). The 1/4 retention point (12 Full layers) falls near the start of the critical regime, while 1/8 retention (6 Full layers) is deep within it. This explains why 1/4 retention with search achieves near-baseline quality (49.9 vs. 50.2) while 1/8 retention even with search degrades notably (46.1 vs. 50.2): 1/4 retention preserves the model just before the steep loss increase, while 1/8 retention removes layers deep in the critical regime where each additional conversion incurs substantial loss increase.

• Pipeline parallelism acceleration of greedy search (Section 3.1.2): The paper reports that splitting the model into $P$ pipeline stages and searching blocks sequentially allows up to $P$ layers to be converted per step, reducing total forward passes by roughly $P\times$. The paper does not report the specific value of $P$ or the wall-clock time required for a full search, which is a practical omission — practitioners need to know whether the search is minutes, hours, or days on standard hardware. Given the 30B model size and a context length of 200K, even $P=4–8$ would still require hundreds of forward passes, each processing 768 samples of 200K tokens, making this a substantial one-time cost. The paper implicitly assumes this cost is amortized over the model's deployment lifetime.

• Stability across calibration sets (Section 3.1.2): The paper states that the greedy search "results are stable across different calibration sets, indicating that this importance ranking is an intrinsic model property rather than a data artifact." No quantitative metric of stability (e.g., overlap between search results from different seeds, variance in Long Avg across calibration sets) is reported. This is a claim that would benefit from specific evidence: what fraction of the selected F layers are identical across different calibration sets? How much does the final Long Avg vary? The qualitative statement is reassuring but leaves room for future work to characterize the robustness more precisely.

• Low-cost of T_cache memory (Section 3, Figure 2 caption): The paper explicitly states that T_cache "requires no additional GPU memory beyond what standard DSA already allocates" because standard DSA already allocates memory for each layer's index tensor, and IndexCache simply repurposes the memory that would have been allocated for the removed indexers. This is an important robustness point: IndexCache's speedup does not come at the cost of increased memory pressure, which would be a significant concern for long-context serving where KV cache memory is already the primary bottleneck. The paper does not benchmark the memory footprint explicitly but the design (one shared buffer per layer, overwritten at each F layer) makes this claim credible.

• General and reasoning benchmark robustness across configurations: Looking across Tables 2, 3, and 4, the General & Reasoning Avg is consistently more robust to indexer removal than Long Avg. In Table 2 at 1/4 uniform interleaving, Long Avg drops by 7.2 points while G&R Avg drops by only 0.8 points (74.6 → 73.8). At 1/8 uniform, Long Avg drops 14.9 points vs. 4.6 points for G&R Avg. This pattern holds in Table 4 on GLM-5: at 1/4 uniform, Long Avg drops 5.7 points (78.4 → 72.7) while the general benchmarks in Figure 1 remain "nearly identical." This suggests that the reasoning benchmarks (AIME, GPQA, LiveCodeBench, IFBench) rely more on the model's learned knowledge and reasoning procedures, which are robust to which specific tokens are attended to, while the long-context benchmarks (especially MRCR v2, GraphWalks) require precise token-level retrieval that is sensitive to indexer mismatch. The paper does not discuss this differential sensitivity explicitly, but it is a consistent pattern across all reported experiments.

Critical Assessment

The experiments presented in Section 4 and the appendices provide substantial evidence for IndexCache's primary claims, but several limitations in the experimental design affect the scope and certainty of those claims.

Claim: IndexCache removes 75% of indexer computations with negligible quality degradation.

Evidence: Table 2, 1/4 retention with searched pattern: Long Avg 49.9 vs. 50.2 baseline (−0.3 points), G&R Avg 74.9 vs. 74.6 (+0.3). Table 3, 1/4 uniform interleaving with training-aware: Long Avg 50.6 vs. 51.0 baseline (−0.4), G&R Avg 74.1 vs. 74.2 (−0.1).

Assessment: The claim is well-supported for the 30B model at 1/4 retention under both the training-free (with greedy search) and training-aware (with uniform pattern) approaches. The degradation is genuinely negligible — within the noise floor of benchmark evaluation. However, the claim's generality is constrained by several factors:

The 1/4 retention claim holds for the searched pattern but not for uniform interleaving in the training-free setting. The difference is stark: 49.9 vs. 43.0 Long Avg. This means practitioners who cannot afford the greedy search (large models with limited calibration data) cannot simply use a uniform 1/4 pattern and expect negligible degradation — they must either invest in the search or accept quality loss. The paper's messaging that "75% of indexer computations can be removed" is accurate but the which 75% matters enormously, and the paper provides the tooling (greedy search) to determine this at a one-time cost that is not benchmarked for wall-clock time.

At 1/8 retention, the claim breaks down. Even with the searched pattern, Long Avg drops to 46.1 vs. 50.2 baseline (−4.1 points). This is not "negligible" by any reasonable standard. The paper is candid about this: the loss curve's elbow marks ~1/4 retention as the practical limit of the training-free approach. The 1/8 retention results are included to demonstrate where the method fails, not to claim it works, which is methodologically honest and useful for practitioners. The training-aware approach was not tested at 1/8, leaving open the question of whether retraining could push the feasible retention ratio further.

The "negligible" characterization depends on benchmark aggregation. Looking at individual benchmarks at 1/4 with searched pattern: GraphWalks is at 47.4 vs. 49.6 (−2.2 points), which some practitioners might consider non-negligible depending on their application. AA-LCR at 43.8 vs. 43.6 (+0.2) and MRCR v2 at 25.1 vs. 24.5 (+0.6) actually improve. The averaging into Long Avg masks this heterogeneity — if a downstream application depends specifically on GraphWalks-type reasoning, the 2.2-point drop may be more concerning than the Long Avg suggests. The paper does not break down the variance or provide per-benchmark confidence intervals, making it difficult to assess whether these per-benchmark differences are statistically significant or within sampling noise.

Claim: IndexCache delivers up to 1.82× prefill speedup and 1.48× decode speedup.

Evidence: Table 1, Figure 3. At 200K context length, 1/4 retention: prefill 19.5s → 10.7s (1.82×), decode 58.0 → 86.0 tok/s (1.48×).

Assessment: These speedup numbers are derived from wall-clock measurements on actual hardware (NVIDIA H100 node, 8-way dp, SGLang) and represent genuine inference acceleration. Several contextual factors affect their interpretation:

The speedup is context-length-dependent. At 10K context, the prefill speedup is only 1.27× (at 1/4 retention), because the indexer accounts for a smaller fraction of total compute (27% at 10K vs. 81% at 200K). The paper appropriately emphasizes the long-context regime because that is where IndexCache's value proposition is strongest, but a system serving mixed context lengths would see an average speedup weighted by the context length distribution. If the majority of requests are short (<10K tokens), the effective speedup would be substantially lower than the headline 1.82×.

The decode speedup under full KV cache saturation is 1.51× at 200K (197 → 297 tok/s). This is the metric most relevant to production throughput (tokens served per second per GPU). The gap between per-request decode speedup (1.48×) and full-throughput decode speedup (1.51×) is small, suggesting that the indexer bottleneck is similarly severe under low and high concurrency. This is expected because the indexer cost is per-token and per-layer regardless of batching.

The speedup numbers are for the 30B model specifically. The 744B GLM-5 results are described as "at least 1.3× improvement in both prefill latency and decode throughput at context lengths beyond 100K" at 1/4 retention. The lower speedup (1.3× vs. 1.48–1.82×) for the larger model may reflect differences in the indexer's share of total compute: larger models have more FFN and attention parameters relative to the indexer, so the indexer contributes a smaller fraction of total latency even at long contexts. The paper does not provide a detailed latency breakdown for GLM-5, so this cannot be verified. The 1.2× end-to-end speedup for GLM-5 at 1/2 retention (Figure 1) is lower than the 1.3× for 1/4 at long contexts, which is expected since fewer indexers are removed.

The speedups are measured without accounting for the difficulty estimation cost of pattern search. In the training-free setting, the greedy search requires significant one-time computation that is not amortized into the reported speedup numbers. This is defensible — the search is done once offline and the pattern is then applied to all future inference — but the one-time cost should be quantified for practitioners evaluating whether to adopt the method. For a 744B model, the search cost could be substantial (hundreds of forward passes at 200K context length), and the paper does not provide guidance on how this scales.

Claim: Training-aware IndexCache enables uniform patterns to match full-indexer accuracy.

Evidence: Table 3: uniform interleaving at 1/2 achieves Long Avg 51.6 vs. 51.0 DSA baseline; at 1/4 achieves 50.6 vs. 51.0.

Assessment: This claim is well-supported at the tested retention ratios and for the specific training recipe used. However, the training-aware experiments have a significant limitation that the paper acknowledges:

The training pipeline is shortened (5,000 total steps vs. full DSA training). The DSA baseline for the training-aware experiments was also trained under this shortened pipeline, so the internal comparison (IndexCache vs. DSA in Table 3) is fair. But the absolute performance numbers differ from the fully-trained DSA in Table 2 (51.0 vs. 50.2 Long Avg), making it difficult to directly compare the training-free and training-aware approaches. The paper says the shortened pipeline "closely matches the performance of full DSA training," but the 0.8-point Long Avg difference suggests there may be residual quality on the table that full training would capture. Whether the training-aware IndexCache would maintain its advantage over the training-free approach under full training is not tested.

The training-aware approach was only evaluated with uniform interleaving. The paper does not explore whether training with the multi-layer distillation loss could enable even more aggressive sharing patterns (e.g., 1/8 retention) that are infeasible in the training-free setting. Given that training eliminates the pattern sensitivity, one might hypothesize that the feasible retention ratio could push beyond 1/4. This is a missed opportunity to characterize the asymptotic limit of the training-aware approach.

The training-aware approach requires access to the full training pipeline. For models that are obtained as pre-trained checkpoints without training infrastructure (the majority of open-source model users), training-aware IndexCache is not applicable. The paper positions this appropriately — as a method for model developers who train from scratch or do continued pre-training — but the practical addressable audience for the training-aware variant is narrower than for the training-free variant.

Claim: Cross-layer index redundancy extends to production-scale models (GLM-5).

Evidence: Table 4, Figure 1. At 1/4 with searched pattern: Long Avg 78.0 vs. 78.4 baseline (−0.4). Figure 1 shows "nearly identical" performance on the Artificial Analysis Index at 1/2 retention.

Assessment: This is preliminary evidence rather than a full validation. Only five long-context benchmarks are reported for GLM-5 (Table 4), and only at 1/2 and 1/4 retention with training-free IndexCache. The training-aware approach is explicitly deferred: "We plan to apply training-aware IndexCache to this production-scale model in the near future." The general and reasoning benchmarks are only shown in Figure 1 for the 1/2 retention case, not tabulated with specific numbers. The paper does not report inference speedup breakdowns for GLM-5 (prefill, decode, full throughput) comparable to Table 1 — only the summary "~1.2× end-to-end speedup" for 1/2 retention and "at least 1.3×" for 1/4 at >100K context. This limits the ability to assess whether the scaling trends observed on the 30B model (growing speedup with context length, larger speedup for prefill than decode) hold at 744B scale.

Missing Experiments and Baselines

Several experiments would strengthen the paper's claims but are absent:

No combination of IndexCache with other inference optimizations. DSA models in production are likely served with additional optimizations — quantization (the indexer already uses FP8, but core attention may use INT4/INT8), speculative decoding, continuous batching, KV cache compression. IndexCache reduces indexer computation specifically, and its interaction with these other optimizations is unexplored. For instance, if KV cache compression (like MiniCache or SwiftKV) is also applied, does the indexer's share of total compute increase (because core attention becomes cheaper), making IndexCache's relative benefit larger? Or do these optimizations compound multiplicatively? The paper's isolated benchmarking is appropriate for a first paper but leaves integration questions unanswered.

No latency breakdown at varying batch sizes. The prefill and decode speedups are reported for single-concurrency (per-request) and full-KV-cache saturation scenarios. Real serving systems operate at varying batch sizes where the tradeoff between indexer cost and core attention cost may shift — the indexer's cost scales linearly with batch size during prefill (because each sequence's indexer runs independently), while core attention's cost scaling depends on the attention implementation. A sweep over batch sizes would characterize the operating regimes where IndexCache provides the greatest benefit.

No comparison to alternative indexer reduction methods. The paper frames indexer reduction as a novel problem and positions IndexCache as the first solution. This is fair, but there are alternative approaches one could imagine: reducing the indexer's precision below FP8, reducing the number of indexer heads, or using a smaller $k$ for the indexer and a larger $k$ for core attention. These are not directly comparable because they trade off quality for speed rather than exploiting redundancy, but a brief comparison would contextualize IndexCache's efficiency-quality tradeoff against these simpler alternatives.

No sensitivity analysis for the calibration set size. The paper uses a batch size of 768 with a context length of 200K for the greedy search. How much does the search quality depend on the number of batches or the context length of the calibration data? Could a smaller calibration set (e.g., 100 samples at 50K context) produce patterns of similar quality? This is practically important because the calibration set construction cost may dominate the search cost for some deployment scenarios.

No comparison of the greedy search against random search. The paper compares greedy search to uniform interleaving and similarity-based selection, but not to random search (randomly selecting which layers to retain given a fixed number of Full layers). A random search baseline, even with a small number of trials (e.g., 10 random patterns, pick the one with lowest LM loss), would help quantify how much of the greedy search's benefit comes from the greedy selection strategy versus simply having some non-uniform pattern. If random search achieves 48.0 Long Avg at 1/4 (vs. 49.9 for greedy and 43.0 for uniform), that would suggest the gain is primarily from avoiding uniform patterns; if random search achieves 45.0, the greedy algorithm's informed selection is providing substantial additional value.

Limited scale diversity in the 30B experiments. All training-free experiments use a single 30B MoE model with 47 layers. The cross-layer overlap properties (Appendix A, Figure 4) — block structure, early-late distinction, sharp transition boundaries — may depend on model depth, width, and architecture. A shallower model (e.g., 12 layers) may have less redundancy to exploit. A denser model (without MoE) may have different attention dynamics. The GLM-5 experiments partially address scale diversity but use only training-free IndexCache and a subset of benchmarks. A systematic study across model scales and architectures would strengthen the claim that cross-layer index redundancy is a universal property.

Where the Claims Hold and Where They Don't

The "negligible quality degradation" claim holds when:

• The retention ratio is ≥ 1/4 (for training-free with greedy search) or ≥ 1/4 (for training-aware with uniform interleaving).
• The evaluation is at 200K context length (the training and calibration context length). The paper does not test whether patterns optimized at 200K generalize to shorter or longer contexts — this is a notable gap given that the indexer's behavior may differ at different context lengths (the cross-layer overlap may not be invariant to $L$).
• The model is a DSA model with a similar architecture (MLA, MoE, 47 layers). The paper does not test on dense DSA models or DSA models with different numbers of layers.

The "negligible quality degradation" claim weakens or fails when:

• The retention ratio drops to 1/8 (training-free): Long Avg drops 4.1 points even with search.
• Uniform interleaving is used without greedy search (training-free) at 1/4: Long Avg drops 7.2 points.
• The specific benchmark is highly sensitive to token-level selection precision (GraphWalks, MRCR v2), where even searched patterns at 1/4 retention lose 1–2 points.

The inference speedup claims hold when:

• The context length is long (>60K) so the indexer dominates total compute.
• The model size and architecture are such that the indexer accounts for a substantial fraction of latency. For very large models where FFN and attention dominate at all context lengths, the absolute speedup from indexer reduction would be smaller. The GLM-5 results (1.3× vs. 1.48–1.82×) already suggest this scaling effect.
• The application is latency-sensitive for prefill (time-to-first-token) or throughput-sensitive for decode. IndexCache does not improve output quality or change the model's capabilities — it is purely an efficiency optimization.

The training-aware approach's advantage over training-free holds when:

• Training infrastructure and the base model are available (continued pre-training is feasible).
• The sharing pattern is known before training begins (to set up the multi-layer distillation groups). The paper does not explore whether a model trained with one pattern (e.g., F S S S F S S S) can be switched to a different pattern at inference time without quality loss. If the adaptivity is pattern-specific, the training-aware model loses the flexibility to adjust retention ratio post-training, which the training-free model retains.

Overall, the experimental section provides thorough evidence for the paper's core claims within the scope tested (30B DSA model, 200K context, 1/4 retention with search, specific benchmarks). The GLM-5 results provide encouraging scalability evidence but are preliminary. The main experimental gaps are the lack of multi-scale architecture testing, the absence of integration with complementary inference optimizations, the missing calibration set sensitivity analysis, and the deferred training-aware large-scale validation. The paper's methodological strength is its inclusion of negative results (similarity-based search failure, uniform interleaving degradation, the 1/8 retention limit) that clearly delineate where the method works and where it doesn't, providing practitioners with actionable guidance for adoption.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Remains External to All Reported Efficiency Gains

The assumption or constraint. The training-free greedy search that determines which layers retain indexers requires O(N^2/P) forward passes over a calibration set of 768 samples at 200K context length (Section 3.1.2). This one-time cost is not amortized into any of the reported speedup numbers (Table 1, Figure 3), meaning the headline 1.82× prefill speedup and 1.48× decode speedup are measured after the pattern has already been discovered and do not account for the resources expended to find it. The paper explicitly separates this cost from inference:

"our experiments do not account for this cost largely for simplicity" (extending the analogous statement about difficulty estimation from Section 3.2 — here applied to the pattern search that serves the same function for training-free IndexCache).

The consequence. For a practitioner deploying a frozen DSA model, the total cost of adopting training-free IndexCache is pattern search cost + (inference cost × number of inferences). The amortization breakeven point — how many inference requests are needed before the per-inference savings exceed the one-time search cost — is never characterized. For a 30B model with 47 layers at 200K context, the greedy search requires hundreds of forward passes, each processing 768 sequences of 200K tokens. The computational cost of this search likely exceeds the cost of serving many thousands of inference requests without IndexCache. If the model is deployed in a low-volume setting or if the pattern must be re-computed for different context length regimes or data distributions, the one-time cost may dominate any inference-time savings.

Furthermore, for models where pipeline parallelism is limited (small models, single-GPU deployments), the acceleration factor P is small, and the search cost grows toward the full O(N^2) bound. The paper reports that pipeline parallelism reduces forward passes "by roughly P×" (Section 3.1.2) but provides no absolute wall-clock times or FLOP counts for the search, making it impossible for practitioners to estimate the search cost for their specific hardware and model configuration.

What evidence exists in the paper. The paper never benchmarks the greedy search cost — no wall-clock time, FLOP count, GPU-hours, or dollar cost is reported for any model scale. The stability claim ("results are stable across different calibration sets," Section 3.1.2) is stated qualitatively without quantitative overlap metrics, so it is unclear whether a smaller calibration set (fewer batches, shorter context) would produce comparable patterns, which would directly reduce the search cost. The similarity-based approach (Appendix C) is computationally cheaper (N forward passes to build the matrix vs. O(N^2/P) for greedy search) but is shown to produce patterns that perform no better than uniform interleaving — meaning the paper rules out the cheap alternative without providing a cost analysis of the expensive one that works.

Mitigation status. The paper does not address search cost quantification or amortization analysis. The training-aware approach (Section 3.2) partially sidesteps this limitation by eliminating the need for pattern search entirely — uniform interleaving suffices after retraining — but training-aware IndexCache imposes its own cost (continued pre-training, Section 4.1) that is far larger than the greedy search. For the training-free setting, which is the primary use case for practitioners with frozen models, the search cost remains an unquantified barrier to adoption. The paper suggests future work on direct difficulty estimation from question text (extending Section 3.2's analogy) but provides no concrete direction for cheap pattern estimation.

---

The Training-Free and Training-Aware Approaches Are Never Both Applicable Simultaneously, Creating a Forced Choice With No Clear Selection Criteria

The assumption or constraint. IndexCache presents two configuration methods as complementary — training-free for frozen models, training-aware for models under development — but the paper provides no guidance on which to use when both are technically possible. A model developer who controls the training pipeline faces a genuine decision: invest the one-time search cost to find a pattern for an already-trained model, or invest the much larger training cost to produce a model that works with simple uniform patterns? The paper evaluates these approaches on different models (training-free on a fully-trained DSA model in Table 2, training-aware on a shortened-training-pipeline DSA model in Table 3) with different baselines (Long Avg 50.2 vs. 51.0), making direct comparison impossible.

The consequence. The approaches impose fundamentally different deployment constraints. Training-free IndexCache produces a pattern that is tied to a specific model checkpoint and must be re-computed if the model is updated, but allows flexible adjustment of the retention ratio post-hoc (you can run the greedy search again for a different K). Training-aware IndexCache bakes the sharing structure into the model weights — once trained, the model presumably works best with the pattern used during training. If a practitioner wants to deploy at different retention ratios for different use cases (e.g., 1/2 for high-quality tasks, 1/4 for high-throughput tasks), a training-aware model trained with one pattern may not support this flexibility, while a training-free model can have patterns searched separately for each ratio. The paper does not test whether a training-aware model trained with a 1/4 uniform pattern degrades when deployed with a 1/2 uniform pattern or a searched pattern.

Additionally, the training-aware experiments use a shortened training pipeline (1,000 warm-up + 4,000 sparse training steps, Section 4.1) that "closely matches the performance of full DSA training." The paper does not verify that the training-aware advantage — uniform patterns matching full-indexer quality — persists under full-length training. It is possible that with more training steps, the layer-specific indexer specialization that makes uniform patterns fragile in the training-free setting would re-emerge, reducing the training-aware approach's advantage.

What evidence exists in the paper. Tables 2 and 3 report on different DSA baselines with different absolute performance (50.2 vs. 51.0 Long Avg), preventing direct head-to-head comparison of training-free searched patterns versus training-aware uniform patterns. The w/o cross-layer loss ablation in Table 3 is the closest comparison point — it shows that training each indexer only against its own layer (simulating the training-free scenario but with retraining) produces Long Avg 49.8 at 1/2 retention with uniform interleaving, versus 51.6 with the cross-layer loss. But this ablation uses the same shortened training pipeline as the training-aware model, so it does not represent the fully-trained model used in Table 2, and the comparison to the training-free searched pattern (50.3 Long Avg at 1/2) is across different baselines.

Mitigation status. The paper acknowledges the training-aware approach as future work for GLM-5 (Section 4.5: "We plan to apply training-aware IndexCache to this production-scale model in the near future") but does not address the deployment flexibility tradeoff or provide criteria for choosing between the two approaches. The question of whether a training-aware model's robustness to sharing patterns generalizes across retention ratios and pattern geometries is entirely unexplored.

---

All Results Are on a Single Model Architecture (MLA-Based MoE) With a Single Sparse Attention Mechanism (DSA), Leaving Generality Unverified

The assumption or constraint. Every experiment in the paper — 30B results in Tables 1–3, GLM-5 results in Table 4 and Figure 1 — uses models with Multi-head Latent Attention (MLA) and Mixture-of-Experts (MoE) architecture, with DSA as the sparse attention mechanism. The paper claims in Section 5.2 that IndexCache's "core principle extends to any sparse attention method that does not rely on a fixed sparse pattern but rather involves a dynamic token selection step," and explicitly mentions MoBA (Lu et al., 2025) and NSA (Yuan et al., 2025) as candidates, but provides no experimental evidence for any architecture other than DSA.

The consequence. The paper's central empirical claim — that cross-layer index redundancy enables removing 75% of indexers with negligible quality loss — may not transfer to other sparse attention mechanisms for several reasons. DSA's lightning indexer is explicitly designed as a lightweight approximation to full attention, trained via distillation. Its outputs may exhibit more cross-layer stability than other selection mechanisms precisely because it is trained to mimic a smooth, layer-consistent full-attention target. Alternative mechanisms with different design choices — MoBA's block-level routing, NSA's hardware-aligned selection patterns — may have fundamentally different cross-layer dynamics. A block-level router that makes coarse-grained decisions (entire blocks of tokens selected or not) might exhibit more stability across layers (because block boundaries are coarser), enabling even more aggressive sharing, or less stability (because block-level granularity amplifies the impact of selection differences when they do occur). Neither direction is tested.

The architecture-specificity also applies to the MLA backbone. MLA compresses key-value representations through low-rank projections, which may make attention distributions smoother and more layer-consistent than standard multi-head attention. The cross-layer top-k overlap heatmap (Appendix A, Figure 4) is specific to the 30B MLA-based DSA model; models with standard attention or other efficient attention variants (grouped-query attention, multi-query attention) may exhibit different overlap patterns, different block structures, and different sensitivity to indexer removal.

What evidence exists in the paper. The paper provides zero experiments on non-DSA sparse attention, non-MLA architectures, or even non-MoE DSA models. The sweep across model scales (30B → 744B) tests scalability within the same architectural family but does not test generality across families. The similarity-based approach failure (Appendix C) — where a method that should work in theory fails in practice due to cascading error propagation — serves as a cautionary parallel: cross-layer redundancy may exist in other architectures but the pattern of sensitivity (which layers are critical) is likely architecture-specific, and the greedy search methodology may need to be validated independently for each new architecture.

Mitigation status. The paper acknowledges the scope limitation implicitly by couching its generality claim as an expectation ("could similarly benefit," Section 5.2) rather than a demonstrated result. The extension to MoBA and NSA is mentioned but not pursued. The broader claim that "cross-layer index reuse [will] become a standard component of efficient inference pipelines" (Section 6) is aspirational and depends on validation that is not provided here.

---

The Method's Performance Depends on Context Length in Ways That Are Only Partially Characterized

The assumption or constraint. All pattern searches and training-aware training are conducted at a fixed context length of 200K tokens (Section 4.1: "a context length of 200K" for both the calibration set and training data). The benchmark evaluations also use a 200K-token context window (Appendix D). However, the inference speedup results (Table 1, Figure 3) are reported at four different context lengths (10K, 60K, 120K, 200K), and the paper acknowledges that the indexer's share of total compute — and therefore IndexCache's benefit — varies dramatically with context length (27% of prefill time at 10K vs. 81% at 200K). This creates a tension: the pattern is optimized at 200K, but the inference may occur at any context length.

The consequence. The cross-layer top-k overlap that justifies index reuse is measured at 200K context (Appendix A). But the attention distributions that produce this overlap may change with context length. At short contexts (e.g., 4K tokens), the model may attend to a broader fraction of the available tokens, making the top-k selection less stable across layers because more positions are near the selection boundary. At extremely long contexts beyond 200K, attention may become more concentrated on a smaller fraction of tokens, potentially making the overlap higher — but the indices that are shared may be optimized for a 200K attention distribution that differs from the 500K or 1M distribution. The greedy-searched pattern determined at 200K may be suboptimal at 10K or 500K.

This matters practically because production serving systems handle requests at widely varying context lengths. If the pattern is fixed (as the paper assumes), a system that serves a mix of short and long requests may see degraded quality on short requests where the 200K-optimized pattern is mismatched to the actual attention dynamics. Alternatively, a system could use different patterns for different context length regimes, but this would multiply the already-unquantified pattern search cost by the number of regimes and add complexity to the serving infrastructure.

What evidence exists in the paper. Table 1 reports inference speedups across context lengths but does not re-evaluate downstream benchmark quality at context lengths other than 200K. RULER (the one benchmark that explicitly varies context length from 4K to 128K, Appendix D) could provide indirect evidence — if the 200K-optimized pattern degrades disproportionately at shorter lengths, this would appear in the RULER scores — but the paper reports only aggregate RULER scores (Tables 2–4) without breaking them down by context length. The 1/4 searched pattern achieves RULER 87.6 vs. 87.9 DSA baseline (Table 2), suggesting aggregate RULER performance is preserved, but this does not rule out length-dependent degradation that averages out.

The training-aware approach may be less sensitive to this issue because the multi-layer distillation trains each indexer against attention distributions from all served layers at the training context length, and the resulting consensus distribution may be more robust to context length changes. However, this is speculation — the paper does not test training-aware IndexCache at contexts other than the training length.

Mitigation status. The paper does not address context length generalization. The extrapolation statement in Section 4.2 — "extrapolating to longer contexts (>200K), IndexCache is expected to deliver even greater speedups" — refers only to the speedup trend from reducing indexer cost share, not to whether the pattern or the model quality remains valid at those lengths. The absence of length-dependent quality evaluation is a significant gap for a method whose primary value proposition is at long contexts, given that the pattern optimization is conducted at a single point in context-length space.

---

Hard Problems at Extreme Sharing Ratios Remain Unsolved, and the Performance Cliff at 1/8 Retention Reveals a Fundamental Limit of the Frozen-Model Approach

The assumption or constraint. The training-free greedy search produces patterns that recover near-baseline quality at 1/4 retention, but at 1/8 retention — retaining only 6 of 47 indexers — even the searched pattern drops Long Avg from 50.2 to 46.1 (Table 2), a "non-negligible" 4.1-point decline that the paper acknowledges as a failure. The per-step loss curve in Section 3.1.2 shows a sharp "elbow" after approximately 35 layers are converted to Shared, corresponding roughly to the 1/4 retention threshold, beyond which each additional indexer removal causes rapidly increasing loss.

The consequence. This elbow represents a hard capability bound for the training-free approach: there exists a subset of layers whose indexers are genuinely necessary in the frozen model, and no pattern search — greedy, exhaustive, or otherwise — can circumvent the fact that when these indexers are removed, the model loses access to critical token selection information that cannot be reconstructed from cached indices. This bound is an intrinsic property of the trained model's layer-specific indexer specialization, not an artifact of the search strategy.

For practitioners, this means there is no path to removing more than ~75% of indexers without either accepting significant quality degradation or retraining the model (training-aware IndexCache). The training-aware approach was not tested at 1/8 retention, so it is unknown whether retraining can push the feasible retention ratio further — the elbow may shift but likely does not disappear entirely, because some degree of layer-specific attention focus is presumably necessary for the model's representational capacity. If the elbow persists under training-aware training, it would establish a fundamental limit on cross-layer index reuse for any approach, training-aware or not.

The pattern of which benchmarks degrade at 1/8 is also instructive. GraphWalks drops from 49.6 to 43.8 even with the searched pattern (−5.8 points); MRCR v2 drops from 24.5 to 21.7 (−2.8 points); LiveCodeBench v6 drops from 71.4 to 69.6 (−1.8 points, Table 2). The long-context retrieval and graph reasoning tasks that require precise token-level attention suffer disproportionately, while instruction following (IFBench: 58.4 → 58.1) is essentially unaffected. This differential sensitivity means the 1/8 failure mode is not uniform — if a deployment primarily serves tasks like IFBench (instruction following), 1/8 retention might be acceptable; if it serves GraphWalks-style reasoning, it is not. The paper does not provide task-specific guidance for setting retention ratios.

What evidence exists in the paper. Table 2 provides the full 1/8 results; the loss curve figure in Section 3.1.2 visualizes the elbow that makes 1/8 infeasible. The paper is transparent that this ratio fails, stating that at 1/8 "the resulting decline in long-context performance at this extreme sparsity becomes non-negligible." The training-aware experiments (Table 3) stop at 1/4 retention, leaving open the question of whether the training-aware approach can extend the feasible range. The GLM-5 results (Table 4) show the same pattern — 1/4 searched pattern preserves quality (78.0 vs. 78.4), suggesting the elbow exists at similar relative retention ratios across model scales — but no 1/8 results are reported for GLM-5.

Mitigation status. The paper treats 1/8 as a failed configuration rather than an open problem to be solved. The training-aware approach is proposed as a potential path to pushing the retention ratio further but is not tested at 1/8. The task-specific sensitivity to extreme sharing — which benchmarks degrade, which are robust — is not characterized systematically beyond the individual numbers in Table 2. A practitioner who needs to push beyond 1/4 retention receives no guidance on whether the training-aware approach can help, or whether certain task families should be served with different retention ratios.

---

Sequential Dependency of Shared Layers Creates an Implicit Accuracy-Latency Tradeoff That Is Not Quantified

The assumption or constraint. IndexCache's design dictates that each Shared layer inherits indices from the nearest preceding Full layer (Section 3: src(ℓ) = max{j < ℓ: c_j = F}). This means the number of consecutive Shared layers between Full layers determines the maximum "staleness" of cached indices — the further a Shared layer is from its Full-layer anchor, the more the model's hidden state has evolved since the indices were computed, and the less accurate those indices may be for the Shared layer's current attention needs. Uniform interleaving (e.g., F S S S F S S S) creates fixed staleness; searched patterns create variable staleness. The paper implicitly treats staleness as a binary property (acceptable if the greedy search selected that layer as Shared, unacceptable otherwise) rather than a continuous degradation.

The consequence. The staleness-distance relationship implies a tradeoff between indexer elimination (which requires long runs of S layers) and attention fidelity (which benefits from frequent F layers to refresh indices with current hidden states). The paper's results suggest this tradeoff is manageable at 1/4 retention — the searched patterns place F layers at intervals that keep staleness within acceptable bounds — but the tradeoff is never explicitly characterized. A practitioner who needs to balance throughput (more S layers) against quality (more F layers) cannot use the paper to predict how quality degrades as a function of maximum consecutive S layers rather than total number of S layers, even though these are correlated but distinct design parameters.

This also interacts with the context-length generalization issue (discussed in a separate limitation): at longer contexts, attention distributions may shift more slowly across layers (because the most relevant tokens remain relevant for more consecutive layers), making staleness less harmful. At shorter contexts, attention may shift more rapidly, making the same pattern produce worse results because the cached indices are stale by the time they reach later Shared layers in a block. The paper provides no analysis of whether the searched patterns' block lengths (number of consecutive S layers) correlate with benchmark degradation, or whether patterns with shorter maximum block length perform systematically better.

What evidence exists in the paper. The searched patterns for the 30B model are listed in Appendix B and contain variable block lengths. For the 1/4 retention pattern (FSFSFSSSSFSSSFSSFFSSFSSFSSSSFSSSFSSSSFSSSSSSSSS), the runs of consecutive S layers include blocks of 4, 3, 3, 2, 4, 3, and 9 S layers. The final block of 9 consecutive S layers (layers 39–47 are mostly S, with one F at position 40) is particularly long and may be responsible for a disproportionate share of the residual degradation at 1/4. The paper does not analyze whether breaking long S-runs into shorter blocks by inserting additional F layers would improve quality, which would directly test the staleness hypothesis. The training-aware approach's robustness to uniform patterns (fixed staleness of 3 at 1/4) suggests that retraining eliminates the staleness sensitivity, but this is confounded with the multi-layer distillation objective.

Mitigation status. The paper does not discuss staleness, maximum consecutive S layers, or the block-length-quality tradeoff. The greedy search implicitly optimizes over this tradeoff (since it can place F layers to break long S-runs if that improves LM loss), but the mechanism is not analyzed or explained. A controlled experiment varying maximum block length while holding the total number of S layers constant would directly characterize this tradeoff and provide actionable guidance for pattern design, but it is absent.

7. Implications and Future Directions

How This Work Changes the Landscape

IndexCache is not a paradigm shift in the sense of introducing a new architecture or attention mechanism — it is a reframing of the inference bottleneck in sparse attention that has both immediate practical consequences and conceptual implications for how the field should design future sparse models.

Reframing the bottleneck from core attention to the selection mechanism. The dominant narrative in sparse attention research has been that the core attention computation is the enemy — reduce quadratic attention to linear-or-better, and inference becomes fast. DSA was the flagship success of this narrative: it reduced core attention to $O(Lk)$ while preserving quality through distillation. IndexCache's reframing is to point out that this success shifts the bottleneck rather than eliminating it. The profiling data in the paper's introduction makes this concrete: at 200K context length, the indexer — the lightweight module designed to enable sparsity — consumes 81% of prefill time. The indexer was treated as acceptable overhead, a small price for making the expensive thing cheap. IndexCache demonstrates that this overhead is not small at scale, and more importantly, that most of it is redundant.

This reframing matters because it changes what problem the field should be optimizing. Before IndexCache, the research agenda for sparse attention was "make selection cheaper per-FLOP" (FP8 arithmetic, low-rank projections, fewer heads) or "make core attention sparser" (smaller $k$, dynamic $k$, hierarchical selection). After IndexCache, a new dimension opens: "the selection mechanism is redundant across layers — share it." This is a different category of optimization than per-component efficiency; it exploits a structural property of deep networks (cross-layer representation stability) rather than a computational property of individual modules (FLOP reduction). The paper's demonstration that this redundancy is strong enough to support 4× indexer reduction with negligible quality loss (Table 2: 1/4 searched pattern, Long Avg 49.9 vs. 50.2 baseline) establishes cross-layer sharing as a first-class optimization axis for sparse attention, alongside per-component efficiency and core attention sparsity.

Extending the cross-layer sharing principle beyond full-attention oracles. Prior cross-layer sharing methods — TidalDecode, Kascade, OmniKV, HySparse — all depend on full attention anchor layers as the ground-truth token selection oracle. IndexCache demonstrates that the same sharing principle works when full attention is absent entirely, using DSA's learned, approximate indexer as the shared oracle. This is not an obvious extension: one could have expected that learned indexer outputs, being approximations rather than exact attention computations, would be too noisy or too layer-specialized to support sharing. The paper's heatmap (Appendix A, Figure 4) shows otherwise — adjacent indexers share 70–100% of their top-2048 tokens — and the downstream benchmark results confirm this overlap is practically exploitable.

The conceptual implication is that the selection mechanism — whatever form it takes — is the redundant component, not the attention computation itself. This decouples the cross-layer sharing principle from the specific architecture and opens the door to applying it to any dynamic sparse attention method. The paper explicitly names MoBA's block-level routing and NSA's hardware-aligned selection as candidates, and the logic extends to any method where each layer independently selects which tokens to attend to. If this generality holds — and the paper provides only the DSA evidence, so it remains a hypothesis — then cross-layer selection sharing should become a standard design consideration for all future sparse attention architectures, not just DSA.

Reconciling training-free fragility with training-aware robustness. A subtle but important conceptual contribution is the paper's demonstration that the pattern sensitivity problem — why uniform interleaving fails catastrophically at 1/4 retention in the frozen model (Long Avg 43.0 vs. 50.2) but matches full-indexer quality after retraining (Long Avg 50.6 vs. 51.0) — is an artifact of training-inference objective mismatch, not an inherent property of cross-layer sharing. In the frozen model, each indexer was trained to serve only its own layer; reusing its output for other layers creates distributional shift that cascades through the network. After retraining with the multi-layer distillation loss, each retained indexer is trained to serve all layers in its block, and each Shared layer is trained to expect inherited indices. The mismatch disappears, and with it the sensitivity to which specific pattern is used.

This is a specific instance of a general principle with implications beyond IndexCache: many "sensitivity" problems that appear when post-hoc modifying inference behavior — pruning, quantization, KV cache compression, attention sparsification — may be resolvable not by finding cleverer post-hoc configurations but by aligning the training objective with the inference modification from the start. If you know at training time that certain layers will share indices, train them to do so; the sensitivity was never intrinsic to the modification, only to the mismatch between how the model was trained and how it is being used. The ablation removing the cross-layer distillation loss (Table 3, "w/o cross-layer loss": Long Avg drops from 51.6 to 49.8) confirms that it is specifically the multi-layer training objective, not just any retraining, that produces this robustness. This finding should encourage the field to invest in training-aware sparse architectures where the sparsity pattern is a first-class training objective rather than a post-hoc inference optimization.

Raising the importance of verifier/indexer quality as the primary bottleneck for efficient inference. A cross-cutting insight from the paper is that the quality of the selection mechanism — in DSA's case, the indexer — is what ultimately limits how aggressively it can be shared and how much speedup is achievable without quality loss. The 1/8 retention failure mode (Long Avg 46.1 even with search, Table 2) is not a failure of the sharing principle but a failure of the remaining indexers to produce indices that cover the attention needs of all their served layers. The per-step loss elbow in Section 3.1.2 — roughly 35 of 47 layers can be converted to Shared before loss increases sharply — represents the point where the retained indexers' outputs can no longer approximate the attention distributions of the layers they serve with sufficient fidelity.

This redirects attention from search algorithms (how to select which layers share) to indexer quality (how to train indexers whose outputs are robust to sharing). The training-aware approach's success in eliminating pattern sensitivity suggests that the primary lever for pushing the feasible retention ratio beyond 1/4 is better indexer training (multi-layer distillation, larger indexer capacity, more distillation data), not better pattern search. This is a practical reorientation: if a team wants to deploy a model with 1/8 or 1/16 indexer retention, their resources are better spent on improving indexer training than on developing more sophisticated search algorithms.

Making cross-layer sharing a standard component of sparse attention pipelines. The paper's closing claim — "as sparse attention becomes the default for frontier LLMs (DeepSeek-V3.2, GLM-5), we expect cross-layer index reuse to become a standard component of efficient inference pipelines" — is aspirational but grounded. The GLM-5 results (Table 4, Figure 1) demonstrate that the method works at production scale with minimal engineering complexity (one conditional branch per layer). The 1.2–1.3× end-to-end speedup at 744B parameters with "nearly identical" benchmark performance is a concrete deployment win that requires no hardware changes, no new kernel development, and no modification to the serving infrastructure beyond the inference loop. If this result replicates across other sparse attention architectures (MoBA, NSA) and model families, cross-layer selection sharing becomes a "free lunch" — a purely software-level optimization with measurable speedup and negligible downside — that is difficult for any production deployment to ignore.

Follow-Up Research This Work Enables

Characterizing the staleness-distance tradeoff with controlled block-length experiments. The paper's searched patterns contain variable numbers of consecutive Shared layers (Appendix B: the 30B 1/4 retention pattern includes runs of 2, 3, 4, and even 9 consecutive S layers), but there is no analysis of whether the length of S-runs — the "staleness" of cached indices — independently affects quality beyond the total number of indexers removed. A controlled experiment would fix the total number of Full layers (e.g., 12 F layers for 1/4 retention) and vary the distribution: one pattern with uniform spacing (F S S S F S S S ... , all blocks of length 3), another with short blocks (F S F S F S ... interspersed with longer blocks to maintain the same F count), and another with long blocks (F S S S S S S S F ... , a few blocks of length ~8). If the uniform-spacing pattern performs best at a given retention ratio, staleness is a first-order concern and pattern design should minimize maximum block length. If all patterns perform equivalently, the total number of F layers dominates and staleness is a second-order effect. The paper's existing infrastructure (greedy search with LM loss evaluation) can directly support this experiment by constraining the search to patterns with bounded maximum S-run length and comparing the resulting loss-quality curves. This would provide actionable guidance for practitioners: is it worth inserting an extra F layer to break a 9-layer S-run, or is that 9-layer run harmless if the total indexer count is low enough?

Training-aware IndexCache at 1/8 and 1/16 retention to find the fundamental sharing limit. The training-free approach hits a hard wall at 1/4 retention (Long Avg drops from 50.2 to 46.1 at 1/8 even with search, Table 2). The training-aware approach was only tested at 1/2 and 1/4 retention with uniform interleaving, where it matched or exceeded the baseline. The natural follow-up is to push the training-aware approach to 1/8 and 1/16 retention — training DSA models from scratch or via continued pre-training with the multi-layer distillation loss and uniform interleaving at these extreme ratios. If training-aware IndexCache at 1/8 achieves Long Avg within 1–2 points of the full-indexer baseline, this would establish that the training-free wall was an artifact of layer-specific indexer specialization, not a fundamental limit on cross-layer sharing. If training-aware IndexCache also hits a wall at ~1/4 retention (or perhaps at 1/8), that wall would represent a genuine limit on how much the indexer's output can be compressed across layers — the point where even a jointly-trained indexer cannot cover the attention needs of too many disparate layers. This experiment would require full DSA training pipelines at scale (30B model, 200K context length), making it computationally expensive but definitive. The paper's shortened training pipeline (5,000 steps) could be used for an initial sweep, with the best ratio validated under full training.

Cross-architecture validation on MoBA or NSA with block-level selection. The paper's strongest generality claim — that IndexCache's principle extends to "any sparse attention method that does not rely on a fixed sparse pattern" (Section 5.2) — is entirely unvalidated. A direct follow-up would apply the training-free greedy search to a MoBA model (Lu et al., 2025), which uses block-level mixture routing to select which blocks of tokens each layer attends to. MoBA's selection granularity is coarser than DSA's per-token selection (entire blocks of tokens are selected or not), which may produce higher cross-layer overlap (block-level decisions are coarser and thus more stable) or lower overlap (the block boundaries introduce quantization artifacts that amplify differences). The experiment would: (1) compute the pairwise overlap of selected blocks across layers in a trained MoBA model (analogous to Appendix A, Figure 4); (2) run the greedy search with per-token LM loss to find sharing patterns; (3) evaluate on long-context benchmarks at 1/2, 1/4, and 1/8 retention; and (4) compare the loss curve elbow position to the DSA results. If MoBA supports 1/4 retention with quality preservation, the generality claim is strengthened and the principle is validated across selection granularities. If MoBA's overlap is lower and the feasible retention ratio is correspondingly worse, this bounds the principle's applicability and identifies selection granularity as a key parameter governing cross-layer redundancy. A parallel experiment on NSA (Yuan et al., 2025) with its hardware-aligned sparse patterns would test whether dynamic (learned) selection exhibits more or less cross-layer redundancy than DSA's distillation-based selection.

Amortizing the pattern search cost through meta-learning or cheap difficulty prediction. The unquantified cost of the greedy search — hundreds of forward passes on 768 samples of 200K context — is the primary barrier to training-free IndexCache adoption. Two approaches could directly address this. First, meta-learning across model scales: run the greedy search on multiple model sizes (e.g., 1B, 7B, 30B DSA models with the same architecture) and analyze whether the selected F-layer positions are consistent across scales. If the pattern at 7B and 30B overlap by >80%, then the search can be done once on a small model and transferred to larger ones, reducing the cost by the ratio of model FLOPs. Second, cheap pattern prediction from model weights or attention statistics: train a lightweight classifier that takes per-layer features (indexer parameter norm, attention entropy on a small calibration set, position in the network) and predicts whether a layer will be in the "easy" or "critical" regime of the greedy search. The training data would be the greedy search results from a few models; the prediction target would be the order in which layers are converted to S. If such a classifier achieves >90% accuracy in predicting F-layer positions at 1/4 retention on held-out models, the pattern search cost is reduced to a single forward pass for feature extraction plus classifier inference. The paper's existing stability claim — "results are stable across different calibration sets" — provides weak evidence that the importance ranking is a structural property that a classifier could learn, but a dedicated experiment is needed to test whether this property transfers across models.

Integrating IndexCache with complementary inference optimizations to measure compound speedups. The paper benchmarks IndexCache in isolation on standard DSA serving (SGLang, 8-way dp, H100). Real production deployments combine multiple optimizations: quantization (INT4/INT8 core attention), speculative decoding, continuous batching, KV cache compression (MiniCache, SwiftKV), and prefix caching. Each of these changes the relative cost share of the indexer. Quantization makes core attention cheaper, increasing the indexer's share of total compute and thus IndexCache's relative benefit. KV cache compression reduces memory pressure but may not affect indexer cost, leaving IndexCache's absolute benefit unchanged but its relative benefit higher (since total compute decreases but the eliminated indexer compute stays the same). Speculative decoding shifts the bottleneck toward the draft model and away from the target model's attention, potentially reducing IndexCache's decode benefit. A systematic integration study on a DSA model, measuring compound speedup (IndexCache + quantization), (IndexCache + KV cache compression), and (IndexCache + speculative decoding), would provide practitioners with a complete deployment optimization roadmap. The key measurement is whether the speedups compound multiplicatively (IndexCache's 1.48× decode speedup × quantization's 1.3× = 1.92× total) or sub-multiplicatively (overlapping bottlenecks reduce the compound gain). The SGLang serving framework used in the paper already supports many of these optimizations, making this experiment engineering-heavy but feasible.

Dynamic, context-length-adaptive sharing patterns for mixed-length serving. The paper's patterns are static — the same layers are Full or Shared regardless of the input's context length. But the indexer's cost share varies dramatically with context length (27% at 10K, 81% at 200K), and the cross-layer overlap likely varies as well (attention distributions at short contexts may be less concentrated, reducing overlap and making sharing more harmful). A serving system that handles mixed context lengths could benefit from context-length-adaptive patterns: use a 1/2 retention pattern at short contexts (where the indexer is cheap and quality is more sensitive to sharing) and switch to a 1/4 or 1/8 pattern at long contexts (where indexer cost dominates and some quality degradation is acceptable in exchange for latency reduction). This requires: (1) measuring benchmark quality for patterns optimized at different context lengths and tested at different context lengths (a matrix of {pattern context} × {evaluation context} to quantify cross-length generalization); (2) implementing a lightweight context-length detector that selects the pattern at inference time with near-zero overhead; (3) characterizing the latency-quality Pareto frontier across context lengths to determine the optimal switching points. If patterns optimized at 200K generalize well to 100K and 300K (which the paper hypothesizes but does not test), the switching logic could be simple (two patterns, short vs. long). If patterns are highly length-specific, a more sophisticated approach — perhaps a continuous parameterization of the sharing ratio as a function of context length — would be needed. The RULER benchmark's multi-length evaluation (4K–128K) provides an existing testbed for this experiment, though RULER scores would need to be broken down by context length rather than aggregated.

Extending multi-layer distillation to the full DSA training pipeline to establish the true training-aware ceiling. The paper's training-aware experiments use a shortened training pipeline (5,000 total steps) that "closely matches" but does not equal full DSA training. The critical question for model developers is: if you train a DSA model from scratch with the multi-layer distillation loss and a uniform 1/4 pattern using the full training budget (tens of thousands of steps, full data), does the performance match a full-indexer model trained with the same budget? The shortened pipeline leaves open the possibility that with more training, the full-indexer model would pull ahead — the multi-layer distillation might help at moderate training budgets but saturate earlier than single-layer distillation, making it inferior at scale. A full-training comparison would require the computational resources of a full DSA training run (comparable to training a production 30B model), making it expensive but definitive. The measurement would be the gap between training-aware IndexCache (uniform 1/4) and standard DSA (all-Full) at training convergence, evaluated on the full benchmark suite. If the gap stays within 0.5 points on Long Avg, training-aware IndexCache is a strict improvement: it matches quality while providing inference speedup with zero pattern search cost. If the gap widens with training, there is a fundamental tradeoff between the representational capacity gained from per-layer indexers and the efficiency gained from sharing, and the optimal point depends on the training budget.

Practical Applications and Downstream Use Cases

Production serving of long-context DSA models at scale (DeepSeek-V3.2, GLM-5 deployments). The most immediate application is in the inference serving infrastructure for production DSA models. The GLM-5 results (Table 4, Figure 1) demonstrate that training-free IndexCache at 1/2 retention preserves "nearly identical" performance on the full Artificial Analysis Index while delivering ~1.2× end-to-end speedup. For a model serving millions of requests per day, a 1.2× speedup translates directly to a ~17% reduction in GPU-hours — equivalent to running the same workload on 5 GPUs instead of 6. The engineering cost of adoption is minimal: one conditional branch per layer in the inference loop, no additional memory, no new kernels, and a one-time pattern search (whose cost the paper does not quantify but which is amortized over the model's deployment lifetime). For the 1/4 retention configuration on the 30B model, the speedups are larger — 1.82× prefill, 1.48× decode at 200K context (Table 1) — making this configuration attractive for throughput-constrained deployments (batch inference, offline evaluation, data generation pipelines) where the marginal quality difference (−0.3 Long Avg vs. baseline) is acceptable in exchange for nearly doubling prefill throughput. The elasticity of the speedup with context length (stronger at longer contexts) makes IndexCache particularly valuable for use cases that push context limits: long document summarization, repository-scale code analysis, multi-turn agent trajectories with accumulated history. A deployment that primarily serves >100K-token requests would see effective speedups closer to the 1.82× ceiling, while a deployment serving mostly <10K-token requests would see more modest 1.2–1.3× speedups.

Cost-efficient data generation and self-improvement pipelines using sparse attention models. When LLMs are used to generate training data for distillation, rejection sampling, or self-improvement loops (STaR, ReST, self-play), inference cost often dominates the total compute budget. A pipeline that generates millions of long-context completions for training a student model or for filtering high-quality reasoning traces can benefit directly from IndexCache's indexer reduction. The training-free variant is particularly well-suited because the quality tolerance for data generation is often higher than for user-facing applications — a 0.3-point Long Avg degradation on the generator model may be imperceptible in the student model trained on its outputs, especially if the student is subsequently fine-tuned. The 1/4 retention configuration (1.48× decode speedup at 200K) would nearly halve the inference cost of generating long reasoning traces, chain-of-thought completions, or agent trajectories. Furthermore, if the data generation pipeline is part of an iterative self-improvement loop (generate → filter → train → generate), the speedup compounds across iterations. The paper's observation that IndexCache sometimes improves reasoning benchmarks (AIME 2025: 92.6 vs. 91.0, GPQA-Diamond: 78.6 vs. 77.6 at 1/4 searched, Table 2) — attributed to a mild regularization effect from removing redundant indexer computations — is particularly relevant: the generator model with IndexCache may produce better training data than the full-indexer model on certain tasks, making the efficiency gain a pure improvement rather than a tradeoff.

On-device or edge deployment of sparse attention models with reduced compute budgets. The paper's 30B model is not an on-device model, but the principle scales down. Sparse attention is increasingly used in smaller models (1B–7B parameters) for on-device deployment (e.g., MiniCPM4, phone-scale LLMs with long-context capabilities), where the indexer's quadratic cost is even more problematic because the total compute budget is severely constrained. A 7B DSA model running on a phone GPU with a 50K-token context may find the indexer consuming 60–70% of the already-limited compute, and IndexCache at 1/4 retention would proportionally reduce this to 15–18% of the original total. The training-free variant is applicable to any pre-trained DSA model regardless of size — the greedy search cost would be lower for smaller models, and the pattern could be searched once on a server and deployed to edge devices. The paper's demonstration that the method adds zero memory overhead (T_cache reuses existing index tensor allocations, Figure 2 caption) is critical for edge deployment where memory is often the primary constraint even more than compute.

Model development with cross-layer sharing as a design principle from the start. The training-aware results (Table 3: uniform 1/4 interleaving matches full-indexer quality) suggest a shift in how DSA models should be trained going forward. A model developer starting a new DSA training run can adopt the multi-layer distillation loss and a uniform 1/4 pattern from the beginning, producing a model that is architecturally identical to standard DSA but runs 1.48–1.82× faster at inference with no quality degradation, no deployment-time optimization, and no pattern search. This is not a post-hoc optimization applied to a finished model — it is a training methodology that produces an intrinsically more efficient model. Given that the multi-layer distillation loss is mathematically equivalent to distilling against averaged target distributions (Proposition 1) and introduces no additional hyperparameters beyond the sharing pattern, the implementation cost is low: modify the distillation loss computation to sum over served layers, and keep the rest of the training pipeline unchanged. If this methodology becomes standard for DSA training, inference efficiency improves "for free" at the cost of a slightly more complex training objective — a tradeoff that is almost certainly net-positive given that inference costs dominate total cost of ownership for deployed models. The paper's findings for GLM-5 (Section 4.5: "We plan to apply training-aware IndexCache to this production-scale model in the near future") suggest the authors view this as the natural endpoint of their work.

When to Prefer This Method

The paper does not position IndexCache against named alternative methods for reducing indexer cost — it is presented as the first method to address this specific bottleneck, with comparisons only against internal baselines (uniform interleaving, similarity-based search, the full-indexer DSA baseline). There is no head-to-head comparison against alternative indexer reduction strategies (reducing indexer precision below FP8, reducing indexer heads, reducing $k$ for the indexer with a larger $k$ for core attention) because the paper frames these as quality-degrading parameter reductions while IndexCache exploits redundancy. The two variants of IndexCache (training-free and training-aware) are positioned as complementary rather than competing — one for frozen models, one for models under development — and the paper provides no explicit tradeoff criteria for choosing between them beyond the obvious: if you can retrain, use training-aware; if you cannot, use training-free with greedy search.

A conditional "prefer X when" decision rule is therefore not applicable here — the paper does not articulate a tradeoff against alternatives, and the internal choice between the two IndexCache variants is already exhaustively characterized by the constraint that determines which is feasible (training access).