Inference Optimization2026~101 minarXiv:2604.22782

Stochastic KV Routing: Enabling Adaptive Depth-Wise Cache Sharing

Anastasiia Filippova, David Grangier, Marco Cuturi +1AppleApr 3, 2026cs.LG

🎯 Pitch

A model’s KV cache for a single book of context can exceed the size of the model weights themselves, yet this paper shows that randomly assigning each layer to reuse a previous layer’s cache during training lets you drop 50–75% of all caches at inference with minimal loss—and sometimes even improves performance by over 50%, turning a memory crisis into free regularization.

1. Executive Summary

This paper introduces Random Cross-Layer Attention (R-CLA), a training scheme that enables depth-wise KV cache sharing by randomly routing each layer's attention to the key–value states of a preceding layer during training, making models robust to arbitrary cache-sharing strategies at inference time. Evaluated on decoder-only Transformers including Llama-3.1-8B, Mistral-7B, and Qwen3-8B across five question-answering benchmarks (HotpotQA, MSMarco, RepLiQA, SQuAD v2, TriviaQA), R-CLA allows dropping 50–75% of layers' caches with dramatically less degradation than baseline models — base models suffer catastrophic collapse at 25% retention while R-CLA preserves substantial capability, and at full retention R-CLA frequently matches or exceeds standard self-attention (e.g., +51% F1 on HotpotQA for Llama-3.1-8B). Inference benchmarks on a Qwen3-8B-scale architecture confirm that cache sharing yields up to 4× KV cache memory reduction and ~22% throughput improvement at 8K context length, while pre-training experiments on Qwen3-1.7B demonstrate that deeper models with shared caches consistently outperform shallower baselines under identical memory budgets — establishing that depth with cache sharing is preferable to reduced depth, but only when the model has been trained with stochastic KV routing to tolerate the sharing patterns.

2. Context and Motivation

The Core Problem: KV Cache Memory Dominates Inference Costs

The fundamental problem this paper addresses is deceptively simple: the memory required to store key–value (KV) states during autoregressive generation dwarfs the memory needed for the model's parameters and input tokens, making it the primary bottleneck in serving transformer language models at scale.

To understand why this matters, we need to walk through what happens during inference. When a transformer decoder generates text autoregressively — predicting one token at a time, each conditioned on all previous tokens — it would be catastrophically wasteful to recompute the key and value projections for every previous token at each generation step. The standard solution is the KV cache: during the initial pre-fill phase, the model processes the entire input prompt in parallel, computing key ( $K$ ) and value ( $V$ ) tensors at every layer for every token and storing them in memory. During the subsequent sampling phase, only the new token's $K$ and $V$ are computed and appended to this cache, while all previous tokens' representations are simply loaded from memory. This avoids redundant computation at the price of a large memory footprint.

The paper quantifies just how large this footprint is. For Llama-2-7B, a single token's KV cache occupies approximately 512 KB (Section 1):

$\text{Cache per token} = 2 \times \text{#layers} \times \text{#kv\_heads} \times \text{head\_dim} \times 2 \text{ bytes}$

The leftmost factor of 2 accounts for storing both $K$ and $V$ ; the rightmost factor of 2 accounts for FP16 precision (2 bytes per floating-point value). For more modern architectures like Llama-3-8B, this drops to 128 KB/token (primarily through Grouped-Query Attention reducing $\text{#kv\_heads}$ ), and for Qwen3-8B it is approximately 144 KB/token — still enormous relative to the original token representation of 2–4 bytes.

The paper makes this concrete with a striking comparison: "caching the context of a book such as Alice's Adventures in Wonderland would consume roughly 1.4 times the memory of the Llama-2-7B model weights themselves." This is a data expansion of over 100,000×: a handful of bytes representing token IDs explodes into hundreds of kilobytes of floating-point tensors. The paper frames this as paradoxical — language models are often described as compression machines that encode training data into their parameters, yet they require massive working memory to manage a single runtime context.

Why This Problem Is Critically Important

The KV cache memory footprint drives several interconnected practical constraints on LLM deployment (Section 1):

It limits batch size. Since the KV cache is stored per-request in GPU high-bandwidth memory (HBM), the total memory consumed scales as $\text{batch\_size} \times \text{sequence\_length} \times \text{cache\_per\_token\_per\_layer} \times \text{num\_layers}$ . For a fixed GPU memory budget, larger caches mean fewer concurrent requests can be served, reducing throughput and increasing cost per query. The paper's inference benchmarks (Section 4.3, Table 5) demonstrate this concretely: at 8K context with batch size 16, the baseline configuration with full per-layer caching runs out of memory on an 80GB GPU, while cache sharing with group size 4 completes successfully.

It constrains context length. Long-context applications — processing entire documents, maintaining long conversations, or analyzing codebases — are memory-limited rather than compute-limited. The KV cache for a sequence of length $T$ with $L$ layers grows as $O(L \times T)$ , and for models with 32+ layers and contexts of 32K+ tokens, this can consume tens of gigabytes per request. This directly limits the kinds of applications that can be deployed.

It introduces memory bandwidth bottlenecks. As the paper notes in Algorithm 1 (Section 3.1.1), standard inference requires loading the entire KV cache from HBM to compute units (SRAM) at every layer for every generated token. The compute cost of attention is relatively small; the dominant cost is loading these large tensors from memory. Reducing the cache size therefore improves not only peak memory but also latency and throughput by reducing memory traffic.

It drives serving costs. In production deployments, the GPU memory occupied by KV caches directly translates to dollars — either through requiring more or larger GPUs to serve a given workload, or through limiting the number of users who can be served concurrently. This is especially acute for applications with long contexts (document QA, code analysis, multi-turn conversations) or high throughput requirements.

Prior Approaches: Three Axes of Attack

The paper organizes prior work on KV cache reduction into three categories (Section 2), and argues that each has fundamental limitations that motivate the need for a complementary approach.

Axis 1: Temporal Eviction and Compression

The most heavily researched direction attempts to reduce the cache along the time axis — deciding which tokens' KV states to keep and which to discard. Early work like StreamLLM (Xiao et al., 2024) observed that attention patterns exhibit strong recency bias: the model pays most attention to recent tokens and a few initial "attention sink" tokens. Keeping only a small sliding window plus these sinks can significantly reduce cache size.

However, this approach faces a fundamental challenge: token importance is query-dependent. A token that seems irrelevant for answering question $Q_1$ might be crucial for answering the follow-up question $Q_2$ about a different aspect of the document. Methods like SnapKV (Li et al., 2024) use attention scores to identify important tokens, but these scores are computed with respect to the current query — they can't anticipate what future queries will need. The paper cites OracleKV (Zhu et al., 2025) and KVZip (Kim et al., 2025) as attempts to predict future relevance from past queries, but notes these methods "either risk information loss or maintain a full index in memory that offsets the communication gains."

There are additional practical issues with temporal eviction. Methods focused on cache updates during generation (H2O by Zhang et al., 2023; FastGen by Ge et al., 2023) "do not reduce the peak memory footprint during the pre-filling phase for long contexts" — the initial cache must still be built in full before eviction can begin. And the computation required to decide which tokens to evict — computing attention scores, clustering, or maintaining indices — consumes resources that partially offset the memory savings.

Axis 2: Architectural Modifications

The most successful reductions in KV cache size have arguably come from changing the model architecture itself rather than post-hoc pruning. Grouped-Query Attention (GQA) (Ainslie et al., 2023) shares KV states across multiple query heads within a single layer: instead of each attention head having its own $K$ and $V$ , groups of heads share a single KV pair. This reduces the cache by a factor equal to the ratio of query heads to KV heads (8:1 in many modern models, hence the 128 KB/token for Llama-3-8B vs. 512 KB/token for Llama-2-7B which used full multi-head attention). Multi-Query Attention (MQA) (Raffel et al., 2020) takes this to the extreme, with all query heads sharing a single KV pair.

Cross-Layer Attention (CLA) (Brandon et al., 2024) extends the sharing idea across layers rather than heads: pairs of adjacent layers share a single KV cache, halving the cache depth. More radically, State Space Models (SSMs) like Mamba (Dao and Gu, 2024) and hybrid architectures like Kimi-k1.5 (Team et al., 2025) and Nemotron-Nano-9B-v2 (Basant et al., 2025) interleave attention layers with SSM layers that require no KV cache at all, dramatically reducing the overall footprint.

KV quantization (Hooper et al., 2024) represents a complementary architectural change: rather than reducing the number of stored states, it reduces the precision (e.g., from FP16 to INT4 or INT8), directly cutting memory per element.

The paper positions its contribution as orthogonal to architectural modifications — "our proposed depth eviction is orthogonal to these architectural improvements and can be combined with GQA or quantization for compounded gains" (Section 2). This is an important claim: R-CLA doesn't compete with GQA or quantization; it stacks on top of them.

The axis the paper focuses on — sharing KV caches across layers — has been explored but, the authors argue, has not been made practical. The key prior works and their limitations:

XC-Cache (Monteiro et al., 2024a) demonstrated that a single shared cache for all layers is feasible, but the approach requires a separate bidirectional encoder to transform the shared representations for each layer's consumption. This introduces a significant overhead: "adding to the cache requires a full encoder forward pass," making cache updates expensive during generation.

Layer-Condensed KV Cache (Wu and Tu, 2024) proposes a shared single-layer cache but "suffers from a significant increase in time-to-first-token due to the need for sequential pre-filling or multiple forward passes." The model must estimate what the top layers' output would have been, requiring iterative refinement that adds latency exactly when it matters most — during the pre-fill phase when users are waiting for the first token.

MiniCache (Liu et al., 2024) takes a post-hoc approach, merging pairs of late layers' caches without retraining. But "its gains are modest since the model is not trained to handle this type of weight sharing" — the layer-specific feature alignments learned during standard training break when caches are suddenly merged.

CommonKV (Wang et al., 2025b) and KVSharer (Yang et al., 2024) are training-free methods that identify shareable layers via similarity metrics. While avoiding retraining is attractive, these achieve only moderate compression (~30% for KVSharer) because they must work within the constraints of what the model already tolerates.

LISA (Mu et al., 2024) trains lightweight adapter networks to reconcile cross-layer attention differences, but requires post-hoc training of alignment networks specific to each sharing pattern.

Wu et al. (2025) provide a systematic study of various cross-layer sharing configurations but "require pre-training from scratch for each" specific pattern — fine for analysis but impractical for deployment across diverse hardware.

The common thread across these limitations is rigidity: each method produces a model tied to a specific sharing pattern, requiring retraining or additional components to change. This motivates the paper's central innovation — stochastic training that produces a single model robust to arbitrary sharing strategies.

Where Existing Approaches Fall Short: The Hidden Cost of the Status Quo

Beyond the specific limitations of individual methods, the paper identifies a deeper problem with the standard inference paradigm: the KV cache represents a massive utilization gap. The paper's Figure 1 and the 100,000× data expansion calculation are not just colorful analogies — they point to a structural inefficiency. A transformer processes information hierarchically, with early layers capturing surface features and later layers capturing more abstract semantics. It is intuitively implausible that each of the 32+ layers requires a completely independent, fully detailed representation of every token to perform its function. Recent work (cited by the authors) suggests that models exhibit high inter-layer redundancy, and that "a full per-layer cache is likely unnecessary" (Section 1).

But prior approaches to exploiting this redundancy have all hit the same wall: the model wasn't trained to tolerate it. Standard transformer training creates a rigid dependency between each layer $l$ and its own specific KV states $(K_l, V_l)$ . The query projection $Q_l$ is trained to interact with features computed at exactly the same depth. When you suddenly force layer $l$ to attend to $(K_{l'}, V_{l'})$ from a different layer $l'$ , the feature alignment breaks, and performance degrades — sometimes catastrophically, as the paper's baseline results demonstrate (Table 2: base models at 25% retention achieve near-zero F1 on most tasks).

This is the gap the paper aims to fill: a training method that breaks this rigid dependency during training, so that the model can tolerate flexible, deployment-time cache sharing without degradation.

How This Paper Positions Itself

The paper positions R-CLA as a training-time solution to an inference-time problem, drawing a deliberate parallel to GQA's approach of sharing within layers: "Just as GQA shares KV states across heads, we follow a similar approach to CLA and share them across layers, though we use a randomized training scheme to ensure models remain robust to various cache sharing strategies" (Section 2).

What makes this positioning distinctive relative to prior depth-wise sharing work is the emphasis on flexibility and deployment generality. The paper explicitly frames the problem in terms of diverse hardware constraints:

"Crucially, this flexibility allows a single model to be deployed across diverse hardware environments, from high-end clusters retaining 100% of the cache to edge devices retaining only a fraction of it, without the need to train separate models for each observed hardware constraint."

This is a significant practical claim. In real-world deployment pipelines, the same model checkpoint might need to serve on H100 clusters with 80GB of HBM, on A100 instances with 40GB, or on edge devices with tighter constraints. Training separate models for each target is expensive and fragments the deployment pipeline. R-CLA promises a single model that can be configured at deployment time — simply choose how many layers to cache based on available memory, and the model handles it gracefully.

The paper also positions its approach as mechanistically distinct from two superficially similar ideas. First, structured dropout (Fan et al., 2019) skips entire layers during training to make the model robust to reduced depth at inference — but R-CLA "every layer performs its full computation; only the source of the KV states is randomized." The model's depth remains intact; what changes is where each layer gets its keys and values. Second, deterministic CLA (Brandon et al., 2024) trains with a fixed sharing pattern — but R-CLA's stochasticity ensures the model sees many different patterns during training, building robustness to the specific pattern chosen at deployment.

Finally, the paper positions its contribution within a broader vision of inference optimization that stacks with existing techniques. The abstract and conclusion both emphasize orthogonality: R-CLA can be combined with GQA (already standard in modern architectures), with KV quantization, and potentially with temporal eviction methods. The paper isn't proposing to replace these approaches but to add a new dimension — the depth dimension — to the optimization toolkit.

The Core Research Question

Reading between the lines, the paper is driven by a specific hypothesis: the standard transformer training procedure creates an artificial coupling between layer depth and KV representation that is not necessary for model performance, and breaking this coupling through randomized training can unlock substantial memory savings at minimal accuracy cost. The experiments are designed to test this hypothesis across three regimes: (1) pre-training from scratch to verify training stability and compare depth-with-sharing against reduced-depth baselines, (2) fine-tuning to assess whether existing pre-trained models can be adapted efficiently, and (3) deployment-time inference to measure actual throughput and memory benefits.

3. Technical Approach

3.1 Reader Orientation (Accessible Technical Breakdown)

This paper develops a training method that teaches transformer language models to tolerate having some layers' key-value caches deleted at inference time. The system is a stochastic training procedure — Random Cross-Layer Attention (R-CLA) — that, during training, randomly forces each layer to borrow the key-value states from a different (earlier) layer instead of using its own, producing a single model that can be deployed with any cache-sharing pattern chosen at runtime based on available hardware memory.

3.2 Big-Picture Architecture (Diagram in Words)

The system has three major components that interact across training and inference:

The Base Transformer Decoder — a standard autoregressive language model with $L$ layers, each performing self-attention with its own query ( $Q_l$ ), key ( $K_l$ ), and value ( $V_l$ ) projections. During standard inference, every layer's $(K_l, V_l)$ pair is cached in GPU memory for all tokens, creating the memory bottleneck.
The Random Cross-Layer Attention (R-CLA) Training Mechanism — a modification to the forward pass during training (and fine-tuning) that, with probability $1-p$ , replaces a layer's own keys and values with those from a randomly selected earlier layer. With probability $p$ , the layer performs standard self-attention. This stochastic process breaks the rigid dependency between each layer and its own specific KV states, building robustness into the model's representations.
The Cache Sharing Strategy (Deployment-Time Configuration) — a deterministic subset $S \subseteq \{1, \ldots, L\}$ of layer indices whose KV caches are actually stored in memory during inference. Layers not in $S$ reuse the cache of the nearest preceding layer that is in $S$ . The same trained model can be deployed with different strategies (e.g., caching every 2nd layer for 50% retention, every 4th layer for 25% retention) without retraining.

Information flows as follows: during training, each forward pass processes input tokens through all $L$ layers in sequence. At each layer $l$ , a Bernoulli random variable is sampled — if it is 1, the layer computes and attends to its own $(K_l, V_l)$ ; if 0, a random earlier layer $l' < l$ is sampled uniformly, and layer $l$ computes its own query $Q_l$ but uses $(K_{l'}, V_{l'})$ from the earlier layer. The model is trained end-to-end with the standard language modeling loss on next-token prediction. At inference time, the cache sharing strategy $S$ is fixed — only layers in $S$ compute and store KV states; all other layers reuse the nearest available cache.

3.3 Roadmap for the Deep Dive

First, the standard inference architecture and why the KV cache dominates memory — this establishes the exact problem R-CLA must solve and introduces the notation for layer-specific keys and values.
Second, the formal definition of cross-layer attention (CLA) and cache sharing strategies — these are the mathematical objects that describe what happens when a layer uses another layer's cache, and how we specify which layers are cached at deployment time.
Third, the Random Cross-Layer Attention (R-CLA) training procedure — the core contribution: why stochasticity is necessary, how the Bernoulli sampling and uniform layer selection work, and what properties this induces in the trained model.
Fourth, the cache sharing mapping function $\mu(l)$ that determines which cache each layer uses at inference — this is the bridge between training-time randomness and deployment-time determinism.
Fifth, the training configurations and hyperparameters for both pre-training and fine-tuning experiments — the concrete numbers that make these experiments reproducible.
Sixth, the inference efficiency implementation — how depth-wise cache sharing translates to actual memory savings and throughput gains on GPU hardware.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a training methodology paper whose core idea is that randomly routing attention across layers during training creates a model whose representations are invariant to which specific layer produced the key-value states, enabling flexible depth-wise cache sharing at inference without per-deployment retraining.

Standard Transformer Inference and the KV Cache Bottleneck

We begin by establishing the standard inference procedure that R-CLA modifies. A transformer decoder consists of $L$ layers. For a given layer $l \in \{1, \ldots, L\}$ , the standard self-attention mechanism receives a hidden state input $H_l \in \mathbb{R}^{T \times d}$ where $T$ is the sequence length and $d$ is the hidden dimension. From this input, the layer computes three projections:

$Q_l = H_l W_l^Q, \quad K_l = H_l W_l^K, \quad V_l = H_l W_l^V$

where $W_l^Q$ , $W_l^K$ , and $W_l^V$ are learned weight matrices specific to layer $l$ .

What these compute: three distinct representations of the input tokens — queries represent "what information each token is looking for," keys represent "what information each token can provide to others," and values represent "the actual content to be aggregated." The attention output is then:

$\text{Output}_l = \text{Attn}(Q_l, K_l, V_l) = \text{softmax}\left(\frac{Q_l K_l^T}{\sqrt{d_k}}\right) V_l$

where $d_k$ is the dimension of each key vector (the head dimension). The softmax produces an $T \times T$ attention weight matrix; multiplying by $V_l$ produces a weighted sum of value vectors.

The KV cache is the pair $(K_l, V_l)$ for every layer $l$ . During autoregressive generation, the model processes the prompt in the pre-fill phase (computing $K_l, V_l$ for all prompt tokens at all layers in parallel, then storing them), and during the sampling phase it computes only the new token's $K_l, V_l$ at each layer while loading all previous tokens' cached states from memory.

The paper quantifies the memory cost (Section 3.1.1):

$M = L \times (T + t) \times S$

where $M$ is the total KV cache memory footprint, $L$ is the number of layers, $T$ is the number of prompt tokens, $t$ is the number of generated tokens, and $S$ is the size of the KV state for a single token at one layer.

What this equation computes: the total GPU memory consumed by the KV cache during generation. It grows linearly with both sequence length and model depth, and in typical deployments (long contexts, deep models) it dominates the model parameter memory. For Llama-2-7B with 32 layers and full multi-head attention, $S$ = 512 KB per token per layer; for a 40K-token book, $M \approx 32 \times 40000 \times 512 \text{ KB} \approx 655 \text{ GB}$ — more than the 14 GB of model weights (Figure 1).

Why this matters for R-CLA's design: the term $L \times S$ in the memory equation is what depth-wise cache sharing attacks. If we can reduce the effective number of independently cached layers from $L$ to $|S|$ (the number of layers we actually store), we reduce the cache footprint proportionally. The challenge is that standard training creates a rigid dependency between each layer $l$ and its own specific $(K_l, V_l)$ — the query projection $W_l^Q$ is trained to interact with features at exactly the same depth, and suddenly substituting $(K_{l'}, V_{l'})$ from a different layer breaks this alignment.

Cross-Layer Attention (CLA): Formal Definition

Cross-layer attention is the operation that R-CLA makes robust. The paper defines it as follows (Section 3.1.2):

$\text{Output}_l = \text{Attn}(Q_l, K_{l'}, V_{l'}) \quad \text{where } l' \leq l$

What this equation computes: the attention output at layer $l$ when it uses the key and value states produced by an earlier (or same) layer $l'$ . The layer still computes its own query $Q_l$ from its current hidden state — only the source of $K$ and $V$ changes. When $l' = l$ , this reduces to standard self-attention. When $l' < l$ , the model reuses cached states from earlier in the network.

Why this form: this is the minimal modification to standard attention that enables cache sharing. The query projection remains layer-specific because each layer processes information at a different level of abstraction — a later layer's query is looking for different patterns than an early layer's query. But the key and value projections are shared because they represent the information content of tokens, and the paper's hypothesis is that this content doesn't need to be recomputed independently at every layer. The constraint $l' \leq l$ ensures causality: a layer cannot attend to states that haven't been computed yet (in the forward pass order).

This is conceptually similar to the cross-attention mechanism in encoder-decoder transformers, where the decoder attends to the encoder's key-value states. Here, later decoder layers "cross-attend" to earlier decoder layers' states. The paper explicitly draws this connection (Section 2): "These works provide strong evidence for the hypothesis that Transformer decoders can function with a shared set of KV states, much like the cross-attention mechanism in encoder-decoder models."

A cache sharing strategy is formalized as a set $S \subseteq \{1, \ldots, L\}$ of layer indices that are authorized to maintain a KV cache in memory (Section 3.1.3). The goal of deployment is to minimize $|S|$ (the number of cached layers) while maintaining model performance.

For any layer $l$ , the system determines which cache to use via the mapping function:

$\mu(l) = \begin{cases} l & \text{if } l \in S \\ \max\{j \in S \mid j < l\} & \text{if } l \notin S \end{cases}$

What this function computes: for each layer $l$ , it returns the index of the layer whose KV cache $l$ should use. If layer $l$ is itself cached ( $l \in S$ ), it uses its own cache ( $l$ ). If layer $l$ is not cached ( $l \notin S$ ), it uses the cache of the nearest preceding layer that is cached — found by taking the maximum element of $S$ that is strictly less than $l$ .

Why this form: the "nearest preceding cached layer" rule minimizes the depth gap between a layer and the cache it reuses. The intuition is that representations change gradually across layers, so layers $l$ and $l-1$ have more similar key-value distributions than layers $l$ and $l-5$ . By defaulting to the nearest available cache, the strategy minimizes the distributional mismatch that each layer must tolerate. The function also ensures that every layer has a valid cache to use: since layer 1 is always cached (it has no preceding layer), there is always at least one element in $S$ less than any $l > 1$ .

What makes this efficient in implementation: Algorithm 2 in the paper shows that when consecutive layers share a cache, the KV tensors can be loaded from GPU memory once and reused for multiple layer computations without additional memory traffic. Only layers $l \in S$ trigger Load and Update operations on the cache; non-cached layers reuse the CurrentCache variable that was loaded by the most recent cached layer.

This efficiency is the key practical motivation. The memory bandwidth bottleneck in standard inference (Algorithm 1) comes from loading $(K_l, V_l)$ from HBM to SRAM at every layer. Under cache sharing, the number of memory loads drops from $L$ to $|S|$ , which directly reduces latency and improves throughput — even beyond the peak memory savings.

Random Cross-Layer Attention (R-CLA) Training: The Core Method

The fundamental problem that R-CLA solves: a standard pre-trained model "relies on specific feature alignments between $Q_l$ and $(K_l, V_l)$ ." When you force a layer to use another layer's cache at inference time, these alignments break, causing performance degradation. The solution is to train the model to be invariant to the source of keys and values.

The R-CLA training procedure (Section 3.2): during training, for every layer $l$ and every forward pass:

Sample a decision variable $d \sim \text{Bernoulli}(p)$ — a coin flip with probability $p$ of landing on 1.
If $d = 1$ : the layer performs standard self-attention, computing $\text{Attn}(Q_l, K_l, V_l)$ .
If $d = 0$ : the layer performs cross-layer attention, using $(K_{l'}, V_{l'})$ where $l'$ is sampled uniformly from $\{1, \ldots, l-1\}$ . The attention operation becomes $\text{Attn}(Q_l, K_{l'}, V_{l'})$ .

The probability $p$ is a hyperparameter controlling how often layers use their own cache versus a borrowed one. The paper experiments with $p \in \{0.25, 0.5, 0.6, 0.75\}$ for pre-training and uses $p = 0.6$ for all fine-tuning experiments (meaning 40% of attention operations are cross-layer on average).

Why randomness rather than a fixed pattern: training with a single deterministic sharing pattern (e.g., always share between layers $l$ and $l-2$ ) would overfit the model to that specific pattern. The model would learn to align its query projections with the specific key-value distributions from exactly one earlier layer, but would still fail when deployed with a different pattern. The randomization serves as a form of data augmentation over sharing configurations — the model sees many different $(l, l')$ pairs during training and must learn query projections that work with a wide variety of key-value distributions.

This is mechanistically distinct from two similar-sounding ideas. Structured dropout (Fan et al., 2019) skips entire layers during training — the layer's computation is dropped entirely, reducing effective depth. In R-CLA, "every layer performs its full computation; only the source of the KV states is randomized." The layer still computes $Q_l$ , still performs attention, and still contributes to the forward pass — it just might use someone else's keys and values. Deterministic CLA (Brandon et al., 2024) always shares between fixed layer pairs (e.g., layers 2 and 3 always share) — the model never sees variation in the sharing pattern.

The paper's ablation study (Section 4.2.3, Table 3) directly tests the value of randomness by comparing R-CLA against CLA@k (deterministic sharing in groups of $k$ ) and RD-CLA@k (deterministic groups with stochastic application — the pattern is fixed but a coin flip decides whether to use it or local KV). The results show that under matching retention levels (the exact pattern the deterministic model was trained for), CLA@k can match R-CLA. But when retention changes at deployment time (which is the whole point — a single model serves diverse hardware), only R-CLA maintains performance across levels. This confirms that the randomness during training is what builds generalization across sharing patterns.

What physically happens during an R-CLA forward pass: the paper's Figure 4 illustrates this. During training, each layer independently flips its coin. In one batch, layer 3 might self-attend while layer 4 cross-attends to layer 1 and layer 5 cross-attends to layer 2. In the next batch, the pattern is different. After training across many batches and many random configurations, the query projection $W_l^Q$ at each layer has learned to extract relevant information from key-value states regardless of which earlier layer produced them — the representations become depth-invariant with respect to the information they encode.

The training loss is the standard causal language modeling objective — next-token prediction cross-entropy — applied to the model's output logits. There is no auxiliary loss for the attention routing; the model learns to cope with random cache routing purely through the pressure to predict tokens correctly.

Pre-Training Configuration (Section 4.1.1)

The pre-training experiments serve to verify that R-CLA does not destabilize training and to compare deeper models with cache sharing against shallower models with full caching under identical memory budgets.

Architecture: Qwen3-1.7B-style decoder-only Transformer. The paper uses a model with 28 layers and the standard Qwen3 attention configuration (Grouped-Query Attention with 8 KV heads). For R-CLA variants, the full 28-layer architecture is used; the cache sharing probability $p$ is varied in $\{0.25, 0.5, 0.6, 0.75\}$ .

Data: A subset of the OpenWeb corpus (Gokaslan et al., 2019), with a fixed context length of 2,048 tokens.

Optimization: AdamW (Loshchilov and Hutter, 2017) with $\beta_1 = 0.9$ , $\beta_2 = 0.99$ , weight decay 0.1, gradient clipping with maximum norm 0.1. The learning rate follows a linear warm-up to $1 \times 10^{-4}$ over the first 5% of training steps, followed by cosine decay for the remainder.

Compute budget: All models are trained for an identical token budget of 34 billion tokens, which the paper notes is "Chinchilla-optimal" (Hoffmann et al., 2022) for a 1.7B-parameter model — meaning the training tokens are roughly 20× the parameter count, the ratio at which pretraining compute is used most efficiently.

Baselines: Shallower Transformers where the number of layers corresponds to the effective cache size under R-CLA. For example, against an R-CLA model with 28 layers and $p = 0.5$ (meaning roughly 14 layers' worth of caches are retained at inference), the baseline is a 14-layer Transformer with standard training and full per-layer caching. This is a fair comparison because both models have the same KV cache memory footprint at inference; the question is whether keeping the extra layers but sharing caches is better than simply removing them.

Hardware: NVIDIA H100 GPUs.

Fine-Tuning Configuration (Section 4.2)

The fine-tuning experiments test whether R-CLA can be applied to existing pre-trained models (rather than requiring training from scratch) and whether the stochastic training induces a regularization effect in data-constrained settings.

Base models: Qwen3-8B (Yang et al., 2025a), Mistral-7B (Jiang et al., 2023), and Llama-3.1-8B (Dubey et al., 2024). These represent three distinct model families with different pre-training recipes, attention configurations, and sizes.

Training data: A curated dataset from five QA sources covering diverse reasoning requirements: HotpotQA (multi-hop reasoning — requires synthesizing information from multiple context passages), SQuAD v2 (includes unanswerable questions), MSMarco (machine reading comprehension), TriviaQA (knowledge-intensive QA), and RepLiQA (questions about fictional content — ensures the model must retrieve from context rather than relying on parametric knowledge). The combined dataset consists of (context, question, answer) triples.

Data augmentation (Section 4.2.1): Two strategies are applied to improve robustness. First, input component order is randomized with probability 0.5: half the examples follow the standard context → question → answer format, and half follow question → context → answer. The paper notes that "placing the question first theoretically benefits the model by allowing the query to condition the context encoding," but this causal advantage is rarely available at test time, so the randomization serves to diversify the training distribution and prevent the model from depending on a specific input structure. Second, for HotpotQA specifically, multiple context passages are provided (some of which are irrelevant confounders), and for each training example three variations are generated with different permutations of the concatenated context passages to prevent position bias.

Training hyperparameters: All models train for 50,000 steps with batch size 128 and maximum input length 8,192 tokens. The optimizer is AdamW with $\beta_1 = 0.9$ , $\beta_2 = 0.95$ , weight decay 0.1. The learning rate warms up linearly to $5 \times 10^{-6}$ over the first 1.5% of training steps, then decays linearly to 0.

R-CLA configuration: During fine-tuning, $p = 0.6$ — meaning each layer has a 40% chance of using cross-layer attention on each forward pass. The paper does not sweep $p$ for fine-tuning experiments; 0.6 is presented as the default configuration.

Comparison baseline: Standard fine-tuning of the same base models with identical data, hyperparameters, and compute budget, but using only standard self-attention (no cross-layer routing during training). At evaluation time, both the R-CLA and baseline models are tested under the same deterministic cache sharing strategies (e.g., retaining every 2nd layer, every 4th layer) with the mapping function $\mu(l)$ as defined above.

The Training Dynamics of R-CLA: Why It Works

Understanding why R-CLA works requires examining the training dynamics it induces. The paper provides evidence through several observations:

Training stability (Table 1, Figure 6 in Appendix A): Across all values of $p$ , pre-training loss curves are smooth and stable — there is no sign of training instability or divergence even at $p = 0.75$ (where three-quarters of attention operations use borrowed caches). The evaluation loss increases modestly from 2.424 at $p = 0$ (standard training) to 2.461 at $p = 0.75$ , representing less than a 2% degradation. This is important because it shows that the randomness does not create optimization difficulties — standard hyperparameters transfer from self-attention training without needing special tuning.

Regularization-like effect (Appendix B, Figure 7): During fine-tuning, R-CLA models show slower learning (higher training loss throughout) compared to standard fine-tuning. For Llama-3.1-8B, the baseline model overfits — validation loss begins increasing after approximately 400 steps — while the R-CLA model "delays the onset of overfitting." For Qwen3-8B, R-CLA "learns consistently more slowly." The paper interprets this as evidence that "in a data-constrained fine-tuning setting, this means R-CLA models can train for more epochs before entering the overfitting regime." The randomness acts as a regularizer by preventing the model from memorizing layer-specific feature alignments that would not generalize to cache sharing.

Deeper models with sharing beat shallower models without (Figure 5): The pre-training results show that for a fixed KV cache memory budget, R-CLA models using the full 28-layer architecture with depth-wise cache sharing consistently achieve lower evaluation loss than shallower baseline Transformers with the same effective number of cached layers. This is the paper's strongest architectural claim: the extra layers provide representational capacity that benefits the model, even when those layers must share caches. The depth is not wasted — it provides hierarchical processing that improves predictions, while the cache sharing prevents the memory cost from scaling with depth.

Why this is not trivial: One might think that if layers are sharing caches, the extra layers add no new information and are just dead weight. The results contradict this. Even when a layer reuses a previous layer's keys and values, it still computes its own query projections and its own feed-forward network transformations. The query determines what information to extract from the shared representations, and different layers can extract different patterns from the same key-value states. The feed-forward network then transforms the attention output, enabling layer-specific processing. So depth with sharing provides strictly more computation than reduced depth, at the same memory cost.

Inference-Time Efficiency Implementation (Section 4.3)

The paper measures actual throughput and memory benefits using a Qwen3-8B-scale architecture (36 layers, 4096 hidden dimension, 32 attention heads, 8 KV heads, head dimension 128) with bfloat16 precision on a single 80GB GPU.

Cache sharing implementation: Under cache sharing with group size $g$ , every $g$ consecutive layers share a single KV cache. The first layer in each group (the "leader") computes and stores $K, V$ ; the remaining $g-1$ layers skip their $K$ and $V$ projections entirely and reuse the leader's cached states. This means for a model with 36 layers and group size $g=4$ , only 9 layers produce and store KV states.

What makes this efficient: The implementation skips two things for non-leader layers. First, it skips the key and value projections — the matrix multiplications $H_l W_l^K$ and $H_l W_l^V$ are not computed at all, saving FLOPs. Second, it avoids allocating memory for $(K_l, V_l)$ — the KV cache entries for non-leader layers simply don't exist in GPU memory. The attention computation itself — loading $K, V$ from HBM and computing $\text{softmax}(QK^T)V$ — still occurs at every layer, but it operates on the leader's cached states rather than layer-specific ones.

The paper notes that these measured gains "represent a conservative lower bound" because "backend-level optimizations could go further: for instance, keeping the shared K,V in SRAM across consecutive layers, or fusing their attention computations into a single memory load." In the current implementation, consecutive non-leader layers each independently load the leader's cache from HBM; a fused implementation could load it once and keep it in SRAM while processing the entire group, further reducing memory bandwidth.

Measured benefits (Tables 4 and 5):

KV cache memory scales as $1/g$ : at 8K tokens, the cache drops from 1,170 MB (baseline, $g=1$ ) to 293 MB ( $g=4$ ) — a 4× reduction.
Decode throughput improves by approximately 22% at 8K context (34.0 tok/s baseline → 41.6 tok/s with $g=4$ ), due to skipping K/V projections on non-leader layers.
Peak memory savings grow with context length: at 32K tokens, $g=4$ saves 3.5 GB versus baseline.
Batch size scaling (Table 5): at batch size 16 with 8K context, the baseline configuration ( $g=1$ ) runs out of memory on the 80GB GPU, while $g=4$ completes successfully. This is the most practically significant result — it means cache sharing directly enables serving more concurrent users on the same hardware, which translates to lower cost per query.

Time-to-first-token (TTFT) shows minimal improvement under cache sharing (e.g., 297 ms → 286 ms at 8K context with $g=4$ ). This is expected because the pre-fill phase is compute-bound rather than memory-bound — the dominant cost is the $O(T^2)$ attention computation over all prompt tokens, not the KV storage. The primary benefits of depth-wise cache sharing are in the memory-bound sampling phase (throughput) and peak memory footprint (enabling larger batches and longer contexts), not in pre-fill latency.

4. Key Insights and Innovations

Innovation 1: The Inference-Time Cache Is a Deployment Property, Not a Training Property

The dominant framework in prior depth-wise cache sharing work treats the sharing pattern as a choice made at training time, frozen into the model architecture. CLA (Brandon et al., 2024) hardcodes which layers share; Wu et al. (2025) pre-train separate models for each configuration; LISA (Mu et al., 2024) trains adapter networks specific to one pattern; post-hoc methods like MiniCache (Liu et al., 2024) and KVSharer (Yang et al., 2024) pick one sharing scheme at application time and the model lives with it. The underlying assumption across all of these is that the cache configuration is a model-level property — you train it, you ship it, and that's what you get.

R-CLA challenges this assumption at the conceptual level. The paper's core intellectual move is to decouple the training objective from the deployment configuration by treating cache sharing as a runtime-settable parameter rather than an architectural constant. During training, the model sees a randomized variety of sharing patterns; at deployment, a deterministic pattern is chosen — but crucially, different deployments of the same trained model can choose different patterns without retraining. This is stated explicitly:

"Crucially, this flexibility allows a single model to be deployed across diverse hardware environments, from high-end clusters retaining 100% of the cache to edge devices retaining only a fraction of it, without the need to train separate models for each observed hardware constraint."

This framing shifts the problem from "how do we design the optimal sharing pattern?" (the question prior work tries to answer) to "how do we train a model that is invariant to which pattern is chosen?" — a fundamentally different question with a fundamentally different solution (stochasticity during training). It's the difference between optimizing for a point estimate and optimizing for robustness across a distribution.

The significance of this reframing extends beyond KV caches. It suggests a general principle for deploying large models across heterogeneous hardware: rather than training separate models for each target, train a single model with stochastic ablations of the resource being conserved, and select the conservation level at runtime. This is analogous to how slimmable neural networks (Yu et al., 2018) train once for multiple width configurations, applied here to the depth dimension of attention state storage.

Evidence that this decoupling actually works comes from Figure 2 and Table 2: a single R-CLA-trained model maintains performance across 100%, 50%, and 25% cache retention, while base models collapse at reduced retention. The ablation in Table 3 provides the critical causal evidence — CLA@k (trained for one specific sharing pattern) degrades when evaluated at a different retention level, while R-CLA (trained with randomness) maintains performance across levels. The randomness during training is not just a regularization hack; it's the mechanism that creates deployment-time flexibility.

This is a fundamental conceptual shift rather than an incremental improvement. It changes what it means to "solve" the KV cache problem: the goal is no longer finding the single best sharing pattern, but producing a model that can adapt to whatever pattern the deployment hardware demands.

A natural baseline for reducing KV cache memory is simply removing layers: a model with 14 layers and full per-layer caching has the same cache footprint as a 28-layer model where adjacent layers share caches. Prior work has implicitly assumed these are roughly equivalent, or that the shallower model might even be better since it avoids any cross-layer mismatch. The paper provides the first clean empirical test of this tradeoff, and the result is a counterintuitive architectural insight: the deeper model with cache sharing consistently outperforms the shallower model with full caching at equal memory budgets, but only when trained with R-CLA.

Figure 5 and the associated pre-training results (Section 4.1.2) make this case directly. The R-CLA model with 28 layers and p=0.5 — effectively retaining ~14 layers' worth of caches at inference — achieves lower evaluation loss than a 14-layer baseline Transformer trained on the same 34B tokens. This holds across the full range of effective cache sizes tested (7, 14, 17, 21, and 28 layers' worth of caches). The implication is that the extra layers provide genuine representational benefit: even when a layer reuses another layer's keys and values, it still performs its own query projection (extracting different information from the shared representations) and its own feed-forward network computation (transforming that information). Depth is not wasted just because the KV states are shared.

What makes this insight distinctive is that it identifies a false equivalence in how the field has thought about KV cache optimization. Shrinking the model isn't the same as sharing caches within a full-depth model, and the difference favors sharing — provided the model has been trained to tolerate it. This is not obvious a priori. One could reasonably argue that a 14-layer model trained for 14 layers of computation is better than a 28-layer model trained to approximate 14 layers of attention. The experimental result shows the opposite, but it also shows that this only holds with R-CLA training — an untrained model forced into cache sharing at deployment would perform worse than the shallower baseline, which is exactly why prior post-hoc methods achieve only modest compression.

This finding has implications for model architecture design beyond KV cache optimization. It suggests that transformer depth provides benefits that are partially separable from the attention mechanism's KV states — the feed-forward networks and query projections at each layer contribute value even when the attention inputs are shared. This aligns with work on the role of feed-forward layers as key-value memories (Geva et al., 2020) and supports the view that transformer depth is not just about refining attention patterns but about applying successive transformations to representations.

The innovation here is empirical rather than methodological: it establishes a new Pareto frontier showing that depth-with-sharing dominates reduced-depth at identical memory budgets, provided the right training procedure is used. This is a fundamental finding that should inform future architecture design for memory-constrained deployment.

Innovation 3: Stochastic KV Routing Acts as a Data-Augmentation Regularizer That Mitigates Overfitting

The paper observes that R-CLA fine-tuning frequently preserves or improves performance at full cache retention compared to standard fine-tuning on the same data (Table 2: +51% F1 on HotpotQA for Llama-3.1-8B, +53% for Qwen3-8B, +30% on SQuAD v2 for Llama-3.1-8B, +102% on TriviaQA for Qwen3-8B). These are not marginal improvements — they are large, consistent gains on a task (question-answering) where the fine-tuning dataset is modest (the combined dataset from five QA sources) relative to model size.

The paper interprets this as a regularization-like effect from the stochastic training, and the training dynamics evidence supports this interpretation. Appendix B (Figure 7) shows that R-CLA models learn more slowly than standard fine-tuning — higher training loss throughout — and for Llama-3.1-8B, the baseline model begins overfitting (validation loss increasing) around step 400 while R-CLA "delays the onset of overfitting." The randomness in attention routing prevents the model from memorizing layer-specific feature alignments that would not generalize, effectively acting as a strong regularizer in the data-constrained fine-tuning regime.

What makes this an innovation rather than just a nice side effect is that it repurposes an inference-efficiency technique as a generalization-improvement technique. The stochastic routing was designed to build robustness to cache sharing; the fact that it also improves full-cache performance suggests that standard transformer training creates overly specific layer-wise dependencies that hurt generalization even when the full cache is available. Breaking these dependencies during training — forcing the model to extract information from representations at different depths — produces representations that are not only robust to cache sharing but also more general.

This is not just a claim about stochasticity as regularization (dropout already does that). The mechanism is specific: R-CLA regularizes the depth-specificity of representations. Standard training allows Query_l to become specialized to Key_l and Value_l — representations at exactly the same depth. R-CLA forces Query_l to work with key-value distributions from a wide range of earlier layers, preventing this specialization. The resulting representations capture information in a more depth-invariant way, which turns out to be beneficial even when the full cache is available.

The ablation (Table 3) helps isolate this effect. At full retention, deterministic CLA@k can match R-CLA (e.g., CLA@4 on TriviaQA at 100%), suggesting the performance gains at full cache come primarily from the information bottleneck created by KV sharing rather than the randomness itself. But under reduced retention, only the fully stochastic R-CLA maintains performance, showing that randomness is necessary for generalization across sharing patterns. The two effects — regularization from the bottleneck and robustness from the randomness — are complementary but mechanistically distinct.

This is best classified as an empirical discovery with conceptual implications: stochastic depth-wise routing during training provides a new form of regularization that is particularly effective in data-constrained fine-tuning, and this regularization is a distinct benefit beyond the primary goal of enabling cache sharing. It opens the question of whether similar stochastic routing techniques could be applied to other axes (heads, attention windows, FFN modules) for regularization purposes even when inference efficiency is not the goal.

Innovation 4: The Depth Dimension Is Orthogonal and Stackable — A Systematic Case for Modular Efficiency

The paper makes a deliberate architectural argument that depth-wise cache sharing is orthogonal to, and stackable with, the two other major approaches to KV cache reduction: temporal eviction and intra-layer sharing (GQA/quantization). This is stated explicitly in Section 2 and reinforced in the conclusion, but the paper goes beyond merely asserting orthogonality — it provides the architectural decomposition that makes the case systematic.

The total KV cache memory can be factored as:

Memory ∝ (number of stored layers) × (sequence length) × (bytes per KV element) × (KV heads per layer)

Each factor corresponds to a different class of optimization:

Number of stored layers → depth-wise sharing (R-CLA's target)
Sequence length → temporal eviction (H2O, SnapKV, PyramidKV, etc.)
Bytes per KV element → quantization (KVQuant, etc.)
KV heads per layer → architectural changes (GQA, MQA)

Prior work has largely operated within single factors. GQA reduces the heads factor; temporal eviction reduces the sequence length factor; quantization reduces the bytes-per-element factor. R-CLA's contribution is to show that the depth factor — which had been explored only in rigid or post-hoc ways — can be systematically optimized with stochastic training. And because these factors are multiplicative in the memory equation, gains compound: applying GQA reduces the cache by ~4× (from 32 heads to 8 KV heads); adding depth-wise sharing at 25% retention reduces it by another ~4×; adding INT4 quantization reduces it by another ~4× — the combined reduction is 4 × 4 × 4 = 64×, all operating on independent factors with no interference.

What makes this framing an innovation is that it converts an empirical technique into a modular design principle. The paper is not just saying "our method works and can be combined with other things" — it's providing the factorization that shows why the methods don't interfere, and demonstrating that the depth dimension is the underexploited factor in this factorization. The inference benchmarks (Section 4.3) use GQA architectures (Qwen3-8B style with 8 KV heads), confirming that the gains from depth-wise sharing are measured on top of GQA — the 4× cache reduction from g=4 sharing multiplies with the ~4× reduction from GQA that's already baked into the architecture.

This modularity insight has practical significance for system designers. It means that investments in different efficiency techniques are not zero-sum — improving the depth-wise sharing robustness of a model doesn't reduce the headroom available for quantization or temporal eviction. Each technique attacks a different term in the memory product, and they can be developed and deployed independently. This is a conceptual reframing of the KV cache optimization landscape from a collection of competing methods to a set of orthogonal, composable optimizations.

The paper doesn't experimentally demonstrate the full stack (R-CLA + temporal eviction + quantization simultaneously), which it acknowledges as a limitation, but the factorization provides the theoretical justification for expecting compound gains, and the existing results confirm that depth-wise sharing and GQA stack cleanly. This turns the paper's contribution from a single-point solution into a principled call for multi-axis optimization, with depth-wise sharing filling the previously underutilized axis.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on two distinct setups. For pre-training, a subset of the OpenWeb corpus (Gokaslan et al., 2019) is used with a fixed context length of 2,048 tokens. For fine-tuning, the authors curate a combined question-answering dataset from five sources: HotpotQA (multi-hop reasoning), SQuAD v2 (includes unanswerable questions), MSMarco (machine reading comprehension), TriviaQA (knowledge-intensive QA), and RepLiQA (fictional content requiring strict context retrieval rather than parametric knowledge). All fine-tuning data consists of (context, question, answer) triples. The authors apply data augmentation: input component order is randomized with 0.5 probability (context → question vs. question → context), and for HotpotQA specifically, three variations are generated per example with different permutations of concatenated context passages to prevent position bias.
Base model(s). Three model families are evaluated in the fine-tuning setting: Llama-3.1-8B (Dubey et al., 2024), Mistral-7B (Jiang et al., 2023), and Qwen3-8B (Yang et al., 2025a). These span three distinct pre-training recipes and attention configurations. For pre-training from scratch, a Qwen3-1.7B-style decoder-only Transformer is used with the standard Qwen3 architecture (28 layers, Grouped-Query Attention with 8 KV heads). For inference efficiency benchmarks, a Qwen3-8B-scale architecture is used: 36 layers, 4096 hidden dimension, 32 attention heads, 8 KV heads, head dimension 128, bfloat16 precision.
Metrics. For pre-training, the primary metric is evaluation loss (cross-entropy on held-out data from the OpenWeb corpus). For fine-tuning, the primary metric is F1 score on each QA dataset, with Exact Match (EM) and ROUGE-L reported in Appendix D (Table 6) as secondary metrics. For inference efficiency, KV cache memory (MB), peak GPU memory (MB), time-to-first-token (TTFT) in milliseconds, and decode throughput in tokens per second are measured.
Baselines. The paper compares against several baselines:
- Standard self-attention models — the same base architectures fine-tuned or pre-trained with identical data, hyperparameters, and compute budget but using only standard self-attention (no cross-layer routing during training). These are labeled "Base" in all result tables.
- Shallower Transformers — for pre-training, baseline models with fewer layers (7, 14, 17, 21, or 28 layers) where the number of layers corresponds to the effective cache size under R-CLA. For example, an R-CLA model with 28 layers and ~14 effective cached layers is compared against a 14-layer baseline Transformer.
- Deterministic CLA (CLA@k) — fixed sharing schemes where groups of k layers share KV states deterministically during training (e.g., layers 0 to k−1 share one KV). This is compared in the ablation study (Section 4.2.3, Table 3).
- Random-Deterministic CLA (RD-CLA@k) — the same fixed sharing scheme as CLA@k but with stochastic application: at each forward step, a coin flip determines whether the layer attends to the fixed shared layer or its local KV. This isolates the effect of randomness within a fixed pattern (Table 3).
Generation budget / compute accounting. For pre-training, all models are trained on an identical token budget of 34 billion tokens, which the paper notes is Chinchilla-optimal for a 1.7B-parameter model. For fine-tuning, all models train for 50,000 steps with batch size 128 and maximum input length 8,192 tokens — identical compute budgets per experiment. For inference efficiency, the "generation budget" is not the relevant metric; instead, cache sharing configurations are defined by the group size g (number of consecutive layers sharing one cache), and measurements are taken at equal context lengths and batch sizes.
Cross-validation / statistical protocol. The paper does not report a formal cross-validation protocol. The test evaluations for fine-tuning are performed on the standard test splits of the five QA datasets. For pre-training, evaluation loss is reported on held-out validation data from the OpenWeb corpus. No confidence intervals or statistical significance tests are reported for the fine-tuning F1 scores, which is a methodological limitation for a paper making claims about performance improvements across five datasets and three model families. The pre-training experiments appear to use single training runs (Table 1 and Figure 5 show single curves, not averages over seeds), so the reported evaluation losses should be understood as point estimates without variance quantification.

Main Quantitative Results

The pre-training experiments (Section 4.1) test two claims: (1) R-CLA does not destabilize training, and (2) deeper models with cache sharing outperform shallower models with full caching at equal memory budgets.

Training stability (Table 1, Figure 6 in Appendix A). The evaluation loss for Qwen3-1.7B pre-trained with varying R-CLA probability p shows minimal degradation:

`p`	Eval Loss
0.00 (standard)	2.424
0.25	2.428
0.50	2.441
0.60	2.446
0.75	2.461

The loss increases by less than 2% even at p = 0.75, where three-quarters of attention operations use cross-layer routing. Figure 6 in Appendix A shows smooth training loss curves across all p values, confirming that R-CLA "incurs no training instability even under aggressive p, and hyperparameters transfer reasonably well from standard self-attention models to R-CLA ones." This is a non-trivial finding because introducing stochasticity into the attention routing could plausibly create optimization difficulties — the fact that standard hyperparameters work without modification is practically important.

Depth with sharing vs. reduced depth (Figure 5). The key architectural finding: across all cache sizes tested, R-CLA models using the full 28-layer architecture with depth-wise cache sharing consistently achieve lower evaluation loss than shallower baseline Transformers trained with the same token budget. The paper states this concisely: "Deeper models with shared cache consistently outperform shallower models, showing that cache sharing preserves the benefit of depth."

This result directly challenges the implicit assumption that removing layers and sharing caches are equivalent ways to reduce memory. The deeper model provides more total computation (more query projections, more feed-forward network transformations) even when attention inputs are shared, and this additional computation translates to better predictions at equal memory cost — but only when the model has been trained with R-CLA to tolerate the sharing.

Headline results (Table 2). The fine-tuning experiments test three models (Llama-3.1-8B, Mistral-7B, Qwen3-8B) across five QA datasets at three cache retention levels: 100% (full cache), 50% (cache every 2nd layer), and 25% (cache every 4th layer). The results demonstrate two distinct phenomena: catastrophic degradation of base models under cache sharing, and preservation or improvement of R-CLA models at full retention.

Cache sharing degradation in base models: At 25% retention, base models collapse to near-zero performance on most tasks. For Qwen3-8B on HotpotQA, F1 drops from 0.233 (100%) to 0.011 (25%); on TriviaQA, from 0.146 to 0.005; on SQuAD v2, from 0.705 to 0.042. For Mistral-7B, similar patterns: HotpotQA drops from 0.215 to 0.031; MSMarco from 0.310 to 0.069; RepLiQA from 0.649 to 0.084. Llama-3.1-8B is somewhat more robust but still degrades severely: HotpotQA drops from 0.203 to 0.080; MSMarco from 0.301 to 0.047.

The magnitude of these drops is important to internalize: these are not marginal degradations but near-total loss of function. When a base model is forced to share caches at 25% retention, it effectively cannot perform the QA task at all. This confirms the paper's central premise: standard training creates a rigid layer-specific KV dependency that breaks catastrophically when caches are removed, and post-hoc sharing without training adaptation is not viable at aggressive retention levels.

R-CLA preserves performance under cache sharing: At 25% retention, R-CLA models retain substantial capability where base models fail. The relative improvements (∆% column in Table 2) are massive, but the absolute numbers tell the more important story. For Llama-3.1-8B with R-CLA at 25% retention: HotpotQA F1 = 0.237 (vs. 0.080 for base), MSMarco = 0.217 (vs. 0.047), RepLiQA = 0.307 (vs. 0.073), SQuAD v2 = 0.628 (vs. 0.257), TriviaQA = 0.291 (vs. 0.137). These are not small numbers — R-CLA at 25% retention frequently outperforms the base model at 100% retention (e.g., Llama-3.1-8B on HotpotQA: R-CLA at 25% gets 0.237 vs. base at 100% gets 0.203).

For Qwen3-8B, the pattern is starker because the base model's collapse is more severe (likely due to differences in pre-training): at 25% retention, R-CLA achieves 0.098 on HotpotQA (base: 0.011), 0.058 on MSMarco (base: 0.027), 0.065 on RepLiQA (base: 0.029), 0.284 on SQuAD v2 (base: 0.042), and 0.131 on TriviaQA (base: 0.005). While these absolute numbers are lower than Llama-3.1-8B's (consistent with different base model capabilities), the preservation of function relative to the base model under identical degradation conditions is dramatic.

Full-retention performance improvement (regularization effect): At 100% retention, R-CLA frequently outperforms standard fine-tuning. The most striking gains are on HotpotQA: Llama-3.1-8B shows +51.1% (0.203 → 0.306), Qwen3-8B shows +53.1% (0.233 → 0.357), Mistral-7B shows +12.4% (0.215 → 0.242). On SQuAD v2: Llama-3.1-8B +30.0% (0.583 → 0.758), Mistral-7B +30.8% (0.573 → 0.750). On TriviaQA for Qwen3-8B: +101.7% (0.146 → 0.295).

Some tasks show minimal change or slight degradation at full retention: RepLiQA for Llama-3.1-8B (−0.9%, 0.655 → 0.649) and Mistral-7B (−0.2%, 0.649 → 0.648). The paper attributes the pattern to a "regularization-like effect from stochastic training, beneficial in data-constrained settings" (Section 4.2.2). The training dynamics evidence in Appendix B (Figure 7) supports this: R-CLA models learn more slowly and delay the onset of overfitting compared to standard fine-tuning.

Consistency across model families: The three model architectures show qualitatively consistent patterns — R-CLA outperforms base at reduced retention and either matches or exceeds base at full retention — but there are quantitative differences in magnitude. Llama-3.1-8B shows the largest absolute performance and strongest full-retention gains. Qwen3-8B shows the most dramatic relative improvements at reduced retention (because its base model degrades most severely). Mistral-7B shows more modest but consistent improvements. The paper does not analyze why these differences exist, though they likely reflect differences in pre-training data, architecture details, and the degree of layer-specific specialization in the original models.

Multi-metric consistency (Appendix D, Table 6): The F1 trends are replicated across Exact Match (EM) and ROUGE-L. For Llama-3.1-8B on HotpotQA at 25% retention: EM improves from 0.035 (base) to 0.153 (R-CLA, +339.6%), ROUGE-L from 0.088 to 0.240 (+171.6%). The pattern holds across all model-dataset-retention combinations, confirming that the F1 improvements reflect genuine answer quality improvements rather than metric artifacts.

Headline finding: R-CLA's unstructured randomness consistently outperforms deterministic sharing patterns (CLA@k) and stochastic variants of fixed patterns (RD-CLA@k) under varying cache retention levels. At a fixed retention level matching what a deterministic model was trained for, CLA@k can match or exceed R-CLA (e.g., CLA@4 on TriviaQA at 100% retention: 0.376 vs. R-CLA 0.360; CLA@2 on SQuAD v2 at 25%: 0.397 vs. R-CLA 0.628 — actually worse, illustrating that even at its "matching" retention, deterministic CLA often underperforms). But when retention changes, CLA@k degrades sharply while R-CLA maintains performance.

The key comparisons from Table 3 on Llama-3.1-8B:

At 100% retention (full cache): CLA@4 performs best on HotpotQA (0.331 vs. R-CLA 0.306) and TriviaQA (0.376 vs. 0.360), while R-CLA performs best on MSMarco (0.318 vs. best CLA 0.323 for RD-CLA@2 — effectively a tie), and CLA@4 matches R-CLA on SQuAD v2 (0.768 vs. 0.758). The performance is broadly comparable at full retention, with no single method dominating. This suggests that at full cache, the regularization benefit comes primarily from the information bottleneck of sharing itself, not from the randomness of the sharing pattern.

At 50% retention: R-CLA dominates. On HotpotQA: 0.305 (R-CLA) vs. 0.149 (CLA@2) vs. 0.134 (RD-CLA@4). On MSMarco: 0.324 (R-CLA) vs. 0.184 (CLA@2) vs. 0.211 (RD-CLA@4). On RepLiQA: 0.597 (R-CLA) vs. 0.237 (CLA@2) vs. 0.270 (RD-CLA@4). On SQuAD v2: 0.740 (R-CLA) vs. 0.387 (CLA@2) vs. 0.521 (RD-CLA@2, the best non-R-CLA). The gaps are large and consistent.

At 25% retention: The pattern persists and intensifies. HotpotQA: 0.237 (R-CLA) vs. 0.089 (CLA@2) vs. 0.071 (RD-CLA@4). MSMarco: 0.217 (R-CLA) vs. 0.065 (CLA@2) vs. 0.076 (RD-CLA@4). RepLiQA: 0.307 (R-CLA) vs. 0.087 (CLA@2) vs. 0.076 (RD-CLA@4). SQuAD v2: 0.628 (R-CLA) vs. 0.290 (CLA@2) vs. 0.313 (RD-CLA@4).

The role of randomness within fixed patterns (RD-CLA vs. CLA): RD-CLA@k generally outperforms CLA@k at retention levels different from what it was trained for, but still substantially underperforms R-CLA. For example, at 50% retention on HotpotQA: RD-CLA@2 = 0.179 vs. CLA@2 = 0.149 — adding stochasticity to the fixed pattern helps, but not enough to close the gap to R-CLA (0.305). This isolates the benefit of exposing the model to a diverse set of sharing patterns during training (R-CLA's unstructured randomness), not just stochastic application of a single pattern (RD-CLA's coin flip on a fixed structure).

Visual summary (Appendix C, Figure 8): The F1 vs. cache retention curves for all five variants across all five QA tasks show R-CLA (red solid line) maintaining a relatively flat performance slope from 100% to 25% retention, while CLA@k and RD-CLA@k variants (dashed lines) show sharp degradation as retention moves away from their trained level. R-CLA is the only method that maintains competitive performance across all retention levels from a single training run.

Interpretation: The ablation cleanly demonstrates that randomness during training serves two distinct roles. The first is creating an information bottleneck (KV sharing itself), which provides regularization and can improve full-retention performance — this is shared by deterministic CLA. The second is building robustness to arbitrary sharing patterns at deployment time, which requires the model to see many different (l, l') pairs during training — this requires the unstructured randomness of R-CLA. The paper's central claim that R-CLA produces a single model adaptable to arbitrary deployment-time sharing strategies is directly supported by this ablation: no deterministic or semi-stochastic variant achieves comparable robustness across retention levels.

Inference Efficiency Benchmarks (Tables 4 and 5)

KV cache memory reduction (Table 4): The primary practical benefit of depth-wise cache sharing is demonstrated empirically. At 8K input tokens with batch size 1 on a Qwen3-8B-scale architecture:

Baseline (g=1, full per-layer caching): KV cache = 1,170 MB, peak memory = 19,319 MB.
Cache sharing with g=4: KV cache = 293 MB (4× reduction), peak memory = 18,455 MB (864 MB saved).

The KV cache reduction scales linearly with group size: at 8K tokens, going from g=1 to g=4 reduces the cache from 1,170 to 293 MB; extrapolating to g=8 would yield ~146 MB. The peak memory savings are smaller than the KV cache savings because peak memory includes model parameters, activations, and other buffers that are not reduced by cache sharing. However, the savings grow with context length: at 32K tokens, g=4 saves 3,456 MB (30,305 → 26,849 MB peak memory).

Throughput improvement (Table 4): At 8K context with batch size 1, decode throughput improves from 34.0 tok/s (baseline) to 41.6 tok/s (g=4), a 22.4% increase. This improvement comes from skipping key and value projections on non-leader layers, not from reduced memory traffic in the attention computation itself (the paper notes that attention computation loads K,V from HBM regardless of whether it's a leader's or shared cache — a backend optimization to keep shared K,V in SRAM across consecutive layers would further improve this).

The throughput gains are consistent across context lengths: at 2K, throughput improves from 32.9 to 43.0 tok/s (+30.7%); at 16K, from 34.2 to 41.1 tok/s (+20.2%); at 32K, from 22.8 to 26.1 tok/s (+14.5%). The relative gain decreases at longer contexts because the attention computation itself (which is O(T^2) in the pre-fill and memory-bandwidth-bound in sampling) becomes a larger fraction of total time, and attention computation is not reduced by cache sharing — only the K/V projection overhead is eliminated.

Time-to-first-token (TTFT): TTFT shows minimal improvement under cache sharing: at 8K context, 297 ms (baseline) → 286 ms (g=4), a 3.7% reduction. This is expected because the pre-fill phase is compute-bound rather than memory-bound: the dominant cost is the O(T^2) attention computation over all prompt tokens in parallel, and skipping K/V projections on some layers is a small fraction of this compute. The paper acknowledges this implicitly by not emphasizing TTFT as a primary benefit of depth-wise cache sharing.

Batch size scaling (Table 5): The most practically significant result: at 8K context with batch size 16, the baseline configuration (g=1) runs out of memory on an 80GB GPU, while cache sharing with g=4 completes successfully (peak memory 60,306 MB, throughput 8.0 tok/s). This directly demonstrates that depth-wise cache sharing enables higher serving capacity on the same hardware — if a deployment needs to serve 16 concurrent requests at 8K context, it literally cannot do so without cache sharing (or other memory reduction techniques). At batch size 8, the baseline consumes 44,897 MB peak memory with throughput 12.8 tok/s, while g=4 uses 37,985 MB (15.4% less) with throughput 14.7 tok/s (14.8% higher).

The throughput scaling with batch size shows diminishing returns: at batch size 2, g=4 achieves 40.3 tok/s vs. baseline 33.3 tok/s (+21.0%); at batch size 4, 25.9 vs. 22.7 (+14.1%); at batch size 8, 14.7 vs. 12.8 (+14.8%). The throughput improvement is maintained across batch sizes but doesn't compound — it's a roughly constant ~15-22% gain from skipping K/V projections, with the absolute throughput limited by the growing memory bandwidth pressure of loading larger caches for larger batches.

Ablation Studies and Robustness Checks

R-CLA probability p in pre-training (Table 1): The evaluation loss increases monotonically but gently with higher p — from 2.424 (p=0, standard) to 2.461 (p=0.75) — representing less than 2% degradation despite 75% cross-layer attention probability. Training remains stable across all values. The paper does not report fine-tuning results at different p values; all fine-tuning uses p=0.6. This is a gap: we don't know whether p=0.6 is optimal for fine-tuning, or whether different models/tasks benefit from different p.

Deterministic vs. stochastic CLA (Table 3, Appendix C): As discussed in detail above, the ablation demonstrates that unstructured randomness (R-CLA) is necessary for deployment-time flexibility across retention levels, while deterministic sharing (CLA@k) can match R-CLA only at its specifically trained retention level. The stochastic-within-fixed-pattern variant (RD-CLA@k) provides intermediate results, confirming that both the diversity of sharing patterns and the randomness of application contribute to robustness.

Training dynamics under R-CLA fine-tuning (Appendix B, Figure 7): For both Qwen3-8B and Llama-3.1-8B, R-CLA (p=0.6) shows consistently higher training loss throughout fine-tuning compared to standard training. For Llama-3.1-8B, the standard model's validation loss begins increasing after approximately 400 steps (overfitting), while R-CLA's validation loss remains flat or continues decreasing. For Qwen3-8B, both models show decreasing validation loss, but R-CLA trains more slowly — consistent with the pre-training observation that R-CLA incurs a small efficiency cost in terms of tokens needed to reach the same loss. The paper interprets this as evidence of a regularization-like effect: "In a data-constrained fine-tuning setting, this means R-CLA models can train for more epochs before entering the overfitting regime."

Pre-training training curves (Appendix A, Figure 6): The training loss curves for Qwen3-1.7B with varying p values (0, 0.25, 0.5, 0.6, 0.75) are smooth and well-behaved, with no divergence or loss spikes. The curves are nearly parallel after an initial transient, indicating that the optimization dynamics are qualitatively similar across p values — R-CLA doesn't introduce new optimization challenges, just a slightly higher asymptotic loss (consistent with the harder learning problem of training with randomized attention routing).

Multi-metric evaluation (Appendix D, Table 6): The F1 trends are replicated across Exact Match (EM) and ROUGE-L. The relative improvements are often larger for EM than F1 (e.g., Llama-3.1-8B on HotpotQA at 25% retention: EM +339.6% vs. F1 +196.2%), suggesting that R-CLA's benefits are particularly pronounced for exact answer matching rather than partial overlap. ROUGE-L improvements closely track F1 improvements, as expected for extractive QA tasks. The consistency across metrics rules out the possibility that the F1 gains are an artifact of a specific scoring method.

Model family robustness: The fine-tuning results span three distinct model families (Llama-3.1, Mistral, Qwen3) with different pre-training recipes, architectures, and scales (7B-8B parameters). The qualitative patterns are consistent: R-CLA preserves performance under cache sharing across all models, and frequently improves full-retention performance. The quantitative differences (Llama-3.1 showing largest absolute gains, Qwen3 showing largest relative gains at reduced retention) suggest that R-CLA's benefits may depend on the base model's pre-training characteristics, but the paper does not investigate this systematically.

Task diversity: The five QA datasets span multi-hop reasoning (HotpotQA), reading comprehension with unanswerable questions (SQuAD v2), knowledge-intensive QA (TriviaQA), standard machine reading (MSMarco), and fictional content requiring strict context grounding (RepLiQA). R-CLA's benefits are consistent across these task types, suggesting that the approach generalizes across different reasoning requirements rather than being specific to a particular task format. However, all tasks are extractive or generative QA over provided contexts — we have no evidence about R-CLA's behavior on tasks that don't involve context retrieval (e.g., open-ended generation, summarization, classification).

Critical Assessment

Claim 1: "R-CLA allows dropping 50-75% of layers' caches with dramatically less degradation than baseline models"

What the experiments demonstrate: Table 2 and Figure 2 provide strong evidence for this claim. At 50% retention, R-CLA models maintain performance close to full-retention levels across all model-dataset combinations, while base models degrade substantially. At 25% retention, the gap widens — base models often collapse to near-zero performance while R-CLA models retain substantial capability. The ablation in Table 3 further confirms that this robustness is specific to the stochastic training; deterministic sharing patterns do not achieve comparable retention-flexibility.

What the experiments do NOT demonstrate: The paper evaluates only three retention levels: 100%, 50%, and 25%. We don't know the shape of the degradation curve between these points — is performance roughly linear in retention, or is there a threshold effect where degradation accelerates below some critical retention? We also don't know whether the model can handle even more aggressive sharing (e.g., 12.5% retention, or a single shared cache for all layers) — the pre-training experiments with p=0.75 suggest the model can operate with heavy sharing, but no inference-time evaluation at >75% cache reduction is reported.

What would strengthen this claim: Evaluating at finer granularity of retention levels (e.g., every 10% from 100% to 10%) to characterize the degradation curve. Testing extreme retention (single cache for all layers) to find the fundamental limit. Reporting statistical confidence intervals on the F1 scores to establish whether the claimed "dramatically less degradation" is statistically reliable.

Claim 2: "At full retention, R-CLA frequently preserves or improves performance compared to standard full-cache baselines"

What the experiments demonstrate: The full-retention column in Table 2 shows consistent improvements on HotpotQA (+51% for Llama-3.1, +53% for Qwen3, +12% for Mistral), SQuAD v2 (+30% for Llama-3.1, +31% for Mistral, +3% for Qwen3), and TriviaQA (+10% for Llama-3.1, +6% for Mistral, +102% for Qwen3). MSMarco and RepLiQA show smaller or negligible gains. The training dynamics in Figure 7 provide a plausible mechanism: R-CLA delays overfitting in data-constrained fine-tuning.

What the experiments do NOT demonstrate: The "regularization-like effect" interpretation, while plausible, is not directly tested. We don't know whether the full-retention improvement is due to (a) better generalization from stochastic training, (b) more effective use of the same training data because the model is forced to learn depth-invariant representations, or (c) some other mechanism. The paper does not run a control experiment that isolates the regularization effect — for example, comparing R-CLA at full retention against standard training with explicit regularization (dropout, weight decay tuning, data augmentation) to see if the same gains can be achieved through other means.

Additionally, the "frequently" qualifier is doing work: RepLiQA shows slight degradation for Llama-3.1 (−0.9%) and Mistral (−0.2%), and Qwen3-8B on SQuAD v2 shows only +3.3%. The benefits are task-dependent and model-dependent in ways the paper doesn't analyze. The claim that R-CLA "preserves or improves performance" is technically true (it never substantially degrades), but the magnitude of improvement varies from negligible to dramatic, and the paper provides no predictive framework for when to expect large vs. small gains.

What would strengthen this claim: Controlled experiments comparing R-CLA to standard training with matched regularization (e.g., increased dropout, weight decay tuning). Analysis of which task or model characteristics predict the magnitude of full-retention improvement. Multiple random seeds to confirm the gains are not due to favorable initialization.

Claim 3: "Deeper models with shared caches consistently outperform shallower baselines under identical memory budgets"

What the experiments demonstrate: Figure 5 shows this for pre-training on a 1.7B-scale model across multiple cache sizes. The result is clean and the comparison is fair: same token budget, same effective cache memory, different depth (full depth with sharing vs. reduced depth without sharing). The finding is architecturally significant.

What the experiments do NOT demonstrate: This result is established only for pre-training on a 1.7B-scale model with the OpenWeb corpus. We don't know whether it generalizes to larger models (8B+) or to fine-tuning settings. The paper does not run the equivalent comparison for Llama-3.1-8B or Qwen3-8B — i.e., training a shallower version of these models on the QA fine-tuning data and comparing against the full-depth R-CLA version at equal cache budgets. We also don't know whether the advantage of depth-with-sharing over reduced-depth holds across different training budgets — it's tested only at 34B tokens (Chinchilla-optimal for 1.7B). At much larger token budgets (overtraining), the advantage might diminish if the shallower model can compensate with more training.

Additionally, the evaluation metric is perplexity (evaluation loss). We don't know whether the perplexity advantage translates to downstream task performance — a shallower model with higher perplexity could still perform comparably on specific tasks if the extra depth primarily helps with aspects of language modeling that don't transfer to the tasks of interest.

What would strengthen this claim: Extending the depth-vs-sharing comparison to larger models and fine-tuning settings. Evaluating on downstream tasks, not just perplexity. Testing across multiple training budgets to see if the advantage persists or diminishes.

What the experiments demonstrate: Tables 4 and 5 directly measure these quantities on a Qwen3-8B-scale architecture. The 4× KV cache reduction and ~22% throughput improvement at 8K context are clearly documented. The batch size scaling results (Table 5) showing that cache sharing enables larger batch sizes on fixed hardware are robust and practically important.

What the experiments do NOT demonstrate: The throughput measurements are for a single specific implementation that the paper acknowledges is a "conservative lower bound." The paper speculates about backend optimizations (keeping shared K,V in SRAM, fusing attention computations) but doesn't implement or measure them. The actual throughput gains with optimized kernels could be substantially different — potentially larger (if memory bandwidth savings from loading the cache once per group are realized) or smaller (if the fused attention computation becomes compute-bound rather than memory-bound).

The experiments use a single GPU (80GB H100). Multi-GPU deployment scenarios (tensor parallelism, pipeline parallelism) might show different scaling behavior because KV cache distribution across devices changes the memory bottleneck pattern. The paper also doesn't measure end-to-end latency for real requests — only TTFT and per-token decode throughput. A user-facing metric like time-to-completion for a full generation would integrate pre-fill and decode costs and provide a more complete picture.

The paper's claim of orthogonality with temporal eviction and quantization is asserted but not experimentally verified. We don't see measurements of R-CLA combined with KV quantization or temporal eviction — the 4× memory reduction is from depth-wise sharing alone, and whether this 4× multiplies cleanly with quantization's 4× to yield 16× total reduction is plausible but untested.

What would strengthen this claim: Implementing and measuring the suggested backend optimizations. Testing on multi-GPU configurations. Measuring end-to-end generation latency for representative workloads. Demonstrating compound savings from combining R-CLA with KV quantization and/or temporal eviction.

Overall Experimental Design Assessment

Strengths:

The three-model, five-dataset fine-tuning evaluation provides good coverage and demonstrates robustness across architectures and tasks.
The ablation study (Table 3) cleanly isolates the role of randomness from the role of sharing, providing causal evidence for the paper's central mechanism.
The inference efficiency benchmarks use realistic model scales (8B-class architecture) and show practically meaningful metrics (peak memory, batch size limits, throughput).
The depth-vs-sharing pre-training comparison is a well-designed architectural test that challenges a common assumption.

Weaknesses and gaps:

No statistical rigor: Single training runs without confidence intervals means we cannot assess whether the reported F1 improvements are statistically significant or within noise. For a paper making claims about performance improvements across many model-dataset combinations, this is a significant omission.
Limited retention granularity: Only three retention levels (100%, 50%, 25%) are tested. The shape of the degradation curve — and whether there are cliff effects — is unknown.
No combination with other KV cache reduction methods: The paper's claim of orthogonality with GQA, quantization, and temporal eviction is not experimentally validated. We don't know whether R-CLA's benefits persist or change when combined with these techniques.
Fine-tuning evaluation is QA-only: While QA is a strong test of context retention (since answers must be extracted from the provided context), we have no evidence about R-CLA's behavior on other task types — summarization, code generation, classification, dialogue — that might have different sensitivity to cache sharing.
No evaluation at scale extremes: The paper doesn't test very long contexts (32K+ tokens for fine-tuning evaluation), very large models (70B+), or very aggressive retention (<25%). These are precisely the regimes where KV cache memory is most problematic, and where R-CLA's claims would be most impactful.
Fixed fine-tuning hyperparameters: All fine-tuning uses p=0.6 and 50,000 steps. We don't know whether these are optimal, or whether different tasks/models benefit from different p values.
The "regularization" interpretation is correlational: The training curves show slower learning and delayed overfitting, but this is observation, not mechanism. No experiment isolates whether the full-retention improvement is due to better generalization, better use of training data, or some other factor.
No comparison to temporal eviction baselines: The paper motivates depth-wise sharing as complementary to temporal eviction but doesn't compare against temporal eviction methods (H2O, SnapKV, PyramidKV) to establish that depth-wise sharing provides benefits beyond what temporal eviction alone achieves.

Conditions on claims:

The claim that R-CLA "allows dropping 50-75% of layers' caches" holds for the specific QA tasks and models tested, but we cannot extrapolate to arbitrary tasks or retention levels.
The full-retention performance improvement claim holds "frequently" — it is task-dependent and model-dependent in ways the paper does not predict or explain. Some tasks (RepLiQA) show negligible benefit.
The depth-vs-sharing architectural claim holds for 1.7B-scale pre-training on OpenWeb; generalization to other scales, training budgets, or downstream tasks is untested.
The inference efficiency claims hold for the specific hardware configuration tested (single H100, Qwen3-8B-scale, bfloat16); generalization to other hardware or multi-GPU setups is plausible but unverified.

6. Limitations and Trade-offs

The assumption or constraint. R-CLA requires training the model with stochastic KV routing — either during pre-training from scratch or through fine-tuning of an existing model. The paper is explicit about this requirement in its Limitations section:

"Enabling R-CLA requires access to training resources; future research might investigate post-hoc methods or lightweight adapters that can induce similar cross-layer robustness without full parameter updates."

The method provides no mechanism for applying cache sharing to an already-trained model without further training. A practitioner with a deployed model (e.g., Llama-3.1-8B-Instruct) who wants to reduce KV cache memory cannot apply R-CLA without running a fine-tuning procedure on task-specific data, which requires access to training data, compute resources, and expertise that may not be available in all deployment contexts.

The consequence. This creates a chicken-and-egg problem for many practical deployments. The motivation for depth-wise cache sharing is reducing inference costs on already-deployed models. But R-CLA's primary mechanism is training-time intervention — you need to train the model before you can benefit from cache sharing. For organizations that do not train or fine-tune their own models (e.g., users of API-hosted models, teams deploying open-weight models as-is), R-CLA is not directly applicable. The paper's statement that future work could investigate "post-hoc methods or lightweight adapters" is an acknowledgment that the current method does not address the most common deployment scenario: taking an existing pre-trained model and applying cache sharing at inference time without training.

What evidence exists in the paper. The catastrophic degradation of base models at reduced retention (Table 2) is the negative evidence: without R-CLA training, models fail at cache sharing. The positive evidence — R-CLA fine-tuning restores performance — comes at the cost of running a full fine-tuning procedure on curated task-specific data (50,000 steps, batch size 128, 8,192-token contexts — computationally non-trivial for 7-8B parameter models). The paper does not experiment with cheaper adaptation methods (e.g., training only adapter layers, or using a small number of fine-tuning steps). The minimum amount of training required to induce cache sharing tolerance is unknown.

Mitigation status. The paper acknowledges this as an explicit limitation (quoted above) and suggests post-hoc methods and lightweight adapters as future work. No experiments explore training-free or low-cost adaptation. For practitioners, this limitation means R-CLA is best suited for organizations that already fine-tune models for their tasks and can add stochastic KV routing to their training pipeline, rather than for teams looking to optimize inference of models they use as-is.

Fine-Tuning Evaluation Is Restricted to a Single Task Family with Modest Context Lengths

The assumption or constraint. All fine-tuning evaluations are conducted on question-answering tasks with a maximum input length of 8,192 tokens. The paper justifies this choice by arguing that QA tasks "critically depend on parsing and retaining information provided in the input prompt" and that "QA serves as the perfect proxy for assessing whether R-CLA renders shared cache models able to perform at parity with models that have access to the full context representation" (Section 4). However, the assumption that robustness on QA transfers to other task types, longer contexts, or generation-heavy workloads is untested.

The consequence. The claim that R-CLA produces models robust to cache sharing is empirically supported only for extractive QA over contexts up to 8K tokens. We do not know whether the same robustness holds for:

Long-context tasks (32K–128K tokens): At these lengths, the KV cache bottleneck is most severe, but the attention patterns that matter for long-range retrieval may be more sensitive to cache sharing. A token relevant to a query at position 30,000 might require precise layer-specific key representations that get lost when caches are shared across large depth gaps.
Generative tasks (summarization, open-ended dialogue, code generation): These require maintaining coherent state across many generated tokens, where small representation errors from cache sharing could compound across autoregressive steps in ways that don't affect single-answer QA.
Tasks requiring parametric knowledge: RepLiQA was included specifically because it uses fictional content, forcing models to retrieve from context rather than relying on memorized knowledge. But this is only one dataset — the interaction between cache sharing and the model's ability to integrate parametric knowledge with context is unexplored.
Multi-turn conversations: Cache sharing could affect the model's ability to maintain conversational state across turns, especially when earlier turns' KV states are shared across layers and later queries need to retrieve specific details.

What evidence exists in the paper. The inference efficiency benchmarks (Section 4.3) test memory and throughput up to 32K tokens, but these are engineering measurements (how much memory is saved, how fast tokens are generated) — they do not evaluate whether the quality of model outputs at 32K context under cache sharing is comparable to full-cache quality. The fine-tuning experiments that actually measure output quality stop at 8,192 tokens. The paper does not report any perplexity or task performance metrics at contexts longer than 8K tokens under cache sharing.

Mitigation status. Not addressed. The paper's Limitations section states: "Our fine-tuning evaluation is focused on QA tasks; while these are a strong proxy for context retention under cache sharing, broader task evaluation would strengthen the conclusions." This is an honest acknowledgment but does not resolve the uncertainty. The framing of QA as a "strong proxy" is a hypothesis, not a demonstrated fact, and the paper provides no evidence that QA robustness predicts robustness on other task types. A practitioner deploying R-CLA for code generation or long-document summarization would need to run their own evaluation since the paper provides no guidance on whether the QA results generalize.

The assumption or constraint. The memory and throughput numbers in Tables 4 and 5 are measured at group size g=4 (caching every 4th layer, 25% retention). These efficiency gains are engineering measurements — they tell us how much memory is saved and how much faster tokens are generated when the cache sharing is applied. But they do not incorporate the task performance cost of that sharing. The paper presents the efficiency results separately from the task performance results, and the two sets of numbers are never combined into a single tradeoff analysis.

The consequence. A practitioner reading the paper might conclude that depth-wise cache sharing delivers 4× memory reduction and ~22% throughput improvement. But those numbers come from the hardware benchmarks (Section 4.3), which measure any model running with g=4 sharing — they don't depend on whether the model was R-CLA-trained. The task performance numbers (Table 2) show that an R-CLA-trained model at 25% retention does maintain substantial capability compared to a base model at 25% retention, but Table 2 also shows that even R-CLA models degrade somewhat at reduced retention. For Llama-3.1-8B at 25% retention on RepLiQA, F1 drops from 0.649 (100%) to 0.307 (25%) — that's still a 53% degradation from full-retention performance, even if it's dramatically better than the base model's 89% degradation (0.649 → 0.073). The 4× memory savings and the 22% throughput gain come at a real task performance cost that is not zero.

For some datasets, the degradation is modest (SQuAD v2: 0.758 → 0.628, a 17% relative drop at 25% retention for Llama-3.1-8B). For others, it is substantial (RepLiQA: 53% drop). For Qwen3-8B, the absolute performance at 25% retention — while much better than the collapsed base model — is low in absolute terms (0.098 on HotpotQA, 0.058 on MSMarco, 0.065 on RepLiQA). Whether these performance levels are acceptable depends on the application, but the paper does not provide guidance on this tradeoff.

What evidence exists in the paper. The efficiency benchmarks (Tables 4 and 5) report memory, TTFT, and throughput at different group sizes. The task performance results (Table 2, Figure 2) report F1 scores at different retention levels. These are in separate sections and are never combined into a unified efficiency-vs-accuracy tradeoff curve. Figure 3 shows cache size vs. F1 for different models and retention levels, which is the closest the paper comes to unifying these dimensions, but this figure plots cache size (an architectural quantity) against F1, not actual throughput or peak memory against F1. The engineering benchmarks use a Qwen3-8B-scale architecture that may not exactly match the Llama-3.1-8B and Mistral-7B models used in the task evaluations, making it unclear whether the throughput numbers from Section 4.3 apply to the same models that achieve the F1 scores in Table 2.

Mitigation status. The paper does not explicitly address this as a limitation, but the structure of the evaluation separates efficiency claims from quality claims in a way that could mislead a casual reader. Figure 3 does show cache size vs. F1 simultaneously (with R-CLA dominating the Pareto frontier), but "cache size" in KB/token is an architectural constant, not a direct measure of deployment efficiency (which depends on batch size, context length, and hardware-specific memory bandwidth). A more complete analysis would measure actual throughput and peak memory against actual task F1 for the specific model-task combinations evaluated, producing a deployment-relevant efficiency-quality frontier. The current presentation makes it easy to read "4× memory reduction" and "preserves performance" in adjacent sentences without noticing that "preserves" means "preserves relative to the collapsed base model under the same conditions," not "preserves relative to full-retention R-CLA performance."

No Demonstration That Difficulty Estimation or Adaptive Strategies Are Unnecessary — the Method Is Static

The assumption or constraint. R-CLA produces a single model that can be deployed with any fixed, static cache sharing strategy S chosen at deployment time based on available hardware memory. The strategy is uniform: every n-th layer is cached, regardless of the input, the task, or the generation step. The cache sharing is "set and forget" — once you choose g=4 for a deployment, all inputs and all generation steps use the same sharing pattern.

The consequence. This uniformity likely leaves efficiency on the table. The paper's own analysis of inter-layer redundancy (citing Monteiro et al., 2024a; Wu and Tu, 2024) suggests that different layers contribute differently to task performance, and it is unlikely that a uniform sampling of every 4th layer is optimal. Some layers might be more critical for certain types of information processing (e.g., early layers for surface features, middle layers for syntactic structure, late layers for semantic integration), and losing their specific KV states could hurt more than losing others. A non-uniform strategy — caching more layers in critical depth ranges and fewer in redundant ones — might achieve better performance at the same memory budget, or equivalent performance at a smaller budget. Additionally, the optimal sharing pattern might depend on the input: a simple factual query might need fewer cached layers than a complex multi-hop reasoning problem, and a dynamic strategy that adapts the sharing based on input difficulty or generation phase (pre-fill vs. sampling) could outperform a static one.

The paper never experiments with non-uniform or dynamic strategies. The mapping function μ(l) always uses the "nearest preceding cached layer" rule (Equation 3.2), which is simple and efficient but assumes that proximity in layer index is the best predictor of representational compatibility. This may not be true — two layers far apart in depth might have more similar KV distributions than adjacent layers, especially if the model has learned specialized functions at different depths.

What evidence exists in the paper. No experiments test non-uniform cache sharing strategies, dynamic strategies, or input-dependent strategies. All results use uniform retention (every 2nd layer, every 4th layer). The paper's ablation (Table 3) tests different group sizes (g=2, g=4) but these are still applied uniformly — the comparison is between different uniform strategies, not between uniform and non-uniform ones. The pre-training experiments (Figure 5) effectively test uniform sharing at various granularities by varying p, but again within the uniform paradigm. The paper does not cite or discuss work on layer importance profiling that could inform non-uniform strategies, nor does it analyze which layers' caches are most critical to retain.

Mitigation status. Not addressed as a limitation. The paper frames the flexibility of R-CLA as a strength — "a single model adaptive to arbitrary sharing strategies at inference time" — but the experiments only explore a narrow subset of the strategy space (uniform retention). The ability to handle arbitrary strategies is demonstrated only in the sense that the model tolerates different uniform retention levels; whether it would also tolerate non-uniform patterns is plausible but untested. A practitioner who wants to push memory savings further might find that a non-uniform strategy works better, but the paper provides no guidance on how to design such a strategy or whether R-CLA training makes the model robust to them. This is a missed opportunity, since the stochastic training should in principle prepare the model for any pattern of (l, l') pairs, not just uniform ones — but this hypothesis is not tested.

Training Overhead Is Amortized Over Inference Savings Only Under High-Volume Deployment

The assumption or constraint. R-CLA training incurs a computational cost — either pre-training from scratch with stochastic KV routing (Section 4.1) or fine-tuning for 50,000 steps (Section 4.2) — that must be amortized over the inference savings from cache sharing. The paper does not account for this training cost in any of its efficiency calculations. The inference benchmarks (Section 4.3) measure savings during inference but do not factor in the cost of obtaining the model that enables those savings.

The consequence. The net benefit of R-CLA depends on the ratio of inference volume to training cost. For a model that will serve billions of tokens over its lifetime (high inference-to-training ratio), the fine-tuning cost is negligible and the inference savings dominate. For a model that will serve a small number of queries (e.g., a research prototype, a model evaluated on a fixed benchmark, a deployment with low traffic), the fine-tuning cost could exceed the inference savings. The paper's fine-tuning experiments use 50,000 steps with batch size 128 and max sequence length 8,192 — a rough calculation: 50,000 × 128 × 8,192 ≈ 52 billion tokens processed during fine-tuning. If inference savings from cache sharing amount to a ~22% throughput improvement, you would need to serve approximately 52B / 0.22 ≈ 236 billion tokens at inference just to break even on the fine-tuning compute, assuming the cost per token is the same during training and inference (which it isn't — inference is cheaper per token but the order of magnitude still suggests non-trivial break-even volume).

This calculation is approximate and depends on many factors (hardware, model size, whether fine-tuning hyperparameters could be reduced), but the paper provides no breakeven analysis or guidance on when the training cost is justified. The implicit assumption is that the training cost is always worth paying, which is true for high-volume production deployments but not for all use cases.

What evidence exists in the paper. The paper provides all the numbers needed for a breakeven calculation (50,000 fine-tuning steps, batch size 128, max length 8,192, ~22% throughput improvement) but does not perform the calculation itself. The training dynamics in Appendix B (Figure 7) show that R-CLA fine-tuning trains more slowly than standard fine-tuning — the training loss is higher throughout — which means the 50,000 steps of R-CLA training may be less "efficient" in terms of loss reduction per step than standard training, though they produce a more robust model. The pre-training experiments (Table 1) show a small but real perplexity cost from R-CLA training (2.424 → 2.461 at p=0.75), meaning that to achieve the same perplexity as a standard model, an R-CLA model would need more training tokens — but the paper doesn't quantify how many more.

Mitigation status. Not addressed. The paper does not discuss training cost amortization, breakeven analysis, or the inference volume required to justify fine-tuning. The Limitations section mentions that "enabling R-CLA requires access to training resources" but frames this as a resource availability issue, not an efficiency tradeoff. A practitioner deciding whether to adopt R-CLA needs to know not just that the fine-tuning is possible, but whether it is worth it given their inference volume. The paper provides no tools for making this assessment.

The Method Assumes the Base Model's Query Projections Can Learn to Be Depth-Invariant — but This May Not Hold for All Architectures or Pre-Training Regimes

The assumption or constraint. R-CLA's central mechanism — training query projections Q_l to work with key-value states from arbitrary earlier layers — assumes that the information content in (K_{l'}, V_{l'}) is sufficiently general that a later layer's query can extract what it needs regardless of which earlier layer produced the representations. This is an assumption about the representational similarity across layers in transformer decoders: that the features at layer 5 and layer 15, while computed at different depths, encode information in a format that is sufficiently compatible with the same query projection.

The consequence. If this assumption fails — if transformer layers develop qualitatively different kinds of representations at different depths (e.g., early layers encoding positional and surface features, late layers encoding abstract semantic features in a fundamentally different "format") — then R-CLA training faces an impossible task: a later layer's query projection must learn to extract abstract semantic information from early-layer representations that simply don't contain it. The R-CLA training loss would remain high, and the model would either fail to learn the task or learn to ignore the cross-layer attention and rely solely on the feed-forward network at each layer for depth-specific processing.

The paper's results suggest this is not a fatal problem for the models and tasks tested — R-CLA training converges and produces functional models. But the degree to which this assumption holds likely varies across architectures (different widths, depths, attention configurations), pre-training regimes (different data mixtures, training objectives), and model scales. A model pre-trained with very deep layers (e.g., 70B+ parameters with 80+ layers) might develop more pronounced representational discontinuities across depth than the 7-8B models tested. A model pre-trained with a different architecture (e.g., parallel attention + FFN rather than sequential, or with different normalization placements) might have different cross-layer representational compatibility. The paper's experiments cover three model families at similar scales (7-8B parameters) — we have no evidence about how this assumption holds at different scales or for architecturally different models.

What evidence exists in the paper. The pre-training loss at different p values (Table 1) provides indirect evidence: even at p=0.75 (where 75% of attention operations use cross-layer routing), the evaluation loss degrades by less than 2% compared to standard training. This suggests that for this specific architecture (Qwen3-1.7B-style) and training setup (34B tokens on OpenWeb), the representational similarity across layers is high enough that R-CLA doesn't catastrophically impair learning. But this is a single data point — we don't know whether other architectures would show larger degradation, and we don't know whether the 2% perplexity degradation at p=0.75 translates to acceptable downstream task performance (the paper doesn't evaluate downstream tasks for the pre-trained models).

The fine-tuning results (Table 2) show that R-CLA generally improves full-retention performance — which is evidence against the assumption being a problem, since if cross-layer representations were incompatible, R-CLA training would hurt rather than help. But the improvements are task-dependent and model-dependent (RepLiQA shows negligible benefit; Qwen3-8B shows massive relative gains in part because its base model performs poorly to begin with), suggesting that the ease of learning depth-invariant representations varies across model-task combinations in ways the paper doesn't analyze.

Mitigation status. Not addressed as a limitation. The paper does not analyze cross-layer representational similarity, measure how compatible KV states are across different depth gaps, or investigate whether certain layers are more "shareable" than others. The uniform random sampling of l' from {1, ..., l-1} (Section 3.2) treats all earlier layers as equally valid sources of KV states, which implicitly assumes complete representational compatibility. A more nuanced approach might weight the sampling distribution to favor layers with more similar representations, or might adapt the sampling based on measured representational similarity during training. The paper's conclusion that "future work could explore adaptive strategies that dynamically adjust the cache depth based on the complexity of the input query or the current generation step" acknowledges the possibility of more sophisticated strategies but doesn't address the more fundamental question of whether all layers' KV states are equally suitable for sharing.

7. Implications and Future Directions

How This Work Changes the Landscape

R-CLA introduces a conceptual reframing rather than a paradigm shift: it converts depth-wise KV cache sharing from an architectural design choice (fixed at training time) into a runtime configuration parameter (selectable at deployment). This distinction matters because it changes what it means to "solve" the KV cache problem. Prior methods — CLA (Brandon et al., 2024), LISA (Mu et al., 2024), the systematic study by Wu et al. (2025), post-hoc merging like MiniCache (Liu et al., 2024) — all produce models tied to a single sharing pattern. The optimal pattern for an H100 cluster with 80GB of HBM may differ from what an edge device with 8GB can support, and these methods require separate training or adapter-training runs per target. R-CLA produces one model that can be deployed at 100%, 50%, or 25% retention (and plausibly any level in between) with the same weights. This is a deployment flexibility argument, not just an accuracy argument.

The reframing has a second dimension: R-CLA demonstrates that stochastic training perturbations can induce invariance to inference-time resource constraints. This principle — train with randomized ablations of the resource you want to conserve, then select the conservation level at deployment — extends beyond KV caches. The same logic could apply to attention head pruning, FFN dimension reduction, or activation quantization: instead of training separate models for each efficiency target, train once with stochastic resource sampling and deploy at whatever level the hardware supports. The paper doesn't explore these extensions, but the conceptual template is clear and broadly applicable.

The paper also provides empirical resolution to a tension in prior depth-wise sharing work. Several prior methods demonstrated feasibility (XC-Cache, Layer-Condensed KV Cache) but at the cost of additional components (separate encoders, sequential pre-filling, learned adapters) that hurt throughput or latency. Others demonstrated simplicity (deterministic CLA, post-hoc merging) but with limited compression or sharp degradation outside the trained pattern. R-CLA shows that simple architectural reuse (no new components) + stochastic training achieves both substantial compression (50-75% cache reduction) and deployment flexibility without the throughput penalties that plagued prior methods (Tables 4-5 show 22% throughput improvement, not degradation). This synthesis of simplicity and effectiveness redirects attention: the bottleneck was never the sharing architecture itself — it was the training procedure's failure to prepare the model for sharing. Future research on depth-wise efficiency should focus on training methodology rather than on designing more complex sharing mechanisms.

The regularization-like effect at full retention (Table 2: +51% F1 on HotpotQA for Llama-3.1-8B, +53% for Qwen3-8B, +30% on SQuAD v2) is a secondary but potentially influential finding. It suggests that standard transformer training creates overly specific layer-wise dependencies that hurt generalization, and that R-CLA's stochastic routing is a form of structured regularization targeting depth-specificity. This opens a new axis for regularization research — not just "add noise" or "drop units," but "break the rigid layer-to-layer feature alignment that standard training encourages." Whether this regularization effect transfers to pre-training at scale (beyond the 1.7B model tested) or to tasks beyond QA is unknown, but the magnitude of the effect in data-constrained fine-tuning (50,000 steps on modestly-sized QA datasets) is large enough to warrant serious investigation.

One research direction becomes less attractive as a result of this work: designing fixed, one-size-fits-all sharing patterns and training separate models for each. The ablation (Table 3) shows that deterministic CLA@k can match R-CLA at its trained retention level, but degrades sharply when retention changes. Since real-world deployments face heterogeneous hardware constraints, a method that requires per-target training is inherently less practical than one that produces a single flexible model. Research effort is better spent on improving the stochastic training recipe (better sampling distributions, curriculum learning over sharing ratios, adaptive p scheduling) than on optimizing deterministic sharing patterns. The paper similarly casts doubt on post-hoc sharing without training adaptation — base models collapse at 25% retention (Table 2), confirming that layer-specific feature alignments are real and breaking them requires training intervention.

Follow-Up Research This Work Enables

Characterize the minimum effective training budget for inducing cache sharing tolerance. The paper fine-tunes for 50,000 steps with batch size 128 — a substantial compute investment (~52B tokens processed). A critical practical question is how much training is necessary: can 5,000 steps achieve 80% of the benefit? Can LoRA adapters trained for a few hundred steps induce sufficient robustness? The training dynamics in Figure 7 show that R-CLA learns slowly and delays overfitting, suggesting that the stochastic routing creates a harder learning problem that may genuinely require more steps. A well-designed experiment would take a fixed base model (e.g., Llama-3.1-8B), apply R-CLA fine-tuning for {1K, 5K, 10K, 25K, 50K} steps, and evaluate cache sharing robustness at each budget using the same five QA datasets. This would produce a training budget vs. cache sharing tolerance curve that tells practitioners exactly how much compute they need to invest for their desired retention level. A parallel experiment could test parameter-efficient methods (LoRA, IA3, prompt tuning) — do small adapter updates suffice, or does the model need full-weight training to reorganize query projections for depth-invariance?

Test whether R-CLA-trained query projections are genuinely depth-invariant or merely tolerant of a narrower distribution shift. The paper's mechanism claim is that R-CLA forces Q_l to learn to extract information from KV states regardless of which earlier layer produced them. This claim is testable: take an R-CLA-trained model, compute the representational similarity (e.g., CKA, PWCCA, or mutual information) between (Q_l) and (K_{l'}, V_{l'}) for all pairs l, l', and compare against a standard-trained model. If R-CLA genuinely induces depth-invariance, the similarity should be more uniform across l' (the query should be equally compatible with keys from layer 5 and layer 15). If R-CLA only teaches tolerance of a narrow distribution shift (e.g., the query learns to handle the specific K distributions from the layers most commonly sampled during training), similarity would still vary with l'. This experiment is low-cost (no additional training needed — just analyze the already-trained models) and would provide mechanistic evidence for or against the paper's central claim. A strong follow-up could also probe which layers benefit most from cache sharing: do early layers' KV states contain information that is inherently more "shareable" (e.g., surface features used by many later layers) while late layers' KV states are more specialized? The uniform random sampling in R-CLA (l' ~ Uniform{1, ..., l-1}) treats all earlier layers as equally valid sources. If representational similarity is not uniform, a weighted sampling scheme during training — where l' is sampled proportional to some similarity metric — might improve robustness or reduce the training budget needed.

Evaluate R-CLA on long-context tasks (32K+ tokens) where the KV cache bottleneck is most severe. The paper's fine-tuning experiments cap at 8,192 tokens, but the inference benchmarks (Tables 4-5) show that the memory benefits of cache sharing grow with context length (at 32K tokens, g=4 saves 3.5 GB peak memory). The critical unknown is whether the quality of the model's outputs under cache sharing degrades at long contexts. Long-context retrieval exhibits different attention patterns than short-context QA — models learn to attend to sparse, relevant tokens across large distances (the "needle in a haystack" phenomenon). If cache sharing blurs the key representations at the specific layers responsible for long-range retrieval, performance could degrade more severely at 32K tokens than at 8K. A follow-up should evaluate R-CLA-trained models on standard long-context benchmarks (e.g., LongBench, ∞BENCH, RULER, or needle-in-a-haystack tasks) at context lengths of 8K, 16K, 32K, and 64K tokens, under varying cache retention levels (100%, 50%, 25%). The key metric is not just F1 at a given length, but the degradation slope — how much faster does performance drop with context length under cache sharing compared to full cache? If the degradation slope is steeper for shared-cache models, that would bound the practical applicability of depth-wise sharing for long-context deployments. This experiment would also reveal whether R-CLA training needs to be conducted at long context lengths to induce robustness — the paper's fine-tuning uses 8K contexts, and it's plausible that models need to experience long-range attention with stochastic KV routing during training to handle it at inference.

Demonstrate that R-CLA stacks multiplicatively with KV quantization and temporal eviction. The paper's central architectural claim is that depth-wise sharing is orthogonal to other KV cache reduction axes (Section 2, Section 5). This claim is theoretically justified by the factorization of cache memory into independent multiplicative factors, but it is experimentally untested. A direct experiment: take a model fine-tuned with R-CLA, apply KV cache quantization (e.g., INT4 via KVQuant-style methods) and temporal eviction (e.g., SnapKV or PyramidKV), and measure both task performance and memory footprint at the combined compression level. For example, with R-CLA at 25% retention (4× depth compression) plus INT4 quantization (4× precision compression) plus 50% temporal eviction (2× sequence compression), the total theoretical reduction is 4 × 4 × 2 = 32×. Does the model actually achieve 32× memory reduction while maintaining useful performance, or do the methods interact negatively (e.g., quantization errors compounding with cross-layer representational mismatch)? The paper's Qwen3-8B-scale inference benchmarks (Section 4.3) already measure memory and throughput for depth-wise sharing alone; extending these measurements to include quantization and temporal eviction would either validate the orthogonality claim or reveal previously hidden interactions. A negative result (worse-than-multiplicative degradation) would be equally valuable — it would reveal that the "independent axes" framing is an oversimplification and that joint optimization of these techniques is needed.

Test R-CLA on decoder-only architectures that interleave attention with non-attention layers (MoE, SSM hybrids). The paper states that "we have not evaluated R-CLA on Mixture-of-Experts (MoE) architectures, though the method should be directly applicable since it only modifies the attention KV source" (Limitations). This is worth testing, but the more interesting question is whether R-CLA's benefits extend to hybrid architectures where not all layers have KV caches to begin with. Models like Kimi-k1.5 (Team et al., 2025) and Nemotron-Nano-9B-v2 (Basant et al., 2025) interleave standard attention layers with SSM layers that require no cache. In such architectures, the attention layers are spaced farther apart, meaning that when a later attention layer shares a cache with an earlier one, the depth gap is larger (in terms of total model depth). If R-CLA's robustness depends on the representational similarity between the sharing layers, larger depth gaps could make the learning problem harder. A follow-up should take a pre-trained hybrid model, apply R-CLA fine-tuning, and evaluate whether the same retention levels (50%, 25%) are achievable when attention layers are interleaved with non-attention blocks. Additionally, the interaction between R-CLA and MoE routing is interesting: in MoE transformers, different tokens may be processed by different experts at each layer, potentially creating token-dependent KV representations that are harder to share across layers. Does R-CLA training need to account for expert routing stochasticity as well as layer stochasticity? This experiment would stress-test the generality of the stochastic training principle beyond the homogeneous decoder architectures tested in the paper.

Investigate whether R-CLA's full-retention performance gains translate to pre-training at scale and to generative tasks. The +30-50% F1 improvements at full retention during fine-tuning (Table 2) are large but observed only in a specific regime: data-constrained fine-tuning of 7-8B models on QA tasks. Two critical extensions: (1) Pre-training at scale: If R-CLA is applied during the pre-training of a large model (e.g., 70B+ parameters, trillion-token budgets), does the regularization effect persist, or does the abundance of pre-training data wash it out? The pre-training experiment in the paper (Qwen3-1.7B, 34B tokens, evaluation loss only) shows a small perplexity penalty from R-CLA (2.424 → 2.461 at p=0.75), which might translate to worse downstream performance if the perplexity gap doesn't close with more tokens. A follow-up should pre-train models at a meaningful scale (e.g., 7B parameters on 200B+ tokens) with and without R-CLA, evaluate on standard benchmarks (MMLU, HellaSwag, ARC, etc.), and determine whether the full-retention benefit observed in fine-tuning is a data-constrained phenomenon or a general property of stochastic training. (2) Generative tasks: All evaluations are on extractive QA. On tasks where the model generates long-form text (summarization, story generation, dialogue), KV cache sharing could interact with autoregressive generation dynamics differently — small errors in key-value representations at early generation steps could compound across hundreds of generated tokens, producing drift that isn't captured by single-answer F1 metrics. A follow-up should fine-tune with R-CLA on summarization (CNN/DailyMail, XSum) or open-ended generation tasks and evaluate both automatic metrics (ROUGE, BERTScore) and human judgments of coherence and factual consistency under varying cache retention.

Practical Applications and Downstream Use Cases

Enabling higher batch sizes on fixed hardware for throughput-critical QA deployments. Table 5 demonstrates the most immediately actionable finding: at 8K context, batch size 16, the baseline configuration (g=1, full per-layer caching) runs out of memory on an 80GB GPU, while cache sharing (g=4) completes successfully with 8.0 tok/s throughput. For a production QA system serving thousands of queries per minute — common in enterprise search, customer support, or document analysis — this directly translates to serving more concurrent users on the same GPU fleet. With R-CLA fine-tuning (Table 2), the quality cost at 25% retention is task-dependent: for SQuAD v2, F1 drops from 0.758 to 0.628 (17% relative) while enabling 4× larger batches; for MSMarco, from 0.318 to 0.217 (32% relative). A practitioner can consult Figure 3 or Table 2 for their specific task and decide whether the throughput gain justifies the accuracy tradeoff. The key operational benefit: a single R-CLA-trained model can be deployed at 100% retention during low-traffic periods (when latency matters more than throughput) and at 25% retention during peak load (when throughput matters more than per-query accuracy), without swapping model weights or maintaining multiple serving configurations.

On-device or edge deployment of LLMs for context-grounded applications. The paper highlights that R-CLA-trained models can be deployed on "edge devices retaining only a fraction of [the cache]" (Section 3.2). Mobile devices, laptops, and embedded systems often have 4-16GB of total RAM (not HBM), making KV cache memory a hard constraint even for quantized models. A 7B-parameter model already occupies ~14GB in FP16 for weights alone, and a full KV cache for a modest 4K-token context could add several GB, exceeding device limits. R-CLA at 25% retention reduces the KV cache by 4× — for Llama-3.1-8B, from approximately 4.9 GB to 1.2 GB for a 40K-token context (Figure 1). Combined with 4-bit weight quantization (which reduces the 14GB of weights to ~3.5GB), the total memory footprint for an 8B model with cache sharing at 4K context could fit within 8GB, making on-device deployment feasible for applications like document-grounded QA, meeting transcription summarization, or privacy-sensitive personal assistants that must process context locally. The paper's RepLiQA results (Table 2: Llama-3.1-8B at 25% retention achieves 0.307 F1 vs. 0.655 at full cache) provide a realistic estimate of the quality tradeoff for context-grounded tasks.

Cost reduction for batch inference pipelines processing large document collections. Organizations that run batch inference over large corpora — e.g., summarizing millions of documents, extracting structured data from legal contracts, or scoring candidate answers — are throughput-bound and memory-sensitive. The paper's throughput improvement (~22% at 8K context, Table 4) from skipping K/V projections on non-leader layers directly reduces cost per document, and the peak memory savings enable processing longer documents on the same GPUs. For a batch pipeline processing 1 million 16K-token documents with a 7B model: baseline throughput at batch size 8 is 12.8 tok/s (Table 5), while g=4 sharing achieves 14.7 tok/s and uses 6.9 GB less peak memory, potentially allowing batch size increase to 16 (which is impossible for the baseline, as Table 5 shows OOM). This could reduce total processing time by 25-40% depending on the batch size scaling, directly translating to lower GPU-hour costs. The fine-tuning cost (50,000 steps) is amortized over millions of inference queries, making the training investment negligible. The key practical consideration: the document processing task must be one where R-CLA fine-tuning data can be curated — the paper uses QA, but the same principle applies to summarization or extraction if task-specific fine-tuning data is available.

Flexible deployment across heterogeneous hardware fleets without maintaining multiple model versions. Cloud inference providers and organizations with mixed GPU fleets (e.g., some H100 nodes, some A100 nodes, some T4 nodes for low-priority workloads) face a model management problem: a model tuned for peak performance on H100s may exceed memory limits on A100s, and training separate models per GPU type fragments evaluation, monitoring, and deployment pipelines. R-CLA offers a solution: ship one model, configure the cache retention at deployment time based on available GPU memory. An H100 with 80GB can run at 100% retention for maximum quality; an A100 with 40GB can run at 50% retention; a T4 with 16GB can run at 25% retention — all from the same model checkpoint. The paper's Figure 2 and Table 2 provide the quality-at-retention curves needed to set these deployment parameters. The operational simplification is significant: one training run, one model registry entry, one set of evaluation benchmarks, and runtime configuration toggles rather than model swaps for different hardware tiers.

Dependency tree

Papers this one builds on, walking backwards through the same domain and shared topics.

Accelerating Speculative Decoding with Block Diffusion Draft Trees2026· inference-optimization← in-domain predecessor
- DMax: Aggressive Parallel Decoding for dLLMs2026· inference-optimization← in-domain predecessor
- DFlash: Block Diffusion for Flash Speculative Decoding2026· inference-optimization← similar topic
- DEER: Draft with Diffusion, Verify with Autoregressive Models2025· inference-optimization← similar topic
- TurboDiffusion: Accelerating Video Diffusion Models by 100-200 Times2025· inference-optimization← similar topic

What it influenced

Later papers in the same chain. Direction reversed.

UniPrefill: Universal Long-Context Prefill Acceleration via Block-wise Dynamic Sparsification2026· inference-optimization→ in-domain predecessor
Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding2025· inference-optimization→ similar topic
- Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference2025· code→ LLM-judged similar
- Locality-aware Parallel Decoding for Efficient Autoregressive Image Generation2025· inference-optimization→ LLM-judged similar
- KLASS: KL-Guided Fast Inference in Masked Diffusion Models2025· inference-optimization→ LLM-judged similar
- Fast and Accurate Causal Parallel Decoding using Jacobi Forcing2025· inference-optimization→ LLM-judged similar
- Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speed2025· training-methods→ LLM-judged similar
- DC-VideoGen: Efficient Video Generation with Deep Compression Video Autoencoder2025· vision→ LLM-judged similar
- Efficient Autoregressive Video Diffusion with Dummy Head2026· inference-optimization→ LLM-judged similar
- DreamLite: A Lightweight On-Device Unified Model for Image Generation and Editing2026· multimodal→ LLM-judged similar
- Rectified Sparse Attention2025· inference-optimization→ in-domain predecessor
- Fast Inference from Transformers via Speculative Decoding2022· inference-optimization→ similar topic
- Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads2024· inference-optimization→ similar topic
- Break the Sequential Dependency of LLM Inference Using LOOKAHEAD DECODING2024· inference-optimization→ similar topic