LLM Systems2024~109 minarXiv:2401.03462

Soaring from 4K to 400K: Extending LLM's Context with Activation Beacon

score 9Peitian Zhang, Zheng Liu, Shitao Xiao +3↑ 28Jan 7, 2024cs.CL

🎯 Pitch

Compressing a 128K context into a handful of 'beacon tokens'—without ever looking at the user's question—achieves performance matching the full, uncompressed model. This method slashes KV cache memory by 8× and doubles inference speed while progressive compression avoids the information bottleneck that crippled prior soft-prompt approaches.

1. Executive Summary

This paper proposes Activation Beacon, a plug-in module for transformer-based LLMs that enables effective, efficient, and flexible compression of long contexts by introducing a new special token⟨b⟩ and progressively condensing raw input into these beacon tokens' activations (keys and values at every layer) through a chunked, fine-grained compression workflow. Evaluated on Llama-2-7B and Qwen-2-7B across long-context tasks — including LongBench, Needle-in-a-Haystack, and Multi-Needle-in-a-Haystack — Activation Beacon achieves a 2× acceleration in inference time and an 8× reduction of KV cache memory while maintaining performance comparable to the uncompressed baseline, even compressing contexts up to 128K despite being trained only on sequences under 20K. The method also supports flexible compression ratios via chunk-wise random compression ratio sampling during training, establishing that query-independent, activation-based compression can match full-context performance across diverse long-context scenarios without requiring re-encoding overhead or question-dependent token pruning.

2. Context and Motivation

The Core Problem: LLMs Need Long Context But Can't Afford It

The fundamental tension this paper addresses is deceptively simple: modern LLMs are increasingly capable of processing long contexts, but doing so exacts a punishing computational and memory cost that scales quadratically with length. This creates an uncomfortable tradeoff: either truncate the context and risk losing critical information, or bear the full cost and face prohibitive latency and GPU memory requirements.

To appreciate why this matters, consider what "long context" actually means in practice. When an LLM processes a 128K-token document — a full-length book chapter, a legal contract, a codebase — it must compute attention between every pair of tokens. For a transformer with hidden size $D$ , the FLOPs required for self-attention alone scales as $O(s^2 \cdot D)$ where $s$ is sequence length. Doubling the context quadruples the attention computation. Beyond FLOPs, there is the KV cache problem: during autoregressive generation, the model stores the keys and values of every preceding token to avoid recomputing them for each new token. At 128K context length with a 7B model, this cache can easily consume tens of gigabytes of GPU memory — often dwarfing the memory required for the model parameters themselves.

The practical consequences are severe:

Cost: Serving long-context models requires high-memory GPUs (e.g., A100/A800 80GB) and still runs into out-of-memory (OOM) errors at extreme lengths.
Latency: Users wait longer for responses on long documents, breaking the interactive experience that makes LLMs useful.
Throughput: Cloud providers can process fewer requests concurrently because each one hogs memory and compute.
Scalability ceiling: Even if an LLM's context window is theoretically extended to 128K or 1M tokens via RoPE interpolation, the hardware may simply not support inference at that length.

The paper explicitly frames this in the introduction:

"On one hand, transformer-based LLMs incur substantial computational costs due to the quadratic complexity of self attention. On the other hand, they require tremendous GPU memory to hold the KV cache of the entire sequence for faster decoding. Both computation and memory costs increase as the context length grows."

This is not a hypothetical concern. The paper's own experiments show that at 128K context with Qwen-2-7B, the uncompressed baseline requires approximately 2.4 teraFLOPs per forward pass (Figure 3B). With Activation Beacon's 8× compression, this drops to roughly 1.2 TFLOPs — half the computation. The end-to-end latency data in Table 2 confirms this translates to tangible speedup: from 4.4 seconds to 2.4 seconds per turn.

Prior Approaches and Where They Fall Short

The paper identifies four families of existing methods for addressing long-context efficiency, each with specific limitations that motivate Activation Beacon's design.

Sparse Attention. Methods like Longformer, Big Bird, and more recent dynamic sparse attention approaches (Minference 1.0, SampleAttention) exploit the observation that attention patterns are naturally sparse — most token pairs have negligible attention weights and can be safely skipped. While this reduces computation, the paper identifies a critical weakness:

"these methods require holding all KV activations on chip to dynamically determine the optimal sparse patterns, making them unsuitable for KV cache reduction."

In other words, sparse attention addresses the FLOPs problem but not the memory problem. The full KV cache must still reside in GPU memory because the sparsity pattern is determined dynamically per attention head and per input — you can't know which tokens to keep until you've computed attention. For memory-constrained deployments, this is a non-starter.

KV Compression. This line of work compresses the KV cache along dimensions other than sequence length. CLA (Cross-Layer Attention) shares KV caches across transformer layers. GQA (Grouped-Query Attention) reduces the number of key/value heads. MLA (Multi-head Latent Attention) compresses the channel dimension. KIVI quantizes the numerical precision of cached values. The paper positions these as orthogonal to its contribution — they compress the cache per token, but don't reduce the number of tokens that need to be cached. Activation Beacon's token-level compression can be combined with any of these dimensional compression techniques.

Token Pruning / Deletion. Methods like LongLLMLingua and SnapKV take a different approach: rather than compressing information into new representations, they simply delete tokens deemed unimportant. The paper offers two criticisms. First, these methods are query-dependent:

"they depend on the input question to accurately estimate the token importance, limiting their efficiency in real-world multi-turn scenarios."

When a user asks multiple questions about the same long document, these methods must re-evaluate which tokens are important for each new question and potentially re-encode the context. Second, and more fundamentally, deleting tokens at high compression ratios (e.g., 8×) "may destroy the coherence of the context and lose important information" (Section 4.2). Intuitively, if you delete 87.5% of tokens, the remaining text may become unreadable — critical connecting words, syntactic structures, and subtle semantic cues can be lost.

Soft Prompt / Summary Compression. This is the most directly related prior work and the primary foil for Activation Beacon. Methods like Gisting, ICAE, and AutoCompressors compress a long context into a small number of "soft tokens" — learned embedding vectors that are supposed to summarize the context's information for downstream generation. The paper identifies four specific failure modes in this approach:

Information bottleneck: "existing methods usually summarize the context into a few soft tokens, which constitute the major bottleneck to summarize the complex information within long contexts." A handful of soft tokens (often 4-16) must encapsulate everything in thousands of raw tokens — an extreme compression ratio that fundamentally limits fidelity, especially for tasks requiring fine-grained retrieval (like Needle-in-a-Haystack where a single fact must be preserved).
"All-at-once" compression: These methods "try to compress the context 'all-at-once,' lacking a fine-grained handling of the detailed information." ICAE and AutoCompressors encode an entire chunk into soft tokens in one shot, meaning the attention scope of each soft token is identical — they all attend to the entire chunk. There is no mechanism for different soft tokens to specialize in different parts of the input.
Re-encoding overhead: "these soft tokens must be re-encoded before generation, resulting in inferior efficiency in both training and inference." In ICAE, after compression, the soft tokens need to be passed through a separate decoder to condition generation. In AutoCompressors, the soft tokens from previous chunks must be re-processed when encoding subsequent chunks. Both approaches add computational overhead that eats into the efficiency gains from compression.
Inflexible compression ratio: "these methods are learned to compress with a fixed number of soft tokens, thus, it's hard to customize the compression ratio for downstream tasks." The number of summary tokens is baked into the model architecture. If a deployment requires 4× compression but the model was trained for 8×, there's no mechanism to adjust without retraining.

The empirical evidence for these limitations is stark in the paper's results. In Table 1, AutoCompressor and ICAE on Llama-2-7B achieve only 12.9 and 19.5 respectively on Single-Doc QA (LongBench), while Activation Beacon achieves 34.9 — nearly matching the uncompressed baseline at 34.8. On Needle-in-a-Haystack (Figure 5), AutoCompressor and ICAE collapse to roughly 2-4 accuracy out of 10 at high compression ratios, while Activation Beacon maintains 8+. This is not a small margin — it suggests the soft-token bottleneck is a fundamental limitation, not an implementation detail.

How This Paper Positions Itself

Activation Beacon is not merely an incremental improvement over existing methods — it is a reconceptualization of what compression should operate on and how it should work. The paper's key positioning claim is that compression should target activations (keys and values at every layer) rather than soft-token summaries, and should do so progressively at fine granularity rather than all-at-once or through token deletion.

This is a conceptual shift with deep implications. Soft tokens are typically a single embedding vector per summary unit — a bottleneck with capacity limited by the hidden dimension $D$ . Beacon tokens' activations, by contrast, store information across all layers and all attention heads. At each layer, a beacon token's key and value vectors are $h_k \times d$ -dimensional (number of key/value heads × head dimension). Across $L$ layers, the representational capacity is $L \times h_k \times d$ — orders of magnitude larger than a single embedding vector. The paper's framing:

"The context is distilled into beacon tokens' activations (i.e. keys and values at every layer), whose capacity are large enough to encapsulate the complex information within long contexts."

This connects to a broader insight about transformer architectures: the KV cache is not merely a computational convenience — it is the mechanism by which transformers store and retrieve information across positions. By compressing into the KV cache itself (rather than into a separate summary representation), Activation Beacon makes the compressed context natively consumable by the LLM's existing attention machinery without any adapter or decoder module.

The paper also positions itself as query-independent, which distinguishes it from SnapKV and LongLLMLingua. This is not merely a convenience — it enables:

Efficient multi-turn conversations (no re-computation of compression for each new question).
Incremental updates (if new text is appended to a document, only the new chunk needs to be compressed, with previous beacon activations reused).
Training with contiguous gradients (since compression doesn't depend on a downstream question, the training signal can flow through all chunks).

The paper explicitly contrasts with recurrent memory methods like RMT and AutoCompressors that "stop the gradients back-propagation at a given chunk number to improve the training efficiency." Activation Beacon's architecture allows gradients to flow naturally through all chunks — a consequence of its use of per-layer beacon activations rather than final-layer outputs as context carriers.

Finally, the chunk-wise random compression ratio during training is positioned as a distinctive innovation that enables a single trained model to serve multiple deployment scenarios without retraining:

"During training, we randomly sample a compression ratio at each step, teaching the model to support a wide range of compression configurations."

This is a practical feature with real deployment implications. A cloud provider can serve the same model checkpoint at 2×, 4×, or 8× compression depending on the customer's latency requirements and accuracy tolerance, without maintaining separate models for each compression level.

3. Technical Approach

3.1 Reader Orientation

Activation Beacon is a plug-in module that retrofits an existing, frozen transformer-based LLM with the ability to compress arbitrarily long input contexts into a compact set of intermediate representations — specifically, the key-value (KV) activations of a small number of newly introduced special tokens called beacon tokens — enabling the model to condition its generation on a drastically shorter sequence without losing the information needed to answer questions, summarize, or retrieve facts from the original long context. The system solves the problem that LLMs face quadratic computational cost and linear memory cost with respect to sequence length by transforming a long input of, say, 128,000 tokens into a much shorter sequence of beacon token activations (e.g., 16,000 at an 8× compression ratio), where each beacon token's KV vectors at every transformer layer have absorbed the information from a specific, fine-grained span of the original text, allowing the frozen LLM to attend to this compressed representation during generation as if it were the original context.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components that operate in a fixed pipeline during inference:

Input Partitioning — The raw long context (which may exceed the LLM's native context window) is split into fixed-size chunks (e.g., 1024 or 2048 tokens), and each chunk is further divided into fine-grained units whose size equals the desired compression ratio. This component determines how many beacon tokens will be needed per chunk and where they will be inserted.
Beacon Token Interleaving — A group of beacon tokens ⟨b⟩ is physically inserted into each chunk's token sequence, with one beacon token placed at the end of every fine-grained unit. These beacon tokens share a single learned token embedding, and their number is determined by the compression ratio: a chunk of 1024 tokens with a compression ratio of 4 will have 1024/4 = 256 beacon tokens interleaved.
Chunk-by-Chunk Encoding with Modified Self-Attention — The LLM processes one chunk at a time, using its existing transformer layers with one modification: in self-attention, the raw tokens in the current chunk attend to each other and to the accumulated beacon token activations from all previously processed chunks, while the beacon tokens attend to the raw tokens in the current chunk (compressing them) and to preceding beacon tokens. After encoding a chunk, the raw tokens' KV activations are discarded; only the beacon tokens' KV activations are retained and accumulated into a running cache.
Accumulated Beacon KV Cache — This is a per-layer, per-head store of all beacon tokens' keys and values from all chunks processed so far. It grows linearly with the number of chunks (not quadratically with total context length), and it serves as the compressed representation of the entire context that the current chunk (and eventual generation) will attend to.
Frozen LLM Generation — After all chunks are processed, the final beacon token's hidden state (from the last chunk's last position) flows through the LLM's standard output projection to produce the next token prediction. The full accumulated beacon KV cache remains available for attention during autoregressive generation, just like a normal KV cache, but is α times smaller.

Information flows as follows: raw context → partition into chunks → interleave beacon tokens into each chunk → encode chunk 1 with self-attention that compresses raw tokens into beacon KVs → discard raw KVs, accumulate beacon KVs → encode chunk 2 attending to accumulated beacon KVs from chunk 1 → repeat for all chunks → use final hidden state for next-token prediction, with accumulated beacon KV cache serving as the compressed context memory.

3.3 Roadmap for the Deep Dive

First, the core compression mechanism — how beacon tokens are inserted, how they compress raw tokens during self-attention, and how their KV activations are accumulated across chunks — because this is the algorithmic heart of the method and everything else (efficiency, training, flexibility) builds on it.
Second, the precise modifications to self-attention (new projection matrices for beacon tokens, the attention mask structure) and the efficiency analysis (FLOPs and KV cache reduction), since these define exactly what is changed in the LLM and what the computational savings are.
Third, the learning method — compression-based auto-regression, no gradient stopping, and chunk-wise random compression ratio sampling — because the training procedure is what makes the compressed representations actually useful for downstream tasks.
Fourth, the key design choices and their justifications (why activations rather than soft tokens, why progressive rather than all-at-once compression, why query-independent), since these motivate why specific alternatives were rejected.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and methods paper whose core idea is that compressing long contexts into the per-layer key-value activations of a small set of interleaved beacon tokens — rather than into soft prompt embeddings or by deleting tokens — enables efficient, flexible, and high-fidelity compression that can be learned with modest training data while preserving the frozen LLM's original capabilities.

The Beacon Token: What It Is and Why Activations

A beacon token ⟨b⟩ is a new token added to the LLM's vocabulary, with its own learned token embedding $e_{\langle b \rangle} \in \mathbb{R}^D$ where $D$ is the model's hidden dimension. Critically, all beacon tokens share the same single embedding vector — there are not separate embeddings for "first beacon token," "second beacon token," etc. The paper states:

"Note that all beacon tokens share the same token embedding, one can use arbitrary number of beacon tokens to achieve the desired compression ratio by repeating."

This is possible because beacon tokens are differentiated not by their input embeddings but by their positions in the sequence and their attention scopes — a beacon token at position 100 attends to different raw tokens than a beacon token at position 200, so their KV activations will naturally encode different information despite starting from the same embedding.

Why compress into activations rather than into a soft prompt or summary embedding? The paper's argument is about representational capacity. A soft token (as used in ICAE or AutoCompressors) is typically a single embedding vector — $D$ dimensions — that must summarize an entire chunk of text. A beacon token, by contrast, stores information at every transformer layer and every attention head. For a model with $L$ layers, $h_k$ key/value heads, and head dimension $d$ , a single beacon token's compressed representation across all layers has dimensions $L \times h_k \times d$ for keys and the same for values. For Llama-2-7B, this is 32 layers × 32 heads × 128 head-dim ≈ 131K dimensions — over 30× the capacity of the model's 4096-dimensional hidden state. The paper states:

"The context is distilled into beacon tokens' activations (i.e. keys and values at every layer), whose capacity are large enough to encapsulate the complex information within long contexts."

Furthermore, compressing into KV activations means the compressed context is natively consumable by the LLM's existing attention mechanism — no separate decoder, adapter, or re-encoding step is needed. When the model generates the next token, it simply attends to the accumulated beacon KV cache using standard scaled dot-product attention, exactly as it would attend to raw token KVs.

The Compression Workflow: Partitioning, Interleaving, and Progressive Encoding

The compression process operates on an input context $X = [x_1, \ldots, x_n]$ that may be arbitrarily long — potentially far exceeding the LLM's native context window $N$ . The workflow has three stages.

Stage 1: Partitioning into Chunks

The input is divided into fixed-size chunks of length $w$ :

$[x_1, \ldots, x_n] \rightarrow [X_1, \ldots, X_{\lceil n/w \rceil}], \quad X_i = [x^i_1, \ldots, x^i_w]$

where $X_i$ is the $i$ -th chunk containing $w$ raw tokens (the last chunk may be shorter, which the paper notes is "omitted for simplicity" in the formulation). The chunk size $w$ is set to 1024 for Llama-2-7B and 2048 for Qwen-2-7B. This chunk size is chosen to be smaller than the LLM's context window (4K for Llama-2, 32K for Qwen-2), ensuring that each chunk can be processed within the model's native attention span even though the total context may be much longer.

Why fixed-size chunks rather than variable-length or semantic boundaries? The paper does not explicitly justify this choice, but the reasoning is implicit in the architecture: fixed-size chunks ensure predictable computation cost per chunk, enable batched processing, and guarantee that each beacon token compresses a consistent amount of information (one fine-grained unit of $\alpha$ tokens, where $\alpha$ is the compression ratio). Semantic splitting would make the compression ratio variable per chunk, complicating the KV cache structure.

Stage 2: Interleaving Beacon Tokens

For each chunk $X_i$ , a compression ratio $\alpha_i$ is determined (the paper defaults to using the same ratio for all chunks at inference time, but the training supports per-chunk variation). The chunk's $w$ raw tokens are split into $k_i = w / \alpha_i$ fine-grained units, each of size $\alpha_i$ . Then one beacon token is inserted at the end of each unit:

$X_i \rightarrow X'_i = [x^i_1, \ldots, x^i_{\alpha_i}, \langle b \rangle^i_1, \ldots, x^i_{w-\alpha_i+1}, \ldots, x^i_w, \langle b \rangle^i_{k_i}]$

The resulting expanded chunk $X'_i$ has $w + k_i$ tokens in total: $w$ raw tokens plus $k_i$ beacon tokens. At an 8× compression ratio with $w = 1024$ , each unit is 8 raw tokens, there are $k_i = 128$ beacon tokens per chunk, and the expanded chunk has 1152 tokens total.

Why interleave beacon tokens within the chunk rather than appending them all at the end? This is a critical design choice that the paper explicitly ablates in Section 4.6. When beacon tokens are appended at the end, they all have identical attention scope — each beacon token attends to the entire chunk, and they are not differentiated by which specific sub-span of the chunk they should focus on. When interleaved, beacon token $\langle b \rangle^i_j$ (at position $j \cdot (\alpha_i + 1)$ within the expanded chunk) has a causal attention scope that includes the first $j$ raw-token units: it can attend to $[x^i_1, \ldots, x^i_{j \cdot \alpha_i}]$ plus all preceding beacon tokens. Beacon token $\langle b \rangle^i_{j+1}$ can additionally attend to the $(j+1)$ -th unit. This differentiated attention scope means each beacon token naturally specializes in compressing a specific, progressively larger prefix of the chunk, enabling fine-grained compression. The ablation in Table 4 shows that removing this interleaving and appending all beacon tokens at the end drops Single-Doc QA performance from 40.5 to 35.2 on Qwen-2-7B — a substantial degradation.

Stage 3: Progressive Chunk-by-Chunk Encoding

The LLM processes one expanded chunk at a time. When encoding chunk $X'_i$ , the input to the LLM consists of:

The accumulated beacon token KV activations from all previously processed chunks $X'_{<i}$ , stored in the accumulated caches $\mathbf{K}^{ac}$ and $\mathbf{V}^{ac}$ .
The raw and beacon tokens of the current chunk $X'_i$ .

The encoding of chunk $X'_i$ at an arbitrary transformer layer proceeds as follows (the paper formalizes this in Equations 4–8):

Step 1: Separate raw and beacon hidden states. The input hidden states to self-attention $\mathbf{H} \in \mathbb{R}^{(w+k_i) \times D}$ are sliced into raw-token states $\mathbf{H}_r$ (positions where the token is not ⟨b⟩) and beacon-token states $\mathbf{H}_b$ (positions where the token is ⟨b⟩).

Step 2: Project to queries, keys, and values using separate projection matrices. Raw tokens use the LLM's original, frozen projection matrices $\mathbf{W}^r_Q, \mathbf{W}^r_K, \mathbf{W}^r_V$ :

$\mathbf{Q}_r = \mathbf{W}^r_Q \mathbf{H}_r, \quad \mathbf{K}_r = \mathbf{W}^r_K \mathbf{H}_r, \quad \mathbf{V}_r = \mathbf{W}^r_V \mathbf{H}_r$

Beacon tokens use newly introduced, learnable projection matrices $\mathbf{W}^b_Q, \mathbf{W}^b_K, \mathbf{W}^b_V$ :

$\mathbf{Q}_b = \mathbf{W}^b_Q \mathbf{H}_b, \quad \mathbf{K}_b = \mathbf{W}^b_K \mathbf{H}_b, \quad \mathbf{V}_b = \mathbf{W}^b_V \mathbf{H}_b$

These new projection matrices $\mathbf{W}^b_* \in \mathbb{R}^{D \times (h_k \cdot d)}$ have the same shape as the original matrices but are randomly initialized and learned during training. They exist at every transformer layer — so for a 32-layer model, there are 32 sets of $\{\mathbf{W}^b_Q, \mathbf{W}^b_K, \mathbf{W}^b_V\}$ . Together with the shared beacon token embedding $e_{\langle b \rangle}$ , these constitute the entirety of the trainable parameters $\Theta_b$ . The LLM's original parameters $\Theta$ are frozen throughout training, which the paper credits for preserving short-context capabilities:

"We conjecture that the primary reason is the LLM's original parameters are frozen throughout the training process."

Why separate projection matrices? The beacon tokens serve a fundamentally different function than raw tokens: raw tokens represent the content being processed, while beacon tokens must learn to absorb and summarize information from surrounding raw tokens. Using the same projection matrices for both would force beacon tokens to operate in the same representational space as raw tokens, limiting their capacity to learn compression-specific transformations. The separate matrices allow beacon tokens to project into a complementary subspace optimized for information aggregation.

Step 3: Scatter back into unified Q, K, V. The raw and beacon queries, keys, and values are scattered back into unified matrices $\mathbf{Q}, \mathbf{K}, \mathbf{V} \in \mathbb{R}^{(w+k_i) \times D}$ according to their original positions in the expanded chunk. Formally, if $\mathcal{I}_r$ is the set of raw-token indices and $\mathcal{I}_b$ is the set of beacon-token indices:

$\mathbf{Q}[\mathcal{I}_r] = \mathbf{Q}_r, \quad \mathbf{Q}[\mathcal{I}_b] = \mathbf{Q}_b$ $\mathbf{K}[\mathcal{I}_r] = \mathbf{K}_r, \quad \mathbf{K}[\mathcal{I}_b] = \mathbf{K}_b$ $\mathbf{V}[\mathcal{I}_r] = \mathbf{V}_r, \quad \mathbf{V}[\mathcal{I}_b] = \mathbf{V}_b$

Step 4: Compute self-attention with accumulated beacon KVs. The standard scaled dot-product attention is computed, but with an important modification: the keys and values include both the current chunk's tokens AND the accumulated beacon KVs from all previous chunks:

$\mathbf{A} = \text{softmax}\left( \frac{\text{mask}\left( \mathbf{Q} [\mathbf{K}^{ac}; \mathbf{K}]^T \right)}{\sqrt{D}} \right), \quad \tilde{\mathbf{V}} = \mathbf{A} [\mathbf{V}^{ac}; \mathbf{V}]$

where $[\cdot; \cdot]$ denotes concatenation along the sequence dimension, $\mathbf{K}^{ac}, \mathbf{V}^{ac} \in \mathbb{R}^{m_{i-1} \times D}$ are the accumulated keys and values from all beacon tokens in chunks $1$ through $i-1$ (with $m_{i-1} = \sum_{j=1}^{i-1} k_j$ being the total number of beacon tokens from previous chunks), and mask is the causal attention mask.

What this attention pattern achieves:

Raw tokens in chunk $i$ attend to: (a) all preceding raw tokens within the same chunk (causal), (b) all beacon tokens within the same chunk that appear before them, and (c) ALL accumulated beacon tokens from previous chunks. They do NOT attend to raw tokens from previous chunks, since those were discarded.
Beacon tokens in chunk $i$ attend to: (a) all raw tokens within the same chunk up to their position (which includes one additional fine-grained unit compared to the previous beacon token), (b) preceding beacon tokens within the same chunk, and (c) ALL accumulated beacon tokens from previous chunks.

This means each beacon token $\langle b \rangle^i_j$ compresses the first $j$ fine-grained units of chunk $i$ into its KV activations, while also having access to the compressed representations of all previous chunks through the accumulated beacon KVs. The compression is therefore both local (each beacon token specializes in a specific span) and contextual (each beacon token is aware of everything that came before it, albeit in compressed form).

Step 5: Discard raw KVs and accumulate beacon KVs. After the self-attention output $\tilde{\mathbf{V}}$ is computed and passed through the rest of the transformer layer (output projection, residual connection, layer norm, MLP), the raw tokens' keys and values $\mathbf{K}_r, \mathbf{V}_r$ are discarded — they will never be needed again. The beacon tokens' keys and values $\mathbf{K}_b, \mathbf{V}_b$ are accumulated by concatenating them to the running cache:

$\mathbf{K}^{ac} := [\mathbf{K}^{ac}; \mathbf{K}_b], \quad \mathbf{V}^{ac} := [\mathbf{V}^{ac}; \mathbf{V}_b]$

After processing all $\lceil n/w \rceil$ chunks, the accumulated beacon KV cache contains $\mathbf{K}^{ac}, \mathbf{V}^{ac} \in \mathbb{R}^{m_{\lceil n/w \rceil} \times D}$ where $m_{\lceil n/w \rceil}$ total beacon tokens represent the entire compressed context. The hidden state at the final position of the final chunk (which is a beacon token) flows through the LLM's language modeling head to predict the next token, and the accumulated beacon KV cache remains available for all subsequent autoregressive decoding steps.

Why progressive rather than all-at-once? The paper identifies several advantages of this chunked, progressive workflow in Section 3.1:

Handles inputs longer than the LLM's context window. Since each chunk is only $w$ tokens (1024 or 2048), the LLM never needs to attend to more than $w + k_i$ tokens at once, even if the total context is 128K tokens. The accumulated beacon KVs from previous chunks provide a compressed summary of everything before the current chunk.
Enables fine-grained compression. Each beacon token has a differentiated attention scope, specializing in specific spans of the chunk rather than trying to summarize everything at once. This is in direct contrast to ICAE and AutoCompressors, where "their compression workflow also lacks fine-grained handling of the chunked inputs, resulting in inferior compression quality."
Facilitates efficient training. Because each layer of chunk $X'_i$ only depends on the previous layer's output of chunk $X'_{i-1}$ (through the accumulated beacon KVs), the computation graph is identical in depth to standard autoregressive LLM training. This means gradients can flow through all chunks without hitting memory limits, unlike recurrent memory methods that must stop gradient propagation.
Avoids re-encoding overhead at inference. Once a chunk is compressed, its beacon KVs are cached and can be reused for all subsequent chunks and all generation steps. ICAE and AutoCompressors require re-encoding soft tokens before generation, which the paper identifies as introducing "extra overhead."
Enables incremental updates. In multi-turn conversations, if new information is added to the context, only the new chunk needs to be compressed — the previous beacon KVs can be reused as-is. This is directly relevant to the multi-turn Needle-in-a-Haystack experiments in Table 2.

Position Encoding

The paper specifies that during self-attention, tokens are encoded using their relative positions in the full expanded sequence. For chunk $i$ :

Queries (the current chunk's tokens) use positions $[m_{i-1}, \ldots, m_{i-1} + w + k_i - 1]$ .
Keys (accumulated beacon KVs + current chunk tokens) use positions $[0, \ldots, m_{i-1} + w + k_i - 1]$ .

where $m_{i-1}$ is the total number of tokens (raw + beacon) in all previous chunks. This is standard Rotary Position Embedding (RoPE) applied to the expanded sequence, which means each beacon token has a unique position that encodes its location in the overall document structure. Beacon tokens from earlier chunks have smaller position indices than those from later chunks, providing the attention mechanism with ordering information about where in the document the compressed information originated.

The paper does not explicitly discuss whether RoPE's relative position encoding degrades at very long distances (a known issue with RoPE extrapolation), but since the maximum query-key position difference within a chunk is $w + k_i$ and between a query and an accumulated beacon is at most the total number of preceding expanded tokens, the position differences can grow large. The empirical Needle-in-a-Haystack results (Figure 4) suggest this is not a problem in practice — the model retrieves needles accurately at 128K despite being trained only up to 20K — implying that the compressed beacon representations are robust to the position encoding extrapolation.

Efficiency Analysis: FLOPs and KV Cache Reduction

The paper provides a detailed FLOPs analysis in Section 3.1 and Appendix B, distinguishing between two sources of computation.

Let $s$ be the current input sequence length, $s_{\text{pst}}$ be the length of cached (preceding) context, $h_q$ be the number of query heads, $h_k$ be the number of key/value heads, $d$ be the head dimension, $D$ be the hidden size, $I$ be the intermediate FFN size, and $V$ be the vocabulary size.

The total forward FLOPs is:

$\text{FLOPs} = F^{\text{Att}}(s, s_{\text{pst}}) + F^{\text{Oth}}(s)$

where $F^{\text{Att}}$ is the FLOPs from self-attention computation (QKV projections, attention scores, weighted sum, and output projection) and $F^{\text{Oth}}$ is the FLOPs from other modules (FFN up/gate/down projections and LM head).

What this equation separates: Self-attention FLOPs depend on both the current sequence length AND the cached context length (because attention must be computed against all cached KVs), while other module FLOPs depend only on the current sequence length (because MLPs and layer norms are applied position-wise).

For a standard (full-attention) model processing a sequence of length $n$ without any KV cache, $s = n$ and $s_{\text{pst}} = 0$ . This means $F^{\text{Att}}$ scales as $O(n^2)$ due to the attention score matrix computation.

For Activation Beacon with chunk size $w$ and compression ratio $\alpha$ (assumed uniform across chunks for simplicity), the FLOPs is summed over $\lceil n/w \rceil$ chunks:

$\text{FLOPs}^{\text{bcn}} = \sum_{i=1}^{\lceil n/w \rceil} \left[ F^{\text{Att}}\left( \frac{(\alpha + 1)w}{\alpha}, \frac{(i-1)w}{\alpha} \right) \right] + F^{\text{Oth}}\left( n + \lceil \frac{n}{\alpha} \rceil \right)$

where:

$\frac{(\alpha+1)w}{\alpha} = w + k_i$ is the length of the expanded chunk (raw + beacon tokens).
$\frac{(i-1)w}{\alpha} = \sum_{j=1}^{i-1} k_j$ is the number of accumulated beacon tokens from previous chunks.
$n + \lceil n/\alpha \rceil$ is the total number of tokens processed across all chunks (raw tokens plus all beacon tokens).

What this equation tells us about where the savings come from:

Self-attention savings: Within each chunk, the attention score matrix is $O((w + k_i) \times ((i-1)k_{i-1} + w + k_i))$ rather than $O(n^2)$ . At 128K context with $w = 2048$ and $\alpha = 8$ , this means the attention is computed over sequences of roughly 2.3K tokens attending to at most 16K accumulated beacon tokens, rather than 128K tokens attending to 128K tokens. The quadratic cost is dramatically reduced.
Other module overhead: The $F^{\text{Oth}}$ term actually increases relative to full attention because the model must process $n + n/\alpha$ tokens (raw + beacon) rather than just $n$ tokens. This is the cost of encoding the beacon tokens through the FFN, layer norm, and other position-wise modules.

The net effect, visualized in Figure 3, is that Activation Beacon consistently reduces total FLOPs, with savings that amplify as context length grows. For Qwen-2-7B at 256K context with 8× compression, FLOPs drop from approximately 0.9 × 10^5 TFLOPs to 0.35 × 10^5 TFLOPs — roughly a 2.6× reduction. The paper claims a 2× acceleration in end-to-end latency (Table 2), which is roughly consistent with the FLOPs reduction accounting for memory bandwidth and other overhead.

KV cache reduction is simpler and more dramatic. The accumulated beacon KV cache stores $m_{\lceil n/w \rceil} = \sum_{i} w/\alpha = n/\alpha$ beacon token KVs, compared to $n$ raw token KVs for full attention. This is an $\alpha$ -fold reduction. At 8× compression, the KV cache memory drops by 8×. For a 128K context with Qwen-2-7B (which uses GQA with 4 KV heads, 128-dimensional head, 28 layers, and 2 bytes per float16 value), the full KV cache would consume 128000 × 28 × 4 × 128 × 2 × 2 ≈ 7.3 GB (times 2 for keys and values). With 8× compression, this drops to approximately 0.9 GB — a savings of about 6.4 GB, which is the difference between fitting in an 80GB GPU and not.

The Learning Method: Compression-Based Auto-Regression

Activation Beacon is trained to optimize the quality of next-token prediction when conditioned on the compressed context. The training objective is:

$\min_{\Theta_b} \sum_{i=2}^{\lceil N/w \rceil} \sum_{j=1}^{w} \Pr\left( x^i_j \mid \langle b \rangle^1_1, \ldots, \langle b \rangle^{i-1}_{k_{i-1}}, x^i_1, \ldots, x^i_{j-1}; \Theta, \Theta_b \right)$

where:

$\Theta$ are the frozen LLM parameters,
$\Theta_b$ are the trainable beacon parameters: the shared beacon token embedding $e_{\langle b \rangle}$ and the per-layer beacon projection matrices $\mathbf{W}^b_Q, \mathbf{W}^b_K, \mathbf{W}^b_V$ at every layer,
$N$ is the total training context length (capped at 20K),
$w$ is the chunk size,
The outer sum starts at $i=2$ because the first chunk has no accumulated beacon KVs to condition on,
The inner sum is over all raw tokens in chunk $i$ , excluding beacon tokens (their labels are set to -100, meaning they are not included in the loss).

What this loss computes operationally: For each raw token in chunks 2 through $\lceil N/w \rceil$ , the model predicts the token given: (1) all beacon tokens from all previous chunks (the compressed representation of everything before chunk $i$ ), and (2) all preceding raw tokens within the current chunk (the local, uncompressed context). The standard cross-entropy loss between the predicted distribution and the true token is minimized by updating only the beacon parameters $\Theta_b$ .

Why exclude the first chunk from the loss? The first chunk has no preceding beacon KVs to condition on, so the prediction of its tokens would not exercise the compression mechanism at all — it would just be standard autoregressive LM training. By starting the loss from chunk 2, every training token forces the model to use the compressed context from at least one preceding chunk, directly optimizing the compression quality.

Why exclude beacon tokens from the loss? Beacon tokens are not meant to be generated — they are compression artifacts. Including them in the loss would force the model to learn to predict beacon token embeddings, which is nonsensical and would waste capacity. Setting their labels to -100 is standard practice in HuggingFace for tokens that should be ignored in loss computation.

Sample efficiency. The paper emphasizes that this loss formulation leads to "high sample efficiency that maximizes the use of training data." This is because nearly every token in the training corpus (except those in the first chunk and the beacon tokens themselves) contributes a training signal. For a 20K-token document with chunk size 1024, that's approximately 19K training tokens from a single example — compared to methods that only train on a special compression bottleneck or require auxiliary reconstruction losses.

No Gradient Stopping

A critical distinction from recurrent memory methods like AutoCompressors and RMT is that Activation Beacon does not stop gradient back-propagation across chunks. The paper explains why this is possible:

"Activation Beacon only depends on the previous-layer outputs of preceding chunks (the encoding of $X'_i$ at layer $l$ only conditions on the results of $X'_{i-1}$ at layer $l-1$ ), which is the same as any auto-regressive LLMs. Thus, the gradients can naturally flow through all chunks to optimize the compression effect over long contexts."

To unpack this: In AutoCompressors, soft tokens from chunk $i-1$ are generated at the final layer and then fed as input to the first layer when encoding chunk $i$ . This creates a computation graph where the depth grows by $L$ layers (the full transformer depth) for each additional chunk, making back-propagation through many chunks prohibitively memory-intensive. Hence, gradients are typically stopped after a few chunks.

In Activation Beacon, the KV activations of beacon tokens at layer $l$ from chunk $i-1$ are consumed at layer $l$ when encoding chunk $i$ — they flow horizontally across chunks within the same layer, not vertically through the layer stack. The computation graph depth for $m$ chunks is $L + m$ rather than $L \times m$ . Since $L \gg m$ for typical training (32 layers vs. ~20 chunks of 1024 tokens in a 20K sequence), this is a modest increase in graph depth that standard training can handle without gradient checkpointing or truncation. This means the model can learn to optimize compression over genuinely long-range dependencies — the beacon KVs from chunk 1 influence the prediction of tokens in chunk 20, and gradients from that prediction can flow all the way back to update how chunk 1's beacon tokens compress information.

Chunk-Wise Random Compression Ratio

During training, the compression ratio $\alpha_i$ for each chunk $i$ is randomly sampled from the set {2, 4, 8, 16, 32} independently for each chunk. At inference time, a single compression ratio is chosen and applied uniformly to all chunks.

Why random per chunk rather than fixed per training example? The paper ablates this in Table 4. When the compression ratio is sampled once per training instance (all chunks in a document get the same ratio) rather than per chunk, Single-Doc QA performance drops from 40.5 to 37.7. The paper hypothesizes that chunk-wise random sampling is better because:

"the chunk-wise setting facilitates better learning of the compression functionality."

The likely mechanism: With per-chunk random ratios, the model sees the same beacon tokens compressing at different granularities within the same document — sometimes 2×, sometimes 32× — forcing it to learn robust compression strategies that work across a range of compression densities. The model also learns to handle the "boundary" between chunks compressed at different ratios, since the accumulated beacon KVs will contain a mix of high-resolution (2×) and low-resolution (32×) compressed representations. This mixed-ratio training may teach the model to extract information from whatever compression quality is available, making it robust to the uniform ratio applied at inference.

Why the set {2, 4, 8, 16, 32}? The paper doesn't explicitly justify these specific values, but they form a geometric progression covering a useful range of compression ratios. 2× provides minimal compression with high fidelity; 32× provides maximum compression with some information loss. The fact that the model is trained on this range enables it to support any of these ratios at inference time by simply changing the number of beacon tokens per chunk. The paper recommends 8× as the default:

"Generally, we recommend to use x8 compression ratio as it preserves most information with high efficiency."

Training Data and Procedure

The training proceeds in two phases, with hyperparameters detailed in Section 4.1.

Pre-training phase:

Data: 1B tokens sampled from RedPajama, with an EOS token appended to each document. Documents shorter than 2,400 tokens or longer than 20,480 tokens are filtered out.
Why RedPajama? This is an open-source reproduction of the LLaMA training corpus, providing diverse general-domain text that teaches the beacon tokens to compress natural language across many domains without overfitting to any specific task format.
Batch size: 8
Learning rate: 5 × 10^{-5}, with linear decay and no warmup
Optimizer: AdamW (implied by standard practice; the paper does not explicitly name the optimizer but the field defaults to AdamW for LLM fine-tuning)
Frozen parameters: All original LLM parameters $\Theta$ are frozen; only $\Theta_b$ (beacon embeddings and per-layer beacon projection matrices) are updated.

Fine-tuning phase:

Data: Three sources: (1) LongAlpaca (long-context QA and summarization), (2) BookSum (chapter-level book summarization), and (3) 16K synthesized QA instances (13K from books, 3K from papers) generated by prompting GPT-3.5-turbo to produce 4 question-answer pairs per text segment. Additionally, 5,000 pre-training samples are mixed in "to mitigate forgetting." All fine-tuning data is formatted as multi-turn conversations with context length limited to 20,480 tokens.
Why synthetic QA data? The paper notes that the synthetic data allows controlling context length by concatenating different numbers of segments using Template A.1 (shown in Appendix A). This provides targeted training on the exact task format (question-answering over long contexts) that the model will be evaluated on.
Learning rate: 1 × 10^{-5}, with linear decay and no warmup (10× lower than pre-training to avoid catastrophic forgetting of the pre-trained compression behavior).
Batch size: 8
Total training cost: The paper states that training "can be quickly accomplished" with 1B pre-training tokens and 30K fine-tuning samples on a single 8×A800 (80G) machine.

Generation: How the Compressed Context Is Used for Decoding

After all chunks are compressed into the accumulated beacon KV cache, the model generates the response autoregressively. The paper emphasizes a key architectural property:

"Activation Beacon unifies generation and compression operations within a single forward pass of the LLM. That is to say, the hidden states of the last input token $H[R_r[-1]]$ is directly used to decode the next token without resorting to another decoder model."

Concretely, the hidden state at the final position of the final chunk (which is always a beacon token, since beacon tokens are appended at the end of each unit) flows through the standard LM head (a linear projection to vocabulary size followed by softmax) to produce the probability distribution over the first generated token. For subsequent tokens, the standard autoregressive process unfolds: each newly generated token's KV activations are appended to the KV cache (alongside the accumulated beacon KVs), and attention is computed against all cached KVs.

What the attention pattern looks like during generation: Each generated token attends to: (1) all preceding generated tokens (standard causal attention), and (2) the entire accumulated beacon KV cache (the compressed representation of the input context). It does NOT attend to any raw context tokens, since those were discarded after compression. This means the attention computation during generation is over $g + m$ tokens where $g$ is the number of generated tokens so far and $m$ is the total number of beacon tokens — $\alpha$ times smaller than attending to the full raw context of length $n$ .

Design Choices and Their Justifications: A Summary

Why compress into activations (KVs) rather than soft tokens?

Capacity argument: KV activations exist at every layer and every head, providing $L \times h_k \times d$ dimensions of representational capacity per beacon token vs. $D$ for a soft token embedding. For Llama-2-7B, this is 32 × 32 × 128 = 131,072 dimensions for keys alone, vs. 4,096 for one embedding vector — a 32× capacity increase.
Architectural integration: KV activations are the native "memory" format of transformers. The frozen LLM already knows how to attend to and extract information from KVs; no adapter, decoder, or re-encoding is needed. The paper explicitly contrasts with ICAE, which requires a separate decoder to convert soft tokens back into a usable format.
Empirical evidence: Table 1 shows AutoCompressor and ICAE (both soft-token methods) achieving 12.9 and 19.5 on Single-Doc QA, while Activation Beacon achieves 34.9 — a 2-3× improvement that the paper attributes to the activation-based compression being less bottlenecked.

Why progressive, fine-grained compression rather than all-at-once?

Differentiated attention scopes: Interleaving beacon tokens within a chunk gives each one a unique span of raw tokens to specialize in. The ablation in Table 4 shows that removing this (appending all beacon tokens at chunk end) causes a 5.3-point drop on Single-Doc QA (40.5 → 35.2).
Handling arbitrary lengths: Progressive chunking means the model never needs to attend to more than $w + k_i$ tokens at once, enabling compression of contexts far longer than the training length (128K vs. 20K, as demonstrated in Figure 4).
Training efficiency: No need for gradient stopping because the computation graph depth scales as $L + m$ not $L \times m$ .

Why query-independent compression rather than question-dependent token pruning?

Multi-turn efficiency: In multi-turn conversations (Table 2), query-independent compression means the context is compressed once and reused for all questions. Question-dependent methods like SnapKV and LongLLMLingua must re-evaluate token importance for each new question, leading to latency that grows with the number of turns: at 3 turns on 128K context, SnapKV takes 10.7 seconds while Activation Beacon takes 3.0 seconds (a 3.6× speedup).
Training signal: Because compression is independent of any specific downstream question, the training loss can be computed on every token in the context (excluding the first chunk), maximizing sample efficiency. A question-dependent compressor would only receive training signal from tokens that are relevant to the specific question asked, wasting most of the training data.

Why freeze the original LLM parameters?

Preserving short-context capabilities: Table 3 shows that Activation Beacon's performance on MMLU, ARC-C, BoolQ, and GSM8K is nearly identical to the original LLM (e.g., Qwen-2-7B MMLU: 70.1 → 69.1). The paper explicitly conjectures that freezing is responsible for this preservation.
Training efficiency: Only a tiny fraction of parameters are trained (beacon embedding + per-layer beacon projections). For a 7B model with 32 layers and hidden size 4096, the beacon parameters are approximately 4096 + 32 × 3 × 4096 × 4096 ≈ 1.6B parameters in the projection matrices alone if they are full-rank, but in practice these are low-rank or the key/value projection dimensions are smaller (e.g., $h_k \times d$ is typically 4096 or smaller). The actual trainable parameter count is a small fraction of the 7B total.
Avoiding catastrophic forgetting: Freezing prevents the model from drifting away from its pretrained language understanding capabilities while learning the compression task, which is especially important given the relatively small training dataset (1B tokens for pre-training, 30K examples for fine-tuning).

4. Key Insights and Innovations

Innovation 1: Compression Targets Activations, Not Embeddings — A Capacity Argument That Changes What "Summary" Means

The most fundamental conceptual move in this paper is what gets compressed. Prior work in context compression — Gisting, ICAE, AutoCompressors — treats compression as the problem of producing a small number of embedding vectors (soft tokens) that summarize the context. These soft tokens live in the model's input embedding space (dimension $D$ ) and serve as a bottleneck through which all contextual information must flow to influence generation. The field's implicit assumption, inherited from the prompt engineering and prefix-tuning traditions, was that a handful of learned embedding vectors could capture the essential information from a long context.

Activation Beacon rejects this assumption entirely. It compresses into per-layer, per-head key-value activations — the same representational format that transformers use internally to store and retrieve information across positions. This is not an incremental improvement in compression quality; it is a fundamentally different answer to the question "What is the right representation to compress into?"

The significance is easiest to see through the capacity lens. A standard soft token in ICAE is one vector of dimension $D = 4096$ (for Llama-2-7B) that must represent an entire chunk of text. A beacon token's compressed representation, by contrast, spans $L \times h_k \times d$ dimensions — 32 layers × 32 key/value heads × 128 head-dim = 131,072 dimensions per beacon token for Llama-2-7B, roughly a 32× increase in raw representational capacity. But the argument is not merely about dimensionality; it is about representational structure. The per-head, per-layer decomposition means different attention heads can specialize in different aspects of compression — one head might focus on named entities, another on syntactic structure, another on numerical values — without interference, because each head's KV space is independent. A soft token embedding provides no such structured decomposition; all information is compressed into a single flat vector, forcing different types of information to compete for the same dimensions.

The paper provides circumstantial evidence for this capacity argument through the stark performance gap between Activation Beacon and soft-token methods. In Table 1 on LongBench Single-Doc QA, AutoCompressor achieves 12.9 and ICAE achieves 19.5, while Activation Beacon achieves 34.9 — matching the uncompressed baseline at 34.8. On Needle-in-a-Haystack with Llama-2-7B (Figure 5), soft-token methods collapse to roughly 2-4 accuracy out of 10 at high compression ratios, while Activation Beacon maintains 8+. These are not marginal differences that could be attributed to better training data or hyperparameters; they represent a qualitative gap between "compression that loses critical information" and "compression that preserves it." The paper does not attempt a controlled ablation varying only the representational format (soft tokens vs. KVs) while holding all else equal, so the capacity argument remains partly conjectural, but the empirical gap is large enough to strongly suggest a fundamental bottleneck in soft-token methods that activation-based compression escapes.

This insight has implications beyond Activation Beacon itself. It suggests that the transformer's internal KV cache is not merely a computational convenience to be optimized away, but rather a privileged representational format for contextual memory — one that the model is already trained to read and write through its attention mechanism. Future work on context compression, retrieval-augmented generation, or memory-augmented LLMs might productively target KV-space representations rather than input-space embeddings. The paper's success in freezing the entire base LLM while only training the beacon-specific projection matrices further suggests that this KV-space compression can be learned as a wrapper around a frozen model, without modifying the model's own representational conventions.

Innovation 2: Progressive Fine-Grained Compression Replaces "All-at-Once" Summarization — And It's the Attention Scope That Matters, Not Just the Chunking

Many prior compression methods (AutoCompressors, ICAE) already segmented long contexts into chunks to handle sequences exceeding the LLM's context window. The paper's contribution here is not chunking per se — it is the observation that how information is distributed across compression tokens within a chunk determines compression fidelity, and that interleaving compression tokens at fine granularity (with differentiated attention scopes) is qualitatively superior to appending them all at the end (with identical attention scopes).

This is subtle enough to be worth unpacking carefully. When ICAE compresses a chunk of 1024 tokens into, say, 16 soft tokens, all 16 soft tokens are generated from the same chunk — they all have access to the entire chunk's information via cross-attention or an encoder-decoder bottleneck. They are differentiated only by their positions and the stochasticity of the generation process, not by any structural assignment of which parts of the chunk each should focus on. The result is 16 tokens that each attempt to summarize the whole chunk, with no mechanism for distributed, complementary compression.

Activation Beacon's interleaving scheme (one beacon token after every $\alpha$ raw tokens) creates a fundamentally different information allocation. Beacon token $\langle b \rangle_j$ has a causal attention scope that includes raw tokens 1 through $j \times \alpha$ , plus all preceding beacon tokens. Beacon token $\langle b \rangle_{j+1}$ additionally sees the next $\alpha$ raw tokens. This means the incremental information each beacon token encodes is precisely the $\alpha$ new tokens it can see that the previous beacon token could not. The beacon tokens form a progressive chain where each one specializes in compressing a specific local span while maintaining context from earlier spans through attention to preceding beacon tokens. The paper calls this "fine-grained compression" and the ablation in Table 4 quantifies its importance: removing the interleaving and appending all beacon tokens at chunk end drops Qwen-2-7B Single-Doc QA from 40.5 to 35.2.

What makes this an intellectual contribution rather than an implementation detail is that it identifies differentiated attention scope as the mechanism by which compression quality scales with the number of compression tokens. In soft-token methods, adding more soft tokens provides diminishing returns because each additional token sees the same information and can only add incremental representational capacity without access to new content. In Activation Beacon, adding more beacon tokens (by reducing $\alpha$ ) simultaneously increases the number of compression tokens AND narrows the span each one must compress, improving fidelity along two dimensions at once. This explains why Activation Beacon maintains high quality even at modest compression ratios (2×, 4×) while scaling gracefully to aggressive compression (16×, 32×) — each beacon token's job gets proportionally harder as $\alpha$ increases, but there is no sudden collapse because the attention scope mechanism naturally distributes the load.

The finding also resolves a tension in prior work. AutoCompressors found that training with longer back-propagation through time (more chunks with gradients flowing) improved compression quality, but at the cost of prohibitive memory requirements. Activation Beacon achieves effective long-range compression without deep gradient graphs by using per-layer (rather than final-layer) information transfer across chunks, as discussed in Section 3. This means the progressive compression workflow and the training efficiency are not independent design choices — they are enabled by the same architectural property (horizontal information flow through KV activations rather than vertical flow through the layer stack).

Innovation 3: Query-Independent Compression as a Principled Design Choice, Not Just a Convenience — And What It Reveals About Multi-Turn Efficiency

The paper draws a sharp line between query-dependent compression (SnapKV, LongLLMLingua) and query-independent compression (Activation Beacon), but the contribution is not merely the choice — it is the empirical demonstration that query-independent compression can match query-dependent methods in quality while dramatically outperforming them in multi-turn efficiency, and the architectural insight into why this is possible.

Query-dependent methods work by asking: "Given this specific question, which tokens in the context are important?" They use attention scores or other importance heuristics to prune tokens that seem irrelevant to the query. This is intuitively appealing — different questions about the same document need different information, so why not compress adaptively? The field's default assumption, implicit in the success of retrieval-augmented generation and attention-based importance scoring, was that query-aware compression would be strictly better than query-agnostic compression because it can allocate the compression budget where it matters most.

Activation Beacon demonstrates that this assumption breaks down under realistic multi-turn deployment. Table 2 tells the story clearly: on a 3-turn Multi-Needle-in-a-Haystack task with Qwen-2-7B at 128K context, Activation Beacon achieves 9.10 accuracy with 2.98 seconds latency, while SnapKV achieves 8.85 accuracy with 10.66 seconds latency — nearly identical quality but 3.6× faster. LongLLMLingua is even worse: 1.50 accuracy at 27.75 seconds. The efficiency gap widens with each additional turn because query-dependent methods must re-compute compression for each new question, while Activation Beacon compresses once and reuses the result.

But the deeper insight is about why query-independent compression can match or exceed query-dependent compression in quality. Query-dependent pruning at high compression ratios (e.g., 8×, meaning 87.5% of tokens are deleted) faces an inherent tension: the tokens deemed "unimportant" for one question may contain information that is essential for understanding the document's structure, maintaining coherence, or answering subsequent questions. By deleting these tokens, the pruned context becomes a degraded, potentially incoherent version of the original. Activation Beacon, by compressing into dense KV representations rather than deleting tokens, preserves the relational structure of the context — even if individual tokens are not explicitly stored, the attention patterns that linked them are distilled into the beacon KVs. This is speculative (the paper does not analyze what information beacon KVs actually encode), but it offers a plausible explanation for the quality parity in Table 1 and Table 2.

The implications extend beyond efficiency. Query-independent compression enables incremental context updates in streaming or conversational settings — if a user appends new information to a document, only the new chunk needs to be compressed, with previous beacon KVs reused. It also simplifies training, because the compression objective is decoupled from any specific downstream task, allowing the full context to serve as a training signal rather than only the tokens relevant to a particular question. This may explain why Activation Beacon achieves strong results with only 1B pre-training tokens and 30K fine-tuning samples — every token contributes to the compression objective, unlike query-dependent methods that only receive signal from tokens that pass the importance filter.

Innovation 4: Chunk-Wise Random Compression Ratio Sampling as a Meta-Training Strategy — Teaching Flexibility Without Architectural Modification

Most compression methods bake the compression ratio into the architecture: ICAE uses a fixed number of soft tokens per chunk, AutoCompressors use a fixed summary length, and token pruning methods use a fixed sparsity ratio or budget. Changing the compression level requires retraining or redesigning the model. Activation Beacon introduces a training strategy — randomly sampling the compression ratio independently for each chunk during training — that produces a single model checkpoint capable of operating at any compression ratio in {2, 4, 8, 16, 32} without modification at inference time.

The innovation is not the sampling itself, but the recognition that varying the compression ratio during training teaches the model a more general compression capability. The paper's ablation in Table 4 confirms this: replacing per-chunk random ratio sampling with per-instance fixed ratio sampling (where all chunks in a document share the same ratio, but different documents get different ratios) drops Single-Doc QA from 40.5 to 37.7. The per-chunk variation is what matters — the model must learn to compress at different granularities within the same document, and to attend to accumulated beacon KVs that may have been compressed at different ratios depending on their originating chunks.

Why might per-chunk variation be more effective than per-instance variation? The paper does not fully explain this, but a plausible mechanism is that per-chunk random ratios force the model to handle heterogeneous compression quality within a single forward pass. When encoding chunk 5, the accumulated beacon KVs from chunks 1-4 might have been compressed at 2×, 32×, 8×, and 4× respectively. The attention mechanism must learn to extract useful information from this mixed-quality compressed context — some parts are high-fidelity, others are coarse summaries. This is a harder learning problem than attending to uniformly compressed context, and solving it likely produces more robust compression representations that degrade gracefully as the ratio increases.

The practical significance is substantial for deployment. A cloud provider serving Activation Beacon can offer a single model checkpoint that supports multiple compression tiers: 2× for accuracy-critical applications, 8× for balanced workloads, and 32× for latency-sensitive applications. No model swapping, no per-tier fine-tuning, no architectural variants. This is a meaningful operational simplification that reduces the barrier to adopting context compression in production systems where different users or tasks have different latency budgets.

The paper also demonstrates that this flexibility does not come at a quality cost relative to ratio-specialized training. Figure 5 shows Activation Beacon maintaining top accuracy across all compression ratios on Needle-in-a-Haystack, whereas competitors (AutoCompressor, ICAE) are typically optimized for a single compression level and degrade sharply when used at others. This suggests that ratio-specialized training may actually produce more brittle models that overfit to a specific compression granularity, while the randomized training produces more general, robust compression behavior.

Innovation 5: The Diagnostic Power of the "Frozen LLM" Design — Separating Compression Capability from Language Modeling Capability

While many papers freeze pretrained parameters during adapter or LoRA fine-tuning as a practical convenience, Activation Beacon's decision to freeze the entire base LLM (7B parameters) while training only the beacon-specific components (a small fraction of total parameters) serves a deeper scientific purpose: it provides a clean ablation that compression capability can be acquired without altering the model's language understanding, and that the KV cache can serve as a learnable compression interface without retraining the attention mechanism that reads from it.

The paper explicitly validates this separation. Table 3 shows that Activation Beacon's performance on standard short-context benchmarks (MMLU, ARC-C, BoolQ, GSM8K) is virtually unchanged from the original model — Qwen-2-7B drops only from 70.1 to 69.1 on MMLU, a 1.0-point difference that could easily be noise. This is not guaranteed by freezing; adding beacon tokens and training new projection matrices could still distort the model's representations if the beacon tokens' activations interfered with the raw tokens' processing. The fact that short-context capabilities are preserved suggests that the learned beacon projections operate in a subspace that is complementary to, rather than competing with, the model's existing representational space.

This finding has implications for the broader question of how to augment frozen LLMs with new capabilities. It provides evidence that the KV cache — which is normally treated as a passive byproduct of forward propagation — can be actively and learnably manipulated to store compressed information, and that the frozen attention mechanism can read from this manipulated cache without retraining. This opens the door to other KV-cache-based augmentation techniques: learned memory modules that write to the KV cache, compression modules for retrieval-augmented generation, or multi-modal adapters that inject non-text information into the KV stream. The key enabling insight is that the attention mechanism's interface — "queries attend to keys, producing weighted sums of values" — is general enough that it does not require the keys and values to come from the same distribution as the pretraining data, as long as the attention patterns they induce are useful for the task.

The paper's comparison with the fine-tuned uncompressed baseline (Full-FT in Table 1) further reinforces this separation. Full-FT receives the same training data as Activation Beacon but updates all model parameters; it achieves comparable performance (34.8 vs. 34.9 on Single-Doc QA for Llama-2-7B). This means that freezing the LLM, far from being a limitation, is sufficient to match a fully fine-tuned model on compression-conditioned generation — and it preserves short-context capabilities that full fine-tuning might degrade (the paper does not report Full-FT's short-context scores, so this remains a hypothesis). The architectural implication is that long-context compression and language modeling are functionally separable capabilities that can be implemented in different parameter subsets of the same model, with the KV cache serving as the interface between them.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary evaluation benchmark is LongBench, a bilingual, multitask benchmark for long context understanding. The paper uses the specific split with 32K maximum context length, covering five task categories: Single-Document QA, Multi-Document QA, Summarization, Few-Shot Learning, and Code Completion. Additionally, the paper evaluates on Needle-in-a-Haystack (NIAH) following the official settings from gkamradt (2023), using ChatGPT to estimate accuracy on a 1-to-10 scale, and Multi-Needle-in-a-Haystack from NeedleBench, where 3 different needles are inserted at different positions and the model is asked to retrieve one specific needle per turn in a multi-turn conversation setting, with experiments repeated 20 times with distinct needle positions. Short-context evaluation uses MMLU, ARC-Challenge, BoolQ, and GSM8K to verify that compression does not degrade the backbone LLM's original capabilities.
Base model(s). The paper applies Activation Beacon to Llama-2-7B (chat) and Qwen-2-7B (instruct). Llama-2-7B is chosen specifically because the two most important compression baselines — AutoCompressors and ICAE — are implemented on Llama-2, enabling direct comparison without confounding model differences. The paper states it "believe[s] this model [Llama-2-7B] is representative of the capabilities of many contemporary LLMs" (Section 4.1). Qwen-2-7B provides a second model family for robustness, and a Qwen-2-72B variant is used only for FLOPs analysis (Figure 3C), not for compression quality evaluation. All backbone LLM parameters are frozen throughout training.
Metrics. For LongBench tasks, the metric is task-specific accuracy (exact match or equivalent, following the LongBench evaluation protocol) reported as a percentage. For Needle-in-a-Haystack, accuracy is a 1-to-10 score estimated by ChatGPT evaluating whether the retrieved information matches the ground-truth needle. For Multi-Needle-in-a-Haystack, accuracy is the fraction of needles correctly retrieved across 20 trials (reported on a scale where 10 = perfect, implying each of 20 trials contributes up to 0.5 points to the aggregate score, though the exact scoring granularity is not fully specified — the paper reports values like 9.75, 9.45, 9.10, suggesting a fine-grained scoring rubric). For short-context benchmarks, standard accuracy metrics are used. For efficiency, end-to-end latency is measured in seconds (compression + generation time).
Baselines. The paper compares against:
- Full: The uncompressed backbone LLM with its native context window (4K for Llama-2, 32K for Qwen-2), using truncation from the middle for contexts exceeding the window.
- Full-FT: The uncompressed backbone LLM fine-tuned on the same training data as Activation Beacon, with the full 32K context (no truncation). This serves as an upper bound for what compression can achieve — if Activation Beacon matches Full-FT, compression is effectively lossless for the task.
- AutoCompressors (Chevalier et al., 2023): Soft-token-based context compression that segments long contexts into chunks and compresses each into summary tokens. Implemented only for Llama-2.
- ICAE (Ge et al., 2024): In-Context Autoencoder that compresses context into a small number of soft tokens using a separate encoder-decoder architecture. Implemented only for Llama-2.
- LongLLMLingua (Jiang et al., 2023): A token-pruning method that deletes unimportant tokens based on question-dependent importance estimation.
- SnapKV (Li et al., 2024): A KV-cache compression method that retains only the most important KV pairs based on attention patterns, also question-dependent.
All compression baselines are "fine-tune[d] their official checkpoints using the same training data" as Activation Beacon to ensure fair comparison (Section 4.1).
Generation budget / compute accounting. The paper measures compute in two complementary ways. For FLOPs analysis (Figure 3), the total forward FLOPs is computed analytically using the formulas in Appendix B, which account for self-attention computation (dependent on both current sequence length and cached context length) and other module computation (FFN, LM head, dependent only on current sequence length). For latency measurements (Table 2), end-to-end wall-clock time in seconds is reported, capturing both compression and generation time. For KV cache memory, the reduction factor is directly proportional to the compression ratio — an 8× compression means 8× fewer KV entries stored. The paper does not use a unified "generation budget" metric (e.g., number of tokens generated) across all experiments; rather, efficiency is evaluated separately through FLOPs curves, latency tables, and memory reduction claims.
Cross-validation / statistical protocol. The paper does not employ cross-validation for the LongBench evaluations — these are single-run results on the standard LongBench test set. For Multi-Needle-in-a-Haystack, experiments are "repeated 20 times for each model with distinct needle positions" and the average accuracy is reported, providing some statistical reliability. For Needle-in-a-Haystack, the evaluation appears to be a single run per (context length, depth) pair, with accuracy estimated by ChatGPT (which introduces its own variance, unaccounted for). No confidence intervals, standard deviations, or statistical significance tests are reported for any experiment. This is a notable limitation — the paper claims near-lossless compression (e.g., 40.5 vs. 41.0 on Single-Doc QA for Qwen-2-7B), but without variance estimates, it is unclear whether these differences are statistically meaningful or within noise.

Main Quantitative Results

LongBench Compression Effectiveness (Table 1)

Headline result: Activation Beacon achieves compression quality that matches or nearly matches the uncompressed fine-tuned baseline (Full-FT) while substantially outperforming all other compression methods across all five LongBench task categories.

For Llama-2-7B:

Single-Doc QA: Activation Beacon (34.9) slightly exceeds Full-FT (34.8) and dramatically outperforms AutoCompressors (12.9), ICAE (19.5), LongLLMLingua (21.5), and SnapKV (24.2). The full uncompressed baseline (Full) achieves only 24.7 because it must truncate context beyond 4K, whereas Activation Beacon compresses the full 32K context into 4K-equivalent activations, effectively recovering information that truncation loses.
Multi-Doc QA: Activation Beacon (27.5) matches Full-FT (27.5) exactly, versus AutoCompressors (16.4), ICAE (19.2), LongLLMLingua (18.8), and SnapKV (22.6).
Summarization: Activation Beacon (25.0) exceeds Full-FT (23.2), suggesting that compression may actually help summarization by filtering out distracting details — though this difference (1.8 points) may not be statistically significant without variance estimates.
Few-Shot: Activation Beacon (61.4) nearly matches Full-FT (61.8), versus LongLLMLingua (49.5) and SnapKV (60.1). AutoCompressors (23.8) and ICAE (24.8) fail catastrophically here — this is the task where soft-token compression is most clearly inadequate, likely because few-shot learning requires preserving the exact formatting and content of multiple examples.
Code: Activation Beacon (57.8) matches Full-FT (57.8) and Full (57.7) exactly. This is notable because code has strict syntactic structure that token-deletion methods might disrupt — LongLLMLingua drops to 53.2 and ICAE collapses to 27.8.

For Qwen-2-7B (which has a native 32K context window, so Full and Full-FT both see the full context):

Single-Doc QA: Activation Beacon (40.5) nearly matches Full-FT (41.0) and exceeds Full (38.8). LongLLMLingua drops to 24.7, SnapKV achieves 38.7.
Multi-Doc QA: Activation Beacon (40.3) nearly matches Full-FT (40.6). LongLLMLingua (20.3) and SnapKV (37.6) both substantially underperform.
Summarization: All methods cluster tightly: Full (26.7), Full-FT (26.8), Activation Beacon (26.8), SnapKV (26.2). LongLLMLingua (26.3) is also competitive here — summarization appears to be the task least sensitive to compression method, likely because it requires extracting gist-level information that even aggressive compression can preserve.
Few-Shot: Activation Beacon (68.4) nearly matches Full-FT (68.5) and Full (70.1 — interestingly, the unfine-tuned Full model performs best here, suggesting the fine-tuning data may slightly degrade few-shot capability). LongLLMLingua drops to 55.9.
Code: Activation Beacon (66.4) slightly exceeds Full-FT (66.1). LongLLMLingua (50.1) is far behind.

Key pattern across both models: Activation Beacon consistently achieves within 1–2 points of Full-FT (and sometimes exceeds it), while soft-token methods (AutoCompressors, ICAE) fail dramatically on most tasks and token-pruning methods (LongLLMLingua) degrade substantially at high compression ratios. SnapKV is the strongest baseline but cannot compress contexts longer than the LLM's window, limiting its applicability. The paper attributes this gap to the capacity of activations versus soft tokens and the fine-grained compression mechanism.

A note on adaptive vs. uniform compression for Llama-2: The paper states that for Llama-2-7B, "we set adaptive compression ratio, translating to x2 compression for 4K-8K contexts, x4 compression for 8K-16K contexts, and x8 compression for 16K-32K contexts," while for Qwen-2-7B, "we apply a uniform compression ratio of x4." This makes the Llama-2 results less directly comparable to baselines that use a uniform compression ratio, though the paper's chunk-wise random ratio training should make the model robust to such variation. It also means Llama-2's 32K context is compressed to roughly 4K activations (matching its native window), while Qwen-2's 32K context is compressed to 8K activations — a less aggressive compression that may partially explain Qwen-2's higher absolute scores.

Needle-in-a-Haystack (Figure 4)

Headline result: Activation Beacon accurately retrieves the needle "most of the time" across context lengths up to 128K (for Qwen-2-7B) and 32K (for Llama-2-7B), despite being trained only on contexts under 20K, demonstrating that the compressed representations preserve fine-grained factual information and that the compression capability generalizes to lengths far exceeding the training distribution.

For Llama-2-7B (Figure 4A): The heatmap shows predominantly green cells (scores 5–10) across all context lengths from 4,096 to 32,768 tokens and all depth percentages from 0% to 100%. There is a small region of lower scores (shown in lighter/yellow coloring) at depth percentages around 60–80% and context lengths beyond approximately 26K, but the overall pattern is strong retrieval throughout. The paper annotates this with a summary score of 5.0 (or 1.0, the notation is ambiguous — likely indicating the average or minimum score), though the visual heatmap suggests most cells are well above 5.

For Qwen-2-7B (Figure 4B): The context extends to 131,072 tokens, far beyond the model's 20K training length. The heatmap shows strong retrieval (green, scores 7–10) across nearly all context lengths and depths, with a small band of slightly lower scores at approximately 50–75K context length and depth percentages around 60–80%. The paper annotates summary scores of 5.0, 9.0, and 7.0 (these may represent different metrics or model variants — the annotation in the figure is ambiguous, but the visual pattern clearly shows high accuracy throughout). This is a genuinely surprising result: the model was never trained to compress contexts longer than 20K tokens, yet its compression mechanism transfers to 128K without degradation. This implies that the progressive chunk-by-chunk compression workflow does not accumulate errors as more chunks are processed — each new chunk's beacon tokens attend to accumulated beacon KVs from all previous chunks, and the quality of this attention does not decay with distance.

The paper emphasizes that this is query-independent compression — the model has "no prior knowledge of what to compress and what not to" — making the high retrieval accuracy particularly notable. Query-dependent methods like SnapKV can focus compression on tokens relevant to a specific question, but Activation Beacon must preserve all potentially relevant information in the compressed representation, and Figure 4 shows it largely succeeds.

Multi-Needle-in-a-Haystack: Multi-Turn Efficiency (Table 2)

Headline result: Activation Beacon achieves near-identical accuracy to the uncompressed Full-FT baseline across all turns (1-turn, 2-turn, 3-turn) while providing 1.8× to 3.6× latency reduction compared to query-dependent compression baselines, with the efficiency advantage growing as the number of turns increases.

For Llama-2-7B at 32K context with 8× compression:

1-Turn: Activation Beacon accuracy 9.75 vs. Full-FT 9.75 (identical). Latency: 1.153s vs. 1.336s (1.16× speedup). AutoCompressors: 1.60 accuracy, 2.135s latency — lower quality and slower. ICAE: 2.15 accuracy, 1.182s — faster but terrible accuracy. LongLLMLingua: 2.05 accuracy, 2.813s. SnapKV: 1.00 accuracy, 0.859s — fastest but worst accuracy (the paper does not explain why SnapKV's accuracy is so low at 1.00; it may indicate a failure mode on this specific task).
2-Turn: Activation Beacon: 9.40 accuracy, 1.356s. Full-FT: 9.45 accuracy, 1.532s. The latency gap widens (1.13× speedup). AutoCompressors: 1.50 accuracy, 2.561s. SnapKV: 1.00 accuracy, 1.656s.
3-Turn: Activation Beacon: 9.05 accuracy, 1.638s. Full-FT: 9.10 accuracy, 1.726s. Gap widens further (1.05× speedup — the gap narrowing in relative terms, but Activation Beacon remains faster in absolute terms). AutoCompressors: 1.50 accuracy, 2.994s — 1.8× slower than Activation Beacon. LongLLMLingua: 2.00 accuracy, 7.034s — 4.3× slower. SnapKV: 1.00 accuracy, 2.199s.

For Qwen-2-7B at 128K context with 8× compression:

1-Turn: Activation Beacon: 9.70 accuracy, 2.445s. Full-FT: 9.75 accuracy, 4.399s — 1.8× speedup. LongLLMLingua: 2.00 accuracy, 10.455s. SnapKV: 9.45 accuracy, 3.955s — competitive accuracy but 1.6× slower than Activation Beacon.
2-Turn: Activation Beacon: 9.35 accuracy, 2.773s — latency barely increases (only 0.33s from 1-turn to 2-turn) because the compressed context is reused. Full-FT: 9.50 accuracy, 5.254s. SnapKV: 8.95 accuracy, 7.803s — SnapKV's latency nearly doubles from 1-turn to 2-turn because it must re-evaluate token importance for the second question. LongLLMLingua: 1.55 accuracy, 19.768s.
3-Turn: Activation Beacon: 9.10 accuracy, 2.981s — still only 0.54s more than 1-turn. Full-FT: 9.20 accuracy, 6.153s. SnapKV: 8.85 accuracy, 10.659s — 3.6× slower than Activation Beacon. LongLLMLingua: 1.50 accuracy, 27.751s — 9.3× slower.

The critical efficiency insight from these numbers: Activation Beacon's latency grows very slowly with turn count (1.153 → 1.356 → 1.638 for Llama-2; 2.445 → 2.773 → 2.981 for Qwen-2) because compression is performed once and reused. In contrast, query-dependent methods show latency that scales nearly linearly with turn count, as each new question requires re-compressing the context. The paper explicitly states Activation Beacon "is 1.8x faster than AutoCompressor because it does not have to re-encode the soft tokens from previous chunks" and achieves "9.3x and 3.6x acceleration upon LongLLMLingua and SnapKV given three turns" for Qwen-2-7B (Section 4.3). The accuracy numbers show Activation Beacon consistently within 0.05–0.10 of Full-FT across all turns, confirming near-lossless compression quality even in multi-turn retrieval.

Compression Flexibility Across Ratios (Figure 5)

Headline result: Activation Beacon maintains top accuracy across compression ratios from 2× to 16× and context lengths from 1K to 32K, while soft-token baselines degrade sharply at higher ratios and SnapKV cannot compress contexts beyond the LLM's window size.

For Needle-in-a-Haystack with Llama-2-7B at 1K context (Figure 5A): All methods perform well at 2× and 4× compression (scores 8–10). At 8×, Activation Beacon and SnapKV maintain ~9, while AutoCompressors drops to ~4 and ICAE to ~5. At 16×, Activation Beacon stays at ~8–9, SnapKV also ~8, while AutoCompressors and ICAE collapse to ~2–3. LongLLMLingua shows a steady decline from ~9 at 2× to ~3 at 16×.

At 4K context (Figure 5B): The pattern intensifies. Activation Beacon maintains ~8–9 across all ratios. SnapKV also strong at ~8. AutoCompressors and ICAE are at ~4–5 even at 2×, degrading to ~2 at 16×. LongLLMLingua declines from ~7 to ~2.

At 32K context (Figure 5C): Only 8× and 16× are shown (since 2× and 4× would produce context lengths exceeding Llama-2's 4K window for the compressed representation — though this is somewhat inconsistent with the paper's claim that Activation Beacon can handle this). Activation Beacon maintains ~8 at 8× and ~7 at 16×. AutoCompressors and ICAE score ~2–3 at 8× and ~1–2 at 16×. SnapKV is absent from the 32K panel because it cannot compress contexts longer than the LLM's window size — a fundamental limitation the paper highlights: "it fails to compress inputs longer than the LLM's window size, which may limit its practical usage" (Section 4.4). LongLLMLingua achieves ~4 at 8× and ~2 at 16×.

The paper's recommendation of 8× as the default compression ratio is supported by these results: at 8×, Activation Beacon achieves 8+ accuracy across all context lengths, while 16× shows a slight but noticeable degradation (to ~7 at 32K) that may matter for high-stakes applications.

Short-Context Capabilities Preservation (Table 3)

Headline result: Activation Beacon preserves the backbone LLM's short-context performance almost perfectly, with degradation of at most 1.0 point across four standard benchmarks.

For Llama-2-7B: MMLU drops from 47.5 to 46.6 (−0.9), ARC-C from 48.5 to 48.4 (−0.1), BoolQ from 86.2 to 86.5 (+0.3), GSM8K from 9.2 to 9.3 (+0.1). The differences are within noise for all benchmarks except possibly MMLU, though without variance estimates this cannot be confirmed.

For Qwen-2-7B: MMLU drops from 70.1 to 69.1 (−1.0), ARC-C unchanged at 62.7, BoolQ from 87.1 to 87.2 (+0.1), GSM8K from 76.0 to 76.2 (+0.2). Again, differences are minimal.

The paper attributes this preservation to freezing the LLM's original parameters: "We conjecture that the primary reason is the LLM's original parameters are frozen throughout the training process" (Section 4.5). This is a non-trivial result — adding new tokens (beacon tokens) and new projection matrices could theoretically distort the model's internal representations even with frozen parameters, if the new projections interact with shared components (e.g., layer norm, which normalizes over the concatenated sequence of raw and beacon tokens). The fact that short-context performance is preserved suggests that the beacon projections operate in a subspace that does not interfere with raw token processing when no beacon tokens are present (or when they are present but the context is short).

FLOPs Reduction (Figure 3)

Headline result: Activation Beacon consistently reduces forward FLOPs across all model scales and context lengths, with savings that amplify as context length grows, reaching approximately 2.6× reduction at 256K context for Qwen-2-7B with 8× compression.

For Llama-2-7B (Figure 3A): At 4K context, all methods are similar (~0.25 TFLOPs) because overhead from encoding beacon tokens roughly offsets attention savings at short lengths. At 32K, 8× compression reduces FLOPs from ~2.0 to ~1.2 TFLOPs (1.7× reduction). At 256K, full attention requires ~3.8 TFLOPs while 8× compression requires ~1.8 TFLOPs (2.1× reduction). The 2× and 4× compression curves lie between the 8× curve and the full-attention line, as expected.

For Qwen-2-7B (Figure 3B): The pattern is similar but the absolute FLOPs are lower because Qwen-2 uses GQA (Grouped-Query Attention) which reduces key/value head computation. At 4K, all methods ~0.15 TFLOPs. At 128K, 8× compression reduces from ~2.3 to ~1.2 TFLOPs (1.9× reduction). At 256K, full attention requires ~2.8 TFLOPs while 8× compression requires ~1.0 TFLOPs (2.8× reduction — the paper claims "more than x4 reduction at 256K context" but the graph appears to show a smaller gap; the discrepancy may arise from how the FLOPs are calculated or the specific 256K data point).

For Qwen-2-72B (Figure 3C): Larger model, larger FLOPs. At 4K, ~1.0 TFLOPs for all methods. At 128K, 8× compression reduces from ~10.0 to ~4.5 TFLOPs (2.2× reduction). At 256K, full attention requires ~12.5 TFLOPs while 8× compression requires ~4.8 TFLOPs (2.6× reduction).

The key trend: the FLOPs savings from Activation Beacon increase with context length because the self-attention cost (which is dramatically reduced by compression) dominates the total FLOPs at long contexts, while the overhead from encoding beacon tokens in other modules (FFN, etc.) becomes proportionally smaller. At short contexts (4K), the overhead dominates and compression provides minimal savings — this is why the paper focuses on long-context scenarios.

Ablation Studies and Robustness Checks

Fine-grained compression (beacon token interleaving vs. appending): Removing the fine-grained interleaving and instead appending all beacon tokens at the end of each chunk drops Single-Doc QA performance from 40.5 to 35.2 on Qwen-2-7B (Table 4). This 5.3-point drop is the largest single-factor degradation in the ablation, confirming that differentiated attention scopes — where each beacon token specializes in compressing a specific, progressively larger span of the chunk — is critical to compression quality. When beacon tokens are appended at chunk end, they all have identical attention scopes (the entire chunk), which the paper interprets as losing the fine-grained distribution of compression responsibility.

Chunk-wise vs. instance-wise random compression ratio: Replacing the per-chunk random compression ratio sampling with per-instance random sampling (where all chunks in a document share the same ratio, but different documents get different ratios) drops Single-Doc QA from 40.5 to 37.7 (Table 4). This 2.8-point gap suggests that training with heterogeneous compression ratios within a single document teaches the model more robust compression strategies than training with uniform ratios per document. The paper does not fully explain why, but a plausible mechanism (discussed in Section 4, prior sections) is that per-chunk variation forces the model to attend to accumulated beacon KVs that were compressed at different granularities, learning to extract information from mixed-quality compressed contexts.

Pre-training and fine-tuning ablation: Removing the pre-training phase (1B tokens from RedPajama) drops performance from 40.5 to 34.9, while removing the fine-tuning phase drops performance to 35.5 (Table 4). Both phases are important, with pre-training providing a 5.6-point gain and fine-tuning adding another 5.0 points on top of the pre-trained checkpoint (the default configuration achieves 40.5, implying the pre-trained-only model achieves approximately 35.5 if we subtract — but the paper reports 34.9 for "w/o Fine-tuning," which is close to 35.5). The paper notes that "both stages are useful, and the combination of both leads to the optimal performance" and suggests that "the compression quality of Activation Beacon can be further enhanced given more abundant and targeted training" — implying that the 1B token pre-training budget may not saturate performance.

Needle-in-a-Haystack across compression ratios and context lengths (Figure 5): This is both a main result and an ablation, showing that Activation Beacon's quality degrades gracefully as compression ratio increases, while soft-token methods collapse. At 32K context with 8× compression, Activation Beacon achieves ~8 accuracy vs. ~2–3 for AutoCompressors and ICAE. At 16×, Activation Beacon still achieves ~7 while competitors are at ~1–2. This confirms that the model genuinely supports flexible compression ratios without retraining, as claimed.

Multi-turn latency scaling (Table 2): While not a controlled ablation per se, the latency numbers serve as an implicit ablation of query-independent vs. query-dependent compression. Activation Beacon's latency grows slowly with turn count (2.445 → 2.773 → 2.981 for Qwen-2-7B) because compression is reused, while SnapKV's latency grows much faster (3.955 → 7.803 → 10.659) because compression is re-computed for each question. At 3 turns, this produces a 3.6× latency advantage for Activation Beacon — a direct consequence of the query-independent design choice.

Base model preservation (Table 3): This serves as an ablation confirming that freezing the LLM parameters successfully prevents catastrophic forgetting of short-context capabilities. Without this check, the impressive long-context results could be compromised if the model became worse at standard tasks. The negligible degradation (≤1.0 point across all benchmarks) validates the frozen-LLM design.

Missing ablations: Several experiments that would strengthen the paper are absent. (1) No ablation on chunk size w — the paper uses 1024 for Llama-2 and 2048 for Qwen-2, but does not explore how compression quality varies with chunk size. Smaller chunks mean more beacon tokens (more KV cache entries) but each chunk is processed with less context; larger chunks mean fewer beacon tokens but each must compress more information. (2) No ablation on the set of compression ratios used during training ({2, 4, 8, 16, 32}) — what happens if the model is tested at a ratio not seen during training (e.g., 3× or 64×)? (3) No ablation on the number of beacon tokens per unit (fixed at 1) — could multiple beacon tokens per unit improve compression by allowing specialization? (4) No ablation comparing Activation Beacon's learned beacon projections to using the original projection matrices for beacon tokens (which would test whether separate projection matrices are necessary or merely the learned initialization provides benefit). (5) No direct comparison with a soft-token baseline that uses the same progressive interleaving workflow but stores summary embeddings rather than KV activations — this would isolate the representational format (activations vs. embeddings) from the workflow (progressive vs. all-at-once).

Critical Assessment

Claim 1: Activation Beacon achieves comparable performance to the uncompressed baseline while providing 2× acceleration and 8× KV cache reduction.

Assessment: Broadly supported by Table 1 (LongBench) and Table 2 (Multi-Needle), but with important nuances.

The comparable performance claim holds for LongBench: across both Llama-2-7B and Qwen-2-7B, Activation Beacon is within 0–1.5 points of Full-FT on every task category (Table 1). The largest gaps are -1.0 on Qwen-2 Single-Doc QA (40.5 vs. 41.0) and -0.5 on Qwen-2 Multi-Doc QA (40.3 vs. 40.6). These are small enough to be practically meaningful — a user would likely not notice the difference. For Multi-Needle-in-a-Haystack (Table 2), the gap is typically 0.05–0.10 accuracy points per turn, which is essentially lossless. However, the paper does not report statistical significance, and the test sets are small (LongBench Single-Doc QA likely has a few hundred questions; Multi-Needle uses 20 trials). Minor differences could be within sampling noise.

The 2× acceleration claim is supported for specific configurations. In Table 2, Qwen-2-7B at 128K shows 4.399s for Full-FT vs. 2.445s for Activation Beacon at 1-turn — 1.8× speedup, slightly less than 2×. At 3-turn, it is 6.153s vs. 2.981s — 2.06× speedup, meeting the claim. For Llama-2-7B at 32K, the speedup is more modest: 1.336s vs. 1.153s at 1-turn (1.16×), rising to 1.726s vs. 1.638s at 3-turn (1.05×). The 2× figure is therefore configuration-dependent: it holds for long contexts (128K) with Qwen-2 but not for shorter contexts (32K) with Llama-2. The paper's abstract claims "a 2x acceleration in inference time" without qualification, which overstates the typical case. A more precise statement would be "up to 2× acceleration at 128K context, with smaller gains at shorter contexts."

The 8× KV cache reduction claim is analytically true: the accumulated beacon KV cache stores n/α entries versus n for full attention, so at 8× compression, the reduction is exactly 8×. The paper does not provide empirical memory measurements (e.g., GPU memory usage in GB), only analytical claims. This is acceptable since the relationship is deterministic, but empirical confirmation would strengthen the claim — memory savings in practice can be less than analytical savings due to padding, fragmentation, and framework overhead.

Claim 2: Activation Beacon outperforms existing context compression methods across various compression configurations.

Assessment: Strongly supported for soft-token methods (AutoCompressors, ICAE) and token-pruning methods (LongLLMLingua), but the comparison with SnapKV is more nuanced.

The gap between Activation Beacon and soft-token methods is large and consistent. On Llama-2-7B Single-Doc QA (Table 1), Activation Beacon (34.9) is 2.7× better than ICAE (19.5) and 2.7× better than AutoCompressors (12.9). On Few-Shot, it is 2.6× better than AutoCompressors (61.4 vs. 23.8). These are not marginal improvements — they represent a qualitative difference in compression capability. The Needle-in-a-Haystack results (Figure 5) reinforce this: at 32K context and 8× compression, Activation Beacon achieves ~8 accuracy vs. ~2–3 for soft-token methods. The evidence for soft-token inferiority is overwhelming.

The comparison with SnapKV is less clear-cut. On LongBench (Table 1), SnapKV is competitive: on Llama-2-7B Single-Doc QA, SnapKV achieves 24.2 vs. Activation Beacon's 34.9 — a 10.7-point gap, but SnapKV cannot process contexts beyond 4K for Llama-2 (it uses the truncated 4K input), making the comparison somewhat unfair. On Qwen-2-7B Single-Doc QA, SnapKV achieves 38.7 vs. Activation Beacon's 40.5 — only a 1.8-point gap, and SnapKV likely operates on the full 32K context. On Multi-Doc QA, SnapKV achieves 37.6 vs. 40.3 — a 2.7-point gap. On Summarization and Few-Shot, the gaps are 0.6 and 1.3 points respectively. On Multi-Needle-in-a-Haystack (Table 2), SnapKV achieves 9.45 vs. Activation Beacon's 9.70 at 1-turn (0.25 gap), 8.95 vs. 9.35 at 2-turn (0.40 gap), and 8.85 vs. 9.10 at 3-turn (0.25 gap). These are small differences.

The paper's argument against SnapKV is primarily about applicability rather than quality: SnapKV "cannot compress context longer than the backbone LLM's window" and its query-dependent nature makes it inefficient in multi-turn settings (3.6× slower at 3-turn). These are legitimate and important practical limitations, but they do not mean Activation Beacon outperforms SnapKV in compression quality — the quality is roughly comparable, with a small edge to Activation Beacon. The paper should more clearly distinguish between "outperforms in compression quality" and "outperforms in efficiency and applicability." For single-turn question-answering over in-window contexts, SnapKV is a strong and computationally simpler baseline that nearly matches Activation Beacon.

The comparison with LongLLMLingua is unambiguous: it underperforms in both quality (e.g., 24.7 vs. 40.5 on Qwen-2 Single-Doc QA) and efficiency (e.g., 27.751s vs. 2.981s at 3-turn 128K). The token-deletion approach simply cannot maintain coherence at high compression ratios.

Claim 3: Activation Beacon enables flexible compression ratios via random sampling during training.

Assessment: Supported, but with an important unexamined limitation.

Figure 5 demonstrates that the same model checkpoint works at 2×, 4×, 8×, and 16× compression during inference, with graceful degradation as ratio increases. This is a genuine practical advantage — no other method in the comparison offers this flexibility. The ablation in Table 4 confirms that chunk-wise random ratio sampling is important for achieving this flexibility (removing it drops performance from 40.5 to 37.7).

However, the paper does not test whether the model generalizes to compression ratios outside the training set {2, 4, 8, 16, 32}. What happens at 3×, 6×, 12×, or 64×? The model might handle interpolation (e.g., 3×, which is between 2× and 4× seen during training) but struggle with extrapolation beyond 32×. The training set includes powers of 2, so it is unclear whether the model has learned a general "compress at any ratio" capability or a "compress at these 5 specific ratios" capability. The paper recommends 8× as the default, which is one of the training ratios — a conservative choice that avoids testing extrapolation. Future work should evaluate at non-power-of-2 ratios to determine the limits of flexibility.

Claim 4: Long-context compression generalizes to lengths far exceeding training (20K training vs. 128K evaluation).

Assessment: Impressively supported by Figure 4, but the mechanism is not directly tested.

Figure 4 shows Activation Beacon on Qwen-2-7B accurately retrieving needles at 128K context despite being trained only on contexts under 20K — a 6.4× length extrapolation. This is a surprising and important result. However, the paper does not ablate why this generalization works. Is it because the progressive chunk-by-chunk workflow prevents error accumulation? Because the RoPE position encoding extrapolates well? Because the beacon token attention patterns are length-agnostic? The paper claims the generalization "validates our tailored compression mechanism and learning method can preserve the fine-grained contextual information" (Section 4.2), but this is an assertion, not a mechanistic explanation. A controlled experiment varying chunk size at test time or analyzing attention patterns at long vs. short contexts would strengthen the claim considerably.

Additionally, the Needle-in-a-Haystack task has a specific structure (a single fact in a sea of irrelevant text) that may be particularly favorable to compression — the model only needs to preserve one piece of critical information, which a well-trained compression mechanism might do easily even at extreme lengths. Performance on more demanding long-context tasks (e.g., multi-hop reasoning over 128K tokens, long-form summarization of book-length texts) may not extrapolate as cleanly. The LongBench evaluation is capped at 32K, leaving open whether the near-lossless compression observed at 32K continues to 128K on complex reasoning tasks.

Missing Baselines and Experiments

No comparison with sparse attention methods (e.g., Minference, SampleAttention, StreamingLLM). The paper argues that sparse attention methods "require holding all KV activations on chip" and are "unsuitable for KV cache reduction" (Section 2), but does not empirically validate this claim. A latency and memory comparison with at least one sparse attention method would clarify the practical tradeoffs.

No comparison with KV cache quantization methods (e.g., KIVI). The paper correctly notes that token-level compression is orthogonal to numerical compression and "can be jointly used" (Section 2), but does not demonstrate this combination. Showing that Activation Beacon + KIVI achieves multiplicative memory savings (8× token reduction × 4× precision reduction = 32× total) would be a powerful demonstration.

No scaling study beyond 7B parameters. The FLOPs analysis in Figure 3C uses Qwen-2-72B, but no compression quality or latency results are reported for this model. It is unknown whether the 1B token pre-training budget and 30K fine-tuning samples — sufficient for 7B models — would also suffice for 72B. Larger models might require proportionally more training data to learn effective beacon projections, or their greater representational capacity might enable faster learning. Without this scaling data, the paper's claims apply only to the 7B scale.

No analysis of what beacon tokens actually encode. The paper makes arguments based on representational capacity (KV activations provide more capacity than soft tokens), but never probes what information is actually stored in beacon KVs. Visualization of attention patterns from generated tokens to beacon tokens, probing classifiers, or analysis of per-head beacon KV specialization would strengthen the capacity argument considerably. As it stands, the claim that KVs provide "large enough" capacity is supported only indirectly by the performance gap with soft-token methods — there could be other explanations (better training, better workflow, interactions with frozen LLM) for the gap.

No evaluation on tasks requiring temporal reasoning or coreference resolution over the compressed context. LongBench includes some of these, but dedicated evaluation on datasets like SCROLLS, NarrativeQA, or LRA would test whether the progressive compression mechanism preserves the sequential and relational structure of long contexts, or whether it loses information about event ordering and entity relationships.

Overall, the experimental evidence strongly supports the paper's core claims — that activation-based, progressive, query-independent compression can match uncompressed performance while providing substantial efficiency gains — but the claims are narrower than the paper's rhetoric sometimes suggests. The 2× acceleration is configuration-dependent, the superiority over SnapKV in quality (not just efficiency) is marginal, the flexibility across compression ratios is demonstrated only for ratios seen during training, and the generalization to extreme lengths is shown only on a specific retrieval task. These limitations do not undermine the paper's contribution, but they should qualify how its results are interpreted and deployed.

6. Limitations and Trade-offs

The Difficulty Estimation Overhead Is Not Accounted For in the Headline Efficiency Numbers

The assumption or constraint. The paper's compute-optimal framework — selecting the best compression strategy per difficulty bin — requires estimating each prompt's difficulty before allocating the test-time budget. The method for doing so (2048 sample generations per question, scored by the PRM or ground-truth) is extraordinarily expensive. The authors acknowledge this explicitly in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

In a deployed system, a prompt arrives, the system must first estimate its difficulty, and then execute the selected strategy. The cost of the estimation step is excluded from all budget calculations and efficiency comparisons.

The consequence. The paper's core efficiency claim — "more than 4× better efficiency over a standard best-of-N baseline" — is computed after difficulty is already known, without amortizing the cost of learning it. In practice, generating 2048 samples per question to estimate difficulty consumes more compute than the largest test-time budgets studied (256–512 generations). This means the $4\times$ figure is an upper bound on achievable efficiency rather than a realized deployment gain, unless difficulty estimation can be made dramatically cheaper. The compute-optimal policy's actual cost is difficulty estimation + strategy execution, and the former may dominate the latter for most practical budgets. A practitioner deciding whether to deploy this method needs to know that the current difficulty estimation procedure costs more than the problem-solving budget itself — a fact that contradicts the paper's framing of the method as efficiency-improving.

What evidence exists in the paper. The paper provides no experimental measurement of the difficulty estimation cost or any amortization analysis. The cost is discussed qualitatively in Section 3.2 ("we acknowledge that estimating difficulty in this way still incurs additional computation cost") and Section 8 flags it as future work, but no numbers are provided. The omission means all efficiency curves (Figures 4, 8, 9) implicitly assume difficulty is known for free.

Mitigation status. The paper proposes two mitigations but evaluates neither. First, it suggests using the PRM's predicted final-answer score (averaged over 2048 samples) rather than ground-truth correctness, which removes the need for oracle labels but does not reduce the 2048-generation cost. Second, it flags future work on "pretraining or finetuning models to directly predict difficulty of a question" (Section 8), but no such model is developed. An alternative approach — adaptive difficulty estimation that uses a small initial sample and dynamically adjusts — is mentioned in Section 3.2 as an exploration-exploitation tradeoff but not implemented. Until a cheap difficulty estimator is validated, the compute-optimal policy is a conceptual contribution rather than a deployable system.

Hard Problems Remain Essentially Unsolved: Test-Time Compute Cannot Create Capability From Nothing

The assumption or constraint. The compute-optimal framework assumes that within the base model's proposal distribution, correct solutions exist for some non-trivial fraction of attempts. When the base model's pass@1 is near zero, no amount of test-time compute — search, revision, or their combination — can recover correct answers. The paper explicitly acknowledges this boundary, stating in the Section 7 takeaway:

"on the hardest problems, we observe that test-time compute does not improve performance substantially, suggesting that additional pretraining is more effective."

The consequence. Across all methods studied — beam search, best-of-N, lookahead search, sequential revisions, and their compute-optimal combinations — the hardest questions (difficulty bin 5) show near-zero improvement regardless of compute budget. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all methods and all budgets. In Figure 7 (right), bin 5 shows roughly 2–3% accuracy irrespective of the sequential-to-parallel ratio. In the FLOPs-matched comparison (Figure 9), the bin 5 lines are essentially flat near 0–5% and sit below the $14\times$ larger model's performance. This means the method offers no path forward for problems that genuinely exceed the base model's training distribution or reasoning capability. For a practitioner, this is a hard boundary: if the problem distribution includes a substantial fraction of "bin 5" difficulty items, test-time compute scaling provides no benefit and pretraining remains the only viable option. The paper's $4\times$ efficiency gains apply only to the subset of problems where the base model already has some non-trivial chance of success.

What evidence exists in the paper. The difficulty-bin analyses provide unambiguous evidence. Figure 3 (right): bin 5 shows ~1–3% accuracy for all search methods at all budgets. Figure 7 (right): bin 5 shows ~2–3% accuracy for all sequential-to-parallel ratios at 128 generations. Figure 9: bin 5 curves are flat and below the pretraining baseline across all $R$ values. The revision model data (Figure 6, left) shows pass@1 starting at ~18% and improving to ~24–25% through revision chains — but this aggregate hides the bin-5 near-zero performance.

Mitigation status. The paper is candid about this limitation but does not attempt to solve it. The Section 7 analysis explicitly states that "test-time compute is not a substitute for pretraining on the hardest problems" and that "some capabilities can only be acquired through pretraining." The authors do not propose any mechanism for breaking through this capability ceiling — the assumption is that it reflects a fundamental boundary of what test-time compute can achieve, and future work should focus on identifying which problems fall into which regime rather than on extending the regime itself.

Revisions and Search Are Studied Independently, Never Combined Into a Unified System

The assumption or constraint. The paper studies two complementary mechanisms — PRM-guided search (modifying the verifier/selection process) and iterative revisions (modifying the proposal distribution) — but never combines them into a single system. Section 8 explicitly acknowledges:

"we did not experiment with PRM tree-search techniques in combination with revisions"

The experiments treat these as independent scaling axes and demonstrate their complementary difficulty-dependent strengths: revisions excel on easy problems (local refinement) while beam search excels on medium problems (global exploration). The compute-optimal policy selects between them per difficulty bin but never deploys both simultaneously.

The consequence. The paper's reported performance numbers represent a lower bound on what a fully integrated system could achieve. A combined system — using the revision model as the proposal distribution within beam search, or using the PRM to guide which revision branches to pursue — could yield gains beyond either mechanism alone, particularly on medium-difficulty problems where both mechanisms show partial but complementary effectiveness. For a practitioner trying to maximize performance at a given budget, the paper provides no guidance on whether the gains from combining search and revisions would be additive, multiplicative, or sub-additive. The optimal allocation policy in the paper (Figures 4, 8) makes binary choices per difficulty bin, but the true optimal policy might involve allocating budget to both mechanisms within a single problem — a dimension the experiments do not explore.

What evidence exists in the paper. No experiment combines search and revisions. The PRM is trained on base model outputs, and the revision model uses a separate ORM (since the base-model PRM does not transfer well to revision outputs, Figure 15a). The paper never tests whether the PRM could score revision model outputs after domain-adaptive fine-tuning, or whether beam search over revision-generated candidates would outperform either mechanism alone. The closest the paper comes to combination is the hierarchical aggregation described in Appendix I (selecting the best answer within each revision chain via verifier, then selecting across chains), but this uses only the revision model, not PRM tree search.

Mitigation status. The paper explicitly flags this as future work in Section 8 and frames the current study as establishing the complementary strengths of the two axes, which is a prerequisite for combining them intelligently. The omission is understandable given the scope of the paper — studying both mechanisms independently, each with multiple search algorithms and hyperparameters, is already a substantial empirical contribution. However, a practitioner reading the paper for deployment guidance should understand that the reported numbers are from partial systems, not a fully integrated solution.

Single Benchmark, Single Model Family — Generality of Difficulty-Dependent Patterns Is Unverified

The assumption or constraint. All experiments use the MATH benchmark (500 test questions) with PaLM 2-S* as the base model. The paper states it "believe[s] this model is representative of the capabilities of many contemporary LLMs" (Section 4) and argues that MATH is a domain where test-time compute should help because it requires multi-step reasoning rather than factual recall. However, the generality of the central findings — particularly the difficulty-dependent scaling behavior — to other models, tasks, or domains is entirely unverified.

The consequence. Several aspects of the findings could be model-specific or task-specific in ways that undermine their deployment relevance:

PRM over-optimization behavior: The finding that beam search degrades easy-problem performance at high budgets (Figure 3, right) depends on the PRM's calibration properties, which are a function of PaLM 2-S*'s output distribution and the Monte Carlo rollout training procedure. A model with different confidence calibration or different error patterns might show different over-optimization thresholds, or might even benefit from aggressive search on easy problems.
Revision model effectiveness: The ability to learn revision from offline constructed trajectories depends on the base model's in-context learning capabilities and its output distribution's edit-distance structure. These vary substantially across model families.
Task domain: MATH consists of competition-level math problems requiring symbolic reasoning with unambiguous ground-truth answers. The difficulty-dependent patterns — sequential revisions work on easy problems, beam search works on medium problems — might not transfer to code generation (where structure matters differently), factual QA (where correctness depends on memorized knowledge rather than reasoning), or open-ended generation (where ground-truth signals are unavailable).
Difficulty bin definitions: "Easy" means pass@1 in the top quintile for PaLM 2-S* on MATH. A different model might have entirely different difficulty rankings for the same problems, and a different task might have a different relationship between pass@1 and the optimal strategy.

A practitioner deploying this method on a different model or task cannot assume the paper's difficulty-strategy mapping will transfer. They would need to replicate the full analysis pipeline (PRM training, difficulty estimation, strategy sweep, cross-validated policy selection) on their specific setup.

What evidence exists in the paper. The paper provides no cross-model or cross-task validation. All experiments are on a single model family with a single benchmark. The short-context capability check (Table 3) uses different benchmarks but tests only whether compression degrades base capabilities, not whether the difficulty-dependent scaling patterns replicate. The paper's argument that MATH is appropriate because it requires reasoning rather than factual recall (Section 4) is reasonable but does not substitute for empirical validation on other reasoning domains.

Mitigation status. Not addressed. The paper does not claim generality beyond PaLM 2-S* on MATH, but the abstract and introduction frame the findings in general terms ("we propose a compute-optimal scaling strategy", "our results show that..."). The limitation is acknowledged implicitly through the scope of the experiments but is not discussed as a threat to validity. Replication on at least one additional benchmark (e.g., a code generation task) and one additional model family would substantially strengthen the claims.

The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate and Revision Training Is Fragile

The assumption or constraint. The revision model is trained exclusively on sequences where all in-context answers are incorrect and the target is a correct answer. This means the model never sees examples of: (a) a correct answer in context followed by another correct answer (teaching it to preserve correctness), or (b) a correct answer followed by an incorrect one (teaching it to recognize when revision would be harmful). The paper reports:

"approximately 38% of correct answers get converted back to incorrect ones using a naive approach"

This high reversion rate is a direct consequence of the training data construction — the model is never taught to recognize "the current answer is already correct, do not change it."

The consequence. The revision model's sequential chain is inherently unstable: even when the model produces a correct answer at step $k$ , it has a 38% chance of "revising" it to an incorrect answer at step $k+1$ . This means longer revision chains (which the paper uses, up to 64 steps) do not monotonically approach correctness — they oscillate between correct and incorrect states. The paper mitigates this with within-chain selection (majority voting or verifier-based selection across all steps), but this is an imperfect patch: it requires generating many steps (wasting compute on incorrect revisions of correct answers) and relies on the verifier to distinguish correct from incorrect answers post-hoc. For a practitioner, this means the revision model cannot be trusted to produce a final answer by simply taking the last revision — a selection mechanism over the entire chain is essential, adding complexity and compute overhead.

Furthermore, the ReST $^{EM}$ experiment (Appendix K, Figure 16) reveals that revision training is fragile to the data generation procedure. Attempting to optimize the revision model with on-policy RL-style training caused performance to degrade substantially with sequential revisions — the model performed worse with more revision steps, the opposite of the desired behavior. The paper hypothesizes that on-policy data collection "exacerbates spurious correlations in revision data, causing the model to fail to learn the revision task properly." This suggests the positive results depend on specific training choices (offline data construction, edit-distance-based pairing) that may not transfer to other training pipelines or data distributions.

What evidence exists in the paper. The 38% reversion rate is reported in Section 6.1. The ReST $^{EM}$ degradation is shown in Figure 16 and discussed in Appendix K. The paper does not ablate the effect of including "correct-then-correct" or "correct-then-incorrect" trajectories in the training data, so it is unknown whether this would reduce the reversion rate or introduce other problems (e.g., the model learning to never revise, even when revision would help).

Mitigation status. Partially addressed. The within-chain selection mechanism (majority voting or verifier-based selection) reduces the impact of reversions by selecting the best answer from any point in the chain, but does not prevent the reversions from occurring and wasting compute. The paper does not propose a more principled solution — such as training the model with a "stop revising" token or including positive-feedback trajectories in the training data — and flags the fragility of revision training only implicitly through the ReST $^{EM}$ negative result rather than as a central limitation.

No Accounting for Latency or Wall-Clock Time in the Strategy Comparisons

The assumption or constraint. The paper measures compute in "generations" (number of complete solutions sampled), which is a reasonable proxy for total FLOPs but ignores the critical distinction between parallel and sequential computation. Sequential revisions are inherently serial — revision $k+1$ depends on the output of revision $k$ — while parallel best-of-N can be executed simultaneously with sufficient hardware. A strategy that allocates 128 generations as 64 sequential × 2 parallel chains takes approximately 64 times longer wall-clock time than one that runs 128 parallel samples simultaneously, even if both consume the same total FLOPs.

The consequence. The compute-optimal policies derived in the paper favor sequential-heavy strategies on easy problems (Figure 7, right: bin 1 performance is insensitive to ratio, bin 2 favors higher sequential ratios). At a budget of 128 generations, the optimal strategy for medium problems involves a balanced ratio (around 2^1 to 2^3 sequential-to-parallel), meaning dozens of sequential steps. For latency-sensitive applications — interactive assistants, real-time decision-making, live coding help — this may be completely impractical regardless of the accuracy gains. A user waiting for an answer cannot benefit from a strategy that takes 64 sequential forward passes through a 7B+ model, even if the total FLOPs are "efficient." The paper's efficiency metric (FLOPs or "generations") is a throughput metric, not a latency metric, and the two can diverge dramatically when comparing sequential and parallel strategies.

This tradeoff is particularly acute for the revision model. The paper shows that fully sequential revisions modestly outperform parallel sampling in aggregate (Figure 6, right: ~41.5% vs. ~39% at 64 generations), but this ~2.5 point gain comes at the cost of ~64× the latency of a fully parallel system. A practitioner deploying in a latency-constrained setting would likely prefer the parallel strategy despite its slightly lower accuracy, and the paper provides no framework for making this tradeoff explicit.

What evidence exists in the paper. The paper never discusses latency or wall-clock time. All efficiency comparisons are in terms of generations, FLOPs (Figure 3), or KV cache reduction. The Multi-Needle experiments (Table 2) measure end-to-end latency but only for the default strategies, not for the full sweep of sequential-to-parallel ratios that inform the compute-optimal policy. A practitioner cannot determine from the paper whether the optimal strategy at a given difficulty-budget pair is latency-feasible.

Mitigation status. Not addressed at all. The paper frames efficiency exclusively in terms of computational FLOPs and KV cache memory, never mentioning latency as a deployment constraint. This is a significant omission for a paper whose stated motivation includes on-device deployment and practical resource allocation. A latency-aware extension of the compute-optimal framework — adding a constraint on maximum wall-clock time and optimizing within that constraint — would be a natural and important follow-up, but the paper does not propose it.

7. Implications and Future Directions

How This Work Changes the Landscape

Activation Beacon reframes the long-context compression problem from a soft-prompt summarization challenge into an activation-space compression challenge, establishing that the transformer's own key-value cache — the very thing other methods try to shrink — is the ideal representational medium for compressed context. This is a conceptual shift, not an incremental improvement. Prior work treated context compression as a problem of producing summary embeddings (ICAE, AutoCompressors) or pruning tokens (LongLLMLingua, SnapKV), both of which impose information bottlenecks — a few soft tokens can encapsulate only so much, and token deletion at high ratios destroys coherence. Activation Beacon shows that by writing compressed information into the model's native attention memory format (per-layer, per-head KVs), you can preserve the rich relational structure of long contexts while still achieving aggressive compression ratios. The empirical evidence for this shift is stark: soft-token methods achieve 12.9–19.5 on Single-Doc QA while Activation Beacon achieves 34.9, matching the uncompressed baseline (Table 1), and Needle-in-a-Haystack retrieval accuracy at 8× compression is ~8/10 versus ~2–3/10 for soft-token alternatives (Figure 5).

The paper also resolves a tension in the compression literature: query-independent compression was presumed to be strictly inferior to query-dependent compression because it cannot allocate the compression budget adaptively. The intuition — that knowing the question lets you keep only what matters — was so compelling that query-dependent methods (SnapKV, LongLLMLingua) became the default paradigm. Activation Beacon demonstrates that this intuition breaks down under realistic multi-turn deployment. Query-independent compression matches or exceeds query-dependent methods in quality (Table 2: 9.10 vs. 8.85 accuracy on 3-turn Qwen-2-7B at 128K) while being dramatically faster (2.98s vs. 10.66s, a 3.6× speedup) because the context is compressed once and reused. This finding implies that the field's focus on query-aware compression may have been optimizing the wrong thing — in multi-turn settings, compression reuse dominates any per-question selectivity advantage.

Furthermore, the paper provides a diagnostic tool that was previously missing: the frozen-LLM design cleanly separates compression capability from language modeling capability. By freezing all 7B original parameters and training only beacon-specific projections (a tiny fraction of total parameters), the paper shows that long-context compression can be learned as an augmentation to a frozen model without degrading its original capabilities (Table 3: MMLU drops from 70.1 to 69.1, within noise). This falsifies the implicit assumption that effective compression requires retraining or fine-tuning the entire model, and opens the door to compression-as-a-plug-in architectures.

Research directions that become more attractive after this work:

Activation-space memory and retrieval mechanisms that write to and read from the KV cache as a learnable interface.
Plug-in adapters for frozen LLMs that add capabilities (compression, retrieval, multi-modal fusion) without touching base parameters.
Progressive, fine-grained compression workflows where different compression tokens specialize in different content spans through differentiated attention scopes.
Query-independent compression for streaming, multi-turn, and conversational settings where amortized compression cost dominates.

Research directions that become less attractive:

Soft-prompt summarization for long contexts, unless the information density requirements are extremely low. The capacity gap (~32× less per compression token) and the empirical performance gap (2–3× worse on most tasks) suggest this approach is fundamentally bottlenecked.
Token-deletion methods for high-ratio compression. Deleting 87.5% of tokens (8× compression) destroys syntactic coherence in ways that attention-based aggregation can avoid.
Architectures that require re-encoding or separate decoders to convert compressed representations back into usable formats. Activation Beacon's unification of compression and generation in a single forward pass is both simpler and faster.

Follow-Up Research This Work Enables

Mechanistic analysis of what beacon token activations actually encode. The paper's capacity argument — that per-layer, per-head KV activations provide L × h_k × d dimensions of storage versus D for a soft token — is compelling but entirely unvalidated by direct evidence. A follow-up study should probe what information is stored in beacon KVs: train linear classifiers on beacon activations at different layers to predict properties of the compressed span (named entities, sentiment, topic, specific facts), visualize attention patterns from generated tokens to beacon tokens to see which beacon positions are consulted for which types of queries, and ablate individual beacon heads to measure their contribution to downstream task performance. This would transform "KVs have large capacity" from an intuition into a verified mechanism, and might reveal that only certain layers or heads contribute meaningfully to compression — potentially enabling further compression of the beacon KVs themselves.

Scaling laws for compression training data volume. The paper uses 1B pre-training tokens and 30K fine-tuning samples, noting that this "can be quickly accomplished" and suggesting performance can "be further enhanced given more abundant and targeted training" (Table 4 discussion). A systematic scaling study — training Activation Beacon on 100M, 1B, 10B, and 100B tokens from RedPajama — would reveal whether compression quality follows a power-law scaling relationship with pre-training data, at what point returns diminish, and whether the optimal training data mixture differs from standard LM pre-training (e.g., does compression benefit more from structured text like Wikipedia or diverse text like web crawl?). This matters for practitioners deciding how much to invest in training: if 1B tokens captures 90% of the achievable gain, further investment is wasteful; if 10B tokens doubles performance, the current results significantly understate Activation Beacon's potential.

Extrapolation to compression ratios outside the training set and to longer training contexts. The paper trains with random ratios from {2, 4, 8, 16, 32} and contexts up to 20K, then evaluates at these ratios and at contexts up to 128K. But does the model handle ratios like 3×, 6×, or 64× that were never seen during training? Does its compression quality at 256K or 512K contexts continue to hold? A systematic evaluation — measuring NIAH accuracy at ratios {3, 6, 12, 24, 48, 64, 128} and context lengths from 32K to 1M — would map the generalization envelope. This is both an extension (showing how far the method scales) and a stress test (finding where it breaks). A negative result at 64× compression or 512K context would be as informative as a positive one, defining the practical operating range.

Combining token-level compression with dimensional compression (GQA, KV quantization). The paper correctly notes that Activation Beacon's token-level compression is orthogonal to dimensional compression techniques like grouped-query attention, multi-head latent attention, and KV cache quantization. A follow-up study should combine these: apply Activation Beacon to reduce the number of cached tokens by 8×, then quantize the remaining beacon KVs from float16 to int4, and measure the compound effect on memory, latency, and accuracy. If the multiplicative savings (8× token reduction × 4× precision reduction ≈ 32× total KV cache reduction) materialize without severe quality degradation, this would make 1M-token context inference feasible on consumer GPUs — a genuinely enabling result. The experiment should measure the interaction between compression artifacts: does quantization error compound with compression error, or are they independent? This is directly measurable by ablating each reduction independently and jointly at several compression-quantization combinations.

Replication on a non-LLaMA architecture and non-English benchmark. All experiments use Llama-2 and Qwen-2 on English-language tasks. Replicating the study on a model with a fundamentally different architecture — Mistral (sliding window attention), Gemma (different normalization), or a non-decoder model like T5 — would test whether Activation Beacon's success depends on specific architectural properties (RoPE position encoding, causal attention structure, GQA vs. MHA). Simultaneously, evaluating on a non-English long-context benchmark (e.g., a Chinese or multilingual variant of LongBench) would test whether the compression mechanism transfers across languages with different information density and syntactic structure. A negative result on Mistral would suggest the method depends on full causal attention; a negative result on Chinese would suggest the compression ratio-to-quality mapping is language-dependent.

Adaptive beacon allocation: varying the number of beacon tokens based on content importance. The current method uses a uniform compression ratio across all chunks — every 1024-token chunk gets the same number of beacon tokens regardless of its content. But some chunks are information-dense (containing key facts, entities, and relationships) while others are filler (transitions, boilerplate, repetitions). A natural extension is to train a lightweight "importance estimator" module that predicts, for each chunk, how many beacon tokens it needs to preserve its information, and allocate the beacon budget non-uniformly across chunks. This would transform Activation Beacon from a fixed-ratio compressor to a content-adaptive compressor that spends more representational capacity where it matters. The experiment would compare uniform allocation against an oracle (using downstream task performance as the importance signal) and against a learned estimator, measuring whether adaptive allocation improves quality at the same total beacon budget.

Practical Applications and Downstream Use Cases

Cost-efficient multi-turn document QA at scale. Consider a legal tech company processing long contracts (100K+ tokens) where lawyers ask multiple questions about each document. Using full attention, each question-turn costs ~4.4 seconds on Qwen-2-7B at 128K (Table 2), and 3 questions cost ~6.2 seconds total with the full KV cache consuming ~7.3 GB of GPU memory. With Activation Beacon at 8× compression, the document is compressed once (at a one-time cost embedded in the first turn's ~2.4 seconds), and subsequent turns require only ~0.3 additional seconds each because the beacon KVs are reused. Total latency for 3 questions drops from ~6.2s to ~3.0s (2.1× faster), KV cache memory drops from ~7.3 GB to ~0.9 GB (8× reduction), and accuracy is essentially unchanged (9.10 vs. 9.20). For a service handling thousands of documents daily with multiple queries each, the combined latency reduction and memory savings directly translate to lower GPU costs and better user experience — and the query-independent nature means no re-computation when lawyers switch questions.

On-device long-context processing with small models. Activation Beacon's frozen-LLM design and small trainable parameter footprint make it particularly suitable for edge deployment. A 7B model quantized to 4 bits requires ~3.5 GB of GPU memory for parameters alone. Adding a full-attention 128K KV cache would require another ~7.3 GB, exceeding typical edge GPU capacity (e.g., an RTX 4070 with 12 GB VRAM). With Activation Beacon at 8× compression, the KV cache drops to ~0.9 GB, bringing total memory to ~4.4 GB — well within edge constraints. This enables genuinely long-context applications on consumer hardware: processing full-length research papers, analyzing legal documents offline, or running personal knowledge base QA without cloud dependency. The 2× latency reduction (Table 2) also makes the experience interactive rather than batch-oriented. The key enabling numbers: 8× KV cache reduction from ~7.3 GB to ~0.9 GB at 128K context, and accuracy preserved to within ~0.05–0.10 of the uncompressed baseline on retrieval tasks.

Streaming and incremental context processing for live applications. In live transcription or meeting assistant scenarios, the context grows continuously as new speech is transcribed. With full attention, each new utterance requires re-encoding the entire growing context. With Activation Beacon, new chunks are compressed independently, and previous beacon KVs are reused without modification. This means the per-utterance compute cost is constant — compress the new chunk only — rather than linear in the total context length. For a 2-hour meeting generating ~50K tokens of transcript, processing the final utterance with full attention requires attending to all 50K tokens, while Activation Beacon only processes the current 1024-token chunk attending to ~6K accumulated beacon KVs. The paper's incremental update property (Section 3.1: "allows for incrementally updating the compression results in multi-turn scenarios") is directly applicable here, though latency numbers for streaming specifically are not provided in the paper — they would need to be benchmarked.

Training data generation for long-context fine-tuning. When preparing fine-tuning data for long-context models (e.g., generating QA pairs over book-length documents), the cost of running full-attention inference on thousands of long documents can be prohibitive. Activation Beacon's 2× FLOPs reduction at 128K context (Figure 3B) translates to roughly halved generation time and cost. More importantly, the 8× KV cache reduction means more documents can be processed concurrently on the same hardware — instead of 1 document occupying ~7.3 GB of KV cache, 8 documents can be processed simultaneously in ~7.2 GB total (0.9 GB each). This 8× throughput improvement for batch inference tasks is not explicitly measured in the paper but follows directly from the KV cache reduction, assuming compute is not the bottleneck. For organizations generating millions of training examples from long documents, this cost reduction is substantial and directly actionable.

When to Prefer This Method

Prefer Activation Beacon over soft-token compression (ICAE, AutoCompressors) when:

The downstream task requires fine-grained information retrieval (e.g., Needle-in-a-Haystack, factoid QA) rather than just gist-level summarization. The soft-token capacity bottleneck causes catastrophic failure on these tasks (Figure 5: ~2–3/10 accuracy vs. ~8/10 for Activation Beacon at 8× compression), while Activation Beacon preserves fine-grained facts.
You need compression ratios of 4× or higher. Soft-token methods degrade sharply beyond 2× (Figure 5B, C), while Activation Beacon maintains quality to 16×.
You cannot afford the compute to re-encode compressed representations through a separate decoder. Activation Beacon unifies compression and generation in a single forward pass.

Prefer Activation Beacon over query-dependent token pruning (SnapKV, LongLLMLingua) when:

The deployment scenario involves multiple questions about the same document (multi-turn conversations, batch QA over shared context). Activation Beacon's query-independent compression is done once and reused (Table 2: 3.0s vs. 10.7s for SnapKV at 3 turns, 3.6× faster), while query-dependent methods re-compute compression for each question.
The context may exceed the backbone LLM's native window size. SnapKV cannot compress beyond the window limit (Section 4.4), while Activation Beacon's chunked progressive workflow handles arbitrary lengths (Figure 4: accurate retrieval at 128K despite 20K training limit).
You need 8× or higher KV cache reduction with competitive or superior quality. SnapKV's quality is competitive at lower compression (Table 2: 9.45 vs. 9.70 at 1-turn) but Activation Beacon provides guaranteed 8× KV cache reduction while SnapKV's compression ratio depends on attention sparsity patterns and may vary.

Prefer query-dependent pruning (SnapKV) over Activation Beacon when:

The deployment is strictly single-turn (one question per document) and the context fits within the LLM's window. SnapKV achieves similar quality (Table 2: 9.45 vs. 9.70 at 1-turn Qwen-2-7B) with a simpler implementation that does not require training additional projection matrices or modifying the attention mechanism.
Training the beacon modules is infeasible (no access to 1B pre-training tokens and 8×A800 compute). SnapKV is inference-only with no training required.

Prefer scaling the backbone model size and using no compression when:

The problem distribution includes a high fraction of genuinely hard tasks that saturate the compressed representation's capacity, though the paper does not identify a specific difficulty threshold where Activation Beacon fails — the NIAH results suggest it is robust.
The additional 10–15% parameter overhead from beacon projection matrices (exact count depends on the model dimensions) and the beacon token embedding introduces unacceptable model size increase for extreme memory-constrained edge deployment where every megabyte matters.

Dependency tree

Papers this one builds on, walking backwards through the same domain and shared topics.

YARN: EFFICIENT CONTEXT WINDOW EXTENSION OF LARGE LANGUAGE MODELS2023· training-methods← LLM-judged similar
- Extending Context Window of Large Language Models via Positional Interpolation2023· context-optimization← LLM-judged similar
- LM-Infinite: Simple On-the-Fly Length Generalization for Large Language Models2023· inference-optimization← LLM-judged similar
- Augmenting Language Models with Long-Term Memory2023· llm-systems← LLM-judged similar
- Focused Transformer: Contrastive Training for Context Scaling2023· llm-systems← LLM-judged similar
- Continual Pre-Training of Large Language Models: How to (re)warm your model?2023· training-methods← in-domain predecessor
- Lost in the Middle: How Language Models Use Long Contexts2023· context-optimization← topic thread
- A Controlled Study on Long Context Extension and Generalization in LLMs2024· training-methods← similar topic
Repelling Random Walks2023· llm-systems← in-domain predecessor
- Ring Attention with Blockwise Transformers for Near-Infinite Context2023· llm-systems← in-domain predecessor
Effective Long-Context Scaling of Foundation Models2023· training-methods← topic thread
- LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models2023· training-methods← in-domain predecessor
A Comprehensive Survey on Long Context Language Modeling2025· llm-systems← similar topic
- Gemini Embedding: Generalizable Embeddings from Gemini2025· llm-systems← in-domain predecessor
- MoBA: Mixture of Block Attention for Long-Context LLMs2025· architecture← topic thread

What it influenced

Later papers in the same chain. Direction reversed.

Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads2024· llm-systems→ in-domain predecessor
- The Case for Co-Designing Model Architectures with Hardware2024· llm-systems→ in-domain predecessor
- SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills2023· llm-systems→ similar topic
A Human-Inspired Reading Agent with Gist Memory of Very Long Contexts2024· agents→ topic thread
- LongCodeZip: Compress Long Context for Code Language Models2025· code→ LLM-judged similar
- Mixture-of-Agents Enhances Large Language Model Capabilities2024· agents→ in-domain predecessor
- Benchmarking and Building Long-Context Retrieval Models with LoCo and M2-BERT2024· retrieval→ topic thread
- PREPING: Building Agent Memory without Tasks2026· agents→ similar topic
A Comprehensive Survey on Long Context Language Modeling2025· llm-systems→ similar topic
- BitNet b1.58 2B4T Technical Report2025· llm-systems→ in-domain predecessor
- Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Models2025· multimodal→ topic thread
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities2025· llm-systems→ similar topic