EFFICIENT STREAMING LANGUAGE MODELS WITH ATTENTION SINKS

🎯 Pitch

LLMs can process infinite text streams without fine-tuning—just by keeping a few initial tokens as “attention sinks.” These tokens soak up overwhelming attention scores that would otherwise destabilize the model, enabling stable generation on over 4 million tokens and a 22× speedup.

1. Executive Summary

This paper introduces StreamingLLM, a framework that enables large language models trained with finite attention windows to generalize to infinite sequence lengths during streaming deployment without any fine-tuning. Analyzing Llama-2, MPT, Falcon, and Pythia on the PG-19 benchmark, the authors identify a phenomenon they term attention sinks — initial tokens that accumulate disproportionately high attention scores across most layers and heads regardless of semantic relevance, driven by the SoftMax function's requirement that attention scores sum to one. StreamingLLM preserves these attention sink tokens (as few as four initial tokens suffice) alongside a rolling KV cache of recent tokens, achieving stable language modeling perplexity on sequences of up to 4 million tokens while delivering up to 22.2× speedup over the sliding window recomputation baseline. The paper further demonstrates that pre-training with a dedicated learnable sink token enables streaming deployment with only a single sink token, establishing that attention sinks are a fundamental architectural property of autoregressive Transformers rather than an artifact of specific training data.

2. Context and Motivation

The Core Problem: LLMs Cannot Operate on Infinite Streams

The fundamental challenge this paper addresses is deceptively simple: how do we deploy a large language model to process an infinite-length stream of text when the model was only trained on sequences of finite length? This is not a hypothetical scenario — it's the exact requirement for any persistent LLM application like a multi-round dialogue system, a daily assistant, or a live transcription service. The model needs to keep generating coherent responses indefinitely, building on recent context while discarding distant history.

This problem decomposes into two distinct but coupled challenges, both of which the paper identifies explicitly in Section 1:

Memory explosion from the KV cache. During autoregressive decoding, Transformer-based LLMs cache the Key and Value states of every previously generated token to avoid recomputing attention from scratch. For a conversation lasting hours or days, this cache grows linearly — and unboundedly — with the sequence length, leading to excessive memory consumption and increasing per-token decoding latency (Pope et al., 2022). Figure 1(a) illustrates this: dense attention over $T$ cached tokens has $O(T^2)$ time complexity and requires storing all $T$ KV pairs.
Length extrapolation failure. Even if memory were infinite, LLMs perform poorly when generating text longer than their pre-training attention window. Models like Llama-2 are trained with a fixed context length (e.g., 4096 tokens), and their performance degrades sharply when this limit is exceeded (Press et al., 2022; Chen et al., 2023). This means that even if we could afford to cache all tokens, the model's outputs would become unreliable once the conversation crosses the training length boundary.

These two challenges interact in a particularly frustrating way: the natural solution to the memory problem (discarding old KV states) turns out to catastrophically break the model's performance, creating a bind that the paper sets out to resolve.

Why This Matters: Real-World Deployment Demands Persistence

The importance of this problem extends well beyond academic curiosity. The paper grounds its motivation in concrete deployment scenarios (Section 1 and Appendix A):

Multi-round dialogue systems like ChatGPT or Claude engage in extended conversations with users. An "ideal ChatBot assistant" should be able to "stably work over the content of recent day-long conversations" (Section 1). Current systems handle this by truncating conversation history — either by discarding the oldest turns or by summarizing distant context — both of which lose information that might become relevant later.
Daily assistant applications based on LLMs need to "function seamlessly over extended periods" (Appendix A), basing responses on recent interactions without requiring frequent cache refreshes. Traditional approaches force a reset when the conversation length exceeds the training window, causing the model to lose recent context at arbitrary points.
Streaming transcription and real-time processing demands continuous operation without the latency spikes that come from recomputing cached states or the quality degradation from aggressive truncation.

The paper explicitly frames the need: "Can we deploy an LLM for infinite-length inputs without sacrificing efficiency and performance?" (Section 1). This is not about extending context length to handle longer documents — it's about an entirely different deployment paradigm where the model runs continuously and must remain stable indefinitely.

The Gap Between Context Extension and Streaming

A crucial distinction the paper draws — one that is easy to miss but essential for understanding its contribution — is that progress in context window extension does not solve the streaming problem. The related work (Section 2) cleanly separates three lines of research:

Length extrapolation aims to enable models trained on short sequences to handle longer ones at test time. This includes work on relative position encodings like RoPE (Su et al., 2021) and ALiBi (Press et al., 2022). However, the paper demonstrates that these methods break down when the test sequence length is "vastly greater than the training length" (Section 2). Figure 3 confirms this empirically: dense attention (which relies on the model's extrapolation ability) fails when the input length surpasses the pre-training attention window size across Llama-2, MPT, Falcon, and Pythia. The perplexity spikes sharply — for Llama-2-13B, it jumps to over 5000 (Table 1).

Context window extension focuses on increasing the maximum number of tokens the model can process in a single forward pass. Recent work achieves this through position interpolation (Chen et al., 2023; kaiokendev, 2023; Peng et al., 2023) followed by fine-tuning. But these methods "only extend LLMs' context window to a limited extent" (Section 2) — they don't enable infinite-length operation. You might extend from 4K to 32K or even 128K tokens, but you'll eventually hit whatever that extended limit is. The paper explicitly states this is "orthogonal to our focus" and "could be integrated with our techniques" (Section 2), but fundamentally doesn't address the unbounded-streaming requirement.

Improving long-text utilization targets the problem that models don't effectively use even the context they do have access to (Liu et al., 2023; Li et al., 2023). This is about attention quality, not attention capacity. Again, orthogonal to the streaming problem.

The key insight is that none of these three lines of work provides what streaming deployment requires: the ability to run a model continuously for an unbounded number of tokens with stable performance and bounded memory. The paper's contribution — enabling exactly this — fills a gap that existing approaches either don't address or address only with severe limitations.

Window Attention: The Intuitive Solution That Fails

The most natural approach to bounding KV cache memory while maintaining efficiency is window attention (Beltagy et al., 2020), illustrated in Figure 1(b). The idea is straightforward: maintain a fixed-size sliding window over the most recent $L$ tokens' KV states. Once the cache is full (after the first $L$ tokens), each new token pushes the oldest cached token out, keeping memory usage constant at $O(L)$ and decoding time at $O(TL)$ rather than $O(T^2)$ .

This approach has clear efficiency advantages that make it attractive for deployment:

Constant memory footprint after the cache fills.
Constant per-token decoding time (linear in cache size, not sequence length).
No recomputation of previous states.

The problem — and this is the paper's central empirical finding that motivates the entire investigation — is that window attention catastrophically fails when the initial tokens are evicted. Figure 3 shows this dramatically: as soon as the sequence length exceeds the cache size (meaning the very first tokens are pushed out), perplexity skyrockets. For Llama-2-13B with a cache size of 1024, perplexity jumps from roughly 5.4 to over 5158 (Table 1). The model essentially breaks — it starts producing nonsensical outputs.

This is the critical diagnostic signal. The fact that removing just the initial tokens (not random tokens, not semantically important tokens, but specifically the first few tokens) causes such catastrophic failure tells us something fundamental about how attention is operating in these models.

The Competing Baseline: Sliding Window with Re-computation

Another approach that avoids the window attention failure is sliding window with re-computation, illustrated in Figure 1(c). Rather than maintaining a continuous KV cache, this method rebuilds the KV states for the $L$ most recent tokens from scratch at each decoding step. Because this re-computation uses dense attention within the window, it doesn't suffer from the initial-token eviction problem — the attention computation is always over a contiguous, complete segment of recent text.

The paper acknowledges that this baseline achieves strong performance: "it performs well on long texts" (Section 1), and Figure 3 confirms its perplexity is essentially the oracle for long-sequence quality. However, it has a fatal efficiency drawback: the re-computation of $L$ tokens at each step requires quadratic attention within that window, yielding $O(TL^2)$ total complexity. This makes it "significantly slower" and "impractical for real-world streaming applications" (Section 1). The speedup numbers in Section 4.5 quantify this: StreamingLLM achieves up to 22.2× per-token speedup over this baseline.

The Observational Discovery: Attention Sinks

The paper's motivation crystallizes around an empirical observation that explains why window attention fails. Through systematic visualization of attention maps from all layers and heads of Llama-2-7B (Figure 2) and Llama-2-70B (Figure 13, Appendix G), the authors uncover a consistent and surprising pattern:

In the bottom two layers (layers 0 and 1), attention follows an intuitive "local" pattern — recent tokens receive more attention than distant ones. This is what we might expect from a well-behaved attention mechanism.
Beyond the bottom two layers, a completely different pattern emerges: the model heavily attends to the initial tokens across virtually all layers and all heads, regardless of their semantic content. The first few tokens receive disproportionately large attention scores even when they are entirely unrelated to the prediction task.

This pattern is not model-specific. The paper demonstrates it across Llama-2-7B (Figure 2), Llama-2-70B (Figure 13), and shows that it extends to MPT, Falcon, and Pythia through the consistent failure of window attention at cache boundaries (Figure 3). Even more broadly, attention sinks appear in encoder-only Transformers like BERT (Figure 14, Appendix H) and in Vision Transformers (Darcet et al., 2023, which identified a parallel phenomenon they call "registers").

The quantitative analysis in Appendix F (Figure 12) makes this concrete for longer sequences: when processing 4096-token sequences, the 4096th token allocates more than half of its total attention to the first token in most layers. This is not a subtle bias — it's a dominant effect.

Why Initial Tokens Become Sinks: The SoftMax Constraint

The paper provides a compelling mechanistic explanation for why this happens (Section 3.1). The key is the SoftMax function that normalizes attention scores:

$\text{SoftMax}(x)_i = \frac{e^{x_i}}{e^{x_1} + \sum_{j=2}^{N} e^{x_j}}, \quad x_1 \gg x_j, j \in 2, \ldots, N$

The SoftMax output must sum to 1 across all attended tokens. This means that even when the current query doesn't have a strong match with any particular previous token — when all the $x_j$ values are relatively small or similar — the model still needs to allocate attention somewhere. It cannot assign zero attention to all tokens because the denominator requires at least some mass to be distributed. The model therefore learns to "dump" unnecessary attention onto specific tokens that serve as repositories.

The reason initial tokens are chosen for this role is elegantly explained by the autoregressive structure: "initial tokens are visible to almost all subsequent tokens because of the autoregressive language modeling nature, making them more readily trained to serve as attention sinks" (Section 3.1). In other words, because the first token is part of the context for every single subsequent prediction during training, it's the most convenient and consistently available place to offload excess attention mass.

A critical experiment in Table 1 proves that this is about absolute position, not semantic content. When the first four tokens are replaced with meaningless linebreak tokens ("\n"), the model still heavily attends to these now-semantically-empty positions, and reintroducing them restores perplexity to near-normal levels (5.60 vs. 5.40 for the original initial tokens). This definitively distinguishes the attention sink hypothesis from the alternative explanation that initial tokens happen to be semantically important.

Why Multiple Sinks Are Needed in Existing Models

The paper observes that existing models need approximately four initial tokens as attention sinks, not just one (Table 2). Adding only one or two initial tokens to the rolling cache doesn't fully restore perplexity; four is the threshold where performance saturates. The explanation (Section 3.1) is that these models "didn't include a consistent starting token across all input samples during pre-training." Even though Llama-2 prefixes paragraphs with a <s> token, this happens "before text chunking, resulting in a mostly random token occupying the zeroth position." Without a stable, learnable token at position 0 that the model can rely on across all training examples, the model spreads its attention-sink behavior across the first few positions, learning to treat them collectively as the sink.

This observation directly motivates the pre-training experiment in Section 3.3: if models are trained from scratch with a dedicated, learnable sink token at the start of every training sample, perhaps they can learn to use just that single token as the attention sink, enabling streaming deployment with even fewer preserved tokens.

The Paper's Position: Enabling Streaming Without Fine-Tuning

The paper positions StreamingLLM as a framework that:

Requires no fine-tuning of existing pre-trained models. By simply preserving the attention sink tokens (the first four) alongside a rolling window of recent tokens, the model can operate stably on sequences of arbitrary length. This is crucial because it makes StreamingLLM immediately applicable to any existing autoregressive LLM that uses relative position encoding.
Addresses a fundamentally different problem from context extension. The paper explicitly states: "StreamingLLM efficiently generates coherent text from tokens within the KV cache without extending the LLMs' context length. It suits continuous operation needs with minimal memory use and past data reliance" (Section 1). This decoupling of the pre-training window size from the actual generation length is presented as a novel capability.
Is complementary to, not competitive with, context extension methods. The paper demonstrates this in Section 4.3 by applying StreamingLLM to two context-extended models (LongChat-7b-v1.5-32k and Llama-2-7B-32K-Instruct), showing that StreamingLLM works on top of extended context windows to enable streaming behavior over longer local context.
Identifies a universal architectural property of Transformers that has implications beyond just autoregressive LMs. The attention sink phenomenon is documented in BERT (Appendix H) and connected to concurrent work on vision transformers (Darcet et al., 2023), suggesting that the SoftMax constraint fundamentally shapes attention behavior across all Transformer architectures.
Provides a forward-looking recommendation for future model training: include a dedicated sink token in all pre-training samples to simplify streaming deployment. The pre-training experiments (Section 3.3, Table 3) validate that this doesn't harm standard performance while dramatically improving streaming behavior.

Summary of the Motivation Path

The paper's motivation follows a clear logical chain: (1) Streaming deployment of LLMs is practically important but currently impossible because of the coupled memory and extrapolation problems → (2) Window attention solves the memory problem but catastrophically fails, and we need to understand why → (3) Attention visualization reveals that initial tokens serve as "attention sinks" that absorb excess SoftMax mass, and their eviction dramatically distorts attention score distributions → (4) Preserving these sink tokens alongside a rolling cache is sufficient to restore stable performance → (5) This insight enables a simple, fine-tuning-free method (StreamingLLM) for infinite-length streaming, and suggests better pre-training practices (sink tokens) for future models.

3. Technical Approach

3.1 Reader Orientation

We need to understand the StreamingLLM system, which is a method for modifying how an existing pre-trained language model stores and computes attention during text generation. The system solves the problem that language models break when generating sequences longer than their training length by selectively discarding most past tokens while preserving a tiny number of special "attention sink" tokens — tokens that absorb excess attention scores and stabilize the model's behavior, enabling infinite-length generation without any fine-tuning.

3.2 Big-Picture Architecture (Diagram in Words)

The StreamingLLM system has four major components that work together at inference time:

The pre-trained base language model (e.g., Llama-2, MPT, Falcon, Pythia) — this is the frozen autoregressive Transformer that generates text token by token. It uses relative position encodings (either RoPE or ALiBi) so that attention computation depends on distances between token positions rather than absolute positions.
The KV cache — the stored Key and Value tensors from previous decoding steps that prevent recomputing attention from scratch. In standard dense attention, this cache grows linearly with sequence length. In StreamingLLM, the cache is bounded to a fixed size and divided into two distinct segments: attention sinks (preserved initial tokens) and the rolling KV cache (the most recent tokens).
The attention mechanism with SoftMax normalization — this is the core computation that mixes information from previous tokens. The SoftMax function forces attention scores to sum to 1 across all contextual tokens, which creates the need for attention sinks. StreamingLLM exploits rather than fights this property.
The cache management policy — the rule that determines which tokens stay in the KV cache and which get evicted when new tokens are generated. This is the key operational innovation: when the cache is full, evict the oldest non-sink token, keeping the initial sink tokens permanently and maintaining a sliding window of recent tokens.

Information flows as follows: a sequence of text enters the model → initial tokens (typically the first four) are generated and their KV states are flagged as permanent attention sinks → subsequent tokens fill the rolling cache → when the rolling cache reaches its capacity, the oldest non-sink token's KV state is evicted to make room for the new token → attention is computed over the combination of preserved sink tokens and recent rolling tokens → the model generates the next token based on this bounded context → the process repeats indefinitely.

3.3 Roadmap for the Deep Dive

First, the attention sink phenomenon and its mechanistic cause — why the SoftMax function forces tokens to accumulate attention scores, and why initial tokens are empirically chosen as sinks in autoregressive models. This is the foundation that motivates every design decision.
Second, the empirical study that proves attention sinks are about absolute position, not semantic content — the linebreak substitution experiment (Table 1). This justifies why preserving arbitrary initial tokens works.
Third, the KV cache design and eviction policy — how the cache is partitioned into sink and rolling segments, and how tokens move between these segments during generation. This is the core operational mechanism.
Fourth, the positional encoding strategy — how relative position encodings (RoPE and ALiBi) are adapted to work with non-contiguous token positions in the cache. This is a subtle but critical detail without which the method would fail.
Fifth, the pre-training methodology for future models — training with a dedicated sink token so that streaming deployment requires only a single preserved token instead of four.
Sixth, the relationship to alternative approaches — why SoftMax-off-by-one (Zero Sink) only partially solves the problem, and what that reveals about the nature of attention sinks.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical analysis paper whose core idea is that the SoftMax function in Transformer attention creates an architectural requirement for "attention sinks" — tokens that absorb excess probability mass — and that preserving these sink tokens alongside a rolling window of recent context enables infinite-length streaming deployment without retraining.

The SoftMax Constraint as the Root Cause of Attention Sinks

The central mechanistic insight of the paper is that attention sinks are not a learned behavior that could be trained away, but rather a direct consequence of the SoftMax function that normalizes attention scores. The paper makes this argument through the following chain of reasoning.

The attention computation in a Transformer for a single query token producing attention weights over $N$ previous tokens takes the form:

$\text{SoftMax}(x)_i = \frac{e^{x_i}}{\sum_{j=1}^{N} e^{x_j}}$

where $x_i$ is the raw attention score (the dot product of the query with the i-th key, plus any position bias) for token $i$ , and the denominator sums over all $N$ tokens in the context.

What it computes: This equation converts $N$ raw attention scores (which can be any real numbers) into $N$ normalized weights that are all positive and sum to exactly 1. Each weight represents the proportion of the query's total attention that is directed at token $i$ . The output is a probability distribution over the previous tokens.

Why this form: The SoftMax is a differentiable approximation to the argmax operation that works well for gradient-based training. But it has a critical property that creates the attention sink requirement: there is no mechanism to assign zero attention to any token. The exponential function $e^{x_i}$ is always positive (for any real $x_i$ ), so every single token in the context must receive some fraction of the attention mass. Even if the query has no meaningful relationship with most previous tokens — even if all the raw scores $x_j$ are very negative or very similar — the model must still distribute the total attention mass of 1.0 across all $N$ tokens.

The paper explains this in operational terms: "The nature of the SoftMax function prevents all attended tokens from having zero values. This requires aggregating some information from other tokens across all heads in all layers, even if the current embedding has sufficient self-contained information for its prediction" (Section 3.1). In other words, when the model's current representation already contains everything it needs to predict the next token, the attention mechanism becomes a burden — it must still attend to something — and the model learns to dump this mandatory attention onto designated sink tokens.

The specific form with $x_1 \gg x_j$ for $j \in 2, \ldots, N$ shown in Equation 1 of the paper describes the equilibrium that emerges from training:

$\text{SoftMax}(x)_i = \frac{e^{x_i}}{e^{x_1} + \sum_{j=2}^{N} e^{x_j}}, \quad x_1 \gg x_j, j \in 2, \ldots, N$

where $x_1$ is the raw attention score for the first token, and $x_j$ are scores for all subsequent tokens.

What this form describes: The model learns to assign a disproportionately large raw score $x_1$ to the initial tokens so that the majority of the SoftMax probability mass concentrates on these tokens. The first term $e^{x_1}$ in the denominator dominates the sum, leaving only a small residual probability to be distributed among the remaining $N-1$ tokens. This means the model can satisfy the "must attend to everything" constraint while functionally ignoring most tokens — it attends heavily to the sink, and weakly to everything else based on their actual relevance.

Why this form emerges: During training, the model discovers that consistently placing a large score on initial tokens is the most efficient strategy. Because initial tokens are visible to all subsequent tokens (thanks to autoregressive masking), they can serve as universal sinks regardless of sequence position. If the model tried to use semantically-meaningful intermediate tokens as sinks, those tokens might not be available as sinks for earlier positions (they haven't been generated yet) or might inadvertently bias the attention distribution toward content that interferes with prediction. Initial tokens are always available and can be made semantically neutral through training.

Empirical Proof That Position, Not Semantics, Matters

The paper designs a crucial experiment (Table 1) to distinguish whether initial tokens receive high attention because of their semantic content or because of their absolute position. This is central to the StreamingLLM strategy because if semantics mattered, preserving arbitrary initial tokens wouldn't help.

The experimental setup uses the Llama-2-13B model on the first book of the PG-19 test set and compares three cache configurations:

0 + 1024 (pure window attention): Only the 1024 most recent tokens are in the cache. No initial tokens are preserved. This achieves a perplexity of 5158.07.
4 + 1020 (original initial tokens): The four original initial tokens are preserved alongside the 1020 most recent tokens. This achieves a perplexity of 5.40 — a nearly complete recovery, demonstrating that preserving initial tokens solves the window attention failure.
4"\text{\textbackslash n}" + 1020 (semantically-empty initial tokens): The four original initial tokens are replaced with four linebreak tokens "\n", which carry essentially no semantic meaning. These meaningless tokens are then preserved alongside the 1020 most recent tokens. This achieves a perplexity of 5.60 — comparable to preserving the original initial tokens.

What this experiment demonstrates: The absolute position of the initial tokens (their index in the sequence, which is 0, 1, 2, 3) is what matters for attention sink behavior, not their semantic content. The model has learned to attend to positions 0-3 regardless of what tokens occupy those positions. This is strong evidence that the attention sink mechanism is a learned positional bias rather than content-dependent attention.

This result has two profound implications for the StreamingLLM method: (1) It confirms that preserving the actual initial tokens of any conversation or streaming session — whatever they happen to be semantically — will work as attention sinks, because the model only cares about their position. (2) It suggests that the model has not merely memorized the specific tokens that appeared at initial positions during training, but has learned a more abstract positional bias that generalizes to arbitrary content at those positions.

The KV Cache Design and Eviction Policy

With the attention sink phenomenon established, StreamingLLM's operational mechanism is remarkably simple. The KV cache is conceptually divided into two segments (Figure 4), and the eviction policy treats these segments differently.

Cache initialization (sequence length ≤ cache size): During the initial phase of generation, when the total number of generated tokens is less than or equal to the total cache capacity, the KV cache operates identically to standard dense attention. All tokens' Key and Value tensors are stored. No eviction occurs. The attention computation includes all $T$ tokens, where $T \leq L_{\text{cache}}$ .

Cache partitioning (sequence length > cache size): Once the number of generated tokens exceeds the total cache capacity $L_{\text{cache}}$ , the cache is divided into two segments:

Attention sink segment: The first $k$ tokens' KV states are permanently preserved and never evicted. The paper empirically determines $k = 4$ as the effective number for standard pre-trained models (Table 2). These tokens are typically the very first tokens of the conversation or streaming session — often a <s> token (beginning-of-sequence marker), an initial user query, or the first few words of the model's response, depending on the application.
Rolling KV cache segment: The remaining $L_{\text{cache}} - k$ slots hold the most recent tokens' KV states. This segment operates as a sliding window: when a new token is generated, its KV state is added to the rolling cache, and if the rolling cache is full, the oldest token in the rolling cache is evicted. Crucially, the oldest rolling token is not necessarily the $k$ th token in the original sequence — it's the oldest token still in the rolling segment, which advances forward with each new token.

Eviction policy at step $T > L_{\text{cache}}$ : When generating token $T$ (the $T$ th token in the sequence), the cache contains tokens at positions $[0, 1, \ldots, k-1]$ (the sinks) and $[T - (L_{\text{cache}} - k), \ldots, T-1]$ (the recent rolling window). The oldest token in the rolling segment — at position $T - (L_{\text{cache}} - k)$ — is evicted to make room for the new token. The sink tokens at positions $[0, \ldots, k-1]$ remain permanently.

Attention computation at each step: At each decoding step, attention is computed over exactly $L_{\text{cache}}$ tokens: $k$ sink tokens plus $L_{\text{cache}} - k$ recent tokens. This means the attention computation is $O(L_{\text{cache}})$ per token, independent of the total sequence length $T$ , yielding $O(T \cdot L_{\text{cache}})$ total complexity for generating $T$ tokens. This is linear in $T$ rather than quadratic.

Memory footprint: The memory required to store the KV cache is proportional to $L_{\text{cache}}$ , not $T$ . For a model with $H$ layers, $D$ head dimension, and $N_h$ heads, the KV cache stores $H \times N_h \times D$ values per token (for both Key and Value). With $L_{\text{cache}} = 2048$ and typical model sizes, this translates to constant memory usage independent of how long the model runs.

What this design solves: By permanently preserving the attention sinks, the denominator of the SoftMax function retains the dominant terms ( $e^{x_1}$ through $e^{x_4}$ from the sink tokens) that the model expects. The attention score distribution therefore remains close to what it was during training, even as tokens in the middle of the sequence are discarded. The model never enters the regime where the SoftMax denominator collapses due to missing sink tokens — which is precisely what causes the perplexity spike in window attention.

Why four sink tokens (not one, not eight): Table 2 provides the empirical evidence for the choice of $k=4$ . The experiment tests cache configurations of the form $x + y$ where $x$ initial tokens are preserved and $y$ recent tokens are kept, across four model families:

With $0 + 2048$ (zero sinks), perplexity is catastrophic: 17.90 for Falcon-7B, 460.29 for MPT-7B, 21.62 for Pythia-12B, 3359.95 for Llama-2-7B (the last uses a 4096-total cache so $0+4096$ ).
With $1 + 2047$ (one sink), perplexity drops substantially but not fully: 12.12 for Falcon-7B, 14.99 for MPT-7B, 11.95 for Pythia-12B, 11.88 for Llama-2-7B. For Llama-2-7B especially, the gap from 11.88 to the fully-recovered 9.59 is significant.
With $4 + 2044$ (four sinks), perplexity reaches the effective floor: 12.12 for Falcon, 14.99 for MPT, 12.09 for Pythia, 9.59 for Llama-2-7B.
With $8 + 2040$ (eight sinks), there is negligible further improvement: 12.12, 14.98, 12.02, and 9.54 respectively.

The paper's explanation for why four tokens are needed rather than one is that standard pre-trained models "didn't include a consistent starting token across all input samples during pre-training" (Section 3.1). Even though Llama-2 uses a <s> token, it's "applied before text chunking, resulting in a mostly random token occupying the zeroth position." Without a stable, learnable token consistently at position 0 during training, the model distributes its attention-sink behavior across multiple initial positions, learning to treat positions 0-3 collectively as the sink. This directly motivates the pre-training experiments in Section 3.3.

Positional Encoding Strategy for Non-Contiguous Caches

A subtle but critical design choice in StreamingLLM is how positional information is assigned to tokens in the cache. Because the cache contains tokens from non-contiguous original positions (e.g., positions 0-3 and positions 96-103 instead of 0-7 consecutively), simply using the tokens' original absolute positions would create gaps in the position sequence that the model was never trained to handle.

The core principle: "StreamingLLM focuses on positions within the cache rather than those in the original text. This distinction is crucial for StreamingLLM's performance" (Section 3.2). That is, tokens are assigned new, contiguous position IDs based on their index within the cache, not their original position in the full sequence.

Concrete example from the paper (Figure 4): Suppose the cache currently contains tokens at original sequence positions [0, 1, 2, 3, 6, 7, 8] (four sink tokens at positions 0-3, three recent tokens at positions 6-8, with tokens at positions 4-5 having been evicted). The StreamingLLM assigns these tokens new cache-local positions [0, 1, 2, 3, 4, 5, 6] respectively. The ninth token being generated will be at cache-local position 7. This creates a contiguous position sequence [0, ..., 7] that matches what the model saw during training, where positions were always consecutive integers starting from zero.

Why this works: The model was trained with relative position encodings (RoPE or ALiBi), which only care about the distance between tokens, not their absolute positions. By assigning contiguous cache-local positions, the relative distances between any pair of tokens in the cache are the same as they would be in a normal training sequence of the same length. The distance from the first sink token to the most recent token is $L_{\text{cache}} - 1$ , not the potentially enormous distance in the original sequence. This keeps all relative distances within the range the model was trained to handle.

Implementation for RoPE (Su et al., 2021): Rotary Position Embedding rotates the query and key vectors by an angle proportional to their position before computing attention scores. The paper states: "For encoding like RoPE, we cache the Keys of tokens prior to introducing the rotary transformation. Then, we apply position transformation to the keys in the rolling cache at each decoding phase" (Section 3.2). This means:

Key tensors are stored in the cache without rotary position encoding applied.
At each decoding step, when computing attention, the cache-local position (0, 1, 2, ..., $L_{\text{cache}}-1$ ) is used to apply the rotary transformation to both queries and cached keys.
This ensures that the attention scores reflect the relative position within the cache, not the absolute position in the streaming history.

For the sink tokens, this means they are always treated as being at positions 0, 1, 2, 3 regardless of how many tokens have been generated and evicted in between. The model consistently sees them as the "closest-to-beginning" tokens, which preserves their sink behavior.

Implementation for ALiBi (Press et al., 2022): ALiBi adds a linear bias to attention scores based on the distance between tokens. For StreamingLLM: "the contiguous linear bias is applied instead of a 'jumping' bias to the attention scores" (Section 3.2). That is, the penalty for attending to a token is proportional to its cache-local distance, not the original sequence distance. If the sink token is at cache position 0 and the current query is at cache position $L_{\text{cache}}$ , the bias is based on distance $L_{\text{cache}}$ , not the potentially much larger original distance.

The consequence of getting this wrong: If the model used original sequence positions instead of cache-local positions, the relative distances would become enormous for long-running streams. The rotary frequencies or ALiBi biases would be applied at scales far beyond what was seen during training, causing the position encoding to produce meaningless or degenerate values. This is precisely why dense attention fails when the sequence length exceeds the pre-training window — the position encodings are operating out-of-distribution. StreamingLLM's cache-local position assignment ensures the model always operates within the position ranges it was trained on.

Pre-Training with Dedicated Sink Tokens

The observation that standard models require four initial tokens as attention sinks (because they lack a consistent starting token during training) leads directly to a forward-looking proposal: train future models with a dedicated, learnable sink token at the start of every training sample. Section 3.3 describes experiments that validate this approach.

Training setup: Three 160-million-parameter language models are trained from scratch using the Pythia-160M codebase and training recipe (Biderman et al., 2023) on the deduplicated Pile dataset (Gao et al., 2020). Training is done on an 8xA6000 NVIDIA GPU server for 143,000 steps with a batch size of 256. All Pythia configurations are retained except the batch size reduction. The three variants are:

Vanilla: Standard SoftMax attention with no modifications to the input format. This is the baseline representing current practice.
Zero Sink: Uses SoftMax-off-by-One (Miller, 2023) instead of standard SoftMax. This is equivalent to prepending a virtual token with all-zero Key and Value features to every attention computation.
Learnable Sink Token: A real, learnable embedding vector is prepended to the input of every training sample as the first token. This token is trained alongside all other parameters and can learn to be an effective attention sink.

The SoftMax-off-by-One (Zero Sink) approach: This variant replaces the standard SoftMax with:

$\text{SoftMax1}(x)_i = \frac{e^{x_i}}{1 + \sum_{j=1}^{N} e^{x_j}}$

where $x_i$ is the raw attention score for token $i$ , and the denominator now has an extra constant term $1$ that is independent of any token.

What it computes: This is the standard SoftMax applied to an augmented set of $N+1$ values where one additional value is fixed at $x_0 = 0$ (since $e^0 = 1$ ). This extra term represents a "null" attention target — a virtual token that can absorb attention mass without requiring any actual token in the input. The output is still $N$ normalized weights (one per real token), but they no longer need to sum to exactly 1 — they sum to $1 - \frac{1}{1 + \sum e^{x_j}}$ , with the remaining mass going to the implicit null token.

Why this form: By providing an always-available null target in the SoftMax denominator, this approach removes the requirement that attention mass must be distributed among real tokens. The model can theoretically assign near-zero attention to all real tokens (if all $x_j$ are very negative, the sum approaches 1 for the null token and 0 for all real tokens). This should eliminate the need for attention sinks entirely.

Results of the Zero Sink approach: Table 3 shows that Zero Sink provides only a partial solution:

With $0 + 1024$ (no initial tokens preserved), Zero Sink achieves perplexity 29214, which is actually worse than Vanilla's 27.87. The model still breaks catastrophically without initial tokens.
With $1 + 1023$ (one initial token preserved), Zero Sink achieves 19.90, which is worse than Vanilla's 18.49 with the same configuration.
With $4 + 1020$ (four initial tokens preserved), Zero Sink achieves 18.01, matching Vanilla's 18.05.
For fully recovering performance, Zero Sink still requires four initial tokens (18.01 with 4+1020 vs. 18.01 with 1+1023).

Interpretation: The model trained with SoftMax-off-by-One still learns to rely on initial tokens as attention sinks, even though the mathematical constraint is theoretically removed. This suggests that attention sinks are not purely a SoftMax artifact — they also emerge as a learned computational strategy that the model finds useful. The null token provided by SoftMax1 is insufficient to capture the attention-dumping behavior that the model develops during training. The model prefers using real initial tokens as sinks.

Results of the Learnable Sink Token approach: Table 3 shows dramatically different behavior:

With $0 + 1024$ (no tokens preserved), the Learnable Sink Token model achieves perplexity 1235 — still poor but much better than Vanilla (27.87) or Zero Sink (29214). The model is more robust to missing its sink.
With $1 + 1023$ (only the sink token preserved), perplexity drops to 18.01 — essentially full recovery, matching the best perplexity achieved by Vanilla and Zero Sink with four initial tokens.
With $2 + 1022$ and $4 + 1020$ , perplexity stays at 18.01-18.05 — no further improvement, confirming that one token is sufficient.

Interpretation: The learnable sink token successfully concentrates all attention-sink behavior onto a single position. The model learns during pre-training that this dedicated token (consistently at position 0 in every training sample) is the designated place to dump excess attention. Consequently, during streaming deployment, preserving only this single sink token alongside the rolling cache is sufficient to maintain stable performance.

Why Learnable Sink Token outperforms Zero Sink: The learnable sink token is a real embedding that participates in the attention computation with actual Key and Value representations. During training, the model can optimize these representations to be effective attention absorbers. In contrast, the Zero Sink's null token has all-zero Key and Value features, which are fixed and non-learnable. The model cannot shape the null token's representations to better serve as a sink, limiting its effectiveness. The learnable sink token provides a trainable attention target that the model can adapt to its specific needs.

Convergence and standard performance impact: Figure 6 shows that the pre-training loss curves for models with and without sink tokens are essentially identical, converging at the same rate to the same final loss. Table 4 confirms that adding a sink token does not harm performance on standard NLP benchmarks: accuracy on ARC-Challenge is 19.6% vs. 18.6% for Vanilla, ARC-Easy is 45.6% vs. 45.2%, HellaSwag is 29.8% vs. 29.4%, LAMBADA is 39.9% vs. 39.6%, OpenbookQA is 16.6% vs. 16.0%, PIQA is 62.6% vs. 62.2%, and Winogrande is 50.8% vs. 50.1%. All differences are within normal variance, indicating that the sink token is a neutral addition for standard (non-streaming) use cases.

Attention visualization with sink tokens (Figure 7): The attention maps for models trained with and without sink tokens show qualitatively different behavior. The Vanilla model (left column) shows the same pattern as Llama-2-7B: local attention in early layers (Layer 0), transitioning to heavy attention on multiple initial tokens in deeper layers (Layer 10). The Sink Token model (right column) shows a concentrated attention spike on the dedicated sink token across all layers, with significantly reduced attention to other initial tokens. This visualization directly confirms the mechanism: the sink token absorbs the attention mass that would otherwise be distributed across several initial positions.

Multiple sink tokens during pre-training (Appendix I): The paper explores whether adding two sink tokens during pre-training is even better. Table 10 shows that with two sink tokens, the model requires both to be preserved for stable streaming (perplexity is 25.73 with only one preserved vs. 18.05 with both). The model learns to distribute sink behavior across both tokens rather than concentrating on one. Since this provides no performance benefit and slightly complicates deployment, the recommendation is for a single sink token.

The paper's recommendation: "We recommend training future LLMs with a sink token in all samples to optimize streaming deployment" (Section 3.3). This is a concrete, actionable suggestion for model developers that requires minimal change to training pipelines while enabling dramatically simpler streaming deployment (one preserved token instead of four).

Why the Method Is Called "StreamingLLM"

The name reflects the core operational paradigm that the method enables: the model can process an infinite stream of tokens continuously, without ever needing to reset its cache or discard recent context. This is fundamentally different from:

Batch processing where the model processes a fixed-length input, produces an output, and can be reset between inputs.
Context-extension approaches that allow processing a single longer document but still have a finite maximum length.
Hierarchical or memory-augmented approaches that maintain a compressed representation of distant history.

StreamingLLM enables a deployment mode where the model simply keeps running, token by token, with bounded memory and constant per-token cost, maintaining coherent outputs indefinitely. The attention sinks are the "anchor" that makes this possible — they keep the attention mechanism stable while the rolling window provides the recent context needed for coherent language modeling.

Relationship to SoftMax-Off-by-One and Why It's Insufficient

The paper's analysis of SoftMax-off-by-One (Equation 2, Section 3.3) reveals an important nuance about attention sinks. The standard SoftMax (Equation 1) mathematically requires attention scores to sum to 1, which forces every token to receive some attention mass. SoftMax1 relaxes this by adding a constant 1 to the denominator:

$\text{SoftMax1}(x)_i = \frac{e^{x_i}}{1 + \sum_{j=1}^{N} e^{x_j}}$

where the additional constant term $1$ in the denominator provides an "escape valve" — attention mass can flow to the implicit null token (represented by the constant $e^0 = 1$ ) rather than being forced onto real tokens.

What this was expected to solve: If attention sinks were purely a mathematical artifact of the SoftMax constraint, then removing that constraint with SoftMax1 should eliminate the need for sink tokens entirely. The model trained with SoftMax1 should be able to operate with a pure window attention (no preserved initial tokens) because it could always route excess attention to the implicit null token.

Why it doesn't fully work: The experimental results (Table 3, Zero Sink row) show that SoftMax1 improves robustness slightly but does not eliminate the need for attention sinks. The model still breaks with $0 + 1024$ (perplexity 29214) and still benefits substantially from preserving initial tokens. This indicates that attention sinks are not only a mathematical artifact — they are also a learned computational strategy. The model discovers during training that concentrating attention on a fixed set of tokens (the initial ones) is useful for its internal computations, independent of the SoftMax constraint. The initial tokens may serve as a kind of "bias" or "default state" that the model uses as a reference point for processing other tokens.

The learnable sink token succeeds where SoftMax1 fails because it leans into this learned strategy: rather than trying to eliminate the sink behavior, it provides a dedicated, optimized token to serve as the sink. This is a more cooperative approach — it works with the model's learned attention patterns rather than trying to mathematically prevent them.

Applicability Constraints and Positioning

The paper is explicit about what StreamingLLM does and does not do, which is important for understanding its technical scope:

Requirements for applicability:

The model must use relative position encoding — specifically RoPE or ALiBi, which are the two dominant approaches. The paper does not test absolute position encodings because those would likely fail when cache-local positions differ from training positions.
The model must be autoregressive (decoder-only) so that initial tokens are visible to all subsequent tokens. While the paper demonstrates attention sinks in encoder models like BERT (Appendix H), the streaming deployment scenario is specifically for autoregressive generation.
No fine-tuning is required — the method works on frozen pre-trained models. This is a key practical advantage.

What StreamingLLM does NOT do:

It does not extend the context window. The model can only attend to $L_{\text{cache}}$ tokens at any step. The effective context is the rolling window plus sinks, which is the same size as the original training window (or smaller, depending on configuration). The paper is explicit: "StreamingLLM efficiently generates coherent text from tokens within the KV cache without extending the LLMs' context length" (Section 1).
It does not improve long-term memory. The model has no access to tokens evicted from the rolling cache. Section C (Appendix C) demonstrates this: accuracy on StreamEval drops to zero when the query-answer distance exceeds the cache size (Table 7). The model can only answer questions about information still in its rolling cache.
It is not suitable for tasks requiring long-range dependencies like long-document QA or summarization of entire books. The paper states this limitation clearly in Appendix A: "StreamingLLM is not suitable for tasks that demand long-term memory and extensive data dependency."

What StreamingLLM IS good for:

Streaming applications where continuous operation matters more than long-range recall.
Multi-round dialogues where recent conversation history (within the cache size) is sufficient for coherent responses.
Daily assistant applications that need to run indefinitely without cache refreshes.
Any deployment where the operational requirement is persistence, not extended context.

The paper demonstrates this positioning through the StreamEval benchmark (Section 4.3), which is specifically designed to test a model's ability to answer queries about recent information in a continuous stream — exactly the use case StreamingLLM targets.

4. Key Insights and Innovations

Innovation 1: Identifying Attention Sinks as a Structural SoftMax Constraint, Not a Trainable Behavior

Before this paper, the dominant assumption in the field was that the failure of window attention — where models collapse when the earliest tokens are evicted — was fundamentally about semantic context loss. The intuition was straightforward: initial tokens contain important information (task instructions, conversation framing, document preamble), and discarding them naturally degrades output quality. This assumption implicitly motivated approaches like text truncation heuristics that preserve beginnings and endings of documents (as the LongBench baseline does with its 1750+1750 token split, shown in Table 8), and it suggested that the solution was better context management or longer training windows.

This paper fundamentally reframes the problem. The key conceptual move — and the one that makes everything else possible — is recognizing that the collapse of window attention is not about what the initial tokens mean, but about where they are. The attention sink phenomenon is diagnosed as a structural consequence of the SoftMax normalization, not a semantic dependency. Because SoftMax forces attention scores to sum to 1 across all contextual tokens, the model must allocate some attention mass to every token in the context, even when no token is relevant to the current prediction. The model learns to "dump" this mandatory attention onto tokens that are always available — and in autoregressive architectures, initial tokens are visible to every subsequent position, making them the natural sink.

The linebreak substitution experiment (Table 1) is the decisive diagnostic that proves this reframing. Replacing semantically meaningful initial tokens with meaningless newline characters ("\n") and still recovering performance (perplexity 5.60 vs. 5.40 for original initial tokens) demonstrates that absolute position, not semantic content, is what the model has learned to depend on. This is not a subtle effect — it's a categorical demonstration that the model's reliance on initial tokens is positional rather than informational.

What makes this innovation fundamental rather than incremental is that it reclassifies the problem from a content-engineering challenge to an architectural constraint. The field had been approaching window attention failure as something to fix with better data curation, smarter truncation, or longer training sequences. The attention sink diagnosis reveals that none of those approaches would have worked, because the SoftMax function itself creates an inescapable requirement for some tokens to serve as attention absorbers. You cannot train this behavior away — you can only work with it.

This connects to a broader pattern the paper documents: attention sinks appear not only in autoregressive LMs but also in encoder-only Transformers (BERT's attention to the [SEP] token in Appendix H, Figure 14) and in Vision Transformers (Darcet et al., 2023's "registers"). This cross-architecture prevalence suggests that attention sinks are a universal feature of SoftMax-based self-attention, not an artifact of specific training data or model architectures. The paper's contribution is naming and characterizing this universal phenomenon in the streaming context.

The conceptual payoff is that this reframing enables a solution strategy that would be unimaginable under the old assumption. If initial tokens were important because of their semantic content, preserving them would require content-specific logic — which initial tokens, from which conversation phase, with what preprocessing? But because they're important for their position, the solution is trivially general: just keep the first N tokens, whatever they happen to be. StreamingLLM's elegance flows directly from the correctness of this diagnostic.

Innovation 2: Demonstrating That Bounded-Memory, Infinite-Length Operation Is Achievable Without Fine-Tuning

A significant unspoken assumption in the LLM deployment literature is that any substantial change to a model's operating regime requires retraining or at least fine-tuning. If you want to handle longer sequences, you extend the context window through position interpolation and fine-tuning (Chen et al., 2023; Peng et al., 2023). If you want to improve efficiency, you train sparse attention patterns from scratch (Child et al., 2019; Beltagy et al., 2020). The idea that a model trained with a finite attention window could be deployed — without any parameter updates — to operate on sequences orders of magnitude longer than its training length would have seemed implausible to most practitioners.

StreamingLLM demonstrates exactly this: models trained with 4K-token windows (Llama-2) or 2K-token windows (Falcon, MPT, Pythia) achieve stable language modeling perplexity on sequences of 4 million tokens and beyond (Figure 5), with no gradient updates, no architectural modifications, and no additional training data. The only change is a modified KV cache eviction policy — preserving the first four tokens' KV states alongside a sliding window of recent tokens.

What makes this finding significant beyond its practical utility is what it reveals about the relationship between training and deployment in Transformer models. The fact that a model can generalize to sequence lengths three orders of magnitude beyond its training length — without any form of adaptation — tells us that the length extrapolation failure documented by Press et al. (2022) and Chen et al. (2023) is not actually a failure of the model's learned computations. The model's attention patterns, its feedforward transformations, its entire internal processing pipeline — all of these continue to work correctly at arbitrary lengths. The failure is specifically and narrowly in the position encoding regime and the attention score distribution. By keeping all position encodings within the trained range (via cache-local position assignment, described in Section 3.2) and keeping the attention score distribution stable (via preserved sink tokens), the model operates comfortably in a regime that is mathematically indistinguishable from its training distribution — even while the total sequence length grows without bound.

This is a profound architectural insight: autoregressive Transformers with relative position encodings are architecturally capable of infinite-length operation by design, and their observed failure at long lengths is entirely an artifact of how position information and attention sinks are managed, not of any fundamental computational limitation. The paper does not change the model — it changes the interface between the model and the token stream, and the model turns out to be fully compatible with the new interface.

The significance of this finding is amplified by its contrast with the primary competing baseline that achieves comparable quality: sliding window with re-computation. That method achieves stable long-sequence performance (Figure 3) but at a cost that makes it "impractical for real-world streaming applications" (Section 1) — 22.2× slower than StreamingLLM at equivalent cache sizes (Figure 10). This isn't a small efficiency tradeoff; it's the difference between a method that can be deployed in production streaming contexts and one that cannot. The fact that StreamingLLM achieves this without fine-tuning means it can be immediately applied to any existing relative-position-encoded autoregressive LLM, which is a practical advantage that fine-tuning-requiring approaches cannot match.

The paper further validates the robustness of this finding across model families that use fundamentally different position encoding schemes: Llama-2, Falcon, and Pythia use RoPE (Su et al., 2021) with its rotary frequency mechanism, while MPT uses ALiBi (Press et al., 2022) with its distance-based additive bias. That StreamingLLM works with both (Figure 5) demonstrates that the principle — keep position encodings within trained ranges and preserve attention sinks — is agnostic to the specific position encoding implementation, as long as it's relative rather than absolute.

Innovation 3: Proposing and Validating Sink Token Pre-Training as a Simple, Forward-Looking Design Change

The paper's analysis of why existing models need multiple initial tokens as attention sinks — "these models didn't include a consistent starting token across all input samples during pre-training" (Section 3.1) — leads directly to a prescriptive recommendation: future language models should be pre-trained with a dedicated, learnable sink token at the start of every training sample. This is not a method for improving existing models (StreamingLLM already works on them with four initial tokens preserved), but rather a design principle for the next generation of models that would make streaming deployment even simpler.

The innovation here is the recognition that attention sinks are not something to be worked around — they are something to be designed for. Rather than treating the multi-initial-token sink behavior as an unfortunate quirk of existing training procedures, the paper reframes it as a failure of the training data format to provide a consistent, learnable sink. The remedy is trivially simple in concept (prepend one token to every training example) and validated to be effective (Table 3) and harmless (Table 4, Figure 6).

What makes this an innovation rather than an obvious extension is the comparison with the mathematical alternative. The SoftMax-off-by-One approach (Zero Sink) attempts to eliminate the need for attention sinks entirely by modifying the SoftMax function to include an implicit null token. This is mathematically elegant — it removes the constraint that attention scores must sum to 1, theoretically allowing the model to assign near-zero attention to all tokens when appropriate. If this had worked, it would have been a cleaner solution than training a dedicated sink token, because it would require no changes to the input format and would eliminate the sink concept entirely.

The paper's finding that Zero Sink does not work — that models trained with SoftMax1 still rely on initial tokens as attention sinks and still fail without them (Table 3, Zero Sink row: perplexity 29214 with $0+1024$ ) — is a significant negative result with conceptual implications. It demonstrates that attention sinks are not purely a mathematical artifact that can be designed away by changing the normalization function. They are also a learned computational strategy — the model discovers during training that having a fixed set of tokens to dump attention onto is useful for its internal processing, independent of whether the SoftMax mathematically requires it. The learnable sink token succeeds precisely because it works with this learned strategy: it provides a dedicated, optimized token that the model can shape through training to be the perfect attention absorber.

This has implications beyond streaming deployment. It suggests that attention in Transformers is not purely content-driven — there is a structural component that emerges from the interaction of training dynamics with architectural constraints. The model learns positional attention biases that are adaptive (they serve the model's computational needs) but not semantic (they don't depend on token meaning). This blurs the line between what is "learned" and what is "architectural" in ways that the field has not fully explored.

The practical recommendation — include a sink token in all pre-training samples — is notable for its minimality. It requires:

One additional token embedding (negligible parameter increase).
Prepending this token to every training example (trivial data preprocessing change).
No change to the training objective, architecture, or hyperparameters.

And the payoff is substantial: streaming deployment that requires preserving only one token (the sink) rather than four tokens (the ad-hoc initial tokens that existing models learned to use). In a memory-constrained deployment where every preserved token counts, reducing the sink overhead from four to one is a meaningful improvement. The paper's experiments with two sink tokens (Appendix I, Tables 9 and 10) further validate that one is the sweet spot — additional sink tokens provide no benefit but increase the deployment overhead.

This innovation is incremental in its implementation (it's a data formatting change) but fundamental in its conceptual framing: it treats the attention sink not as a bug to be fixed but as a feature to be optimized.

Innovation 4: Introducing a New Deployment Paradigm That Decouples Training Window Size from Generation Length

The paper's deepest conceptual contribution might be the decoupling of two quantities that the field had implicitly assumed were tightly coupled: the attention window size used during training, and the maximum sequence length a model can handle during deployment. Before StreamingLLM, the relationship was treated as essentially one-to-one: a model trained with a 4K-token window could generate roughly 4K tokens of coherent text, and extending generation beyond that required either extending the training window (through position interpolation and fine-tuning) or accepting severe quality degradation.

StreamingLLM demonstrates that this coupling is an artifact of implementation, not an architectural necessity. By managing the KV cache to keep the model's attention computation within its trained positional range — through the combination of sink token preservation and cache-local position assignment — the model can generate text indefinitely while never exceeding the position ranges or attention score distributions it experienced during training. The model thinks it's always operating on a sequence of length $L_{\text{cache}}$ with normal attention patterns; the fact that tokens at positions $L_{\text{cache}}+1, L_{\text{cache}}+2, \ldots$ are being generated and then partially evicted is invisible to the model's computations.

This paradigm shift has practical implications for how model developers think about deployment. The training window size becomes a runtime configuration parameter (how much recent context to preserve) rather than a hard limit on generation length. A model trained with a 4K window can be deployed with a 2K rolling cache if memory is constrained, or an 8K rolling cache if recent context is important — in both cases generating indefinitely. The StreamEval results with context-extended models (Section 4.3, Figure 9) demonstrate that StreamingLLM works on top of extended context windows, showing that cache size and training window size are independently configurable.

This conceptual reframing also clarifies what the field's various research directions actually address. The paper's taxonomy in Section 2 — length extrapolation, context window extension, and improving long-text utilization — can now be understood as addressing different aspects of a single underlying problem. StreamingLLM addresses the operational stability aspect: can the model keep running? It does not address the capacity aspect: how many tokens of information can the model actually use? This separation — between "does the model work?" and "how much can the model remember?" — is a useful conceptual tool that the field had not articulated clearly before.

The paper is admirably explicit about this boundary. Appendix A states: "StreamingLLM is not suitable for tasks that demand long-term memory and extensive data dependency." Table 7 (Appendix C) quantifies this: accuracy on StreamEval is high when the query-answer distance is within the cache, and drops to zero when it exceeds the cache capacity. This is not a failure of StreamingLLM — it's a precise characterization of what the method does (enable infinite streaming) and doesn't do (extend memory). By making this separation explicit, the paper provides a framework for thinking about which applications need streaming (daily conversation, real-time transcription) and which need long-context processing (book summarization, long-document QA).

This innovation is conceptual rather than technical — it's a new way of thinking about what it means to "handle long sequences" — but conceptual innovations that reorient a field's thinking can be as influential as technical ones.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary evaluations use the PG-19 test set (Rae et al., 2020), which contains 100 long-form books. For language modeling perplexity experiments, the books are concatenated into continuous text streams, enabling evaluation on sequences far longer than individual documents. The paper also uses ARC-Easy and ARC-Challenge (Clark et al., 2018) for streaming question-answering, and introduces StreamEval, a custom benchmark inspired by LongEval (Li et al., 2023), for testing retrieval of information at varying distances within a continuous stream. For the pre-training experiments, models are evaluated on seven standard NLP benchmarks: ARC-Challenge/Easy, HellaSwag (Zellers et al., 2019), LAMBADA (Paperno et al., 2016), OpenbookQA (Mihaylov et al., 2018), PIQA (Bisk et al., 2020), and Winogrande (Sakaguchi et al., 2019).
Base model(s). The paper evaluates four major model families spanning different architectures and scales: Llama-2 (7B, 13B, 70B) (Touvron et al., 2023b), Falcon (7B, 40B) (Almazrouei et al., 2023), Pythia (2.8B, 6.9B, 12B) (Biderman et al., 2023), and MPT (7B, 30B) (Team, 2023). This selection deliberately covers the two dominant relative position encoding schemes: Llama-2, Falcon, and Pythia use RoPE (Su et al., 2021), while MPT uses ALiBi (Press et al., 2022). The diversity ensures that findings are not specific to a single position encoding method or model architecture. For instruction-tuned experiments, Llama-2-Chat variants (7B, 13B, 70B) are used. For context-extended model experiments, LongChat-7b-v1.5-32k (Li et al., 2023) and Llama-2-7B-32K-Instruct (Together, 2023) are employed. For pre-training experiments, 160-million-parameter models are trained from scratch using the Pythia-160M codebase.
Metrics. The primary metric is language modeling perplexity (PPL), computed as the exponential of the negative log-likelihood of the ground-truth next token given the preceding context. Lower perplexity indicates better language modeling performance. For streaming question-answering on ARC datasets, exact match accuracy is used — the model's generated answer at each answer position must exactly match the ground-truth answer. For StreamEval, retrieval accuracy measures whether the model correctly retrieves a specific piece of information (a REGISTER_CONTENT value) from a specified position in the stream. For the pre-trained model evaluation on standard benchmarks, zero-shot accuracy is reported.
Baselines. The paper compares against three primary baselines: (1) Dense attention — standard Transformer attention over all previous tokens, which stores the full KV cache and has $O(T^2)$ complexity (Figure 1a). (2) Window attention (Beltagy et al., 2020) — maintains only the most recent $L$ tokens' KV states in a sliding window, achieving $O(TL)$ complexity (Figure 1b). This is the efficiency-motivated baseline that StreamingLLM aims to improve upon. (3) Sliding window with re-computation — at each decoding step, rebuilds the KV states for the $L$ most recent tokens from scratch using dense attention, achieving the best quality but at $O(TL^2)$ complexity (Figure 1c). This serves as the oracle quality baseline for language modeling perplexity. For the pre-training experiments, the baseline is a Vanilla model trained with standard SoftMax attention and no sink token. For streaming QA on ARC, the one-shot sample-by-sample baseline (processing each question-answer pair independently rather than in a continuous stream) provides an upper bound on what streaming methods can achieve.
Generation budget / compute accounting. Compute is measured in terms of per-token decoding latency (milliseconds) and memory usage (GB), benchmarked on a single NVIDIA A6000 GPU using the Huggingface Transformers library (Wolf et al., 2020). The paper compares these metrics across different cache sizes (256, 512, 1024, 2048, 4096 tokens) for both StreamingLLM and the sliding window with re-computation baseline. This captures the tradeoff between context capacity and computational cost. For language modeling experiments, the "cache config" notation $x+y$ denotes preserving $x$ initial tokens (attention sinks) plus $y$ most recent tokens, with $x+y$ equaling the total cache size. For Llama-2 models, the default cache size is 2048; for Falcon, Pythia, and MPT, it is 1024 — chosen as half of the pre-training window size to "enhance visualization clarity" (Section 4.1).
Cross-validation / statistical protocol. No formal cross-validation or statistical significance testing is reported. Results are computed over the full test sets: 100 books from PG-19 for perplexity (concatenated for long-sequence evaluation), the standard ARC test sets for accuracy, and 100 samples with 100 queries each for StreamEval. For attention visualization, maps are averaged over 256 sentences with consistent lengths (16 or 128 tokens). For the quantitative analysis of attention scores on long inputs (Appendix F), 256 sequences of 4096 tokens each are used, with error bars representing standard deviation across heads within each layer. The pre-training experiments use a single training run per variant (143,000 steps) due to computational constraints, and report results on a single sample from PG-19 for streaming perplexity.

Main Quantitative Results

Language Modeling Perplexity on Long Texts (Section 4.1)

Figure 3 presents the core perplexity results across four model families on texts spanning 20K tokens, comparing all four methods (dense attention, window attention, sliding window with re-computation, and StreamingLLM). The headline finding is that StreamingLLM matches the oracle sliding window with re-computation baseline in perplexity, while window attention catastrophically fails and dense attention degrades when the input exceeds the pre-training window.

The failure modes are stark and consistent across all models. Dense attention exhibits normal perplexity until the input length surpasses the pre-training attention window size (approximately 4K tokens for Llama-2, 2K for others), at which point perplexity spikes sharply — reaching over 5000 for Llama-2-13B (Table 1) and similar catastrophic levels for other models. Window attention (with cache sizes of 1024 or 2048 depending on the model) maintains normal perplexity until the sequence length exceeds the cache size, at which point the initial tokens are evicted and perplexity immediately skyrockets — from approximately 5.4 to over 5158 for Llama-2-13B when going from 1024 to 1025 tokens (Table 1). StreamingLLM, in contrast, maintains stable perplexity throughout the entire 20K-token sequence, with its curve essentially overlapping that of the sliding window with re-computation baseline. This holds for Llama-2-7B, Llama-2-13B, Falcon-7B, MPT-7B, and Pythia-12B.

Figure 5 extends this to the extreme scale, showing perplexity over 4 million tokens across eight models spanning three orders of magnitude in parameter count: Llama-2-7B, Llama-2-13B, Llama-2-70B, Falcon-7B, Falcon-40B, Pythia-2.8B, Pythia-6.9B, Pythia-12B, and MPT-7B, MPT-30B. The key result: perplexity remains stable throughout the entire 4 million token sequence for every model tested. The fluctuations visible in the plots are attributed to book transitions in the concatenated PG-19 test set (the 100 books are concatenated, so perplexity naturally varies when transitioning between books with different domains and writing styles). This stability is remarkable considering these models were trained with attention windows of 2048 or 4096 tokens — StreamingLLM enables generation three orders of magnitude beyond the training length with no degradation.

Table 1 provides the precise numbers that ground the visual patterns. On the first book (65K tokens) of the PG-19 test set using Llama-2-13B: window attention with cache size 1024 achieves perplexity 5158.07, while reintroducing 4 initial tokens alongside 1020 recent tokens drops perplexity to 5.40. The near-1000× reduction in perplexity from preserving just four tokens is the quantitative foundation for the entire StreamingLLM approach. The linebreak substitution experiment in the same table shows $4"\text{\textbackslash n}" + 1020$ achieves 5.60, confirming that the effect is positional rather than semantic.

Table 2 ablates the number of preserved initial tokens across four model families. The key threshold finding: four initial tokens are sufficient to achieve near-optimal perplexity, while one or two are insufficient. For Llama-2-7B with a 4096-total cache: $0+4096$ gives 3359.95 (catastrophic), $1+4095$ gives 11.88 (substantially improved but not fully recovered), $2+4094$ gives 10.51, $4+4092$ gives 9.59 (essentially optimal), and $8+4088$ gives 9.54 (marginal further improvement). Similar patterns hold for Falcon-7B (17.90 → 12.12 → 12.12 → 12.12), MPT-7B (460.29 → 14.99 → 15.00 → 14.99), and Pythia-12B (21.62 → 11.95 → 12.09 → 12.09). The diminishing returns after four tokens are consistent across models, justifying the default choice of four attention sinks.

Streaming Question Answering (Section 4.3)

Table 5 reports accuracy on ARC-Easy and ARC-Challenge for instruction-tuned Llama-2-Chat models in a streaming question-answering setup. All question-answer pairs from the ARC datasets are concatenated into a continuous stream, and the model must answer each question as it appears in sequence. The headline: StreamingLLM matches the one-shot baseline accuracy, while window attention collapses to near-random performance.

For Llama-2-7B-Chat: on ARC-Easy, one-shot achieves 71.25%, StreamingLLM achieves 71.34%, and window attention achieves 3.58% (approaching chance-level for a multi-choice task). On ARC-Challenge: one-shot achieves 53.16%, StreamingLLM achieves 55.03% (slightly higher, likely due to variance), and window attention achieves 1.39%. Dense attention results in Out-of-Memory (OOM) errors, making it unusable for streaming deployment. The pattern holds across model scales: Llama-2-13B-Chat achieves 78.16% / 80.89% / 0.25% on ARC-Easy and 63.31% / 65.61% / 0.34% on ARC-Challenge for one-shot, StreamingLLM, and window attention respectively. Llama-2-70B-Chat achieves 91.29% / 91.37% / 0.12% on ARC-Easy and 78.50% / 80.20% / 0.32% on ARC-Challenge. StreamingLLM matches or slightly exceeds the one-shot baseline across all configurations, while window attention produces effectively random outputs (accuracies of 0.12–3.58%).

A crucial detail: the cache sizes are set to 1024 for all models in this experiment. Since the concatenated ARC questions can exceed this length, window attention breaks once the sequence surpasses 1024 tokens — which is exactly where StreamingLLM's sink preservation mechanism maintains stability.

Figure 9 and the StreamEval benchmark (Section 4.3, described in Figure 8) test a more targeted capability: retrieving specific pieces of information from a continuous stream when queried at regular intervals. In StreamEval, each line of the input contains a REGISTER_CONTENT value associated with a line number. Every 10 lines, the model is queried about the content from 20 lines prior, testing whether recent information remains accessible. The headline from Figure 9: StreamingLLM maintains reasonable accuracy as input lengths approach 120K tokens, while both dense and window attention fail at their respective break points. The exact accuracy values are not explicitly reported in the text for Figure 9, but the visual shows StreamingLLM sustaining performance throughout, dense attention failing at the pre-training text length, and window attention failing at the KV cache size. The paper also demonstrates that StreamingLLM can be combined with context-extended models (LongChat-7b-v1.5-32k and Llama-2-7B-32K-Instruct), with the extended context window "broadening the maximum cache size of streaming LLMs, enabling the capture of broader local information" (Section 4.3).

Efficiency Results (Section 4.5)

Figure 10 presents the latency and memory benchmarks comparing StreamingLLM against the sliding window with re-computation baseline across different cache sizes for Llama-2-7B and Llama-2-13B on a single NVIDIA A6000 GPU. The headline: StreamingLLM achieves up to 22.2× per-token speedup over the re-computation baseline while maintaining comparable memory usage.

For Llama-2-7B, per-token decoding latency:

Cache size 256: StreamingLLM 65ms vs. re-computation 1411ms (21.7× speedup)
Cache size 512: StreamingLLM 45ms vs. re-computation 523ms (11.6× speedup)
Cache size 1024: StreamingLLM 35ms vs. re-computation 223ms (6.4× speedup)
Cache size 2048: StreamingLLM 31ms vs. re-computation 103ms (3.3× speedup)
Cache size 4096: StreamingLLM 31ms vs. re-computation 63ms (2.0× speedup)

For Llama-2-13B:

Cache size 256: StreamingLLM 106ms vs. re-computation 2355ms (22.2× speedup)
Cache size 512: StreamingLLM 75ms vs. re-computation 860ms (11.5× speedup)
Cache size 1024: StreamingLLM 60ms vs. re-computation 361ms (6.0× speedup)
Cache size 2048: StreamingLLM 52ms vs. re-computation 169ms (3.3× speedup)
Cache size 4096: StreamingLLM 48ms vs. re-computation 99ms (2.1× speedup)

The speedup is largest at small cache sizes because the re-computation baseline has quadratic complexity within its window — $O(L^2)$ per step for a window of size $L$ — while StreamingLLM has linear complexity $O(L)$ for the same window. As cache size increases, the quadratic term dominates, making the baseline increasingly expensive while StreamingLLM scales gracefully. The memory footprint is essentially identical between the two methods (within 1-2 GB for all configurations), since both store the same number of KV states — the difference is purely in how those states are computed (incrementally vs. recomputed from scratch).

A key observation from the latency curves: StreamingLLM's per-token latency exhibits only linear growth with cache size (as expected for $O(L)$ attention computation), while the re-computation baseline shows quadratic growth. This confirms that the asymptotic complexity analysis translates directly to practical speedups.

Pre-Training with Sink Tokens (Section 4.2)

Table 3 presents the streaming perplexity of 160M-parameter models trained from scratch with and without sink tokens, evaluated on the first sample of the PG-19 test set. The headline: a model trained with a learnable sink token achieves stable streaming perplexity with only the sink token preserved, while the vanilla model requires four initial tokens.

For the Vanilla model: $0+1024$ achieves 27.87, $1+1023$ achieves 18.49 (substantially worse than full recovery), $2+1022$ achieves 18.05, $4+1020$ achieves 18.05 (full recovery). The model needs multiple initial tokens to stabilize.

For the Zero Sink model (SoftMax-off-by-One): $0+1024$ achieves 29214 (worse than Vanilla — the model still catastrophically fails without any initial tokens), $1+1023$ achieves 19.90, $2+1022$ achieves 18.27, $4+1020$ achieves 18.01. This model still relies on initial tokens as sinks despite the mathematical relaxation of the SoftMax constraint.

For the Learnable Sink Token model: $0+1024$ achieves 1235 (poor but much better than 27.87 or 29214 — some robustness even without the sink), $1+1023$ achieves 18.01 (full recovery with a single token), $2+1022$ achieves 18.01, $4+1020$ achieves 18.02. The model concentrates all sink behavior onto the single dedicated token.

The key comparison is $1+1023$ across variants: Vanilla achieves 18.49 (partial recovery — not sufficient), Zero Sink achieves 19.90 (worse than Vanilla), and Learnable Sink Token achieves 18.01 (full recovery). The learnable sink token reduces the required preserved tokens from four to one while matching or slightly exceeding the best perplexity of the other variants.

Figure 6 demonstrates that the pre-training loss curves for models trained with and without sink tokens are essentially identical, converging at the same rate. This is critical because it shows the sink token is not a tradeoff between streaming capability and standard performance — it's a free addition.

Table 4 extends this to downstream task performance, reporting zero-shot accuracy across seven NLP benchmarks. The Vanilla model and the +Sink Token model show near-identical performance: ARC-Challenge (18.6% vs. 19.6%), ARC-Easy (45.2% vs. 45.6%), HellaSwag (29.4% vs. 29.8%), LAMBADA (39.6% vs. 39.9%), OpenbookQA (16.0% vs. 16.6%), PIQA (62.2% vs. 62.6%), and Winogrande (50.1% vs. 50.8%). All differences are within 0.4 percentage points for most tasks, confirming that the sink token does not harm the model's general language understanding capabilities.

Figure 7 visualizes the attention maps of 160M models trained without (left) and with (right) a sink token, showing the same layers and heads. Observations: (1) Without a sink token, the model shows local attention in lower layers (Layer 0) and increased attention to multiple initial tokens in deeper layers (Layer 10), replicating the pattern seen in Llama-2-7B (Figure 2). (2) With a sink token, there is clear, concentrated attention directed at the sink across all layers, effectively collecting redundant attention. (3) With the presence of the sink token, less attention is given to other initial tokens, supporting the claim that the sink token successfully centralizes attention-sink behavior.

Long-Range Benchmark Evaluation (Appendix D)

Table 8 evaluates StreamingLLM on LongBench (Bai et al., 2023) using Llama-2-7B-chat (max context length 4K), comparing against the default truncation baseline that preserves 1750 initial and 1750 final tokens. This experiment tests whether StreamingLLM can handle tasks requiring long-range context, or whether its cache limitation impairs performance.

The findings are mixed but informative. StreamingLLM with $4+3496$ (4 sink tokens, 3496 recent tokens) underperforms the truncation baseline across all tasks: NarrativeQA (11.6 vs. 18.7), Qasper (16.9 vs. 19.2), HotpotQA (21.6 vs. 25.4), 2WikiMQA (28.2 vs. 32.8), GovReport (23.9 vs. 27.3), MultiNews (25.5 vs. 25.8). This is expected — the $4+3496$ configuration loses the initial input prompt information that is often crucial for understanding the task instructions and document preamble in these long-document benchmarks. However, when StreamingLLM is configured with $1750+1750$ (1750 sink tokens and 1750 recent tokens, matching the truncation baseline's allocation), performance is essentially restored: NarrativeQA (18.2 vs. 18.7), Qasper (19.7 vs. 19.2), HotpotQA (24.9 vs. 25.4), 2WikiMQA (32.0 vs. 32.8), GovReport (26.3 vs. 27.3), MultiNews (25.9 vs. 25.8). This demonstrates that StreamingLLM's effectiveness is contingent on the information within its cache — when the cache contains the same information as the truncation baseline, performance is comparable. The paper correctly interprets this as confirming that StreamingLLM does not extend context length but rather enables stable operation within whatever context is cached.

Ablation Studies and Robustness Checks

Number of initial tokens as attention sinks (Table 2, Section 4.4): The experiment tests cache configurations $0+y$ , $1+y$ , $2+y$ , $4+y$ , and $8+y$ across four model families, where $y$ is adjusted to keep total cache size constant. The finding: one or two initial tokens are insufficient to fully recover perplexity, while four initial tokens provide near-complete recovery with diminishing returns beyond that. For Llama-2-7B, the perplexity progression is 3359.95 → 11.88 → 10.51 → 9.59 → 9.54. The jump from 2 to 4 tokens (10.51 → 9.59) is substantial, while 4 to 8 (9.59 → 9.54) is negligible. This threshold of four is consistent across Falcon-7B, MPT-7B, Pythia-12B, and Llama-2-7B. A non-obvious finding: Falcon-7B and MPT-7B show almost no improvement beyond a single sink token (12.12 with $1+2047$ , $2+2046$ , $4+2044$ , and $8+2040$ are all 12.12 or 14.99–15.00 for MPT), while Pythia-12B and Llama-2-7B show a clear progression from one to four. This suggests that different model families distribute their attention-sink behavior differently across initial positions, possibly due to variations in pre-training data formatting.

Cache size (Table 6, Section 4.4): The experiment varies the cache size while keeping the number of sink tokens fixed at four, measuring perplexity on 400K tokens of the concatenated PG-19 test set. The counterintuitive finding: increasing cache size does not consistently lower perplexity. For Falcon-7B, perplexity decreases from 13.61 at $4+252$ to 12.34 at $4+1020$ but then increases to 12.84 at $4+2044$ . For MPT-7B, perplexity actually increases monotonically with cache size: 14.12 → 14.25 → 14.33 → 14.99. For Pythia-12B, perplexity improves from 13.17 to 12.08 at $4+1020$ and plateaus at 12.09 for $4+2044$ . For Llama-2-7B, perplexity decreases from 9.73 to 9.08 at $4+2044$ but then increases to 9.59 at $4+4092$ . The paper interprets this as evidence that "these models might not maximize the utility of the entire context they receive" (Section 4.4) — a limitation of the current generation of LLMs in effectively attending to and using all available context, which is orthogonal to StreamingLLM but constrains the benefits of larger cache sizes.

Semantic content vs. absolute position of sink tokens (Table 1, Section 3.1): The linebreak substitution experiment: replacing the original initial four tokens with "\n" characters and measuring perplexity. The finding: semantically-empty tokens at the initial positions function as effective attention sinks, nearly matching the original tokens (perplexity 5.60 vs. 5.40). This is a crucial ablation because it rules out the alternative hypothesis that initial tokens are important for their semantic role (e.g., task framing, document context). It directly supports the paper's claim that attention sinks are a positional phenomenon driven by the SoftMax constraint rather than content-driven attention.

Pre-training without sink token vs. with Zero Sink vs. with Learnable Sink Token (Table 3, Section 4.2): The three-way comparison between Vanilla, Zero Sink (SoftMax-off-by-One), and Learnable Sink Token models. The key negative result: Zero Sink does not eliminate the need for attention sinks. Despite mathematically removing the constraint that attention scores must sum to one (by adding a constant to the denominator), the Zero Sink model still fails catastrophically without initial tokens (perplexity 29214 with $0+1024$ ) and still benefits from preserving them (19.90 with $1+1023$ , 18.01 with $4+1020$ ). This demonstrates that attention sinks are not purely a SoftMax artifact — they are a learned computational strategy that the model adopts during training. The Learnable Sink Token succeeds because it works with this learned strategy (providing an optimized sink) rather than trying to mathematically eliminate it. This negative result is conceptually important because it closes off what would have been an even cleaner solution (replacing SoftMax rather than modifying the input format).

Number of sink tokens during pre-training (Appendix I, Tables 9 and 10): The experiment compares training with zero, one, or two dedicated sink tokens. The finding: adding a second sink token provides no benefit. Table 9 shows that two sink tokens do not improve zero-shot accuracy across the seven NLP benchmarks (e.g., ARC-Challenge: 18.6% Vanilla, 19.6% with one sink, 18.7% with two sinks; LAMBADA: 39.6% Vanilla, 39.9% with one sink, 37.5% with two sinks — a slight degradation). Table 10 shows that with two sink tokens, the model requires both to be preserved for stable streaming (perplexity 25.73 with only one preserved vs. 18.05 with both preserved, compared to 18.01 for the single-sink model with one token). The model learns to distribute sink behavior across both tokens rather than concentrating on one, which increases deployment overhead without improving performance. This contrasts with findings in Vision Transformers (Darcet et al., 2023), where multiple "registers" were found beneficial, highlighting that the optimal number of sink tokens may be architecture-dependent.

StreamingLLM with context-extended models (Section 4.3, Figure 9): The experiment applies StreamingLLM to LongChat-7b-v1.5-32k and Llama-2-7B-32K-Instruct, demonstrating that StreamingLLM is complementary to context extension methods. These models have extended training windows (32K tokens) but still benefit from StreamingLLM's bounded-memory streaming capability. The paper frames this as "broadening the maximum cache size of streaming LLMs, enabling the capture of broader local information" — essentially, you can have a larger rolling cache with a context-extended model, but still need the sink mechanism for unbounded streaming.

Query-answer distance in StreamEval (Appendix C, Table 7): The experiment evaluates Llama-2-7B-32K-Instruct on StreamEval with increasing distances between the query and the answer, under different cache configurations ( $4+2044$ , $4+4092$ , $4+8188$ , $4+16380$ ). The finding: accuracy is high when the query-answer distance is within the cache size and drops to zero when it exceeds the cache capacity. For example, with $4+4092$ cache: at line distance 20 (460 tokens), accuracy is 84.60%; at line distance 100 (2300 tokens), accuracy is 61.60%; at line distance 200 (4600 tokens, exceeding the cache), accuracy drops to 0.00%. This demonstrates that StreamingLLM does not extend the model's effective memory — it only preserves what is in the current cache. The model cannot answer questions about information that has been evicted from the rolling window, regardless of how stable its language modeling perplexity remains.

Critical Assessment

The paper makes several central claims, and the experiments provide varying degrees of support. I'll examine each major claim critically.

Claim 1: StreamingLLM enables LLMs to generate coherent text on infinite-length sequences without fine-tuning. The experiments provide compelling evidence for "very long" sequences but do not quite demonstrate "infinite." Figure 5 shows stable perplexity on 4 million tokens across eight models — this is three orders of magnitude beyond the training length and far exceeds what any existing method demonstrates. The perplexity curves show no upward trend that would suggest an approaching failure point. However, 4 million tokens is finite, and the paper cannot prove that no instability would emerge at 40 million or 400 million tokens. This is a practical rather than theoretical limitation — the evidence strongly suggests stability at arbitrary lengths, but the claim of "infinite" is an extrapolation from finite experiments. What the experiments do rigorously demonstrate is that the failure mode observed with dense attention (breakdown at the pre-training window) and window attention (breakdown at cache eviction) does not occur with StreamingLLM. The mechanism — keeping SoftMax attention score distributions stable by preserving sink tokens — provides a theoretical rationale for why this stability should hold at any length, but the experimental evidence is finite.

Claim 2: The attention sink phenomenon explains why window attention fails, and it is driven by absolute position rather than semantic content. This claim is strongly supported by two specific experiments. The linebreak substitution (Table 1: 5.60 vs. 5.40 perplexity for "\n" vs. original initial tokens) directly demonstrates positional dependence. The attention visualization (Figures 2, 11, 12, 13) systematically shows concentrated attention on initial tokens across layers, heads, and model scales. The cross-model consistency (Llama-2, Falcon, MPT, Pythia, and even BERT in Figure 14) demonstrates generality. The quantitative analysis in Figure 12 (Appendix F) showing that the 4096th token allocates substantially more than half its attention to the first token in most layers provides precise measurements. The paper's mechanistic explanation — that the SoftMax constraint forces attention distribution and initial tokens are always visible — is logically coherent and consistent with all observed data. A limitation: the paper does not provide a formal proof that SoftMax necessarily leads to attention sinks, and the Zero Sink experiment (Table 3) shows that SoftMax-off-by-One (which relaxes the sum-to-one constraint) still produces models that rely on initial tokens as sinks. This suggests the mechanism is more complex than pure mathematical necessity — there's a learned component that the paper acknowledges but does not fully dissect.

Claim 3: StreamingLLM achieves up to 22.2× speedup over the sliding window with re-computation baseline. The efficiency results (Figure 10, Section 4.5) directly support this number: 106ms vs. 2355ms for Llama-2-13B with a 256-token cache on an A6000 GPU. However, this 22.2× figure should be contextualized: (a) it's measured at the smallest cache size (256 tokens), and the speedup decreases as cache size increases (2.0–2.1× at 4096 tokens), (b) it's benchmarked on a single GPU with a specific implementation (Huggingface Transformers), and (c) the absolute latency numbers (31–106ms per token for StreamingLLM) may not reflect optimized production serving systems. The 22.2× number is the headline, but the more representative figure for practical cache sizes (1024–2048 tokens) is 3.3–6.4× — still substantial but less dramatic. The memory comparison shows near-identical usage between the two methods, which is expected since both store the same number of KV states.

Claim 4: Pre-training with a dedicated sink token enables streaming deployment with a single preserved token and does not harm standard performance. The pre-training experiments (Section 4.2) provide evidence but with important limitations. The models trained are 160M parameters — three orders of magnitude smaller than the largest models evaluated (Llama-2-70B). The paper does not demonstrate that the sink token approach scales to billion-parameter models. The training run is a single replicate (one seed, 143,000 steps), so there is no evidence about the reliability of the finding across random initializations. The streaming evaluation in Table 3 uses only the first sample of the PG-19 test set rather than a larger corpus. The standard benchmark evaluation (Table 4) shows approximately equal performance, but the differences (e.g., 18.6% vs. 19.6% on ARC-Challenge) are within what could be training variance with a single run. A rigorous demonstration would require multiple training runs with error bars. Despite these limitations, the result is plausible and well-motivated by the analysis of why existing models need multiple sink tokens.

Claim 5: StreamingLLM is suitable for streaming applications but does not extend context length or improve long-term memory. This is the paper's most honestly handled claim. The evidence supporting the limitation is clear: Table 7 (Appendix C) shows accuracy on StreamEval dropping to zero when query-answer distance exceeds cache size, and Table 8 (Appendix D) shows StreamingLLM with $4+3496$ underperforms the truncation baseline on LongBench tasks that require long-range context. The paper explicitly states this limitation in multiple places (Section 1, Appendix A, Appendix C). This transparency strengthens the paper's credibility.

Genuine weaknesses and missing experiments:

No evaluation of generation quality beyond perplexity. All language modeling results are perplexity-based. The paper does not provide qualitative examples of text generated by StreamingLLM over very long sequences, nor does it report metrics like coherence, repetitiveness, or diversity. Perplexity is a reasonable proxy for language modeling quality, but it is known to be imperfect — models can achieve low perplexity while producing degenerate text (e.g., repetitive loops). For a method that claims to enable "stable and efficient language modeling," qualitative examples would strengthen the case.
The 4 million token experiment uses a concatenated test set, not a single continuous document. The PG-19 test set consists of 100 separate books concatenated together. This means the model encounters hard document boundaries (book transitions) that may help "reset" its state in ways that a truly continuous single-document stream would not. The perplexity fluctuations at book boundaries (mentioned by the authors) confirm these boundaries exist. A more rigorous test would use a single extremely long document or a synthetic stream with no natural boundaries.
No comparison to context extension methods at matched context sizes. The paper positions StreamingLLM as complementary to context extension, but doesn't provide a direct comparison. For example: how does StreamingLLM with a 4K cache compare to a context-extended model (e.g., Llama-2-7B-32K) with a 4K cache using window attention? If the context-extended model with window attention also fails when initial tokens are evicted (as the paper would predict), this would strengthen the claim that attention sinks are a universal SoftMax property. If it doesn't fail, that would suggest context extension training changes attention patterns in ways that complicate the sink hypothesis.
No evaluation on tasks beyond language modeling and simple QA. The paper focuses on perplexity and retrieval accuracy (ARC, StreamEval). It does not test tasks that require compositional reasoning over the cached context, such as multi-step inference, mathematical reasoning, or code generation. Since StreamingLLM removes tokens from the middle of the sequence, it's possible that some types of reasoning that depend on coherence across the entire context could be impaired even if perplexity remains stable.
The pre-training experiments use 160M-parameter models, not the scale where attention sinks are most critical. The attention sink phenomenon is most prominent in larger models (Llama-2-70B shows it clearly in Figure 13). Training a 160M-parameter model and extrapolating the sink token findings to billion-parameter scales is reasonable but unvalidated. A 1B or 7B pre-training experiment would substantially strengthen this claim.
No ablation on which layers need attention sinks. The paper shows that attention sinks are most prominent in deeper layers (Figures 2, 12), but StreamingLLM preserves sink tokens for all layers uniformly. Could preserving sink tokens only in layers above a certain depth achieve the same effect with even fewer preserved tokens? This would be a natural ablation that could further optimize the memory footprint.
The positional encoding strategy (cache-local positions) is not ablated. Section 3.2 describes how StreamingLLM assigns contiguous cache-local positions rather than original sequence positions. This is described as "crucial for StreamingLLM's performance," but no experiment compares cache-local vs. original positions. If original positions work poorly (as the paper implies), showing this failure would validate the design. If they work surprisingly well, it would suggest the position assignment strategy is less critical than claimed. This is a notable omission given the centrality of position encoding to the method.
Single GPU, single precision, single framework benchmarks. The efficiency results (Figure 10) use Huggingface Transformers on one A6000 GPU. Production deployments would use optimized inference engines (e.g., vLLM, TensorRT-LLM) with quantization, continuous batching, and other optimizations. The relative speedup of StreamingLLM vs. re-computation might differ substantially in those settings. The paper notes in the Impact Statement that StreamingLLM has been adopted by TensorRT-LLM and other serving solutions, but does not provide benchmarks from those integrations.

These weaknesses do not invalidate the paper's contributions, but they bound the strength of its claims. The evidence for the attention sink phenomenon and the effectiveness of sink preservation for stabilizing window attention is compelling. The evidence for the practical benefits (speedup, deployment simplicity) is solid but context-dependent. The evidence for the forward-looking recommendation (sink token pre-training) is suggestive but preliminary due to the small scale of the pre-training experiments.

6. Limitations and Trade-offs

The Method Fundamentally Discards Information: No Retrieval of Evicted Context

The assumption or constraint: StreamingLLM permanently discards the KV states of tokens that fall out of the rolling cache window. The model has no mechanism—compressed, hierarchical, or otherwise—to access any information from tokens that have been evicted. The paper is admirably transparent about this, stating in Appendix A: "StreamingLLM is not suitable for tasks that demand long-term memory and extensive data dependency, such as long document question-answering (QA) and summarization. However, it excels in scenarios only requiring short-term memory, like daily conversations and short document QA."

The consequence: Any information that ages beyond the rolling cache window becomes permanently inaccessible. If a user refers to something mentioned earlier in a conversation—even something critical like a name, a constraint, or a decision—the model has no way to retrieve it unless it happens to have been preserved as part of the initial attention sink tokens (which is incidental, not designed). This is not a gradual degradation; it is a hard cliff. Table 7 in Appendix C demonstrates this starkly: with a 4+4092 cache configuration, accuracy on StreamEval is 84.60% when the query-answer distance is 460 tokens (within cache), 61.60% at 2300 tokens, and drops to 0.00% at 4600 tokens (exceeding the cache capacity). The model does not get worse at retrieving—it becomes completely incapable. This failure mode is silent: the model continues generating fluent, coherent text (its perplexity remains stable), but the content it produces will be disconnected from any information that has left the window. In a multi-hour conversation, essential context from earlier in the dialogue simply vanishes.

What evidence exists in the paper: Table 7 (Appendix C) provides the direct measurement of the cliff: accuracy is high within the cache, plummets to zero beyond it. Table 8 (Appendix D) shows the consequence on real-world long-context benchmarks: StreamingLLM with 4+3496 substantially underperforms the truncation baseline on NarrativeQA (11.6 vs. 18.7), Qasper (16.9 vs. 19.2), HotpotQA (21.6 vs. 25.4), and 2WikiMQA (28.2 vs. 32.8) because the initial input prompt—which contains task instructions and document framing—has been evicted from the cache. The paper's own StreamEval benchmark (Section 4.3) is explicitly designed to test retrieval of recent information (20 lines back), avoiding the regime where StreamingLLM would fail.

Mitigation status: Not addressed by the paper, and arguably not addressable within the StreamingLLM framework. The paper positions this as a fundamental design tradeoff rather than a bug: "StreamingLLM firstly decouples the LLM's pre-training window size and its actual text generation length, paving the way for the streaming deployment of LLMs" (Section 5). The implication is that other mechanisms—compressed memory, retrieval-augmented generation, context extension—must handle long-term memory, and StreamingLLM handles operational stability. This is a fair decomposition but means that any deployment requiring both persistence and recall must combine StreamingLLM with additional systems that the paper does not develop or evaluate. The StreamEval experiments with context-extended models (LongChat-7b-v1.5-32k and Llama-2-7B-32K-Instruct, Figure 9) demonstrate that larger rolling caches help—they delay the cliff by expanding the window—but do not eliminate the fundamental tradeoff: tokens still get evicted, and information still gets lost.

The 22.2× Speedup Claim Is a Best-Case Figure That Diminishes at Practical Cache Sizes

The assumption or constraint: The paper's headline efficiency claim—"up to 22.2× speedup" over the sliding window with re-computation baseline—is measured at the smallest evaluated cache size (256 tokens) and decreases substantially as cache size increases to more practical values. The 22.2× number comes from a specific measurement: Llama-2-13B on a single NVIDIA A6000 GPU with Huggingface Transformers at cache size 256, where StreamingLLM achieves 106ms per token vs. the re-computation baseline's 2355ms (Section 4.5, Figure 10).

The consequence: A practitioner reading the headline might expect a 22.2× speedup in their deployment, but the actual speedup at cache sizes that provide useful context (1024–4096 tokens) is 2.0–6.4×. The speedup figures from Figure 10 tell the full story:

Cache 256: 22.2× (Llama-2-13B)
Cache 512: 11.5×
Cache 1024: 6.0×
Cache 2048: 3.3×
Cache 4096: 2.1×

The same pattern holds for Llama-2-7B: 21.7× → 11.6× → 6.4× → 3.3× → 2.0×. At the largest cache size (4096), StreamingLLM is only about twice as fast as the re-computation baseline. This is still a meaningful improvement—halving latency for the same quality is valuable—but it is an order of magnitude less dramatic than the headline figure. The reason is that StreamingLLM's latency grows linearly with cache size ( $O(L)$ for the attention computation), while the re-computation baseline grows quadratically ( $O(L^2)$ for recomputing attention within the window). At small $L$ , the quadratic term is modest, but as $L$ grows, the re-computation cost explodes relative to StreamingLLM—which is why the speedup is largest at small cache sizes. A practitioner choosing a cache size for their application will face a direct tradeoff: larger caches provide more recent context (delaying the information-loss cliff documented in Limitation 1) but reduce the relative speedup advantage over the baseline.

What evidence exists in the paper: Figure 10 reports all the numbers transparently. The paper's text explicitly states "up to 22.2× speedup" (Section 1, Abstract, Section 4.5), which is technically accurate for the best-case measurement. However, the paper does not discuss how the speedup varies with cache size or provide guidance on choosing a cache size that balances latency and context retention. The efficiency benchmarks are also limited to a single hardware configuration (NVIDIA A6000) and a single software framework (Huggingface Transformers), without evaluation on optimized inference engines that might change the absolute numbers or relative scaling.

Mitigation status: Not addressed. The paper presents the full cache-size sweep in Figure 10 but does not explicitly discuss the tradeoff or recommend cache sizes for different deployment scenarios. A practitioner must extract this information from the figure themselves. The Impact Statement notes adoption by TensorRT-LLM, Intel Extension for Transformers, and other serving solutions, suggesting that optimized implementations exist, but no benchmarks from those integrations are provided. The relative scaling behavior ( $O(L)$ vs. $O(L^2)$ ) is fundamental and would persist across implementations, but the absolute latencies and the crossover points where StreamingLLM's advantage becomes marginal could shift.

The Pre-Training Sink Token Recommendation Is Validated Only at 160M Parameters—Not at Deployment Scale

The assumption or constraint: The paper's forward-looking recommendation to pre-train models with a dedicated sink token rests on experiments with 160-million-parameter models trained on 8×A6000 GPUs for 143,000 steps using the Pythia-160M codebase (Section 3.3, Section 4.2). The attention sink phenomenon itself is demonstrated at scale—Llama-2-70B shows the same initial-token attention concentration (Figure 13, Appendix G) as Llama-2-7B and the 160M models—but the specific claim that a single learnable sink token can replace four ad-hoc initial tokens during pre-training is verified only at 160M parameters.

The consequence: The gap between 160M parameters and the models where streaming deployment is most valuable (7B–70B parameters) is enormous—roughly 44–438× larger. Several aspects of the sink token approach might interact with scale in ways the paper cannot observe:

Training dynamics: At 160M parameters, the model may not have the capacity to develop the rich, distributed attention patterns that require multiple sink tokens. A 70B-parameter model, with many more attention heads and layers, might learn to use the sink token differently—potentially still relying on multiple initial positions despite the dedicated sink, or alternatively, making the sink token even more effective. The paper cannot distinguish these possibilities.
The single training run problem: Each variant (Vanilla, Zero Sink, Learnable Sink Token) is trained exactly once. Table 4 shows near-identical zero-shot performance (e.g., ARC-Challenge: 18.6% vs. 19.6%, PIQA: 62.2% vs. 62.6%), but with a single training run per variant, there is no way to estimate the variance in these numbers. The 1.0–1.5 percentage point differences could be noise from random initialization, data order, or hardware nondeterminism rather than genuine effects of the sink token.
The streaming evaluation uses a single PG-19 sample: Table 3 evaluates streaming perplexity on the first sample of the PG-19 test set only. This is a single data point. While the attention sink phenomenon is consistent across models and texts (Figures 2, 11, 12, 13), the specific claim that one sink token suffices for a model trained with one sink token is not validated across multiple texts or domains.
Potential for negative transfer: The sink token occupies position 0 in every training sample, consuming some of the model's limited positional budget. At 160M parameters with short training sequences, this is negligible. At 70B parameters with long training sequences, it's probably still negligible, but the paper provides no evidence.

What evidence exists in the paper: Tables 3, 4, 9, and 10; Figures 6 and 7; Appendix I. The evidence that exists is internally consistent and supports the sink token hypothesis at the tested scale. The convergence curves (Figure 6) show identical training dynamics, the benchmark results (Tables 4 and 9) show no degradation, and the attention visualizations (Figure 7) confirm the mechanism. But all of this is at 160M parameters with a single training run per variant.

Mitigation status: The paper does not acknowledge this scale gap as a limitation. The recommendation is stated unqualified: "We recommend training future LLMs with a sink token in all samples to optimize streaming deployment" (Section 3.3). The authors likely believe—with some justification, given the architectural universality of the attention sink phenomenon—that the finding scales. But the experimental evidence provided would not meet the burden of proof for a team deciding whether to modify the pre-training recipe of a 70B-parameter model (a multi-million-dollar endeavor). A 1B or 7B validation experiment would substantially strengthen this claim, but it does not exist in the paper.

The Method Requires Preserving Initial Tokens That May Contain Sensitive, Irrelevant, or Misleading Content

The assumption or constraint: StreamingLLM permanently preserves the KV states of the first ~4 tokens of the sequence for the entire lifetime of the streaming session—potentially millions of tokens and hours or days of wall-clock time. These initial tokens are typically whatever happened to be at the start of the conversation: a <s> token, the first user query, or the first few words of the system prompt. The paper demonstrates (Table 1) that the semantic content of these tokens is largely irrelevant to their function as attention sinks—the model attends to their position, not their meaning—but they nonetheless continue to participate in every single attention computation for every subsequent token generation.

The consequence: This creates several practical concerns that the paper does not discuss:

Privacy and data retention: In a multi-turn conversation, the preserved initial tokens represent a permanent record of the conversation's beginning that cannot be deleted without breaking the model's stability. If a user asks to delete early parts of a conversation (as required by regulations like GDPR's "right to erasure"), removing those initial tokens would catastrophically degrade the model's outputs—just as window attention fails when initial tokens are evicted (Figure 3, Table 1). The model's operational stability is coupled to the permanent retention of specific user data.
Attention pollution: Even though the paper shows that sink tokens receive attention for structural rather than semantic reasons, they are still real tokens with real Key and Value representations that influence the attention output. The SoftMax weights may concentrate on the sink tokens, but the Value vectors of those tokens are still multiplied by those weights and added to the attention output. If the initial tokens contain misleading, toxic, or otherwise problematic content, that content has a permanent, non-removable channel into every subsequent generation step. The paper does not analyze whether initial token semantics leak through the attention mechanism despite the sink behavior, or whether neutralizing initial tokens (e.g., replacing them with padding) changes generation quality beyond perplexity.
Application-specific initial tokens: In different deployment scenarios, the initial tokens might be highly variable. A chatbot might start with a system prompt, a user greeting, or an image description. A code completion system might start with a file header. A transcription system might start with speaker identification. StreamingLLM treats all of these uniformly—the first four tokens are preserved—but the paper does not study whether different types of initial content affect downstream generation quality or whether certain initial token types are more suitable as sinks than others. The linebreak experiment (Table 1) suggests semantics don't matter for perplexity, but perplexity is a coarse metric that may miss subtle effects on generation quality, coherence, or task performance.

What evidence exists in the paper: Table 1 shows that replacing initial tokens with "\n" characters preserves perplexity (5.60 vs. 5.40), suggesting that semantic content of sink tokens has minimal impact on language modeling metrics. However, this is a single experiment on a single model (Llama-2-13B) measuring only perplexity. The paper does not evaluate whether initial token content affects downstream task performance (e.g., QA accuracy, dialogue coherence, factual consistency) or whether certain types of initial content produce better or worse results over very long streams. The attention visualizations (Figures 2, 7, 11, 13) confirm that attention is concentrated on initial tokens, but do not analyze what information from those tokens actually propagates to subsequent layers.

Mitigation status: Not addressed. The paper's framing treats the initial tokens purely as structural attention anchors and does not consider the data governance or content-quality implications of permanently preserving them. A practical mitigation—using neutral placeholder tokens as the first four tokens of every streaming session (inspired by the "\n" experiment)—is implicitly suggested by the results but never explicitly recommended or evaluated for long-term streaming stability. The pre-training sink token proposal (Section 3.3) partially addresses this by providing a dedicated, controlled token whose content can be designed to be benign, but this solution requires training new models and does not help with currently deployed models.

Performance Is Evaluated Predominantly Through Perplexity, Leaving Generation Quality Unexamined

The assumption or constraint: The paper's primary evaluation metric for language modeling quality is perplexity (Figures 3, 5; Tables 1, 2, 3, 6, 10). The streaming question-answering experiments (Section 4.3) use exact-match accuracy as a secondary metric, and StreamEval measures retrieval accuracy. But for the core claim that StreamingLLM enables "stable and efficient language modeling with up to 4 million tokens and more" (Abstract), the evidence is almost entirely perplexity-based.

The consequence: Perplexity measures how well the model predicts the next token in a ground-truth sequence, but it is known to be an imperfect proxy for generation quality. Models can achieve low perplexity while producing degenerate text—for instance, by becoming repetitive, by generating fluent but generic outputs, or by losing topical coherence while maintaining local token-level predictability. These failure modes are particularly relevant for StreamingLLM because:

Repetition loops: A common failure mode in long-form LLM generation is getting stuck in repetitive patterns. Because StreamingLLM only sees the most recent tokens (plus the sink tokens), it might be more susceptible to repetition if the rolling cache contains repetitive text that reinforces itself. Perplexity measured on ground-truth text (which is not repetitive) would not capture this.
Topic drift: Over very long streams, the conversation or document naturally evolves through multiple topics. With only a fixed-size recent window, the model might lose the larger topical context and produce text that is locally fluent but globally incoherent—shifting topics unnaturally or contradicting earlier statements. Perplexity, calculated token-by-token against a pre-written text, cannot detect this because the ground-truth text maintains coherence by construction.
Factual consistency: In a streaming question-answering setting, the model might produce answers that sound plausible but are factually inconsistent with information that has been evicted from the cache. The StreamEval benchmark (Section 4.3) tests this to some extent—measuring whether the model can retrieve specific facts from within the cache—but only for the specific case of exact-match retrieval of line-content pairs. It does not test whether the model maintains a consistent persona, remembers stated preferences, or avoids contradicting itself across topics.

What evidence exists in the paper: The streaming QA experiments (Table 5) provide some generation quality evidence: StreamingLLM matches one-shot accuracy on ARC, demonstrating that the model's ability to answer factual questions does not degrade. StreamEval (Figure 9, Table 7) shows that within-cache retrieval is maintained. But neither of these tests evaluate open-ended generation quality, coherence over long contexts, or the linguistic quality of generated text. The paper does not provide a single example of text generated by StreamingLLM over a long stream, making it impossible for a reader to judge whether the generated text is actually useful in practice. For a method that claims to enable "streaming deployment of LLMs" (Section 5), the absence of qualitative examples or task-based generation evaluations is a significant gap.

Mitigation status: Partially addressed through the StreamEval benchmark and the ARC streaming experiments, which go beyond perplexity to test task performance. But these tests are narrow—they evaluate specific retrieval and question-answering capabilities rather than the open-ended generation quality that would matter for the dialogue and assistant applications the paper envisions. The paper acknowledges in Appendix A that StreamingLLM is not suitable for tasks requiring long-term memory, which implicitly acknowledges that generation quality may suffer when information is evicted, but does not analyze whether generation quality degrades in more subtle ways even when perplexity is stable. A qualitative analysis or a human evaluation of long-form generated text would substantially strengthen the paper's practical claims.

The Position Encoding Strategy Is Described as Crucial but Never Ablated or Formally Justified

The assumption or constraint: Section 3.2 describes how StreamingLLM assigns cache-local positions to tokens rather than using their original sequence positions. The paper states: "StreamingLLM focuses on positions within the cache rather than those in the original text. This distinction is crucial for StreamingLLM's performance" (Section 3.2). The specific procedure: for a cache containing original-position tokens [0, 1, 2, 3, 6, 7, 8] (four sink tokens at 0–3, three recent tokens at 6–8, tokens at 4–5 evicted), the model reassigns positions [0, 1, 2, 3, 4, 5, 6]. For RoPE models, keys are cached before rotary transformation, and the transformation is applied at each decoding step using cache-local positions. For ALiBi models, a contiguous linear bias is applied based on cache-local distances.

The consequence: Despite being described as "crucial," the impact of this design choice is never empirically demonstrated. The paper does not report what happens if original sequence positions are used instead—would the model fail entirely? Degrade gradually? Work surprisingly well? This matters because:

The position encoding strategy is a moving part that could interact poorly with other components. If a practitioner implements StreamingLLM but misunderstands or misconfigures the position assignment, the method might silently underperform or fail. Without an ablation showing the consequence of getting this wrong, the practitioner has no way to debug position-related issues.
The claim that relative position encodings are sufficient for infinite-length operation depends entirely on this strategy working correctly. If cache-local positions introduce subtle distribution shifts—for instance, if the model was trained with positional embeddings that encode absolute position information despite using relative encodings—then StreamingLLM might have a hidden failure mode that emerges only at very long streams.
It is possible that the position strategy matters less than the paper claims. If the model's attention is dominated by the sink tokens (which are always at positions 0–3 regardless of the strategy) and local attention to recent tokens (which have small relative distances), the exact position assignment for intermediate distances might have minimal impact. If this is the case, the method is more robust than the paper implies, which would be valuable information. The paper provides no evidence either way.

What evidence exists in the paper: Absolutely none. The paper does not ablate the position encoding strategy. No experiment compares cache-local positions to original-sequence positions, to no-position-encoding, or to any alternative strategy. The description in Section 3.2 provides the implementation detail and asserts its importance, but no empirical evidence is offered to support the assertion. This is a notable omission given the paper's otherwise thorough empirical approach and the centrality of position encoding to the length-extrapolation problem the method addresses.

Mitigation status: Not addressed. The paper relies entirely on the conceptual argument that keeping positions within the training range should preserve performance, and the indirect evidence that StreamingLLM works (Figures 3, 5) validates the combined approach without isolating the contribution of position encoding. A simple ablation—measuring perplexity with original positions vs. cache-local positions at a fixed cache size—would directly address this and is well within the paper's experimental scope. Its absence weakens the claim that the position encoding strategy is "crucial" and makes the method harder to implement correctly based on the paper alone.

7. Implications and Future Directions

How This Work Changes the Landscape

StreamingLLM causes a reframing of the length extrapolation problem from a model capability challenge to an attention management challenge. Before this work, the prevailing assumption was that LLMs' inability to generate beyond their training sequence length reflected a fundamental computational limitation—models simply didn't learn to process position encodings or attention patterns at unseen scales, and fixing this required either architectural changes (ALiBi, position interpolation) or expensive fine-tuning (Chen et al., 2023; Peng et al., 2023). The problem was "how do we train models to handle longer sequences?"

StreamingLLM demonstrates that this framing was partly wrong. The models already know how to handle longer sequences in terms of their core computations—the feedforward layers, the attention mechanism itself, the token prediction pipeline all work correctly at arbitrary lengths. The failure is specifically and narrowly in two interface-level details: (1) position encodings entering untrained numerical regimes, and (2) the attention score distribution being catastrophically distorted when SoftMax denominators lose the dominant terms from initial sink tokens. Fix these two interface issues—by reassigning cache-local positions and preserving attention sinks—and the same frozen model handles 4 million tokens as comfortably as 4,000. This is not a new capability being added; it's a pre-existing capability being unlocked by removing artificial constraints on how tokens are presented to the model.

This reframing shifts the research agenda in several ways. It redirects attention from training solutions to inference-time KV cache management. If the model's core computations are length-agnostic, then the key design space is not "how do we train better position encodings?" but "how do we manage the KV cache to keep the model's attention computation in a regime it recognizes?" This makes KV cache eviction policies a first-class research area rather than an implementation detail. Methods that evict tokens based on attention scores (H2O, Zhang et al., 2023b), compress tokens into summary representations, or dynamically adjust which tokens are preserved all become more interesting because the paper provides a theoretical framework—attention score distribution stability—for evaluating them.

It clarifies what "length generalization" actually means, decomposing it into three separate sub-problems that had been conflated: (1) operational stability (can the model keep running without producing garbage?), (2) context capacity (how many tokens of information can the model access at once?), and (3) context utilization (does the model effectively use the context it has access to?). StreamingLLM solves (1) without touching (2) or (3), making explicit what had been an implicit confusion in the literature. This decomposition is valuable even if StreamingLLM is not the final solution—it provides a framework for positioning future work. A context extension method that increases the training window from 4K to 32K tokens solves a context capacity problem but still needs an operational stability solution for streaming beyond 32K. A position interpolation method that enables extrapolation to 2× training length solves a different problem from one that enables 1000× training length. The paper's framework makes these distinctions explicit.

It reconciles contradictory observations about initial token importance. Some prior work had observed that LLMs are sensitive to the presence of early tokens (e.g., "lost in the middle" phenomena documented by Liu et al., 2023), but the dominant interpretation was semantic: early tokens contain important task framing or document context. StreamingLLM's linebreak substitution experiment (Table 1) proves that at least part of this sensitivity is purely positional—the SoftMax mechanism itself creates a dependency on initial tokens regardless of their content. This doesn't mean semantics don't matter (they clearly do for task performance), but it means there's a structural floor to the importance of initial tokens that cannot be trained away. This explains why even sophisticated context extension methods still show some degradation at extreme lengths: they're fighting the SoftMax constraint, not just the position encoding.

It connects the LLM literature to a broader phenomenon in Transformers. By documenting attention sinks in BERT (Figure 14, Appendix H) and linking to concurrent work on "registers" in Vision Transformers (Darcet et al., 2023), the paper establishes that attention sinks are not a quirk of autoregressive language models—they are a universal property of SoftMax-based self-attention across architectures and modalities. This has implications beyond streaming deployment. If all Transformers learn to designate certain tokens as attention sinks, then any method that discards or compresses tokens during inference (for efficiency, summarization, or memory management) must account for which tokens serve as sinks. Dynamic token pruning methods (e.g., Anagnostidis et al., 2023) that accidentally evict attention sink tokens will cause the same catastrophic collapse that window attention exhibits. The attention sink concept provides a diagnostic framework for understanding why some token-pruning methods work and others fail.

It redirects attention away from purely mathematical fixes for the SoftMax constraint. The Zero Sink experiment (Table 3) is a significant negative result with implications for a line of work that attempts to "fix" attention by modifying the SoftMax function (Miller, 2023). The fact that SoftMax-off-by-One—which mathematically removes the sum-to-one constraint—does not eliminate attention sink behavior demonstrates that attention sinks are learned computational strategies, not just mathematical artifacts. Models discover during training that having a fixed set of tokens to absorb attention mass is useful, and they will create such tokens even when not strictly forced to. This suggests that SoftMax modifications alone cannot solve the length generalization problem; they must be combined with KV cache management strategies that work with the model's learned attention patterns rather than trying to mathematically prevent them.

Follow-Up Research This Work Enables

Attention-sink-aware token eviction policies. StreamingLLM uses a simple FIFO eviction policy on the rolling cache: the oldest non-sink token is always the one discarded. But not all recent tokens are equally important for future predictions, and the attention sink analysis provides a framework for more sophisticated policies. A strong follow-up would design an eviction policy that scores each token in the rolling cache by its "attention sink potential"—how much attention mass it attracts from later tokens—and preferentially preserves tokens that emerge as secondary sinks within the rolling window. The evaluation would compare this policy against StreamingLLM's FIFO baseline on both perplexity and downstream task performance (StreamEval, LongBench) across multiple cache sizes. The hypothesis is that some tokens in the rolling window naturally acquire sink-like properties as the stream evolves (e.g., section boundaries in a document, speaker turns in a dialogue), and preserving them preferentially could extend the effective memory of the cache without increasing its size. The H2O method (Zhang et al., 2023b) already explores attention-score-based eviction, but without the attention sink framework to explain why certain tokens accumulate high scores. Connecting these ideas could yield eviction policies that are both theoretically grounded and practically effective.

Layer-specific sink token preservation. Figures 2 and 12 show that attention sinks are most prominent in deeper layers—the bottom two layers show local attention patterns, while layers 3+ increasingly concentrate attention on initial tokens. StreamingLLM preserves sink tokens uniformly across all layers, which may waste memory in early layers where sink attention is weak or absent. A natural ablation: preserve sink tokens only from layer K onwards, where K is determined by measuring the attention mass on initial tokens per layer. The experiment would sweep K from 0 to the full layer depth, measuring both perplexity (PG-19) and memory savings (since skipping sink preservation in early layers reduces the KV cache size). The hypothesis is that K can be relatively large (maybe 1/3 to 1/2 of total layers) with minimal perplexity impact, yielding meaningful memory reduction. This is a direct extension of the paper's own observation that "the attention maps in the first two layers exhibit the 'local' pattern" (Section 3.1) but has not been exploited for efficiency. The method is trivially implementable on top of StreamingLLM and would produce clear, quantifiable results.

At-scale validation of sink token pre-training for billion-parameter models. The paper's recommendation to train future LLMs with a dedicated sink token rests on 160M-parameter experiments (Section 3.3). A critical follow-up would validate this at the 1B–7B parameter scale where streaming deployment is practically relevant. The experiment would replicate the paper's setup (Pythia training recipe, deduplicated Pile, identical hyperparameters) at Pythia-1B or Pythia-6.9B scale with and without a sink token, measuring: (1) training convergence (loss curves), (2) downstream benchmark performance (the same seven benchmarks from Table 4, plus additional long-context tasks from LongBench), (3) streaming perplexity with varying numbers of preserved sink tokens (0, 1, 2, 4), and critically (4) multiple training runs (at least 3 seeds) to estimate variance. This experiment would either confirm that the single-sink-token finding scales—strengthening the paper's recommendation considerably—or reveal scale-dependent effects (e.g., larger models distributing sink behavior across more tokens despite the dedicated sink) that would refine our understanding of how attention sinks develop during training. The cost is non-trivial (7B-parameter pre-training runs) but feasible for academic labs with access to moderate GPU clusters, and the result would directly inform whether model developers should adopt the sink token practice.

Qualitative evaluation of long-form streaming generation. The paper evaluates StreamingLLM almost exclusively through perplexity (Figures 3, 5, Tables 1–3, 6) and narrow retrieval accuracy (Table 5, StreamEval). A crucial follow-up would characterize the actual generation quality of models running under StreamingLLM for extended periods. The experiment would run several models (Llama-2-7B, Llama-2-13B) with StreamingLLM cache configurations (e.g., 4+2044, 4+1020) on an open-ended generation task—for instance, continuing a long story, maintaining a multi-turn dialogue, or generating a technical document section by section—for 50K+ tokens. Evaluation would include both automatic metrics (repetition rate at different n-gram levels, semantic coherence via embedding similarity between adjacent segments, topic modeling to detect drift) and human evaluation (fluency, coherence, consistency judgments on sampled segments). The goal is to identify failure modes that perplexity misses: Does the model gradually lose topical coherence? Does it start repeating phrases or sentence structures? Does it contradict information from earlier in the stream (which has been evicted)? This experiment would define the practical operational envelope of StreamingLLM—not just "does it work?" (which perplexity says yes) but "how well does it work, and what are the degradation patterns?"—which is essential for real-world deployment decisions.

Position encoding ablation study. Section 3.2 describes the cache-local position assignment as "crucial for StreamingLLM's performance" but provides no empirical evidence. A direct ablation would compare StreamingLLM under three position assignment strategies at a fixed cache size (e.g., 2048) and stream length (e.g., 100K tokens): (1) cache-local positions (the paper's method), (2) original sequence positions (tokens keep their absolute positions in the full stream), and (3) no position encoding (for models where this is feasible, or by zeroing position biases). The evaluation would measure perplexity (PG-19) and downstream task accuracy (StreamEval). If original positions cause catastrophic failure (as the paper implies), this validates the design and provides a diagnostic for implementers. If original positions work surprisingly well—for instance, if the model's attention is so dominated by sink tokens and local context that position encoding for intermediate distances barely matters—then the method is more robust than claimed and the position assignment strategy is less critical. Either result is valuable: the first confirms a non-obvious design requirement, the second simplifies implementation. The experiment is straightforward and directly addresses a gap in the paper's empirical coverage.

Cross-task attention sink analysis and domain-specific sink optimization. The paper demonstrates attention sinks in language modeling (PG-19) and question answering (ARC, StreamEval), but doesn't analyze whether sink patterns change across task types. A follow-up would measure attention sink behavior—specifically, the distribution of attention mass across positions 0–3 in each layer and head—when the same model processes different input types: standard pretraining text (PG-19), instruction-following prompts, code (using Code Llama), multi-turn dialogue, and structured data (e.g., JSON or tables). If attention sink patterns are stable across tasks, StreamingLLM's uniform 4-sink-token policy is robust. If patterns vary—for instance, code generation showing weaker sink dependence because of the structured nature of the task—then task-adaptive sink preservation (preserving more or fewer sink tokens depending on detected task type) could further optimize the memory-quality tradeoff. The experiment would use Llama-2-7B and Code Llama-7B, measuring per-layer attention mass on initial tokens across at least 100 examples per task type. A substantial finding would be evidence that some tasks naturally create alternative sink tokens (e.g., system prompts in dialogue, function signatures in code) that could reduce reliance on the absolute initial tokens.

Practical Applications and Downstream Use Cases

Persistent conversational agents with bounded memory. The most direct application is in dialogue systems (chatbots, voice assistants, customer support agents) that must maintain coherent conversations over extended periods—hours or days—without requiring cache resets or loss of recent context. StreamingLLM enables these systems to run continuously with a fixed memory budget: a 7B-parameter model with a 4+2044 cache configuration consumes constant GPU memory (~14–19 GB depending on precision, from Figure 10) and maintains stable per-token latency (~31–52ms for Llama-2-7B at cache sizes 2048–4096 on an A6000, Figure 10) regardless of conversation length. The practical benefit is eliminating the engineering complexity of conversation state management—no need for summarization modules, context window resets, or sliding-window-with-recomputation hacks when conversations exceed the training length. For a customer support deployment handling thousands of concurrent conversations, the 22.2× per-token speedup over the re-computation baseline at small cache sizes (Figure 10) translates directly to reduced GPU requirements or increased throughput per GPU. The limitation—that information beyond the rolling cache is permanently lost—is acceptable for dialogue where conversational relevance decays naturally with time and distance.

Real-time transcription and live captioning. Applications that process continuous audio streams (live captioning, meeting transcription, broadcast monitoring) face the same unbounded-stream challenge as dialogue, but with the added constraint that the model sees text as it is produced by an automatic speech recognition system—there are no natural "document boundaries" to reset the cache. StreamingLLM enables these systems to run indefinitely without the periodic cache resets that would cause information loss at arbitrary (and potentially critical) moments. The stable perplexity on 4M+ tokens (Figure 5) means the language model's predictions remain calibrated over multi-hour streams, which is essential for the model to serve as a reliable rescoring or contextual biasing component for the ASR system. The 2.0–6.4× speedup at practical cache sizes (2048–4096, Figure 10) compared to the re-computation baseline means lower end-to-end latency for live applications where every millisecond matters.

On-device assistants with small models and tight memory budgets. The paper's finding that 4 attention sink tokens suffice for models without dedicated sink token pre-training (Table 2) and that the streaming mechanism works across model scales from 2.8B to 70B parameters (Figure 5) has direct implications for on-device deployment. Smartphone or embedded-system assistants running quantized small models (e.g., a 2.8B-parameter model in 4-bit precision) can use StreamingLLM to maintain coherent multi-turn interactions without the memory growth that would exhaust device RAM. The constant KV cache memory—determined by cache size rather than conversation length—is predictable and budgetable, which is essential for resource-constrained environments. The paper's demonstration that the 160M-parameter models trained with a sink token require only a single preserved token (Table 3, 1+1023 achieves 18.01 perplexity) suggests a path to even tighter memory optimization: if future small models are pre-trained with sink tokens, on-device streaming deployment requires preserving exactly one token's KV state as structural overhead.

Continuous monitoring and alerting systems. Applications that continuously process streaming data—social media monitoring, financial news analysis, security log review—need language models that can operate 24/7 without degradation. StreamingLLM provides the operational stability (stable perplexity, no mode collapse) that makes LLM-based monitoring feasible. Because these applications typically only need the most recent context window to detect anomalies or generate alerts, StreamingLLM's bounded-memory design is well-matched to the task: the rolling cache provides sufficient context for immediate analysis, and the attention sinks prevent the catastrophic degradation that would cause false positives or missed detections when the stream exceeds training length. The linear latency scaling with cache size (Figure 10) means that throughput requirements can be met by tuning the cache size to the minimum needed for the detection task, rather than being constrained by the total stream length.