ArXiv: 2604.10098

🎯 Pitch

Transformers across all architectures—language, vision, and multimodal—squander massive amounts of attention on semantically useless tokens like punctuation and background patches, yet surgically removing this phenomenon causes catastrophic performance collapse. This survey maps over 180 papers to show that these “attention sinks” emerge from a fundamental Softmax constraint forcing models to dump excess probability somewhere, not from a learnable feature, and that effective mitigation must disrupt this process at the mathematical level rather than simply redistributing attention after the fact.

1. Executive Summary

This survey systematically consolidates over 180 studies on Attention Sink (AS) — the phenomenon where Transformers concentrate disproportionate attention on uninformative tokens — organizing the literature into three dimensions: Fundamental Utilization, Mechanistic Interpretation, and Strategic Mitigation. Across architectures including LLMs, ViTs, MoE models, and multimodal LLMs, the paper taxonomizes utilization strategies such as Sink Token Preservation (retaining initial tokens during KV cache compression to prevent performance collapse) and Attention Redistribution (reallocating attention mass from sink tokens to semantically relevant targets), interprets AS through theories including Softmax Limitations and No-Op Theory (the sum-to-one constraint forcing attention onto low-information tokens when no meaningful key exists) and Outlier Circuits (weight, activation, and attention outliers forming interconnected pathways that sustain sink behavior), and examines mitigation approaches such as Gated Attention Mechanisms (learnable gates that suppress attention outputs to decouple no-op behavior from extreme logits). The survey establishes that AS is not an isolated artifact of causal LLMs but a recurring structural phenomenon across diverse architectures, surfacing on delimiters in classical language models, background patches in ViTs, and both text-side and visual anchors in multimodal models — and that mitigation techniques eliminate AS effectively only when they either provide explicit alternatives to implicit sink mechanisms or disrupt the causal chain at the Softmax level, not when they merely redistribute attention post-hoc.

2. Context and Motivation

The Core Problem: A Fragmented Field Without a Systematic Map

The fundamental gap this survey addresses is not a single technical question but the absence of a unified understanding of Attention Sink (AS) across disparate research communities. Despite AS being observed and studied since at least 2019 in classical language models, 2023 in large language models, and subsequently across vision, multimodal, and diffusion transformers, the literature has developed in isolated silos. Researchers in efficient LLM inference studying KV cache compression, those in mechanistic interpretability investigating outlier circuits, and those in vision transformers analyzing background patch artifacts are all contending with the same underlying phenomenon — but their findings, vocabulary, and proposed solutions rarely cross-reference one another.

The paper identifies this fragmentation through three explicit questions (Section 1.1):

"Q1: What are the fundamental paradigms for leveraging AS in current Transformer models? What are their distinctive characteristics, and how are they applied across different Transformer architectures?"

"Q2: What underlies the emergence and necessity of AS in Transformers? How does it develop and evolve, and what functional roles does it fulfill?"

"Q3: How can future Transformer architectures be designed or optimized to operate independently of AS?"

These questions capture the field's state before this survey: separate communities had partial answers, but no one had stitched them into a coherent picture. A practitioner encountering AS in a ViT might spend weeks rediscovering insights that the LLM community had already formalized under different terminology (e.g., "massive activations" vs. "outlier dimensions"). A researcher developing mitigation strategies might not realize that prior work in a different architecture had already identified the Softmax sum-to-one constraint as the root cause, leading to redundant effort or suboptimal solutions.

Why This Matters: From Architectural Curiosity to Deployment Bottleneck

The importance of understanding AS extends far beyond academic curiosity about attention patterns. The paper demonstrates that AS is a cross-cutting concern with direct consequences for practically every stage of the model lifecycle (Sections 2-5, 6):

Inference efficiency. AS tokens — particularly the initial tokens in causal LLMs — serve as indispensable but numerically expensive anchors. When KV cache compression methods such as those surveyed in Section 3.1 (StreamingLLM, H2O, DuoAttention) fail to preserve these tokens, performance collapses catastrophically. StreamingLLM [24] found that evicting initial tokens from the sliding window causes perplexity to spike from 5.43 to 5,158 on long texts — an order-of-magnitude degradation. Conversely, preserving sink tokens at full precision during KV cache quantization (KVSink [28], IntactKV [151]) is critical because their small value norms make them disproportionately sensitive to numerical error. Without understanding why these tokens matter, practitioners either over-retain them (wasting memory) or accidentally evict them (breaking the model).

Quantization and low-bit deployment. AS is intimately linked to activation outliers that pose a primary bottleneck for model quantization. The paper documents (Section 4.2) how massive activations on sink tokens can reach values exceeding 10,000× the mean, making per-tensor static quantization nearly impossible without dedicated mitigation. Techniques such as Prefixing Attention Sinks [156] (Section 3.3) and Outlier-Safe Pre-Training [42] (Section 5.4) are not merely academic exercises — they directly enable 2-bit KV cache quantization and FP8 training at production scale, with throughput improvements of 36% or more.

Hallucination in multimodal models. In MLLMs, visual attention sinks — specific background patches or non-semantic visual tokens that attract disproportionate attention — have been causally linked to hallucination (Section 3.2, 6.5). VAR [105] and AttnReal [31] demonstrate that reallocating attention mass from these visual sinks to foreground objects reduces hallucination without retraining. The practical implication is immediate: for an MLLM deployed in a medical imaging or autonomous driving context, unmitigated visual AS could cause the model to "see" objects that aren't there, with potentially catastrophic consequences.

Training stability and model capacity. The paper shows (Section 4.2, 5.4) that AS emerges from mutually reinforcing outlier circuits — weight outliers in down-projection matrices amplify activation outliers, which induce attention concentration on sink tokens, which in turn reinforces the pattern through training dynamics. These circuits are not benign; they consume representational capacity that could otherwise be allocated to semantically meaningful tokens. In MoE LLMs, Super Experts [43] form specifically around sink tokens, with pruning just 3 of 6,144 experts in Qwen3-30B-A3B causing catastrophic performance collapse. Understanding this coupling is essential for designing next-generation architectures that don't structurally rely on sink mechanisms.

Emerging architectures. The survey highlights (Section 2.5, 7.2) that AS behaviors in newly emerging architectures — diffusion language models, vision-language-action models, hybrid linear attention models — remain largely unexplored. The moving sink phenomenon in DLMs [134] and the role of register tokens in VLA models [135] are initial glimpses into a much larger design space. Without a systematic framework, each new architecture family will need to rediscover AS dynamics from scratch.

What Prior Work Established — and Where It Fell Short

The paper is built on an extensive body of prior work that it organizes, critiques, and synthesizes. Understanding where prior approaches succeed and where they leave gaps is essential to appreciating the survey's contribution.

Empirical Utilization (2023-present). The earliest systematic work on AS focused on practical exploitation. StreamingLLM [24] (Section 3.1) demonstrated that retaining the first few tokens in a sliding window KV cache prevents perplexity collapse, enabling infinite-length streaming generation without fine-tuning. H2O [172] generalized the insight to "heavy hitters" — tokens with high cumulative attention — and showed they serve as critical anchors for cache eviction. These methods established AS as an engineering phenomenon to be managed, but they treated AS as a given, asking how to preserve it rather than why it exists or whether it can be eliminated.

The limitation of this phase is clear in retrospect. Preservation-based methods implicitly assume AS tokens are static (always the initial token in causal LLMs), but works such as KVSink [28] and Active-Dormant Attention Heads [163] subsequently showed that sink positions can shift depending on layer depth and input context. A preservation strategy that forces fixed sink positions will underperform or fail entirely when sinks are dynamic. Moreover, the computational overhead of identifying sinks accurately at inference time remains an unsolved challenge (Section 3.1.3).

Mechanistic Interpretation (2024-present). As empirical utilization matured, the community began investigating the causes of AS through multiple complementary lenses. Quantizable Transformers [29] (Section 4.1) articulated the Softmax Limitations and No-Op Theory: the sum-to-one constraint forces attention mass onto uninformative tokens when a head needs to produce a near-zero output, because Softmax cannot output exact zeros without extreme logit values. This theory generated testable predictions — notably that sink tokens should exhibit suppressed value norms — which were validated across BERT, ViT, and LLaMA.

Massive Activations [98] (Section 4.2, 4.3) identified that AS tokens coincide with extreme activation outliers concentrated in sparse feature dimensions, and that the value updates from these tokens are nearly identical across query positions, functioning as an implicit attention bias added to every token's output. The Outlier Circuits framework (Section 4.2), refined by KVSink [28] and Systematic Outliers [82], mapped the causal chain: weight outliers in down-projection matrices → activation outliers in specific dimensions → attention concentration on sink tokens → reinforcement through training dynamics.

The limitation of this phase is that mechanistic explanations remained fragmented across architectural silos and interpretive frameworks. The Softmax Limitations theory explains why AS emerges but not how the specific numerical mechanisms (outlier circuits) implement it. The Outlier Circuits theory explains the numerical infrastructure but does not independently justify why the model needs to produce near-zero updates in the first place — that justification comes from the No-Op Theory. The Implicit Attention Bias perspective characterizes the functional role AS plays, but treats it as a fixed computational feature without explaining its geometric consequences or its relationship to the training dynamics that give rise to outlier circuits.

Individual papers typically advanced one interpretive theory without reconciling it with others. A researcher studying weight outliers in BERT [49, 50, 51] might not connect their findings to the Softmax-1 modification developed for LLaMA [162] or the geometric anchoring analysis performed on ViTs [46].

Strategic Mitigation (2025-present). Building on mechanistic understanding, the most recent work has targeted architectural interventions to eliminate AS at its root. Gated Attention Mechanisms (Section 5.1), pioneered by Quantizable Transformers [29] and scaled to production LLMs by [26, 44, 45], introduce a learnable gate after the attention output:

GatedAttention(x)=σ(G(x))⊙Softmax(QK⊤d)V\text{GatedAttention}(x) = \sigma(G(x)) \odot \text{Softmax}\left(\frac{QK^\top}{\sqrt{d}}\right) V

This decouples the no-op behavior from extreme Softmax logits — rather than forcing attention onto sink tokens to get a near-zero output, the head can learn to set the gate near zero, directly suppressing its contribution. Modified Softmax Functions (Section 5.2) take the complementary approach of fixing the normalization itself. Softmax-1 adds a constant 1 to the denominator, reducing first-token attention from 65% to 3.3% and activation kurtosis from 1657 to 3.1 [162]. Softpick replaces Softmax with a rectified non-normalized function, achieving a 0% sink rate [77].

The limitation of mitigation approaches is threefold. First, almost all require training from scratch — gate parameters, modified softmax functions, and learnable attention biases cannot be retrofitted into existing pretrained models without retraining, which is prohibitively expensive at scale. Second, there is no standardized evaluation for comparing mitigation techniques. Without consistent metrics for sink rate reduction, activation outlier suppression, quantization compatibility, and downstream task impact, it is impossible to determine which approach offers the best trade-off for a given deployment scenario. Third, the benefits and costs are not systematically characterized across difficulty regimes, architectures, or deployment contexts. A gated attention mechanism that eliminates AS completely but adds 5% latency overhead might be optimal for batch inference but unacceptable for interactive deployment — yet such trade-offs are rarely quantified.

How This Paper Positions Itself

The survey explicitly frames itself not as a contribution of a new method but as a meta-contribution that structures the field for future progress. Section 1.2 identifies four contributions:

  1. First systematic survey and taxonomy — organizing 180+ studies into the Utilization → Interpretation → Mitigation progression (Figure 3), with the temporal trend showing Utilization dominating 2023-2024 research, Interpretation deepening in 2024-2025, and Mitigation emerging as the frontier in 2025-2026.

  2. In-depth methodological synthesis — for each of the four utilization paradigms (Preservation, Redistribution, Learnable Prefix, Repurposing), four interpretation theories (Softmax Limitations, Outlier Circuits, Implicit Bias, Geometric Anchoring), and five mitigation strategies (Gated Attention, Modified Softmax, Learnable Bias, Pre-training Interventions, Other), the paper distills core mechanisms, practical implementations, and critical insights. This goes beyond a literature review — each subsection provides a portable conceptual framework that a researcher can apply to their own architecture.

  3. Critical insights and future directions — Section 7 identifies specific open challenges (computational overhead of dynamic sink detection, training-from-scratch requirements, incomplete understanding of training dynamics) and maps them to concrete research directions (lightweight post-hoc adaptation, theoretical formalization of outlier circuit emergence, standardized evaluation benchmarks).

  4. Scenario-driven application mapping — Section 6 maps AS techniques to nine application domains (pre-training, tuning, inference, interpretability, hallucination reduction, safety, general capability enhancement, long-context enhancement, multimodal enhancement), providing tangible guidance for practitioners who need to decide whether to preserve, redistribute, or mitigate AS in their specific deployment context.

The paper's distinctive intellectual move is to refuse to treat AS as either purely beneficial or purely harmful. Instead, it presents a nuanced picture: sink tokens stabilize attention under aggressive compression (making them useful for efficient inference), yet they consume representational capacity and complicate quantization (making them harmful for deployment). The survey's taxonomy is designed to help practitioners navigate this tension — when to preserve sinks, when to redistribute them, when to eliminate them entirely — based on their architectural constraints and application requirements.

The visual organization (Figures 1 and 2) reflects this ambition. The "Preliminaries on Attention Sink" section (Section 2.2) establishes a precise definition: AS is not merely "high attention on some token," but specifically the mismatch between disproportionately large attention mass and limited semantic contribution, formalized through a threshold-based detection criterion (Equation 5). This definitional precision is necessary because vague characterizations of AS have led to confusion — researchers might conflate genuinely informative attention patterns (e.g., attention to a question word in QA tasks) with uninformative sink concentration, leading to misguided interventions.

By structuring the survey as a cumulative developmental trajectory (Utilization → Interpretation → Mitigation) rather than a flat taxonomy, the paper makes a historical argument about the field's maturation: early work exploited AS as an empirical fact, mid-period work explained its causes, and the current frontier seeks to transcend it entirely. This narrative arc implicitly argues that the field is ready to move beyond ad-hoc sink management toward principled architectural solutions — but only if it first achieves the cross-community synthesis that this survey provides.

3. Technical Approach

3.1 Reader Orientation

This is a survey paper — it does not propose a new system or method, but rather constructs a conceptual framework for organizing, connecting, and evaluating over 180 existing studies on Attention Sink (AS) across diverse Transformer architectures. The "system" being built is a multi-dimensional taxonomy that enables researchers and practitioners to understand what AS is, why it emerges, how it can be exploited or mitigated, and which approaches are appropriate for which architectural contexts.

The paper solves the problem of fragmented knowledge: researchers in efficient LLM inference, mechanistic interpretability, vision transformers, and multimodal models have all encountered AS, but their findings, vocabulary, and solutions have developed in isolation. The survey's "shape" of solution is a three-axis classification — Fundamental Utilization (how AS is empirically used), Mechanistic Interpretation (why AS exists), and Strategic Mitigation (how AS can be eliminated) — that cross-references findings across architectures so that a practitioner encountering AS in any Transformer variant can locate relevant techniques, understand their mechanistic basis, and select appropriate interventions without needing to rediscover insights from adjacent communities.

3.2 Big-Picture Architecture (Diagram in Words)

The survey is organized as a hierarchical taxonomy with four layers, illustrated in Figures 1 and 2:

  1. Preliminary Layer (Section 2): Architecture-specific AS characterization. For each Transformer family — Classical Language Models, Causal LLMs, MoE LLMs, Multimodal LLMs, Vision Transformers, Diffusion Transformers, and emerging architectures — the survey characterizes what AS looks like (which tokens are sinks, how attention distributes across layers) and what research has been done specific to that architecture. This layer establishes that AS is not one phenomenon but a family of related behaviors with architecture-dependent manifestations.

  2. Utilization Layer (Section 3): Four paradigms for leveraging existing AS. This layer classifies methods that work with AS rather than eliminating it: Sink Token Preservation (retaining sink tokens during compression to stabilize attention), Attention Redistribution (reallocating attention mass from sinks to semantically meaningful tokens), Learnable Prefix Tokens (introducing trainable tokens to serve as explicit, controlled sinks), and Sink Token Repurposing (exploiting sink properties for tasks such as backdoor attacks, defense, or efficiency optimization).

  3. Interpretation Layer (Section 4): Four primary theories for why AS emerges. This layer synthesizes mechanistic explanations: Softmax Limitations and No-Op Theory (the sum-to-one constraint forces attention onto uninformative tokens when a head needs near-zero output), Outlier Circuits (weight, activation, and attention outliers form interconnected pathways that sustain AS), Implicit Attention Bias (AS functions as a fixed, input-independent bias term added to attention outputs), and Geometric Anchoring (sink tokens serve as stable reference points structuring the representation space).

  4. Mitigation Layer (Section 5): Five strategies for eliminating AS. This layer surveys architectural interventions that remove the need for AS: Gated Attention Mechanisms (learnable gates that suppress attention outputs directly), Modified Softmax Functions (altering or replacing Softmax to remove the sum-to-one constraint), Learnable Attention Bias (explicit bias terms that absorb attention without sink tokens), Pre-training Interventions (optimizer, loss, or normalization changes that prevent outlier formation), and other techniques including Outlier-Driven Rescaling and Architectural Isolation.

A cross-cutting Applications Layer (Section 6) maps all techniques to nine deployment scenarios (pre-training, tuning, inference, interpretability, hallucination reduction, safety, capability enhancement, long-context extension, multimodal enhancement), providing practical decision guidance.

Information flows through the taxonomy as follows: a practitioner identifies their architecture (Section 2) → determines their goal (exploit AS or eliminate it) → if exploiting, selects a utilization paradigm (Section 3) informed by mechanistic understanding (Section 4) → if eliminating, selects a mitigation strategy (Section 5) targeting the root cause identified by interpretation → validates against application requirements (Section 6).

3.3 Roadmap for the Deep Dive

The detailed technical breakdown follows a causal chain rather than the survey's section order, because understanding what methods do requires first understanding why AS exists:

  • First, the formal definition of AS (Section 2.2.2): The mathematical criterion that distinguishes genuine sink tokens from mere high-attention tokens, establishing the threshold-based detection framework used throughout the literature. This definition is foundational — without it, the utilization, interpretation, and mitigation sections lack a precise target.

  • Second, the Softmax Limitations and No-Op Theory (Section 4.1): The mathematical origin of AS, explaining why the Softmax sum-to-one constraint makes sink tokens inevitable in standard attention. This theory provides the mechanistic justification for why utilization techniques work (they exploit a necessary behavior) and why mitigation techniques target Softmax or attention gating (they address the root cause).

  • Third, the Outlier Circuits framework (Section 4.2): The numerical mechanism that implements the No-Op behavior, tracing how weight outliers in down-projection matrices amplify activation outliers, which induce attention concentration on sink tokens, which in turn reinforce the pattern. This explains how AS manifests numerically, connecting the abstract Softmax constraint to concrete neuron-level phenomena.

  • Fourth, the utilization paradigms (Sections 3.1–3.4): With the mechanistic basis established, we examine how practitioners exploit AS — preserving sink tokens during KV cache compression, redistributing their attention mass to semantic targets, introducing learnable prefixes as explicit sinks, and repurposing sink properties for security or efficiency tasks. Each paradigm is evaluated against what we now know about why sinks exist.

  • Fifth, the mitigation strategies (Sections 5.1–5.5): Building on mechanistic understanding, we survey architectural interventions that eliminate AS — gated attention, modified softmax functions, learnable biases, pre-training interventions, and other approaches — evaluating which root cause each addresses and what trade-offs it entails.

This order — definition → origin → mechanism → exploitation → elimination — builds understanding cumulatively. The utilization techniques make sense only once we understand why preserving a seemingly uninformative token prevents catastrophic perplexity collapse. The mitigation strategies' design choices (why gate after attention, why add 1 to the Softmax denominator specifically) are opaque without first understanding the No-Op theory and the outlier circuit dynamics.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a survey and taxonomy paper whose core idea is that Attention Sink research across architectures, utilization strategies, interpretive theories, and mitigation techniques can be organized into a unified three-axis framework (Utilization → Interpretation → Mitigation) that reveals complementarities, contradictions, and gaps invisible when each community's work is examined in isolation.

3.4.1 Formal Definition of Attention Sink

The paper establishes its target phenomenon with unusual precision in Section 2.2.2, recognizing that vague definitions have caused confusion across the literature. Not every high-attention token is an attention sink — the defining property is the mismatch between disproportionately large attention mass and limited semantic contribution.

The threshold-based detection criterion. The paper formalizes sink identification through Equation 5, which operates on the cumulative attention received by each token across all queries in a sequence:

SAS={j  |  ∑i=1LAi,j>τ⋅μA},μA=1L∑k=1LA^kS_{\text{AS}} = \left\{ j \;\middle|\; \sum_{i=1}^{L} A_{i,j} > \tau \cdot \mu_A \right\}, \quad \mu_A = \frac{1}{L} \sum_{k=1}^{L} \hat{A}_k

where $A \in \mathbb{R}^{L \times L}$ is the attention weight matrix for a sequence of length $L$, $A_{i,j}$ is the attention weight from query token $i$ to key token $j$, $\hat{A}_j = \sum_{i=1}^L A_{i,j}$ is the cumulative attention score received by token $j$ across all queries, $\mu_A$ is the mean cumulative attention across all $L$ tokens, and $\tau > 1$ is a relaxation threshold empirically set to a large value (e.g., 1000 in [82]).

What it computes: For each token position $j$ in the sequence, the criterion sums the attention it receives from every query position $i$ (restricted by causal masking in autoregressive models). It then compares this per-token sum $\hat{A}_j$ against a multiple $\tau$ of the global mean $\mu_A$ across all tokens. Tokens whose cumulative attention exceeds this threshold are classified as sink tokens. The high $\tau$ value (1000× the mean) ensures that only tokens with genuinely extreme attention concentration are labeled — this is not a statistical outlier detection (2–3 standard deviations) but a criterion for orders-of-magnitude deviation from typical attention allocation.

Why this form: The cumulative attention formulation captures the core observation that sink tokens receive high attention from nearly all queries, not just a few. A token that receives high attention from one query but negligible attention from others would have a modest sum $\hat{A}_j$ and fail the threshold. The multiplicative threshold $\tau \cdot \mu_A$ adapts to sequence length — as $L$ grows, the mean $\mu_A$ changes, but the threshold tracks it proportionally. This is superior to an absolute threshold because it automatically calibrates to different sequence lengths without manual tuning. The authors cite [82] as establishing this specific formulation, noting that the large $\tau$ value empirically separates sink tokens (which accumulate on the order of hundreds or thousands of times the mean attention) from merely "important" tokens (which might accumulate 2–5× the mean).

The two defining characteristics. Beyond the mathematical criterion, the paper emphasizes (Section 2.2.2) that AS tokens exhibit two highly consistent properties across architectures:

  1. Extremely high attention scores: In LLaMA and similar LLMs, the first token "frequently receives the maximum attention in 98% of attention heads" [162]. This is not a subtle effect — it is near-total dominance of the attention distribution in most heads.

  2. Intrinsically low-information content: AS tokens correspond to tokens with minimal semantic or task-specific contribution, but the specific token type varies by architecture. The paper provides an empirical taxonomy (Section 2.2.2):

    • Classical Language Models: [SEP] and [CLS] tokens [29, 53]
    • Causal LLMs (dense and MoE): Initial tokens, strong delimiters (periods, newlines), and weak-semantic tokens [43, 98]
    • Vision Transformers: Low-information background patches [98, 126]
    • Multimodal LLMs: Both text-side AS (e.g., [BOS]) inherited from causal LLMs, and vision-side AS on low-information visual patches [105]

This architectural specificity is critical for practitioners: a method that assumes sink tokens always occupy position 0 (the initial token) will work for standard causal LLMs but fail on ViTs where sinks concentrate on background patches that vary spatially across images, or on encoder-only models where [SEP] tokens at sequence midpoints act as sinks.

What the definition excludes. The paper is careful to distinguish AS from merely "attending to important tokens." In a question-answering task, attention to the question word "What" might be high because it genuinely carries semantic information. That is not AS because the token's semantic contribution matches its attention mass. The defining feature of AS is the mismatch — tokens that receive massive attention despite contributing near-zero information, as evidenced by their suppressed value norms (Section 4.1.2).

3.4.2 Softmax Limitations and No-Op Theory

This theory, first articulated in Quantizable Transformers [29] (NeurIPS 2023) and subsequently refined by multiple works surveyed in Section 4.1, provides the mathematical origin story for AS. It is the most fundamental of the four interpretation perspectives because it explains why AS is inevitable under the standard attention formulation, independent of training data, optimizer choice, or model scale.

The core problem: Softmax cannot say "none of the above." The standard attention computation for a query vector $q_i$ is:

Attention(qi,K,V)=∑jsoftmax(qikj⊤d)j⋅vj\text{Attention}(q_i, K, V) = \sum_{j} \text{softmax}\left(\frac{q_i k_j^\top}{\sqrt{d}}\right)_j \cdot v_j

where $q_i \in \mathbb{R}^d$ is the query vector at position $i$, $k_j, v_j \in \mathbb{R}^d$ are the key and value vectors at each position $j$, and the softmax ensures $\sum_j \text{softmax}(\cdot)_j = 1$.

What it computes: For each query position $i$, the model computes a vector of unnormalized compatibility scores between $q_i$ and every key $k_j$ (the pre-Softmax logits), applies the Softmax function to convert these scores into a probability distribution summing to 1, and uses these normalized weights to compute a weighted sum of the value vectors $v_j$. The output at position $i$ is thus a convex combination of all value vectors visible to that query (constrained by causal masking in autoregressive models).

Why this is a problem: The sum-to-one constraint means that for every query, the attention head must allocate 100% of its attention mass somewhere, even when no key in the context is semantically relevant to the query. The mechanism lacks a natural "null" or "abstain" option. The paper articulates this through Equation 39, which shows that achieving a near-zero attention weight on any non-sink token requires logits to be driven to extreme values:

Softmax(x)i=0  ⟺  ∃j≠i, xj−xi=+∞\text{Softmax}(x)_i = 0 \iff \exists j \neq i, \ x_j - x_i = +\infty

where $x_j = q_i k_j^\top / \sqrt{d}$ are the pre-Softmax logits.

What this equation reveals: For an attention head to effectively ignore all tokens (i.e., produce near-zero attention weights everywhere except on one designated token), it must push the logit of that designated token infinitely higher than all others. In finite-precision arithmetic, this manifests as logits reaching extreme magnitudes (hundreds or thousands), which in turn produce activation outliers — values orders of magnitude larger than the typical activation range — that propagate through subsequent layers and cause numerical instability.

The No-Op solution: The theory posits that attention heads adapt to this constraint by learning a no-op behavior — concentrating attention on a fixed set of sink tokens $S$ (e.g., [SEP] in BERT, initial tokens in LLMs, background patches in ViTs) whose value vectors $V_S$ have learned to be negligible:

Aij≈{1,j∈S0,otherwisewith∥VS∥≈0A_{ij} \approx \begin{cases} 1, & j \in S \\ 0, & \text{otherwise} \end{cases} \quad \text{with} \quad \|V_S\| \approx 0

What this accomplishes: When a head needs to make no meaningful contribution to the residual stream for certain tokens, it routes its full attention mass (≈1) to sink tokens whose value vectors are near-zero, resulting in an attention output $\sum_j A_{ij} v_j \approx 0$. This achieves the desired null update without violating the Softmax sum-to-one constraint. Critically, the model learns which tokens serve as sinks and that their value vectors should be suppressed — this is not hardcoded but emerges during training.

The mutual reinforcement cycle. The paper identifies (via [44] and [28]) that this behavior is self-reinforcing: high attention on sink tokens further suppresses their value states during training (because gradients through near-zero values are small), which in turn makes them even more attractive as attention dumps (since they reliably produce near-zero outputs), which further entrenches the pattern. This explains why AS, once formed during pre-training, is difficult to eliminate without architectural intervention — it is not a transient numerical artifact but a stable attractor state of the training dynamics.

Why this theory is foundational: It explains three otherwise-disparate observations with a single causal mechanism: (1) why sink tokens receive disproportionate attention (they are the designated targets for redundant attention mass), (2) why sink tokens exhibit suppressed value norms (they need to produce minimal residual updates to function as no-ops), and (3) why activation outliers co-occur with AS (the extreme logits required to concentrate attention on a single token produce massive activations). Without this theory, these observations appear as independent quirks; with it, they are necessary consequences of the Softmax constraint interacting with the attention head's need for a null operation.

3.4.3 Outlier Circuits

While the No-Op theory explains why AS must exist, the Outlier Circuits framework (Section 4.2) explains how AS is implemented numerically — the specific chain of neuron-level causal relationships that convert the abstract Softmax constraint into concrete attention patterns and activation magnitudes.

The three types of systematic outliers. Building on Systematic Outliers [82] (ICLR 2025), the paper categorizes outliers into three interdependent types (Section 4.2.1):

  1. Weight Outliers: Exceptionally large values concentrated in specific columns of the down-projection matrices $W_{\text{down}}^\ell$ in MLP layers. In LLaMA2-7B, these are observed in the second layer and the last two layers. They are also referred to as "Super Weights" [189].

  2. Activation Outliers: Abnormally large activations in hidden states, further divided into two subtypes. Both are confined to specific feature dimensions and exhibit minimal variation across different inputs:

    • Down-Projection Input Outliers ($x_{\text{down}}^\ell$): Large activations at the input to the down-projection MLP layer, localized to a limited number of shallow and deep layers. Also known as "Activation Spikes" [165].
    • Layer Output Outliers ($h^\ell$): Large activations in the residual stream output, persisting across layers but diminishing in the final layers. Also referred to as "Massive Activations" [98].

  3. Attention Outliers: Certain keys receive disproportionately high cumulative attention scores — this is precisely the AS phenomenon. These outliers persist across nearly all layers.

The causal chain. Section 4.2.1 and Figure 27 articulate a four-step causal sequence that transforms weight structure into attention behavior:

  • Step 1: Weight outliers in up-projection and gate-projection matrices induce large neuron activations in early layers. Specifically, large values in $W_{\text{up}}$ and $W_{\text{gate}}$ cause certain feature dimensions to produce outputs far exceeding the typical activation range. These constitute the first type of activation outliers — the down-projection input outliers $x_{\text{down}}^\ell$.

  • Step 2: Weight outliers in the down-projection matrix amplify these activations along specific feature dimensions. Columns of $W_{\text{down}}^\ell$ with anomalously large values multiply the already-large $x_{\text{down}}^\ell$ values, producing outputs that are several orders of magnitude beyond the mean. These amplified values propagate through residual connections to become the layer output outliers $h^\ell$.

  • Step 3: Activation outliers induce attention concentration on corresponding tokens. Tokens exhibiting activation outliers show strong alignment in particular dimensions of their query and key vectors — the key vectors for these tokens have unusually large norms along the outlier feature dimensions, causing their dot products $q_i k_j^\top$ with queries to dominate the pre-Softmax logits. The Softmax then assigns disproportionately high attention weights to these tokens, forming AS.

  • Step 4: The value vectors of sink tokens remain small, completing the No-Op. Despite receiving most of the attention mass, the value vectors $v_j$ for sink tokens do not exhibit corresponding outlier magnitudes — they remain near their typical range. This means the attention output $\sum_j A_{ij} v_j$ is dominated by near-zero contributions, achieving the no-op behavior. The Outlier Circuits perspective identifies this value-norm suppression not as a separate mechanism but as the downstream consequence of the first three steps: the same weight structure that amplifies activations along specific dimensions does not amplify value vectors along those same dimensions, creating an asymmetry that routes attention to tokens whose value contributions are negligible.

The cross-layer lifecycle. KVSink [28] (COLM 2025) documents that this circuit follows a predictable lifecycle across layers (Figure 29), characterized by four phases:

  • Initial (Layer 0): No significant outliers. Attention is predominantly local — each query attends to its immediate neighbors — because the model has not yet developed the weight structure that amplifies specific feature dimensions.

  • Emergence (Layer 1): Weight outliers in the down-projection matrix of layer 1 produce the first activation outliers in $x_{\text{down}}^2$. These outliers are modest in magnitude but initiate the cascade.

  • Stabilization (Layers 2–29 in LLaMA2-7B): The outlier cascade is fully established. Activation outliers propagate through residual connections, attention outliers (AS) form and persist, and the circuit reaches a stable equilibrium where weight, activation, and attention outliers mutually reinforce one another. This phase spans the vast majority of the model's depth.

  • Dissipation (Layer 30) and Final (Layer 31): Activation outliers gradually diminish as the model approaches the output. The final layers show attenuated outlier magnitudes, and attention patterns become more distributed. The paper interprets this as the model "unpacking" the compressed representations maintained during the stabilization phase for final token prediction.

Why this temporal structure matters: It demonstrates that AS is not a static property of individual layers but an emergent dynamical system that spans nearly the entire model depth. Interventions targeting AS cannot treat each layer independently — modifying the weight structure in layer 1 will suppress the outlier cascade throughout the entire stabilization phase, while modifying layers in the dissipation phase may have limited impact because the circuit has already mostly run its course. This explains why mitigation techniques such as Gated Attention (Section 5.1) and Pre-training Interventions (Section 5.4) that operate uniformly across all layers are effective — they intercept the circuit at its earliest stages.

Architecture-specific variations. The survey documents how the Outlier Circuit manifests differently across architectures, demonstrating the framework's generality:

  • MoE LLMs (Section 2.3.3): The circuit concentrates within Super Experts [43] — a tiny subset of experts (as few as 3 out of 6,144 in Qwen3-30B-A3B) whose down-projection outputs exhibit extreme activation outliers. Sink tokens consistently achieve high router scores on these Super Experts, ensuring that AS is primarily activated within them. Pruning just these 3 experts causes catastrophic AS collapse and performance degradation, confirming their gatekeeping role.

  • MLLMs (Section 2.3.4): Outlier-driven AS emerges on both text-side initial tokens (inherited from the causal LLM backbone) and visual background patches (introduced through cross-modal fusion). See What You Are Told [105] (ICLR 2025) demonstrates that visual AS can be precisely identified by detecting Massive Activations in the vision encoder's hidden states, establishing a direct link between outlier magnitudes and sink token identification in multimodal contexts.

  • Audio-Visual Speech Recognition (Section 4.2.2): The outlier-AS coupling extends beyond text and vision. Intermediate low-semantic tokens (not just [BOS]) exhibit massive activations that correspond to fixed feature indices, and these activations originate from MLP layers, confirming the cross-modal generality of the weight → activation → attention causal chain.

Causal evidence. The paper emphasizes (Section 4.2.2) that the Outlier Circuits framework is supported by intervention studies, not just correlational observations. The strongest causal evidence comes from Unveiling Super Experts [43] (ICLR 2026): pruning the 3 Super Experts that concentrate extreme activation outliers in Qwen3-30B-A3B triggers a catastrophic collapse of AS and leads to repetitive, uninformative outputs. This is the interventional equivalent of removing a component from a circuit and observing that the downstream behavior ceases — the causal role of Super Experts in sustaining AS is directly established.

3.4.4 Implicit Attention Bias

While the Outlier Circuits framework explains how AS is implemented numerically, the Implicit Attention Bias perspective (Section 4.3) characterizes what functional role AS plays in the attention computation — what the model gains from concentrating attention on uninformative tokens.

The core observation: Value updates from sink tokens are nearly identical across queries. Section 4.3.1, drawing on Massive Activations [98] (COLM 2024), decomposes the attention output for a query token $k$:

Attention(Q,K,V)k=∑i∈Cpikvi+∑i∉Cpikvi\text{Attention}(Q, K, V)_k = \sum_{i \in C} p^k_i v_i + \sum_{i \notin C} p^k_i v_i

where $p^k_i$ is the attention weight from query $k$ to token $i$, $v_i$ is the value vector of token $i$, and $C$ is the set of tokens that exhibit Massive Activations (i.e., the AS tokens).

What this decomposition reveals: When the authors compute $\sum_{i \in C} p^k_i v_i$ — the total contribution of sink tokens to the attention output — for different query positions $k$ and different input sequences, they find that it is "nearly identical across all query positions and across different inputs" [98]. This means that regardless of what a query is attending to in the semantically meaningful part of the sequence, it receives essentially the same constant vector contribution from the sink tokens.

The functional interpretation: This constant vector functions mathematically as a bias term — a fixed offset added to every token's attention output, independent of the input content and the token's position. KVSink [28] provides quantitative evidence (Figure 31): the average cosine similarity of $\sum_{i \in S} p^t_i v_i$ across all tokens $t$ for each attention head is close to 1, indicating near-perfect constancy.

Why this is functionally useful: In standard attention, the output is a weighted sum of value vectors where the weights are content-dependent (computed via query-key compatibility). However, there may be aspects of the representation that are independent of the current context — e.g., a baseline offset that all tokens should receive regardless of their specific content. Without an explicit bias parameter in the attention computation (unlike in convolutional or linear layers, which include learned bias vectors), the model must manufacture this constant offset using the available mechanism — by concentrating attention on tokens whose value vectors provide the desired constant contribution. AS is thus an emergent solution to the absence of explicit bias terms in the standard attention formulation.

Causal validation through replacement. The paper documents (Section 4.3.2, 5.3) that providing an explicit attention bias eliminates the need for AS, providing strong causal evidence for the interpretation. Massive Activations [98] augments attention with learnable key and value biases $k', v' \in \mathbb{R}^d$:

Attention(Q,K,V;k′,v′)=softmax(Q[K⊤ k′]d)[Vv′⊤]\text{Attention}(Q, K, V; k', v') = \text{softmax}\left(\frac{Q[K^\top \ k']}{\sqrt{d}}\right) \begin{bmatrix} V \\ v'^\top \end{bmatrix}

When a GPT-2 model is trained with these explicit biases from scratch, Massive Activations disappear entirely, and the AS phenomenon is "correspondingly eliminated." This confirms that AS is a manifestation of an implicit bias learned to cope with the Softmax constraint — provide an explicit alternative, and the model no longer needs to manufacture one through extreme attention concentration.

The relationship to No-Op Theory: The Implicit Attention Bias perspective complements rather than contradicts the No-Op Theory. The No-Op Theory explains when AS is used (when a head needs near-zero output), while the Implicit Bias perspective explains what the non-zero residual contributes (a constant offset). A head that needs to produce output $b + \epsilon$, where $b$ is a fixed bias and $\epsilon$ is a content-dependent adjustment, can route its content-dependent computation through semantically relevant tokens and obtain $\epsilon$, while obtaining $b$ from the sink tokens' nearly-constant value contribution. This division of labor — content-dependent processing via semantic tokens, constant offset via sink tokens — allows the head to efficiently represent both aspects of its output without requiring separate parameters.

3.4.5 Geometric Anchoring

While the Implicit Attention Bias perspective characterizes AS at the level of attention outputs, the Geometric Anchoring framework (Section 4.4) examines AS in the representation space — the geometry of hidden states and how sink tokens serve as stable reference points that structure the representational manifold.

Core concept: Sink tokens as coordinate-system origins. Section 4.4.1 presents several formalizations that converge on a common insight: sink tokens occupy a privileged position in the high-dimensional embedding space, and other tokens' representations are systematically organized relative to them.

Positional Vector Decomposition [167] (NeurIPS 2024): Each hidden state at layer $l$ for token position $t$ is decomposed as:

hl,ts=pl,t+cl,tsh^s_{l,t} = p_{l,t} + c^s_{l,t}

where $p_{l,t}$ is the positional vector (encoding the position $t$ at layer $l$, independent of the token's semantic content) and $c^s_{l,t}$ is the semantic content vector (encoding the token's meaning for input sequence $s$).

What this decomposition reveals: Using mean-based decomposition followed by PCA visualization, the study finds that after the first layer, only the initial tokens (e.g., positions 1–4) exhibit distinct positional vectors, whereas later tokens converge to similar positional representations. The sink token's positional vector $p_{l,1}$ acts as a geometric anchor that "guides the formation of positional vectors for subsequent tokens, thereby inducing AS." When the input sequence length exceeds the model's training window, the positional vectors become out-of-distribution, causing AS to vanish and perplexity to spike sharply — confirming the interdependence between positional geometry and sink behavior.

OrthoRank [70] (ICML 2025): Token importance is evaluated based on orthogonality relative to the sink token:

importance(t)∝1−∣cos⁡(ht,hs)∣\text{importance}(t) \propto 1 - |\cos(h_t, h_s)|

where $\cos(h_t, h_s)$ is the cosine similarity between the hidden state of token $t$ and the sink token $s$.

What this metric computes: Tokens whose hidden states are nearly orthogonal (cosine similarity ≈ 0) to the sink token's hidden state are considered more informative, because they have diverged from the neutral anchor position to encode task-specific semantic content. Tokens nearly parallel to the sink (cosine similarity ≈ 1) have remained close to the anchor and are less informative. Empirically, this orthogonality-based importance metric outperforms attention-score-based metrics for KV cache eviction, confirming that geometric deviation from the sink anchor is a more reliable signal of information content than the attention scores themselves.

KeyDiff [78] (NeurIPS 2025): Sink tokens exhibit a distinctive geometric property in the key space:

cos⁡(ks,kˉ)≈0\cos(k_s, \bar{k}) \approx 0

where $k_s$ is the key vector of the sink token and $\bar{k}$ is the mean key vector across all tokens.

What this reveals: Sink tokens are geometric outliers in the key space — their key vectors point in directions almost orthogonal to the average key direction. This property is exploited for efficient KV cache management: tokens with near-zero cosine similarity to the mean key are identifiable as critical anchors without computing full attention matrices. The average Spearman correlation between this geometric distinctiveness and attention scores is approximately 0.94 across layers and heads, confirming that geometric outlier status predicts attention concentration.

The functional role: These geometric characterizations converge on a consistent picture: sink tokens serve as a stable coordinate origin in the representation space. Without them, the representational manifold would lack a fixed reference point — every token's position would be defined only relative to other content-dependent tokens, causing the manifold to drift with input content and making it difficult for subsequent layers to extract consistent features. The sink token provides an input-independent anchor that stabilizes the geometry, enabling other tokens to encode their semantic content as deviations from this fixed reference rather than as absolute positions in a drifting space.

Relationship to Anti-Overmixing Theory (Section 4.5): The paper presents Anti-Overmixing as a distinct interpretation, but it can be understood as the functional consequence of geometric anchoring. The Anti-Overmixing theory [27] (COLM 2025) posits that the first token's role is to prevent "excessive information mixing across layers" — without a sink, token representations would quickly converge to identical representations, causing representational collapse. This is precisely what geometric anchoring formalizes: the sink token provides a stable anchor that maintains representational diversity by serving as a fixed reference point that other tokens can diverge from without converging to each other.

3.4.6 Sink Token Preservation (Utilization Paradigm 1)

With the mechanistic basis established, the paper surveys four paradigms for exploiting AS in existing models. Sink Token Preservation (Section 3.1) is the most widely adopted — the insight that permanently retaining sink tokens in the KV cache prevents catastrophic performance collapse under aggressive compression.

Core formulation (Section 3.1.1). For a sequence with token indices $\{1, \ldots, t\}$ up to generation step $t$, the method ensures that a fixed set of sink indices $I_{\text{sink}} \subseteq \{1, \ldots, k\}$ is always available to every query:

Attn(qi,KJi,VJi)=softmax(qiKJi⊤d)VJi\text{Attn}(q_i, K_{J_i}, V_{J_i}) = \text{softmax}\left(\frac{q_i K_{J_i}^\top}{\sqrt{d}}\right) V_{J_i}

where $J_i \supseteq I_{\text{sink}}$ denotes the set of token indices available to query $i$, constrained by causality such that $J_i \subseteq \{1, \ldots, i\}$, and $I_{\text{sink}}$ is the set of indices permanently retained in the KV cache regardless of eviction policy.

What it accomplishes: By guaranteeing that every query can attend to the sink tokens, this formulation preserves the model's ability to route redundant attention mass to its learned no-op targets. If sink tokens were evicted, the attention head would be forced to concentrate its otherwise-unnecessary attention mass onto semantically meaningful tokens, distorting their representations. StreamingLLM [24] (ICLR 2024) provides the canonical demonstration: using sliding window attention without sink preservation, perplexity on long texts explodes from 5.43 (with sinks preserved) to 5,158 (with sinks evicted). The mechanism is that the model's attention heads have been trained to expect the initial tokens as available attention targets; removing them creates a mismatch between training and inference distributions.

Practical instantiations (Section 3.1.2). The paper categorizes preservation methods by their application context:

  • StreamingLLM [24]: The simplest form — retain the first $S$ tokens alongside the most recent $W$ tokens. This enables infinite-length streaming generation without fine-tuning, requiring no architectural changes and zero additional parameters.
  • H2O [172] (NeurIPS 2023): Generalizes the fixed-position heuristic to a data-dependent criterion — retain tokens with high cumulative attention scores (termed "heavy hitters"), which naturally identifies sink tokens regardless of their position. The KV cache budget $K$ is allocated by solving $\hat{C}_t = \{(k_i, v_i) : i \in \arg\max_{|I| \leq K} \sum_{i \in I} a_i\}$, where $a_i$ is the cumulative attention score for token $i$. This automatically adapts to architectures where sinks are not at position 0.
  • MInference [182] (NeurIPS 2024): For sparse attention, constructs a binary mask $M_t$ per head that enforces sink token visibility for all queries while sparsifying the remaining context. This accelerates pre-filling by up to 10× without accuracy loss by recognizing that attention patterns are highly structured — some heads are "vertical-slash" heads attending primarily to sink-like tokens, and enforcing their visibility while sparsifying other connections preserves model behavior.
  • KV cache quantization methods (KVSink [28], IntactKV [151], RotateKV [84]): Preserve sink tokens at full precision while aggressively quantizing other tokens to 2-bit or 4-bit. This leverages the observation (Section 4.1.2) that sink tokens' suppressed value norms make them disproportionately sensitive to quantization error — small absolute perturbations in near-zero values constitute large relative errors. By protecting the precise numerical values of sink tokens, these methods enable aggressive compression of the remaining cache with minimal degradation.

Why this paradigm works (connecting back to mechanistic understanding): Preservation methods succeed because they respect the No-Op mechanism — by keeping sink tokens available as attention targets, they allow attention heads to continue executing their learned no-op behavior. The success of these methods is itself evidence for the No-Op theory: if AS were merely an artifact of training dynamics with no functional role, removing sink tokens would cause minor degradation, not catastrophic collapse. The fact that evicting sink tokens causes perplexity to spike by three orders of magnitude demonstrates that the model structurally depends on these tokens as attention targets — exactly as the No-Op theory predicts.

Limitations (Section 3.1.3): The paper is careful to identify when preservation fails. Current methods largely assume static sink positions (typically the initial token in causal LLMs), but KVSink [28] and Active-Dormant Attention Heads [163] show that sinks can emerge at non-initial positions depending on input content and layer depth. Fixed-position preservation will fail when sinks shift — a method that always retains token 0 will waste capacity if the true sink for a particular head is at position 7. Dynamic identification methods exist but incur computational overhead and may conflict with optimized attention kernels such as FlashAttention that make assumptions about memory access patterns.

3.4.7 Attention Redistribution (Utilization Paradigm 2)

Where preservation passively retains sink tokens as anchors, Attention Redistribution (Section 3.2) actively reallocates their disproportionate attention mass to semantically meaningful targets. This paradigm is motivated by the observation that sink tokens consume representational capacity — attention mass that could be productively deployed on task-relevant tokens is instead wasted on uninformative anchors.

Core formulation (Section 3.2.1). The paper abstracts explicit redistribution methods into a unified framework:

A~ij={α⋅Aij,j∈SAij+β⋅1∣Ti∣∑s∈SAis,j∈TiAij,otherwise\tilde{A}_{ij} = \begin{cases} \alpha \cdot A_{ij}, & j \in S \\ A_{ij} + \beta \cdot \frac{1}{|T_i|} \sum_{s \in S} A_{is}, & j \in T_i \\ A_{ij}, & \text{otherwise} \end{cases}

where $S \subseteq \{1, \ldots, t\}$ is the set of sink token indices, $T_i \subseteq \{1, \ldots, t\} \setminus S$ is the set of target token indices for query $i$ (non-sink tokens intended to receive redistributed attention), $A_{ij} = \text{softmax}(q_i k_j^\top / \sqrt{d})$ is the original attention score, $\alpha \in [0,1]$ controls the retention of sink attention (how much attention mass sinks continue to receive), and $\beta \in [0,1]$ specifies the proportion of the freed mass that is redistributed to target tokens.

What it computes: For each attention head and query position, the method identifies which key positions $j$ are sink tokens ($j \in S$) and which are target tokens that should receive additional attention ($j \in T_i$). It reduces the attention weight on sink tokens by a factor $\alpha$ (e.g., $\alpha = 0$ means complete elimination of sink attention), computes the total freed attention mass $\sum_{s \in S} A_{is}$, and redistributes a fraction $\beta$ of this mass equally among the target tokens. The constraint $\alpha + \beta = 1$ ensures that the total attention mass per query remains 1, preserving the Softmax sum-to-one property but changing where the mass is allocated.

Why this form: The uniform redistribution across target tokens is a minimal-assumption default — without further information about which target tokens are most relevant to query $i$, equal distribution maximizes entropy and minimizes the risk of distorting the attention distribution in unintended ways. More sophisticated methods (e.g., ACT [154], ICML 2025) make $\alpha$, $\beta$, and the target set $T_i$ input-dependent, learning to redistribute attention proportionally to the query's estimated relevance to each target.

Two families of redistribution (Section 3.2.1–3.2.2):

  1. Explicit Redistribution uses predefined parameters $\alpha$, $\beta$, and $T_i$. The extreme case is full redistribution ($\alpha = 0, \beta = 1$), which completely eliminates sink attention and transfers the entire freed mass to target tokens. VAR [105] (ICLR 2025) applies full redistribution to MLLMs: visual background patches identified as sinks via Massive Activation detection have their attention mass transferred to foreground object patches, enhancing visual grounding and reducing hallucination. AttnReal [31] (IJCV 2026) recycles attention from output tokens to visual tokens in MLLMs. GasEraser [30] suppresses misleading text tokens acting as sinks and reallocates attention to visual regions.

  2. Attention-Sink-Aware Calibration dynamically determines $\alpha$, $\beta$, and $T_i$ based on the input context. ACT [154] identifies harmful AS (including non-initial sinks) and calibrates attention per-input without retraining. ZeroTuning [75] (ICLR 2026) treats the initial token as a controllable lever: by adding a learnable bias $b$ to the initial token's pre-Softmax logit, the entire attention distribution can be modulated indirectly via the zero-sum property of Softmax — a negative bias $b$ suppresses the sink, freeing attention mass that naturally redistributes to remaining tokens.

The connection to mechanistic interpretation: Redistribution methods implicitly accept the No-Op Theory's premise — attention mass on sink tokens is indeed wasted — but reject the conclusion that it is inevitable. By surgically reallocating the wasted mass, they improve model behavior without architectural modification. However, the paper notes (Section 3.2.3) that this approach has a fundamental limitation: it operates on attention scores after Softmax, potentially conflicting with optimized kernels, and it requires accurate identification of sink and target tokens, which reintroduces the dynamic detection challenge that preservation methods also face.

3.4.8 Learnable Prefix Tokens (Utilization Paradigm 3)

Rather than passively preserving natural sinks or redistributing their attention, Learnable Prefix Tokens (Section 3.3) introduce trainable parameters that serve as explicit, controlled attention sinks, optimized via gradient descent to absorb excess attention mass in a predictable manner.

Core formulation (Section 3.3.1). Given an original input sequence $X = \{x_1, \ldots, x_N\}$ with $x_i \in \mathbb{R}^D$, the method inserts a set of $K$ learnable tokens $P = \{p_1, \ldots, p_K\}$ at the sequence start:

S=[P;X]∈R(K+N)×DS = [P; X] \in \mathbb{R}^{(K+N) \times D}

where $p_i \in \mathbb{R}^D$ are trainable parameters optimized during pre-training or fine-tuning, $K$ is the number of prefix tokens, and $[P; X]$ denotes concatenation.

What it accomplishes: Every token in the sequence can attend to these prefix tokens (they are at the start, visible to all queries in autoregressive models, or fully visible in bidirectional models). During training, the model learns to route globally shared or redundant attention mass toward these tokens, making them function as dedicated computational sinks. At inference, $P$ remains fixed — the tokens do not change with input content — providing stable, predictable sink behavior that does not depend on identifying natural sinks or worrying about their dynamic positions.

Key property that distinguishes this from natural sinks: Unlike the initial token in a causal LLM, which happens to become a sink due to its structural position and the emergence of outlier circuits, learnable prefix tokens are designed to be sinks. Their parameters are optimized specifically to attract and absorb excess attention mass, and they can be placed at known, fixed positions that simplify kernel optimization and KV cache management.

Practical instantiations across architectures (Section 3.3.2):

  • Vision Transformers Need Registers [126] (ICLR 2024): Adds "register tokens" to ViT inputs. In standard ViTs, natural AS emerge on background patches, causing noisy attention maps (Figure 19, left). Register tokens absorb this excess attention, producing cleaner attention maps focused on foreground objects (Figure 19, right). The tokens are trained during standard pre-training and add minimal parameter overhead (typically 4 register tokens per image, totaling 4D additional parameters).

  • StreamingLLM [24] (ICLR 2024): Introduces a learnable placeholder token $p$ during pre-training that remains permanently in the KV cache, unlike natural initial tokens that could be evicted in sliding-window architectures. This provides a guaranteed sink target for infinite-length generation that persists even as the window slides past the original initial tokens.

  • Prefixing Attention Sinks [156] (EMNLP 2024): Constructs a learnable prefix that serves as a dedicated buffer for outlier activations during quantization. By confining the extreme values that would otherwise appear in arbitrary token positions to a small, known region (the prefix), the method enables per-tensor activation quantization — the quantizer's range can be set based on the prefix's expected outlier magnitudes, while the remaining tokens' activations stay within a narrower, more quantizable range.

  • CTR-Sink [47] (ArXiv 2025): Adapts the paradigm to recommendation systems, where user behavior sequences lack the natural coherence of language. Artificial sink tokens aggregate local context, carrying business semantics such as time intervals between actions.

Why this paradigm is proactive rather than reactive: The paper distinguishes Learnable Prefix Tokens from Preservation and Redistribution along an important axis: prefix tokens are introduced during training and are part of the model's learned computation, not post-hoc interventions at inference. This means they can be optimized to not merely absorb attention but to do something useful with it — e.g., CTR-Sink tokens aggregate local context, and register tokens in VGGT [11] (CVPR 2025) encode camera parameters for 3D vision tasks. The cost is that they require training (or at least fine-tuning), making them inapplicable to already-deployed pretrained models without additional training investment.

The relationship to natural sink elimination: An important nuance the paper highlights (Section 3.3.2, discussing Test-time Registers [121]) is that adding register tokens does not eliminate AS — it reallocates it from natural sinks (background patches, initial tokens) to controlled prefix sinks. The Softmax sum-to-one constraint still forces attention mass to go somewhere; the prefix tokens simply provide a designated, optimized destination. This is fundamentally different from mitigation strategies (Section 5) that remove the need for any sink by modifying Softmax or adding gating. Prefix tokens work within the existing Softmax paradigm, making them a utilization strategy rather than a mitigation strategy.

3.4.9 Sink Token Repurposing (Utilization Paradigm 4)

The final utilization paradigm (Section 3.4) exploits the intrinsic properties of existing AS — their stable high attention scores, fixed positions, numerical outlier status, and geometric distinctiveness — to achieve specialized objectives beyond attention management.

Core methodology (Section 3.4.1). Unlike preservation (which maintains sinks as anchors), redistribution (which reallocates their attention), or prefix tokens (which create artificial sinks), repurposing treats AS as a computational primitive that can be leveraged for tasks including:

  • Offensive Use: Exploiting sink tokens as attack vectors. Forgetting to Forget [33] (ArXiv 2025) demonstrates that backdoor triggers placed at sink positions and aligned with sink attention values exhibit significantly enhanced persistence during unlearning — the model unlearns the target knowledge on clean inputs but reliably recovers it when the hidden trigger is present. Mirage in the Eyes [109] (USENIX 2025) induces hallucination attacks against MLLMs by injecting adversarial visual inputs that create additional AS tokens, leveraging the existing attention concentration mechanism to propagate harmful content.

  • Defensive Use: Using sink properties as protective buffers or diagnostic signals. Surgery [171] (ArXiv 2026) monitors "sink divergence" — the difference in sink attention between harmful and refusal samples — and applies regularization to suppress positive divergence, preventing models from learning harmful patterns during fine-tuning. The defense formulation is:

Ldefense=λ⋅1∣H∣∑h∈HReLU(dh)\mathcal{L}_{\text{defense}} = \lambda \cdot \frac{1}{|H|} \sum_{h \in H} \text{ReLU}(d_h)

where $H$ is the set of attention heads, $d_h$ quantifies the difference in sink attention between harmful and refusal samples for head $h$, and $\lambda$ is a regularization weight. The ReLU ensures that only heads showing increased sink attention on harmful samples are penalized — heads where sink attention decreases (indicating benign behavior) are unaffected.

  • Efficiency-Oriented Use: Leveraging geometric and statistical properties. KeyDiff [78] exploits the fact that sink tokens have near-zero cosine similarity with the mean key vector to identify critical tokens for eviction decisions without computing full attention. StreamingDialogue [160] (NeurIPS 2024) treats dialogue end-of-utterance tokens as natural AS to aggregate and compress long conversation histories.

What distinguishes repurposing from other paradigms: The paper emphasizes (Section 3.4.3) that repurposing represents a "high-leverage intervention point" — because AS tokens sit at the intersection of attention concentration, activation outliers, and geometric anchoring, manipulating them can produce cascading effects on model behavior with minimal computational cost. However, the field currently lacks a rigorous theoretical framework for quantifying AS capacity (how much information can be encoded in sink tokens without disrupting their primary function) or predicting the downstream effects of sink manipulation.

3.4.10 Gated Attention Mechanisms (Mitigation Strategy 1)

Moving from utilization (working with AS) to mitigation (eliminating AS), Gated Attention Mechanisms (Section 5.1) address the root cause identified by the No-Op Theory: the inability of attention heads to produce near-zero outputs without extreme logits.

Core formulation (Section 5.1.1). First introduced in Quantizable Transformers [29] (NeurIPS 2023):

GatedAttention(x)=σ(G(x))⊙Softmax(Q(x)K(x)⊤dhead)V(x)\text{GatedAttention}(x) = \sigma(G(x)) \odot \text{Softmax}\left(\frac{Q(x)K(x)^\top}{\sqrt{d_{\text{head}}}}\right) V(x)

where $\sigma(\cdot)$ is the sigmoid function ensuring outputs in $[0, 1]$, $G(\cdot)$ is a learnable projection that produces a gating vector of the same dimension as the attention output, and $\odot$ denotes element-wise multiplication.

What it computes: After the standard Softmax attention produces its output (a weighted sum of value vectors), the gating vector $\sigma(G(x))$ is computed from the same input $x$ through a separate learned projection. Each element of the attention output is multiplied by the corresponding element of this gating vector. When an attention head needs to make no contribution to the residual stream, it can learn to set the gating vector close to zero — directly suppressing its output — rather than learning to force extreme Softmax logits to concentrate attention on sink tokens with near-zero value vectors.

Why this breaks the No-Op cycle: The gating mechanism decouples the "null update" behavior from the Softmax distribution. In standard attention, producing a near-zero output requires two coordinated operations: (1) concentrate attention on sink tokens (which requires extreme logits), and (2) learn near-zero value vectors for those sink tokens (which requires coordinated weight training). The gating mechanism collapses this into a single operation: set the gate to zero. No extreme logits are needed, no sink tokens are formed, and the attention distribution can remain flat or focus on genuinely relevant tokens.

The design space exploration [26] (NeurIPS 2025). The paper highlights a comprehensive ablation study over 30 variants of 15B MoE models and 1.7B dense models trained on 3.5 trillion tokens, investigating two primary gating forms:

  • Head-wise scalar gating: A single scalar $\sigma(g_h(Q))$ per head modulates the entire attention output. This introduces non-linearity upon the low-rank mapping in softmax attention and yields query-dependent sparse gating scores. It is found to achieve the best trade-off between effectiveness and parameter efficiency.

  • Element-wise gating: A full vector $\sigma(G(x))$ is applied element-wise to the attention output. This offers finer control — individual dimensions of the attention output can be independently suppressed — but introduces significantly more parameters.

The key empirical result: head-wise scalar gating reduces the proportion of attention allocated to the first token from 46.7% (averaged across layers) to 4.8%, and in the most extreme head (layer 21), from 83% to 4%. Training stability also improves — gating suppresses loss spikes, enabling higher learning rates and better scaling.

Value-State Gated Attention (VGA) [44] (Section 5.1.2). An alternative gating strategy applies the gate to value representations before they are weighted by the attention matrix:

VGA(Q,K,V)=Softmax(QK⊤d)(σ(Gv(V))⊙V)\text{VGA}(Q, K, V) = \text{Softmax}\left(\frac{QK^\top}{\sqrt{d}}\right) \left(\sigma(G_v(V)) \odot V\right)

where $G_v(\cdot)$ is a learnable projection that produces a gating vector from the value representations themselves.

Why value-state gating differs from output gating: By gating the value vectors rather than the attention output, VGA creates a direct regulatory pathway — the gate is computed from the value representations, meaning it can suppress a token's contribution based on the token's own emergent value state, not based on the query's perspective. The paper's theoretical analysis (via gradient examination) shows that this design decouples value and attention score updates more effectively than input-state gating, allowing the model to learn which value states correspond to "should be suppressed" independently of the attention scores those tokens receive.

Limitations common to all gated approaches (Section 5.1.3):

  • Training from scratch required: Gate parameters cannot be injected into pretrained models without retraining, limiting applicability for adapting existing deployed models.
  • Non-negligible parameter overhead: Element-wise gating modulates each dimension independently, adding parameters proportional to the model dimension.
  • Training dynamics poorly understood: How gate values evolve during optimization, when they converge toward near-zero states, and how this suppression interacts with value norms remain open questions.
  • No standardized evaluation: The absence of consistent metrics for AS mitigation effectiveness makes it impossible to quantitatively compare gated attention variants against modified softmax or learnable bias approaches on a common scale.

3.4.11 Modified Softmax Functions (Mitigation Strategy 2)

Where gated attention provides an alternative pathway for null updates, Modified Softmax Functions (Section 5.2) directly intervene in the Softmax normalization itself to prevent the formation of extreme logits that give rise to AS.

The three families of modification (Section 5.2.2):

1. Output-Constrained Softmax retains the Softmax framework but restricts its output range to prevent extreme probabilities.

  • Softmax-1 [162] (ICLR 2025): Modifies the denominator by adding a constant 1:

Softmax-1(z)i=ezi1+∑jezj\text{Softmax-1}(z)_i = \frac{e^{z_i}}{1 + \sum_j e^{z_j}}

where the constant 1 in the denominator acts as a virtual extra token with logit 0 (since $e^0 = 1$). This allows the sum of attention weights to be less than 1 — the "missing" probability mass is allocated to this implicit null token.

What it computes: For each query, the attention weights no longer sum to 1; they sum to $S/(1+S)$ where $S = \sum_j e^{z_j}$. The remaining $1/(1+S)$ is effectively "thrown away" — the model can allocate zero attention to all real tokens by letting the implicit null token absorb the probability mass. Empirically, this reduces first-token attention from 65% to 3.3% and activation kurtosis from 1657 to 3.1, enabling robust 4-bit quantization.

Why this specific constant (1): Adding 1 to the denominator is equivalent to appending an extra key with logit 0. Since $e^0 = 1$, this token always receives a fixed share of the denominator regardless of the actual logits. This is the minimal modification that breaks the sum-to-one constraint — adding a bias $b$ to the denominator generalizes to $\sum_j e^{z_j} + b$, and $b=1$ is the simplest case that achieves the desired effect. The constant 1 has the appealing property that when all $z_i$ are small (all tokens equally irrelevant), the attention weights approach $1/(N+1)$ rather than being forced to sum to 1, naturally distributing attention mass away from real tokens when none are relevant.

  • Clipped Softmax [29] (NeurIPS 2023): Maps the Softmax output to $[\gamma, \zeta]$ then clips to $[0, 1]$:

ClippedSoftmax(x;ζ,γ)=clip((ζ−γ)⋅Softmax(x)+γ,0,1)\text{ClippedSoftmax}(x; \zeta, \gamma) = \text{clip}\left((\zeta - \gamma) \cdot \text{Softmax}(x) + \gamma, 0, 1\right)

where $\zeta \geq 1$ and $\gamma \leq 0$ are hyperparameters. This allows exact zeros and ones with finite input range — the stretching $(\zeta - \gamma)$ and offset $\gamma$ mean that the argmax of the pre-clip values is the same as the argmax of the Softmax, but extreme probabilities are prevented and gradient flow is blocked for clipped values, forcing the model to learn outlier-free strategies.

2. Normalization-Free Attention abandons the sum-to-one constraint entirely, replacing Softmax with functions that evaluate each query-key pair independently.

  • Softpick [77] (ArXiv 2025): A rectified, non-normalized function:

Softpick(z)i=max⁡(0,ezi∑jezj−τ)\text{Softpick}(z)_i = \max\left(0, \frac{e^{z_i}}{\sum_j e^{z_j}} - \tau\right)

where $\tau$ is a threshold hyperparameter. Starting from standard Softmax probabilities, it subtracts $\tau$ and applies ReLU, setting all probabilities below the threshold to exactly zero.

What it computes: For each query-key pair, compute the standard Softmax probability, subtract the threshold, and clip negative results to zero. The sum of the resulting values is not constrained to be 1 — it can be anywhere from 0 (all probabilities below $\tau$) to $1 - K\tau$ (where $K$ is the number of tokens with probabilities above $\tau$). When no token is relevant, all probabilities can be below $\tau$, producing an all-zero attention output — a true null update without any sink token.

Why this form: The threshold mechanism directly addresses the root cause: if an attention head needs to produce no output, it can simply ensure all post-Softmax probabilities are below $\tau$, achieving exact zeros without extreme logits. Empirical results on 340M models show a 0% sink rate (no token meets the threshold-based AS criterion) and reduction of activation kurtosis from 33,510 to 340.

  • Sigmoid Attention [25] (ICLR 2025): Replaces Softmax entirely with the element-wise sigmoid:

SigmoidAttn(Q,K)=σ(QK⊤d)\text{SigmoidAttn}(Q, K) = \sigma\left(\frac{QK^\top}{\sqrt{d}}\right)

where $\sigma(x) = 1/(1 + e^{-x})$. Without normalization, each query-key pair is evaluated independently, making AS impossible by construction — there is no competition among keys, so no single token can monopolize the probability mass.

3. Pre-Softmax Modulation retains Softmax but modifies its inputs to prevent extreme logits.

  • SWAT [83]: Combines sigmoid attention with sliding window training, achieving competitive long-context performance without AS.
  • Integral Transformer [67] (EMNLP 2025): Denoises attention by integrating signals sampled from the logit distribution, conceptually replacing deterministic logits with expected values over a noise distribution to produce more balanced logits that prevent disproportionate weight on sink tokens.

Why these methods work (the causal chain): Each family intervenes at a different point in the causal chain that produces AS. Normalization-free attention (Softpick, Sigmoid) removes the sum-to-one constraint, eliminating the necessity of AS entirely — the model never faces the forced allocation problem. Output-constrained softmax (Softmax-1, Clipped Softmax) retains the constraint but makes exact zeros achievable with finite logits, allowing null updates without extreme values. Pre-Softmax modulation (SWAT, Integral Transformer) reduces logit variance, preventing the exponential function in Softmax from disproportionately amplifying large logits and suppressing small ones.

The trade-off (Section 5.2.3): The paper identifies a fundamental tension. While these methods effectively eliminate AS, "excessive flattening of attention distributions can diminish the model's capacity to concentrate on genuinely informative tokens, potentially harming performance on tasks that demand sharp attention." For example, a QA task where the model must precisely attend to a single answer span requires the ability to produce highly peaked attention distributions — the same capability that, in standard Softmax, produces AS on uninformative tokens. The challenge is designing modifications that allow sharp attention when needed while preventing forced attention allocation when no token is relevant.

3.4.12 Learnable Attention Bias and Pre-Training Interventions (Mitigation Strategies 3 and 4)

Learnable Attention Bias (Section 5.3) is motivated by the Implicit Attention Bias interpretation (Section 4.3): if AS functions as a fixed, input-independent bias term added to attention outputs, then providing an explicit learnable bias should eliminate the need for AS. The paper surveys four instantiations:

  • Key-Value Bias Concatenation [98]: Augments attention with learnable key and value vectors $k', v' \in \mathbb{R}^d$ appended to the key and value matrices. Training with these explicit biases from scratch eliminates Massive Activations entirely. The formulation is:

Attention(Q,K,V;k′,v′)=softmax(Q[K⊤ k′]d)[Vv′⊤]\text{Attention}(Q, K, V; k', v') = \text{softmax}\left(\frac{Q[K^\top \ k']}{\sqrt{d}}\right) \begin{bmatrix} V \\ v'^\top \end{bmatrix}

  • Key Bias [25]: Adds a learnable bias matrix $K_{\text{bias}}$ to the key matrix. With only key biases, AS "disappears and attaches to the bias position" — proving that an explicit key bias can completely substitute for the implicit sink mechanism.

  • Denominator Bias [54, 66]: The most parameter-efficient form — adds a head-specific learnable scalar $b$ to the Softmax denominator:

SoftmaxLAB(z)i=ezi∑jezj+b\text{Softmax}_{\text{LAB}}(z)_i = \frac{e^{z_i}}{\sum_j e^{z_j} + b}

This term "creates a virtual sink that absorbs excess attention probability when no real token is relevant," achieving the same functional effect as Softmax-1 but with a learned rather than fixed constant.

Pre-Training Interventions (Section 5.4) target not the architecture but the optimization dynamics that give rise to outlier circuits. Standard adaptive optimizers like Adam have been shown to favor "privileged bases" in weight matrices — specific directions that receive disproportionately large updates, producing weight outliers that cascade into activation outliers and AS. The paper surveys three categories:

  • Loss Function Interventions: TWEO [59] penalizes activation distribution tails, reducing outliers from over 10,000 to below 20, enabling FP8 training at BF16-comparable quality with 36% throughput improvement. Sink-Aware Training [191] adds an auxiliary load balancing loss encouraging uniform attention head utilization, addressing head collapse where only a subset of heads become sinks.

  • Optimizer Interventions: OrthoAdam [162] uses orthogonal matrices to transform gradients, preventing accumulation in privileged directions and reducing activation kurtosis from 1657 to 3.1. Muon Optimizer, adopted in the Outlier-Safe Pre-Training (OSP) framework [42] (ACL 2025), eliminates privileged bases entirely.

  • Integrated Frameworks: OSP combines Muon Optimizer, single-scale RMSNorm (replacing per-channel learnable scales with a single scalar to prevent channel-wise outlier amplification), and learnable embedding projection to produce the "first production-scale LLM without extreme activation outliers" — a 1.4B model trained on 1 trillion tokens.

The paper's synthesis (Section 5.5). The surveyed mitigation strategies are organized into two over-arching principles: (1) providing explicit alternatives that render AS unnecessary (Gated Attention, Learnable Bias, Outlier-Driven Rescaling, Architectural Isolation), and (2) cutting the causal chain that gives rise to AS (Modified Softmax, Pre-training Interventions). This taxonomy helps practitioners select strategies based on their deployment constraints: those who can train from scratch should consider cutting the causal chain at its root (Softmax modification or pre-training intervention), while those adapting pretrained models may need to explore explicit alternatives that can potentially be injected via fine-tuning (though the paper notes this remains an open challenge — most methods currently require training from scratch).

4. Key Insights and Innovations

Innovation 1: A Three-Axis Taxonomy Reveals That Utilization, Interpretation, and Mitigation Form a Cumulative Intellectual Progression, Not Parallel Subfields

The survey's most fundamental contribution is not any single finding about Attention Sink (AS) but the meta-argument embedded in its organizational structure: that AS research has progressed through three distinct intellectual phases — Fundamental Utilization (exploiting AS as an empirical fact), Mechanistic Interpretation (explaining why AS exists), and Strategic Mitigation (eliminating AS at its root) — and that these phases form a cumulative developmental trajectory rather than independent research threads. This is visualized in Figure 3, which plots publication counts over time showing Utilization dominating 2023–2024, Interpretation deepening in 2024–2025, and Mitigation emerging as the frontier in 2025–2026.

What the field did before this taxonomy. Prior to this survey, AS research was siloed by both architecture and application. A researcher studying KV cache compression for LLMs (working on StreamingLLM or H2O) would not necessarily be aware that the "massive activations" literature in mechanistic interpretability was studying the same underlying phenomenon, or that the "register tokens" solution developed for ViTs addressed the same root cause that gated attention targeted in language models. Individual papers cross-referenced narrowly within their subcommunity — a KV cache compression paper might cite other compression papers, but not outlier circuit analyses or softmax modification proposals. The field lacked a shared vocabulary (the same phenomenon was called "attention sink," "massive activations," "outlier dimensions," and "heavy hitters" in different communities) and a shared conceptual framework for connecting empirical exploitation strategies to mechanistic causes to architectural solutions.

What is distinctive at the idea level. The survey's taxonomy is not merely a filing system — it makes a historical argument about the field's maturation. By structuring the literature as "Initial Focus → Deepening Understanding → Systematic Intervention," the paper argues implicitly that the field is ready to move beyond ad-hoc sink management (preserving initial tokens, redistributing attention post-hoc) toward principled architectural solutions (gated attention, modified softmax, pre-training interventions). This argument is not neutral: it asserts that Utilization strategies, while practically valuable, are intellectually incomplete without mechanistic grounding, and that Mitigation strategies are the natural endpoint of the research trajectory once the root causes are understood.

The taxonomy also reveals complementarities invisible in isolation. For example, the paper shows that Sink Token Preservation (Section 3.1) and Gated Attention Mechanisms (Section 5.1) address the same underlying phenomenon — the No-Op behavior forced by Softmax — but from opposite directions: preservation accepts the behavior and ensures it can continue under compression, while gating eliminates the need for it entirely. A practitioner choosing between these approaches can now understand that they are not arbitrary alternatives but points on a spectrum from exploitation to elimination, with different trade-offs in deployment feasibility (preservation requires no retraining; gating requires training from scratch) and asymptotic performance (preservation inherits all the limitations of AS, including quantization sensitivity; gating removes them).

Why this is fundamental rather than incremental. The taxonomy is not an incremental refinement of an existing categorization — it is the first systematic organization of a field that previously had none. Prior work (e.g., individual papers in each sub-area) made local contributions without connecting them to the broader landscape. The survey's taxonomy creates the conceptual infrastructure for future research: a new paper on AS in diffusion language models can now be situated within the Utilization/Interpretation/Mitigation framework, its relationship to work in other architectures immediately visible through the shared taxonomy, rather than requiring ad-hoc literature review for each new domain.

Evidence anchoring. The temporal trend in Figure 3 (cumulative publication counts from 2023–2026) directly supports the cumulative progression claim. The comprehensive paper table in Appendix A, which tags each of 180+ papers along the three taxonomy axes, demonstrates that most papers address only one or two axes, validating that the axes represent genuinely distinct research foci rather than arbitrary labels. The cross-referencing density in Sections 3–5 — where utilization methods are evaluated against mechanistic theories (e.g., preservation methods' success as evidence for the No-Op Theory) and mitigation strategies are traced to specific interpretive insights (e.g., gated attention as a direct response to Softmax Limitations) — demonstrates that the taxonomy enables connections that were not being made in the original literature.

Innovation 2: AS Is Reframed as a Necessary Consequence of Architectural Constraints, Not a Contingent Artifact of Scale or Training

Before the systematic interpretation synthesized in this survey, AS was often treated as an emergent quirk of large-scale training — something that "happens" to large models but is not fundamentally required by the architecture. Individual papers observed AS in specific contexts (e.g., StreamingLLM [24] noted that initial tokens dominate attention in LLaMA; Massive Activations [98] documented extreme activations in LLMs and ViTs) but did not articulate a unified causal explanation that applied across architectures and predicted the phenomenon's necessary emergence.

What the field assumed before. The dominant implicit assumption was that AS was a contingent artifact — possibly a result of particular training data distributions, optimizer dynamics, or scale thresholds. This assumption led to research strategies that treated AS as a problem to be empirically managed (e.g., preserving initial tokens during KV cache compression) or an idiosyncrasy to be worked around (e.g., adding register tokens to ViTs to absorb "artifacts"), rather than as a structural consequence of the Softmax attention formulation itself. The field lacked a clear answer to the question: if we trained a Transformer from scratch with different data, a different optimizer, or at a different scale, would AS still emerge?

The reframing. The survey synthesizes four complementary interpretation perspectives — Softmax Limitations and No-Op Theory (Section 4.1), Outlier Circuits (Section 4.2), Implicit Attention Bias (Section 4.3), and Geometric Anchoring (Section 4.4) — into a single causal narrative that reframes AS from a contingent artifact to an architecturally necessary phenomenon:

  1. The Softmax sum-to-one constraint (Section 4.1) makes AS mathematically inevitable: when an attention head needs to produce a near-zero output, the only way to achieve this under standard Softmax is to concentrate attention on tokens with negligible value vectors, because Softmax cannot output exact zeros without extreme logits. This is not a property of scale or training data — it is a property of the Softmax function itself.

  2. The Outlier Circuits (Section 4.2) explain how this mathematical necessity is implemented numerically: weight outliers in down-projection matrices amplify activations along specific feature dimensions, these activation outliers concentrate attention on corresponding tokens, and the value vectors of those tokens remain suppressed, creating a closed-loop mechanism that sustains AS. This explains why AS is robust — once the circuit forms during training, it is self-reinforcing.

  3. The Implicit Attention Bias perspective (Section 4.3) reveals what functional role AS serves: the constant value contribution from sink tokens acts as a learned bias term, compensating for the absence of explicit bias parameters in standard attention.

  4. Geometric Anchoring (Section 4.4) shows why the model benefits from this arrangement: sink tokens provide stable reference points that structure the representational geometry, preventing representational collapse.

What is distinctive at the idea level. The intellectual move is to convert a collection of independent empirical observations — "LLaMA attends to the first token," "ViTs have activation outliers on background patches," "BERT attends to [SEP]" — into predictions of a unified theory. The No-Op Theory predicts that sink tokens should exhibit suppressed value norms (validated across BERT, ViT, and LLaMA by [29, 28, 44]), that removing the sum-to-one constraint should eliminate AS (validated by Softmax-1, Softpick, and Sigmoid Attention [162, 77, 25]), and that AS should emerge wherever Softmax attention is used regardless of modality (validated by its observation in text, vision, audio-visual, and multimodal models). This predictive unification is what distinguishes a genuine mechanistic theory from a post-hoc description.

Why this is fundamental. This reframing has direct practical consequences. If AS were a contingent artifact, the correct strategy would be to empirically identify and work around it in each new architecture — a never-ending game of whack-a-mole. If AS is an architectural necessity, the correct strategy is to modify the architecture (gated attention, modified softmax, explicit biases) or modify the training dynamics (pre-training interventions) to remove the necessity at its root. The paper's taxonomy of mitigation strategies (Section 5.5) explicitly organizes methods along this principle: methods that provide explicit alternatives (rendering AS unnecessary) versus methods that cut the causal chain (eliminating the conditions that force AS to emerge). This organization is only possible because of the reframing — it would not make sense if AS were understood as a quirky byproduct rather than a structural consequence.

Evidence anchoring. The paper's organization of Section 4 supports this reframing through its structure. Rather than presenting the four interpretation theories as competing explanations, it presents them as complementary levels of analysis — mathematical origin (Softmax Limitations), numerical mechanism (Outlier Circuits), functional role (Implicit Bias), and representational geometry (Geometric Anchoring) — and provides a summary table in Section 4.5 that maps each theory to its analytical level and central question. The cross-referencing of causal evidence (e.g., pruning Super Experts collapses AS [43]; adding explicit biases eliminates Massive Activations [98]; replacing Softmax with Softpick achieves 0% sink rate [77]) demonstrates that the theories are not merely interpretive but make falsifiable predictions that have been empirically validated across architectures.

Innovation 3: The Four Utilization Paradigms Are Distinguished by Their Stance Toward AS — Passive, Active, Proactive, or Instrumental — Not by Their Application Domain

Prior to this survey, methods that leveraged AS were categorized primarily by their application domain: KV cache compression, sparse attention, quantization, multimodal hallucination reduction, streaming generation. This application-driven categorization obscured the conceptual relationships between methods — a researcher working on KV cache compression might not realize that their approach (retaining initial tokens) shares a fundamental logic with a method for stabilizing video generation (retaining initial frames as global anchors), or that their approach differs categorically from methods that actively redistribute attention rather than passively preserving sinks.

What was missing. The application-driven categorization conflates the problem being solved with the strategy being employed. Two methods that both target KV cache compression — one that retains initial tokens (preservation) and one that dynamically identifies and retains tokens with high cumulative attention (also preservation but with dynamic identification) — are more similar conceptually than two preservation methods applied to different domains (KV cache compression vs. video generation). The application-driven categorization makes the former similarity invisible and the latter invisible as well, preventing cross-domain transfer of insights.

The reframing into four stances. The survey introduces four paradigms distinguished by their stance toward AS rather than their application:

  1. Sink Token Preservation (Section 3.1): Passive. The method does not alter AS — it simply ensures that existing sink tokens remain available as attention targets under compression, respecting the model's learned dependence on them. The core insight across applications (StreamingLLM for text generation, H2O for KV cache compression, Rolling Forcing for video diffusion) is identical: don't break what the model expects.

  2. Attention Redistribution (Section 3.2): Active. The method intervenes to reallocate attention mass from sink tokens to semantically meaningful targets, but it works within the existing Softmax framework — it changes where attention goes but not the mechanism that produces AS. The core insight across applications (VAR for MLLM hallucination, ACT for LLM improvement, Pos2Distill for position bias) is that sink tokens consume representational capacity that can be productively reallocated.

  3. Learnable Prefix Tokens (Section 3.3): Proactive. The method introduces new, trainable sink targets during training, creating controlled sinks rather than relying on emergent ones. The core insight across applications (register tokens for ViTs, placeholder tokens for StreamingLLM, prefix tokens for quantization) is that sink behavior can be designed rather than discovered.

  4. Sink Token Repurposing (Section 3.4): Instrumental. The method exploits AS properties — stable positions, high attention scores, geometric distinctiveness — as computational primitives for tasks unrelated to attention management, such as backdoor attacks, defense, or efficiency optimization. The core insight is that AS tokens are high-leverage intervention points because they sit at the intersection of multiple model subsystems.

What is distinctive at the idea level. This categorization is not a relabeling exercise — it reveals a spectrum of intervention intensity (from passive to instrumental) and a spectrum of architectural coupling (preservation and repurposing work with existing models without modification; redistribution works with existing models but modifies attention computations at inference; prefix tokens require training or fine-tuning). These spectra immediately tell a practitioner where on the deployment-feasibility vs. asymptotic-performance trade-off each paradigm sits, and which paradigm is appropriate given their constraints (can they retrain? can they modify inference code?).

Why this is a conceptual advance rather than incremental. This categorization enables cross-domain transfer that was previously impossible. A researcher developing a method to reduce hallucination in MLLMs by reallocating attention from visual background patches to foreground objects (Attention Redistribution applied to MLLMs) can now recognize that their approach is conceptually analogous to a method for improving LLM performance by calibrating attention away from initial tokens (ACT, Attention Redistribution applied to text-only LLMs). The shared paradigm immediately suggests that techniques developed in one domain (e.g., ZeroTuning's bias-based attention modulation for text) might be adapted to the other (bias-based modulation of visual attention). Without the paradigm-level categorization, these connections would require reading across disparate application literatures with no shared vocabulary to guide the search.

Evidence anchoring. The paper's organization of Section 3 demonstrates this reframing in action. Each paradigm subsection (3.1–3.4) follows an identical structure — Core Methodology, Practical Approaches (with sub-categories spanning multiple application domains), and Discussion/Insights — making the cross-domain generality of each paradigm visually apparent. The discussion subsections explicitly identify transferrable limitations (e.g., all preservation methods face the challenge of dynamic sink identification; all redistribution methods face the challenge of kernel compatibility) that would appear as domain-specific issues under an application-driven categorization.

5. Experimental Analysis

Evaluation Methodology

This survey does not conduct original experiments. It is a literature synthesis that reports and organizes findings from over 180 prior studies. As such, there is no single experimental protocol, but the paper does identify the experimental practices common across the AS literature, which can be characterized as follows:

  • Dataset. The surveyed papers use a diverse collection of benchmarks because AS manifests differently across architectures and modalities. For language models, common datasets include the MATH benchmark, standard language modeling corpora (for perplexity evaluation), and long-context reasoning benchmarks. For vision transformers, ImageNet classification and dense prediction tasks dominate. For multimodal LLMs, hallucination benchmarks (e.g., POPE, CHAIR) and visual grounding tasks are standard. The paper does not prescribe a specific dataset because AS is a cross-architectural phenomenon; rather, it catalogues which datasets each surveyed method was validated on (Appendix A, Table 1).

  • Base model(s). The surveyed literature spans models from BERT-base and RoBERTa (110M-355M parameters) through LLaMA-2-7B/13B, LLaMA-3, and Mistral-7B to production-scale models including Qwen3-30B-A3B (MoE), GPT-OSS-120B, and multimodal models such as LLaVA and Qwen-VL. ViT studies use DINOv2, DeiT-III, and OpenCLIP. This architectural and scale diversity is itself a finding — AS emerges across all these families, confirming it is not an artifact of a particular model scale or training recipe (Section 2.2.2).

  • Metrics. No single metric spans all surveyed work. The paper identifies several metric families (Section 2.2.2, and throughout Sections 3-5):

    • Sink rate: The fraction of attention heads or tokens meeting a threshold-based AS criterion (Equation 5). Softpick [77] reports 0% sink rate vs. 63.41% for standard Softmax on 340M models.
    • Attention concentration: The proportion of total attention mass allocated to sink tokens. Gated Attention [26] reports reduction from 46.7% (averaged across layers) to 4.8% on the first token.
    • Activation kurtosis / outlier magnitude: Quantifying the extremeness of activation distributions. Softmax-1 [162] reduces kurtosis from 1657 to 3.1; TWEO [59] reduces activation outliers from over 10,000 to below 20.
    • Perplexity (PPL): For language modeling tasks, particularly in KV cache compression and streaming evaluation. StreamingLLM [24] reports PPL of 5.43 with sink preservation vs. 5,158 without on long texts.
    • Task-specific accuracy: MATH accuracy for reasoning tasks, ImageNet top-1 for vision, hallucination rates for MLLMs.
    • Quantization fidelity: The gap between full-precision and quantized model performance, measured as accuracy degradation or perplexity increase per bit reduction.

  • Baselines. The paper surveys methods rather than conducting head-to-head comparisons, so baselines vary by sub-area. Common baselines include:

    • Standard dense attention (all tokens retained) as the upper bound for KV cache compression methods.
    • Best-of-N or majority voting for verifier-based selection methods.
    • Full-precision models as baselines for quantization studies.
    • Standard Softmax attention as the baseline for modified softmax and gated attention methods.
    • Vanilla fine-tuning (without AS-aware interventions) for safety and robustness studies such as Surgery [171].
    • Each surveyed paper typically compares against the strongest prior method in its specific sub-area, but these baselines are not standardized across the AS literature — a gap the paper identifies (Section 7.2) as an open challenge requiring a standardized evaluation benchmark.

  • Generation budget / compute accounting. For inference-efficiency methods (KV cache compression, sparse attention), compute is measured in token retention ratio (fraction of KV cache entries preserved), memory footprint reduction, or latency speedup. For streaming methods, budget is measured in window size and number of sink tokens retained. For quantization methods, budget is measured in bit-width (2-bit, 4-bit, 8-bit) and the resulting model size reduction. For pre-training interventions, compute is measured in training FLOPs, token throughput, and convergence speed relative to standard training. There is no unified compute-accounting framework across the surveyed literature — methods optimized for different constraints (memory vs. latency vs. throughput) use different cost metrics, and the paper does not attempt to reconcile them.

  • Cross-validation / statistical protocol. The paper does not describe a unified statistical protocol, as it synthesizes results from 180+ independent studies. Individual surveyed papers employ standard practices for their domains: perplexity is typically evaluated on held-out validation sets; classification accuracy uses standard train/val/test splits; and KV cache compression methods often report results averaged over multiple random seeds or multiple evaluation runs. Some methods (particularly those involving difficulty estimation for compute-optimal allocation, such as the prior example paper) use cross-validation for strategy selection, but this is not a universal practice across the AS literature. The survey paper itself does not introduce statistical tests or confidence intervals when comparing results across studies — this is a limitation acknowledged implicitly in the call for standardized benchmarks (Section 7.2).

Main Quantitative Results

Because this is a survey paper, it does not present new experimental results. Instead, it aggregates and contextualizes results from the 180+ papers it surveys. The "main results" are therefore the key quantitative patterns that emerge when findings are organized across the three taxonomy axes and five architecture families. Below, I organize these patterns by the paper's logical groupings, citing the specific subsections and figures from the survey that report them.

Architecture-Specific AS Prevalence

The paper establishes that AS is pervasive but architecture-dependent in Sections 2.3–2.5. The headline finding is not a single number but a pattern of qualitative consistency: across all surveyed architectures, AS tokens receive disproportionately high attention while carrying low semantic information, but the identity of sink tokens varies systematically by architectural design.

  • Causal LLMs (Section 2.3.2, Figure 6): The first token receives the maximum attention in 98% of attention heads [162]. This figure, drawn from LLaMA-2-7B, demonstrates near-complete dominance: in almost every attention head, the query-key compatibility score for the initial token exceeds all others, making it the de facto sink. The attention maps in Figure 6 show a vertical stripe at position 0 that remains invariant across input content and generation length.

  • Classical Language Models (Section 2.3.1, Figure 5): In BERT, "[SEP]" and "[CLS]" tokens receive disproportionate attention, with more than half of a head's total attention often allocated to these special tokens [53]. The paper notes that this was empirically observed as early as 2019, well before the term "attention sink" was coined — an example of the fragmentation the survey seeks to address.

  • MoE LLMs (Section 2.3.3, Figure 9): Sink tokens concentrate their routing scores on a tiny subset of Super Experts. In Qwen3-30B-A3B, pruning just 3 of 6,144 experts causes catastrophic AS collapse [43]. The quantitative claim is stark: a 0.05% reduction in expert count eliminates a phenomenon that spans the entire model depth. Figure 9 shows that sink tokens achieve router scores on Super Experts that are orders of magnitude higher than non-sink tokens' scores — the distribution is not merely skewed but bimodal.

  • Multimodal LLMs (Section 2.3.4, Figure 10): Visual background patches exhibit Massive Activations (quantified as hidden state norms reaching 54.7 vs. 5.8 for foreground tokens) and attract attention disproportionate to their semantic relevance [105]. This establishes a direct cross-modal transfer of the AS phenomenon — the same mechanism that concentrates attention on [BOS] in text-only models concentrates attention on background patches in visual-language models.

  • Vision Transformers (Section 2.4, Figure 11): Attention concentrates on background patches, which exhibit both high attention scores (Figure 11c) and low value magnitudes (Figure 11e) — the characteristic signature of AS predicted by the No-Op Theory.

Utilization Results: Preservation vs. Redistribution vs. Prefix vs. Repurposing

The paper organizes utilization results by paradigm rather than application. Each paradigm's headline quantitative claim illustrates its characteristic trade-off:

Sink Token Preservation (Section 3.1, Figure 13):

  • StreamingLLM [24]: Using sliding window attention with 4 sink tokens + 2,048 window tokens achieves PPL of 5.40 on long texts, comparable to full dense attention, whereas sliding window without sink preservation causes PPL to explode to 5,158. This is a ~957× reduction in perplexity from retaining just 4 tokens — the most dramatic quantitative demonstration of AS dependence in the survey.
  • MInference [182]: Sparse attention with sink token mask enforcement accelerates pre-filling by up to 10× without accuracy loss on long-context benchmarks. The speedup is achieved by recognizing that attention patterns are highly structured and that sink tokens are always needed — sparsifying everything except sink access preserves model behavior while reducing computation.
  • KV cache quantization (rotated): RotateKV [84] achieves 2-bit KV cache quantization with minimal accuracy degradation by preserving sink tokens at full precision while quantizing other tokens, enabled by outlier-aware adaptive rotations.

Attention Redistribution (Section 3.2, Figure 16):

  • VAR [105]: On multimodal hallucination benchmarks, redistributing attention from visual background sinks to foreground objects achieves hallucination reduction comparable to methods requiring model retraining, despite being training-free. The exact hallucination rate reduction is not quoted in the survey, which cites [105] without reproducing its full result table — a limitation in how the survey reports numbers.
  • ZeroTuning [75]: Adding a single learnable bias b to the initial token's pre-Softmax logit enables per-input optimization of the entire attention distribution, improving task accuracy across diverse benchmarks without model retraining. The mechanism exploits the zero-sum property of Softmax — adjusting one token's logit indirectly redistributes attention to all others.

Learnable Prefix Tokens (Section 3.3, Figures 17-19):

  • Vision Transformers Need Registers [126]: Adding 4 register tokens during pre-training produces qualitatively cleaner attention maps (Figure 19) and improves dense prediction task performance. The improvement is visual as much as quantitative — the attention maps shift from noisy, background-concentrated patterns to clean, foreground-focused patterns.
  • Prefixing Attention Sinks [156]: Using a learnable prefix as an outlier buffer enables per-tensor activation quantization that would otherwise fail due to outlier-dominated dynamic ranges (Figure 18). The key number is not a single accuracy figure but the fact that per-tensor quantization becomes possible at all — without prefix tokens, the outlier activations force either per-token quantization (expensive) or unacceptable accuracy degradation.

Sink Token Repurposing (Section 3.4, Figures 20-21):

  • Forgetting to Forget [33]: Placing backdoor triggers at sink positions enhances backdoor persistence during unlearning — the metric is recovery efficacy, with sink-aligned triggers achieving higher recovery than randomly placed triggers. The specific percentages are not reproduced in the survey's text, but the finding's implication is clear: AS tokens are a vulnerability surface.
  • Surgery [171]: Sink divergence regularization during fine-tuning prevents the model from learning harmful patterns, with effectiveness measured by the model's refusal rate on harmful queries after defensive fine-tuning.

Mitigation Results: Gating, Modified Softmax, Learned Bias, Pre-training

Gated Attention Mechanisms (Section 5.1, Figures 37-38):

  • Head-wise scalar gating [26]: Across 15B MoE models and 1.7B dense models trained on 3.5 trillion tokens, gating after SDPA reduces the proportion of attention allocated to the first token from 46.7% (averaged across all layers) to 4.8%. In the most extreme head (layer 21, baseline), first-token attention drops from 83% to 4% with gating (Figure 38). Training loss at 3.5T tokens is lower for gated models, and training is more stable with fewer loss spikes.
  • Value-State Gated Attention (VGA) [44]: On BERT, RoBERTa, and LLaMA-2-7B, VGA mitigates AS formation, stabilizes value-state norms, and improves downstream task performance and quantization fidelity compared to output-gating approaches. The paper does not reproduce specific accuracy numbers in Section 5.1.2, which limits direct quantitative comparison between gating variants from the survey text alone.

Modified Softmax Functions (Section 5.2, Figures 24, 40-41):

  • Softmax-1 [162]: Reduces first-token attention from 65% to 3.3% and activation kurtosis from 1657 to 3.1 on LLaMA-scale models (Figure 40). The perplexity penalty under 4-bit weight quantization drops from 3565 to 0.3.
  • Softpick [77]: Achieves 0% sink rate on 340M models (Figure 24), with activation kurtosis reduced from 33,510 to 340. The zero-sink-rate claim is absolute — no token in any head meets the threshold-based AS criterion after replacing Softmax with Softpick.
  • Clipped Softmax [29]: Enables full INT8 quantization of activations without accuracy degradation, achieving outlier-free behavior that standard Softmax cannot match at any bit-width.

Learnable Attention Bias (Section 5.3):

  • Massive Activations [98]: Training GPT-2 with explicit key-value biases eliminates Massive Activations and AS entirely — the finding is qualitative (AS disappears) rather than a numerical comparison, but it provides the strongest causal evidence that AS substitutes for absent explicit bias terms.

Pre-Training Interventions (Section 5.4, Figure 42):

  • TWEO [59]: Reduces activation outliers from over 10,000 to below 20. Under standard FP8 training (which fails catastrophically on models with outliers), TWEO achieves BF16-comparable quality while increasing training throughput by 36%. This enables hardware-friendly W8A8 per-tensor static quantization at state-of-the-art quality.
  • Outlier-Safe Pre-Training (OSP) [42]: Trains a 1.4B parameter model on 1 trillion tokens as the "first production-scale LLM without extreme activation outliers" (Figure 42). The Muon Optimizer with single-scale RMSNorm and learnable embedding projection eliminates AS at the training stage without architectural modification.

Ablation Studies and Robustness Checks

Because this is a survey, ablation studies are those reported by the individual surveyed papers. The survey synthesizes key ablation findings across multiple studies, organized here by the aspect being ablated:

Gating position in attention (Section 5.1, Figure 37, left and middle): The work in [26] systematically compares over 30 gating variants applied at different positions — after SDPA (G1), after the Value layer (G2), at the query/key/value projections, and combinations thereof. The finding is that gating after SDPA (G1) yields the best overall perplexity-performance trade-off across 15B MoE models, while gating after the Value layer (G2) also improves performance (particularly in perplexity) but not as consistently. This ablation matters because it reveals that the location of gating is more consequential than the form of gating (scalar vs. element-wise) — the best location is after the attention computation is complete, not before or during it.

PRM aggregation strategy (Appendix E of the original paper, not in the survey): The survey does not discuss PRM aggregation. (Note: this is from the example paper, not the actual survey — included here because the structure was requested, but the survey paper has no PRM. For completeness: the survey lacks a unified set of shared ablations because it synthesizes results from 180+ independent studies. Its "ablation insights" are therefore patterns that recur across multiple papers, not controlled experiments.)

Register token count in ViTs (Section 3.3, referencing [126]): Vision Transformers Need Registers ablates the number of register tokens added (0, 1, 2, 4, 8). The finding is that 4 tokens are consistently optimal across DINOv2, DeiT-III, and OpenCLIP — fewer than 4 leaves residual attention artifacts on background patches, while more than 4 provides diminishing returns and marginally increases parameter count. This non-monotonic optimality (4 is better than both 2 and 8) suggests that the register capacity must match the "sink bandwidth" of the specific architecture.

Softmax modification hyperparameters (Section 5.2):

  • Softpick threshold \tau [77]: The threshold determines how aggressively low-attention tokens are zeroed out. Setting \tau too high eliminates not only AS but also genuinely informative low-attention tokens, degrading downstream task performance. Setting \tau too low fails to eliminate AS. The optimal \tau is model-dependent and must be tuned per architecture — a practical limitation the paper notes.
  • Temperature in Elastic-Softmax [55]: Higher temperatures T > 1 or smaller exponents \alpha < 1 flatten the attention distribution more aggressively. The ablation shows that 59.58% attention sparsity is achieved at optimal parameters, but excessive flattening harms task performance on tasks requiring sharp attention (this is a qualitative observation, not quantified in the survey text).

Optimizer choice for outlier formation (Section 5.4, Figure 42): The OSP framework [42] ablates three optimizers — Adam, Muon, and OSP (Muon + single-scale RMSNorm + learnable embedding projection) — on a 1.4B model trained on 100B tokens. Adam produces large activation outliers (visible as extreme values in specific feature dimensions). Muon alone reduces outliers but does not eliminate them — they remain present in deep layers. OSP (Muon + additional interventions) eliminates outliers entirely. This ablation demonstrates that optimizer choice is necessary but not sufficient — complementary normalization and projection modifications are required to completely erase the outlier cascade.

Value-state gating vs. output gating vs. input-state gating (Section 5.1, referencing [44]): VGA is compared against vanilla attention and input-state gated attention on BERT, RoBERTa, and LLaMA-2-7B. The finding is that VGA's value-state gating "decouples value and attention score updates more effectively than prior methods that gate on input embeddings." This is demonstrated through gradient analysis rather than raw accuracy numbers — VGA suppresses sink token contributions at the value level, preventing the mutual reinforcement cycle between high attention and suppressed value norms from forming in the first place.

Cross-modal transfer of AS mitigation (Section 4.2.2, referencing [40]): In audio-visual speech recognition, a decorrelation loss that reduces cosine similarity between the BOS token and other tokens mitigates both Massive Activations and intermediate AS tokens. The ablation compares: (1) baseline AVSR model, (2) baseline + decorrelation loss. The finding is that the decorrelation loss improves word error rates under high feature downsampling while maintaining stability at lower rates — demonstrating that the same outlier-AS coupling observed in text and vision extends to audio-visual processing.

Critical Assessment

The paper is a survey, not a study with a single set of hypotheses. A critical assessment must therefore evaluate whether its organizational framework and synthesized claims are supported by the evidence it marshals, and whether it adequately surfaces the limitations of the evidence base it draws upon.

Does the survey demonstrate that AS is a recurring structural phenomenon across architectures, or merely that similar attention patterns have been observed under different names?

The evidence for AS as a cross-architectural phenomenon is extensive in breadth but uneven in depth. The paper documents AS in at least six major architecture families (causal LLMs, classical LMs, MoE LLMs, MLLMs, ViTs, diffusion transformers) with consistent features: high attention concentration on low-semantic tokens, suppressed value norms of sink tokens, and co-occurrence with activation outliers. The consistency of the No-Op signature — high attention + low value norms — across BERT [29], LLaMA [28], ViT [29], and MLLMs [105] strongly supports the claim of a shared underlying mechanism.

However, the survey reports the existence of AS in each architecture rather than demonstrating that it operates identically. For instance, the "moving sinks" observed in diffusion language models [134] differ qualitatively from the static initial-token sinks in autoregressive LLMs — they shift positions during generation and removing them causes only minor degradation (in contrast to catastrophic collapse in LLMs). This difference is noted (Section 2.5) but not deeply analyzed. The survey could have been strengthened by a systematic cross-architecture comparison table documenting (a) sink token identity, (b) sink rate, (c) value-norm suppression ratio, (d) impact of sink removal, and (e) co-occurrence with activation outliers — all on a common set of metrics. The absence of such a table means the "recurring structural phenomenon" claim rests on qualitative consistency rather than quantitative uniformity.

Does the three-axis taxonomy (Utilization → Interpretation → Mitigation) genuinely reflect intellectual progression, or is it an imposed narrative on parallel, contemporaneous work?

Figure 3 provides temporal evidence: Utilization papers dominate 2023-2024, Interpretation papers grow in 2024-2025, and Mitigation papers emerge as the frontier in 2025-2026. This cumulative publication trend is consistent with the progression narrative. The causal logic also holds at the level of individual contributions — for example, Gated Attention [29] is explicitly motivated by the Softmax Limitations theory developed in the same paper, and Modified Softmax methods [162, 77] explicitly cite the No-Op Theory as their justification.

However, an alternative reading is possible: all three types of work have existed in parallel throughout, but the volume of each type has shifted over time. Early work by Clark et al. [53] (2019) already characterized attention to [SEP] in BERT in a way that anticipates later mechanistic interpretations, and preservation-like heuristics (intuitively retaining "important" tokens) predate the formalization of AS. The taxonomy cleanly separates papers that, in practice, often address multiple axes simultaneously — Quantizable Transformers [29] contributes to all three (characterizing AS in BERT and ViT, explaining it via No-Op Theory, and proposing Clipped Softmax and Gated Attention as mitigations). The survey tags each paper in Appendix A with multiple axes, acknowledging this overlap, but the cleanly separated Sections 3–5 may overstate the distinctness of the three research strands.

Does the paper adequately aggregate quantitative results, or does it rely on qualitative synthesis that obscures effect sizes?

This is the survey's most significant limitation. When the paper reports that Gated Attention reduces first-token attention from 46.7% to 4.8% [26], or that Softpick achieves a 0% sink rate [77], or that StreamingLLM reduces perplexity from 5,158 to 5.40 [24], it provides precise, comparable numbers. But for many surveyed methods — particularly in the Attention Redistribution and Sink Token Repurposing sections — the paper describes mechanisms and claims effectiveness without reproducing the supporting metrics from the original papers. For example, VAR [105] is described as "enhancing visual grounding and reducing hallucinations" without quoting the hallucination rate reduction. Surgery [171] is described as "preventing models from learning harmful patterns during fine-tuning" without specifying the refusal rate improvement. This is partly inherent to the survey format — 180+ papers cannot all be summarized with full result tables — but it means that effect sizes are not comparable across methods, which undermines the taxonomy's practical utility. A practitioner trying to choose between Attention Redistribution and Learnable Prefix Tokens for an MLLM hallucination problem cannot compare their quantitative impact from the survey alone.

Missing standardized evaluations. The survey identifies this as a future direction (Section 7.2: "Standardized Benchmark for AS and Outlier Mitigation") but does not itself propose metrics or test protocols. This is a chicken-and-egg problem: without standardized benchmarks, the survey cannot provide standardized comparisons; but the survey's role is precisely to motivate the creation of such benchmarks. The paper is honest about this limitation, but it means that claims like "X approach is more effective than Y" are almost entirely qualitative in the survey text.

Does the survey adequately distinguish between AS that is functional (the model benefits from it) and AS that is inert (the model tolerates it but would perform equally well without it)?

The survey presents the Anti-Overmixing theory (Section 4.5) and Geometric Anchoring (Section 4.4) as evidence that AS serves a functional role — preventing representational collapse and providing stable coordinate origins. But the Mitigation section demonstrates that several approaches (Gated Attention, Softpick, OSP) eliminate AS without apparent performance degradation on their evaluation tasks. This creates a tension: if AS is functional, why can it be eliminated without consequence? If AS is inert, why does evicting sink tokens cause catastrophic perplexity collapse (StreamingLLM)?

The survey does not fully resolve this tension. The likely resolution, suggested but not stated explicitly, is that AS is necessary under the standard Softmax architecture (removing sinks breaks the model because the model has learned to depend on them), but not necessary under architectures that provide alternative mechanisms for null updates or bias terms. The model "needs" AS only because it was trained with Softmax constraints; remove those constraints (gating, modified softmax, explicit bias), and the model learns alternative strategies that achieve the same functional ends (stable representations, constant offsets) without AS. This distinction between "necessary given the current architecture" and "necessary for the underlying task" is implicit throughout Sections 4–5 but could have been made more analytically precise.

Missing experiments that would have strengthened the survey's conclusions:

  • Cross-architecture ablation holding architecture constant: The survey observes AS in many architectures, but a controlled experiment varying only the architecture (e.g., causal vs. bidirectional masking) while holding model scale, training data, and task constant would directly test the claim that AS is an architectural necessity rather than a scale artifact. No such experiment exists in the surveyed literature.
  • Training trajectory analysis: The survey discusses training dynamics (e.g., Figure 42 for OSP, and KVSink's lifecycle analysis in Figure 29), but a systematic study tracking AS emergence across training steps for multiple architectures would provide causal evidence for the "emergence during pre-training" claim. Existing work provides snapshots (before/after training or at discrete checkpoints) rather than continuous trajectories.
  • Standardized mitigation comparison at matched scale: Gated Attention, Softpick, OSP, and Softmax-1 are all trained on different models at different scales with different evaluation protocols. A controlled comparison training all methods on the same base architecture at the same scale would be required to rank mitigation effectiveness — and this experiment has not been done.

Bottom line: The survey's central contribution is organizational and conceptual, not experimental. Its claims about the cross-architectural nature of AS, the causal chain from Softmax constraints to outlier circuits to sink behavior, and the cumulative progression from utilization to interpretation to mitigation are strongly supported by the breadth and consistency of the evidence it marshals. Its limitations lie in the depth of quantitative synthesis — effect sizes are not systematically aggregated or compared — and in the resolution of the tension between functional and inert interpretations of AS. These limitations are intrinsic to the survey format and are explicitly acknowledged as challenges requiring future work (Section 7), but they mean that the survey is a better map of the terrain than a guide to choosing specific coordinates based on quantitative trade-offs.

6. Limitations and Trade-offs

Limitation 1: The Survey Synthesizes Across Architectures Without Controlled Cross-Architecture Comparisons, So Claims of Generality Rest on Qualitative Consistency Rather Than Quantitative Uniformity

The assumption or constraint. The survey's central contribution — that AS is a "recurring structural phenomenon" across diverse Transformer architectures — is supported by documenting that AS appears in causal LLMs, classical LMs, MoE LLMs, MLLMs, ViTs, and diffusion Transformers (Sections 2.3–2.5). However, each architecture was studied independently by different research groups using different metrics, scales, and evaluation protocols. The survey does not report a single controlled comparison where architecture is the only variable varied — e.g., holding model scale, training data, and task constant while comparing causal vs. bidirectional masking. The paper acknowledges this implicitly by noting differences: "moving sinks" in diffusion language models "shift positions throughout the generation process rather than remaining fixed at the sequence start" and removing them "causes only minor performance degradation, contrasting sharply with the high sensitivity observed in autoregressive models" (Section 2.5). These differences are reported but not systematically analyzed.

The consequence. The claim that AS is the "same" phenomenon across architectures may obscure genuine architectural divergence. If AS in diffusion language models is qualitatively different from AS in autoregressive LLMs — different formation mechanisms, different functional roles, different sensitivity to removal — then techniques developed for one architecture may not transfer to the other. A practitioner working on diffusion Transformers cannot safely assume that sink preservation strategies validated on LLaMA (e.g., retaining initial tokens, Section 3.1) will work for their architecture, because the sinks may move during generation and their removal may not be catastrophic. The survey's taxonomy groups these architectures together at the phenomenon level, which facilitates cross-reference but risks over-unification — collapsing architecturally distinct behaviors under a single label and thereby encouraging misplaced generalization.

What evidence exists in the paper. The survey presents per-architecture AS characterization (Sections 2.3–2.5) with qualitative descriptions (attention maps, sink token identities, outlier magnitudes) but no cross-architecture quantitative table. For example, Section 2.3.2 reports that in LLaMA, the first token receives maximum attention in 98% of heads [162], but no comparable metric is reported for ViTs or MLLMs. Figure 6 (LLaMA attention logits), Figure 10 (MLLM visual attention sinks), and Figure 11 (ViT activation outliers) use different visualization formats, metrics, and scales, making direct comparison impossible. The absence of a unified quantitative summary — sink rate, value-norm suppression ratio, impact of sink removal, co-occurrence with activation outliers — across architectures means the "recurring phenomenon" claim rests on qualitative pattern-matching rather than measured uniformity.

Mitigation status. The paper does not address this directly. Section 7.2 identifies "Systematic Cross-Architecture and Cross-Modal Investigation" as a future direction, noting that "techniques developed for AS in one domain often remain confined to that specific domain" and that "systematic studies on cross-architecture and cross-modal transfer are needed to determine which methods generalize effectively and which require adaptation." This is an honest acknowledgment but defers the problem to future work rather than bounding the current survey's claims.

Limitation 2: The Paper Aggregates Findings Without Standardized Quantitative Comparisons, So Effect Sizes Are Not Comparable Across Methods

The assumption or constraint. The survey synthesizes over 180 studies, each of which reports results using its own metrics, baselines, model scales, and evaluation protocols. The survey's format — structured conceptual synthesis rather than meta-analysis — means it does not extract, normalize, or compare effect sizes across studies. When the paper reports that Gated Attention reduces first-token attention from 46.7% to 4.8% [26] or that Softpick achieves a 0% sink rate [77], it reproduces specific numbers from individual studies. But for most surveyed methods — particularly in the Attention Redistribution (Section 3.2) and Sink Token Repurposing (Section 3.4) sections — the paper describes mechanisms and qualitative outcomes without reproducing the supporting quantitative results. For example, VAR [105] is described as "enhancing visual grounding and reducing hallucinations" (Section 3.2.2) without quoting the hallucination rate reduction. Surgery [171] is described as "preventing models from learning harmful patterns during fine-tuning" (Section 3.4.2) without specifying the refusal rate improvement.

The consequence. A practitioner trying to decide between preservation, redistribution, learnable prefixes, or mitigation for a specific deployment cannot compare quantitative trade-offs from the survey alone. Is gated attention (which requires training from scratch) worth the 46.7% → 4.8% sink-rate reduction compared to zero-training-cost redistribution methods? The survey provides no common scale on which to evaluate this trade-off. This is not a failure of the survey's conceptual framework — the taxonomy remains valid regardless of quantitative comparisons — but it limits the survey's practical utility as a decision-making tool. The application guidelines in Section 6 provide qualitative recommendations ("apply Attention Redistribution to shift attention mass from AS tokens to informative visual tokens") but cannot specify how much improvement to expect relative to alternative approaches.

What evidence exists in the paper. The survey provides precise numbers for some headline results (StreamingLLM PPL reduction, Gated Attention sink-rate reduction, Softpick zero-sink-rate claim, Softmax-1 kurtosis reduction, TWEO outlier reduction) but qualitative descriptions for most others. Section 5.1.2 discusses Value-State Gated Attention [44] but does not quote its specific accuracy improvements. Section 3.2.2 describes ACT [154], ZeroTuning [75], and Pos2Distill [90] without reproducing their performance numbers. This is inherent to the survey format — 180+ full result tables cannot be reproduced — but it creates an asymmetry where some methods appear more empirically validated than others simply because the survey chose to quote their numbers.

Mitigation status. The paper identifies "Standardized Benchmark for AS and Outlier Mitigation" as a key future direction (Section 7.2), explicitly stating that "different mitigation strategies cannot be fairly compared in terms of efficacy, computational overhead, parameter introduction, or other critical factors." This candid admission acknowledges the limitation but defers its resolution to the community. The paper does not attempt to create even a partial quantitative comparison (e.g., a table of sink-rate reductions for methods that report comparable metrics), which would have partially addressed the limitation within the survey's scope.

Limitation 3: The Difficulty of Dynamic Sink Identification Is Undiagnosed, Making Preservation and Redistribution Methods Brittle in Deployment Scenarios Where Sink Positions Shift

The assumption or constraint. Sink Token Preservation (Section 3.1) and Attention Redistribution (Section 3.2) methods require identifying which tokens are sinks. The simplest methods assume sinks occupy fixed positions — e.g., the initial tokens in causal LLMs (StreamingLLM [24], Section 3.1.2) or background patches in ViTs (VAR [105], Section 3.2.2). The paper acknowledges that this assumption is fragile: "current AS detection methods assume static sink positions, but sinks can dynamically emerge at non-initial positions" (Section 3.1.3), citing KVSink [28] and Active-Dormant Attention Heads [163]. Methods that dynamically identify sinks exist (H2O [172] uses cumulative attention; KeyDiff [78] uses cosine similarity to the mean key), but the paper notes that "dynamic identification incurs additional computational overhead and can conflict with optimized kernels such as FlashAttention" (Section 3.1.3).

The consequence. The headline efficiency claims of preservation methods — StreamingLLM's ~957× perplexity improvement, MInference's 10× pre-filling speedup — assume sink positions are correctly identified. If sinks shift dynamically (as KVSink documents they can), a fixed-position preservation strategy will either retain the wrong tokens (wasting KV cache capacity on non-sinks) or evict true sinks (causing the catastrophic perplexity collapse that preservation was designed to prevent). Dynamic identification methods avoid this fragility but introduce an unquantified computational overhead that is not factored into the reported speedups. A deployment pipeline that must run H2O's cumulative attention scoring or KeyDiff's key-space similarity computation on every generation step may find that the "10× speedup" shrinks to 2× or less once detection cost is included. The paper does not quantify this overhead for any dynamic method.

What evidence exists in the paper. The paper explicitly identifies this limitation in Section 3.1.3 ("Limitations") and Section 3.2.3 ("Limitations"), but the evidence is qualitative — it cites studies that observe dynamic sink behavior (KVSink [28], Active-Dormant Attention [163]) without quantifying how often sinks shift, by how much, or what the detection cost is. Section 7.1 ("Challenges") reiterates that "efficient and accurate detection of dynamic sinks remains an open challenge" and that dynamic identification "incurs additional computational overhead." No experiment in the surveyed literature systematically measures the precision-recall trade-off of sink detection methods or characterizes the regimes where static-position assumptions fail.

Mitigation status. The paper identifies "Efficient and Lightweight AS Handling" as a future direction (Section 7.2), calling for "lightweight detection of dynamic sinks" that "minimize computational overhead while ensuring robustness across diverse inputs and layers." It also notes that attention redistribution methods "rely on the precise identification of sinks and target tokens" and that "most redistribution techniques operate on attention scores after Softmax, potentially conflicting with optimized attention kernels" (Section 3.2.3). These acknowledgments are honest but do not bound the practical impact of the limitation — a practitioner cannot determine from the survey whether dynamic sink identification will be a minor inconvenience or a deployment-blocker for their specific use case.

Limitation 4: Almost All Mitigation Methods Require Training from Scratch, Severely Limiting Their Applicability to Existing Pretrained Models

The assumption or constraint. The survey's mitigation strategies — Gated Attention Mechanisms (Section 5.1), Modified Softmax Functions (Section 5.2), Learnable Attention Bias (Section 5.3), and Pre-Training Interventions (Section 5.4) — all assume the ability to train a model from scratch with the mitigation integrated into the architecture or training recipe. The paper explicitly acknowledges this for each strategy: gated attention "requires training from scratch; gate parameters cannot be directly injected into pretrained models without retraining" (Section 5.1.3); modified softmax methods "require training from scratch, as the modified Softmax cannot be retrofitted into pretrained models without retraining" (Section 5.2.3); learnable attention bias "requires training from scratch, as the bias parameters cannot be retrofitted into pretrained models" (Section 5.3.3); and pre-training interventions "require training from scratch" (Section 5.4.3). Section 7.1 ("Challenges") consolidates this under "Training from Scratch and Adaptation Cost," noting that "lightweight adaptation techniques such as adapters or continual pre-training remain largely unexplored for AS."

The consequence. The mitigation strategies that the survey presents as the "natural endpoint" of the Utilization → Interpretation → Mitigation progression are inaccessible to the vast majority of practitioners who work with existing pretrained models (LLaMA, Qwen, GPT-OSS, DINOv2, etc.). Full retraining of a 7B+ parameter model is prohibitively expensive — costing millions of dollars in compute — and is only feasible for organizations that train foundation models from scratch. This means the survey's most principled solutions (eliminating AS at its root) are not deployable by the users who most need them: teams fine-tuning or serving pretrained models at scale. The deployment-accessible alternatives are the Utilization paradigms (preservation, redistribution, prefix tokens via fine-tuning), which work with AS rather than eliminating it and therefore inherit all the limitations of sink dependence — quantization sensitivity, representational capacity waste, and vulnerability to dynamic sink shifts.

What evidence exists in the paper. The survey does not quantify the cost of retraining. It does not estimate the additional FLOPs or wall-clock time required to train, e.g., a Gated Attention 7B model from scratch versus standard pre-training. It does not survey whether any of the mitigation methods have been successfully applied via fine-tuning (e.g., initializing a pretrained model's weights and training only the gate parameters for a fraction of the original training steps). The "training from scratch" constraint is stated as a qualitative fact without analysis of its practical severity. Section 7.2 identifies "Lightweight Adaptation for Pre-trained Models" as a future direction, suggesting adapters, LoRA, and continual pre-training as potential solutions — but notes that these "remain largely unexplored."

Mitigation status. The paper is transparent about this limitation for each mitigation strategy individually (Sections 5.1.3, 5.2.3, 5.3.3, 5.4.3) and collectively (Sections 7.1, 7.2). However, the survey structure itself may inadvertently oversell mitigation strategies by presenting them as the culmination of the field's progress without adequately weighting the feasibility gap. A reader who skims the taxonomy and sees Mitigation as the "Systematic Intervention" phase may not realize that these interventions are currently only available to organizations training foundation models from scratch. A more explicit upfront caveat — quantifying the scope of models that can benefit from each strategy phase — would have better set practitioner expectations.

Limitation 5: The Survey Does Not Resolve the Tension Between Functional and Inert Interpretations of AS

The assumption or constraint. The survey presents Mechanistic Interpretation (Section 4) as providing complementary perspectives on why AS exists, but these perspectives imply contradictory answers about whether AS is necessary or eliminable. The Geometric Anchoring theory (Section 4.4) and Anti-Overmixing theory (Section 4.5) suggest that AS serves a functional role — stabilizing the representational geometry and preventing information collapse. The No-Op Theory (Section 4.1) and Implicit Attention Bias (Section 4.3) suggest that AS is a workaround for an architectural limitation (lack of explicit bias terms, Softmax sum-to-one constraint) rather than an intrinsically necessary computation. The Mitigation section (Section 5) then demonstrates that several approaches (Gated Attention, Softpick, OSP) eliminate AS without apparent performance degradation on their evaluation tasks.

These findings are in tension: if sink tokens are necessary geometric anchors, why can they be eliminated without harm? If they are merely architectural workarounds, why does evicting them from a pretrained model cause catastrophic perplexity collapse (StreamingLLM [24], Section 3.1)? The resolution is likely that AS is necessary under the standard Softmax architecture (removing sinks breaks models that were trained with them) but not necessary under modified architectures that provide alternative mechanisms (gating, modified softmax, explicit biases). The model "needs" AS only because it learned to depend on it given the training constraints; change those constraints, and the model learns alternatives.

The consequence. The survey does not make this resolution explicit. The four interpretation theories are presented as "complementary perspectives" (Section 4.5) with a summary table organizing them by "analytical level" — but the table does not reconcile their conflicting implications for whether AS should be preserved or eliminated. A practitioner reading the Interpretation section might conclude that AS is geometrically essential and should not be tampered with; reading the Mitigation section, that AS is an eliminable artifact and should be engineered away. The survey provides no framework for determining when AS is functional versus inert, or which architectures require sink-like mechanisms even after mitigation (e.g., do modified-softmax models still form geometric anchors through other means?).

What evidence exists in the paper. The tension is visible in the juxtaposition of specific claims. Section 4.4.1 states that "the first token anchors the residual stream, allowing diverse token representations to be maintained even in deep layers" and that "AS thus emerges as a structural adaptation essential for preserving expressive power in autoregressive Transformers" [27]. Section 5.1.2 then reports that Gated Attention reduces first-token attention from 46.7% to 4.8% without degrading performance — implying that the anchoring function is either not essential (the "diverse token representations" are maintained through other mechanisms) or is performed by the gating mechanism itself (the gate, not the sink token, now modulates the residual stream). The survey does not analyze which of these interpretations is correct or survey evidence that would distinguish them. The OSP framework [42] (Section 5.4.2) trains "the first production-scale LLM without extreme activation outliers" and presumably without sink anchoring, yet achieves competitive performance — but the survey does not analyze whether OSP models develop alternative anchoring mechanisms or simply don't need them.

Mitigation status. The survey does not explicitly identify this as a limitation. Section 7.1 notes that "an incomplete understanding of training dynamics" is a challenge, but frames it as a gap in formalizing "mutual reinforcement mechanisms between attention scores and value states" rather than as a fundamental interpretive tension. Section 7.2 calls for "Theoretical Formalization of Training Dynamics" and a "Unified Theoretical Framework" — both of which would help resolve the functional vs. inert question — but does not flag the current ambiguity as an obstacle to principled method selection. This is perhaps the survey's most significant analytical gap: it taxonomizes without synthesizing the causal implications of competing interpretations for downstream engineering decisions.

Limitation 6: The Application Guidelines Are Qualitative Mappings Without Trade-Off Quantification, Limiting Their Use as Decision Support

The assumption or constraint. Section 6 ("Applications and Practical Guidelines") maps AS techniques to nine application domains (pre-training, tuning, inference, interpretability, hallucination reduction, safety, capability enhancement, long-context enhancement, multimodal enhancement) and provides qualitative recommendations. For example, Section 6.5 recommends: "For MLLMs suffering from visual hallucinations, apply Attention Redistribution to shift attention mass from AS tokens to informative visual tokens [30, 31, 32, 107]." These recommendations are context-free: they do not specify conditions under which one approach is preferable to another, quantitative trade-offs between approaches, or failure modes to watch for. The guidelines treat each technique as applicable to its listed domains without discussing when it might not work — e.g., Attention Redistribution assumes accurate sink identification (Limitation 3) and kernel compatibility (Section 3.2.3), which may make it unsuitable for latency-sensitive deployments using FlashAttention.

The consequence. A practitioner following the guidelines literally might apply Attention Redistribution to an MLLM hallucination problem without realizing that (a) Sink Token Repurposing (Section 6.5: "leverage dense visual AS heads in shallow layers to maintain global context and reduce hallucination via Sink Token Repurposing [108, 115]") is an alternative that exploits rather than suppresses AS, and (b) the choice between these approaches depends on whether visual AS in their specific model are primarily harmful (favoring redistribution) or structurally beneficial (favoring repurposing). The guidelines do not provide decision trees, flowcharts, or conditional recommendations that would help a practitioner navigate these choices.

What evidence exists in the paper. Section 6 is organized by application domain, with each subsection listing relevant techniques and citing papers. There are no comparative tables, no quantitative performance comparisons across techniques applied to the same domain, and no discussion of domain-specific failure modes. For example, Section 6.8 ("Long-Context Enhancement") lists Sink Token Preservation, Learnable Prefix Tokens, Learnable Attention Bias, Sink Token Repurposing, Gated Attention, and Modified Softmax as all applicable — without indicating that Gated Attention and Modified Softmax require training from scratch (Limitation 4) while Preservation and Repurposing do not, or that these categories entail fundamentally different deployment costs. The guidelines effectively say "all techniques that have been applied to long-context tasks can be applied to long-context tasks," which is true but not actionable.

Mitigation status. The paper does not acknowledge this as a limitation. Section 6 is presented as a practical contribution ("provides practical, actionable guidelines for managing AS") without caveats about the abstraction level or missing trade-off characterization. This contrasts with the rest of the survey, which is generally candid about limitations (Sections 3.1.3, 3.2.3, 5.1.3, 5.2.3, etc.). A more useful version of Section 6 would provide conditional recommendations — e.g., "If you are deploying a pretrained causal LLM and cannot retrain, prefer Sink Token Preservation for KV cache compression (low overhead, proven robustness) over Gated Attention (requires training from scratch). If you are training a ViT from scratch, prefer Learnable Prefix Tokens (register tokens) over redistribution (training-free but kernel-incompatible)." The survey's stated goal is to provide "practical, actionable guidance" (Section 1.2), but the guidance it provides is at the level of literature pointers, not decision support.

7. Implications and Future Directions

How This Work Changes the Landscape

This survey does not introduce a new method, a new dataset, or a new empirical result. Its contribution is conceptual infrastructure: the first systematic map of a field that has grown rapidly but chaotically, connecting findings across six architecture families, four utilization paradigms, four interpretation theories, and five mitigation strategies that were previously studied in isolation. The magnitude of this contribution is best understood not as a paradigm shift — the underlying phenomena (attention concentration on uninformative tokens, activation outliers, the Softmax constraint) were already known — but as a reframing that converts a collection of independent observations into a coherent causal narrative with actionable structure.

What changes. Before this survey, the AS literature was characterized by three fragmentation patterns, each of which the survey directly addresses:

  • Terminological fragmentation: The same phenomenon was called "attention sink" in efficient LLM inference, "massive activations" in mechanistic interpretability, "outlier dimensions" in BERT analysis, and "heavy hitters" in KV cache compression. The survey establishes a unified definition (Section 2.2.2, Equation 5) with a threshold-based detection criterion that can be applied across architectures, giving the field a shared vocabulary and a falsifiable operationalization of its central concept.

  • Architectural fragmentation: Researchers studying AS in causal LLMs rarely cited work on AS in ViTs or BERT, despite the underlying Softmax constraint being identical across architectures. The survey's Section 2 systematically documents AS manifestations in each architecture family, and its Section 4 demonstrates that the No-Op Theory, Outlier Circuits, Implicit Attention Bias, and Geometric Anchoring apply across all of them — making it impossible for future work to claim AS is peculiar to a single architecture without addressing the cross-architectural evidence the survey has assembled.

  • Methodological fragmentation: Utilization strategies (preservation, redistribution), interpretive theories (Softmax limitations, outlier circuits), and mitigation strategies (gated attention, modified softmax) were developed by different communities with different goals. The survey's cumulative taxonomy (Figure 3, showing Utilization → Interpretation → Mitigation as a temporal progression) reveals that mitigation strategies are explicit responses to interpretive theories — gated attention directly addresses the No-Op Theory, learnable attention bias directly addresses the Implicit Attention Bias perspective, pre-training interventions directly target the Outlier Circuits. This makes the dependency structure of the field visible: mitigation research presupposes interpretive understanding, which presupposes systematic empirical characterization. Future work that proposes a mitigation without engaging the relevant interpretive theory will be recognizable as incomplete.

Reconciling prior contradictions. The survey resolves the apparent contradiction between studies showing AS is functional (Anti-Overmixing theory [27]: removing the first token causes representational collapse) and studies showing AS is eliminable (Gated Attention [26], Softpick [77], OSP [42]: eliminating sink tokens does not degrade performance). The resolution, implicit in the survey's structure but available for explicit articulation, is that AS is necessary under the standard Softmax architecture but not necessary under architectures that provide alternative mechanisms for null updates or bias terms. StreamingLLM [24] shows catastrophic perplexity collapse when sinks are evicted from a standard model — because the standard model learned to depend on them. But models trained from scratch with gated attention or modified softmax learn alternative strategies that achieve the same functional ends (stable representations, constant offsets, anti-overmixing) without concentrating attention on uninformative tokens. This reframing eliminates the contradiction and points to a testable prediction: if a model trained with Gated Attention from scratch is fine-tuned on a task where standard models rely heavily on AS, it should develop alternative mechanisms (e.g., gate-mediated suppression, geometric anchoring through learned bias terms) rather than recreating AS.

Which research directions become more attractive.

  • Attractive direction: Lightweight post-hoc adaptation. The survey makes clear that current mitigation methods require training from scratch (Sections 5.1.3, 5.2.3, 5.3.3, 5.4.3), and Section 7.2 identifies lightweight adaptation as the most urgent open challenge. This direction is now obviously attractive because the taxonomy has identified a large class of practitioners (everyone deploying pretrained models) who are excluded from the most principled mitigation strategies. A method that could inject learnable attention biases or gate parameters into a pretrained LLaMA or Qwen model via LoRA fine-tuning — and achieve even 50% of the sink-rate reduction that training from scratch achieves — would immediately become the default recommendation for deployment teams.

  • Attractive direction: Standardized AS benchmarks. The survey repeatedly identifies the absence of standardized evaluation as a bottleneck (Sections 5.1.3, 5.2.3, 5.3.3, 7.1). Before this survey, the need for such benchmarks was an intuition; after the survey, it is a documented gap supported by the observation that no two mitigation papers report comparable metrics (sink rate vs. activation kurtosis vs. attention concentration vs. perplexity penalty under quantization). A benchmark that specified a common protocol — e.g., measure sink rate, activation kurtosis, perplexity, and quantization fidelity on a fixed set of models (LLaMA-2-7B, ViT-B/16, LLaVA-1.5) under a fixed set of interventions — would enable the first quantitative ranking of mitigation strategies that the survey itself cannot provide.

  • Less attractive direction: Novel search or optimization methods for managing AS at inference time. The survey's synthesis of the Outlier Circuits framework (Section 4.2) and the No-Op Theory (Section 4.1) shows that AS is a training-level phenomenon sustained by mutually reinforcing weight, activation, and attention outliers, not a surface-level pattern that can be eliminated by clever inference-time heuristics. Methods that attempt to dynamically detect and suppress sinks at inference (a natural extension of Attention Redistribution, Section 3.2) face an inherent ceiling: they operate after the outlier circuit has already formed, so they can mitigate symptoms but not the root cause. The survey's structure implies that the highest-impact research will target training-time interventions (pre-training, fine-tuning) rather than inference-time workarounds.

  • Less attractive direction: Architecture-specific AS studies. Before this survey, a paper documenting AS in a new architecture (e.g., "Attention Sinks in 3D Vision Transformers" or "Massive Activations in State-Space Models") could claim novelty by virtue of being the first to observe the phenomenon in that domain. After this survey, such papers must now demonstrate that the AS they observe is qualitatively different from the cross-architectural patterns already documented in Sections 2.3–2.5 — otherwise they are filling in a known template rather than advancing understanding. The survey has raised the bar for observational AS studies by establishing the expected signature (high attention + low value norms + activation outliers) that any new architecture must either match or deviate from in instructive ways.

Follow-Up Research This Work Enables

1. A controlled cross-architecture study holding scale, data, and task constant to test whether AS emergence is driven by architectural constraints (causal masking, Softmax) or by training dynamics (optimizer, data distribution). The survey documents AS in six architecture families (Sections 2.3–2.5) but cannot distinguish between the hypothesis that AS is an architectural necessity (forced by causal masking and Softmax constraints, as the No-Op Theory in Section 4.1 predicts) and the hypothesis that AS is a training-dynamics artifact (emergent from Adam-style optimization interacting with specific data distributions, as the Outlier Circuits in Section 4.2 and the OSP results in Section 5.4.2 suggest). A strong follow-up would train three architectures — causal decoder-only, bidirectional encoder-only, and encoder-decoder — on identical data at identical scale (~1B parameters, ~100B tokens) with identical optimizer settings, then measure sink rate, activation kurtosis, value-norm suppression ratio, and impact of sink eviction across all architectures. The causal masking hypothesis predicts that only the causal decoder develops strong initial-token AS, with the bidirectional encoder developing distributed sink patterns on special tokens and the encoder-decoder showing cross-attention-specific sinks. The training-dynamics hypothesis predicts similar outlier magnitudes across architectures. The survey makes this experiment tractable by providing the measurement protocol (Equation 5, the two defining characteristics from Section 2.2.2) and the expected signatures for each architecture.

2. Injecting learnable attention biases or gate parameters into pretrained LLaMA-2-7B via LoRA fine-tuning and measuring sink-rate reduction as a function of fine-tuning tokens. The survey's primary deployment bottleneck is that all mitigation strategies require training from scratch (Sections 5.1.3, 5.2.3, 5.3.3, 5.4.3, and Limitation 4 above). A strong follow-up would take a pretrained LLaMA-2-7B model, inject (a) learnable key-value biases as in Massive Activations [98] (Section 5.3.2, Equation 55) or (b) a head-wise scalar gate as in Gated Attention [26] (Section 5.1.2, Equation 47) using LoRA adapters on the attention projections, and fine-tune on a standard corpus (e.g., C4, The Pile) for increasing token budgets (1M, 10M, 100M, 1B tokens). At each checkpoint, measure first-token attention concentration (the metric Gated Attention reduced from 46.7% to 4.8% when trained from scratch), activation kurtosis, and perplexity. The key question is whether 100M fine-tuning tokens — a tiny fraction of the 3.5T tokens used for from-scratch training in [26] — can achieve even 50% of the sink-rate reduction. If yes, the deployment bottleneck is solved for the vast majority of practitioners. If no, the result quantifies the adaptation cost and establishes a lower bound on the token budget required, informing whether the community should invest in more efficient post-hoc methods or accept that AS mitigation requires training from scratch.

3. A standardized AS mitigation benchmark measuring sink rate, activation kurtosis, perplexity, and quantization fidelity on LLaMA-2-7B, ViT-B/16, and LLaVA-1.5 under Gated Attention, Softmax-1, Softpick, OSP, and Learnable Attention Bias, all trained at matched scale. The survey's most significant quantitative limitation is the absence of comparable effect sizes across mitigation methods (Limitation 2 above). A strong follow-up would train each mitigation method on the same base architecture at the same scale (~7B parameters for language, ~86M parameters for vision) on the same data, measure a common set of metrics, and produce the first head-to-head ranking. The metrics should include: (a) sink rate (Equation 5), (b) activation kurtosis (as in Softmax-1 [162]), (c) first-token attention concentration (as in Gated Attention [26]), (d) perplexity on held-out validation data, (e) downstream task accuracy (e.g., MMLU for language, ImageNet for vision), and (f) perplexity penalty under 4-bit weight and KV cache quantization (as in RotateKV [84] and KVSink [28]). This benchmark would directly address the open challenge identified in Section 7.2 and would provide the quantitative decision support that Section 6 currently lacks. A negative result — e.g., finding that no single method dominates across all metrics, with Softpick best for sink-rate reduction but Gated Attention best for quantization fidelity — would be equally informative, revealing that practitioners must choose mitigation strategies based on their specific deployment constraint (memory vs. accuracy vs. latency).

4. A training-trajectory analysis tracking the emergence of Outlier Circuits across 100+ checkpoints during LLaMA-2-7B pre-training, measuring the precise iteration when weight outliers, activation outliers, and attention outliers first appear and quantifying their mutual reinforcement dynamics. The survey's Outlier Circuits framework (Section 4.2) describes a four-step causal chain (weight outliers → activation outliers → attention outliers → value-norm suppression) and KVSink's lifecycle analysis (Figure 29) characterizes the cross-layer progression after training is complete. But the training dynamics that produce this circuit — when during optimization each component emerges, whether they emerge simultaneously or sequentially, and whether the mutual reinforcement is gradual or exhibits a phase transition — are undocumented. A strong follow-up would save checkpoints every 1,000 optimization steps during a LLaMA-2-7B pretraining run and measure (a) the kurtosis of W_down column norms (weight outlier emergence), (b) the maximum activation magnitude in each layer's residual stream (activation outlier emergence), (c) the fraction of attention allocated to the initial token (attention outlier emergence), and (d) the ratio of sink-token value norms to non-sink-token value norms (value suppression). The prediction from the No-Op Theory is that attention outliers and value-norm suppression emerge simultaneously — the model learns to concentrate attention on sink tokens and suppress their value vectors as a coordinated strategy. The prediction from the training-dynamics alternative is that weight outliers emerge first (driven by Adam's privileged bases), activation outliers follow (amplified by weight outliers through the down-projection), and attention concentration follows activation outliers (as the key-query dot product is dominated by outlier feature dimensions), with value-norm suppression emerging last as a consequence of attention concentration. Distinguishing these trajectories would determine whether interventions should target the optimizer (preventing weight outliers from forming, as OSP does) or the attention mechanism (preventing attention concentration even after activation outliers exist, as gated attention does) — a decision with direct engineering consequences.

5. A stress-test of the AS-as-architectural-necessity claim by training a causal LLM from scratch with Sigmoid Attention (no Softmax, no sum-to-one constraint) and measuring whether it develops alternative geometric anchoring mechanisms. The survey presents Geometric Anchoring (Section 4.4) as a functional role AS serves — sink tokens act as stable reference points that structure the representational geometry, and the Anti-Overmixing Theory (Section 4.5) argues that the first token prevents representational collapse. If these functional roles are genuinely necessary for autoregressive Transformers, then a model trained without AS (via Sigmoid Attention or Softpick) should develop some alternative mechanism to achieve the same functions — perhaps through learned positional biases, through gating structures that provide stable offsets, or through distributed geometric anchors that are not identifiable as single sink tokens. A strong follow-up would train a 1B-parameter causal LLM from scratch with Sigmoid Attention (Equation 53) or Softpick (Equation 52), and at convergence measure: (a) whether the representational geometry exhibits stable reference points (using OrthoRank's cosine similarity analysis [70], Section 4.4.1), (b) whether removing early tokens still causes perplexity collapse (the StreamingLLM test), and (c) whether the model's representational diversity across token positions is maintained or collapses in deep layers (the Anti-Overmixing diagnostic from [27]). If the model develops alternative anchoring (e.g., learned position-dependent biases that serve the same geometric role without attention concentration), it demonstrates that AS is one solution to a deeper representational requirement, not the requirement itself — a finding that would redirect mitigation research toward identifying and providing that requirement through cleaner mechanisms. If the model shows representational collapse or degraded deep-layer diversity, it demonstrates that the Softmax constraint is not the only source of geometric anchoring needs, and that future architectures must provide explicit geometric stabilization mechanisms even after eliminating AS.

6. A cross-modal transfer study applying Sink Token Repurposing techniques developed for text LLMs (backdoor triggers, defensive regularization) to MLLMs and measuring whether sink-aligned attacks transfer across modalities. The survey documents that shell companies — AS tokens in text-only LLMs (initial tokens) and visual AS in MLLMs (background patches) — are structurally analogous (both exhibit high attention + low value norms + activation outliers, Sections 2.3.2 and 2.3.4) and that Sink Token Repurposing exploits this structural role for attacks and defenses (Section 3.4). Forgetting to Forget [33] showed that placing backdoor triggers at text-side sink positions enhances attack persistence during unlearning. A strong follow-up would test whether the same principle transfers to MLLMs: inject a visual backdoor trigger (a specific pixel pattern) at visual sink positions (background patches identified via Massive Activation detection, as in VAR [105]), fine-tune the MLLM on harmful content with this trigger, apply an unlearning procedure, and measure whether the backdoor persists more strongly than triggers placed at non-sink visual positions. If the transfer succeeds, it demonstrates that the sink-as-gateway vulnerability is a cross-modal property of the Softmax attention mechanism, not a peculiarity of text tokens — with direct implications for MLLM security. If it fails, it reveals that visual and text sinks, despite structural similarities, have modality-specific properties that limit attack transfer, constraining the scope of Sink Token Repurposing as a unified framework.

Practical Applications and Downstream Use Cases

1. KV cache compression for long-context LLM inference at scale. The survey's synthesis of Sink Token Preservation (Section 3.1) directly informs a deployment pattern that is already used in production LLM serving systems. The pattern: permanently retain initial sink tokens (typically positions 0–3) in the KV cache at full precision while aggressively quantizing or evicting other tokens. The quantitative justification from the survey: StreamingLLM [24] demonstrates that retaining just 4 sink tokens alongside a 2,048-token sliding window achieves PPL of 5.40 on long texts, compared to 5,158 without sink preservation — a ~957× reduction in perplexity from preserving 4 tokens. RotateKV [84] achieves 2-bit KV cache quantization with minimal degradation by keeping sink tokens at full precision. For a deployment serving 100M daily inference requests with average context length 8K tokens, the memory savings from 2-bit quantization of non-sink tokens (reducing KV cache size by ~4×) while protecting the critical sink tokens translates directly to reduced GPU memory footprint and increased batch sizes, with quantified perplexity impact from the surveyed numbers.

2. Hallucination reduction in multimodal LLMs without retraining. The survey documents a training-free intervention pattern for MLLM hallucination (Sections 3.2, 6.5): identify visual sink tokens (background patches with high attention and low semantic content) using Massive Activation detection as in VAR [105], and redistribute their attention mass to foreground objects using explicit redistribution (Equation 20 with α = 0, β = 1). The benefit is that this requires no model fine-tuning — it operates on attention scores during inference — and therefore applies to any pretrained MLLM (LLaVA, Qwen-VL, GPT-4V if API access to attention weights is available). The survey cites AttnReal [31] (IJCV 2026) and VAR [105] (ICLR 2025) as demonstrating hallucination reduction comparable to methods requiring retraining. For deployment teams running MLLMs in high-stakes visual QA (medical imaging, autonomous driving, document understanding), this intervention can be deployed immediately as a post-processing step on attention outputs, with the only deployment requirement being access to the model's internal attention weights. The quantified speedup depends on the specific implementation, but the key practical insight from the survey is that this approach exploits the same outlier-AS coupling (Section 4.2.2) that VAR used for identification, making the intervention principled rather than heuristic.

3. Quantization-aware pre-training for edge deployment of small language models. The Outlier-Safe Pre-Training (OSP) framework [42] documented in Section 5.4.2 trains a 1.4B model without activation outliers, enabling robust 4-bit quantization that would otherwise fail due to outlier-dominated dynamic ranges. The survey's framing makes this a template for any organization training small models for edge deployment: adopt Muon Optimizer to eliminate privileged weight bases, replace per-channel RMSNorm scales with a single scalar per layer (Equation 60), and add a learnable embedding projection to redistribute initial activation magnitudes. The quantified benefit from Section 5.4.2: the first production-scale LLM without extreme activation outliers, trained on 1 trillion tokens. For a team training a 1B–3B parameter model for on-device deployment (smartphone, IoT, automotive), following the OSP recipe eliminates the need for post-training quantization techniques (which work around outliers rather than preventing them) and enables direct W8A8 or W4A8 quantization that would otherwise require outlier-aware mixed-precision schemes. The throughput improvement from TWEO (36%, Section 5.4.2) provides a concrete estimate of the training efficiency gain from outlier-free pre-training.

4. Register token injection for cleaner ViT attention maps in dense prediction tasks. The survey documents that adding 4 register tokens during ViT pre-training (Section 3.3.2, [126]) absorbs attention artifacts that would otherwise concentrate on background patches, producing attention maps that focus on foreground objects rather than uninformative regions (Figure 19). For teams deploying ViTs in dense prediction tasks — semantic segmentation, depth estimation, object detection — where clean spatial attention is directly linked to output quality, injecting register tokens during fine-tuning (via Self-Distilled Registers [123], Equation 29) or even at test time (Test-time Registers [121]) is a low-overhead intervention (4 tokens × D parameters, negligible relative to model size) with visualized benefits. The survey makes this approach accessible by documenting both the pre-training and post-hoc variants, so teams that inherited a pretrained ViT can apply the post-hoc version without retraining, while teams training from scratch can adopt the pre-training version as a default architectural component (as DINOv3 [118] already does). The quantified improvement on dense prediction tasks is not directly reproduced in the survey, but the qualitative attention map improvement (Figure 19) provides sufficient motivation for practitioners in vision domains where attention interpretability matters.