Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing

🎯 Pitch

Transformer attention heads produce crippling activation outliers because they try to act as no-ops by saturating the softmax to near-zero, a behavior that forces preceding layers to generate unbounded inputs. The authors introduce clipped softmax and gated attention, two architectural changes that let models explicitly output exact zeros without large inputs, eliminating outliers pre-training and enabling straightforward W8A8 quantization with no loss.

1. Executive Summary

This paper diagnoses and eliminates the activation outlier problem that prevents transformers from being easily quantized to INT8, proposing two architectural modifications—clipped softmax (stretching the softmax output to allow exact zeros at finite input range) and gated attention (an explicit learned sigmoid gate modulating each attention head's output)—that can be dropped into standard transformer pre-training. The authors analyze and train BERT-base, OPT, and ViT models, demonstrating that in almost all cases both methods reduce maximum infinity norm and kurtosis of attention-layer outputs by orders of magnitude while maintaining or improving floating-point performance, yielding W8A8 post-training quantized models within 1% of the original accuracy without additional fine-tuning or mixed-precision workarounds. The core mechanistic insight is that attention heads learn to attend to delimiter tokens or uninformative background patches as a "no-op" on the residual, driving softmax inputs toward infinity and producing strong outliers in the feed-forward network outputs of the preceding layer—establishing that outlier-free quantization is achievable by giving the model explicit, finite-range mechanisms to produce near-zero attention outputs rather than forcing it to approximate them through unbounded softmax saturation.

2. Context and Motivation

The Core Problem: Activation Outliers Block Simple Quantization of Transformers

The paper addresses a concrete, practical obstacle that confronts anyone trying to make transformer models run efficiently: modern transformers develop extreme outliers in their activations during training, and these outliers make quantizing those activations to low bitwidths (like 8-bit integer) catastrophically difficult. The problem is not just that quantization becomes slightly less accurate — it's that without addressing outliers, a quantized model's performance can collapse entirely, with perplexity exploding from ~4.5 to >1000 in the BERT-base case, as shown by the baseline W8A8 results in Table 2.

This is a specific technical problem within the broader challenge of making large transformer models deployable. The paper's introduction frames it crisply: "quantizing transformers is not always trivial. When quantizing the activations of a transformer, significant problems arise with outliers in specific layers." The key observation that makes this frustrating rather than merely challenging is that these outliers are highly structured — they appear in only a handful of embedding dimensions (often fewer than 10 out of 768 for BERT-base), consistently across layers and input sequences, and account for the vast majority of the tensor's dynamic range. Because of these few extreme values, a per-tensor quantizer faces an impossible choice: set the quantization range wide enough to cover the outliers (incurring massive rounding error on the 99%+ of non-outlier values) or clip the outliers (incurring massive clipping error). Either way, accuracy degrades severely.

The problem's practical significance stems from two converging trends. First, quantization is the primary technique for reducing the inference cost of neural networks, enabling deployment on edge devices, reducing datacenter power consumption, and lowering latency. As Horowitz [23] quantified and the paper cites, fixed-point integer operations are substantially more energy-efficient than floating-point operations — a fact that makes INT8 inference highly desirable for production systems. Second, transformer models keep getting larger, making the inference cost problem more acute. The paper cites prior work showing that modern large language models exhibit outliers "after every linear layer, including query, key, and value projection layers" [13], meaning the problem compounds with model scale rather than being confined to a few isolated layers.

The theoretical significance is equally compelling. The presence of these outliers is not a random artifact of training dynamics — it reflects something the network is actively trying to learn. The paper sets out to understand why the network learns outliers, which makes this a diagnostic investigation as much as an engineering solution. If outliers are a symptom of a limitation in the standard transformer architecture that forces the model to work around it via pathological activation values, then fixing the architecture could address the root cause rather than patching the symptoms.

The Status Quo: Patch the Quantizer, Not the Model

Prior to this work, the research community had developed a repertoire of techniques to make quantization work despite outliers, but all of them came with significant trade-offs. The paper groups these into families:

Finer quantization granularity. Rather than using a single scale factor per tensor (per-tensor quantization), approaches like GPT3.int8() [13] use per-channel or per-token quantization, where each row or column gets its own quantization parameters. This allows the non-outlier dimensions to use a tight range while the outlier dimensions use a wide range — essentially partitioning the problem rather than solving it. The cost, as the paper notes, is that "this requires implementing specific hardware for input-channel quantization" or more complex mixed-precision kernels. General-purpose integer matrix multiplication hardware expects uniform quantization across the reduction dimension, so per-channel quantization requires custom kernels that may not map efficiently to existing accelerators.

Mixed-precision or higher-bitwidth activations. Another family of solutions keeps outlier-prone activations in FP16 or uses a non-standard numeric format while quantizing everything else. While this preserves accuracy, it fundamentally undermines the benefit of quantization — you're still paying for floating-point operations on some fraction of the network, and the implementation complexity of mixing precisions in a single graph is substantial.

Post-training fixes that massage the model. SmoothQuant [67] introduces per-channel scaling factors between weights and activations to smooth the outliers across dimensions, reducing the maximum dynamic range at the cost of making the weight distributions slightly less uniform. Outlier Suppression [62, 63] applies shifting and scaling transformations to the activation tensors post-hoc. These approaches work, but the paper characterizes them as "frequently tedious and either require retraining the network, require implementing specific hardware... or require parts of the activations to still be in higher bit-widths."

Quantization-aware training (QAT). Instead of post-training fixes, you could train the network from scratch with simulated quantization noise, which indeed produces quantizable models. However, QAT requires running the full training pipeline — access to training data, hyperparameters, and compute budget — which isn't always available, especially for practitioners who receive a pre-trained model checkpoint and want to deploy it.

The critical limitation shared by all these approaches is that they treat the outlier as a quantization problem, not as a model design problem. They adapt the quantization scheme to accommodate outliers rather than preventing outliers from arising in the first place. This is the gap the paper identifies: "In contrast, in this work, we want to address the root cause of the problem and understand why outliers are learned in the first place."

A Deeper Gap: No Causal Explanation for Outliers Existed

Before this paper, the community knew that outliers exist, where they appear (specific hidden dimensions), and that clipping them hurts task performance [31, 49]. What was missing was a mechanistic explanation for why the network creates them. Without understanding the functional role of outliers — what the network is using them to accomplish — it's impossible to design an architectural fix that preserves the desired behavior while eliminating the pathological side effect.

The paper cites prior outlier detection and characterization work [4, 13, 31] that established their existence and empirical properties, but notes that these works didn't explain the causal chain from training objective → learned behavior → outlier emergence. Similarly, attention analysis work by Clark et al. [8] and Kovaleva et al. [30] had observed that attention heads sometimes attend exclusively to delimiter tokens like [SEP] or punctuation and speculated this acts as a "no-op," but this observation hadn't been connected to the outlier phenomenon or quantization difficulty.

This gap is significant because without a causal understanding, any attempt to remove outliers risks removing functionally important behavior. If outliers are, as some prior work suggested, "crucial for model predictions" and setting them to zero "significantly degrades model task performance" [31, 49], you can't just clip them or regularize them away — you need to give the model an alternative way to achieve the same goal.

Specific Shortcomings of Prior Approaches That This Paper Addresses

Moving beyond general categorizations, the paper identifies several specific technical shortcomings that motivate its two proposed solutions:

The softmax function cannot express exact zeros with finite inputs. This is a mathematical property, not an engineering choice. For any finite vector $x$ , $\text{softmax}(x)_i > 0$ for all $i$ . If an attention head wants to produce an output that is exactly zero for a given token — to perform a genuine no-operation on the residual — it must drive the softmax input toward infinity for the non-attended tokens. The paper formalizes this observation in Equation 2: $\text{softmax}(x)_i = 0 \iff \exists j \neq i, \; x_j - x_i = +\infty$ . In practice, "near-zero" requires large-but-finite values, and these large values propagate backward through the network.

Standard attention lacks an explicit "abstain" mechanism. In a residual network, the identity mapping is the default — adding zero to the hidden state preserves it unchanged. But the attention mechanism as defined by Vaswani et al. [60] always produces some output. Even if the values $V$ for delimiter tokens are small, the attention-weighted sum is never precisely zero. The model has to learn workarounds: attend to special tokens, push the value projections for those tokens toward zero, and push the softmax inputs toward extremes to saturate the attention distribution. Each of these workarounds introduces numerical pathologies somewhere in the network.

LayerNorm amplifies the problem into a feedback loop. The paper's Figure 4 diagrams this precisely: outliers from the FFN output in layer $L$ pass through LayerNorm (which normalizes them, reducing their magnitude relative to the tensor statistics), then through the attention mechanism of layer $L+1$ , which needs the large dynamic range in the softmax input to achieve the desired attention distribution. Since LayerNorm dampens magnitudes, the FFN of layer $L$ has to produce even larger outliers at its output to survive normalization and still provide sufficient dynamic range downstream. This creates an escalation: "as softmax will never output exact zeros, it will always back-propagate a gradient signal to grow bigger outliers. The outliers will thus tend to become stronger in magnitude, the longer the network is trained."

The problem spans architectures and modalities. The paper demonstrates the same pattern — attention heads attending to low-information tokens/paches with small value outputs, creating outliers in preceding layers — in BERT (delimiter tokens like [SEP], periods, commas), ViT (background patches), and OPT (the causal LM variant). This universality suggests the problem is architectural, not domain-specific. Any transformer with softmax attention, residual connections, and LayerNorm will tend to develop this behavior given sufficient training, because the architecture lacks a clean way to express "skip this update."

How This Paper Positions Itself

The framing in Sections 1 and 3 makes explicit that this is a root-cause analysis paper with architectural solutions, not another quantization workaround paper. The key sentence is: "We show that strong outliers are related to very specific behavior of attention heads that try to learn a 'no-op' or just a partial update of the residual."

The paper doesn't compete with SmoothQuant, GPT3.int8(), or QAT approaches on their own terms (better quantization schemes). Instead, it proposes to change the source distribution so that no special quantization handling is needed in the first place. The goal is stated plainly in the introduction: "We hope to make transformers easy to quantize from the get-go without needing any post-processing."

The specific positioning relative to the literature includes:

Vs. post-training quantization fixes [13, 62, 63, 67]: These adapt quantization to work with outliers. This paper changes the model so outliers don't arise. The methods are complementary — you could combine them — but the paper argues that architectural fixes are more fundamental and eliminate the need for the more complex quantization schemes.
Vs. quantization-aware training [3, 16, 26]: QAT forces the model to be robust to quantization noise during training. This paper achieves quantizability without simulating quantization during training, maintaining standard FP16/FP32 training and achieving good PTQ results. The implication is that architectural fixes are simpler and more robust — they don't require maintaining a QAT pipeline.
Vs. prior attention analysis [8, 30]: These papers observed the delimiter-attending "no-op" behavior but didn't connect it to outliers or quantization. This paper provides the causal link: the architectural limitation (no exact-zero attention output) → the learned workaround (attending to delimiters with extreme softmax saturation) → the numerical pathology (large outliers in preceding FFN outputs) → the practical problem (catastrophic quantization failure).
As a new pre-training protocol: The paper explicitly recommends "a new pre-training protocol that significantly reduces the magnitude of outliers yielding way more quantization-friendly models that can be effortlessly quantized using PTQ without strong degradation of performance." This positions the work not as a paper-about-quantization, but as a paper-about-architecture whose downstream benefit is easier quantization.

The Two Proposed Fixes as Architectural "Permission"

A useful way to understand the paper's framing is that both clipped softmax and gated attention are mechanisms for explicitly granting the model a capability it was previously forced to approximate.

Clipped softmax says: "Instead of asymptotically approaching zero by driving inputs to infinity, we'll let you output exactly zero once the softmax falls below a threshold." The parameter $\gamma < 0$ controls how far below zero the softmax output is stretched before clipping, effectively defining a threshold $\frac{-\gamma}{\zeta - \gamma}$ below which the output becomes exactly zero. This decouples "I want to ignore this token" from "I need unboundedly large softmax inputs," removing the driving force for outlier escalation.

Gated attention says: "Instead of using the attention mechanism to approximate a no-op by attending to low-value tokens, here's an explicit gate you can close." The sigmoid-gated formulation $\text{Gated\_attention}(x) = \text{sigmoid}(G(x)) \odot \text{softmax}(\dots)V(x)$ separates two concerns: the attention mechanism decides where to attend (for heads that do want to update), and the gate decides whether to update at all. The model can now close the gate without manipulating the softmax toward extreme values.

Both approaches are fundamentally addressing the same architectural limitation — the inability to produce a true no-op — through different mechanisms (a more flexible softmax, or an explicit multiplicative gate). The paper frames them as independent and both effective, giving practitioners options depending on their preferences (clipped softmax is nearly zero-overhead; gated attention adds a small number of parameters but gives more explicit control).

3. Technical Approach

3.1 Reader Orientation

This paper proposes two drop-in modifications to the standard transformer attention mechanism—clipped softmax and gated attention—that prevent the emergence of activation outliers during pre-training by giving the model explicit, finite-range mechanisms to produce near-zero attention outputs rather than forcing it to approximate a “no-op” through unbounded softmax saturation. The core insight driving both designs is that attention heads attempting to not update the residual stream currently approximate this behavior by attending almost exclusively to delimiter tokens or uninformative patches whose value projections are learned to be near-zero, which mathematically requires pushing softmax inputs toward infinity and thereby creates the large-magnitude outliers that catastrophically break post-training quantization.

3.2 Big-Picture Architecture (Diagram in Words)

The system being modified is a standard transformer encoder or decoder, with the changes confined entirely to the multi-head self-attention sublayer. The architecture has five components:

The base transformer (BERT, OPT, or ViT) with its standard sequence of layers, each containing multi-head self-attention followed by a feed-forward network, both wrapped with residual connections and LayerNorm.
The softmax attention mechanism within each head, which computes attention probabilities from query-key dot products and uses them to weight the value projections.
The LayerNorm and residual stream, which normalize activations between sublayers and add the attention output back to the input—this is the path through which outliers in one layer propagate to become inputs requiring even larger dynamic range in the next.
Proposed modification A: Clipped softmax—replaces the standard softmax with a stretched-and-clipped variant that can output exact zeros at finite input range, removing the mathematical impossibility that previously forced the model to push activations toward infinity.
Proposed modification B: Gated attention—adds a lightweight learned sigmoid gate that multiplies each attention head's output token-wise, giving the model an explicit "update or don't update" control separate from the attention probabilities.

Information flows as follows: the hidden state from the previous layer enters the attention sublayer → query, key, and value projections are computed as usual → attention probabilities are computed from query-key similarities → if using clipped softmax, the probabilities are stretched by factors $\gamma$ and $\zeta$ then clipped to $[0,1]$ ; if using gated attention, the gating module computes per-token sigmoid probabilities from the same input → the attention-weighted values are computed as usual → if using gated attention, the gating probabilities multiply the attention output element-wise → the result is added to the residual stream and passed through LayerNorm into the FFN.

3.3 Roadmap for the Deep Dive

First, the outlier formation hypothesis from Section 3 of the paper—the four-step causal chain from "attention head wants a no-op" to "FFN outliers grow unboundedly"—because every design decision in the two proposed methods follows directly from intercepting this chain at specific points.
Second, clipped softmax: the mathematical definition (Equation 4), the role of $\gamma$ and $\zeta$ , and the parameterization $\gamma = -\alpha / T$ that makes the method adaptive to sequence length.
Third, gated attention: the mathematical definition (Equation 5), the design of the gating module $G$ , and the role of bias initialization in controlling the initial gate state.
Fourth, the training methodology so that you understand what "pre-training from scratch" means in the context of the experiments, what changes when using each method, and what stays identical to standard training recipes.
Fifth, the quantization setup applied identically to all models post-training, so that you understand what "W8A8 PTQ" concretely means and why the metrics (maximum infinity norm, kurtosis) are chosen.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a diagnostic-and-fix paper whose core idea is that activation outliers in transformers are an architectural artifact of the softmax function's inability to output exact zeros at finite input range, and that two simple architectural changes—clipped softmax and gated attention—can remove the root cause by giving attention heads explicit mechanisms to produce near-zero outputs without pathological activation magnitudes.

The Outlier Formation Hypothesis (The Causal Chain That Motivates Both Fixes)

Before presenting the methods, the paper lays out a four-step causal chain explaining how the architectural limitation of softmax attention produces activation outliers. Understanding this chain is essential because each proposed method intercepts it at a specific point.

Step 1: Attention heads want to perform a no-op on some tokens. In a residual network, the default behavior is the identity mapping—adding zero to the hidden state preserves it unchanged. But the attention mechanism as defined by Vaswani et al. always produces some non-zero output. If an attention head's function is not applicable to certain tokens (or to any tokens at all), the head must learn to approximate a zero update. The empirical evidence (Figures 2 and 3) shows this is done by attending almost exclusively to tokens with low information content—delimiter tokens like [SEP], periods, and commas in BERT, or background patches in ViT.

Step 2: The softmax cannot produce exact zeros without infinite input range.

$\text{softmax}(x)_i = \frac{\exp(x_i)}{\sum_{j=1}^d \exp(x_j)}$

where $x \in \mathbb{R}^d$ is the vector of pre-softmax logits for a single query position, $d$ is the number of key positions (sequence length), and $\text{softmax}(x)_i$ is the attention probability assigned to the $i$ -th key.

What it computes: for each query position, the softmax converts an unnormalized vector of similarity scores (dot products between the query and all keys, scaled by $1/\sqrt{d_{\text{head}}}$ ) into a probability distribution over key positions, where each element is in $(0, 1)$ and all elements sum to 1. The output dictates how much each key's value contributes to the attention output for that query.

Why this form: the exponential ensures positivity and makes the distribution concentrate on the largest logits, while the denominator normalizes so that the output is a valid probability distribution. However, the exponential is strictly positive, meaning $\text{softmax}(x)_i > 0$ for any finite $x_i$ —even if $x_i$ is $-1000$ , the output is not zero, just extremely small. This is the mathematical property that creates the problem.

The paper formalizes the limit case:

$\text{softmax}(x)_i = 0 \iff \exists j \neq i, \; x_j - x_i = +\infty$

where $j$ is some other position in the sequence.

What it states: for an attention probability to be exactly zero, there must exist at least one other position whose pre-softmax logit exceeds the current position's logit by an infinite amount. Since infinite values are impossible in a finite-precision floating-point system, exact zeros are unreachable.

Why this matters for the hypothesis: if an attention head wants to attend only to delimiter tokens and not at all to content tokens, it needs the attention probabilities for content tokens to be as close to zero as possible. The only way to approach zero in standard softmax is to make the delimiter tokens' logits much larger than the content tokens' logits. The larger the difference, the closer to zero the un-attended probabilities become. But since zero is only asymptotically reachable, there is always a gradient signal pushing the logit differences to increase.

Step 3: To produce sufficiently large softmax logits after LayerNorm, the FFN output in the previous layer must have extreme magnitude. The paper's Figure 4 diagrams this precisely. The large dynamic range needed in the softmax input of layer $L+1$ must come from somewhere—specifically, from the output of the feed-forward network in layer $L$ , which flows through a residual connection and a LayerNorm before entering the attention mechanism of layer $L+1$ . But LayerNorm subtracts the mean and divides by the standard deviation across the feature dimension, which compresses the dynamic range. To still produce sufficiently large logits after this normalization, the FFN output of layer $L$ must have values that are extreme enough to survive the normalization and still dominate the softmax input. These extreme values are the "strong outliers" observed in practice—typically appearing in only a few hidden dimensions (Figure 1 shows that dimensions #123, #180, #225, #308, #381, #526, and #720 account for essentially all outliers in BERT-base).

Step 4: The feedback loop causes outliers to grow over training time. Because the softmax never outputs exact zeros, the gradient $\partial y_i / \partial x_j \neq 0$ for all $i, j$ . This means the outlier-producing dimensions always receive a gradient signal, and since the loss landscape rewards more extreme saturation (it produces attention distributions closer to the desired one-hot pattern), the outliers tend to grow in magnitude the longer the network trains. This is not a transient training artifact but a structural consequence of the architecture.

With this causal chain established, the two proposed methods intervene at different points:

Clipped softmax intervenes at Step 2 by changing the softmax function itself so that exact zeros are achievable with finite input range, breaking the feedback loop that drives logits toward infinity.
Gated attention intervenes at Step 1 by giving the model an alternative mechanism to produce a no-op—closing a gate—so that the attention head no longer needs to approximate a no-op through its attention distribution in the first place.

Clipped Softmax: Mathematical Definition and Design Rationale

The paper proposes to replace the standard softmax in Equation 3 with a clipped variant:

$\text{clipped\_softmax}(x; \zeta, \gamma) = \text{clip}((\zeta - \gamma) \cdot \text{softmax}(x) + \gamma, \; 0, \; 1)$

where $x \in \mathbb{R}^d$ is the pre-softmax input vector (query-key dot products scaled by $1/\sqrt{d_{\text{head}}}$ ), $\zeta \geq 1$ is the upper stretch factor, $\gamma \leq 0$ is the lower stretch factor, $\text{softmax}(x)$ is the standard softmax output in $(0, 1)^d$ , $\text{clip}(\cdot, 0, 1)$ clamps each element to the interval $[0, 1]$ , and the output is a vector in $[0, 1]^d$ that may contain exact zeros and exact ones.

What it computes: the function takes the standard softmax output (which lies in $(0,1)$ ), stretches it linearly from the interval $(0,1)$ to $(\gamma, \zeta)$ , and then clips back to $[0,1]$ . The stretching is an affine transformation: first multiply by $(\zeta - \gamma)$ (which is at least 1 since $\zeta \geq 1$ and $\gamma \leq 0$ ), then add $\gamma$ (which is non-positive). Values of the original softmax that are smaller than $\frac{-\gamma}{\zeta - \gamma}$ become negative after stretching and are clipped to zero. Values larger than $\frac{1-\gamma}{\zeta - \gamma}$ exceed 1 after stretching and are clipped to one. Values in between remain in $(0,1)$ .

Why this form: the affine stretch followed by clipping achieves two things that standard softmax cannot. First, by allowing $\gamma < 0$ , the softmax output is stretched to include negative values, and those negative values become exact zeros after clipping. This means the model can achieve $\text{clipped\_softmax}(x)_i = 0$ with finite values of $x$ —specifically, whenever the original softmax output falls below the threshold $-\gamma / (\zeta - \gamma)$ . This removes the asymptotic behavior that forced logits toward infinity. Second, the gradient is zero for any value that lands in the clipping region (below 0 or above 1), meaning the feedback loop that previously caused unbounded growth is broken: once a probability is clipped to zero or one, it stops contributing gradients. Third, this formulation follows a precedent for sigmoid functions [40, 45], adapted here for the softmax's multi-dimensional output.

The paper's hyperparameter investigation (Table 1 and Table 8) reveals that only the lower clipping matters for outlier suppression. The key finding is quoted precisely:

"most of the improvement happens when we use $\gamma < 0$ (clipping at zero)... using $\zeta > 1$ (clipping at one) yields similar results to the vanilla softmax. Finally, when we combine both $\gamma < 0$ and $\zeta > 1$ , the results seem similar to just clipping at zero. We, therefore, conclude that for dampening outliers, only the lower-range clipping allows exact zeros matter."

This means in practice, the upper stretch factor $\zeta$ can be set to 1 (no upper stretching), and the method reduces to:

$\text{clipped\_softmax}(x; \gamma) = \text{clip}((1 - \gamma) \cdot \text{softmax}(x) + \gamma, \; 0, \; 1)$

where $\gamma < 0$ stretches the softmax downward by $|\gamma|$ before clipping.

Clipped Softmax: The $\gamma = -\alpha / T$ Parameterization

The paper observes that the optimal value of $\gamma$ depends on the sequence length $T$ , because the average attention probability is $1/T$ (each row of the $T \times T$ attention matrix sums to 1). They propose a parameterization that accounts for this:

$\gamma = -\frac{\alpha}{T}$

where $\alpha > 0$ is a new hyperparameter (independent of sequence length) and $T$ is the sequence length.

What it computes: the lower stretch factor $\gamma$ is set to be inversely proportional to the sequence length, with $\alpha$ controlling the strength. For a sequence of length $T = 128$ and $\alpha = 1$ , this gives $\gamma = -1/128 \approx -0.0078$ . For a longer sequence of $T = 512$ , the same $\alpha = 1$ gives $\gamma = -1/512 \approx -0.0020$ —a weaker stretch because longer sequences have smaller average probabilities and need less stretching to reach the clipping threshold.

Why this form: the clipping threshold in the stretched-then-clipped softmax is $-\gamma / (1 - \gamma)$ when $\zeta = 1$ . With $\gamma = -\alpha/T$ , this threshold becomes approximately $\alpha / T$ for small $\alpha/T$ . Since the average softmax output is $1/T$ , any probability below the average by a factor of $\alpha$ gets clipped to zero. This makes the method adaptive: the same $\alpha$ works across different sequence lengths because it defines the threshold relative to the uniform-attention baseline, not in absolute terms.

The paper validates this parameterization empirically in Figure 6 (Section 5.2), training BERT-6L with sequence lengths from 32 to 256 and $\alpha \in \{1/4, 1/2, 1, 2, 4, 8\}$ . The results show that $\alpha \in [2, 4]$ "significantly dampens the magnitude of outliers while maintaining good FP16 perplexity across all explored sequence lengths." This provides a practical recipe: choose $\alpha$ in that range, compute $\gamma = -\alpha/T$ for the target sequence length, and use that value throughout training.

The threshold analysis in the paper can be made explicit: with $\zeta = 1$ and $\gamma < 0$ , the zero-clipping threshold is $-\gamma / (1 - \gamma)$ . For $\gamma = -0.03$ (the best-performing value for BERT-base with $T = 128$ in Table 1), this threshold is $0.03 / 1.03 \approx 0.029$ . Any original softmax output below approximately 0.029 becomes exactly zero after stretching and clipping, while values above remain in the stretched range. For $T = 128$ , the uniform probability is $1/128 \approx 0.0078$ , so the threshold is about 3.7 times the uniform baseline—positions that the model considers substantially less relevant than average get completely ignored.

Gated Attention: Mathematical Definition and Design Rationale

The second proposed method adds an explicit multiplicative gate to the attention output:

$\text{Gated\_attention}(x) = \text{sigmoid}(G(x)) \odot \text{softmax}\left(\frac{Q(x)K(x)^T}{\sqrt{d_{\text{head}}}}\right) V(x)$

where $x$ is the input to the attention layer, $G(\cdot)$ is a learned gating function (parameterized by a small neural network), $\text{sigmoid}(G(x))$ produces per-token gating probabilities in $(0, 1)$ , $\odot$ denotes element-wise multiplication across the token axis, and everything else is the standard multi-head self-attention defined in Equation 3.

What it computes: the standard attention output is computed identically to the baseline—query-key dot products produce attention probabilities, which weight the value projections, yielding a per-token attention output vector. Independently, the gating function takes the same input $x$ and produces a scalar probability $\pi_t \in (0,1)$ for each token position $t$ . This scalar multiplies the entire attention output vector for that token—every feature dimension is scaled by the same gate value. If the gate is close to zero, the attention output is nearly zeroed out regardless of what the attention probabilities and values computed. If the gate is close to one, the attention output passes through unmodified.

Why this form: this is a separation of concerns. In standard attention, the decision of where to attend (controlled by the softmax over keys) is entangled with the decision of whether to update (which the model approximates by attending to low-value delimiter tokens). Gated attention separates these: the softmax controls which tokens to aggregate information from, and the gate controls whether the aggregated information should be added to the residual at all. The gate can be closed ( $\pi_t \approx 0$ ) to produce a near-exact no-op without requiring any manipulation of the attention distribution, the value projections, or the softmax saturation. Conversely, the softmax is now free to distribute probability mass meaningfully among content tokens without having to reserve mass for delimiter tokens just to approximate a no-op.

This formulation is illustrated in Figure 5, which shows the gating probabilities $\pi = \text{sigmoid}(G(x))$ being multiplied element-wise with the standard attention output, producing the final gated attention result before the residual addition.

Gated Attention: Gating Module Design

The gating function $G$ operates on a per-head, per-token basis. The input $x$ has shape $(T, d_{\text{model}})$ , which is reshaped into $(n_{\text{heads}}, T, d_{\text{head}})$ for multi-head attention. For each head $i \in \{1, \dots, n_{\text{heads}}\}$ , the gating function $G_i: \mathbb{R}^{d_{\text{head}}} \to \mathbb{R}$ maps the $d_{\text{head}}$ -dimensional representation of each token to a scalar logit. The gating probability is then obtained via the sigmoid:

$\tilde{\pi}_{i,t} = G_i(x_{i,t,:}) \quad \forall t \in \{1, \dots, T\}$ $\pi_{i,:} = \text{sigmoid}(\tilde{\pi}_{i,:})$

where $x_{i,t,:} \in \mathbb{R}^{d_{\text{head}}}$ is the input representation for head $i$ , token $t$ , $G_i$ is the per-head gating function, $\tilde{\pi}_{i,t} \in \mathbb{R}$ is the pre-sigmoid logit, and $\pi_{i,t} \in (0, 1)$ is the final gating probability.

What it computes: for every token position and every attention head, the gating module takes the per-head input features and produces a scalar between 0 and 1 that controls how much of that head's attention output for that token contributes to the residual stream. A value of 0 means "completely suppress this head's output for this token"; a value of 1 means "let the attention output pass through fully."

Why this form: per-head gating gives each head independent control over whether to update each token's representation. Per-token gating is necessary because the no-op behavior is token-specific—a head might want to update content word representations while not updating delimiter tokens, and different tokens in the same sequence should be gated independently. The sigmoid nonlinearity ensures the gate output is strictly between 0 and 1, providing a smooth differentiable mechanism that the model can learn to open or close.

The paper explores several parameterizations of $G_i$ , summarized in Table 4:

Linear: $G_i$ is a single linear layer mapping $d_{\text{head}} \to 1$ . This is the simplest and most lightweight option, adding $n_{\text{heads}} \cdot (d_{\text{head}} + 1) \approx d_{\text{model}}$ parameters per attention layer—equivalent to approximately one extra token embedding. The paper notes this adds "less than 0.009% of the total model size" for BERT-base.
MLP: $G_i$ is a two-layer feed-forward network with a hidden size $n_{\text{hid}}$ and ReLU nonlinearity (mapping $d_{\text{head}} \to n_{\text{hid}} \to 1$ ). This is more expressive but adds $n_{\text{heads}} \cdot (n_{\text{hid}}(d_{\text{head}} + 2) + 1)$ parameters per layer.
All-heads-linear: a single linear layer mapping $d_{\text{model}} \to n_{\text{heads}}$ that produces gating logits for all heads simultaneously, allowing cross-head information mixing. This adds $n_{\text{heads}} \cdot (d_{\text{model}} + 1)$ parameters.

The paper presents results primarily for the Linear variant, as it is the most lightweight while still effective. In the main results (Table 2), the "Gated attention" rows for BERT and OPT use either Linear or MLP (the specific configuration is indicated in the extended results tables: for BERT, $\text{GA, MLP } (n_{\text{hid}} = 4)$ in Table 5; for OPT, $\text{GA, Linear } (\pi_{\text{init}} = 0.25)$ in Table 6).

Gated Attention: Bias Initialization Strategy

The bias of the gating function's linear layer is a critical hyperparameter that controls the initial gate state. With small random weights (following He initialization [22]), the pre-sigmoid output $G_i(\cdot) \approx b_{\text{init}}$ at the start of training, so the initial gating probability is:

$\pi_{\text{init}} \approx \text{sigmoid}(b_{\text{init}})$

where $b_{\text{init}}$ is the bias initialization value and $\pi_{\text{init}} \in (0, 1)$ is the expected gate openness at training start.

What this controls: setting $b_{\text{init}}$ to a large positive value (e.g., $b_{\text{init}} = 2.0$ , giving $\pi_{\text{init}} \approx 0.88$ —very open gates) means the gated attention initially behaves almost identically to standard attention, and the model must learn to close gates where needed. Setting $b_{\text{init}}$ to a large negative value (e.g., $b_{\text{init}} = -2.0$ , giving $\pi_{\text{init}} \approx 0.12$ —mostly closed gates) means attention outputs are initially suppressed, and the model must learn to open gates where attention is useful.

Why this matters: this creates a tradeoff. Starting with open gates ( $\pi_{\text{init}} \approx 1$ ) means the initial training dynamics closely match standard pre-training, which is safe for convergence but may take many iterations for gates to learn to close—and during those iterations, the model might still develop outliers. Starting with closed gates ( $\pi_{\text{init}} \approx 0$ ) immediately suppresses all attention outputs, which prevents outliers from forming but starts far from the standard attention behavior, potentially degrading convergence. The paper's experiments (Figure 7) show that the method is robust across a wide range of $\pi_{\text{init}} \in [0.25, 0.9]$ for BERT and $\pi_{\text{init}} \in [0.1, 0.5]$ for ViT, with the full range of values producing good floating-point performance and significantly reduced outliers.

The default settings used in the main results (Tables 5, 6, 7) use $\pi_{\text{init}} \in \{0.25, 0.5\}$ —starting with gates slightly closed or in a neutral state, providing a balance between initial stability and outlier prevention.

Quantization Method (Applied Post-Training to All Models)

To evaluate the quantizability of models trained with the proposed methods, the paper applies a uniform post-training quantization (PTQ) procedure identically to all models. The quantization scheme follows the standard formulation from Jacob et al. [26]:

$\hat{x} = q(x; s, z, b) = s \cdot \left(\text{clip}\left(\left\lfloor \frac{x}{s} + z \right\rceil, \; 0, \; 2^b - 1\right) - z\right)$

where $x$ is the quantizer input (a weight or activation tensor), $s \in \mathbb{R}^+$ is the scale factor determining the step size between quantization levels, $z \in \mathbb{Z}$ is the zero point (the integer value corresponding to real zero), $b \in \mathbb{N}$ is the bitwidth (8 for all main experiments), $\lfloor\cdot\rceil$ is round-to-nearest, and $\hat{x}$ is the de-quantized approximation of $x$ after round-trip through the quantized representation.

What it computes: the real-valued input $x$ is divided by the scale $s$ to map it to the quantization grid, shifted by the zero point $z$ so that zero maps to the correct integer, rounded to the nearest integer, clipped to the representable range $[0, 2^b - 1]$ , shifted back, and multiplied by $s$ to produce a quantized approximation in floating point. During inference, the integer values are used directly for computation; the floating-point simulation is used for evaluation.

Why this form: uniform affine (asymmetric) quantization is one of the most commonly used schemes because it maps efficiently to integer arithmetic hardware: the scale and zero point can be folded into the matrix multiplication via requantization steps. Symmetric quantization (used for weights in this paper) restricts $z = 0$ , which simplifies computation but can waste half the quantization range if the distribution is not zero-centered. Asymmetric quantization (used for activations) allows $z \neq 0$ , better utilizing the quantization bins for distributions with non-zero means, like ReLU outputs.

The quantization configuration applied uniformly in the main experiments:

Weights: symmetric uniform quantization ( $z = 0$ ). For BERT and ViT, min-max range estimation is used (the scale is set to $\max(|x|) / (2^{b-1} - 1)$ so that the full symmetric range is covered). For OPT, MSE range estimation is used (the scale is chosen to minimize the mean squared error between original and quantized weights) because the paper found this performs better for OPT.
Activations: asymmetric uniform quantization with static range estimation. The ranges are determined by passing 16 batches of calibration data (randomly sampled from the respective training sets) through the model and collecting activation statistics using an exponential moving average with momentum 0.9—this is the "running min-max" estimator [32]. For OPT, percentile-based range estimation (99.99% and 99.999% percentiles instead of min/max) is also explored, with the best configuration selected per experiment.
All weights and activations (both inputs and outputs of every layer) are quantized to 8 bits, with the exception of the final linear layer for BERT and OPT models (left in higher precision).
Each PTQ experiment is repeated 3 times with different random subsets of calibration data, and mean ± standard deviation is reported.

Outlier Metrics: Maximum Infinity Norm and Kurtosis

To quantify how successful the proposed methods are at removing outliers, the paper uses two metrics applied to the output of the attention layer:

Maximum infinity norm: $\max \|x\|_\infty$ averaged across the validation set—the largest absolute value anywhere in the attention layer's output, averaged over all validation examples. This directly measures the magnitude of the most extreme outliers. For vanilla BERT-base, this is 735 ± 55; for clipped softmax with $\gamma = -0.025$ , it drops to 21.5 ± 1.5—a reduction of over 30×.
Average kurtosis: the kurtosis of the activation distribution, averaged across all layers. Kurtosis measures the "tailedness" of a distribution—higher kurtosis means more probability mass in the tails relative to a normal distribution, which is a signature of outliers. For vanilla BERT-base, this is 3076 ± 262; for clipped softmax, it drops to 80 ± 6.

These metrics are chosen because prior work [4, 6] has shown they correlate well with quantizability: tensors with high infinity norm require large quantization ranges (losing precision on the bulk of values), and tensors with high kurtosis have a small fraction of values dominating the range. Low values of both metrics indicate a distribution that is well-suited to uniform quantization with a single scale factor.

Training Methodology (What Changes vs. What Stays the Same)

The paper does not propose a new training algorithm or optimization procedure. The standard pre-training recipes for BERT [14], OPT [74], and ViT [15] are followed exactly, with the only change being the replacement of the standard softmax with clipped softmax or the addition of the gating module in every attention layer. All other hyperparameters—optimizer, learning rate schedule, batch size, weight decay, dropout, data augmentation, number of training steps—remain identical to the standard recipes.

For BERT-base (109M parameters), the training follows Devlin et al. [14]: masked language modeling on BookCorpus + English Wikipedia, sequence length 128, batch size 256, $10^6$ training steps, AdamW optimizer with max learning rate $10^{-4}$ , linear warmup over $10^4$ steps followed by linear decay, weight decay 0.01, gradient clipping at 1.0, dropout 0.1. FP16 mixed precision is used via HuggingFace Accelerate.

For OPT-125m, the training follows Zhang et al. [74]: causal language modeling on the same dataset, sequence length 512, effective batch size 192 (batch size 48 with 4 gradient accumulation steps), 125,000 steps, AdamW with $(\beta_1, \beta_2) = (0.9, 0.95)$ , max learning rate $4 \times 10^{-4}$ , 2000 warmup steps then linear decay, weight decay 0.1, gradient clipping 1.0, dropout 0.1. An important note: the paper found that applying weight decay to LayerNorm $\gamma$ parameters (not done by default) reduces outliers even in the vanilla model, so some experiments include this change.

For ViT-S/16 (22M parameters), the training follows the PyTorch Image Models pipeline [64]: ImageNet-1K at 224×224 resolution, 16×16 patches, batch size 512, 300 epochs, AdamW, cosine learning rate schedule from $10^{-3}$ down to $10^{-5}$ , weight decay 0.03, RandAugment, Mixup, CutMix, label smoothing 0.1, color jitter 0.4. After preliminary experiments showing that outliers already originate after the patch embeddings, the paper adds a LayerNorm after the patch embeddings (which is absent in the default ViT definition) as an additional architectural change that works synergistically with the proposed methods.

For the scaling experiments with larger OPT variants (350m and 1.3B), training is reduced to $10^5$ steps with batch size 256 due to compute constraints.

Each configuration is trained two times with different random seeds, and mean ± standard deviation is reported for all metrics.

Summary of Design Choices and Their Justifications

Clipped softmax uses only $\gamma < 0$ with $\zeta = 1$ : Table 1 empirically shows that upper clipping ( $\zeta > 1$ ) provides no additional benefit for outlier suppression—only the ability to reach exact zeros matters. This simplifies the method to a single hyperparameter.
The $\gamma = -\alpha/T$ parameterization: this makes the method adaptive to sequence length, with the clipping threshold relative to the uniform-attention baseline rather than an absolute value. The paper validates that $\alpha \in [2, 4]$ works across sequence lengths from 32 to 256.
Gated attention uses per-head independent gating, not shared across heads: each head may have different no-op requirements, so independent gates give maximum flexibility. Per-token gating is essential because no-op behavior is token-specific.
Linear gating is the default, not MLP: Table 5 shows that Linear and MLP ( $n_{\text{hid}} = 4$ ) perform similarly on BERT-base, so the simpler, cheaper Linear variant is preferred for its minimal overhead (∼1 extra token's worth of parameters per layer).
Bias initialization for gated attention is in a moderate range (not fully open, not fully closed): Figure 7 shows that $\pi_{\text{init}} \in [0.25, 0.5]$ provides a good balance—gates start partially open so training dynamics are stable, but the model can learn to close them quickly enough to prevent outlier formation.
Quantization uses simple per-tensor PTQ with static ranges, not per-channel or QAT: the whole point of the paper is that with architectural fixes, simple quantization works. Using more advanced quantization techniques would obscure whether the improvement comes from the architecture or the quantizer.

4. Key Insights and Innovations

Innovation 1: Outlier Formation as an Architectural Pathology, Not a Training Artifact

The paper's most fundamental contribution is transforming the activation outlier problem from a mysterious empirical nuisance into a well-understood architectural pathology with a clear causal mechanism. Prior to this work, the field knew that outliers existed in specific hidden dimensions [4, 13, 31], that they were important for model predictions [31, 49], and that they made quantization extremely difficult—but nobody could explain why the model learned them in the first place. The dominant framing treated outliers as an intrinsic property of transformer training dynamics, something to be accommodated through cleverer quantization schemes rather than something that could be eliminated at the source.

What makes this paper's diagnostic contribution genuinely novel is the causal chain it uncovers: (1) the softmax function mathematically cannot produce exact zeros with finite inputs → (2) attention heads that want to perform a no-op on the residual must therefore drive softmax inputs toward infinity to asymptotically approach zero attention → (3) these large inputs must survive LayerNorm compression, placing extreme demands on the FFN output of the preceding layer → (4) the softmax gradient is everywhere non-zero, creating a perpetual feedback loop that amplifies outliers over training time. This is not a list of correlated observations—it is a mechanistic explanation that connects the architectural limitation (softmax positivity) to the learned behavior (attending to delimiter tokens) to the numerical pathology (FFN outliers in specific dimensions) to the practical consequence (catastrophic quantization failure).

The significance of this causal understanding extends beyond the specific fix. It reframes the outlier problem as evidence of a capability gap in the transformer architecture—the inability to express a clean "do nothing" operation on the residual—rather than as an optimization quirk. This shift in perspective is what makes the paper's solutions principled rather than heuristic. You don't need to understand why clipping activations at 6 standard deviations hurts performance [31]; you need to understand what functional behavior those extreme values enable, and then give the model an alternative way to achieve that behavior. The paper demonstrates this universality by showing the same pattern—attention heads attending to low-information tokens/patches with small value outputs, generating outliers in specific preceding-layer dimensions—across BERT, OPT, and ViT (Figures 1–3, Figures 9–17 in the appendix), establishing that this is a general architectural phenomenon rather than a BERT-specific curiosity.

This is fundamentally a diagnostic breakthrough disguised as an architectural fix paper. The methods (clipped softmax, gated attention) are validated by the quality of the diagnosis that motivated them. The paper's approach—work backward from the observed outlier pattern to the attention behavior that requires it, then to the mathematical limitation that forces it—represents a template for how to debug neural network training pathologies: don't just regularize the symptom; ask what the network is trying to do that requires the pathological values, and give it a better tool for that job.

Innovation 2: The "Architectural Permission" Framing—Removing Constraints Rather Than Adding Constraints

The dominant approach to the outlier problem before this paper was constraining the model or the quantizer: SmoothQuant [67] constrains the activation range by mathematically redistributing scales between weights and activations; GPT3.int8() [13] constrains the quantization scheme to handle outliers as a special case; Outlier Suppression [62, 63] constrains the outlier dimensions via post-hoc shifting and scaling; QAT [3, 16] constrains the training process to be robust to quantization noise. All of these approaches add complexity to work around the outlier phenomenon.

The paper inverts this logic entirely. Both clipped softmax and gated attention are mechanisms that remove a constraint from the architecture—specifically, the constraint that the attention function must always produce a strictly positive, non-zero output for every token. The softmax's (0,1) range, normally seen as a feature (it produces a valid probability distribution), is identified here as a limitation: it prevents the model from expressing "I want zero contribution from this token" without driving internal activations toward infinity. Clipped softmax removes this constraint by stretching the softmax output to include negative values, then clipping—exact zeros become possible at finite input range. Gated attention removes it even more directly by adding an explicit multiplicative gate that can zero out the attention output regardless of what the softmax computes.

This "architectural permission" framing is conceptually distinctive because it treats the model not as something that needs to be regularized or constrained into better behavior, but as something whose existing architecture prevents it from expressing the behavior it wants to learn, forcing pathological workarounds. The implication is profound: when a neural network learns extreme values, before reaching for a regularizer, ask what the architecture prevents it from doing cleanly. This reframing has implications beyond quantization—it suggests a general design principle for neural architectures: wherever the model is observed learning "hacks" (extreme values, brittle solutions, degenerate representations), the first question should be "what would we need to add or change so the model can express the desired behavior without the hack?"

The evidence for this reframing is most compelling in the qualitative results (Figure 8 and Appendix A.1). Models trained with clipped softmax and gated attention both still exhibit no-op-like behavior—they learn to suppress attention outputs for certain tokens. But with clipped softmax, this is done through the clipping threshold (attention probabilities below ~0.029 become exact zeros) rather than through unbounded softmax saturation. With gated attention, it's done through the explicit gate (which can be closed to near-zero) rather than through manipulating the attention distribution at all. The functional behavior is preserved—it must be, because the no-op capability is genuinely useful for the model—but the mechanism no longer requires pathological activation values. This is evidence not just that the methods work, but that the diagnostic framing is correct: outliers were indeed a symptom of the model working around an architectural limitation, and removing the limitation eliminates the symptom while preserving the function.

Innovation 3: Empirical Resolution of the "No-Op" Hypothesis with Cross-Architecture and Cross-Modality Evidence

Prior work had speculated that attention heads attending to delimiter tokens act as a "no-op" [8, 30], but this was an interpretation of attention patterns, not a causal explanation linked to concrete numerical consequences. The key missing pieces were: (1) evidence that the no-op behavior is causally upstream of the outliers (not just correlated with them), and (2) evidence that this mechanism is universal across architectures and modalities, not specific to BERT or NLP.

The paper provides both. The causal direction is established by the outlier analysis in Section 3 and Appendix A. Figure 1 shows that >97% of outliers in BERT-base's FFN outputs correlate with delimiter token positions ([SEP], ".", ","). Figure 2 shows that the attention heads corresponding to these outlier dimensions (hidden dimensions #180 → head #3, #720 → head #12) assign almost all probability mass to those same delimiter tokens, and those tokens have small values in V. The product attention_weights × values is therefore small—the attention head's output is a soft near-zero. But crucially, the paper shows this is not mere correlation: the direction of dependence is that the attention head in layer L+1 needs large softmax inputs to achieve this attention pattern, which requires large FFN outputs in layer L. The outliers appear in the FFN output of layer L because the attention head in layer L+1 needs them—the causal arrow runs from the downstream attention behavior to the upstream FFN values.

The cross-modality universality is a significant empirical contribution on its own. The paper demonstrates the identical pattern in vision transformers (Section 3, Figures 3 and 9, Appendix A.2): outliers in ViT's later layers (peaking at layers #10–11) correlate with background patches; the corresponding attention head in the next layer (head #1, corresponding to hidden dimensions #48 and #43 where the vast majority of outliers occur) allocates most probability mass to those same background patches; and those patches have distinctly smaller value magnitudes than non-outlier patches. The fact that the same mechanism appears in autoregressive decoders (OPT) and vision transformers establishes that this is not a BERT artifact or an NLP artifact—it is a general consequence of the softmax attention + residual connection + LayerNorm architecture, arising whenever the model needs to selectively not update certain token/patch representations.

This universality finding is what elevates the paper from "a fix for BERT quantization" to "a fundamental architectural insight about transformers." It means the outlier problem will appear in any sufficiently-trained transformer regardless of domain, and that the proposed fixes should generalize correspondingly. The paper doesn't just assert this—it validates it empirically by training BERT (encoder, NLP), OPT (decoder, NLP), and ViT (encoder, vision) with both methods and showing consistent outlier suppression across all three (Table 2).

Innovation 4: The `γ = -α/T` Parameterization as a Sequence-Length-Adaptive Design Principle

Incremental parameter tuning is the norm in deep learning—try a grid of values on a validation set and pick the best one. The clipped softmax hyperparameter γ could easily have been treated this way: just sweep γ ∈ {-0.01, -0.02, -0.03} for each model and report the best result. What makes the paper's treatment of γ an innovation is the recognition that the optimal stretch factor depends on sequence length in a principled way, and the proposal of a parameterization that makes the method adaptive.

The reasoning is clean and connects directly to the mechanism being fixed. Each row of the T × T attention matrix sums to 1, so the average attention probability is 1/T. The clipping threshold in clipped softmax is approximately α/T when γ = -α/T with ζ = 1. This means the threshold for producing an exact zero is a fixed multiple of the uniform-attention baseline—attention probabilities that are α times smaller than average get clipped. The hyperparameter α therefore has the same semantic meaning across sequence lengths: it controls how much below average a probability must be before the model gives up on it entirely.

The empirical validation (Figure 6, Section 5.2) shows that α ∈ [2, 4] works well across sequence lengths from 32 to 256, providing practical evidence that the parameterization succeeds at its goal—making the hyperparameter transferable across sequence lengths. But the intellectual contribution is larger than the empirical result. It demonstrates a design principle for architectural modifications that depend on array dimensions: identify the natural scale of the quantity being modified (here, the average attention probability 1/T), and define hyperparameters relative to that scale. This principle could guide the design of other sequence-length-dependent architectural modifications in transformers, making it a methodological contribution beyond the specific clipped softmax application.

This is an incremental but intellectually satisfying refinement: the method doesn't need this parameterization to work (you could just grid-search γ), but the parameterization makes the method robust and transferable in a way that reflects understanding of the underlying mechanism.

5. Experimental Analysis

Evaluation Methodology

Dataset. Three separate training and evaluation pipelines are used for the three model families. For BERT and OPT (language modeling): pre-training uses the concatenation of BookCorpus [77] and English Wikipedia (specifically, the English subset of Wiki-40b, containing cleaned-up text with pre-defined training/validation splits). Evaluation reports perplexity (masked language modeling for BERT, causal language modeling for OPT) on the Wikipedia validation set. For the qualitative outlier analysis in Section 3 and Appendix A, the authors fine-tune BERT-base on the MNLI dataset (from the GLUE benchmark [61]) and analyze the MNLI-m validation set. For ViT: training and evaluation use ImageNet-1K [11, 52] at 224×224 resolution with 16×16 patches; evaluation reports top-1 accuracy on the ImageNet validation set.
Base model(s). Three model families spanning architectures and modalities: BERT-base-uncased (109M parameters, encoder-only, bidirectional attention, masked language modeling objective), OPT-125m (125M parameters, decoder-only, causal attention, causal language modeling objective), and ViT-S/16 (22M parameters, encoder-only vision transformer, image classification objective). For scaling experiments, OPT-350m and OPT-1.3B variants are additionally trained. The models are chosen to represent the three canonical transformer architectures (bidirectional encoder, autoregressive decoder, vision encoder), explicitly testing whether the outlier phenomenon and proposed fixes generalize. The paper argues BERT-base and ViT-S/16 are representatively-sized models where PTQ without outlier handling fails dramatically, making them informative testbeds.
Metrics. Three categories of metrics are tracked. Task performance: masked language modeling perplexity for BERT, causal language modeling perplexity for OPT (both on Wikipedia validation set), and top-1 accuracy for ViT (on ImageNet validation set) — all evaluated in both full floating-point precision (FP16 for BERT/OPT, FP32 for ViT) and after W8A8 post-training quantization. Outlier magnitude metrics: maximum infinity norm (max ‖x‖_∞ averaged across the validation set) and kurtosis of x averaged across all layers, where x is the output of an attention layer — these are applied to the FP16/FP32 model to assess how many outliers exist before quantization. Both are established in prior work [4, 6] as correlating well with quantizability. Quantized performance metrics: W8A8 perplexity or accuracy, with each PTQ experiment repeated 3 times using different random subsets of calibration data, reporting mean ± standard deviation. For the qualitative analysis (Section 3), outlier counts per hidden dimension and per token/patch position are aggregated across the validation set.
Baselines. The primary baseline is vanilla softmax pre-training — the same model architecture, dataset, and training recipe, but with standard softmax attention. This is the "Vanilla" row in all results tables (Tables 1, 2, 3, 5, 6, 7, 8, 9, 10). The paper does not compare against prior quantization workarounds (e.g., SmoothQuant, GPT3.int8(), Outlier Suppression) as baselines for the core experiments, because the central claim is not "our quantization scheme is better than theirs" but rather "our architectural changes make simple PTQ work." Comparisons against prior work are conceptual rather than head-to-head experimental — the paper positions itself as eliminating the need for those workarounds, not outperforming them at their own task.
Generation budget / compute accounting. This is not a test-time compute scaling paper, so there is no "generation budget" in the usual sense. The relevant cost accounting is training compute (GPU-hours on Nvidia A100 GPUs, summarized in Appendix D, Table 11) and architectural overhead (extra parameters and runtime for the proposed modifications). The paper reports that clipped softmax is "only marginally more expensive" than vanilla softmax (BERT: 93.6 vs. 92.8 hours, ViT: 104.0 vs. 101.8 hours, Appendix D, Table 11). Gated attention with Linear gates adds 3-8% runtime overhead depending on the model (BERT: 97.7 hours, OPT: 55.7 hours, ViT: 110.8 hours) and approximately d_model extra parameters per attention layer, which the paper notes is "less than 0.009% of the total model size" for BERT-base. The FLOPs-matched tradeoffs studied in Section 7 of the referenced example paper (pretraining vs. inference compute) are not part of this paper's experimental design.
Cross-validation / statistical protocol. Each pre-training configuration is run two times with different random seeds, and means ± standard deviations are reported for perplexity, accuracy, infinity norm, and kurtosis. For quantization, each PTQ experiment is repeated three times with different random subsets of calibration data used for activation range estimation, with means ± standard deviations reported for the quantized metric. The BERT-6L experiments studying the γ = -α/T parameterization (Section 5.2, Figure 6) sweep across five sequence lengths T ∈ {32, 64, 128, 192, 256} and six values of α ∈ {1/4, 1/2, 1, 2, 4, 8}, with each configuration trained once for 500,000 steps — the purpose is parameter sensitivity analysis rather than significance testing. For the large-scale OPT experiments (350m and 1.3B, Table 3), models are trained once due to compute constraints, with PTQ repeated 3 times for the standard deviation on the quantized metric.

Main Quantitative Results

Clipped Softmax Hyperparameter Sweep (Section 5.1)

The headline result from Table 1 is that the lower stretch factor γ < 0 is what matters for outlier suppression, while the upper stretch factor ζ > 1 provides no additional benefit. For BERT-base:

Vanilla softmax (γ = 0, ζ = 1): FP16 perplexity 4.49 ± 0.01, max infinity norm 735 ± 55, average kurtosis 3076 ± 262, W8A8 perplexity 1294 ± 1046.
Clipped softmax with γ = -0.03, ζ = 1 (lower clipping only): FP16 perplexity 4.41 ± 0.01, max infinity norm 20 ± 1, average kurtosis 80 ± 6, W8A8 perplexity 4.55 ± 0.01.
Clipped softmax with γ = 0, ζ = 1.03 (upper clipping only): FP16 perplexity 4.49 ± 0.00, max infinity norm 741 ± 66, average kurtosis 1707 ± 1249, W8A8 perplexity 1469 ± 646 — essentially indistinguishable from vanilla, confirming that upper clipping does nothing for outliers.
Clipped softmax with γ = -0.03, ζ = 1.03 (both lower and upper clipping): FP16 perplexity 4.43 ± 0.03, max infinity norm 22 ± 3, average kurtosis 73 ± 8, W8A8 perplexity 4.56 ± 0.05 — similar to lower-clipping-only, confirming ζ is irrelevant.

The intermediate values of γ show a monotonic trend: as |γ| increases from 0.003 to 0.03, the max infinity norm drops from 688 to 20 and the average kurtosis drops from 2149 to 80, while FP16 perplexity actually improves slightly (from 4.49 to 4.41). The paper interprets this as evidence that "the network is helped with learning the 'no-op' updates more easily" — the explicit mechanism for producing zeros removes a burden, and performance may slightly benefit.

The same pattern is confirmed for ViT in Table 8, where γ = -0.003 (with ζ = 1) achieves FP32 accuracy 78.26 ± 0.06%, max infinity norm 99 ± 36, W8A8 accuracy 76.49 ± 0.48%, compared to vanilla (γ = 0, ζ = 1): FP32 accuracy 78.80 ± 0.42%, max infinity norm 426 ± 69, W8A8 accuracy 71.27 ± 0.88%. Upper clipping alone (γ = 0, ζ up to 1.01) produces no meaningful improvement. This cross-architecture replication strengthens the conclusion that only the zero-clipping capability matters.

Clipped Softmax: `γ = -α/T` Parameterization Across Sequence Lengths (Section 5.2)

Figure 6 presents the key result: the γ = -α/T parameterization makes clipped softmax adaptive to sequence length, with α ∈ [2, 4] working well across T ∈ {32, 64, 128, 192, 256}. For BERT-6L trained on WikiText-103:

Across all sequence lengths, α ∈ {1/4, 1/2} (weak stretch) produces FP16 log-perplexity similar to or slightly better than vanilla softmax, but the maximum infinity norm remains high (within roughly an order of magnitude of vanilla), meaning outliers are not sufficiently suppressed.
α ∈ {8} (very strong stretch) suppresses outliers extremely effectively (maximum infinity norm drops by approximately 100× or more across sequence lengths) but consistently degrades FP16 log-perplexity relative to vanilla, especially at short sequence lengths — the stretch is too aggressive, over-clipping attention probabilities that were genuinely informative.
α ∈ {2, 4} achieves the best tradeoff: maximum infinity norm drops by approximately 10-100× (note the logarithmic y-axis in Figure 6b) while FP16 log-perplexity remains close to or slightly better than vanilla across all sequence lengths. The paper states that these values "significantly dampen the magnitude of outliers while maintaining good FP16 perplexity across all explored sequence lengths."

The fact that the same α range works across sequence lengths validates the parameterization's design rationale — the clipping threshold α/T is relative to the uniform-attention baseline 1/T, so its semantic meaning ("clip probabilities that are α times smaller than average") is sequence-length-invariant.

Gated Attention: Bias Initialization Sweep (Section 5.3)

Figure 7 presents the impact of bias initialization for the Linear gated attention on BERT-6L and ViT. The key finding is that the method is robust across a wide range of initial gate openness, but extremes degrade performance. For BERT-6L (Figure 7a):

π_init around 0.5-0.9 (gates start open to partially open): FP16 log-perplexity is close to or slightly better than vanilla, and W8A8 log-perplexity is dramatically better — roughly 1-2 orders of magnitude lower (note that Figure 7a plots relative log-perplexity on what appears to be a linear scale, but the W8A8 improvement is unambiguous from the quantized perplexity values in Table 5).
π_init = 0.12 (gates start mostly closed): FP16 performance degrades noticeably relative to vanilla, and W8A8 performance, while still much better than vanilla's quantized result, is worse than the moderate-π_init configurations. The gates starting too closed makes it hard for the model to learn to open them when attention is needed.
π_init ≈ 0.88 (gates start mostly open): FP16 performance is comparable to vanilla, but outlier suppression is weaker — the gates haven't learned to close before some outlier formation occurs.

For ViT (Figure 7b), a similar pattern holds but shifted toward lower π_init values: π_init ∈ [0.1, 0.5] works well, with π_init ≈ 0.12 showing the best quantized accuracy, while π_init ≈ 0.88 shows little improvement over vanilla in quantized performance — almost all the way open at initialization allows outliers to form before the gates learn to close.

The paper interprets the wide acceptable range ([0.25, 0.9] for BERT, [0.1, 0.5] for ViT) as "relative robustness of our method to this hyperparameter." The difference between BERT and ViT optimal ranges may reflect differences in training duration, model capacity, or the nature of the no-op requirement in vision vs. language.

Main Results: Across All Three Architectures (Section 5.4)

Table 2 is the central results table, comparing vanilla pre-training against the best clipped softmax and gated attention configurations for BERT, OPT, and ViT. The headline is: in almost all cases, both methods reduce outlier metrics by 1-3 orders of magnitude, maintain or slightly improve floating-point performance, and bring W8A8 quantized performance within ~1% or ~0.1 perplexity of the floating-point baseline.

BERT-base (perplexity, lower is better):

Vanilla: FP16 4.49 ± 0.01, max inf norm 735 ± 55, avg kurtosis 3076 ± 262, W8A8 1294 ± 1046.
Clipped softmax (γ = -0.025 from Table 5): FP16 4.39 ± 0.00, max inf norm 21.5 ± 1.5, avg kurtosis 80 ± 6, W8A8 4.52 ± 0.01 — a ~286× reduction in max inf norm, ~38× reduction in kurtosis, and W8A8 perplexity essentially matching the FP16 baseline.
Gated attention (MLP, n_hid = 4, from Table 5): FP16 4.45 ± 0.03, max inf norm 39.2 ± 26.0, avg kurtosis 201 ± 181, W8A8 4.65 ± 0.04 — slightly higher variance but still a ~19× reduction in max inf norm, ~15× reduction in kurtosis, and W8A8 perplexity within 0.2 of the FP16 baseline.

OPT-125m (perplexity, lower is better):

Vanilla: FP16 15.84 ± 0.05, max inf norm 340 ± 47, avg kurtosis 1778 ± 444, W8A8 21.18 ± 1.89.
Clipped softmax (γ = -12/512 from Table 6): FP16 16.29 ± 0.07, max inf norm 63.2 ± 8.8, avg kurtosis 19728 ± 7480, W8A8 37.20 ± 2.40 — this is the one failure case. The max inf norm does decrease (~5×), but kurtosis actually increases dramatically, and W8A8 perplexity is substantially worse than vanilla. The paper explicitly notes: "At the moment, we do not have an explanation of why this is the case and leave it for future work."
Gated attention (Linear, π_init = 0.25, from Table 6): FP16 15.55 ± 0.05, max inf norm 8.7 ± 0.6, avg kurtosis 18.9 ± 0.9, W8A8 16.02 ± 0.07 — a ~39× reduction in max inf norm, ~94× reduction in kurtosis, and W8A8 perplexity within 0.47 of the FP16 baseline, which is slightly better than the vanilla FP16 perplexity (15.55 FP16 for gated attention vs. 15.84 FP16 for vanilla).

ViT-S/16 (accuracy, higher is better):

Vanilla: FP32 80.75 ± 0.10%, max inf norm 359 ± 81, avg kurtosis 1018 ± 471, W8A8 69.24 ± 6.93%.
Clipped softmax (γ = -0.0001, from Table 7): FP32 80.89 ± 0.13%, max inf norm 73.7 ± 14.9, avg kurtosis 22.9 ± 1.6, W8A8 79.77 ± 0.25% — a ~4.9× reduction in max inf norm, ~44× reduction in kurtosis, and W8A8 accuracy drops less than 1.2 percentage points from the FP32 baseline.
Gated attention (Linear, π_init = 0.5, from Table 7): FP32 81.01 ± 0.06%, max inf norm 79.8 ± 0.5, avg kurtosis 19.9 ± 0.3, W8A8 79.82 ± 0.11% — very similar to clipped softmax, with W8A8 accuracy within 1.2 percentage points of FP32.

A noteworthy detail: the ViT results include an additional architectural change — adding LayerNorm after the patch embeddings — because the paper found that "distinct outliers already originate after the patch embeddings." The Table 7 results include a vanilla baseline with this LayerNorm (FP32 80.98 ± 0.08%, max inf norm 81.1 ± 2.5, W8A8 79.62 ± 0.06%), which already shows dramatically fewer outliers than the vanilla baseline without the extra LayerNorm (max inf norm 359 ± 81). The proposed methods provide additional improvement on top of this stronger baseline.

The cross-architecture pattern is clear: except for clipped softmax on OPT, both methods produce models whose FP16/FP32 performance is competitive with or slightly better than vanilla, whose outlier metrics are reduced by 1-3 orders of magnitude, and whose W8A8 quantized performance is within ~1% accuracy or ~0.5 perplexity of the floating-point baseline — a qualitative change from the vanilla models, where quantization causes catastrophic degradation (BERT W8A8 perplexity 1294 vs. 4.49 FP16; ViT W8A8 accuracy 69.24% vs. 80.75% FP32; OPT W8A8 perplexity 21.18 vs. 15.84 FP16 without the weight-decay-on-LN fix).

Scaling to Larger Models (Section 5.4, Table 3)

Table 3 addresses the question of whether gated attention remains effective at scale. For OPT-350m and OPT-1.3B, trained for 10^5 steps (fewer than the 125k steps for OPT-125m due to compute constraints):

OPT-350m:

Vanilla: FP16 perplexity 13.19, max inf norm 253, avg kurtosis 2689, W8A8 37.52 ± 3.84.
Gated attention: FP16 perplexity 13.01, max inf norm 65.4, avg kurtosis 261, W8A8 14.42 ± 0.06 — a ~3.9× reduction in max inf norm, ~10× reduction in kurtosis, and W8A8 perplexity within 1.41 of FP16.

OPT-1.3B:

Vanilla: FP16 perplexity 12.13, max inf norm 428, avg kurtosis 2756, W8A8 989.6 ± 175.
Gated attention: FP16 perplexity 12.21, max inf norm 67.2, avg kurtosis 444, W8A8 29.95 ± 0.42 — a ~6.4× reduction in max inf norm, ~6.2× reduction in kurtosis, and W8A8 perplexity improved from ~990 (essentially a broken model) to ~30 (usable, ~2.5× the FP16 perplexity, but dramatically better than without the method).

The paper notes that the kurtosis reduction at 1.3B (~6.2×) is less dramatic than at 125m (~94×), but the quantized perplexity improvement is still massive — from a model that has essentially collapsed (989.6 perplexity) to one that is within a factor of ~2.5 of its floating-point performance. Given the reduced training steps (10^5 vs. 125,000), this may understate the method's effectiveness at scale when trained to convergence. The paper acknowledges compute constraints as the limiting factor for the scaling experiments.

Qualitative Results (Section 5.5)

Figure 8 provides side-by-side visualizations of the attention patterns learned by vanilla softmax, clipped softmax, and gated attention for BERT-base on the same data sequence. The key qualitative finding is that both methods still produce no-op-like behavior, but through clean mechanisms that don't require pathological activation values.

Vanilla softmax (Figure 8a, attention head #3, layer #11): The attention probabilities show almost all mass on [SEP] and punctuation tokens. The values for those tokens are near-zero (~0.02 magnitude), so the attention output is very small — a soft no-op achieved through saturating the softmax, which requires large FFN outputs in the previous layer.
Clipped softmax (Figure 8b, attention head #8, layer #11): The attention probabilities show a more diffuse distribution — the softmax no longer needs to saturate because probabilities below the clipping threshold simply become exact zeros. The attention output is correspondingly small, but without requiring extreme softmax logits. The paper notes that "smaller attention weights are generally more diffused while higher weights are more saturated (which comes from the stretching and clipping)."
Gated attention (Figure 8c, attention head #5, layer #11): The gating probabilities π = sigmoid(G(x)) are shown alongside the attention probabilities and values. The gate is partially closed for most tokens (show in blue/green, values around 0.2-0.4) and more open for some (yellow, values around 0.6-0.8), modulating the attention output without the attention distribution itself needing to be extreme. The paper notes that "the output of the softmax is significantly different since the update of the hidden representation is now further modulated by gating probabilities."

The paper reports finding "similar patterns in multiple attention heads, but the exact head indices where we observed such patterns depend on random initialization" — the no-op behavior is learned consistently across heads, but which specific heads take on the no-op role varies with random seed.

Appendix A.1 (Figures 14 and 15) provides additional examples across multiple data sequences for clipped softmax and gated attention, confirming that the qualitative pattern generalizes.

Ablation Studies and Robustness Checks

Clipped softmax stretch parameter ζ (upper clipping): Tables 1 and 8 show that ζ > 1 provides no benefit beyond γ < 0 alone. For BERT-base, γ = -0.03, ζ = 1 achieves W8A8 perplexity 4.55 ± 0.01; γ = -0.03, ζ = 1.03 achieves 4.56 ± 0.05 — statistically indistinguishable. For ViT, γ = -0.003 achieves W8A8 accuracy 76.49%; γ = -0.003, ζ = 1.003 achieves 76.37% — again indistinguishable. The paper concludes that "only the lower-range clipping allows exact zeros matter" and uses ζ = 1 going forward. Non-obvious implication: this means the model does not need to learn complete one-hot attention — it just needs to learn to completely ignore some tokens. Producing exact ones (fully focused attention on a single token) is apparently not a requirement that standard softmax struggles with.

Gating architecture variants: Table 5 compares Linear, MLP (n_hid = 4), MLP (n_hid = 64), and All-heads-linear gating for BERT-base. All configurations reduce outlier metrics and improve W8A8 perplexity substantially vs. vanilla. Linear gating with π_init = 0.75 achieves W8A8 perplexity 4.88 ± 0.22; MLP with n_hid = 4 achieves 4.65 ± 0.04; All-heads-linear achieves 4.67 ± 0.03. The differences are relatively small, indicating that the gating mechanism itself matters more than the expressivity of the gating function. The paper selects the simpler Linear variant for most experiments because it adds minimal parameters. Table 6 for OPT compares Linear vs. All-heads-linear (with and without weight decay on LayerNorm γ), finding similar performance between them. Non-obvious finding: the All-heads-linear variant, which mixes information across attention heads, does not provide a clear advantage over per-head independent linear gates, suggesting the gating decision does not require cross-head coordination.

Weight decay on LayerNorm γ for OPT: Table 6 reveals a significant finding not emphasized in the main text but important for OPT practitioners. The vanilla OPT-125m without weight decay on LN γ achieves max inf norm 340 ± 47 and W8A8 perplexity 21.18 ± 1.89. Adding weight decay on LN γ (Table 6, "Vanilla" with LN γ wd = ✓) reduces this to max inf norm 87.7 ± 31.9 but slightly worsens FP16 perplexity (15.96 vs. 15.84) and makes W8A8 worse (39.46 ± 16.59 vs. 21.18 ± 1.89). This is a partial but incomplete fix. The gated attention variants with weight decay on LN γ achieve the best overall results: max inf norm 7.3-10.8 and W8A8 perplexity 16.02-16.23 for the Linear variants. The paper notes: "we found that applying the weight decay on LayerNorm weights γ (which isn't the case, by default) has a strong effect on reducing the outliers' magnitude while yielding the comparable FP16 performance." Non-obvious insight: the interaction between LayerNorm weight decay and gated attention is synergistic — each reduces outliers partially, and together they achieve the best results.

LayerNorm after patch embeddings for ViT: Table 7 includes a split between ViT trained without and with an extra LayerNorm after the patch embeddings. The vanilla model without extra LN: max inf norm 359 ± 81, W8A8 accuracy 69.24 ± 6.93%. Vanilla with extra LN: max inf norm 81.1 ± 2.5, W8A8 accuracy 79.62 ± 0.06% — a dramatic improvement from this single architectural change. The paper's proposed methods are tested on top of this stronger baseline and provide incremental improvement: clipped softmax (with LN) achieves W8A8 79.77 ± 0.25%; gated attention (with LN) achieves 79.82 ± 0.11%. Non-obvious finding: the ViT architecture as commonly defined (without LayerNorm after patch embeddings) has a pre-existing outlier problem at the embedding level that is distinct from the softmax-driven outlier problem in later layers. Adding the LayerNorm largely fixes this, and then the proposed methods address the remaining outlier sources. This suggests that quantizability issues in vision transformers have at least two independent causes.

Fine-tuning a pre-trained model with gated attention (Appendix B.6): Table 9 addresses a practical concern: do you need to pre-train from scratch, or can you fine-tune a pre-trained model with gated attention added post-hoc? For OPT-1.3B fine-tuned for 4,000 steps on BookCorpus + Wikipedia: vanilla fine-tuning achieves FP16 perplexity 29.46, max inf norm 79.3, avg kurtosis 2086. Fine-tuning with gated attention (gates initialized to π_init = 0.5, multiplied by 2 to approximate vanilla attention output initially, plus a small activation regularization at FFN outputs) achieves FP16 perplexity 29.18, max inf norm 50.9, avg kurtosis 665. This is a ~1.6× reduction in max inf norm and ~3.1× reduction in kurtosis — meaningful but far less dramatic than pre-training from scratch with gated attention (where max inf norm dropped from 428 to 67.2, a ~6.4× reduction). Non-obvious finding: fine-tuning can partially reduce outliers, but the pre-existing outliers from vanilla pre-training are never fully eliminated because they are already embedded in the model's representations — the gates can modulate future updates but cannot undo the existing outlier values.

Low-bit quantization beyond W8A8 (Appendix B.7): Table 10 pushes BERT-base to W6A8, W4A8, and W6A6 configurations to test whether the quantizability improvements extend to more aggressive compression. For vanilla BERT: W8A8 perplexity 1294 ± 1046, W6A8 (MSE weight ranges) 6.49 ± 0.38, W4A8 (MSE weight ranges) 6.52 ± 0.02, W6A6 42.8 ± 11.7. For clipped softmax: W8A8 4.52 ± 0.01, W6A8 (MSE) 4.56 ± 0.01, W4A8 (MSE) 4.90 ± 0.02, W6A6 6.64 ± 0.14. For gated attention: W8A8 4.65 ± 0.04, W6A8 (MSE) 4.71 ± 0.03, W4A8 (MSE) 5.02 ± 0.03, W6A6 5.90 ± 0.11. Key finding: the proposed methods maintain dramatically better quantized performance across all bitwidths, but the absolute degradation at W4A8 and W6A6 is still noticeable (clipped softmax: 4.90 W4A8 vs. 4.39 FP16; gated attention: 5.02 W4A8 vs. 4.45 FP16), indicating that while architectural fixes make W8A8 essentially lossless, pushing to very low bitwidths still requires additional quantization techniques. The paper notes that "achieving good results with low-bit activation quantization in general is a challenging problem" and recommends combining the methods with more advanced quantization techniques for sub-8-bit targets.

Negative result: Clipped softmax on OPT (Table 6): The paper transparently reports that clipped softmax fails on OPT-125m. Across all explored γ values (from -1/512 to -12/512), W8A8 perplexity remains worse than vanilla (37.20 vs. 21.18 for the best clipped softmax vs. vanilla, comparing the "LN γ wd = ✓" condition). Kurtosis increases substantially (19727 for the best perplexity vs. 2080 for vanilla). This is the only clear failure case across all architectures and methods, and the paper's acknowledgment — "At the moment, we do not have an explanation of why this is the case and leave it for future work" — is an honest admission of an unresolved puzzle.

Negative result: ReST^EM revision model degradation (Appendix K, Figure 16): Not applicable to this paper — this is a finding from the reference example paper, not the paper being analyzed. There is no ReST^EM experiment in the paper under analysis. (This is an integrity check: do not hallucinate content from the example.)

Critical Assessment

The paper makes two major claims: (1) activation outliers in transformers are caused by attention heads learning to approximate a no-op by attending to delimiter tokens with small value outputs, which requires pushing softmax inputs toward infinity and producing large FFN outputs in preceding layers, and (2) clipped softmax and gated attention eliminate this root cause, producing models that can be quantized to W8A8 via simple PTQ with minimal accuracy loss.

Claim 1 (causal mechanism): Supported by strong correlational evidence, but not experimentally verified through intervention. The outlier analysis in Section 3 and Appendices A.1-A.2 is observational: it shows that outlier dimensions in layer L correspond to attention heads in layer L+1 that attend to delimiter tokens with small values, and that this pattern appears consistently across BERT, OPT, and ViT. This establishes a robust correlation and is consistent with the proposed causal chain. However, the paper never performs the direct causal intervention that would prove the hypothesis experimentally: training a model with the standard architecture but preventing the no-op behavior (e.g., by masking delimiter tokens during pre-training and seeing whether outliers still form elsewhere), or directly measuring whether the gradient signal through the softmax to FFN outputs in layer L is larger for dimensions that become outliers. The causal claim rests on the plausibility of the mechanism and the success of the architectural fixes that the mechanism motivated. The fact that the fixes work is consistent with the mechanism being correct, but does not constitute a direct test of it — the fixes could work for reasons other than the hypothesized mechanism. The paper's qualitative results (Figure 8) show that both methods still learn no-op-like behavior through different mechanisms, which is suggestive but not definitive. A stronger causal test would involve, for example, measuring whether the gradient norm through the softmax to preceding-layer FFN outputs is systematically larger for outlier dimensions than non-outlier dimensions in a vanilla model, and whether this gradient imbalance disappears in the clipped softmax and gated attention models.

Claim 2 (architectural fixes eliminate outliers and enable simple W8A8 PTQ): Largely supported, but with important caveats about scope and evaluation. The evidence is compelling for the models tested: BERT-base shows W8A8 perplexity within 0.03-0.16 of FP16 (Table 2); ViT-S/16 within 1.0-1.2 percentage points of FP32 top-1 (Table 2); OPT-125m with gated attention within 0.47 perplexity of FP16 (Table 2). However, several qualifications apply:

Single dataset per model, no fine-tuned downstream evaluation: Perplexity and ImageNet top-1 accuracy are reasonable metrics for pre-training quality, but it is an open question whether the architectural modifications affect fine-tuning performance on downstream tasks. A model that achieves identical pre-training perplexity but learns different internal representations might transfer differently to GLUE tasks or object detection. The paper provides no fine-tuning experiments for BERT or ViT, and no generation quality evaluation for OPT beyond perplexity. This matters because the outlier dimensions have been shown to be important for task predictions [31, 49]; if clipped softmax or gated attention change how those dimensions are used, downstream task performance could differ even if perplexity matches.
Limited scale: The largest model tested with pre-training from scratch is OPT-1.3B (Table 3), and that model was trained for only 10^5 steps — substantially fewer than the 125,000 steps used for OPT-125m and far fewer than the training of production-scale models (which are typically 10-100× larger and trained on 10-100× more tokens). The scaling trend from Table 3 is encouraging (gated attention helps at 350m and 1.3B), but the kurtosis reduction weakens at 1.3B (~6.2× vs. ~94× at 125m), and the W8A8 vs. FP16 perplexity gap is larger (29.95 vs. 12.21, a ~2.5× ratio) than at 125m (16.02 vs. 15.55, a ~1.03× ratio). Whether these trends continue or reverse at larger scales is unknown.
The W8A8 PTQ configuration is simple but not state-of-the-art: The paper uses per-tensor static quantization with min-max or MSE range estimation, which is a reasonable baseline but substantially weaker than the quantization techniques it positions itself against. GPT3.int8(), SmoothQuant, and Outlier Suppression all employ more sophisticated quantization schemes specifically designed to handle outliers. The paper's claim is that its models "can be effortlessly quantized using PTQ without strong degradation of performance," and this is demonstrated for the simple quantizer used. But the paper never shows that its models + simple PTQ outperform vanilla models + state-of-the-art quantization techniques (e.g., SmoothQuant) at the same bitwidth. This is not a fatal omission — the paper's framing is architectural rather than competitive with quantization methods — but it means the practical value proposition relative to existing post-training fixes is unquantified. A practitioner reading this paper cannot determine from the reported results whether they should adopt clipped softmax pre-training or simply apply SmoothQuant to their existing vanilla model.
The clipped softmax failure on OPT is unexplained: The fact that clipped softmax degrades kurtosis and W8A8 perplexity on OPT-125m (Table 6) while working well on BERT and ViT is a significant limitation. Since OPT is the only autoregressive decoder tested with clipped softmax, it raises the question of whether the method systematically fails on decoder architectures or on causal language modeling objectives. The paper doesn't investigate this further (e.g., testing clipped softmax on a small GPT-2 variant or on encoder-decoder architectures), leaving a gap in understanding the method's applicability domain. The failure is particularly puzzling because the hypothesized mechanism (softmax saturation → outliers) should apply equally to causal attention. Potential hypotheses not explored: the causal attention mask changes the effective sequence length T per position, which the γ = -α/T parameterization doesn't account for; the autoregressive training objective creates different gradient dynamics through the softmax; or OPT's specific architectural choices (pre-LayerNorm placement) interact badly with clipped softmax in ways that don't affect BERT's post-LayerNorm placement.
The γ = -α/T parameterization is validated only up to T = 256: The sequence length sweep in Figure 6 covers 32 to 256, but modern language models routinely use sequence lengths of 2048, 4096, or more. Extrapolating the parameterization to much longer sequences is not empirically validated, and the behavior of the α ∈ [2, 4] recommendation at those lengths is unknown. At T = 4096, γ = -2/4096 ≈ -0.0005, which is a very weak stretch — the clipping threshold becomes α/T ≈ 0.001, meaning only attention probabilities below 0.1% of the uniform baseline get clipped. Whether this is still sufficient to prevent outlier formation at scale is untested.
No training-from-scratch comparison with QAT: The paper argues that architectural fixes are simpler than quantization-aware training. But a comparison where the same model architecture, same training budget, and same quantization scheme are used, with the only difference being vanilla + QAT vs. clipped softmax/gated attention + PTQ, would directly test this claim. Without it, the relative merits of "fix the architecture and use PTQ" vs. "use QAT on the standard architecture" are unknown — QAT might achieve equally good quantized performance without architectural changes, at the cost of requiring a QAT training pipeline instead of a standard one. The paper's framing ("without any additional effort" vs. QAT's requirement to "train the entire network for more epochs") is reasonable but the tradeoff in practice depends on whether the practitioner already has a PTQ pipeline or a QAT pipeline.
Computational overhead reporting is incomplete: Table 11 reports training time in hours, but does not report inference time or memory consumption for the quantized models. Since the paper's motivation is efficient inference via INT8 quantization, an end-to-end latency and throughput comparison (vanilla FP32/FP16 vs. vanilla W8A8 with SmoothQuant vs. clipped softmax/gated attention W8A8) would directly address the practical motivation. Without this, the paper demonstrates that PTQ accuracy is preserved but doesn't close the loop on whether the quantized models are actually faster or more memory-efficient than their floating-point counterparts in deployment.
The qualitative analysis uses a fine-tuned model (MNLI) but the pre-training experiments use the pre-trained model directly: Section 3's outlier analysis is performed on BERT-base fine-tuned on MNLI, while the pre-training results in Section 5 are on BERT-base evaluated on MLM perplexity. If outlier patterns change during fine-tuning (e.g., the model develops new no-op requirements for the fine-tuning task), the diagnostic from Section 3 may not perfectly map to the pre-training setting where the architectural fixes are applied. This is a minor concern since the paper also shows outliers in pre-trained checkpoints (the vanilla BERT-base in Table 2 has max inf norm 735), but it's worth noting that the detailed head-level analysis comes from a fine-tuned model.
Missing experiment: combining clipped softmax with gated attention. The paper presents them as independent modifications but never tests whether applying both simultaneously yields further improvements, or whether they are redundant (both addressing the same root cause through different mechanisms might mean combining them provides no additional benefit). This is a natural ablation that would strengthen the paper's understanding of the mechanism.

Overall assessment: The paper succeeds at its primary goal: establishing a mechanistic hypothesis for why transformers develop activation outliers, and showing that two architectural modifications motivated by that hypothesis produce models that are dramatically more quantization-friendly under simple PTQ. The evidence is consistent, replicated across three architectures, and the improvements are large enough (1-3 orders of magnitude in outlier metrics) that they are unlikely to be statistical artifacts. The paper's main weaknesses are the lack of downstream task evaluation, the limited scale of training-from-scratch experiments, the unexplained clipped softmax failure on OPT, and the absence of direct comparisons against state-of-the-art quantization methods applied to vanilla models — all of which leave the practical value proposition less sharply defined than it could be. The paper is best understood as a diagnostic contribution with a validated architectural solution, rather than as an exhaustive deployment playbook. A practitioner considering adopting these methods would need to: (1) verify that the FP16 performance improvements hold at their target model scale and on their downstream tasks, (2) compare against applying SmoothQuant or similar to a vanilla model (since that may achieve comparable W8A8 accuracy with less training cost if a pre-trained vanilla checkpoint already exists), and (3) if using clipped softmax on autoregressive decoders, validate that the OPT failure case doesn't generalize to their model family.

6. Limitations and Trade-offs

6.1 Clipped Softmax Fails on OPT Without Explanation

The constraint. Clipped softmax — which works well on BERT and ViT, reducing max infinity norm by ~30× and ~5× respectively — degrades performance on OPT-125m. Table 6 shows that across all explored γ values (from -1/512 to -12/512), W8A8 perplexity remains worse than vanilla (37.20 ± 2.39 vs. 21.18 ± 1.89 for the best clipped softmax vs. vanilla, under the "LN γ wd = ✓" condition where both are comparable). Worse, kurtosis increases dramatically (19,727 ± 7,479 for the best-perplexity clipped softmax vs. 2,080 ± 1,460 for vanilla) — clipped softmax makes the outlier problem actively worse in the causal LM decoder setting. The paper is transparent about this:

"The only case where our method failed to perform well was the clipped softmax applied to OPT. At the moment, we do not have an explanation of why this is the case and leave it for future work."

The consequence. This failure is consequential because OPT is the only autoregressive decoder tested with clipped softmax. It is therefore unknown whether the failure is specific to OPT (its pre-LayerNorm placement, its training recipe, its weight initialization scheme), or whether clipped softmax systematically fails on all autoregressive decoder architectures and/or causal language modeling objectives. If the latter, clipped softmax's applicability would be restricted to encoder-only architectures, which would substantially limit its practical relevance given that most large-scale deployment of transformers today uses decoder-only models (GPT family, LLaMA family, etc.). The hypothesized mechanism — softmax saturation → FFN outliers — should apply equally to causal attention, so the failure suggests an unmodeled interaction with autoregressive training dynamics or the causal attention mask that the paper's diagnostic framework does not capture.

What evidence exists in the paper. Table 6 provides the full sweep of γ values for OPT-125m. The quantitative story is clear: FP16 perplexity remains roughly flat across γ values (15.86–16.29 vs. 15.96 for vanilla), but kurtosis explodes (from ~2,000 to ~19,000 at γ = -12/512) and W8A8 perplexity never improves. The paper does not provide any diagnostic analysis for this failure — no attention pattern visualizations for OPT with clipped softmax, no per-layer infinity norm analysis, no comparison of gradient norms through the softmax. The failure is reported but not investigated.

Mitigation status. Not addressed. The paper explicitly states it has no explanation and leaves the investigation to future work. Gated attention works well on OPT (Table 6: max inf norm 8.7 ± 0.6, W8A8 16.02 ± 0.07), so for OPT-like decoders the paper effectively recommends gated attention as the sole viable method, but provides no guidance on whether this pattern generalizes. A practitioner considering clipped softmax for a decoder-only LLM has no evidence either way from this paper.

6.2 Difficulty Estimation Cost Is Not Accounted For and No Cheap Alternative Exists

The constraint. Both proposed methods require training the model from scratch with the architectural modification in place — they are pre-training protocol changes, not post-hoc fixes. The paper acknowledges that this is expensive:

"One of the drawbacks of our proposed framework is that it requires training from scratch, which could be expensive when applied to very large models."

The fine-tuning experiment in Appendix B.6 partially addresses this concern (showing that fine-tuning a pre-trained model with gated attention for 4,000 steps reduces max inf norm from 79.3 to 50.9 — a meaningful but far smaller improvement than training from scratch, where the reduction is ~39×), but the fine-tuned model still has substantially larger outliers than a model trained from scratch with gated attention. For clipped softmax, there is no fine-tuning option — it is a drop-in replacement for the softmax function itself, and adding it to a pre-trained model would require the model to learn new attention distributions that are fundamentally incompatible with the pre-trained weights.

The consequence. The practical value proposition depends on the comparison: (cost of training from scratch with clipped softmax / gated attention) vs. (cost of applying a post-training quantization workaround like SmoothQuant or GPT3.int8() to an existing vanilla checkpoint). For practitioners who receive a pre-trained model (the most common deployment scenario), the paper's methods are not applicable without re-running pre-training — which for large models (7B+ parameters) is a multi-million-dollar proposition that most practitioners cannot afford. The paper's own compute budget (Appendix D: ~1,400 GPU-days on A100s total, ~320 GPU-days for the main results) is modest by modern standards and does not include any model larger than 1.3B parameters. The scaling experiments in Table 3 train OPT-350m and OPT-1.3B for only 10^5 steps (vs. 125,000 for OPT-125m), which is substantially less than convergence — the authors acknowledge this is "due to compute constraints." The true cost of training a converged large model with these modifications from scratch is unknown and likely substantial.

What evidence exists in the paper. Table 11 reports training times: BERT-base (92.8 hours vanilla, 93.6 clipped softmax, 97.7 gated attention on A100), OPT (53.6/54.4/55.7 hours), ViT (101.8/104.0/110.8 hours). These are modest for the model sizes tested. But the paper provides no cost comparison against applying SmoothQuant or GPT3.int8() to a pre-trained vanilla checkpoint, which would typically take minutes to hours of calibration on a single GPU. The fine-tuning experiment (Table 9) provides partial evidence that gated attention can be retrofitted, but the outlier reduction is substantially weaker than training from scratch, and no W8A8 quantization result is reported for the fine-tuned model — so we do not know whether the residual outliers are small enough to enable simple PTQ.

Mitigation status. Partially addressed via the fine-tuning experiment (Appendix B.6), but only for gated attention and only at one model scale (OPT-1.3B). The paper does not report quantized performance of the fine-tuned model, so the practical endpoint — "can I take someone else's pre-trained OPT-1.3B, fine-tune with gated attention for cheap, and get W8A8 PTQ to work?" — is not answered. The paper acknowledges the training-from-scratch cost as a drawback but frames their work as establishing a new pre-training protocol, which inherently accepts this cost as the price of architectural improvement.

6.3 No Downstream Task Evaluation — Pre-Training Metrics May Not Generalize

The constraint. All results report pre-training metrics: masked language modeling perplexity for BERT, causal language modeling perplexity for OPT, and ImageNet top-1 accuracy for ViT. The paper never fine-tunes the modified models on downstream tasks (e.g., GLUE for BERT, generation quality benchmarks for OPT, object detection or segmentation for ViT) and reports those results. This is a significant gap because prior work [31, 49] has shown that the outlier dimensions are functionally important for task performance — setting them to zero degrades model predictions. The paper's own qualitative analysis (Section 3) is performed on BERT-base fine-tuned on MNLI, demonstrating that the no-op behavior and associated outliers appear in the fine-tuned model, but the training-from-scratch experiments never close the loop by fine-tuning the modified pre-trained models and comparing downstream performance.

The consequence. A model that achieves identical pre-training perplexity to vanilla but uses different internal mechanisms (clipped softmax's hard zeros, gated attention's multiplicative suppression) might transfer differently to downstream tasks. For example, if the no-op behavior that standard attention achieves through delimiter-token attention is used differently during fine-tuning than during pre-training, the modified architecture might learn representations that are suboptimal for task-specific adaptation. Conversely, if the proposed methods help the model learn cleaner representations (as the slight FP16 improvements in Table 2 suggest), they might transfer better. Neither direction is tested. A practitioner choosing between vanilla pre-training + SmoothQuant and clipped softmax pre-training + simple PTQ cannot determine from the paper's evidence whether their downstream task accuracy will be preserved, improved, or degraded.

What evidence exists in the paper. None. All evaluation is on pre-training validation metrics. This limitation is not explicitly acknowledged in the paper. The qualitative MNLI analysis (Section 3) is strictly diagnostic (showing the existence of the no-op/outlier pattern) and does not report fine-tuning accuracy for the modified models. This is a notable absence given that the outlier dimensions were previously shown to be important for task predictions [31], which the paper cites but does not experimentally engage with.

Mitigation status. Not addressed. The paper does not mention downstream evaluation as future work. This is one of the most significant gaps for deployment decisions — without knowing how the modifications affect fine-tuning, the practical recommendation to "train all transformers with these modifications" is premature.

6.4 The Causal Mechanism Is Hypothesized, Not Experimentally Verified

The constraint. The paper proposes a specific causal chain: attention heads want a no-op → softmax cannot produce exact zeros → softmax inputs must be large → FFN outputs in previous layer must be extreme to survive LayerNorm → outliers form → quantization fails. This chain is supported by strong correlational evidence (the co-occurrence of outlier dimensions, delimiter-attending heads, small value outputs, and per-layer outlier magnitude peaks in Figures 1–3 and Appendices A.1–A.2), but the paper does not perform any interventional experiment that would directly test the causal direction. The paper's primary validation of the hypothesis is the success of the architectural fixes that it motivated — but as a verification strategy, this is logically indirect: the fixes could work for reasons other than the hypothesized mechanism, and confirming the mechanism would require directly manipulating the causal chain and observing the predicted effect.

The consequence. Without direct causal verification, the paper's diagnostic framework — while plausible and consistent with all observations — may be incomplete or partially incorrect. Alternative (or additional) mechanisms could contribute to outlier formation that are not addressed by clipped softmax or gated attention. For instance, outliers might also arise from the FFN's own representational needs independent of downstream attention requirements; from LayerNorm's gradient dynamics creating positive feedback on specific dimensions; or from interactions with the token embedding that project certain dimensions into regimes where outliers are likely. The clipped softmax failure on OPT (Section 6.1) hints at an unmodeled factor in the causal chain. If the mechanism were fully understood, OPT's failure should be predictable or at least diagnosable post-hoc — but the paper "does not have an explanation." This suggests the mechanistic understanding, while substantially better than prior work, is not complete.

What evidence exists in the paper. The evidence is entirely correlational and consistency-based. Section 3 and Appendices A.1–A.2 show that outlier dimensions in layer L correspond to attention heads in layer L+1 that attend to delimiter tokens/background patches with small values. Table 1 shows that γ < 0 (enabling exact zero attention probabilities) is what matters for outlier suppression, consistent with the hypothesis. Figure 8 shows that the proposed methods still exhibit no-op behavior but through clean mechanisms. What is missing: no measurement of gradient norms through the softmax to preceding-layer FFN outputs showing that outlier dimensions receive systematically larger gradients in vanilla models; no experiment where the hypothesized causal pathway is blocked (e.g., forcing uniform attention on delimiter tokens) to see if outliers still form; no comparison of FFN output distributions before and after LayerNorm to directly verify the "surviving normalization" claim.

Mitigation status. Not addressed. The paper does not frame the causal mechanism as a hypothesis requiring further verification — it presents it as the established explanation, with the architectural fixes as confirmation. The failure on OPT is treated as an exception to an otherwise correct theory rather than as evidence that the theory is incomplete. The paper does not suggest specific experiments to directly test the causal chain.

6.5 Scaling Behavior Beyond 1.3B Parameters and Standard Training Durations Is Unknown

The constraint. The largest model trained from scratch with the proposed methods is OPT-1.3B (Table 3), trained for only 10^5 steps — substantially fewer than the 125,000 steps used for OPT-125m and far fewer than the training duration of production-scale models. For BERT and ViT, only the base/small variants are tested (109M and 22M parameters, respectively). The paper explicitly acknowledges this scope limitation:

"While we studied the scalability of our method for models up to 1.3B size, we haven't explored the case of very large transformers that are trained for way longer."

The paper speculates that "given the fundamental understanding of the issue underlying our solutions, we expect the same effect on larger-scale models," but provides no evidence beyond the 125M → 350M → 1.3B trend.

The consequence. The scaling trend in Table 3 raises concerns about extrapolation. The kurtosis reduction from gated attention weakens as model size increases: ~94× at 125M, ~10× at 350M, ~6.2× at 1.3B. The W8A8-to-FP16 perplexity ratio grows from ~1.03× at 125M (16.02 vs. 15.55) to ~2.5× at 1.3B (29.95 vs. 12.21). If these trends continue to larger models, gated attention might provide insufficient outlier suppression at 7B, 13B, or 70B parameters to enable simple 8-bit PTQ. The reduced training steps at larger scales confound this trend — it is unclear whether the weakening effect is due to model scale, insufficient training, or both. Additionally, very large models trained on much more data may develop outlier patterns that the base/small-scale analysis doesn't capture, especially since Dettmers et al. [13] observed that in large enough models, outliers "start appearing after every linear layer, including query, key, and value projection layers" — a more pervasive pattern than what is analyzed here.

The γ = -α/T parameterization is validated only for sequence lengths 32–256 (Figure 6), but large language models routinely use sequence lengths of 2048–8192+. Extrapolating the α ∈ [2, 4] recommendation produces very weak stretch factors at long sequences (e.g., γ ≈ -0.0005 at T = 4096, α = 2). Whether such weak stretching still prevents outlier formation, or whether α needs to scale with sequence length in some other way, is unknown.

What evidence exists in the paper. Table 3 provides the only scaling data — three model sizes for OPT (125M, 350M, 1.3B), each trained for fewer steps than full convergence. The sequence length sweep in Figure 6 covers 32–256 for BERT-6L only. No scaling data exists for BERT beyond base or for ViT beyond S/16.

Mitigation status. The paper acknowledges the limited scale and states an expectation (without evidence) that the effects will generalize. No experiments or theoretical analysis are provided to justify this expectation. The paper suggests this as future work implicitly, but does not explicitly call for large-scale validation. Given the compute cost of training multi-billion-parameter models from scratch, this limitation is practically significant — it means a team considering adopting these methods for a large-scale training run must take a bet that the scaling trends from the paper's small-scale experiments hold, without empirical confirmation.

6.6 No Direct Comparison Against State-of-the-Art Quantization Methods Applied to Vanilla Models

The constraint. The paper evaluates quantization using simple per-tensor static PTQ with min-max or MSE range estimation — a deliberately basic quantizer, since the claim is that the architectural fixes make even "simple PTQ" work well. However, the paper never applies state-of-the-art quantization techniques (SmoothQuant [67], GPT3.int8() [13], Outlier Suppression [62, 63], or more advanced PTQ methods like AdaRound [44] or BRECQ [35]) to the vanilla models and reports those results alongside the proposed methods. The paper's comparisons are strictly against "vanilla model + simple PTQ," not "vanilla model + best available quantization."

The consequence. The practical value proposition is underspecified. A practitioner deciding whether to adopt clipped softmax pre-training faces this comparison: (A) train from scratch with clipped softmax, then apply simple PTQ → W8A8 accuracy ~X%; vs. (B) use an existing vanilla pre-trained checkpoint, apply SmoothQuant or GPT3.int8() → W8A8 accuracy ~Y%. The paper provides X (for BERT, ViT, OPT with gated attention) but never measures Y on its own models and datasets. If Y is close to X (i.e., existing quantization workarounds already recover most of the FP16 performance), then the value of re-training with architectural modifications is marginal — the benefit would be simplicity of the quantizer, not improved accuracy. If Y is substantially worse than X, then the architectural fixes provide a clear accuracy advantage that may justify re-training. The paper's framing ("we want to address the root cause... so that no special quantization handling is needed") makes simplicity the primary selling point, but simplicity alone is not quantified — it is unclear how much accuracy a practitioner sacrifices by choosing the "simple PTQ on vanilla model" path vs. the "architectural fix + simple PTQ" path.

The paper's own low-bitwidth experiments (Appendix B.7, Table 10) show that even with architectural fixes, pushing below 8-bit activations still requires more sophisticated quantization (MSE range estimation, and performance degrades noticeably at W4A8 and W6A6), suggesting that at very low bitwidths the architectural fixes alone are insufficient and must be combined with other techniques — which blurs the paper's "no special handling needed" value proposition.

What evidence exists in the paper. None. The paper does not run SmoothQuant, GPT3.int8(), Outlier Suppression, or any other quantization workaround on its vanilla models. The paper's "Background and related work" section (Section 2) describes these methods and positions the paper as addressing a different problem (root cause vs. workaround), but the experimental sections contain no head-to-head comparisons. This is a deliberate framing choice — the paper positions itself as an architectural contribution, not a quantization competition — but it leaves the deployment tradeoff unquantified.

Mitigation status. Not addressed. The paper does not acknowledge this as a limitation or suggest future work comparing against state-of-the-art quantization methods. The paper's claim is that the models can be "effortlessly quantized using PTQ without strong degradation of performance," which is demonstrated for the simple PTQ pipeline used. Whether existing quantization methods achieve similarly good results on vanilla models with comparable effort is an unanswered empirical question that affects the practical recommendation.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a diagnostic reframing with architectural consequences — it does not invent a new quantization technique, a new training objective, or a new model architecture from scratch. Rather, it establishes that the activation outlier problem in transformers is not an intrinsic training artifact but a symptom of a specific architectural limitation: the softmax function's inability to express exact zeros at finite input range. This reframing shifts the field's response to outliers from "how do we quantize around them?" to "why does the architecture force the model to create them, and can we fix that?"

The magnitude of this shift is significant but bounded. It does not replace post-training quantization workarounds (SmoothQuant, GPT3.int8(), Outlier Suppression) — those remain the practical answer for practitioners deploying existing pre-trained models. What it does is change the design target for future pre-training runs: if you are training a transformer from scratch, the paper provides evidence that two simple architectural changes (clipped softmax or gated attention) make the resulting model amenable to simple per-tensor PTQ with effectively no accuracy penalty (W8A8 within ~1% accuracy or ~0.1 perplexity of FP16 across BERT and ViT, Table 2). The downstream consequence for the research community is that architectural quantizability should be treated as a first-class pre-training objective, not an afterthought addressed through post-hoc quantization engineering.

The paper resolves a specific prior contradiction obliquely but clearly. Earlier work [31, 49] established that the outlier dimensions are "crucial for model predictions" and that "clipping them or setting to zero the corresponding parameters significantly degrades the model task performance." This finding created a tension: outliers are both necessary (for function) and harmful (for quantization). The paper's diagnostic framework dissolves this tension by showing that what is necessary is not the outliers themselves, but the functional behavior they enable — the ability of attention heads to perform a no-op on the residual for certain tokens. The outliers are a pathological side effect of the mechanism the model uses to achieve that behavior (softmax saturation → large FFN outputs). Clipped softmax and gated attention preserve the functional behavior (no-op capability) through clean mechanisms (hard zeros from clipping, or multiplicative gating) that do not produce outliers. The tension between "outliers are necessary" and "outliers are harmful" is thus resolved by separating the function (no-op on the residual) from the mechanism (extreme activation values).

Research directions that become more attractive after this work:

Verifier robustness and over-optimization research is not directly connected to this paper, but the diagnostic methodology — work backward from an observed pathology to the architectural limitation that forces it, then design a mechanism that grants the model a clean way to achieve the same goal — transfers cleanly. Researchers studying reward hacking in RLHF or degenerate outputs in long-context generation could apply the same template: identify what functional behavior the model is trying to express, then ask what architectural constraint forces it to use pathological means.
Architecture co-design with quantization becomes a more focused research agenda. Rather than treating quantization as a post-training compression step, this paper demonstrates that small architectural changes during pre-training can dramatically simplify the quantization problem. The γ = -α/T parameterization in clipped softmax — tying the hyperparameter to sequence length so that its semantic meaning (threshold relative to uniform-attention baseline) is invariant — exemplifies a design principle with broader applicability: architectural modifications for quantizability should be parameterized in terms of the natural scales of the tensors they operate on.
Attention mechanism redesign for purposes beyond quantization — the insight that the standard softmax forces unwanted interactions between "where to attend" and "whether to update" suggests that gated attention or similar decoupling mechanisms might benefit other objectives: interpretability (gates are directly inspectable as "did this head contribute"), pruning (heads with consistently closed gates can be removed), and training stability (removing the softmax saturation feedback loop might reduce sensitivity to learning rate and initialization).

Research directions that become less attractive:

Ever-finer quantization granularity as the primary research direction for transformer quantization — the paper demonstrates that architectural fixes can make simple per-tensor quantization work, reducing the pressure to develop per-channel, per-token, or group-wise quantization schemes that require specialized hardware support. While these techniques remain valuable for models that are already trained, the long-term research trajectory should shift toward making models inherently quantizable rather than making quantizers more tolerant of outliers.
Studying outliers as an isolated phenomenon without connecting them to functional behavior — the paper shows that outlier dimensions are not random but are causally linked to specific attention heads performing specific functions (no-op behavior). Research that treats outliers as a statistical nuisance to be regularized away without understanding what the model uses them for risks removing functionally important behavior, as Kovaleva et al. [31] demonstrated with the catastrophic effects of zeroing outlier dimensions. The paper's approach — diagnose the function first, then design an alternative mechanism — sets a higher bar for future work on neural network pathologies.

Follow-Up Research This Work Enables

Direct causal verification of the outlier formation hypothesis through gradient-based intervention. The paper's causal chain posits that the softmax gradient signal ∂yi/∂xj ≠ 0 drives unbounded growth of softmax inputs, which in turn requires large FFN outputs in the preceding layer to survive LayerNorm. This is plausible but never directly tested. A strong follow-up would instrument a vanilla BERT-base training run to measure, at regular intervals: (1) the gradient norm through the softmax to each hidden dimension of the FFN output in layer L, and (2) whether dimensions that will eventually become outliers (identifiable by their final-state infinity norm) show systematically larger gradient norms early in training, before the outliers become extreme. If the hypothesized mechanism is correct, outlier-precursor dimensions should show elevated gradient norms from the earliest training steps, and this elevation should be absent in clipped softmax and gated attention training runs. The experiment is tractable — BERT-base trains in ~93 hours on an A100 (Table 11), and gradient logging adds manageable overhead. A negative result (no early gradient difference between outlier and non-outlier dimensions) would indicate that the hypothesized mechanism is incomplete and that other factors (e.g., weight initialization interactions, LayerNorm gradient dynamics, or the FFN's own representational preferences) contribute independently to outlier formation.

Diagnose and resolve the clipped softmax failure on autoregressive decoders. The paper reports that clipped softmax fails on OPT-125m (Table 6: kurtosis increases from ~2,000 to ~19,000, W8A8 perplexity degrades from 21.18 to 37.20) but provides no investigation. The most pressing unanswered question is whether this failure is specific to OPT (its pre-LayerNorm placement, its weight initialization, its training recipe) or whether it generalizes to all autoregressive decoder architectures. A systematic follow-up would test clipped softmax pre-training on: (a) OPT-125m with post-LayerNorm placement (matching BERT's architecture more closely) to test whether LayerNorm position is the key factor; (b) a small GPT-2 variant (also autoregressive, but architecturally different from OPT) to test generalization across decoder families; (c) an encoder-decoder model (e.g., T5-small) to test whether the encoder or decoder side (or both) exhibits the failure. For each, the experiment should report not just aggregate perplexity and outlier metrics, but per-head attention pattern analysis and per-layer kurtosis, to localize where in the network clipped softmax diverges from vanilla behavior. The causal attention mask means that the effective sequence length T varies per position (position i attends to i+1 tokens), which the γ = -α/T parameterization does not account for — testing a position-dependent γ_i = -α/(i+1) variant would directly test whether the fixed γ interacts poorly with the variable-length attention window of causal decoders.

Combine clipped softmax and gated attention and evaluate whether they are redundant or complementary. The paper presents them as independent modifications but never tests both together (Section 8 of the hypothetical paper structure would note this as a natural next step). A simple experiment on BERT-base would train four models with identical compute budgets: vanilla, clipped softmax only, gated attention only, and clipped softmax + gated attention. The key measurements would be: (1) W8A8 perplexity — does the combination outperform either method alone, suggesting complementary mechanisms, or is performance identical to the better single method, suggesting redundancy? (2) Attention pattern analysis — with both modifications available, does the model use clipping to zero out certain tokens and gating to modulate the remaining output, or does it predominantly use one mechanism and leave the other effectively unused? (3) Gate saturation analysis — in the combined model, do gates learn to be more binary (close to 0 or 1) because the softmax clipping handles the fine-grained suppression, or do they remain intermediate-valued? This experiment would test the paper's implicit claim that both methods address the same root cause (inability to produce a no-op) through different mechanisms — if they are truly addressing the same limitation, they should be largely redundant; if they provide complementary capabilities, there is an additional mechanism at play that the paper's diagnostic framework does not fully capture.

Evaluate downstream task transfer of models trained with clipped softmax and gated attention. The paper evaluates only pre-training metrics (MLM perplexity, CLM perplexity, ImageNet top-1 accuracy). Prior work [31] showed that outlier dimensions are important for task predictions and that zeroing them degrades performance. Whether the architectural fixes — which change how the no-op behavior is implemented — affect fine-tuning transfer is open. A direct follow-up would take the BERT-base models from Table 2 (vanilla, clipped softmax, gated attention, all at comparable FP16 perplexity) and fine-tune them on the GLUE benchmark, reporting both FP16 and W8A8 performance on each task. The hypothesis to test: if the no-op behavior serves the same function during fine-tuning as during pre-training, the modified models should fine-tune comparably to vanilla. If the no-op mechanism interacts with task-specific attention patterns (e.g., delimiter tokens carry different information in NLI vs. paraphrase detection), the modified models might transfer differently. A negative result (e.g., clipped softmax showing degraded fine-tuning performance on certain GLUE tasks despite matched pre-training perplexity) would indicate that the clean no-op mechanism, while beneficial for pre-training and quantization, imposes a representational constraint that matters for some downstream behaviors. This would motivate task-specific variants or mixed architectures where some heads use standard softmax and others use the modified versions.

Scale γ = -α/T to long-sequence regimes (2048–8192 tokens) and determine whether α remains sequence-length-invariant. Figure 6 validates the parameterization for sequence lengths 32–256, but modern LLMs use much longer contexts. At T = 4096 with α = 2, the stretch factor is γ = -2/4096 ≈ -0.0005, producing a clipping threshold of α/T ≈ 0.0005 — substantially weaker than the γ = -0.03 (threshold ~0.029) that worked for BERT-base at T = 128. A systematic experiment would train a BERT or GPT variant at multiple sequence lengths spanning 128 to 4096 (or the maximum feasible given compute), using the γ = -α/T parameterization with α ∈ [2, 4], and measure whether: (a) max infinity norm remains suppressed at long sequences, or whether the weaker stretch allows outliers to re-emerge; (b) the optimal α shifts with sequence length (e.g., longer sequences might need larger α because the model has more tokens to distribute attention across and thus more opportunities for pathological saturation); (c) the FP16 perplexity penalty for larger α changes with sequence length. If the α ∈ [2, 4] recommendation fails at long sequences, a revised parameterization (e.g., γ = -α/log(T) or a learned per-head γ) would be needed, and the paper's claim of sequence-length invariance would require qualification. This is a practical necessity before the method can be recommended for long-context LLM pre-training.

Fine-tuning with gated attention followed by full W8A8 PTQ evaluation. The fine-tuning experiment in Appendix B.6 (Table 9) shows that adding gated attention to a pre-trained OPT-1.3B and fine-tuning for 4,000 steps reduces max inf norm from 79.3 to 50.9 (a ~1.6× reduction). But the paper does not report W8A8 perplexity for this fine-tuned model, so the question "can I retrofit gated attention onto an existing pre-trained model and enable simple W8A8 PTQ?" is unanswered. A targeted follow-up would: (a) fine-tune OPT-1.3B (or a more accessible model like OPT-350m) with gated attention for longer (e.g., 50,000 steps instead of 4,000) to see if the outlier reduction asymptotically approaches training-from-scratch levels; (b) report W8A8 perplexity after fine-tuning at multiple checkpoints to determine the training duration needed for quantization to become viable; (c) compare against the training-from-scratch results from Table 3 at matched total FLOPs to determine whether fine-tuning or from-scratch training is more compute-efficient for achieving a target W8A8 performance. If fine-tuning can recover most of the training-from-scratch benefit at a fraction of the cost, the practical applicability of gated attention broadens dramatically — it becomes a post-hoc enhancement rather than a pre-training commitment.

Practical Applications and Downstream Use Cases

Pre-training recipes for deployment-targeted models. For teams that control the pre-training pipeline and intend to deploy their models with INT8 inference, the paper provides a drop-in change: replace standard softmax with clipped softmax (γ = -2/T to -4/T) or add linear gated attention (with π_init = 0.5) to every attention layer, and train with the same hyperparameters as before. The quantized benefit is quantified: BERT-base W8A8 perplexity goes from completely broken (1294 ± 1046) to essentially matching FP16 (4.52 ± 0.01) with clipped softmax, or within 0.2 perplexity (4.65 ± 0.04) with gated attention (Table 2). ViT-S/16 W8A8 accuracy improves from 69.24% to 79.77% (clipped softmax) or 79.82% (gated attention), narrowing the gap to FP32 from ~11.5 percentage points to ~1.2 points. For OPT-125m, gated attention takes W8A8 from 21.18 to 16.02 perplexity (vs. 15.55 FP16). The training overhead is minimal — clipped softmax adds essentially no cost (BERT: 93.6 vs. 92.8 hours on A100, a <1% increase; Appendix D, Table 11), and gated attention adds 3–8% depending on the model. This makes the methods immediately applicable for anyone running BERT-scale or ViT-scale pre-training, and a low-risk experiment for larger-scale training runs (with the caveats about scale and downstream evaluation noted above).

Edge deployment of vision transformers without per-channel quantization hardware. The paper's ViT results (Table 7) show that adding a LayerNorm after patch embeddings and using clipped softmax or gated attention reduces max infinity norm from 359 to ~74–80, enabling W8A8 accuracy within 1.2 percentage points of FP32 using simple per-tensor static PTQ. This matters for edge deployment because per-tensor quantization maps directly to standard integer matrix multiplication hardware available on mobile and embedded accelerators (DSPs, NPUs), whereas per-channel quantization often requires custom kernels or mixed-precision support that may not be available or efficient on those platforms. The paper's methods allow a ViT-S/16 to be deployed at INT8 on commodity edge hardware without accuracy loss from 80.98% to 79.77–79.82% (Table 7, with extra LN), which is a concrete deployment recipe for vision applications (image classification, object detection backbones) where floating-point inference is too slow or energy-intensive. The combination of extra LayerNorm + clipped softmax/gated attention is particularly actionable because the extra LayerNorm alone already provides most of the benefit (W8A8 79.62% vs. 79.77–79.82%), and clipped softmax provides a small additional improvement at nearly zero cost.

Scaling up fine-tuning data generation with quantized models. For self-improvement or distillation pipelines where a model is used to generate training data for another model, inference cost dominates because the generator model is run on large corpora. If the generator model can be quantized to INT8 without accuracy loss, the inference cost drops substantially (INT8 operations are more energy-efficient and can use higher-throughput hardware). The paper demonstrates that BERT models trained with clipped softmax or gated attention achieve W8A8 perplexity within 0.1–0.2 of FP16, meaning the quality of generated training data (e.g., masked token predictions for distillation) would be nearly identical to the floating-point generator while running at INT8 throughput. While the paper evaluates only perplexity and not generation quality metrics (e.g., top-k accuracy of masked predictions), the close tracking of W8A8 and FP16 perplexity suggests generation quality is likely preserved. For teams running large-scale BERT-based data generation, the training cost of switching to clipped softmax pre-training (effectively zero additional cost, per Table 11) is recouped by the inference efficiency gain from reliable INT8 deployment.

Fine-tuning pre-trained decoders with gated attention for partial outlier reduction when full re-training is infeasible. The fine-tuning result in Table 9, while not achieving training-from-scratch levels of outlier suppression, suggests a pragmatic pathway: for an existing pre-trained OPT-1.3B, fine-tuning with gated attention for 4,000 steps reduces max infinity norm from 79.3 to 50.9 and average kurtosis from 2,086 to 665. While the paper does not report W8A8 perplexity for this fine-tuned model, the outlier reduction is meaningful, and extending the fine-tuning duration or combining with activation regularization (as the authors did) may yield sufficient quantizability for deployments where per-channel quantization hardware is unavailable but some accuracy degradation is tolerable. This use case is speculative given the missing W8A8 results, but the fine-tuning experiment demonstrates that gated attention can be retrofitted at modest cost (~4,000 steps on 1.3B parameters is a small fraction of full pre-training), making it a low-risk intervention for practitioners who need improved quantizability (if not perfect) from an existing checkpoint and cannot afford to re-train from scratch.

When to Prefer This Method

The paper does not articulate an explicit "prefer X over Y given conditions A, B, C" decision framework comparing clipped softmax or gated attention against named alternatives like SmoothQuant, GPT3.int8(), or Outlier Suppression. The paper positions its methods as a pre-training protocol change rather than as alternatives to post-training quantization techniques, so a head-to-head tradeoff matrix is not part of the paper's own framing. The paper's contribution is establishing that the outlier problem can be prevented architecturally; whether that prevention is preferable to post-hoc treatment in a given deployment scenario depends on factors (availability of pre-trained checkpoints, training budget, target hardware, tolerance for specialized quantization kernels) that the paper does not experimentally explore or provide explicit guidance for. A forced decision rule would impose structure the paper does not provide.

The one clear conditional the paper does provide is architecturally specific: for autoregressive decoder models, prefer gated attention over clipped softmax, based on the unexplained failure of clipped softmax on OPT-125m (Table 6) while gated attention works well (Table 6: max inf norm 8.7, W8A8 16.02). This recommendation is tentative, based on a single failure case without diagnostic follow-up, and the paper acknowledges it does not understand the cause.