Measuring Maximum Activations in Open Large Language Models

🎯 Pitch

Global maximum activations in modern open LLMs span nearly four orders of magnitude at comparable sizes—Gemma3-27B-it peaks at ~7×10⁵ while Qwen3.5 and MoE models linger in the 10²–10³ range—breaking any simple monotonic scaling with parameter count. MoE checkpoints are 14–23× quieter than matched-scale dense models, and the residual stream carries the global maximum in 22 of 24 models, upending the assumption that larger models intrinsically need wider activation ranges. These measured maxima directly predict INT-8 signal-to-noise degradation, making them a deployment-critical property that should be reported alongside every weight release.

1. Executive Summary

This paper measures the maximum activation magnitude (M = max |a|) — the largest absolute activation observed across layers and key components — across 27 checkpoints from 8 modern open LLM families (Qwen2.5/2.5-VL/3/3.5, Gemma2/3, Ling, GPT-OSS) under a unified 5,000-sample multi-domain pipeline. The study treats M as a continuous deployment-relevant model property rather than a binary outlier flag, finding that global maxima span nearly four orders of magnitude at comparable parameter counts — from Qwen3.5 and MoE checkpoints in the 10²–10³ range to Gemma3-27B-it reaching ~7×10⁵ — while MoE checkpoints exhibit 14.0–23.4× lower peaks than matched-scale dense counterparts and the residual stream carries the global maximum in 22/24 checkpoints (with GPT-OSS-20B peaking at the MLP output as the sole exception). A lightweight INT-8 sanity check shows that higher M co-varies with lower signal-to-quantization-noise ratio through activation-scale selection, establishing that maximum activation magnitude is a family-, architecture-, and training-stage-dependent model property — not a simple byproduct of parameter count — that breaks monotonic scaling predictions across families and generations.

2. Context and Motivation

The Core Problem: We Don't Systematically Know How Large Activations Can Get in Modern Open LLMs

The fundamental question this paper tackles is deceptively simple: when you run a modern open-source LLM on real text, what is the largest activation value that can emerge, and what controls its magnitude? This matters because the dynamic range of activations — the gap between typical values and extreme outliers — is a first-order constraint for nearly every downstream deployment decision. The activation dynamic range determines what numerical precision is required for stable inference, how aggressively you can quantize activations to lower bit widths for efficiency, and what scale factors must be chosen to avoid clipping or overflow. The paper frames this as a deployment-oriented question: not "do massive activations exist?" but "how large can they get, where do they appear, and how does this vary across the rapidly diversifying landscape of open models?"

This gap is significant for several practical reasons the authors highlight in Section 1:

Low-bit quantization is bottlenecked by activation range. In per-tensor quantization schemes, a single scale factor is typically chosen to cover the largest observed activation magnitude. If even a tiny fraction of activations are orders of magnitude larger than the rest, that scale factor wastes the vast majority of quantization levels on values that almost never occur, amplifying reconstruction error for the ordinary activations that carry the bulk of useful signal. The problem is succinctly stated: "a small number of extremely large activations can therefore dominate the scale, waste most quantization levels on rarely used values, and amplify reconstruction error" (Section 1, paragraph 2).
Model releases are accelerating without corresponding measurement. The post-2024 explosion of open-weight LLMs has diversified the design space along axes that plausibly affect activation behavior: normalization stacks, gated MLP variants (SwiGLU), mixture-of-experts routing, vision-language modality adaptation, intermediate training checkpoints, and instruction tuning. Yet no prior study provides activation magnitude measurements across this expanded landscape.
Deployment decisions are currently made without model-card statistics for activation range. Practitioners choosing between model families, deciding on quantization bit widths, or setting inference-system numerical tolerances have no principled prior for what activation magnitudes to expect. They rely on rules of thumb inherited from earlier LLaMA-style models that may not generalize.

Conflicting Perspectives: Massive Activations as Both Functional Features and Deployment Obstacles

The paper is motivated by an intellectual tension in the existing literature. Prior work has studied large activations from two largely disconnected angles:

The interpretability/mechanistic line treats massive activations as functional model components. Dettmers et al. (2022) first showed that OPT and BLOOM models develop sparse, high-magnitude "emergent features" whose preservation is necessary for lossless INT8 inference. Sun et al. (2024) sharpened this into the definition of massive activations: token-feature coordinates that are simultaneously large (|x_i| > 100) and locally sparse (at least 1000× the per-token median absolute value). Subsequent mechanistic work explained why these values emerge — Bondarenko et al. (2023) attributed them to attention heads needing a "no-op" route through the residual stream via attention sinks on beginning-of-sequence tokens, and more recent studies (Gu et al., 2025; Sun et al., 2026) have traced when these sinks emerge during pretraining and how residual-stream spikes decouple from local attention sinks. These studies treat massive activations as something to be understood — functional consequences of softmax constraints and residual-stream dynamics.

The quantization/mitigation line treats the same activations as something to be removed. SmoothQuant (Xiao et al., 2023) migrates activation difficulty into weights through algebraically equivalent rescaling; AWQ (Lin et al., 2024) and GPTQ (Frantar et al., 2023) protect salient parameters from quantization error; rotation methods like QuaRot (Ashkboos et al., 2024) and SpinQuant (Liu et al., 2024) use orthogonal transforms to distribute outlier mass across dimensions; FlatQuant (Sun et al., 2025) learns affine flattening transforms; and PrefixQuant (Chen et al., 2024) and KIVI (Liu et al., 2024) target KV-cache-specific outliers. All of these methods are designed to transform away the activation upper bound rather than to measure how it varies across models.

The critical gap the paper identifies is that neither line provides a unified measurement of the activation upper bound itself across modern model families. The interpretability work concentrates on a handful of pre-2024 LLaMA-, OPT-, or BLOOM-style checkpoints with binary existence tests. The quantization work assumes the upper bound is a problem to be solved but calibrates its mitigations on a narrow set of models, typically LLaMA derivatives. As the authors put it: "It remains unclear whether either discovery still holds for the recent wave of post-LLaMA open releases" (Section 1, paragraph 5).

Where Existing Approaches Fall Short

The paper identifies specific limitations in prior work along several axes:

Narrow model coverage: the LLaMA-derivative monoculture. Prior massive-activation studies (Dettmers et al., 2022; Sun et al., 2024; Bondarenko et al., 2023) analyzed a restricted set of model families — primarily OPT, BLOOM, and LLaMA variants. These models share architectural commonalities (pre-norm transformer stacks, standard dense MLP blocks, text-only training) that do not represent the diversity of post-2024 open releases. Models like Qwen2.5/3/3.5 diverge along multiple axes simultaneously: normalization stack design, SwiGLU gating, MoE routing with sparse expert activation, multimodal adaptation for vision-language tasks, and publicly released intermediate training checkpoints that expose the dynamics of ongoing pretraining. Without measurements across this expanded space, practitioners cannot know whether prior findings about activation magnitudes generalize or whether new families exhibit qualitatively different behavior.

Binary classification rather than continuous measurement. The dominant framework from Sun et al. (2024) defines massive activations through a binary criterion: does a model contain any token-feature coordinate exceeding both the absolute (>100) and relative (>1000× median) thresholds? This treats activation magnitude as a categorical property — a model either "has" massive activations or it doesn't. The paper argues this framing is insufficient for deployment decisions. The authors show (Section 3, Table 1, and Figure 2) that the binary criterion and continuous magnitude can disagree: some checkpoints that fail the binary test (e.g., Qwen2.5-1.5B with peak 7,968 but local ratio only ~574) still exhibit large absolute activations that pose quantization risks, while others that pass may have lower absolute peaks than those that fail. The deployment-relevant question is not "are there massive activations?" but "how large is the largest activation, and where does it live?"

No decomposition of what controls peak magnitude. Prior work established that massive activations exist and offered mechanistic accounts for why they form (attention sinks, residual-stream dynamics), but it did not systematically decompose which model properties predict the magnitude. The paper identifies this as a practical gap: "model family, architecture, model generation, and training stage" may all affect peak magnitude, but their relative importance and interaction are unknown (Section 1, paragraph 3). Without this decomposition, practitioners cannot anticipate whether switching from a dense to MoE architecture, from base to instruction-tuned, or from one generation to the next within a family will increase or decrease activation range — each decision is made blindly.

Mismatch between measurement and deployment context. The interpretability literature's canonical experiments (e.g., ablating massive-activation coordinates and measuring perplexity degradation) establish functional importance but do not map to deployment-relevant metrics like quantization reconstruction error. The quantization literature's canonical experiments (e.g., end-to-end perplexity or task accuracy after applying a specific mitigation) provide deployment metrics but are specific to particular mitigation recipes and model versions. Neither provides a model-card statistic — a simple, standardized measurement of maximum activation magnitude that can be reported alongside any open-weight release and used by downstream practitioners to make quantization decisions before applying any specific mitigation. This paper positions M = max |a| as exactly that statistic.

How This Paper Positions Itself

The paper explicitly positions itself as complementary to both the interpretability and quantization lines, not as a replacement or competitor. The key positioning statements appear in Section 5 of the Introduction and are elaborated in Section E (Related Work):

"Our work is therefore complementary to both the interpretability and quantization literatures. Rather than asking only whether a model contains a massive activation under a fixed local criterion, we measure the continuous deployment-relevant statistic M = max |a| under a unified protocol, compare it across recent open families and matched architectural or training contrasts, and connect it to activation-quantization reconstruction error."

This complementarity is operationalized through several design choices:

Measurement-first, not mechanism-first. The paper deliberately stops short of causal claims about why families differ: "We deliberately stop short of causal claims about why families differ; explaining the mechanism behind these differences—and intervening on it—is left as future work" (Section F). The contribution is the measurement infrastructure and the empirical survey itself — what the authors call "the first unified-protocol measurement of the global maximum M = max |a| across 27 post-LLaMA open checkpoints from 8 families" (Section 1, paragraph 5). This is a descriptive contribution that provides inputs to both mechanistic and mitigation research: mechanists can use the family/architecture/training-stage contrasts to design causal experiments, and quantization researchers can use the per-model M values to calibrate their mitigation hyperparameters.

Unified protocol as the key methodological contribution. A central claim is that prior studies are not comparable because they use different corpora, different tokenization, different component hooks, and different aggregation procedures. The paper's pipeline (Section 2, Figure 1) standardizes all of these: the same 5,000-sample multi-domain text corpus (with domain-balanced sampling across math, code, web text, knowledge, Chinese, low-resource languages, and mixed English), re-tokenized with each model family's own tokenizer to avoid tokenizer-mismatch artifacts; identical PyTorch forward hooks on six component classes (embeddings, hidden states, attention outputs, MLP/MoE outputs, SwiGLU gate pre-activations, final LayerNorm); and consistent reporting of global maxima across all hooked components and layers. This protocol makes cross-family comparisons meaningful in a way that aggregating results from disparate prior studies cannot be.

Matched-design comparisons rather than population-level regressions. Rather than fitting a regression of M against parameter count across all 24 checkpoints (which the authors show would be meaningless given the cross-family variation spanning orders of magnitude), the paper isolates specific contrasts: same-family different sizes (Section 5.1, Figure 5), same-scale MoE vs. dense (Appendix C.1, Figure 8), same-family vision-language vs. text-only (Appendix C.2, Figure 9), same-backbone Base vs. Instruct (Appendix C.3, Figure 10), and same-family different training stages (Appendix C.4, Figure 11). This matched-design approach — holding as many factors constant as possible while varying one — provides the first observational decomposition of how scale, architecture, modality, fine-tuning, and training progress each associate with activation peaks. The authors are careful to note these are observational contrasts, not causal interventions (Section D.3), but they nonetheless provide actionable prior information for practitioners.

Deployment validation through INT-8 sanity check. The paper does not claim to provide a complete quantization evaluation. Instead, Section D.2 and Figure 12 show a lightweight probe: for eight representative checkpoints spanning the observed M range, apply two standard per-tensor INT-8 quantization strategies (max-abs scaling and 99.9% clipping) at the peak hidden layer and measure signal-to-quantization-noise ratio (SQNR). The result — that higher M correlates with lower SQNR, with Qwen3.5-0.8B (M ≈ 122) achieving 29.1 dB while Gemma3-4B-it (M ≈ 246K) drops to ~10–14 dB — establishes the deployment relevance of the measurement without overclaiming a general scaling law. The authors explicitly frame this as a "sanity check" rather than a "calibrated dose–response curve" (Section D.2), acknowledging that richer distributional and task-level evaluation would be needed for full quantization assessment.

A call for model-card reporting. The paper's ultimate positioning is prescriptive: "We therefore recommend that M and its layerwise carrier be reported alongside any open-weight release as part of a deployment-oriented model card extension" (Section F). This reframes maximum activation magnitude from a research artifact (something that mechanistic interpretability studies probe) or a mitigation target (something that quantization methods eliminate) into a first-class model property that should be measured, reported, and used for deployment planning — analogous to how parameter count, context length, and benchmark scores are already standard model-card entries.

3. Technical Approach

3.1 Reader Orientation

This paper is an empirical measurement study — not a method proposal, not a causal analysis, but a systematic survey of a single deployment-critical quantity across a broad set of modern open-weight LLMs. The "system" being built is a standardized measurement pipeline that takes any HuggingFace-compatible LLM checkpoint as input and produces a single number — the global maximum activation magnitude $M = \max|a|$ — along with layerwise and component-level breakdowns of where that maximum occurs. The core idea is that this number is a continuous, architecture-and-training-dependent model property that should be measured and reported alongside every open-weight release, replacing the binary "does this model have massive activations?" question from prior work with the deployment-relevant "how large can its activations get, and where?"

The problem it solves is the absence of comparable activation-range data across modern model families. Before this paper, a practitioner choosing between Qwen3.5-27B and Gemma3-27B-it for a low-bit deployment had no way to know that the latter's activation peaks are roughly 4,000× larger. The solution is a unified measurement protocol — fixed corpus, family-specific tokenization, identical hooks, consistent aggregation — applied identically to 27 checkpoints from 8 families, producing directly comparable activation statistics that reveal cross-family variation spanning nearly four orders of magnitude.

3.2 Big-Picture Architecture (Diagram in Words)

The pipeline has three major stages, each with distinct components:

Stage 1 — Data Preparation: A fixed 5,000-sample multi-domain text corpus is constructed once, then re-tokenized separately for each model family using that family's own tokenizer. This produces a set of token sequences where the semantic content is identical across families but the tokenization is family-aligned.

Stage 2 — Activation Measurement: Each checkpoint is loaded with frozen weights. During forward inference on the tokenized corpus, PyTorch forward hooks intercept six classes of activation tensors (embedding outputs, post-residual hidden states, attention outputs, MLP/MoE outputs, SwiGLU gate pre-activations, and final LayerNorm outputs) and stream layerwise statistics into GPU-side accumulators without per-sample synchronization.

Stage 3 — Analysis and Reporting: Per-model JSON files containing recorded statistics (mean, std, RMS, mean absolute value, max/min, and streaming quantile estimates of absolute values) are post-processed to extract the global maximum $M = \max|a|$ across all hooked components and all layers. All figures, tables, and matched-design comparisons are generated from these saved statistics.

Information flows sequentially through these three stages: Stage 1 produces token sequences → Stage 2 consumes them and produces per-model statistics files → Stage 3 consumes those files and produces the paper's analyses. The entire pipeline is offline and deterministic: no parameters are modified, no training occurs, and the same corpus is used for every checkpoint.

3.3 Roadmap for the Deep Dive

First, the formal definition of the primary metric $M$ and the full set of recorded statistics — since all downstream analyses derive from these quantities, and understanding what is (and isn't) measured determines how to interpret every finding.
Second, the corpus construction procedure in detail — since activation maxima are extreme statistics that could be dominated by a single anomalous input, the corpus design (domain balancing, length distribution, re-tokenization strategy) is the foundation for claiming that measured peaks are representative rather than accidental.
Third, the model selection criteria and the full 27-checkpoint suite — since the paper's central contribution is the cross-family survey, understanding which checkpoints were chosen and why (the matched-design logic: same-family scaling, same-scale architectural contrasts, training-stage evolution, Base-vs-Instruct pairs) is essential for evaluating the scope and limits of the findings.
Fourth, the hook-based instrumentation and the streaming accumulation mechanism — since the paper measures activations at six distinct component classes, and where the global maximum lives (residual stream vs. MLP output vs. final norm) is itself one of the key findings.
Fifth, the peak stability validation procedure — since global maxima are by definition sensitive to outliers, the paper must establish that its reported peaks are stable under subsampling before any downstream analysis is credible.
Sixth, the relationship between the continuous metric $M$ and the binary massive-activation criterion from prior work — since the paper explicitly positions itself as reframing the question from "does this model have massive activations?" to "how large can activations get?", we need to understand exactly how the two frameworks connect and where they disagree.

3.4 Detailed, Sentence-Based Technical Breakdown

This is an observational measurement study whose core idea is that the maximum activation magnitude of an open LLM is a continuous, releasable model property that is not predicted by parameter count alone and should be reported alongside model weights as a deployment-oriented model-card statistic. The technical approach is entirely a measurement pipeline — there is no training, no optimization, and no causal intervention.

Primary Metric and Recorded Statistics

The paper's central quantity is $M = \max|a|$ , defined as "the largest absolute activation observed across layers and key components" (Section 1, paragraph 2). This is a single scalar per checkpoint: the global maximum of the absolute value taken over every activation tensor collected by the hooks, across all layers, all token positions, all feature dimensions, and all six component classes. It is what the paper calls "the value plotted in every bar chart and used in every matched-pair ratio" (Section 3.1).

The paper records a richer set of statistics beyond just the maximum, stored as per-model JSON files. For each captured component tensor (at each layer, for each component class), the pipeline records:

Mean ( $\mu$ ): the arithmetic mean of the tensor values
Standard deviation ( $\sigma$ ): the square root of the variance
RMS ( $\sqrt{\mathbb{E}[x^2]}$ ): the root mean square, which captures magnitude including both the mean and spread
Mean absolute value ( $\mathbb{E}[|x|]$ ): the average of absolute values, less sensitive to extreme outliers than RMS
Maximum value: the single largest positive value in the tensor
Minimum value: the single most negative value in the tensor (for activations that can be negative, such as pre-activation values before non-linearities)
Streaming estimates of absolute-value quantiles: log-histogram-based quantile estimation that computes quantiles (e.g., median, 90th, 99th, 99.9th percentile) of the absolute activation values without storing the full distribution

What these statistics enable. The $\max$ gives $M$ for each component class at each layer, which can be aggregated to the global $M$ or plotted as layerwise trajectories (as in Figure 4). The quantiles enable the local-sparsity diagnostic from Sun et al. (2024): for a given token's activation vector $\mathbf{x} \in \mathbb{R}^d$ , a coordinate $x_i$ is a "massive activation" if $|x_i| > 100$ and $|x_i| / \text{median}_{j=1}^d |x_j| > 1000$ . The median required for this test comes from the quantile estimates. The mean and RMS provide context for interpreting how extreme the maximum is relative to typical activation magnitudes.

Why record all of these rather than just $M$ ? The paper's analyses require different statistics at different points. The global $M$ alone tells you how large activations can get but not where (which component class, which layer depth, whether in a single isolated spike or across a broad plateau). The layerwise maxima (shown in Figure 3 as a heatmap and in Figure 4 as trajectories) require per-layer $\max$ values. The local-sparsity diagnostic (Figure 2, Table 1) requires per-token medians from the quantile estimates. The INT-8 sanity check (Figure 12) uses the per-layer $\max$ to set scale factors and needs RMS-like statistics for SQNR computation. Recording the full set of statistics once and storing them as JSON files decouples measurement from analysis: "All figures and tables are generated from the resulting per-model statistics" (Section 2, caption of Figure 1).

Relationship between M and the Top-k values in Table 1. A critical distinction: the "Top-1" values reported in Table 1 come from "a single representative layer chosen for the local-sparsity diagnostic and may therefore differ from M when a different layer carries the global peak" (Section 3.1). This is because the Sun criterion is evaluated per-layer using the token with the highest local ratio, while $M$ is the single largest absolute value anywhere in the model. For example, Qwen2.5-32B shows Top-1 = 22,144 in Table 1 (at the representative passing-criterion layer) but the paper notes its global $M$ is 30,848 (Appendix C.3). This gap arises because the layer selected for Table 1 satisfies the Sun binary criterion (it has a token with both $|x_i| > 100$ and ratio > 1000), but another layer that doesn't satisfy the criterion (perhaps because its median is higher, reducing the ratio) may have a larger absolute peak. The paper is explicit: "the Top-1 entry of 22,144 that appears for Qwen2.5-32B in Table 1 is taken at the representative passing-criterion layer, and is by construction ≤ M" (Appendix C.3).

Corpus Construction

The evaluation corpus is designed to answer a specific threat: that maximum activation statistics are dominated by a single domain, text type, or sequence length, making the reported peaks accidental artifacts of corpus composition rather than stable model properties. The paper addresses this through explicit domain balancing, length stratification, and sample-size validation.

Target composition. The corpus contains exactly 5,000 samples drawn from RedPajama (Shen et al., 2024) sources, partitioned into seven domain buckets with explicit target counts:

850 mathematical or scientific samples
850 code samples
850 English web samples
850 knowledge-oriented samples (encyclopedic, book, or Q&A text)
400 Chinese samples
300 samples in "other low-resource languages" (languages other than English and Chinese)
900 additional English or mixed web samples

The paper states the rationale: "This design reduces the risk that maximum-activation statistics are dominated by a single domain and ensures that the corpus covers formal text, natural web text, knowledge-intensive text, code, and multilingual content" (Section 2.1, paragraph 1). The domain balances are chosen so that no single content type overwhelms the corpus — math, science, code, web text, and knowledge each get roughly equal representation at ~17% of the corpus, with additional coverage for Chinese (~8%) and low-resource languages (~6%).

Why this partitioning strategy rather than uniform or proportional sampling? If the corpus were uniformly sampled from RedPajama without domain control, it would be dominated by English web text (which constitutes the vast majority of web-crawled corpora), and maxima observed on that corpus might not reflect what happens on code or mathematical text. If the corpus were purely code or purely mathematical, it would over-represent formal, structured text at the expense of the natural language that constitutes most real-world usage. The balanced design provides a middle ground: it covers diverse content types in roughly equal proportion so that the observed maximum is not an artifact of any single domain's statistical properties, while still being large enough (5,000 samples) to provide statistical stability.

Sequence-length stratification. Samples are randomly truncated to five discrete length buckets with target proportions:

256 tokens: 1%
512 tokens: 1%
1024 tokens: 2%
2048 tokens: 3%
4096 tokens: 93%

This produces "an average length of approximately 3899 tokens, corresponding to roughly 19.5M tokens in total" (Section 2.1, paragraph 2). The heavy skew toward 4096-token sequences is deliberate: the corpus is "dominated by long-context inputs while retaining a small number of short and medium-length sequences" (Section 2.1, paragraph 2). This reflects the deployment scenario where most inference involves longer contexts, and maximum activations may depend on sequence length (longer contexts mean more token positions over which to take a maximum, and also more opportunity for attention sinks to accumulate mass on beginning-of-sequence tokens over many layers). The small proportion of shorter sequences provides a check on whether maxima are length-dependent.

Why 4096 tokens as the maximum length? The paper does not explicitly justify this choice, but several considerations apply. First, 4096 is a common context length for inference benchmarks and covers the majority of typical usage scenarios. Second, many of the measured checkpoints support much longer contexts (32K, 128K, or more), but measuring at those lengths would dramatically increase computation (a single 128K-token sequence generates 32× more activations than a 4K-token sequence) while providing diminishing returns if activation maxima saturate at moderate lengths. The paper acknowledges this as a limitation in Section D.3: "All measurements use sequences up to 4,096 tokens, so we cannot speak to whether maxima grow, saturate, or shift carriers under 32k–128k contexts."

Family-specific re-tokenization. This is a critical design choice for cross-family comparability. The text corpus is held fixed (the same 5,000 strings), but tokenization is performed separately for each model family using that family's own tokenizer. The paper explains: "models receive semantically identical text but token sequences aligned with their own tokenizer, reducing activation-statistics bias caused by tokenizer incompatibility" (Section 2.1, paragraph 3).

Why does tokenizer alignment matter? Different model families use different tokenizers with different vocabularies, different tokenization algorithms (BPE vs. SentencePiece vs. custom), and different typical token lengths. If you tokenized the corpus once with a single tokenizer and fed those token IDs to all models, you would be feeding semantically different text to different models — the token sequence [101, 2087, 374, ...] might represent "The cat sat" under one tokenizer but garbage under another. Conversely, if you re-tokenized the same text strings with each model's tokenizer, you preserve semantic equivalence while accepting that the token sequence lengths and compositions will differ. This means that models with more aggressive tokenization (fewer tokens for the same text) will process fewer total forward passes, but each forward pass processes the same semantic content. The paper argues this is the right trade-off for measuring model-level properties: you want the model evaluated on the text it was trained to process, not on text distorted by an alien tokenizer.

Limitations of the corpus design. The paper acknowledges in Section D.3 that the corpus is "restricted to English, Chinese, and code domains" and that "activations on long-tail languages, mathematical reasoning chains, or tool-use traces may differ." The 400 Chinese and 300 low-resource-language samples provide coverage but not saturation for multilingual activation behavior. Similarly, the corpus is designed for measurement (covering diverse content types) but is not necessarily representative of any specific deployment distribution — a practitioner deploying a model primarily on code would want code-specific activation measurements.

Model Suite and Selection Criteria

The paper selects 24 checkpoints for the main analysis plus 3 additional Qwen2.5-Instruct checkpoints for the Base-vs-Instruct comparison, for a total of 27 checkpoints from 8 model families. The selection follows three explicit principles (Section 2.2):

Principle 1: Coverage of "recent mainstream open LLM families rather than restricting the study to earlier LLaMA-style models." The families are:

Qwen2.5 (1.5B, 7B, 32B): Dense text-only models serving as a scaling baseline. Qwen2.5 represents a post-LLaMA architecture with its own normalization stack, SwiGLU MLPs, and training recipe (Qwen Team, 2024).
Qwen2.5-VL (3B, 7B, 32B): Vision-language variants of Qwen2.5, trained with multimodal adaptation on top of the text backbone. These enable the vision-language vs. text-only matched comparison.
Qwen3 (1.7B, 8B, 30B-A3B, 32B): The next generation of Qwen (Qwen Team, 2025), including both dense (1.7B, 8B, 32B) and MoE (30B-A3B, where the notation means 30B total parameters with 3B active per token) variants. This family provides the dense/MoE matched comparison (30B-A3B vs. 32B) and the generational comparison (Qwen2.5 → Qwen3).
Qwen3.5 (0.8B, 9B, 27B, 35B-A3B): A further generation that the paper identifies as a "low-outlier counterexample family" (Section 5.3 and Table 2). These checkpoints systematically suppress activation magnitudes compared to Qwen2.5 and Qwen3, providing the clearest evidence that family-level design choices can override parameter-scale effects.
Gemma2 (2B, 9B, 27B): Google's Gemma 2 family (Gemma Team, 2024), included as a non-monotonic scaling case — the 9B checkpoint peaks below the 2B checkpoint, a pattern not seen in other families.
Gemma3 (4B-it, 27B-it): Google's Gemma 3 family (Gemma Team, 2025). These are publicly released as instruction-tuned checkpoints, so the Gemma2/Gemma3 comparison is "not interpreted as a strict base-to-base ablation" (Section 2.2 and Table 2). Gemma3 exhibits the highest peaks in the study (696,320 for the 27B-it checkpoint).
Ling (5T, 10T, 15T, 20T): The Ling-mini series (Ling Team, 2025), released at intermediate training checkpoints after 5, 10, 15, and 20 trillion tokens of training. This enables the training-stage evolution comparison with fixed family and approximately fixed architecture.
GPT-OSS (20B): OpenAI's open-source release (OpenAI, 2025), included as a "component-level special case" — it is the only checkpoint where the global maximum comes from the MLP output rather than the hidden state.

Principle 2: Multiple parameter scales and architectural forms to "separate the effects of scale, family, and architecture." The checkpoints span from 0.8B to 35B parameters (with GPT-OSS at 20B), covering the range where most open-weight deployment happens. Within each family, multiple scales enable the within-family scaling analysis (Section 5.1, Figure 5). Across families, same-scale checkpoints (e.g., ~7–9B: Qwen2.5-7B, Qwen2.5-VL-7B, Qwen3-8B, Qwen3.5-9B, Gemma2-9B; ~27–32B: Qwen2.5-32B, Qwen2.5-VL-32B, Qwen3-32B, Qwen3.5-27B, Qwen3.5-35B-A3B, Gemma2-27B, Gemma3-27B-it) enable the cross-family comparison that reveals the ~4-order-of-magnitude variation at similar parameter counts.

Principle 3: Inclusion of special variants — MoE models, vision-language models, intermediate training checkpoints, and instruction-tuned models — to "examine whether maximum activations change with routing, modality adaptation, training progress, or supervised fine-tuning." Each variant provides a matched comparison where one factor changes while others are held roughly constant:

MoE vs. dense: Qwen3-30B-A3B vs. Qwen3-32B (same family, similar total parameters, different architecture); Qwen3.5-35B-A3B vs. Qwen3.5-27B (same family, similar total, different architecture).
Vision-language vs. text-only: Qwen2.5-VL-7B vs. Qwen2.5-7B and Qwen2.5-VL-32B vs. Qwen2.5-32B (same family, same text backbone, different modality adaptation).
Base vs. Instruct: Qwen2.5-{1.5B, 7B, 32B} vs. Qwen2.5-{1.5B, 7B, 32B}-Instruct (same backbone, same architecture, different training stage).
Training stage: Ling-mini-{5T, 10T, 15T, 20T} (same family, same architecture, different training duration).

What is deliberately excluded. The paper does not include models earlier than the post-LLaMA wave (e.g., LLaMA-1, LLaMA-2, OPT, BLOOM) because those have already been studied by prior work and the paper's contribution is specifically to extend measurement to newer releases. It also does not include closed-source models (GPT-4, Claude, Gemini) because the hook-based measurement protocol requires weight access. And it does not include models much larger than ~35B parameters because the computational cost of measuring activation statistics across 5,000 samples with full forward passes scales with model size, and because the focus is on "deployment-sized" models where low-bit quantization is most practically relevant.

Important caveat about the Gemma3 checkpoints. The paper explicitly notes that "Except for the publicly released Gemma3 checkpoints, which are instruction-tuned models, we treat the main-analysis checkpoints as base or intermediate-training checkpoints" (Section 2.2). This means the Gemma3 peak magnitudes are affected by instruction tuning, and the high values observed (246K for 4B-it, 696K for 27B-it) cannot be cleanly attributed to architecture alone — they reflect both the Gemma3 architecture and the effects of supervised fine-tuning. The paper is transparent about this but it limits the cleanliness of the Gemma2→Gemma3 generational comparison.

Hook-Based Instrumentation and Streaming Accumulation

The measurement mechanism is built on PyTorch's forward hook system. During forward inference, each checkpoint is loaded with its full weights in evaluation mode (no dropout, no parameter updates). Six classes of hooks intercept activation tensors as they are computed:

Embedding outputs: The tensor produced by the token embedding lookup table, before any transformer layer processing. This is the first activation any token generates, representing the sum of token embedding and (for models that use them) positional encoding. Hooking here captures whether the embedding layer itself produces large values, which would be a pre-transformer source of extreme activations.

Layerwise hidden states after residual updates: For each transformer layer $\ell$ , the hidden state tensor after the residual connection has added the attention output and the MLP/MoE output to the previous layer's hidden state. Mathematically, if $h_{\ell-1}$ is the previous hidden state, $a_\ell$ is the attention output at layer $\ell$ , and $m_\ell$ is the MLP/MoE output at layer $\ell$ , then the hooked tensor is $h_\ell = h_{\ell-1} + a_\ell + m_\ell$ . This is the dominant carrier of extreme activations — the paper finds that "22 global maxima occur in layerwise hidden states" (Section 4.3). The residual stream accumulates contributions from every layer, so large values can build up over depth.

Why hook after the residual update rather than before? The post-residual hidden state is what gets fed into the next layer's operations (attention, MLP) and what ultimately feeds into the final LayerNorm before the language modeling head. It is the "official" representation at each depth. Hooking before the residual would miss the accumulation effect, which is precisely where extreme activations can grow.

Layerwise attention outputs: For each layer, the output of the multi-head attention block before it is added to the residual stream. This is the tensor $a_\ell$ in the notation above. Separating attention outputs from MLP outputs allows diagnosis of which sub-layer is generating large values — if attention outputs are large, the source may be attention sinks; if MLP outputs are large, the source may be the feedforward network's expansion.

Layerwise MLP outputs or MoE block outputs: For each layer, the output of the feedforward network (in dense models) or the aggregated expert outputs (in MoE models) before the residual addition. For SwiGLU-style architectures, this is the output after the gating and up-projection have been combined. For MoE models, this is the output after token routing has selected a subset of experts and their outputs have been weighted and summed.

MLP gate pre-activations in SwiGLU-style architectures: In a SwiGLU MLP, the computation involves two linear projections from the hidden dimension $d$ to an intermediate dimension $d_{\text{ff}}$ (often 8/3 × $d$ or 4 × $d$ ), followed by a SiLU (Sigmoid Linear Unit) activation on one branch and element-wise multiplication with the other branch. The "gate pre-activations" are the values before the SiLU non-linearity. The paper hooks these because large pre-activation values can indicate that the feedforward network is operating in a saturated regime, and because they are a separate component class that might carry information about activation dynamics not visible in the post-nonlinearity outputs.

Final LayerNorm outputs: The tensor after the final layer normalization, immediately before the language modeling head (the linear projection to vocabulary logits). This is the activation that directly determines the output distribution. For models with pre-norm architectures, this is typically the last operation before the unembedding matrix.

Why hook all six classes rather than just hidden states? The paper's central findings about carrier components — where the global maximum lives — require comparing maxima across component types. If only hidden states were hooked, the discovery that GPT-OSS-20B's global maximum comes from the MLP output (the "component-level exception") would have been invisible. Similarly, the observation that "all qualifying coordinates occur in layerwise hidden states" for the Sun-criterion-passing checkpoints (Section 4.3) requires having checked other components and found lower peaks there. The six-class hook coverage ensures that the global maximum $M$ is truly global — it accounts for every major activation tensor in the forward pass.

Streaming accumulation mechanism. During inference, activation statistics are accumulated online without storing individual tensors. The paper describes this as "streaming GPU accumulation" with "online collection without per-sample synchronization" (Figure 1, Stage 2). For each hooked tensor, the pipeline updates running estimates of:

Moments: mean and variance are updated using Welford's online algorithm or equivalent, avoiding the need to store all samples to compute these aggregates.
Extrema: the running maximum and minimum are tracked by element-wise comparison with each new tensor, maintaining the single largest and smallest values seen so far.
Quantiles: absolute-value quantiles are estimated using "log-histogram-based quantile estimation" (Figure 1, Stage 2). This likely works by maintaining a histogram of log-transformed absolute values with fixed bin edges (e.g., logarithmically spaced bins from $10^{-6}$ to $10^6$ ), incrementing bin counts as new values arrive, and then estimating quantiles by identifying which bin contains the target cumulative fraction. This is more memory-efficient than storing a sorted list of all observed values and more accurate than moment-based quantile approximations for the extreme tails of the distribution.

Why log-histogram rather than exact quantiles? The activation values span several orders of magnitude (the paper shows peaks from ~10² to ~7×10⁵), and the distribution is highly skewed — the vast majority of activations are small, with a tiny fraction being extreme. A linear histogram would need very fine bins in the low range (where most values live) and very wide bins in the high range (where precision matters less), or would waste most bins on the empty high range. A log-spaced histogram naturally captures multi-order-of-magnitude data with uniform relative precision. It also makes the quantile estimation robust to the exact choice of bin count: with logarithmically spaced bins, the bin widths are proportional to the value magnitude, so the relative error in quantile estimates is roughly constant across the range.

Batch processing. The paper does not specify batch size, but the pipeline uses "forward-only inference" on a GPU. Since no gradients are computed and no optimizer states are maintained, memory consumption is determined by activations and model weights, not by the 2–3× overhead of optimizer states. This means relatively large batch sizes can be used, limited primarily by the peak activation memory of the largest models at 4096-token sequence length. The total corpus of 19.5M tokens across 5,000 samples is processed in multiple forward passes, with statistics accumulated across batches.

The statistics-to-analysis decoupling. A key architectural choice is that "All figures and tables are generated from the resulting per-model statistics" (Section 2.3). The pipeline saves per-model JSON files containing all recorded statistics; downstream analysis scripts read these files and produce every figure and table in the paper without re-running inference. This decoupling has practical benefits: it allows iterative analysis (changing plot parameters, trying different comparisons) without recomputing the expensive forward passes, and it makes the per-model statistics a shareable artifact that other researchers can use for their own analyses.

Peak Stability Validation

Because the global maximum activation is an extreme statistic — by definition, the single largest value out of billions of individual activation scalars — it could be triggered by a small number of anomalous samples (e.g., a single unusually formatted code snippet or a pathological input). If the reported $M$ is dominated by one or two samples, it is not a stable model property but an artifact of corpus composition. The paper validates stability through a subsampling experiment described in Section 2.3.

Procedure. For four representative models, the authors construct "category-proportional subsamples" of 1,000 and 2,000 examples from the original 5,000-example corpus. "Category-proportional" means that each domain bucket is sampled proportionally to its representation in the full corpus — if math/science constitutes 850/5000 = 17% of the full corpus, it constitutes approximately 17% of the subsample. This preserves the domain balance while reducing sample size. Each subsample size (1,000 and 2,000) is repeated 5 times with different random draws, and each repeat runs the full activation scan (forward inference, hook collection, global maximum computation).

Results. The paper reports two summary statistics:

For the 1,000-sample repeats, "The largest coefficient of variation across 1k repeats is 10.1% for Qwen3-30B-A3B" (Section 2.3). The coefficient of variation (CV) is the standard deviation of the observed $M$ values across the 5 repeats divided by their mean. A CV of 10.1% means that the maximum activation magnitude varies by about ±10% of its mean across different random 1,000-sample subsets.
For the 2,000-sample repeats, "the largest coefficient of variation across 2k repeats is 8.2%" (Section 2.3).

Interpretation. These results indicate that the reported maximum activations "are not accidental single-sample artifacts and that the measurements are statistically robust at the scale studied here" (Section 2.3). The key phrase is "at the scale studied here" — the experiment validates that 5,000 samples is sufficient for the peaks to stabilize, but it does not guarantee that a 50,000-sample corpus wouldn't produce higher peaks. The decreasing CV from 1,000 to 2,000 samples (10.1% → 8.2%) suggests that the peaks are approaching their asymptotic values, though complete saturation cannot be verified without testing larger corpora.

Why these four models? The paper doesn't specify which four models were tested, but the choice likely targets the extremes of the observed range — a Qwen3.5 low-peak model, a Gemma-family high-peak model, a Qwen2.5 medium-peak model, and the Qwen3 MoE checkpoint (which shows the highest CV) — to ensure that stability holds across the full magnitude spectrum.

What this validation does NOT establish. The subsampling experiment validates that the measured $M$ is stable under resampling of the specific evaluation corpus, but it does not validate that the corpus itself is representative of all possible text. A model might produce larger activations on, say, highly repetitive code or mathematical proofs with unusual formatting, and those would not be captured unless they appear in the corpus. The paper acknowledges this as a limitation: "maximum activation is an extreme statistic and could be affected by longer contexts, rare inputs, or larger evaluation corpora" (Section D.3). The stability validation addresses sampling noise but not domain coverage.

Relationship to the Binary Massive-Activation Criterion (Sun et al., 2024)

The paper explicitly positions its continuous metric $M$ in relation to the binary criterion from Sun et al. (2024), which the paper calls "the Sun criterion." Understanding this relationship is essential because the two frameworks serve different purposes and can reach different conclusions about the same checkpoint.

The Sun criterion defined. Given a hidden state vector $\mathbf{x} \in \mathbb{R}^d$ for a single token at a single layer, a coordinate $x_i$ is classified as a massive activation if it simultaneously satisfies two conditions:

Absolute magnitude threshold: $|x_i| > 100$
Local sparsity threshold: $|x_i| / \text{median}_{j=1}^d |x_j| > 1000$

At the model level, a checkpoint "passes the criterion if any hidden layer contains at least one token-feature coordinate satisfying both thresholds" (Section 3.1). This is a binary existence test — the model either has at least one such coordinate or it doesn't.

Why both thresholds? The absolute threshold ( $>100$ ) ensures the value is numerically large in an absolute sense — a value of 10 with a ratio of 10,000× would be locally sparse but not a quantization concern because the absolute magnitude is small enough to fit comfortably in standard floating-point representations. The local sparsity threshold ( $>1000\times$ the per-token median) ensures the value is an outlier within its own context — a value of 500 in a tensor where the median is 200 would be absolutely large but not particularly extreme relative to its neighbors, which would make it less problematic for per-tensor quantization (since the scale would already need to accommodate ~200-magnitude values anyway). The combination identifies values that are both large AND isolated.

Application to the 24 checkpoints. The paper reports that "20 of the 24 main-analysis checkpoints pass the Sun criterion" (Section 3.2). The four failing checkpoints are Qwen2.5-1.5B, Qwen3.5-0.8B, Qwen3.5-9B, and Qwen3.5-35B-A3B. The paper identifies two distinct failure mechanisms:

Failure mechanism 1: Large but not sparse (Qwen2.5-1.5B). This checkpoint reaches an absolute peak of 7,968, which comfortably exceeds the $>100$ absolute threshold. However, "the median absolute value within the peak token is 13.9, giving a local ratio of roughly 574 and falling below the 1000× threshold" (Section 3.2). In plain language: this model produces large activations, but they are relatively dense — the surrounding activations are also fairly large (median 13.9), so the peak of 7,968 is "only" about 574 times the median, not 1,000 times. The model "exhibits large but relatively dense activations rather than locally sparse massive activations" (Section 3.2).

Failure mechanism 2: Systematically suppressed scale (Qwen3.5 checkpoints). Qwen3.5-0.8B, Qwen3.5-9B, and Qwen3.5-35B-A3B fail because "their overall activation scale is systematically suppressed" (Section 3.2). In Figure 2, these checkpoints' points cluster in the lower-left region of the scatter plot (low absolute magnitude, low-to-moderate local ratio), never entering the upper-right passing region. The absolute peaks for these models (122, 956, 132) are much lower than for other families at comparable sizes — Qwen3.5-9B peaks at 956 while Qwen2.5-7B peaks at 13,248. The suppression is comprehensive: it's not that these models have locally dense large activations (like Qwen2.5-1.5B), but rather that their activation magnitudes are low across the board, so they neither exceed the absolute threshold nor build up large local ratios.

Why the continuous M differs from the binary criterion, and why it matters. Figure 2 is the key diagnostic. The scatter plot shows, for each checkpoint, each layer's peak token (circles) and each layer's highest local-ratio token (triangles). The passing region is the upper-right quadrant (above the $|x_i| > 100$ horizontal dashed line and to the right of the $|x_i|/\text{median} > 1000$ vertical dashed line). Checkpoints that pass the criterion have at least one point in this region; checkpoints that fail have none.

The critical observation is that the binary criterion and the continuous magnitude $M$ can disagree in both directions. Qwen2.5-1.5B has a peak of 7,968 but fails the criterion; it poses a quantization risk (7,968 is large enough to dominate a per-tensor scale) despite being classified as "no massive activations." Conversely, Qwen3.5-9B has a peak of only 956 but passes the criterion (somewhere in its layers, there is a coordinate with $|x_i| > 100$ and ratio > 1000); its quantization risk is lower than Qwen2.5-1.5B's despite being classified as "has massive activations." The paper states this directly: "some checkpoints failing the binary criterion are the easiest to quantize, while some passing it are the hardest" (Section 1, Contributions bullet 2).

The deployment-motivated reframing. The paper's central methodological move is to treat the binary criterion as a "descriptive bridge to prior work" (Section 3.2) while using $M$ as the primary metric. The justification is practical: "for the deployment-oriented question of this paper — 'how large can the activation magnitude become' — the absolute peak M is the directly relevant quantity" (Section 3.2). A per-tensor INT-8 quantizer doesn't care whether the largest value is locally sparse or locally dense — it only cares what the largest value is, because that determines the scale factor. If $M = 7000$ , the quantizer needs to accommodate values up to 7000 regardless of whether the median is 1 or 100. The local sparsity ratio is informative for understanding why the peak formed (mechanistic analysis) and for designing mitigations (sparse outlier preservation, rotation-based distribution), but it is not the primary risk indicator for scale selection.

Operationalizing the comparison. Table 1 reports, for each checkpoint, the Top-5, Top-10, Top-100, Top 1%, Top 10%, and median absolute values "from a single representative layer chosen for the local-sparsity diagnostic" (Section 3.1). For passing models, this is a hidden layer containing a same-token sparse extreme value (one that satisfies the Sun criterion). For failing models, this is the full-model peak location (the layer where $M$ occurs). The table includes both the absolute values and the median, allowing the reader to compute the local ratio (Top-1 / median) and verify whether the Sun criterion is met.

For example, Qwen2.5-1.5B shows Top-1 = 7,968 and median = 13.9, giving ratio = 7968/13.9 ≈ 573, which is < 1000 — hence the × marker indicating failure. Qwen3.5-27B shows Top-1 = 167.0 and median = 0.225, giving ratio = 167/0.225 ≈ 742, which is also < 1000 — but the absolute peak of 167 is so low that the failure is inconsequential for deployment. The dual reporting (absolute peaks in Figure 6, Sun-criterion annotation via × markers, detailed token-level statistics in Table 1) gives the reader both perspectives.

What the paper does NOT do. The paper does not propose a new definition of massive activations. It does not argue that the Sun criterion is wrong or should be replaced. It does not re-analyze the Sun-criterion failures in terms of the normalization-stack dynamics identified by Sun et al. (2026) — the authors explicitly note that "linking each failing checkpoint to specific normalization or attention dynamics... is left to a mechanistic follow-up" (Section D.3). The contribution is methodological: the observation that the binary criterion and the continuous magnitude can disagree, and the argument that for deployment purposes, the continuous magnitude is the more directly actionable statistic.

4. Key Insights and Innovations

Innovation 1: Reframing Massive Activations from a Binary Existence Test to a Continuous Deployment-Relevant Model Property

The paper's most fundamental conceptual contribution is not a new method or finding, but a reframing of what "massive activations" means for downstream practice. Prior work, crystallized by Sun et al. (2024), defined massive activations through a binary criterion: a coordinate is "massive" if it simultaneously exceeds an absolute threshold (|x_i| > 100) and a local sparsity threshold (|x_i| / median > 1000). Under this framework, a model either "has" massive activations or it doesn't — a categorical diagnosis inherited from the interpretability literature's focus on existence and functional role. The deployment question, however, doesn't care about binary classification. A per-tensor INT-8 quantizer doesn't ask whether the largest activation is locally sparse or locally dense — it asks what the largest value is, because that value alone determines the scale factor and how many quantization levels are wasted.

The paper replaces the binary question with a continuous one: M = max |a|, the global maximum absolute activation observed across all layers and components. This is a scalar model property, directly measurable, directly actionable for quantization scale selection, and comparable across models. The shift from "does this model have massive activations?" to "how large can activations get in this model, and where do they live?" transforms activation magnitude from a research artifact to be diagnosed into a model-card statistic — a deployment-relevant number that should be reported alongside parameter count, context length, and benchmark scores.

Why this is a fundamental shift rather than incremental refinement. The binary criterion and the continuous magnitude can disagree in both directions — and the paper demonstrates this empirically (Figure 2, Table 1). Qwen2.5-1.5B fails the Sun criterion (peak 7,968, local ratio ~574, below the 1000× threshold) but has a larger absolute peak than Qwen3.5-9B (peak 956), which passes the Sun criterion. Under the binary framework, Qwen2.5-1.5B is classified as "no massive activations" despite posing a greater quantization risk. Under the continuous framework, the ordering matches deployment reality: the model with the larger M requires a larger scale factor. This disagreement exposes the binary criterion as an answer to a different question than the one practitioners need answered — it identifies structurally interesting outlier coordinates, not deployment risk. The paper doesn't argue the Sun criterion is wrong; it argues it's answering a mechanistic question while practitioners need a magnitude question answered.

Prior anchoring. Dettmers et al. (2022) treated outlier features as a preservation target for INT8 inference; Sun et al. (2024) sharpened the definition with the dual threshold; Bondarenko et al. (2023) and Gu et al. (2025) traced mechanistic origins. All of these studies ask "where do large activations come from and what do they do?" The paper replaces "where do they come from?" with "how large are they, and how does this vary?" — a reframing that makes the measurement directly actionable for quantization decisions without requiring mechanistic understanding.

Evidence anchor. The disagreement is visualized in Figure 2: the four failing checkpoints avoid the upper-right passing region, but two of them (Qwen2.5-1.5B with M≈8K, Qwen3.5-9B with M≈956) have very different absolute magnitudes. The continuous M values are reported in Figure 6, where the deployment-relevant ordering is immediately visible regardless of Sun-criterion status.

Innovation 2: The Empirical Demonstration That Maximum Activation Magnitude Is Not Predicted by Parameter Count — It Breaks Monotonic Scaling by Orders of Magnitude Across Families

Before this paper, the dominant implicit assumption in quantization and deployment work was that activation range scales monotonically with model size. This was a reasonable extrapolation from the LLaMA-derivative monoculture: within a single family, larger models tend to exhibit larger activations, and since prior studies concentrated on a narrow set of families (OPT, BLOOM, LLaMA variants), the within-family pattern was mistaken for a cross-family law. The paper systematically dismantles this assumption by showing that family and architecture identity override parameter-count effects by factors of 10–100×.

The evidence is starkest in Figure 6: at comparable parameter counts (~27–32B), Qwen3.5-27B peaks at 167, Qwen3-32B at 35,328, Qwen2.5-32B at 30,848, and Gemma3-27B-it at 696,320. That's a span of nearly four orders of magnitude at essentially the same model size — a 4,000× range that no monotonic scaling law could capture. Even within a single 7–9B scale bracket, the range from Qwen3.5-9B (956) to Qwen2.5-VL-7B (8,256) to Gemma2-9B (1,656) to Qwen3-8B (17,664) spans roughly 18×. The paper's matched-design comparisons in Section 5 and Appendix C decompose this variation into identifiable factors: MoE routing suppresses peaks by 14–23× relative to same-family dense counterparts (Figure 8), Qwen3→Qwen3.5 generational change suppresses peaks by ~200× for the ~30B scale (from 35,328 to 167), and Gemma2→Gemma3 generational change amplifies peaks by ~120× at 2B→4B and ~4.4× at 27B (Appendix Figure 7).

Why this is a fundamental empirical finding rather than incremental benchmarking. The paper doesn't just report numbers — it establishes a negative result with practical force: parameter count is an unreliable proxy for activation range. Any deployment pipeline that assumes "larger model = more quantization difficulty" will be wrong for MoE models (which are large by total parameters but low by activation peaks), for Qwen3.5 (which scales to 27B while maintaining peaks under 1,000), and for Gemma3 (which reaches extreme peaks at only 4B parameters). This negative result has immediate practical implications: a practitioner choosing between Qwen3.5-27B and Gemma3-27B-it for low-bit deployment would face a 4,000× difference in activation range despite identical parameter counts — a difference invisible without direct measurement but catastrophic for quantization quality if unanticipated.

Prior anchoring. Within-family scaling (Figure 5) shows that the monotonic assumption holds locally — Qwen2.5, Qwen3.5, and Gemma3 all show increasing peaks with size within the same family. This is consistent with prior observations on LLaMA variants. The paper's contribution is showing that this local monotonicity does not generalize across families. The interpretive studies (Sun et al., 2024; Dettmers et al., 2022) had no reason to test cross-family variation because they studied narrow model sets. The quantization literature (SmoothQuant, AWQ, GPTQ) calibrated mitigations on specific models without asking whether the calibration would hold for other families. The paper provides the first evidence that it likely wouldn't — the quantization hyperparameters that work for Qwen3.5-27B (peak 167) would be wildly inappropriate for Gemma3-27B-it (peak 696,320).

Evidence anchor. Figure 6 shows the full scatter of global maxima across all 24 checkpoints, with family identity encoded by color. The vertical axis is log-scale, so the roughly four-order-of-magnitude span is immediately visually apparent. The matched-design contrasts in Figure 5 (within-family scaling), Figure 8 (MoE vs. dense), Figure 7 (generational trends), Figure 9 (vision-language vs. text), Figure 10 (Base vs. Instruct), and Figure 11 (training-stage evolution) collectively decompose this variation into identifiable factors.

Innovation 3: The Identification of the Residual Stream as the Nearly-Universal Carrier of Extreme Activations — With One Informative Exception

The paper makes a crisp empirical claim: the residual stream (post-residual hidden states) carries the global maximum activation in 22 of 24 main-analysis checkpoints (Section 4.3). This finding has both diagnostic and prescriptive force. Diagnostically, it tells practitioners where to look — if you want to know a model's activation range, inspect the hidden-state peaks, not just the attention or MLP outputs. Prescriptively, it tells quantization researchers where to focus mitigation effort — any quantization scheme that protects attention outputs and MLP outputs but neglects hidden states is protecting the wrong tensors.

The two exceptions are instructive rather than undermining. GPT-OSS-20B's global maximum comes from the MLP output, making it the sole "component-level exception" (Table 2). This tells us that the residual-stream dominance is not a logical necessity — it's an empirical regularity that holds across 8 families, dense and MoE architectures, base and instruction-tuned variants, and multiple generations, but it can be broken by specific architectural choices in GPT-OSS. The Qwen3.5-0.8B checkpoint peaks at the final LayerNorm output, but this is a low-magnitude peak (M = 122) from a systematically suppressed family, making it less a counterexample to the residual-stream claim and more a consequence of scale suppression making the final normalization's modest values visible.

Why this is a practical innovation rather than a mechanistic one. Prior work (Sun et al., 2026; Bondarenko et al., 2023; Gu et al., 2025) provided mechanistic accounts of why residual-stream spikes form — attention sinks on BOS tokens, early-layer step-up blocks under pre-norm transformers, softmax constraints forcing mass allocation. The paper's contribution is not a new mechanistic explanation but an empirical generalization with direct operational implications: across a far wider model landscape than previously tested, the residual stream is systematically the component where you'll find the largest values. This is actionable advice for anyone designing activation hooks, building quantization calibration pipelines, or setting numerical tolerances — inspect the hidden states first and deepest.

Prior anchoring. Prior work focused on hidden states because that's where the "massive activation" coordinates live by definition — Sun et al. (2024) defined their criterion on hidden state vectors. But they didn't systematically compare hidden-state peaks against other component classes to establish that hidden states dominate other components by orders of magnitude. The paper's six-class hook design (embeddings, hidden states, attention outputs, MLP/MoE outputs, SwiGLU gates, final LayerNorm) enables this comparison directly. For the high-peak models (Gemma3, Qwen2.5, Qwen3 dense), the hidden-state peaks exceed attention and MLP outputs by factors of 10–100× — a gap that justifies the prescription to focus quantization effort on the residual stream.

Evidence anchor. Figure 15 in Appendix G.3 shows component-level trajectories for three representative checkpoints: a low-peak model (Qwen3.5-9B), a high-peak model (Gemma3-27B-it), and the GPT-OSS component exception. In the high-peak model, hidden states dominate attention and MLP outputs by roughly 1–2 orders of magnitude across most layers. In GPT-OSS-20B, the MLP curve briefly exceeds the hidden-state curve, confirming the exception. The claim that "22 global maxima occur in layerwise hidden states" is made in Section 4.3 and supported by the full per-model statistics.

Innovation 4: The Discovery That MoE Routing Systematically Suppresses Activation Peaks by 14–23× Relative to Matched-Scale Dense Counterparts — An Observational Finding with Immediate Architectural Implications

The paper's most striking matched-design result is the MoE-versus-dense comparison in Figure 8: Qwen3-30B-A3B peaks at 1,512, compared to Qwen3-32B (dense) at 35,328 — a 23.4× reduction. Qwen3.5-35B-A3B peaks at 132, compared to Qwen3.5-27B (dense) at 167 (global M across all components; the Table 1 Top-1 value of 167 is from the representative layer and is by construction ≤ M) — the reduction factor is 14.0× when compared against the global M. These are not small, noise-level differences; they are order-of-magnitude suppressions that persist across two different model families (Qwen3 and Qwen3.5).

Why this is a significant empirical finding rather than an obvious consequence of sparse activation. The naive intuition would be: MoE models activate fewer parameters per token, so they should have smaller activations. But this intuition doesn't specify how much smaller, or whether the effect would be small (2×) or large (20×), or whether it would generalize across families. The paper quantifies the effect size precisely and shows it holds across families with very different baseline activation scales — Qwen3 dense peaks at ~35K while Qwen3.5 dense peaks at ~167, yet both see MoE variants suppress peaks by similar multiplicative factors. This constancy of the ratio despite the 200× difference in baseline scale suggests a structural effect of sparse routing on activation dynamics, not merely an additive offset.

Caveats and the observational nature of the evidence. The paper is careful to note this is observational, not causal: "with only n = 2 matched pairs and possible unobserved differences in training recipe between paired checkpoints, this contrast is observational rather than causal" (Appendix C.1). The MoE and dense checkpoints may differ not only in routing architecture but also in training data mixture, hyperparameters, or optimization schedule — any of which could contribute to the peak difference. The paper's restraint here is appropriate: it identifies a robust empirical pattern but does not claim to have isolated the mechanism. A causal claim would require controlled experiments varying only the dense/MoE axis while holding all else constant, which the paper does not do.

What makes this finding distinctive in the broader literature. Prior work on MoE models (DeepSeek-V3 technical report; Ling Team, 2025) focused on training efficiency and inference throughput, not activation range. The quantization literature's MoE-aware methods (e.g., QMoE for weight quantization) addressed the storage challenge of many expert weights rather than activation dynamics. To the paper's knowledge, this is the first study to directly measure and compare activation peaks between matched-scale dense and MoE checkpoints from the same families. The finding that MoE routing co-occurs with substantially suppressed activation peaks opens a new axis for deployment optimization: if MoE architecture reduces activation range by an order of magnitude, it may enable more aggressive activation quantization than a same-scale dense model — a benefit orthogonal to the throughput advantages MoE already provides.

Evidence anchor. Figure 8 presents the two matched pairs as side-by-side bar charts with explicit reduction factors annotated. The Qwen3-30B-A3B bar (1,512) sits at roughly 1/23 of the Qwen3-32B bar (35,328). The Qwen3.5-35B-A3B bar (132) sits below the Qwen3.5-27B value on the log-scaled vertical axis. The numerical ratios (23.4× and 14.0×) are stated in Section 5.4 and in the paper's abstract.

Innovation 5: The Demonstration That a Simple Continuous Measurement (`M`) Directly Covaries with Quantization Reconstruction Error — Closing the Loop from Measurement to Deployment Impact

The paper could have stopped at measurement — reporting that activation peaks vary by orders of magnitude across families, identifying residual-stream dominance, documenting MoE suppression. Instead, it takes the crucial additional step of validating that M actually matters for deployment. The INT-8 sanity check in Section D.2 and Figure 12 provides this validation: across eight representative checkpoints spanning the observed M range, higher M correlates with lower signal-to-quantization-noise ratio (SQNR) under both max-abs scaling and 99.9% clipping.

The results are directional rather than a calibrated dose-response curve — Qwen3.5-0.8B (M≈122) achieves 29.1 dB under max-abs scaling, while Gemma3-4B-it (M≈246K) drops to ~10–14 dB — but the directionality is exactly what the deployment argument requires: larger peaks → worse quantization reconstruction → the measurement is a valid risk indicator. The paper is appropriately modest about the scope of this validation, calling it a "sanity check" rather than "a calibrated dose–response curve" and acknowledging that "richer distributional and task-level measurements" would be needed for full quantization evaluation (Section D.2).

What distinguishes this from standard quantization benchmarking. The paper is not proposing a new quantization method, nor is it benchmarking existing methods for accuracy-perplexity tradeoffs. It is testing a specific causal chain: if M determines the scale factor in per-tensor quantization, and if the scale factor determines reconstruction error, then models with higher M should show lower SQNR under identical quantization recipes. The chain holds, establishing M as a valid prior for quantization risk — not a complete predictor, but a signal that should inform decisions about quantization bit width, clipping strategy, and whether to apply advanced mitigations.

The deliberate narrowness of the probe. The paper uses only two simple per-tensor recipes (max-abs and 99.9% clipping) at the peak hidden layer on 8 of 24 checkpoints. It explicitly excludes rotation-based (QuaRot, SpinQuant), prefix-based (PrefixQuant), and affine-flattening (FlatQuant) methods because those are "designed to neutralize M rather than to expose it" (Section D.2). This is a strategic choice: by using the simplest possible quantization — the one most directly affected by the activation upper bound — the paper isolates the M→SQNR relationship without confounding from mitigation-specific hyperparameters. The finding that more sophisticated mitigations can reduce but not eliminate the M-dependence of quantization quality is left implicit; the paper's point is that M shapes the starting condition that any mitigation must overcome.

Evidence anchor. Figure 12 shows grouped horizontal bars for eight models under the two quantization strategies, with SQNR on the horizontal axis and a 20 dB reference line. Right-side annotations report the ratio between the global peak and the 99.9% clipping threshold, quantifying how much mass is clipped. The visual pattern — leftmost bars (Qwen3.5 variants, low M) showing high SQNR, rightmost bars (Gemma3, high M) showing low SQNR — makes the correlation immediately apparent without requiring a statistical test.

5. Experimental Analysis

Evaluation Methodology

Dataset. All measurements use a custom-constructed 5,000-sample multi-domain corpus drawn from RedPajama sources (Shen et al., 2024). The corpus is partitioned into seven domain buckets with explicit target counts: 850 mathematical/scientific samples, 850 code samples, 850 English web samples, 850 knowledge-oriented samples (encyclopedic, book, Q&A text), 400 Chinese samples, 300 samples in other low-resource languages, and 900 additional English or mixed web samples. Samples are randomly truncated to lengths of 256, 512, 1024, 2048, or 4096 tokens with target proportions of 1%, 1%, 2%, 3%, and 93% respectively, producing an average length of approximately 3,899 tokens and roughly 19.5M total tokens. The corpus design ensures that no single domain or text type dominates the activation statistics, and the heavy skew toward 4,096-token sequences reflects the deployment scenario where long-context inference is common.
Base model(s). The main analysis covers 24 checkpoints from 8 open model families: Qwen2.5 (1.5B, 7B, 32B), Qwen2.5-VL (3B, 7B, 32B), Qwen3 (1.7B, 8B, 30B-A3B, 32B), Qwen3.5 (0.8B, 9B, 27B, 35B-A3B), Gemma2 (2B, 9B, 27B), Gemma3 (4B-it, 27B-it), Ling (5T, 10T, 15T, 20T training checkpoints), and GPT-OSS (20B). An additional 3 Qwen2.5-Instruct checkpoints (1.5B, 7B, 32B) are used for the Base-vs-Instruct comparison. Models are selected to span multiple parameter scales (0.8B to 35B), architectural forms (dense, MoE, vision-language), training stages (intermediate checkpoints, instruction-tuned variants), and generations (Qwen2.5→Qwen3→Qwen3.5, Gemma2→Gemma3). The selection explicitly targets "recent mainstream open LLM families rather than restricting the study to earlier LLaMA-style models" (Section 2.2). All checkpoints are public HuggingFace-compatible releases.
Metrics. The primary metric is the global maximum activation magnitude M = max |a|, defined as the largest absolute activation observed across all six hooked component classes (embedding outputs, post-residual hidden states, attention outputs, MLP/MoE outputs, SwiGLU gate pre-activations, final LayerNorm outputs) and all layers in each checkpoint. This is a single scalar per model reported as the global peak. Layerwise analyses use per-layer max |a| values for individual component classes, typically the hidden-state peaks. The Sun criterion (Sun et al., 2024) — checking whether any hidden-layer token-activation coordinate satisfies both |x_i| > 100 and |x_i| / median_j |x_j| > 1000 — is computed as a binary diagnostic but is not the primary metric; it is used as "a descriptive bridge to prior work" (Section 3.2). The INT-8 sanity check (Appendix D.2) additionally reports signal-to-quantization-noise ratio (SQNR) in dB at the peak hidden layer under two per-tensor symmetric quantization strategies.
Baselines. The paper does not compare against baseline methods in the traditional sense — it is an observational measurement study, not a method evaluation. The framework is: measure M under a unified protocol across 24 checkpoints, then compare across checkpoints along matched dimensions (size, architecture, family, generation, training stage). There is no "best-of-N baseline" or "majority voting baseline"; instead, the baseline is the set of per-model statistics that any single checkpoint produces. The INT-8 sanity check compares two quantization strategies as alternative scale-selection recipes — max-abs scaling (scale factor set to the observed maximum) and 99.9% clipping (scale factor set to the 99.9th percentile absolute value, clipping larger values) — but these are probes of how M translates into reconstruction error, not competing baselines for a method.
Generation budget / compute accounting. There is no "generation budget" in the conventional sense because no search or sampling is involved. Every checkpoint processes the identical corpus: 5,000 samples at up to 4,096 tokens each, totaling approximately 19.5M tokens per checkpoint. All inference is forward-only with frozen weights and no parameter updates. The computational cost per checkpoint scales with model size (parameters × tokens processed), but the paper does not compare methods at matched FLOPs — it simply reports what peaks occur when each model processes the same corpus. The primary "budget" is the corpus size (5,000 samples), which was chosen to ensure peak stability without making measurement prohibitively expensive across 27 checkpoints.
Cross-validation / statistical protocol. There is no cross-validation in the conventional sense because there is no model training or hyperparameter selection. The key statistical validation is the peak stability experiment described in Section 2.3: for four representative checkpoints, the authors construct category-proportional subsamples of 1,000 and 2,000 examples from the original 5,000-example corpus, repeat each subsample size 5 times with different random draws, and run the full activation scan on each repeat. The largest coefficient of variation across 1,000-sample repeats is 10.1% (for Qwen3-30B-A3B), and across 2,000-sample repeats is 8.2%. This establishes that the reported M values are not accidental single-sample artifacts and are statistically stable at the 5,000-sample scale. The authors acknowledge that this does not guarantee stability for larger corpora or different domain compositions — it validates robustness to sampling noise within the fixed corpus design, not representativeness of the corpus itself.

Main Quantitative Results

The paper's results are organized as an empirical survey rather than a method evaluation, so the "main results" are the descriptive findings that emerge from comparing M across checkpoints along matched dimensions. The results are presented in Section 5 (What Controls Peak Magnitude?) with supporting detail in Section 4 (Where Maximum Activations Form) and Appendix C (Architectural and Training Factors at Matched Scale).

Global Activation Magnitude Varies by Nearly Four Orders of Magnitude Across Model Families at Comparable Parameter Counts

The headline finding appears in Figure 6: the global maximum activation magnitude M spans from 122 (Qwen3.5-0.8B) to 696,320 (Gemma3-27B-it) across the 24 main-analysis checkpoints — a range of nearly 6,000× on a linear scale, or roughly 3.75 orders of magnitude on a log scale. At comparable parameter counts, the variation is equally dramatic. In the ~27–35B range: Qwen3.5-27B peaks at 167, Qwen3.5-35B-A3B at 132, Qwen3-32B at 35,328, Qwen2.5-32B at 30,848 (global M across all components; the Table 1 Top-1 of 22,144 is at the representative passing-criterion layer), Gemma2-27B at 157,696, Gemma3-27B-it at 696,320, and GPT-OSS-20B at 43,008. This is a span of roughly 4,000× at essentially the same model size bracket. In the ~7–9B range: Qwen3.5-9B peaks at 956, Qwen2.5-VL-7B at 8,256, Qwen3-8B at 17,664, Qwen2.5-7B at 13,248, and Gemma2-9B at 1,656 — still a span of roughly 18× within a narrow size window.

The paper's explicit interpretation is that "Cross-family variation is much larger than within-family scaling variation" (Section 5.2). This is visible in Figure 6, where within-family scaling (shown in Figure 5 as separate lines) produces modest multiplicative increases (Qwen2.5 rises from ~8K to ~31K for a 1.5B→32B increase; Gemma3 rises from ~246K to ~696K for a 4B→27B increase), while cross-family differences at fixed sizes produce order-of-magnitude gaps.

Figure 6 also annotates checkpoints that fail the Sun binary criterion with a × marker (Qwen2.5-1.5B, Qwen3.5-0.8B, Qwen3.5-9B, Qwen3.5-35B-A3B). The annotation visually reinforces the paper's reframing argument: Qwen2.5-1.5B (M ≈ 7,968) is marked as failing despite having a larger peak than several passing checkpoints (e.g., Qwen3.5-9B at 956, which passes), demonstrating that the binary criterion and continuous magnitude carry different information.

The Residual Stream Carries the Global Maximum in 22 of 24 Checkpoints

Section 4.3 reports that across the 24 main-analysis checkpoints, 22 global maxima occur in layerwise hidden states (post-residual hidden states). The two exceptions are GPT-OSS-20B (global maximum from the MLP output) and Qwen3.5-0.8B (global maximum from the final LayerNorm output). If attention is restricted to the 20 checkpoints that satisfy the Sun local-sparsity criterion, "all qualifying coordinates occur in layerwise hidden states" (Section 4.3).

The paper does not report the absolute magnitude differences between hidden-state peaks and attention/MLP peaks, but Figure 15 in Appendix G.3 provides component-level trajectories for three representative checkpoints. In the high-peak model (Gemma3-27B-it), hidden-state maxima exceed attention and MLP maxima by roughly 1–2 orders of magnitude across most layers. In GPT-OSS-20B, the MLP output curve briefly exceeds the hidden-state curve, confirming the component-level exception visually.

The practical implication is clear: "the residual stream is the dominant carrier through which extreme activation magnitudes are propagated and preserved" (Section 4.3). Any deployment-oriented measurement of activation range should inspect hidden-state peaks as the primary signal, and any quantization scheme should prioritize hidden-state tensors for protection or transformation.

Within-Family Scaling Shows Monotonic Increases for Most Families, With Gemma2 as a Notable Exception

Figure 5 presents within-family scaling: for checkpoints of different sizes within the same family and model form, plotting M against parameter count on a log scale. The paper reports that "Most families, including Qwen2.5, Qwen3.5, and Gemma3, show a stable within-family scale effect: the global maximum activation increases with parameter count" (Section 5.1). Specifically:

Qwen2.5: 1.5B → 7B → 32B shows M rising from ~8K → ~13K → ~31K (global M; the Table 1 Top-1 for 32B is 22,144 at the representative layer).
Qwen2.5-VL: 3B → 7B → 32B shows M rising similarly, tracking the text-only variants.
Qwen3: 1.7B → 8B → 32B shows M rising from ~14K → ~18K → ~35K.
Qwen3.5: 0.8B → 9B → 27B shows M rising from 122 → 956 → 167 (the 27B value is surprisingly low, but the trend from 0.8B to 9B is upward; see discussion of the 27B point below).
Gemma3: 4B-it → 27B-it shows M rising from ~246K → ~696K.

The exception is Gemma2, where the 9B checkpoint peaks at 1,656, which is below the 2B checkpoint at 2,992. This "local non-monotonic" case is flagged with a cross marker in Figure 5. The paper notes that "individual checkpoints can still deviate due to training or recipe differences" (Section 5.1), cautioning that even within-family scaling is not rigidly monotonic.

The Qwen3.5-27B point (M = 167) is lower than Qwen3.5-9B (M = 956) in Figure 5, appearing as a downward deflection in the otherwise upward Qwen3.5 curve. The paper does not highlight this explicitly as anomalous, but it is visible in Figure 5 and may reflect the 27B checkpoint's specific training recipe diverging from the smaller Qwen3.5 variants. The Qwen3.5 family is the "low-outlier counterexample family" (Table 2) with systematically suppressed activation scales, so the small absolute magnitude differences (956 vs. 167) are less practically significant than the 4,000× cross-family gaps — both are in the hundreds-to-low-thousands range, a regime where quantization is relatively unproblematic compared to the Gemma3 regime.

Generational Evolution Is Family-Dependent and Non-Monotonic

Appendix Figure 7 compares generational trends at matched sizes. For Qwen across three size groups (~1B, ~7–9B, ~27–32B), the trajectory is an inverted-V: Qwen2.5 → Qwen3 increases, then Qwen3 → Qwen3.5 sharply decreases. The paper reports specific values: at the ~1B scale, Qwen2.5-1.5B (7,968) → Qwen3-1.7B (14,208) → Qwen3.5-0.8B (122); at the ~7–9B scale, Qwen2.5-7B (13,248) → Qwen3-8B (17,664) → Qwen3.5-9B (956); at the ~27–32B scale, Qwen2.5-32B (30,848) → Qwen3-32B (35,328) → Qwen3.5-27B (167). The Qwen3→Qwen3.5 suppression is by a factor of roughly 116× at the 1B scale, 18× at the 8B scale, and 211× at the 32B scale.

For Gemma, the trajectory is a sharp increase: Gemma2-2B (2,992) → Gemma3-4B-it (245,760) at the small scale (~120× increase), and Gemma2-27B (157,696) → Gemma3-27B-it (696,320) at the large scale (~4.4× increase). The paper explicitly notes that the Gemma3 checkpoints are instruction-tuned, so "the Gemma2/Gemma3 comparison is not interpreted as a strict base-to-base ablation" (Section 2.2).

The key conclusion from the generational analysis is that "Maximum activation magnitude does not monotonically shrink or grow with release time; instead, it is highly family-dependent" (Section 5.3). There is no industry-wide trend toward larger or smaller activation peaks — each family's design decisions dominate the generational trajectory.

MoE Routing Co-Occurs with 14.0–23.4× Lower Activation Peaks Relative to Matched-Scale Dense Counterparts

Figure 8 presents the MoE-versus-dense comparison using two matched pairs from the same families at similar total parameter scales. Qwen3-30B-A3B (MoE, 30B total, 3B active per token) peaks at 1,512, compared to Qwen3-32B (dense) at 35,328 — a 23.4× reduction. Qwen3.5-35B-A3B (MoE, 35B total, 3B active per token) peaks at 132, compared to the Qwen3.5-27B dense counterpart's global M (the paper states this value in Appendix C.1 and notes the reduction factor is 14.0×).

The paper interprets this as "unlikely to be explained by active-parameter count alone; it is consistent with an architectural effect of sparse routing and expert structure" (Appendix C.1). The MoE checkpoints activate fewer parameters per forward pass (3B active vs. 32B or 27B active for their dense counterparts), but the peak suppression factor (14–23×) is larger than the active-parameter ratio (~10×), suggesting that sparse routing changes activation dynamics beyond simply reducing the number of active parameters.

The paper explicitly cautions about the strength of evidence: "with only n = 2 matched pairs and possible unobserved differences in training recipe between paired checkpoints, this contrast is observational rather than causal" (Appendix C.1). The two pairs come from different families (Qwen3 and Qwen3.5) with very different baseline activation scales, limiting the statistical basis for generalization.

Vision-Language Adaptation Co-Occurs With Moderate Peak Reductions (1.4–1.6×) at Matched Scale

Figure 9 compares Qwen2.5-VL with text-only Qwen2.5 at 7B and 32B. Qwen2.5-VL-7B peaks at 8,256, which is 1.6× lower than Qwen2.5-7B at 13,248. At 32B, Qwen2.5-VL peaks at 22,144, which the paper reports as 1.4× lower than the text-only Qwen2.5-32B global M of 30,848 (Appendix C.2). The Table 1 Top-1 value of 22,144 for Qwen2.5-32B is from the representative layer and is by construction ≤ the global M.

The paper concludes that "Vision-language checkpoints therefore remain in a high-magnitude regime comparable to their text-only backbones, but modality adaptation and training-recipe changes co-occur with a moderate same-scale shift in the maximum activation" (Appendix C.2). The 1.4–1.6× reductions are modest compared to the 14–23× MoE reductions or the 200× Qwen3→Qwen3.5 suppression — vision-language adaptation does not fundamentally change the activation regime, but it does measurably reduce peak magnitudes. Again, this is observational (n = 2 matched pairs from a single family) and "could in principle reflect unobserved training-recipe differences" (Appendix C.2).

Instruction Tuning Primarily Compresses Late-Layer Peaks, With Preserved Middle-Layer High-Peak Regions

Figure 10 compares Qwen2.5 Base and Instruct checkpoints at matched backbone and parameter scales (n = 3 pairs). At 1.5B, the global peak remains unchanged at 7,968. At 7B, it increases slightly from 13,248 to 13,312 (a 1.005× increase — essentially flat). At 32B, it decreases from 30,848 (global M) to 22,144, a 1.4× reduction.

The paper goes beyond global-peak comparison to provide layerwise analysis (Appendix C.3). For the 32B pair, "the stable high-peak middle-layer regions remain at 1.000× of their base values" — the mid-layer plateau is identical between Base and Instruct. However, late-layer peaks change substantially: "The final-layer peaks decrease from 1,536 to 840 (−45%), from 4,864 to 2,528 (−48%), and from 30,848 to 21,248 (−31%), with only mild compensation in the penultimate layer." The 1.5B and 7B pairs show similar patterns: middle layers are unchanged (multiplicative factors of 1.000× and 1.005×), while final layers drop by 45% and 48% respectively.

The interpretation is that "The global-peak decrease in Qwen2.5-32B-Instruct is therefore best understood as late-layer compression that exposes an already existing mid-layer high-peak region as the global maximum, rather than the creation of a new activation structure" (Appendix C.3). In plain language: instruction tuning doesn't fundamentally restructure where peaks form — it compresses the late-layer activations (where the global peak lived in the Base model), causing the mid-layer plateau (which was already there, just at a slightly lower magnitude) to become the new global maximum. This is a nuanced finding: SFT changes which layer's peak is reported as M, not the underlying presence of high-magnitude activations in the model.

Training Progress Co-Occurs With Monotonically Increasing Activation Peaks at Fixed Architecture

Figure 11 shows the Ling-mini training-stage evolution. As training progresses from 5T to 20T tokens, the global maximum activation rises monotonically: 7,648 → 9,024 → 9,600 → 10,240 (Appendix C.4 reports these values; the figure caption states "5T→20T: 1.54× higher," but the numerical values suggest 10,240/7,648 = 1.34×. The discrepancy between the 1.54× annotation in Figure 11 and the 1.34× computed from the stated values may reflect a labeling error in the figure or a different comparison base; the text in Appendix C.4 states "overall increase of 1.34×"). Regardless of the exact factor, the direction is unambiguous: longer training → larger peaks.

This is a compact finding: even when family, architecture, and approximate parameter scale are held fixed, "training progress itself co-occurs with a gradual increase in extreme activation magnitudes" (Appendix C.4). The magnitude of this effect (1.34× over a 4× increase in training tokens) is small relative to cross-family variation (which spans ~4,000×), but it demonstrates that activation peaks are not static properties of an architecture — they evolve during training. The paper recommends that "Training stage should therefore be recorded as a separate observation dimension alongside model size and architecture" (Appendix C.4).

Two Distinct Layerwise Emergence Patterns Coexist Across Families

Section 4.2 identifies "two patterns" in the layerwise trajectories of hidden-state peak magnitudes, illustrated in Figure 4 with representative curves. The jump-and-plateau pattern shows activation magnitude rising sharply in early or middle layers and then remaining high over a long layer interval. Representative models include Qwen2.5 and GPT-OSS (Figure 4, left panel). The gradual-accumulation pattern shows activation magnitude increasing more smoothly with depth and often reaching its maximum in later layers. Representative models include Qwen3.5 and Gemma (Figure 4, right panel).

The paper states that "The pattern is strongly associated with model family and architecture rather than being a monotonic function of parameter scale" (Section 4.2). The dichotomy is presented as a qualitative description of the depth-normalized heatmap in Figure 3, not as a quantitative classifier. Figure 3 itself is a rich visual: the horizontal axis is normalized depth, color encodes log-scale peak magnitude, and white hollow markers indicate the peak depth bin for each checkpoint. The heatmap reveals that even within families, peak depth has "no universal location across architectures: even within the same family, maxima can occur in shallow, middle, or deep layers" (Section 4.1). The Qwen2.5 family, for instance, shows peak markers spread across early, middle, and late depth bins depending on model size.

Ablation Studies and Robustness Checks

The paper's "ablation" structure differs from conventional method-evaluation papers because there is no method being ablated. The closest analogs are the matched-design comparisons (already described above as main results) and several diagnostic probes that test the robustness of the measurement framework itself.

Peak stability under subsampling (Section 2.3). The most critical robustness check is whether the global maximum M is dominated by a small number of anomalous samples. For four representative checkpoints, category-proportional subsamples of 1,000 and 2,000 examples are drawn 5 times each, and the full activation scan is repeated. The largest CV across 1,000-sample repeats is 10.1% (Qwen3-30B-A3B); across 2,000-sample repeats, it is 8.2%. The decreasing CV with subsample size (from 10.1% at 1K to 8.2% at 2K) suggests the peak is approaching its asymptotic value. The paper interprets this as evidence that "the reported maximum activations are not accidental single-sample artifacts" (Section 2.3). This is not an exhaustive stability guarantee — it validates robustness to sampling noise within the fixed corpus design, not to corpus composition changes — but it establishes that the 5,000-sample scale is sufficient for the measured peaks to stabilize.

Sun criterion failure analysis (Section 3.2, Figure 2). For the four checkpoints that fail the Sun binary criterion (Qwen2.5-1.5B, Qwen3.5-0.8B, Qwen3.5-9B, Qwen3.5-35B-A3B), the paper provides a failure-mode diagnosis using the scatter plot in Figure 2. Each layer's peak token (circles) and highest local-ratio token (triangles) are plotted against absolute magnitude (x-axis, log scale) and local ratio (y-axis, log scale). The dashed thresholds at |x_i| > 100 and ratio > 1000 define the passing region. Qwen2.5-1.5B's points cluster near |x_i| ≈ 8,000 and ratio ≈ 500–600 — above the absolute threshold but below the ratio threshold, confirming the "large but not sparse" diagnosis. The three Qwen3.5 checkpoints' points cluster in the lower-left quadrant (low absolute magnitude, low-to-moderate ratio), confirming the "systematically suppressed scale" diagnosis. This diagnostic validates that the binary criterion and continuous M capture genuinely different signal — not just measurement noise.

Component-level trajectory comparison (Appendix G.3, Figure 15). For three representative checkpoints (Qwen3.5-9B as low-peak, Gemma3-27B-it as high-peak, GPT-OSS-20B as component exception), the paper plots component-level layerwise trajectories for hidden states, attention outputs, and MLP/MoE outputs. This ablation validates the residual-stream dominance claim: in Qwen3.5-9B and Gemma3-27B-it, the hidden-state curve lies substantially above the attention and MLP curves across most layers. In GPT-OSS-20B, the MLP curve briefly exceeds the hidden-state curve in a mid-to-late layer region, confirming the component-level exception. This is a qualitative ablation — it shows the pattern holds across the extremes of the observed M range — but is not quantified (no reported gap sizes, no statistical test of hidden-state vs. component differences).

Per-family layerwise trajectories (Appendix G.2, Figure 14). Each family's checkpoints are plotted as separate subplots with hidden-state peak magnitude on a log-scale vertical axis against normalized depth. This provides a visual ablation of the two-pattern dichotomy: Qwen2.5 shows clear jump-and-plateau (early spike, sustained plateau), Qwen3 shows a similar pattern, Qwen3.5 shows gradual accumulation with low overall magnitude, Gemma2 and Gemma3 show gradual accumulation at very high overall magnitude, Ling shows moderate gradual accumulation, and GPT-OSS shows a jump-and-plateau with a single checkpoint. The subplots confirm that the trajectory shape is strongly associated with family identity, not parameter count — within each subplot, curves for different-sized checkpoints in the same family trace roughly similar shapes, while across subplots the shapes differ qualitatively.

Deployment-oriented tiering (Appendix G.1, Figure 13). Checkpoints are grouped into five order-of-magnitude tiers based on M: T1 low (<2K, 6 checkpoints), T2 moderate (2K–10K, 6 checkpoints), T3 high (10K–50K, 8 checkpoints), T4 very high (50K–200K, 2 checkpoints), T5 extreme (>200K, 2 checkpoints). This is a descriptive grouping, not an ablation, but it serves as a robustness check on the claim that models require different quantization treatment: the distribution across tiers (6, 6, 8, 2, 2) shows substantial diversity, with checkpoints spread across five orders of magnitude, meaning a one-size-fits-all quantization strategy would serve some models poorly.

INT-8 sanity check (Appendix D.2, Figure 12). This is the paper's key deployment-validation probe, not an ablation of a method design choice but a test of the core claim that M matters for quantization. Eight representative checkpoints spanning the observed M range are subjected to two per-tensor symmetric INT-8 quantization strategies (max-abs scaling and 99.9% clipping) at their peak hidden layer, using 128 calibration samples and 256 evaluation samples. SQNR is reported for each combination. The pattern is directional rather than perfectly monotonic: Qwen3.5-0.8B (lowest M, 122) achieves the highest SQNR (29.1 dB under max-abs, 26.3 dB under 99.9% clipping); Gemma3-4B-it (M ≈ 246K) and other high-peak models achieve roughly 10–14 dB under max-abs and further drop to ~0.2–0.4 dB under 99.9% clipping. Qwen3.5-9B and Qwen3.5-35B-A3B (M = 956 and 132 respectively) show lower SQNR than Qwen3.5-0.8B despite having low absolute peaks, indicating that "within-layer distributional structure beyond peak magnitude also affects quantization error" (Appendix D.2). This finding is important: it validates that M is a relevant prior for quantization risk while simultaneously demonstrating that M alone is not a complete predictor — the full activation distribution at the peak layer matters too. The ratio between global peak and 99.9% clipping threshold is annotated on the right side of Figure 12, quantifying the mass being clipped under the 99.9% strategy.

Absence of certain ablations. The paper explicitly identifies several analyses it does NOT perform, which function as a form of gap acknowledgment rather than missing evidence. It does not: (i) extend measurements beyond 4,096-token contexts to test whether maxima grow or saturate at 32K–128K lengths; (ii) control for unobserved training-recipe differences in the MoE-vs-dense and VL-vs-text comparisons (both rest on n = 2 matched pairs); (iii) test whether the INT-8 SQNR-versus-M relationship generalizes to all 24 checkpoints (the sanity check covers only 8); (iv) provide a quantitative classifier for the two-pattern trajectory dichotomy in Section 4; or (v) link Sun-criterion failures to specific normalization or attention dynamics (deferred to "a mechanistic follow-up," Section D.3). These gaps are limitations, not failures of the measurement framework, but they constrain the strength of claims that can be made from the observational contrasts alone.

Critical Assessment

The paper's central claims, introduced in the executive summary and elaborated throughout, are empirical rather than causal. The critical question is whether the reported measurements genuinely support them, and under what conditions the claims generalize.

Claim: "Global maxima span over nearly four orders of magnitude at comparable parameter counts"

This claim is strongly supported by the measurements but with two important nuances. The range from Qwen3.5-0.8B (122) to Gemma3-27B-it (696,320) is approximately 5,700×, or roughly 3.75 orders of magnitude on a log scale — close to the claimed four orders. At matched parameter scales, the range is similar: at ~27–35B, the span from Qwen3.5-27B (167) to Gemma3-27B-it (696,320) is roughly 4,200×. The evidence in Figure 6 directly supports this.

The first nuance is that the extremes of this range come from instruction-tuned checkpoints, not base models. Gemma3-27B-it is instruction-tuned, and its extreme peak (696,320) reflects both the Gemma3 architecture and the effects of SFT. The paper acknowledges that the Gemma3 checkpoints are "publicly released instruction-tuned checkpoints" (Section 2.2) and that the Gemma2/Gemma3 comparison is "not interpreted as a strict base-to-base ablation." A strict base-model-only comparison would show a narrower range — likely from Qwen3.5-9B (956, base) to Qwen3-32B (35,328, base) or Gemma2-27B (157,696, base) — which is still ~165×, or roughly 2.2 orders of magnitude. The four-order claim is accurate for the full set of publicly released checkpoints studied, but practitioners should be aware that instruction tuning is confounded with architecture in the extreme-high-end measurements.

The second nuance is that the corpus is restricted to up to 4,096 tokens. If activation peaks grow with sequence length — a plausible hypothesis given that attention sinks accumulate mass over more tokens — then the maximum peaks at 32K or 128K contexts could be larger than those reported here. The range across families might compress or expand at longer contexts. The paper acknowledges this gap (Section D.3) but does not quantify its potential impact.

Claim: "MoE checkpoints in the 10²–10³ range and Gemma3-27B-it reaching ~7×10⁵"

The specific numbers are supported by Figure 6 and the matched-pair comparisons. Qwen3.5-35B-A3B peaks at 132 (10² range), Qwen3-30B-A3B peaks at 1,512 (10³ range), and Gemma3-27B-it peaks at 696,320 (~7×10⁵). The contrast is real and dramatic.

However, the framing "MoE checkpoints in the 10²–10³ range" may overgeneralize from the two measured MoE checkpoints. Both come from Qwen families (Qwen3 and Qwen3.5), which have systematically different baseline activation scales (Qwen3 dense peaks at ~35K, Qwen3.5 dense peaks at ~167). It is possible that a MoE checkpoint from Gemma3 — if one existed at comparable scale — would show peaks in the 10⁴–10⁵ range, not the 10²–10³ range, because the Gemma baseline is so much higher. The claim is accurate as a report of what was measured but should not be interpreted as "MoE architecture universally produces peaks in the 10²–10³ range regardless of family." The MoE suppression factor (14–23×) is the more generalizable finding; the absolute peak magnitude reflects both the suppression factor and the family's baseline activation scale.

Claim: "Cross-family and cross-generation comparisons break simple monotonic scaling"

This is the paper's most robustly supported claim, with evidence from multiple angles. Figure 5 shows that within-family scaling is roughly monotonic (with Gemma2 as a qualified exception) but produces much smaller variation than between-family differences at fixed size. Figure 6 shows that models of similar parameter count can differ by orders of magnitude in M, directly refuting any universal "larger model = larger activations" rule. Appendix Figure 7 shows that generational trends within the same family are highly non-monotonic and family-dependent — Qwen goes up then down, Gemma goes up sharply. The claim is conservative in its phrasing ("break simple monotonic scaling") and is amply supported.

A missing experiment that would strengthen this claim: showing that the non-monotonicity holds when controlling for training data. The Qwen generations (2.5, 3, 3.5) were trained on different data mixtures with different recipes, so the activation differences could reflect training-data changes rather than architectural evolution. This doesn't weaken the practical claim — a practitioner choosing between Qwen2.5 and Qwen3.5 still faces the 200× peak difference regardless of its cause — but it limits the explanatory power of "generation" as an independent variable.

Claim: "MoE checkpoints exhibit 14.0–23.4× lower peaks than matched-scale dense counterparts"

The claim is directly supported by the two matched pairs in Figure 8, but the evidence base is thin (n = 2 pairs from related families) and observational rather than causal. The suppression factors (23.4× for Qwen3-30B-A3B vs. Qwen3-32B, 14.0× for Qwen3.5-35B-A3B vs. Qwen3.5-27B) are large and consistent in direction, but:

There are only two pairs. This is insufficient for statistical inference about the population of all MoE models. A third pair from a different family (e.g., if DeepSeek-V2 or Mixtral checkpoints were measured) would substantially strengthen the generalization.
The pairs come from related families (Qwen3 and Qwen3.5), which share architectural lineage. The suppression might be specific to Qwen's MoE implementation (routing strategy, expert count, load-balancing loss) and not generalize to other MoE designs.
Training recipes are not controlled. The dense and MoE checkpoints within each family may differ in training data, optimization hyperparameters, or training duration in ways that affect activation magnitude independently of the dense/MoE axis.
The Qwen3.5-35B-A3B vs. Qwen3.5-27B comparison pairs different total parameter counts (35B vs. 27B), so the reduction factor of 14.0× confounds the MoE effect with a 1.3× parameter-count difference.

The claim is best read as "in the two cases we measured, MoE models showed 14–23× lower peaks than their dense counterparts" rather than "MoE architecture reduces peaks by 14–23×." The paper's own caveat — "this contrast is observational rather than causal" — is appropriate.

Claim: "The residual stream carries the global maximum in 22/24 checkpoints"

This claim is straightforwardly supported by the data in Section 4.3 and Figure 15. The claim is an empirical regularity report: "we observed this pattern in 22 of 24 cases." The two exceptions (GPT-OSS-20B, Qwen3.5-0.8B) are identified and discussed. The claim does not assert that the residual stream must carry the global maximum in all models, only that it does in nearly all measured models — a conservative framing that matches the evidence.

A potential concern is that the paper does not report how much larger the hidden-state peaks are than the attention or MLP peaks in the 22 cases where hidden states dominate. If the gap is small (e.g., hidden state at 1,000 vs. MLP at 950), the residual-stream dominance claim is less actionable than if the gap is large (e.g., hidden state at 10,000 vs. MLP at 100). Figure 15 provides qualitative evidence that the gap is large (1–2 orders of magnitude) for high-peak models, but the quantitative gap sizes are not systematically reported for all checkpoints.

Claim: "A lightweight INT-8 sanity check shows that measured maxima co-vary with low-bit reconstruction error via activation-scale selection"

The claim is supported directionally but is not a calibrated dose-response relationship. Figure 12 shows that low-M models (Qwen3.5-0.8B, M=122) achieve high SQNR (~29 dB) while high-M models (Gemma3-4B-it, M≈246K) achieve low SQNR (~10–14 dB). The correlation is in the expected direction.

However, the evidence is limited in several ways:

Only 8 of 24 checkpoints are tested, and the selection is not random — it deliberately spans the observed M range. A full 24-checkpoint regression of SQNR against log₁₀ M would be needed to establish a calibrated relationship.
Only the peak hidden layer is quantized. In a full deployment, all layers are quantized, and the SQNR bottleneck may be at a different layer than the one carrying the global peak. The relationship between global M and end-to-end quantization quality is not tested.
Only two simple per-tensor recipes are tested (max-abs and 99.9% clipping). Advanced quantization methods (SmoothQuant, rotation-based methods, per-channel quantization) are designed to reduce the M-dependence of SQNR specifically. The paper's claim is about the relationship between M and reconstruction error under conditions where M directly determines the scale factor — which is true for the tested recipes — but does not generalize to all quantization strategies.
The relationship is not perfectly monotonic. Qwen3.5-35B-A3B (M=132) shows lower SQNR than Qwen3.5-0.8B (M=122) despite having a similar M, indicating that "within-layer distributional structure beyond peak magnitude also affects quantization error" (Appendix D.2). This means M is not a complete predictor of SQNR even for the simple recipes tested.

The paper's modest framing — calling this a "sanity check" rather than a "calibrated dose–response curve" — accurately reflects the strength of the evidence. The claim as stated ("measured maxima co-vary with low-bit reconstruction error") is supported; a stronger claim ("M predicts SQNR with high accuracy") would not be.

What's Missing: Experiments That Would Have Strengthened the Paper

Context-length scaling. All measurements are at ≤4,096 tokens. Several measured checkpoints support 32K–128K contexts. Measuring at least one checkpoint at 8K, 16K, and 32K would establish whether peaks grow, saturate, or shift carriers at longer contexts — information directly relevant to deployment at modern context lengths.

More matched MoE pairs. The MoE suppression finding is striking but rests on only 2 pairs from related families. Including DeepSeek-V2-Lite (dense vs. MoE at matched scale) or Mixtral-8×7B vs. a same-scale dense counterpart would substantially strengthen the generalization.

A third family for the VL-vs-text comparison. The Qwen2.5-VL comparison provides n = 2 pairs from a single family. Measuring a vision-language variant from another family (e.g., if LLaVA or InternVL checkpoints could be compared to their text backbones) would test whether VL adaptation consistently reduces peaks by ~1.5× or whether the effect is Qwen-specific.

Full-model INT-8 quantization. The sanity check is limited to one layer. Quantizing all layers for at least one low-M and one high-M checkpoint and reporting end-to-end perplexity on held-out text would provide much stronger deployment validation than a single-layer SQNR measurement.

Corpus ablation. The paper validates peak stability under subsampling of the fixed corpus but not sensitivity to corpus composition. Measuring peaks on a purely code corpus, a purely mathematical corpus, and a purely natural-language corpus for a subset of checkpoints would establish whether the cross-family ordering (Qwen3.5 < Qwen2.5 < Gemma3) is stable across domains or whether domain-specific effects could reverse the ordering.

Causal intervention. The paper is explicitly observational and stops short of causal claims, which is methodologically appropriate for a survey paper. However, even a single controlled experiment — e.g., comparing the same Qwen2.5 architecture trained with vs. without attention sinks on BOS tokens, or with vs. without SwiGLU gating — would transform the matched-design comparisons from associations into at least plausibly causal evidence.

Overall Assessment

The experiments genuinely support the paper's descriptive claims: maximum activation magnitudes vary enormously across modern open LLM families, the residual stream is the dominant carrier of these extreme values, and the variation is structured by family, architecture, and training stage rather than by parameter count alone. These findings are well-supported by the unified measurement protocol, the 27-checkpoint coverage, and the matched-design comparisons.

The paper's prescriptive claims — that M should be reported alongside open-weight releases, and that M is a valid prior for quantization risk — are supported directionally by the INT-8 sanity check but are not exhaustively validated. The relationship between M and end-to-end quantization quality under advanced mitigation strategies remains untested, and the paper acknowledges this. The value of M as a model-card statistic is established in principle (it is measurable, it varies substantially, and it correlates with reconstruction error under simple quantization recipes); whether it provides sufficient information to guide quantization decisions without additional per-model profiling is a question the paper does not answer and that likely requires task-level evaluation with specific quantization methods.

The paper's primary strength is its systematic sweep across a diverse model landscape, revealing patterns that no prior study could have identified because no prior study measured activation peaks across this many families. Its primary limitation is the observational nature of the matched-design comparisons, which cannot distinguish causal factors (architecture, training recipe, data mixture) from correlated ones. The paper is transparent about this limitation and positions itself as providing the measurement infrastructure and empirical baselines that enable future causal and mechanistic work — a scope that is both appropriate and well-executed.

6. Limitations and Trade-offs

Scope Limited to Activation Magnitude Only — No Mechanistic Explanation, No Remediation, No Intervention

The assumption or constraint. The paper deliberately measures what the activation peaks are, not why they form or how to change them. It explicitly stops at measurement: "We deliberately stop short of causal claims about why families differ; explaining the mechanism behind these differences—and intervening on it—is left as future work" (Section F). All matched-design comparisons (MoE-vs-dense, Base-vs-Instruct, etc.) are "observational rather than causal" (Appendix C.1), meaning the paper identifies associations between model properties and peak magnitudes without establishing that changing the property would change the peak.

The consequence. A practitioner reading this paper can learn that Gemma3-27B-it has a peak ~4,000× larger than Qwen3.5-27B, but gains no actionable guidance on how to reduce that peak if they need to deploy Gemma3 in a low-bit setting. They don't know whether the peak is caused by the architecture (could be changed by switching from pre-norm to post-norm), the training recipe (could be mitigated by a different optimizer or data mixture), the instruction-tuning process (could be addressed by a different SFT strategy), or some interaction of all three. The paper's findings are therefore diagnostic but not prescriptive — it tells you that a problem exists, not how to fix it. For a deployment team facing a high-M model that they must use (e.g., because of benchmark performance requirements), the paper provides risk awareness but no mitigation strategy beyond the generic advice to apply existing quantization methods (which the paper itself does not evaluate).

What evidence exists in the paper. The matched-design comparisons in Section 5 and Appendix C all include explicit caveats about observational status. For MoE: "with only n = 2 matched pairs and possible unobserved differences in training recipe between paired checkpoints, this contrast is observational rather than causal" (Appendix C.1). For vision-language: "this is observational evidence based on n = 2 matched pairs within a single family" (Appendix C.2). For training stage: "training progress itself co-occurs with a gradual increase in extreme activation magnitudes" (Appendix C.4) — the word "co-occurs" signals association, not causation. The paper's own limitation discussion (Section D.3) lists "linking each failing checkpoint to specific normalization or attention dynamics" as future work.

Mitigation status. The paper does not attempt mitigation or intervention. It treats this as out of scope, positioning itself as providing the measurement baseline that future causal work can build on. The limitation is acknowledged and scoped explicitly, which is intellectually honest, but it means the paper's practical utility is limited to risk assessment — it cannot guide remediation.

The Difficulty Estimation Cost Is Unaccounted for in the Headline Claims — But Repurposed for This Paper, the Analogous Cost Is Corpus Construction and Measurement Overhead

The assumption or constraint. The paper's headline finding — that M varies by nearly four orders of magnitude across families — is produced by a measurement pipeline that requires (a) constructing a carefully balanced 5,000-sample multi-domain corpus, (b) running full forward inference on that corpus with hook instrumentation for each checkpoint, and (c) aggregating statistics over approximately 19.5M tokens per checkpoint. This cost is not included in any deployment-relevance analysis. The paper itself acknowledges that the measurement protocol is designed for survey research, not for lightweight deployment profiling: it uses "streaming GPU accumulation" and "family-specific tokenization" that would need to be replicated for any new model family not already in the study.

The consequence. The paper advocates for M to be "reported alongside any open-weight release as part of a deployment-oriented model card extension" (Section F). But it provides no guidance on how to compute M cheaply. A model developer wanting to report M for their new checkpoint would need to replicate the entire pipeline — corpus construction, hook instrumentation, 5,000-sample forward inference — which is a non-trivial engineering and compute investment. A practitioner wanting to verify M for a model they downloaded would face the same cost. The 5,000-sample corpus at up to 4,096 tokens per sample, processed through a ~30B parameter model, represents a substantial compute budget (roughly 19.5M tokens × ~60 GFLOPs/token ≈ 1.2 exaFLOPs for a 30B model, or approximately several GPU-hours on modern hardware). This cost is acceptable for a one-time survey paper but may be prohibitive for routine model-card reporting. The paper's peak stability experiment (Section 2.3) shows that the 5,000-sample scale is necessary for the peaks to stabilize (CV drops from 10.1% at 1,000 samples to 8.2% at 2,000 samples), so simply reducing sample count would degrade measurement reliability. The paper does not investigate whether a smaller, cheaper corpus could produce a useful approximation of M — for example, whether 100 samples of specifically chosen "high-activation" text (code, math, repetitive patterns) could estimate the 5,000-sample peak within a factor of 2×.

What evidence exists in the paper. The peak stability experiment (Section 2.3) is the only relevant evidence, and it addresses a different question (stability under subsampling) rather than measurement cost. The paper does not report the wall-clock time or FLOPs required per checkpoint, does not compare the full-corpus M to what would be obtained from a smaller corpus, and does not propose a lightweight M-estimation protocol. The gap is unaddressed rather than hidden — the paper simply does not engage with the cost question.

Mitigation status. Not addressed. The paper neither proposes a cheap estimator for M nor quantifies the cost of the full pipeline. Future work on "pretraining or finetuning models to directly predict difficulty of a question" (Section 8 of the original study was not provided, but the paper's limitation discussion in Section D.3 mentions future work on extending to longer contexts and broader domains rather than cheaper measurement) could potentially address this, but the current paper provides no path to low-cost M estimation.

Measurements Capped at 4,096 Tokens — No Evidence for Behavior at Modern Context Lengths (32K–128K Tokens)

The assumption or constraint. All measurements use sequences truncated to a maximum of 4,096 tokens (93% of samples at exactly 4,096, with the remaining 7% at 256, 512, 1,024, or 2,048 tokens). The paper explicitly acknowledges: "All measurements use sequences up to 4,096 tokens, so we cannot speak to whether maxima grow, saturate, or shift carriers under 32k–128k contexts that several of these checkpoints support" (Section D.3).

The consequence. The paper's claim that Gemma3-27B-it peaks at 696,320 is a lower bound on what that model might produce at longer contexts. If activation peaks grow with sequence length — a plausible hypothesis given that attention sinks accumulate mass on BOS tokens over more positions, and residual-stream spikes can compound over more layers with more token interactions — then the true peak at 32K or 128K could be substantially higher. The paper provides no evidence either way. For deployment purposes, this means that M measured at 4K tokens is not a safe upper bound for models deployed at longer contexts. A practitioner running Gemma3-27B-it at 32K context length cannot assume the peak stays at ~7×10⁵; it could be 2×, 5×, or 10× larger. This uncertainty directly affects quantization scale selection, numerical precision requirements, and clipping-strategy design.

The converse is also possible — peaks might saturate at moderate context lengths and not grow further. Attentional softmax saturation, layer normalization, or residual-stream dynamics could impose an effective ceiling. But the paper provides no data to distinguish these scenarios. The 4,096-token limit also means the paper cannot speak to whether peak carrier components shift at longer contexts — for example, whether MLP outputs become dominant when attention sinks saturate, or whether the residual stream's dominance (22/24 checkpoints) holds at 32K.

What evidence exists in the paper. None. The paper does not include any measurements beyond 4,096 tokens, does not extrapolate from shorter contexts to predict longer-context behavior, and does not reference any prior work that establishes the context-length scaling of activation peaks. This is a gap in coverage, not a flawed measurement — the paper is transparent about the limitation.

Mitigation status. Acknowledged but not addressed. Section D.3 lists context length as a limitation and suggests it as future work, but no experiments or analyses are provided. The practical consequence is that practitioners deploying these models at modern context lengths cannot rely on the reported M values as safe upper bounds.

The INT-8 Sanity Check Is Restricted to 8 Checkpoints, 1 Layer, and 2 Simple Recipes — Insufficient to Establish M as a Quantization Risk Predictor Across the Full Deployment Space

The assumption or constraint. The paper's deployment-validation experiment (Appendix D.2, Figure 12) quantizes only the peak hidden layer of 8 representative checkpoints using only two per-tensor symmetric INT-8 recipes (max-abs scaling and 99.9% clipping), with 128 calibration samples and 256 evaluation samples. This is explicitly a "sanity check" rather than "a calibrated dose–response curve" (Section D.2). The paper acknowledges that "end-to-end quantization should be evaluated with richer distributional and task-level measurements."

The consequence. The claim that "higher M correlates with lower SQNR" is supported directionally but cannot be generalized to the full deployment context for several reasons:

First, the layer-quantized is cherry-picked. The peak hidden layer is, by construction, the layer where M matters most for scale selection. Quantizing all layers and measuring end-to-end perplexity or task accuracy would provide a much more realistic assessment of how M translates into deployment degradation. It is possible that a model with a very high peak in one layer but well-behaved activations everywhere else suffers less overall degradation than a model with moderate peaks distributed across many layers.

Second, the quantization recipes are the simplest possible. Max-abs and 99.9% clipping are the most M-sensitive strategies by design. Modern deployment pipelines almost never use these strategies without additional mitigation. SmoothQuant, rotation-based methods (QuaRot, SpinQuant), per-channel quantization, and learned clipping thresholds all substantially reduce the dependence of SQNR on the raw activation maximum. The paper explicitly excludes these because they are "designed to neutralize M rather than to expose it" (Section D.2) — which is methodologically sound for a probe of the M–SQNR relationship, but means the probe does not represent realistic deployment conditions.

Third, only 8 of 24 checkpoints are tested. The selection spans the observed M range but is not random or stratified. A regression of SQNR against log M across all 24 checkpoints would establish whether M is a reliable predictor of quantization difficulty or whether within-layer distributional structure (which the paper notes affects Qwen3.5-9B and Qwen3.5-35B-A3B, which have low M but lower SQNR than Qwen3.5-0.8B) dominates in practice.

Fourth, SQNR is an intermediate metric, not a task metric. The paper measures signal-to-quantization-noise ratio in dB at the peak layer, not perplexity on held-out text or accuracy on a downstream task. SQNR is a reasonable proxy for reconstruction quality, but the relationship between per-layer SQNR and end-to-end task performance is known to be non-linear and architecture-dependent. A model could have low SQNR at one layer but still perform well on downstream tasks if subsequent layers are robust to that noise.

What evidence exists in the paper. Figure 12 and the surrounding discussion in Section D.2 establish a directional relationship but no more. The paper is transparent about the scope of this experiment, calling it a "lightweight INT-8 activation quantization sanity check" and a "deployment-relevance sanity check." It provides SQNR values for 8 models under 2 recipes and notes the approximate correlation with M. The 20 dB reference line in Figure 12 provides a rough quality threshold, but the paper does not claim that models below 20 dB are unusable or that models above 20 dB are safe to quantize.

Mitigation status. The paper explicitly frames this as a sanity check rather than a complete validation and lists "end-to-end quantized tasks" and "a 24-point regression of SQNR against log10 M" as future work (Section D.3). The limitation is acknowledged and scoped, but it means the paper's central prescriptive claim — that M should be reported as a deployment-oriented model-card statistic — rests on a plausibility argument (M affects scale selection, scale selection affects SQNR, therefore M matters) rather than on demonstrated end-to-end deployment impact.

All Measurements Are on a Single Evaluation Corpus, Single Modality (Text), and Single Language-Domain Mix — No Evidence for Generalization Across Deployment Distributions

The assumption or constraint. The entire paper's findings are conditioned on a single 5,000-sample multi-domain corpus with a specific composition: 17% each math/science, code, web text, and knowledge; 8% Chinese; 6% low-resource languages; 18% additional English/mixed web. The paper acknowledges that the corpus "is restricted to English, Chinese, and code domains" and that "activations on long-tail languages, mathematical reasoning chains, or tool-use traces may differ" (Section D.3). The corpus was constructed for balanced coverage, not for representativeness of any specific deployment distribution.

The consequence. The paper's reported M values are estimates of what a model produces on this specific corpus, not universal model properties. If a practitioner deploys Qwen2.5-7B (reported M ≈ 13K) primarily on a code-completion task, the true peak on their code-heavy distribution could be substantially different from the reported value. The paper's domain-balancing strategy reduces the risk that a single domain dominates the peak, but it also means the reported M is an average-of-domains statistic that may not match any real deployment distribution.

More critically, the paper does not test whether the ordering of models by M is stable across domains. A practitioner choosing between Qwen3.5-9B (M ≈ 956 on the balanced corpus) and Gemma2-9B (M ≈ 1,656 on the balanced corpus) might reasonably prefer Qwen3.5-9B for low-bit deployment because its reported peak is lower. But if Qwen3.5-9B produces disproportionately large activations on code while Gemma2-9B does not, the ordering could reverse on a code-heavy deployment distribution. The paper provides no domain-disaggregated peak measurements that would reveal such reversals.

The same concern applies to text types not represented in the corpus at all — highly repetitive text, text with unusual formatting (tables, ASCII art, LaTeX), text in languages not covered (Japanese, Arabic, Hindi), or structured inputs like JSON or YAML. The paper's corpus includes code and web text but does not explicitly cover these edge cases, which could trigger activation patterns not seen in the 5,000 samples.

What evidence exists in the paper. The peak stability experiment (Section 2.3) validates that M is stable under resampling of the same corpus, not that M is stable under domain shifts. The corpus construction details (Section 2.1) describe the domain buckets and length distribution but provide no analysis of per-domain activation peaks. The paper does not report, for example, that 90% of the global maximum comes from code samples or that math samples produce consistently higher peaks than web text. Without this decomposition, the reader cannot assess whether the reported M is representative or domain-specific.

Mitigation status. The paper acknowledges the corpus limitation (Section D.3) but does not address it through experiments. A domain-disaggregated analysis (even just reporting per-domain maxima for a subset of checkpoints) would substantially strengthen confidence that the cross-family ordering is stable. The paper treats this as future work.

The Observational Matched-Design Comparisons Confound Architecture, Training Recipe, and Data Mixture — Precluding Attribution of Peak Differences to Specific Model Properties

The assumption or constraint. The paper's central analytical strategy — matched-design comparisons that "fix" one factor while varying another — assumes that the paired checkpoints differ only in the factor of interest. The paper repeatedly acknowledges that this assumption is not satisfied. For the MoE-vs-dense comparison: "with only n = 2 matched pairs and possible unobserved differences in training recipe between paired checkpoints, this contrast is observational rather than causal" (Appendix C.1). For vision-language: "modality adaptation and training-recipe changes co-occur" (Appendix C.2). For the Base-vs-Instruct comparison: the instruct variants "conflate supervised fine-tuning with downstream RLHF or DPO stages whose training data we do not control" (Section D.3). For generational comparisons: Qwen2.5, Qwen3, and Qwen3.5 differ simultaneously in architecture, training data, and optimization — there is no "all else equal" generational change.

The consequence. None of the paper's matched-design contrasts support causal attribution. Consider the most dramatic finding — the Qwen3→Qwen3.5 peak suppression of ~200× at the ~30B scale. The paper can report that Qwen3.5-27B (M=167) has much lower peaks than Qwen3-32B (M=35,328), but it cannot say why. Possible explanations include: architectural changes in Qwen3.5's normalization stack, different training data that doesn't produce attention-sink dynamics, different optimizer hyperparameters that suppress activation growth, different model scale (27B vs. 32B), or some interaction. A deployment team considering whether to adopt Qwen3.5 specifically for its low activation peaks cannot know whether this property will persist in future Qwen releases, because they don't know what caused it.

The MoE suppression finding faces the same attribution problem. Qwen3-30B-A3B and Qwen3-32B differ in architecture (MoE vs. dense) but may also differ in training data mixture (MoE models often use different data sampling strategies), training duration, or optimization hyperparameters. The 14–23× suppression could be partially or entirely due to these confounds rather than the sparse routing mechanism itself. The paper's language is appropriately cautious ("co-occurs with," "is consistent with," "observational rather than causal"), but the practical implication is that the matched-design comparisons are more useful as hypothesis generation than as decision-making evidence.

The Ling-mini training-stage comparison (Appendix C.4, Figure 11) comes closest to isolating a single factor — it measures checkpoints from the same training run at different token counts — but even here, the training data distribution changes over the course of training (later stages may use different data mixtures or annealing schedules), and the model architecture may be constant but the effective architecture changes as parameters converge.

What evidence exists in the paper. The caveats are stated explicitly for each matched comparison, and Section D.3 reiterates that "these results reveal stable empirical differences but do not by themselves establish causal training mechanisms." The paper's transparency is a strength — it does not overclaim — but it means the actionable content of the matched-design analyses is "these things vary together" rather than "changing X will change M by factor Y."

Mitigation status. The paper does not attempt causal interventions (e.g., training two models with identical data but different architectures, or taking a single checkpoint and comparing its activations before and after applying a specific architectural modification). It identifies causal analysis as future work and positions its contribution as the measurement baseline that enables that future work. This is intellectually honest but limits the paper's prescriptive value for practitioners making architecture or training decisions.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper changes the conversation around LLM deployment by establishing maximum activation magnitude as a first-class model property — something that should be measured, reported, and factored into deployment decisions alongside parameter count, context length, and benchmark scores. This is a methodological reframing with downstream practical force, not a paradigm shift in our understanding of how LLMs work internally. The conceptual move is clean and well-motivated: prior work asked "does this model have massive activations?" (a binary diagnostic inherited from interpretability research) while this paper asks "how large can activations get, and where do they live?" (a continuous measurement oriented toward deployment risk). The difference is not semantic — it produces different practical conclusions, as demonstrated by Qwen2.5-1.5B failing the Sun binary criterion (local ratio only ~574) despite having a peak of 7,968 that would dominate a per-tensor quantization scale, and Qwen3.5-9B passing the Sun criterion despite having a peak of only 956 that poses minimal quantization risk (Section 3.2, Figure 2).

The most landscape-altering empirical finding is the nearly four-order-of-magnitude variation in M at comparable parameter counts (Section 5.2, Figure 6). At the ~27–35B scale, Qwen3.5-27B peaks at 167 while Gemma3-27B-it peaks at 696,320 — a difference of roughly 4,000×. This directly refutes the implicit assumption that activation range scales monotonically with model size, which prior work on LLaMA-derivative families (Dettmers et al., 2022; Sun et al., 2024) had no reason to question because they studied a narrow model set. The finding has immediate practical impact: a deployment team choosing between two ~30B models for INT-8 quantization cannot rely on parameter count as a proxy for activation range — one model may fit comfortably in INT-8 with simple max-abs scaling while the other requires advanced mitigation just to avoid catastrophic reconstruction error. This is a negative result with positive consequences: it tells the field what doesn't work (parameter-count-based heuristics) and what does work (direct measurement), redirecting deployment effort from extrapolation to measurement.

The paper also reconciles a latent tension between the interpretability and quantization literatures. Prior work treated massive activations simultaneously as "functional model components" that serve useful roles (attention sinks for no-op routing, Bondarenko et al., 2023; Gu et al., 2025) and as "deployment obstacles" to be eliminated by quantization methods (SmoothQuant, AWQ, rotation-based approaches). The paper shows these perspectives are complementary rather than contradictory by demonstrating that activation magnitude is continuous, not binary — models can have activations that are large enough to matter for quantization without satisfying the local-sparsity criterion that mechanistic work uses to identify functionally interesting coordinates. A model can be "well-behaved" by the Sun criterion (Qwen2.5-1.5B, peak 7,968) yet still pose quantization challenges, or "pathological" by the Sun criterion (Qwen3.5-9B, peak 956) yet be trivially quantizable. The resolution is that the binary criterion answers a mechanistic question (are there structurally interesting outlier coordinates?) while the continuous magnitude answers a deployment question (what numerical range must the system accommodate?). Both questions are valid; they are simply different.

Several research directions become more attractive in light of this work. Family-level architecture comparisons become a priority for understanding activation dynamics, because the paper shows that family identity explains far more variance in M than parameter count — the Qwen3.5 family systematically suppresses peaks across all sizes, the Gemma3 family amplifies them, and these effects dwarf within-family scaling. This suggests that architectural details (normalization stack design, SwiGLU gating, residual connection structure, attention implementation) are the primary levers controlling activation range, making them a high-value target for mechanistic investigation. MoE architecture attracts new interest as an activation-range mitigation strategy, since the paper's 14–23× suppression factor in matched pairs (Figure 8) is large enough to influence architecture selection for deployment — if MoE routing reduces activation peaks by an order of magnitude, it may enable quantization bit widths that are infeasible for same-scale dense models. Training-stage dynamics become relevant for activation prediction, since the Ling-mini checkpoints show that M increases monotonically with training progress (1.34× from 5T to 20T tokens, Figure 11), suggesting that any model's activation range at release is a snapshot of an evolving property, not a static architecture-determined constant.

Conversely, some research directions become less attractive or require reframing. The search for a universal "activation scaling law" (analogous to Chinchilla scaling laws for pretraining compute) is undercut by the finding that cross-family variation spans orders of magnitude while within-family scaling is relatively modest — any such law would be family-specific to the point of being a restatement of measurement rather than a predictive tool. Binary massive-activation diagnostics (Sun et al., 2024) become less useful as standalone deployment signals, since the paper demonstrates that they can misorder models by quantization risk — the binary criterion is best treated as a mechanistic diagnostic to be used alongside, not instead of, continuous magnitude measurement. One-size-fits-all quantization recipes calibrated on a single model family become suspect, since hyperparameters that work for Qwen3.5-27B (peak 167) would be wildly inappropriate for Gemma3-27B-it (peak 696,320) — calibration must be per-family or per-checkpoint, not per-scale.

Follow-Up Research This Work Enables

Cheap M estimation for routine model-card reporting. The paper's measurement protocol (5,000-sample multi-domain corpus, full forward inference with hooks, streaming accumulation) produces reliable M values but is too expensive for routine use by model developers releasing checkpoints. The critical practical gap is: can M be estimated from a much smaller, targeted corpus — perhaps 100–200 samples of text types known to trigger high activations (code with unusual indentation, math with large numbers, repetitive boilerplate) — and still approximate the 5,000-sample M within a factor of 2×? A strong follow-up would take the 24 checkpoints measured in this paper, construct progressively smaller corpora (5,000 → 2,000 → 500 → 100 samples) with domain-stratified sampling, measure M on each, and report the multiplicative error as a function of corpus size. If 200 well-chosen samples recover M within 1.5× of the full-corpus value across all families, the measurement becomes practical for model cards. If 2,000 samples are needed, the cost remains prohibitive and alternative approaches (training a lightweight activation-range predictor from architecture metadata) become the priority.

Context-length scaling of M from 4K to 128K tokens. The paper's measurements are capped at 4,096 tokens, but modern open models routinely support 32K–128K contexts. The open question is whether M grows, saturates, or shifts carriers as context length increases — and whether the answer is family-dependent. A direct follow-up would take 3–4 checkpoints spanning the observed M range (e.g., Qwen3.5-27B at ~167, Qwen2.5-7B at ~13K, Gemma3-4B-it at ~246K) and measure M at context lengths of 4K, 8K, 16K, 32K, and (if feasible) 64K–128K on a fixed set of long documents (books, long code files). The key measurements are: (a) does M grow sublinearly, linearly, or not at all with context length? (b) does the residual stream remain the dominant carrier, or do attention outputs or MLP outputs become competitive at long contexts? (c) does the cross-family ordering (Qwen3.5 < Qwen2.5 < Gemma) hold at all context lengths, or do some families scale more aggressively with length? If M saturates by 16K, the 4K measurements become safe lower bounds; if M grows with log(context length), extrapolation becomes possible; if M shows family-specific scaling with length, deployment recommendations become context-length-dependent.

Causal intervention on activation peaks via controlled training experiments. The paper's matched-design comparisons are observational — they identify associations between model properties and M but cannot establish causation. A causal follow-up would train small models (e.g., ~100M–300M parameters) under controlled conditions, varying exactly one factor while holding all others identical: (a) pre-norm vs. post-norm transformer architecture with identical initialization and data, (b) standard MLP vs. SwiGLU gating with matched parameter counts, (c) standard attention vs. attention with explicit BOS-token attention sink suppression, (d) different weight initialization scales. The experiment would measure M at multiple checkpoints during training (every 1B tokens) to trace how the peak evolves and whether architectural differences produce immediate or gradual divergence. This would convert the paper's associative findings (e.g., "Qwen3.5 family suppresses peaks") into causal knowledge (e.g., "switching from pre-norm to architecture variant X reduces M by factor Y"), enabling prescriptive architecture guidance.

Domain-disaggregated activation peaks to test cross-domain ordering stability. The paper's unified corpus is domain-balanced but reports only aggregate M. A practitioner deploying a model primarily on code or mathematical text needs to know whether the cross-family ordering (Qwen3.5 < Qwen2.5 < Gemma) holds within their domain. A follow-up would take the 5,000-sample corpus and report per-domain M for each checkpoint: what is the peak on the 850 math/science samples alone? On the 850 code samples alone? On the 850 web text samples alone? The key question is whether any checkpoint's M is dominated by a single domain (suggesting that the aggregate M overstates risk for other domains) and whether any domain-specific ordering reversals occur (e.g., Qwen2.5 producing larger peaks than Gemma on code despite the aggregate ordering being reversed). If domain-specific orderings differ from the aggregate, deployment recommendations become domain-conditional; if orderings are stable, the aggregate M is a sufficient prior.

Full-model INT-8 quantization with end-to-end perplexity and task evaluation. The paper's INT-8 sanity check quantizes only the peak hidden layer of 8 checkpoints and measures SQNR, not task performance. A deployment-validation follow-up would apply per-tensor INT-8 quantization (max-abs and 99.9% clipping) to all linear layers of 4–6 checkpoints spanning the M range, measure perplexity on held-out text, and evaluate on a standard benchmark (e.g., MMLU or GSM8K for the Qwen/Gemma families) to establish whether the M→SQNR relationship at the peak layer translates into M→task-degradation relationship end-to-end. The experiment would also compare against a strong quantization baseline (e.g., SmoothQuant with default hyperparameters, or AWQ) to test whether advanced mitigations eliminate the M-dependence of task performance or merely reduce it. If models with high M (Gemma3-4B-it) show larger perplexity degradation and task-accuracy drops than models with low M (Qwen3.5-0.8B) even under advanced mitigation, then M is a robust prior for quantization difficulty; if advanced mitigation equalizes performance across the M range, then M is a useful signal for simple quantization but not for state-of-the-art deployment pipelines.

MoE activation peak suppression across diverse families to test generalization. The paper's finding that MoE reduces peaks by 14–23× rests on only 2 matched pairs from Qwen families. A generalization follow-up would measure M on MoE checkpoints from other families — DeepSeek-V2-Lite (dense vs. MoE at matched scale), Mixtral-8×7B vs. a same-scale dense Mistral variant, or a Qwen2.5-MoE variant if released — and compute the suppression factor for each pair. The experiment would test whether the 14–23× factor is Qwen-specific (reflecting Qwen's routing strategy or expert structure) or a broader regularity of MoE architectures. If the factor is consistent across families (e.g., always 10–25×), it becomes an architectural design principle for activation-range-conscious model development; if it varies widely (e.g., 2× in one family, 50× in another), then the suppression is mediated by implementation details rather than being a generic MoE property.

Practical Applications and Downstream Use Cases

Model selection for low-bit deployment with activation-range awareness. A deployment team choosing between open-weight models for an INT-8 or INT-4 inference pipeline can use this paper's reported M values as a first-pass filter. If the target deployment uses simple per-tensor quantization (max-abs scaling), models in the T1–T2 tiers (M < 10K: Qwen3.5 family, Qwen3-30B-A3B, Qwen2.5-1.5B, Qwen2.5-VL-3B, Gemma2-2B/9B, Ling-mini checkpoints) are substantially lower risk than models in T4–T5 tiers (M > 50K: Gemma2-27B, Gemma3-4B-it, Gemma3-27B-it). The specific gap is quantified by the INT-8 sanity check (Figure 12): Qwen3.5-0.8B (M=122, T1) achieves 29.1 dB SQNR under max-abs scaling while Gemma3-4B-it (M≈246K, T5) achieves ~10–14 dB — a difference that would produce visibly degraded output quality without additional mitigation. A team that must use Gemma3 for benchmark performance reasons can plan in advance to budget engineering time for SmoothQuant, rotation-based quantization, or per-channel scaling rather than discovering the problem during integration testing.

Activation-scale calibration for custom inference runtimes. Many production inference systems (vLLM, TensorRT-LLM, custom CUDA kernels) require per-model calibration of activation scales for efficient memory allocation and numerical stability. The paper's per-model statistics (layerwise maxima, carrier components, peak depth) provide a starting point for this calibration that is more informative than the default "run a few hundred samples and pray." Specifically, the finding that the residual stream carries the global maximum in 22/24 checkpoints (Section 4.3) tells runtime developers to focus scale-calibration effort on hidden-state tensors rather than attention outputs or MLP outputs. The layerwise heatmap (Figure 3) shows where the peaks live — shallow, middle, or deep layers — enabling targeted calibration of the most vulnerable layers without scanning all layers equally. The jump-and-plateau vs. gradual-accumulation dichotomy (Figure 4) indicates whether peaks are concentrated in a narrow depth band (requiring high-precision accumulation only in that band) or distributed broadly (requiring high precision throughout). For a runtime that supports mixed-precision inference, this enables informed decisions about which layers can safely use lower precision.

Architecture selection for resource-constrained training of new models. Teams developing new LLMs from scratch or via continued pretraining can use the paper's matched-design findings as priors for architecture decisions affecting deployment feasibility. If the target deployment involves low-bit inference on edge devices, the paper suggests: (a) MoE architecture is likely to produce lower activation peaks than dense architecture at matched total parameter count (14–23× reduction observed in Qwen families, Figure 8), potentially enabling more aggressive quantization; (b) Qwen3.5-style architectural choices (the specific normalization and gating variants that produce systematically suppressed peaks across all sizes, from 0.8B to 27B) are worth investigating and replicating rather than defaulting to LLaMA-style pre-norm stacks; (c) instruction tuning does not fundamentally restructure activation peaks — it compresses late layers while preserving mid-layer high-peak regions (Appendix C.3) — so activation-range estimates from base model measurements remain informative for instruct variants at the same backbone. None of these are guaranteed (the paper is observational, not causal), but they provide evidence-based starting points for architecture ablation studies that would otherwise require blind exploration.

Model card extension for open-weight releases. The paper advocates for reporting M alongside model weights as a deployment-oriented model card entry. Concretely, a model card for a new release would include: the global M across all layers and components, the component class carrying M (almost always hidden states; flag if MLP or final norm), the normalized depth of the peak layer, the deployment tier (T1–T5 based on Figure 13), and the INT-8 SQNR at the peak layer under max-abs scaling as a reference point. This adds minimal overhead to the release process if M is measured once during pre-release evaluation and would give downstream practitioners an instant, standardized signal for activation-range risk. The paper's open-source measurement code (https://github.com/clx1415926/Max_act_llm) and per-checkpoint statistics lower the barrier to adoption — model developers can run the pipeline on their own checkpoints and include the resulting M value in their release documentation without developing custom measurement infrastructure. The practical adoption question is whether major open-weight releases (Meta, Mistral, Qwen, DeepSeek, Google) incorporate this or an equivalent metric into their model cards — the paper makes the case for why they should, but institutional adoption depends on factors beyond technical merit.