Inference Optimization2023~126 minarXiv:2309.14592

Efficient Post-training Quantization with FP8 Formats

score 9Haihao Shen, Naveen Mellempudi, Xin He +3↑ 11Sep 26, 2023cs.LG

🎯 Pitch

Skipping INT8 for FP8 quantization lifts the workload pass rate from 65.87% to 92.64% across 75 diverse neural network architectures, effectively solving the one-in-three failure problem that plagues standard integer quantization. Even LayerNorm and other notoriously outlier-prone operations quantize cleanly with FP8, and simply matching the format to your domain—E4M3 for language, E3M4 for vision—gives you near-lossless 8-bit inference out of the box.

1. Executive Summary

This paper studies the advantages of FP8 floating-point data formats for post-training quantization across 75 unique network architectures spanning NLP, computer vision, audio, and recommendation tasks. The authors examine three FP8 representations — E5M2, E4M3, and E3M4 — analyzing how the trade-off between dynamic range (exponent bits) and precision (mantissa bits) affects model accuracy, and introduce a quantization workflow comprising a standard scheme (per-channel weight scaling, per-tensor activation scaling) and an extended scheme (mixed FP8 formats, expanded operator coverage including LayerNorm and BatchNorm, dynamic quantization). The headline result is that FP8 formats achieve substantially higher workload pass rate than INT8 — 92.64% vs. 65.87% at a 1% relative accuracy loss threshold — with E4M3 reaching 96.32% coverage on NLP models, while also enabling quantization of previously intractable operations like LayerNorm without accuracy degradation. The study further establishes that E4M3 is better suited for NLP models whereas E3M4 performs marginally better on computer vision tasks, with the boundary condition that these domain-specific format preferences emerge only when the quantization scheme is adapted per-architecture through the extended tuning workflow.

2. Context and Motivation

The Core Problem: INT8 Quantization Is the Industry Standard, but It Fails on a Large Fraction of Modern Workloads

The fundamental problem this paper addresses is that INT8 quantization — the de facto standard for efficient inference — does not work well on a substantial and growing fraction of deep learning models. The paper reports that INT8 achieves only a 65.87% pass rate (defined as maintaining accuracy within 1% of the FP32 baseline) across the 75 diverse network architectures they tested (Table 2). This means roughly one in three models cannot be reliably quantized to INT8 without meaningful accuracy degradation. For production deployments that depend on quantization for cost and latency reasons, this represents a serious practical bottleneck.

The root cause of INT8's limitations is structural, not incidental. INT8 is a uniform quantization scheme: it divides the representable range into 256 equally-spaced buckets. The step size between adjacent representable values is fixed and determined by the ratio of the data's dynamic range to the number of quantization levels. This creates two interrelated failure modes that the paper identifies (Section 1, Section 2):

First, limited dynamic range. INT8 can only represent values spanning a factor of 256 (approximately 8 bits of range) at whatever resolution its fixed step size provides. When the values in a tensor span a wider range — which happens frequently in activations, particularly in transformer architectures — INT8 must either clip extreme values (losing information about outliers) or stretch its step size so wide that small values are poorly represented. Neither option is satisfactory. The paper's Figure 1 illustrates this with a toy example: a normal distribution with a small fraction of outliers. INT8's uniform grid is forced to span the full range including outliers, resulting in coarse representation of the high-density region near zero where most values actually lie.

Second, outlier sensitivity. The paper emphasizes that large language models (LLMs) in particular exhibit activation outlier patterns that are fundamentally hostile to uniform quantization. These outliers are "significantly larger when compared to the rest of the activations" (Section 1), and their presence has been shown to be amplified by LayerNorm operations (Wei et al., 2022). A small number of extreme values can dictate the step size for the entire tensor, degrading the representation quality of the vast majority of values. Prior mitigations — clipping thresholds learned through calibration or training, mathematical transformations that redistribute magnitude between weights and activations (SmoothQuant, Xiao et al., 2022; Outlier Suppression, Wei et al., 2022) — are patches that work in some cases but do not address the underlying limitation of the uniform format.

Third, inapplicability to certain operator types. The paper notes (Section 3.2) that operations like LayerNorm, BatchNorm, and element-wise operations (Add, Mul) are memory-bound and consume significant execution time, yet prior attempts to quantize them using integer approximation were "unsuccessful in maintaining the model accuracy" (Bhandare et al., 2019; Kim et al., 2021). This forces these operators to remain in higher precision, limiting the end-to-end speedup achievable through quantization and adding data format conversion overhead at operator boundaries.

Why This Problem Is Important

The poor pass rate of INT8 is not merely an academic concern — it has direct and growing consequences for how neural networks are deployed in production.

The shift toward large language models. The last major era of quantization research was driven by computer vision workloads, where INT8 proved highly effective. Convolutional networks with ReLU activations produce well-behaved tensor distributions that are amenable to uniform quantization. But the field has shifted dramatically toward transformer-based architectures — first in NLP, increasingly in vision (ViT), and now in multimodal models and diffusion models — where activation patterns are fundamentally different. The paper explicitly evaluates both traditional CNNs and modern transformers, and the results (Table 2, per-category breakdown) show INT8 achieving only 57.89% pass rate on CV workloads and 67.65% on NLP workloads, making clear that the problem spans both domains but is more acute where transformers dominate.

The economics of inference. For data center deployments serving millions of queries, the cost difference between running a model in FP16/FP32 versus INT8 is enormous — typically 2–4× in throughput and proportional savings in total cost of ownership. When INT8 is viable, it is the clear economic choice. When it fails and the model must be run in higher precision, the cost penalty is severe. A pass rate of 65.87% means that for roughly 35% of models, the deployment team must either accept accuracy degradation, pay the higher precision premium, or invest engineering effort in bespoke quantization solutions — which is itself expensive and may not succeed. For LLMs in particular, where inference costs dominate the total deployment budget, this is an increasingly urgent problem.

The expanding universe of model architectures. The paper evaluates 75 unique architectures spanning text classification, generative language modeling, summarization, machine translation, image classification, image generation (Stable Diffusion), image segmentation, object detection, speech recognition, and recommendation systems (Section 4.1). This diversity is deliberate: it reflects the reality that production ML systems today deploy a heterogeneous mix of model types. A quantization scheme that works for ResNet-50 but fails for BERT, or works for BERT but fails for Stable Diffusion, requires separate deployment pipelines and complicates hardware design. The paper's ambition is to find a unified quantization approach that generalizes across this diversity.

Hardware support is emerging for FP8. The paper is not proposing FP8 in a vacuum. At the time of writing, hardware vendors were actively developing native FP8 support — NVIDIA's H100 GPU includes FP8 tensor core instructions, and Intel's Habana Gaudi2 and forthcoming GPU architectures include FP8 support. The existence of this hardware makes the question of which FP8 format to standardize on, and how to use it in post-training quantization, immediately relevant to practitioners and hardware architects alike. The paper's three-format comparison (E5M2, E4M3, E3M4) directly informs this standardization decision.

Where INT8 Mitigations Fall Short

The paper extensively catalogs prior attempts to address INT8's limitations (Section 1), grouping them into several categories and explaining why each is insufficient as a general solution.

Asymmetric quantization (Jacob et al., 2018; Krishnamoorthi, 2018; Bhalgat et al., 2020) allocates different numbers of bits to the positive and negative ranges using a non-zero offset, better matching the actual distribution of values. This helps when the positive and negative ranges have different magnitudes, but it does nothing to address the fundamental dynamic range limitation — the overall ratio of largest to smallest (non-zero) representable value is still bounded by the bit width.

Non-uniform quantization (Miyashita et al., 2016; Zhou et al., 2017; Cai et al., 2017; Fang et al., 2020; Li et al., 2020) assigns more precision to frequently-occurring values, conceptually similar to floating-point formats. However, these methods typically require specialized hardware support for the non-uniform grid, which has not materialized in commodity inference hardware. INT8's ubiquity is precisely because uniform integer arithmetic is natively supported by CPUs, GPUs, and dedicated accelerators. Non-uniform quantization schemes that lack hardware support are academic exercises rather than deployment-ready solutions.

Per-group and per-channel scaling (Zhou et al., 2016; Mellempudi et al., 2017; Jacob et al., 2018; Krishnamoorthi, 2018) extend effective dynamic range by using independent scale factors for subsets of elements — a channel gets its own scale, or a group of channels gets its own scale. This partially addresses the outlier problem because an outlier in one channel no longer forces the scale of other channels to stretch. However, it introduces computational overhead. Per-channel activation scaling (as opposed to per-channel weight scaling, which is standard and free because weights are constant) requires special kernel implementations that the paper notes "are likely to incur higher compute overheads" (Section 3.1). INT8 with per-tensor activation scaling (the standard, efficient approach) remains vulnerable to outliers.

Learned clipping thresholds (Sung et al., 2015; Zhao et al., 2019b; Bhalgat et al., 2020; Choi et al., 2018; Esser et al., 2020; Zhang et al., 2018a) attempt to find the optimal truncation point that balances outlier exclusion against range coverage. These are effective for moderate outlier distributions but become increasingly fragile as the outlier-to-typical-value ratio grows — which is exactly what happens in LLM activations. Moreover, finding the right threshold requires either calibration data (which may not be representative of deployment data) or training-time optimization (which is incompatible with the post-training quantization paradigm that the paper focuses on).

Mathematical transformations for outlier redistribution — notably SmoothQuant (Xiao et al., 2022) and Outlier Suppression (Wei et al., 2022) — scale activations down and weights up (or vice versa) to mathematically equivalent effect, moving magnitude from the outlier-plagued activations into the well-behaved weights. These are the most effective INT8 mitigations available, and the paper acknowledges their importance by enabling SmoothQuant on NLP models (Section 4.2.1, footnote). However, the paper's results show that even with SmoothQuant enabled, INT8 still achieves only 67.65% pass rate on NLP workloads, substantially below FP8's 92–96%. This makes the point clearly: the improvements from SmoothQuant are real but they are incremental patches on a fundamentally limited format, not a replacement for a format with inherently larger dynamic range.

The fundamental insight: all of these mitigations are attempts to make a uniform format behave more like a non-uniform format in the regions that matter. FP8 formats are inherently non-uniform, with dense representation near zero and sparse representation at large magnitudes, which is precisely the property these mitigations are trying to approximate. The paper's bet is that it is better to use a format that has this property by construction than to retrofit it onto INT8 through increasingly complex pre-processing.

Prior Work on FP8: Fragmented and Training-Focused

The paper positions itself within a growing FP8 literature that it argues has been fragmented in scope and primarily oriented toward training rather than inference deployment.

Early FP8 work (Wang et al., 2018; Mellempudi et al., 2019) focused on the E5M2 format for training tasks, motivated by the wide dynamic range needed to represent gradient values — which can span many orders of magnitude during backpropagation. This established that FP8 was viable for training, but E5M2's limited precision (only 2 mantissa bits, meaning only 4 representable values per exponent step, with the implicit leading bit) makes it a blunt instrument for representing the fine-grained variations in weights needed for accurate inference.

Sun et al. (2019) advanced the field by proposing a hybrid approach: E5M2 for gradients (where dynamic range dominates) and E4M3 for weights and activations (where precision matters more). They also introduced the concept of an exponent bias to shift the representable range of E4M3 to better cover activation outliers. This was the first work to systematically study format selection per-tensor-type, and it demonstrated that combining formats could outperform using either format alone — a principle the current paper extends significantly. However, Sun et al. evaluated on a limited set of models (primarily CNNs for image classification), and their inference experiments were secondary to their training focus.

Subsequent studies (Noune et al., 2022; Kuzmin et al., 2022) explored formats with even fewer exponent bits — E3M4 and E2M5 — and the concept of variable exponent bias where the bias is tuned per-tensor or per-channel to shift the representable range to match the data distribution. This work significantly expanded the design space of FP8 formats and demonstrated that these extreme formats could achieve competitive accuracy on selected benchmarks, but again the evaluation scope was limited.

Micikevicius et al. (2022) presented the most comprehensive prior study, covering both training and inference with E5M2 and E4M3 formats using per-tensor scaling. This work was influential in standardizing FP8 formats for deep learning and included inference results on large language models, including GPT-3 at 6.7B parameters — significantly larger than most prior quantization studies. However, their work did not systematically compare against INT8, did not explore E3M4 for inference, and did not study the interaction between format choice and domain (NLP vs. CV).

How This Paper Positions Itself

The paper carves out a specific and previously unoccupied position in this landscape through several deliberate choices:

1. Production-oriented post-training quantization, not training. The paper explicitly states its focus on "post-training quantization as the preferred approach used in production" (Section 1). This distinguishes it from the training-focused lineage (Wang et al., 2018; Mellempudi et al., 2019; Micikevicius et al., 2022). Post-training quantization is the deployment setting where practitioners have a pre-trained model, no access to the training pipeline, and need to quantize it for efficient inference. The constraints are different: you cannot fine-tune the model or adjust weights to compensate for quantization error. The quantization scheme must work as a one-shot transformation on the frozen model. This makes the problem harder — any accuracy loss is permanent — but also makes the solutions more immediately applicable.

2. Breadth over depth: 75 architectures across 200+ tasks. Where prior FP8 work typically evaluated on a handful of well-known models (ResNet-50, BERT, perhaps one or two LLMs), this paper deliberately selects a large and diverse set of 75 architectures spanning NLP, CV, audio, and recommendation. The paper describes its selection criteria as "randomly selected from a pool of a combination of diversity and popularity from mainstream hubs such as Hugging Face Models and Torch Vision" (Section 4.1). This breadth is the paper's primary evidence for the claim that FP8 quantization "generalizes across different network architectures" (abstract). The 200+ task count arises because many models support multiple evaluation tasks (e.g., a single BERT checkpoint evaluated on MRPC, COLA, STS-B, SST2, etc.).

3. Systematic format comparison including E3M4. Prior inference studies typically compared E5M2 and E4M3. This paper adds E3M4 to the comparison and — critically — discovers that it has domain-specific advantages: E3M4 is marginally better for CV models while E4M3 is better for NLP (Table 2, pass rates of 78.95% vs. 73.68% for CV; 92.11% vs. 96.32% for NLP). This finding could not have emerged from a narrower study, and it has direct implications for hardware designers deciding which formats to implement in silicon.

4. Unified quantization workflow as a contribution, not just format comparison. The paper does not simply report accuracy numbers for different formats. It develops a quantization workflow (Section 3, Figure 2) comprising a standard scheme (common operators, fixed per-operation scaling rules) and an extended scheme (mixed formats, expanded operator coverage, dynamic quantization, BatchNorm calibration). The workflow is designed to be applied procedurally: start with the standard scheme, then incrementally apply extended scheme components based on the specific model's sensitivity. This makes the framework practical — a deployment engineer can follow the recipe without needing to understand the FP8 format details, and the extended scheme provides a "tuning" knobs for models that don't meet accuracy targets under the standard scheme.

5. Accuracy-driven automatic model tuning for quantization. The paper claims it is "the first study to showcase accuracy-driven automatic model tuning for quantization" (Section 1, contributions). This refers to the extended quantization scheme's feedback loop (Figure 2): apply a quantization configuration, evaluate accuracy, and if the accuracy target is not met, incrementally add extended scheme components (mixed formats, dynamic quantization, expanded operator coverage) until the target is met or all options are exhausted. This treats quantization configuration as an optimization problem rather than a one-shot decision, which is a practical framing that prior work did not systematically explore.

6. Direct and fair comparison against INT8. The paper explicitly designs its standard quantization scheme to be "identical to INT8 quantization scheme, allowing a fair accuracy comparison" (Section 3.1). This means the same operator coverage, the same scaling granularity (per-channel for weights, per-tensor for activations), and the same treatment of first/last operators. The differences in accuracy are therefore attributable to the data format itself, not to confounding differences in the quantization recipe. This is methodologically important because format comparisons that use different scaling strategies or operator coverage for different formats tell you about the recipe, not the format.

The Overarching Thesis

The paper's motivating thesis can be stated as: FP8 formats are a better default than INT8 for post-training quantization of modern neural networks, and with a properly designed workflow, they can replace INT8 across the vast majority of workloads with minimal accuracy loss. This is a bold claim because INT8 has been the industry default for nearly a decade, supported by mature tooling (TensorRT, OpenVINO, TFLite) and ubiquitous hardware. Unseating it requires demonstrating not just parity but meaningful superiority across a convincing range of workloads — which is exactly what the paper's experimental design attempts to establish.

3. Technical Approach

3.1 Reader Orientation

This paper develops a post-training quantization workflow — a procedural recipe — that converts a full-precision (FP32) neural network into an FP8-quantized version without any fine-tuning or access to the training pipeline, aiming to reduce inference computation cost while keeping accuracy loss below 1% relative to the FP32 baseline. The core idea is that FP8 formats achieve this because their inherently non-uniform representation — dense grid points near zero, sparse at large magnitudes — matches the long-tailed distributions of neural network tensors better than INT8's rigid uniform grid, which forces a single step size to span both the dense central mass and rare outliers, degrading representation quality.

3.2 Big-Picture Architecture (Diagram in Words)

The quantization system has five major components:

FP8 Data Type Emulation — a software layer that faithfully simulates the numeric behavior of E5M2, E4M3, and E3M4 formats on FP32 hardware by applying rounding rules, exponent biasing, and special-value handling (NaN, infinity, subnormals) as if the computation were running on native FP8 silicon. This is the foundation that makes all experiments reproducible without specialized hardware.
Standard Quantization Scheme — a fixed, broadly-applicable configuration that quantizes the common subset of operators (Convolution, Linear, Embedding) using per-channel weight scaling and per-tensor activation scaling. This scheme is deliberately designed to be identical in operator coverage and scaling granularity to INT8 quantization, ensuring fair format-to-format comparison.
Extended Quantization Scheme — an incremental, tunable layer of configurations applied on top of the standard scheme when accuracy targets are not met. It includes three independent knobs: mixed FP8 formats (assigning different formats to weights vs. activations), expanded operator coverage (quantizing LayerNorm, BatchNorm, MatMul, BatchMatMul, and element-wise operations that the standard scheme leaves in FP32), and dynamic quantization (computing activation scale factors at runtime rather than from calibration data).
BatchNorm Calibration — a CV-specific correction step that recomputes BatchNorm running mean and variance statistics after quantization to compensate for the distribution shift introduced by the quantized activations flowing through preceding layers. This uses a small calibration dataset (recommended 3,000 samples with training-time data augmentation transforms).
Tuning Feedback Loop — a meta-algorithm that starts with the standard scheme, evaluates accuracy, and if the target (≤1% relative loss) is not met, incrementally activates extended scheme components — operator coverage expansion, mixed formats, dynamic quantization — in a structured search until the accuracy target is reached or all options are exhausted.

Information flows sequentially: an FP32 model enters the pipeline → the standard scheme is applied (weights quantized per-channel, activation scales calibrated from a small dataset) → accuracy is evaluated → if below target, BatchNorm calibration is applied (CV models only) and re-evaluated → if still below target, extended scheme components are toggled one at a time with re-evaluation after each change → the first configuration that meets the 1% threshold is selected as the deployed model.

3.3 Roadmap for the Deep Dive

First, the FP8 numeric formats themselves — bit layouts, encoding rules, the mathematical formula converting bits to real values, and the density properties that make them suitable for neural network tensors. This is foundational: every subsequent design decision follows from the format properties.
Second, the standard quantization scheme — scaling factor computation, weight vs. activation treatment, first/last operator exceptions, and why these choices are deliberately matched to INT8 conventions.
Third, the extended quantization scheme's three mechanisms — mixed formats, expanded operator coverage, and dynamic quantization — each motivated by what the standard scheme cannot handle.
Fourth, the BatchNorm calibration procedure specific to CV models, including the data augmentation choice and sample size recommendation.
Fifth, the integrated tuning workflow (Figure 2) that sequences these components, including the stopping criterion and the search strategy.

This order builds from the bit-level representation (what FP8 is) through per-operator quantization rules (how to apply the format) to the meta-strategy (how to deploy across diverse models).

3.4 Detailed, Sentence-Based Technical Breakdown

This is an empirical systems paper whose core idea is that FP8 floating-point formats — when deployed through a structured quantization workflow with tunable knobs — achieve substantially higher post-training quantization success rates than INT8 across a diverse model landscape, and that the optimal FP8 variant depends on the application domain (E4M3 for NLP, E3M4 for CV).

FP8 Numeric Formats: Bit Layout, Encoding Rules, and Representable Range

The paper studies three 8-bit floating-point formats, all following the standard IEEE-like floating-point structure: one sign bit, $e$ exponent bits encoding an unsigned integer, and $m$ mantissa (fraction) bits encoding the fractional part of the significand. The general decoding formula from stored bits to real value is:

$x = (-1)^s \times 2^{e - b} \times (1 + f_1 \times 2^{-1} + f_2 \times 2^{-2} + \dots + f_m \times 2^{-m})$

where $s \in \{0, 1\}$ is the sign bit, $e$ is the unsigned integer formed by the exponent bits (ranging from $0$ to $2^e - 1$ ), $b$ is the exponent bias (a fixed offset that centers the representable range around $1.0$ ), and $f_i \in \{0, 1\}$ are the mantissa bits.

What it computes: Given the 8 stored bits for a single number, this formula produces a real-valued number by (1) determining the sign from $s$ , (2) computing the exponent's power-of-two scaling factor as $2^{e-b}$ , and (3) interpreting the mantissa bits as a binary fraction tacked onto an implicit leading $1$ (the "implicit leading bit" convention, standard in IEEE 754). The mantissa provides precision within each exponent interval: for a given exponent value, there are exactly $2^m$ equally-spaced representable values. The exponent provides dynamic range: each increment of $1$ in the exponent multiplies the representable values in that interval by $2$ , so values double between consecutive exponent steps.

Why this form: This is the standard floating-point decomposition inherited from IEEE 754, chosen because it efficiently represents numbers across many orders of magnitude with variable precision — dense near zero, sparse at large magnitudes. The implicit leading bit ( $1 +$ rather than including the leading bit as a stored bit) gives one extra bit of precision for free, since the leading non-zero bit of a normalized binary number is always $1$ . The exponent bias $b$ shifts the center of the representable range: without a bias, the smallest exponent ( $e=0$ ) would represent values near $2^0 = 1$ , wasting the format's representational capacity on numbers $\geq 1$ . By setting $b$ , the format centers its precision where the model's tensor values actually lie.

The three FP8 formats differ in how they allocate the 7 non-sign bits between exponent and mantissa, and in their encoding rules for special values (zero, subnormals, NaN, infinity). Table 1 in the paper provides the specifications:

E5M2: 5 exponent bits, 2 mantissa bits, bias $b=15$ . This format follows "IEEE-like encoding rules", meaning it reserves the all-ones exponent ( $e=31$ ) for infinities and NaNs, and the all-zeros exponent ( $e=0$ ) for subnormals and zero. Maximum representable normal value is 57,344.0; minimum positive normal value is $1.5 \times 10^{-5}$ . The small mantissa (2 bits = 4 values per exponent step, plus the implicit leading bit giving effectively 3-bit precision) means this format prioritizes dynamic range over precision — it can represent values spanning 7 orders of magnitude but can only distinguish 4 distinct values within each factor-of-2 interval.

E4M3: 4 exponent bits, 3 mantissa bits, bias $b=7$ . Uses "extended encoding": unlike IEEE convention, the all-ones exponent bit pattern is reclaimed to represent useful numeric values rather than infinity/NaN. Infinity is not representable in E4M3; a single unique bit-sequence of all-ones is reserved to represent NaN. This reclaiming adds 16 additional representable values that would otherwise be wasted on infinity encodings. Maximum value 448.0; minimum positive 0.0019. The balance is 4 bits of dynamic range (factor of $2^4 = 16\times$ between minimum and maximum exponent) with 3 bits of precision (8 values per exponent step).

E3M4: 3 exponent bits, 4 mantissa bits, bias $b=3$ . Also uses extended encoding with no infinity. Maximum value 30.0; minimum positive 0.015. This format prioritizes precision (4 mantissa bits = 16 values per exponent step) over dynamic range (only $2^3 = 8\times$ between min and max exponent). The maximum value of 30.0 is notably restrictive — any value above 30.0 must be clipped, which is a serious design consideration for activation tensors that may contain outliers.

All three formats support subnormals: when the exponent bits are all-zero ( $e=0$ ), the implicit leading bit becomes $0$ instead of $1$ , and the exponent is treated as $1 - b$ rather than $0 - b$ . This provides a smooth underflow region between zero and the smallest normal value, gradually losing precision as values approach zero — critical for representing very small weight values without quantizing them to zero.

Why FP8 Handles Outliers Better: Density Properties and Quantization Error

The key difference between FP8 and INT8 is where the representational budget is spent. INT8 divides its 256 levels uniformly across the entire representable range; the step size $\Delta$ is constant and equal to $\text{max\_value} / 127.5$ (for symmetric quantization). FP8 concentrates its grid points where the exponent is smallest — near zero — and has increasingly sparse coverage at large magnitudes.

The paper formalizes this with a density metric derived in Appendix A.1. For a floating-point format with $e$ exponent bits and $m$ mantissa bits, the density (number of representable values per unit interval) in the range $[2^n, 2^{n+1})$ is:

$D_{E(e)M(m)} = \frac{2^m}{2^{n+1} - 2^n} = 2^{m - n}$

where $2^m$ is the number of distinct mantissa patterns (all equally spaced within that exponent interval), $2^{n+1} - 2^n = 2^n$ is the width of the interval, and $n = \lfloor \log_2 N \rfloor$ is the exponent value for a number $N$ in that range.

What it computes: For any decimal number $N$ , this gives the local representational density — how many distinct values the format can distinguish in the immediate neighborhood of $N$ . The formula makes explicit that density decays as $2^{-n}$ : when $N$ is small ( $n$ small), density is high; when $N$ is large ( $n$ large), density is low. INT8 has $D = 256 / \text{range}$ everywhere — constant density, which is high for large values but wastefully sparse for the small values where most tensor data lives.

Why this matters: Neural network weight and activation tensors typically follow long-tailed normal-like distributions: a dense concentration of values near zero containing the vast majority of the information, with a thin tail of larger values (including outliers in activations). FP8's density pattern matches this structure: the 3 $\sigma$ region of the distribution — where most values lie and where precise representation is important for preserving the model's computation — falls in the high-density exponent ranges near zero, while outliers fall in the sparse, high-exponent ranges where precision matters less. INT8 forces a trade-off: if you set the maximum representable value high enough to cover outliers without clipping, the uniform step size stretches, and the dense central region gets poorly represented; if you clip outliers to keep the step size small, the outliers are lost entirely.

Figure 1 illustrates this concretely. The center panel shows the quantization grid distribution for a normal distribution with 1% uniformly-distributed outliers in $[-6, 6]$ . E4M3 and E3M4 show a dense band of grid points around zero (the high-precision region) that covers most of the 3 $\sigma$ region of the data, with grid points thinning out toward the tails. INT8's grid points are equally spaced across the full range, forcing the Outliers to stretch the grid and reduce the number of points under the distribution's central mass. The right panel quantifies the consequence: E4M3 and E3M4 achieve substantially lower mean squared error (MSE) than INT8, while E5M2 — with only 2 mantissa bits — performs worse because its extreme dynamic range comes at the cost of too-coarse precision.

The Standard Quantization Scheme: Weight and Activation Scaling

The standard scheme is the "default configuration applied to common set of operators across different architectures" (Section 3.1). It covers Convolution, Linear, and Embedding layers — the compute-heavy operators that dominate inference time. The scheme is explicitly designed to be "identical to INT8 quantization scheme, allowing a fair accuracy comparison."

Per-channel weight scaling. For weight tensors, each output channel receives its own scale factor, computed as:

$s_w = \frac{\text{float\_max}}{\max(|W_{\text{channel}}|)}$

where $\text{float\_max}$ is the maximum representable normal value of the chosen FP8 format (57,344 for E5M2, 448 for E4M3, 30 for E3M4), and $\max(|W_{\text{channel}}|)$ is the absolute maximum value among all weights in that output channel.

What it computes: For each output channel's weight vector, this computes a single scalar $s_w$ that, when multiplied with every weight in that channel, maps the channel's maximum absolute value exactly to the format's maximum representable value — fully utilizing the available encoding space for that channel without overflow.

Why per-channel: The paper states that "although FP8 formats have sufficient dynamic range to handle common weight distributions, empirical evidence suggests that applying per-channel scaling can reduce rounding errors by effectively utilizing the full encoding space for each channel." In other words, even though FP8 could handle the weights with a single per-tensor scale (its dynamic range is large enough to cover across-channel variation), using per-channel scales gives each channel the full $2^{e+m}$ quantization levels dedicated to its value range, reducing the relative quantization error. This is standard practice in INT8 quantization and is computationally cheap because weight scales are computed once offline and weights are constant at inference time — the per-channel granularity adds no runtime overhead for weight-only quantization.

Per-tensor activation scaling. For activation tensors (inputs to each layer), a single scale factor is computed for the entire tensor:

$s_a = \frac{\text{float\_max}}{\max(|A|)}$

where $\max(|A|)$ is the absolute maximum value in the activation tensor, calibrated from a small set of representative input data (a few hundred to a few thousand samples).

What it computes: A single scalar mapping the entire activation tensor's dynamic range to the format's full encoding space. This is simpler than per-channel activation scaling but means that a single outlier in any spatial/channel position dictates the scale for all positions.

Why per-tensor for activations but per-channel for weights: The paper acknowledges that "some recent studies (Xiao et al., 2022; Wei et al., 2022; Dettmers et al., 2022) have indicated that per-channel activation scaling can benefit INT8 quantization" but explicitly excludes this from the study because such methods "may require special kernel implementations that are likely to incur higher compute overheads." The practical consideration is that per-channel activation quantization requires the scale multiplication to happen inside the matrix multiplication inner loop rather than as a pre-processing step, which breaks standard GEMM kernel assumptions. By sticking with per-tensor activation scaling, the paper ensures that its quantization scheme maps directly to efficient hardware implementations — a deliberate engineering trade-off prioritizing deployability over maximum accuracy.

Range calibration for E4M3 and E3M4. E5M2's massive dynamic range (max value 57,344) means it can handle typical activation outliers without any calibration — the paper applies "direct quantization" for E5M2, meaning each value is simply rounded to the nearest representable FP8 value without any scaling or clipping. For E4M3 and E3M4, whose maximum values (448 and 30) may be exceeded by activation outliers, the paper uses "simple max scaling" — compute the scale as $\text{max\_representable} / \max(|A|)$ and apply it. The paper examined more sophisticated range-calibration methods including "KL divergence, MSE error, and percentile" but found they "did not provide any additional benefits" (Section 3). This is notable because KL-divergence-based calibration is the standard approach for INT8 activation quantization (it clips a small fraction of the largest values to reduce the step size for the bulk of the distribution). For FP8, the format's inherent non-uniformity makes this unnecessary — the outliers naturally fall into sparse, high-exponent regions where they are represented coarsely but don't stretch the precision for small values.

First and last operator exception. For convolutional neural networks only, the first convolution layer (which processes input images) and the last fully-connected layer (which produces classification logits) are kept in higher precision (FP32 or FP16). The paper states these "typically constitute < 1% of the total computation" but are "more sensitive to quantization" (Section 3.1). This exception is inherited from INT8 best practices (Han et al., 2015b; Choi et al., 2018) and is applicable only to convolutional networks — NLP transformer models do not receive this treatment because their embedding and final projection layers are already included in the standard scheme's operator coverage.

The Extended Quantization Scheme: Three Tunable Mechanisms

When the standard scheme fails to meet the 1% relative accuracy loss threshold, the extended scheme provides three independent mechanisms that can be activated incrementally.

Mechanism 1: Expanded Operator Coverage. The standard scheme leaves several operator types in FP32: LayerNorm, BatchNorm (when not folded into preceding convolutions), MatMul, BatchMatMul, and element-wise operations (Add, Mul). These are "memory-bound operations" — their execution time is dominated by data movement rather than arithmetic, so they contribute disproportionately to inference latency even though they involve fewer FLOPs than convolutions or matrix multiplications.

The paper reports that "previous attempts to quantize these operators using integer approximation were unsuccessful in maintaining the model accuracy" (Bhandare et al., 2019; Kim et al., 2021) — the dynamic range requirements of normalization operations and the precision sensitivity of element-wise operations made INT8 quantization degrade accuracy. The extended scheme enables FP8 quantization for these operators, exploiting FP8's wider dynamic range to represent the normalization statistics and residual add connections without accuracy loss.

Figure 9 visualizes the incremental effect: starting from the standard scheme (Conv + Linear, optionally excluding first/last ops), successively adding BatchMatMul/MatMul, then Embedding/EmbeddingBag, then LayerNorm. Each expansion increases the fraction of the model running in FP8 at the potential cost of accuracy. The paper's results show that E4M3 maintains accuracy through all expansion levels on NLP models, while INT8 degrades noticeably.

Mechanism 2: Mixed FP8 Formats. The standard scheme applies a single FP8 format uniformly to all quantized tensors. The extended scheme allows per-tensor-type format assignment: weights and activations within the same operation can use different FP8 variants. The motivation comes from observed distribution differences (Figure 3):

Weight tensors (both CV and NLP) "tend to follow normal distributions with lots of values near zero" — they are precision-bound: representing the fine variations in weight values matters more than covering extreme magnitudes.
Activation tensors in NLP models "show a lot of outliers which demand a larger dynamic range" — they are range-bound: covering the outlier values without clipping is critical.
Activation tensors in CV models "tend to be precision-bounded" — similar to weights, their outlier magnitude is less extreme.

The mixed format strategy assigns E4M3 or E5M2 (more exponent bits) to range-bound tensors and E3M4 (more mantissa bits) to precision-bound tensors. For NLP workloads, the paper's experiments show that using E4M3 for activations (handling outliers) and E3M4 for weights (providing finer precision for the well-behaved weight distribution) produces the best accuracy results on models like BERT, Funnel, and Longformer (Table 5).

Figure 8 provides a concrete example on BERT-base (MRPC task): using a single format (E5M2, E4M3, or E3M4) for both weights and activations produces relatively high quantization error in the Linear operator's output (as measured by MSE between FP32 output and quantized output). Mixing E4M3 for activations with E3M4 for weights reduces the output MSE significantly — from 22,108.65 (best single-format, E3M4) to 919.70, a 24× reduction. The explanation: E4M3's wider dynamic range correctly represents the activation outliers that would otherwise clip or stretch the scale for the entire tensor, while E3M4's higher precision for the weights means the matrix multiplication's inner products accumulate less quantization noise per weight-activation product pair.

Mechanism 3: Dynamic Quantization. The standard scheme uses static quantization: activation scale factors are computed once from calibration data before inference begins, and are fixed for all future inputs. This is "computationally more efficient" (Section 3.2) because the scale computation happens offline and no per-input range computation is needed at runtime.

Dynamic quantization recomputes the activation scale factor $s_a$ for every input at runtime, using the actual min/max of that specific input's activations rather than calibration-derived estimates. This adapts the scale to the particular input's activation range, which can vary significantly — especially in NLP models where different input sequences produce different attention patterns and activation magnitudes.

The paper reports that dynamic quantization "offers no additional benefits to E5M2" — its dynamic range is already large enough that any realistic input fits without scaling — but "observed a noticeable improvement in accuracy for E4M3 and E3M4 formats on selected models" (Section 3.2). Table 6 quantifies this: on BERT-base (MRPC) with E4M3, dynamic quantization improves accuracy by +0.87% (0.9151 vs. 0.9072). On BERT-large (RTE), the improvement is +0.98% (0.7401 vs. 0.7329). These are meaningful gains that can push a model from "below threshold" to "above threshold" in the tuning workflow.

The trade-off is computational: dynamic quantization requires computing the maximum absolute value of each activation tensor online, which adds a small amount of per-inference overhead for scan operations. The extended scheme treats this as a tunable knob — activate dynamic quantization only for models where static quantization fails the accuracy threshold.

BatchNorm Calibration: Correcting Distribution Shift in CV Models

Convolutional neural networks with BatchNorm layers face a specific quantization challenge not present in NLP transformers. BatchNorm normalizes activations using running estimates of mean and variance accumulated during training. When preceding layers are quantized, the distribution of activations feeding into the BatchNorm layer shifts — the quantized outputs have different statistical properties than the FP32 outputs used to compute the running statistics. Using the stale running statistics with quantized inputs introduces a systematic bias that degrades accuracy.

The paper's solution — attributed to Sun et al. (2019), who originally proposed it — is to recalibrate the BatchNorm statistics after quantization. The procedure:

Freeze all quantized weights and scale factors.
Pass a small calibration dataset through the quantized model.
Recompute the running mean and running variance of the activations at each BatchNorm layer using these quantized-forward-pass statistics.
Replace the original (FP32-era) BatchNorm statistics with the recalibrated ones.

The paper studies two key hyperparameters of this calibration: the sample size and the data augmentation strategy.

Sample size trade-off: Figure 7 shows that using 300 samples with training-time augmentation transforms achieves comparable accuracy to 10,000 samples with training transforms, and both significantly outperform using inference-time transforms (3,000 samples with inference transforms performs worse than 300 samples with training transforms). The paper recommends "sample size of 3K with training transform for achieving best results across a wide range of networks."

Training vs. inference transforms: The paper finds that "training transform [is] more effective even at smaller sample sizes (<3K)" (Section 4.3.1). Training transforms include random cropping, flipping, and color jittering — augmentations that increase the diversity of the calibration data and produce more robust BatchNorm statistics that generalize better to the varied inputs the model will see at deployment. Inference transforms (center crop + resize only) produce less diverse calibration activations, requiring more samples to achieve the same statistical quality.

Why this matters for fairness: The INT8 baseline also benefits from BatchNorm calibration, and the paper's workflow diagram (Figure 2) shows this step applied to CV models regardless of quantization format. The accuracy improvements from this step are therefore not unique to FP8; they simply reflect good quantization engineering practice. Including this step ensures the FP8-vs-INT8 comparison is fair — both formats get the same correction.

The Tuning Workflow: Automated Accuracy-Driven Configuration Search

The complete workflow (Figure 2) integrates all components into a procedural pipeline designed for deployment engineers. The paper describes it as "accuracy-driven automatic model tuning for quantization" — a contribution it claims is a first for the quantization literature.

Step-by-step procedure:

Start with the standard quantization scheme: Apply per-channel weight scaling and per-tensor activation scaling to Convolution, Linear, and Embedding layers. Use the FP8 format specified for the trial (E5M2 with direct quantization, or E4M3/E3M4 with max-based range calibration). For CV models, exclude the first convolution and last fully-connected layer (keep them in FP32).
Evaluate accuracy: Run the quantized model on the evaluation dataset and compute accuracy relative to the FP32 baseline. Compute relative accuracy loss as $(\text{acc}_{\text{FP32}} - \text{acc}_{\text{quantized}}) / \text{acc}_{\text{FP32}} \times 100\%$ .
Check stopping criterion: If relative accuracy loss ≤ 1%, the model passes and the current configuration is accepted. No further tuning needed.
Apply BatchNorm calibration (CV models only): If the model has BatchNorm layers and accuracy is below threshold, perform the recalibration procedure described above. Re-evaluate accuracy. If threshold is now met, stop.
Expand operator coverage: Incrementally add operator types to the FP8 quantization set. The paper's results (Figure 9) suggest an order: first add BatchMatMul and MatMul, then Embedding (and EmbeddingBag), and finally LayerNorm. After each expansion, re-evaluate accuracy. Continue expanding as long as accuracy remains above threshold.
Enable mixed FP8 formats: If single-format quantization fails, assign E4M3/E5M2 to range-bound tensors (NLP activations) and E3M4 to precision-bound tensors (weights, CV activations). The assignment is based on the heuristics from Section 3.2: examine tensor distributions for outlier presence (range-bound) vs. compact normal-like shape (precision-bound).
Switch to dynamic quantization: If static quantization still fails, enable per-input activation scale computation for E4M3 or E3M4 formats. This is applied selectively to the specific models that benefit from it (the paper's Table 6 shows improvements for specific BERT variants).
Fall back: If no configuration meets the 1% threshold, the model is considered a quantization failure for that format. The deployment engineer must either accept a larger accuracy loss, use a different format, or keep the model in FP32/FP16.

Why this workflow and not a single recipe: The paper's central empirical finding is that no single quantization configuration works universally. The standard scheme alone achieves high pass rates (74.89% for E5M2, 92.64% for E4M3 across all models in Table 2), but the remaining failures require model-specific tuning. The extended scheme provides knobs that are applied only when needed, preserving the computational simplicity of the standard scheme for models that don't need tuning while providing a path to accuracy recovery for models that do.

The search space and efficiency: The paper does not perform an exhaustive grid search over all combinations of extended scheme knobs — that would require evaluating $2^{\text{\#knobs}}$ configurations per model. Instead, the workflow applies knobs incrementally in a fixed priority order informed by the paper's empirical findings: BatchNorm calibration first (cheap, often sufficient for CV), then operator coverage expansion (increasing fraction of model in FP8), then mixed formats (per-tensor-type format optimization), then dynamic quantization (most expensive at runtime). This greedy approach finds the first acceptable configuration rather than the globally optimal one, trading some optimality for practical efficiency — a deployment engineer can tune a model in a few evaluation runs rather than dozens.

Design Choices and Their Justifications

Why per-channel weight scaling and per-tensor activation scaling: Per-channel weights give each output channel its full encoding precision at no runtime cost. Per-tensor activations avoid the need for specialized GEMM kernels that per-channel activation quantization would require. The asymmetry reflects a deliberate engineering priority: deployability on standard hardware over theoretical optimality.

Why max-based range calibration rather than KL divergence: The paper explicitly tested KL divergence, MSE minimization, and percentile-based calibration for FP8 activation scaling and found "they did not provide any additional benefits" (Section 3). For INT8, KL divergence works because clipping outliers reduces the step size for the dense region — a trade-off that matters when step size is uniform. For FP8, the dense region's precision is already high (because it falls in low-exponent ranges with many mantissa bits) and is unaffected by whether outliers are clipped or not — they occupy separate exponent ranges. The simpler max-based scaling is therefore sufficient.

Why first/last operator exception persists for CNNs but not NLP: CNNs process raw pixel inputs (ranging 0–255) in the first layer, and the quantization error there propagates through the entire network. The final classification layer's output logits directly determine the predicted class — small perturbations matter. For NLP, embeddings are already included in the standard scheme and the final projection layer is just another Linear operator — its sensitivity is not categorically different from intermediate layers.

Why the 1% relative accuracy loss threshold: The paper does not explicitly justify this threshold, but it is a common convention in the quantization literature — it represents a level of accuracy degradation that most production deployments can tolerate, balancing the efficiency gains of quantization against the quality requirements of the application. A stricter threshold (e.g., 0.5%) would lower pass rates for all formats; a looser threshold (e.g., 2%) would raise them. The relative comparison (FP8 vs. INT8) is more informative than the absolute pass rate at any given threshold.

4. Key Insights and Innovations

Innovation 1: Reframing Quantization Format Selection as a Domain-Conditioned Empirical Problem

Prior to this work, the selection of quantization data types — INT8 vs. FP8, and which FP8 variant — was treated as either a hardware-driven decision (what does the silicon support?) or a theoretical exercise (what dynamic range does the format provide?). The dominant implicit assumption was that a single optimal format exists, or at most that E5M2 is for training (gradients need range) and E4M3 is for inference (weights and activations need precision), as established by Sun et al. (2019) and Micikevicius et al. (2022).

This paper makes a conceptual move that reframes the problem: format selection is not universal — it is conditioned on the application domain in a way that is empirically discoverable but not obvious from first principles. The specific finding that E4M3 achieves 96.32% pass rate on NLP while E3M4 achieves 78.95% on CV (Table 2) is not a minor performance detail — it reveals that the optimal exponent-to-mantissa ratio depends on how tensor distributions differ across domains. NLP activations carry large outliers (from attention patterns and LayerNorm amplification), demanding the wider dynamic range of E4M3's 4 exponent bits. CV activations are more compact (ReLU-clipped, BatchNorm-normalized), making E3M4's extra mantissa bit — which doubles the precision within each exponent interval — genuinely more valuable than the lost exponent bit.

What makes this distinctive is that it inverts the standard engineering question. Rather than asking "which format should hardware support?" and then forcing all workloads into that format, the paper asks "given that we can support multiple formats, which ones should we deploy for which workloads, and what is the empirical evidence for those choices?" This is a diagnostic contribution, not a methodological one: it gives practitioners a decision rule (E4M3 for NLP, E3M4 for CV) backed by systematic evidence, and it gives hardware architects data to justify implementing both formats rather than standardizing on one.

The comparison to prior work is instructive. Micikevicius et al. (2022) advocated for E4M3 as the general-purpose inference format and demonstrated its effectiveness on GPT-3, but did not test E3M4 or evaluate across CV models at scale. Kuzmin et al. (2022) explored E3M4 and even E2M5 but on a narrow set of models without the domain comparison. This paper's 75-architecture evaluation provides the statistical power to claim that the domain effect is real and not an artifact of model-specific quirks.

The significance extends beyond raw performance: this finding implies that the FP8 format design space is not a simple "more exponent bits = better for outliers" trade-off. On CV workloads, where outliers are less extreme, the precision gain from an extra mantissa bit actually outweighs the dynamic range loss from giving up an exponent bit. This is non-obvious — one might expect that more dynamic range is always safer for post-training quantization where you cannot fine-tune weights to adapt — and it would remain invisible in a study that evaluated only NLP transformers or only ResNet variants.

Evidence anchor: Table 2 (pass rate breakdown by domain), with the NLP column showing E4M3 at 96.32% vs. E3M4 at 92.11%, and the CV column showing E3M4 at 78.95% vs. E4M3 at 73.68%. Figure 5 provides the per-model granularity, showing that the trend holds across model sizes within each domain.

Innovation 2: The Extended Quantization Scheme as a Formalized Tuning Protocol, Not Just a Collection of Techniques

The individual components of the extended quantization scheme — per-channel scaling, dynamic quantization, mixed precision, operator coverage expansion — all existed in prior literature. What the paper contributes is the meta-algorithm that sequences them: a greedy, incremental tuning protocol with a fixed priority order and a clear stopping criterion (1% relative accuracy loss). This transforms quantization from a one-shot format decision into a search problem over a structured configuration space, where the goal is to find the first acceptable configuration rather than the globally optimal one.

This is a conceptual shift in how post-training quantization is approached. The field's dominant paradigm — embodied by TensorRT, OpenVINO, and TFLite — is to apply a fixed recipe to a model and either accept the result (if accuracy is acceptable) or fall back to higher precision (if not). The recipe may include some calibration choices (KL divergence vs. percentile for activation range), but the overall structure is static: quantize these operators with this granularity, and that's the output. The paper's workflow (Figure 2) says instead: start with the simplest configuration that is broadly applicable, and only add complexity where the model demands it. This is the quantization analog of the "don't optimize prematurely" principle in software engineering — preserve simplicity for the majority of models that don't need tuning, while providing a path to accuracy recovery for the minority that do.

The significance of this contribution is practical rather than theoretical: it turns quantization from an art (requiring expert judgment about which techniques to apply to which model) into a procedure (a sequence of steps with empirically-grounded priority ordering). The priority ordering itself is a contribution derived from the paper's large-scale experiments: BatchNorm calibration first (cheap, often sufficient for CV), operator coverage expansion next (maximizes the fraction of the model in FP8), mixed formats next (per-tensor-type format optimization, more complex to implement), dynamic quantization last (adds runtime overhead). This ordering encodes empirical knowledge — it would be difficult to derive from first principles without running the experiments.

Crucially, the workflow is format-agnostic at the meta-level. The same structure works for E4M3, E3M4, and even INT8 — but the pass rates differ dramatically because the underlying format's properties determine how often the standard scheme alone suffices and how effective the extended scheme's knobs are. This separation between the workflow structure (the meta-algorithm) and the format choice (the parameter) is what makes the contribution generalizable — a new 8-bit format (e.g., E2M5 with a different bias) could be plugged into the same workflow and evaluated without redesigning the tuning protocol.

Evidence anchor: Figure 2 (the workflow diagram) and Figure 9 (visualizing how incremental operator coverage expansion affects accuracy for different formats). Table 2's pass rates demonstrate the workflow's effectiveness: E4M3 achieves 92.64% overall pass rate, far exceeding INT8's 65.87%, using the same workflow structure.

Innovation 3: The Systematic Demonstration That FP8 Handles Previously Unquantizable Operators, Changing the Scope of End-to-End Quantization

A persistent limitation of INT8 quantization has been that certain operator types — LayerNorm, BatchNorm (when not folded), and element-wise operations — resist quantization without significant accuracy degradation. Prior work by Bhandare et al. (2019) and Kim et al. (2021) attempted INT8 approximations for these operators and reported failures. The practical consequence was that quantized models were not truly end-to-end quantized: they included FP32 "islands" for these operators, requiring format conversions at operator boundaries that added latency and complexity.

This paper demonstrates that FP8 formats can quantize these operators without accuracy loss, effectively eliminating the concept of unquantizable operators for a large class of models. Figure 9 shows this systematically: as operators are incrementally added to the FP8 quantization set — from Conv+Linear only, to +BatchMatMul+MatMul, to +Embedding, to +LayerNorm — E4M3 maintains consistent accuracy with low variability across both CV and NLP models, while INT8 degrades or shows high variability. The Stable Diffusion results (Figure 6) provide a visually compelling demonstration: FP8 with LayerNorm quantized produces images indistinguishable from FP32, while INT8 introduces visible artifacts and a degraded FID score.

What makes this distinctive is not the per-operator result (which one might predict from FP8's wider dynamic range) but the systematic evidence that the barrier is format-inherent, not operator-inherent. The operators were not "unquantizable" in any absolute sense — they were unquantizable with INT8's uniform grid. FP8's non-uniform representation makes them tractable. This reframes the problem: rather than developing ever-more-sophisticated integer approximations for normalization operations (a research direction that had produced limited returns), the solution is to adopt a format that naturally handles the required dynamic range.

The practical implications are significant. In a fully end-to-end FP8 model, there are no format conversion boundaries — activations flow from one quantized operator to the next in FP8, eliminating the FP32↔INT8 conversion overhead that plagues INT8 deployments. This simplifies kernel implementations (no mixed-precision dispatch logic), reduces memory bandwidth (no intermediate dequantization buffers), and potentially enables higher throughput. The paper does not benchmark wall-clock performance, but the architectural implication is clear: FP8 enables a cleaner, more efficient inference pipeline architecture than INT8 for models with these operators.

Evidence anchor: Figure 9, showing accuracy loss as operator coverage expands. The LayerNorm column is particularly telling — it's the rightmost addition and represents the operator type that prior work had found most resistant to integer quantization.

Innovation 4: Identifying the Insufficiency of KL-Divergence Calibration for FP8, Breaking a Longstanding Default

For over half a decade, KL-divergence-based range calibration has been the default activation calibration method for INT8 quantization. Migacz (2017) established it as the standard approach in TensorRT, and it has been widely adopted because it elegantly solves INT8's core tension: clip a fraction of the largest values (the outliers) to reduce the step size for the dense region, with the KL divergence between the original and quantized distributions serving as the objective function for selecting the clipping threshold.

This paper reports a negative finding that is significant precisely because it challenges this default: for FP8 formats, KL-divergence calibration, along with MSE minimization and percentile-based methods, "did not provide any additional benefits" over simple max-based scaling (Section 3). The explanation — articulated in the paper's Appendix A.1 density analysis — is that FP8's non-uniform grid makes outlier clipping unnecessary. The outliers naturally fall into sparse, high-exponent regions of the FP8 representation where they are coarsely quantized, while the dense central distribution falls into low-exponent regions with high precision. Clipping outliers to improve central precision — the mechanism that makes KL divergence work for INT8 — provides no benefit because the central precision is already determined by the exponent region, not by the overall range.

This is a conceptual contribution because it reveals that calibration strategies are format-dependent in a non-obvious way. The field had implicitly treated calibration as a universal step — you pick a method (KL, MSE, percentile), you calibrate, you quantize — independent of the target format. This paper shows that the calibration method's value is tightly coupled to the format's representation structure: uniform formats benefit from outlier clipping; non-uniform formats with sufficient exponent range do not. The practical consequence is that FP8 quantization is simpler than INT8 quantization at the calibration stage — no need for sophisticated clipping threshold optimization, just compute the maximum and scale. This simplicity is a genuine practical advantage for deployment engineers who previously needed to tune calibration hyperparameters.

The paper does not overstate this finding — it is presented as an empirical observation rather than a theoretical proof — but its implications are broader than the specific FP8 formats studied. Any floating-point format with sufficient dynamic range relative to the data's outlier magnitude should exhibit the same property, which is a testable prediction that could guide future format design.

Evidence anchor: Section 3 ("we also examined more sophisticated range-calibration methods such as KL divergence, MSE error and percentile which did not provide any additional benefits") and Figure 1 (right), which quantifies the MSE advantage of FP8 over INT8 under the max-based scaling that the paper ultimately recommends.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on more than 200 tasks using over 20 different datasets (Section 4.1). For NLP, these include the GLUE benchmark suite (MRPC, COLA, STS-B, SST2, RTE), lambada-openai for generative language modeling, samsum for summarization, WMT EN-RO for machine translation, and wikitext for language modeling. For computer vision, the paper uses ImageNet ILSVRC 2012 and CIFAR-10 for image classification, the Kaggle Carvana Image Masking Challenge dataset for segmentation, COCO2014 for object detection, and FID evaluation on stable diffusion prompts for image generation. For speech, LibriSpeech is used; for recommendation, Criteo Terabyte. The diversity is deliberate — spanning classification, generation, segmentation, detection, translation, and recommendation — to stress-test the quantization workflow across fundamentally different task types and output modalities.
Base model(s). The paper evaluates 75 unique model architectures selected "randomly from a pool of a combination of diversity and popularity from mainstream hubs such as Hugging Face Models and Torch Vision, as well as individual models from Github based on their popularity" (Section 4.1). The models span: NLP transformers (BERT-base and BERT-large, DistilBert, Longformer, Funnel, XLM-RoBERTa), generative LLMs (Bloom at 7B, 176B; LLaMA at 65B), text generation models (MarianMT, DialogGPT, Pegasus), convolutional neural networks (ResNet-50, DenseNet-121, VGG-13, GoogleNet, ShuffleNetV2, EfficientNet-B0, MobileNetV2, InceptionV3), vision transformers (ViT), generative vision models (Stable Diffusion), object detectors (YoloV3), segmentation models (U-Net), speech models (HuBERT, wav2vec 2.0), and recommendation models (DLRM). The inclusion of models ranging from small (sub-32MB) to very large (Bloom-176B at hundreds of GB) tests the workflow's scalability across parameter count. The paper uses FP32 pre-trained checkpoints as the starting point for all quantization experiments.
Metrics. The primary metric is pass rate — the percentage of evaluated workloads (model + task combinations) that maintain accuracy within 1% relative loss compared to the FP32 baseline (Table 2). For individual models, relative accuracy loss is computed as (acc_FP32 - acc_quantized) / acc_FP32 × 100%. Task-specific metrics follow standard conventions: accuracy for classification (ImageNet, CIFAR-10, GLUE tasks), F1 score for MRPC, Matthews correlation for COLA, Pearson/Spearman correlation for STS-B, perplexity or accuracy for lambada-openai, BLEU for machine translation, FID (Fréchet Inception Distance) for image generation quality. For the 1% threshold, the paper uses relative rather than absolute loss — a model with 90% FP32 accuracy passes if it achieves ≥ 89.1% after quantization.
Baselines. The paper compares against INT8 quantization as the primary baseline, with two variants: static quantization for CV models and dynamic quantization for NLP models (Table 2). This split reflects the paper's choice to give INT8 its best-known configuration per domain rather than a one-size-fits-all recipe. The INT8 baseline uses per-channel weight scaling and per-tensor activation scaling matching the FP8 standard scheme's granularity, with SmoothQuant enabled on NLP models using the default smoothing alpha value (Section 4.2.1 note). KL-divergence-based range calibration is applied for INT8 activation scaling, as is standard practice. The FP32 model serves as the accuracy reference point but is not a competing method — it represents the upper bound that quantization methods attempt to approach.
Generation budget / compute accounting. The paper does not report inference latency, throughput, or FLOPs measurements. Compute is accounted for implicitly through the quantization scheme design choices: the standard scheme quantizes compute-heavy operators (Convolution, Linear, Embedding, MatMul, BatchMatMul) to FP8, and the extended scheme additionally covers memory-bound operators (LayerNorm, BatchNorm, element-wise Add/Mul). The relative computational efficiency of different configurations is discussed qualitatively — static quantization is "computationally more efficient" than dynamic quantization (Section 3.2), and per-channel activation scaling is avoided because it "may require special kernel implementations that are likely to incur higher compute overheads" (Section 3.1) — but no timing numbers are reported. The paper uses a software emulation framework (FP8 Emulation Toolkit + Neural Compressor) that runs quantized operations on FP32 hardware by simulating FP8 rounding behavior, meaning wall-clock speedup measurements would be meaningless; the experiments measure only accuracy, not performance.
Cross-validation / statistical protocol. The paper employs no cross-validation or statistical significance testing. The pass rate is a simple percentage of workloads meeting the 1% threshold, with no confidence intervals reported. Accuracy numbers for individual models are point estimates from single evaluation runs. The paper does not discuss train/validation/test splits for calibration data — calibration samples are drawn from the training set or a separate calibration set, but the distinction between calibration data and evaluation data is not formalized. For the BatchNorm calibration experiments (Figure 7), the paper sweeps sample sizes (300, 3K, 10K) and reports the resulting accuracy, but does not average over multiple calibration runs or report variance. This is a meaningful methodological limitation: with 75 models and 200+ tasks, some of the pass rate differences (e.g., E4M3 at 92.64% vs. E3M4 at 90.04% in Table 2) could be sensitive to small per-model variations — a single model passing or failing the 1% threshold shifts the overall pass rate by approximately 1.3 percentage points. The paper does not address this sensitivity.
Calibration data for activation range estimation. For static quantization, scale factors are computed from a small set of calibration samples — typically a few hundred to a few thousand inputs from the training distribution. The paper does not standardize the calibration sample size across experiments; for CV models, the BatchNorm calibration experiments (Figure 7) explore this variable, but for NLP models and the standard scheme, the calibration sample count is not reported. The extended scheme's dynamic quantization eliminates calibration data dependence for activation scaling, but the initial static configuration still requires it.
Hardware and software stack. The experiments use the FP8 Emulation Toolkit for data type emulation and Neural Compressor for model quantization (Section 4.1). Both operate on standard FP32 hardware (CPUs/GPUs), meaning all "FP8" results are simulated — actual FP8 silicon would have slightly different rounding behavior and no performance measurements are possible. This is standard practice in pre-silicon quantization research and does not invalidate the accuracy comparisons, but it means the paper provides no evidence about actual speedup or energy efficiency.

Main Quantitative Results

Workload Pass Rate: The Central Finding

The paper's primary quantitative claim is that FP8 formats substantially outperform INT8 in workload coverage. Table 2 reports the pass rate — percentage of workloads meeting the ≤1% relative accuracy loss criterion — for each format and quantization approach:

Format	Quantization	CV Pass Rate	NLP Pass Rate	Overall Pass Rate
E5M2	Direct	55.26%	78.42%	74.89%
E4M3	Static	73.68%	96.32%	92.64%
E4M3	Dynamic	71.05%	92.11%	88.74%
E3M4	Static	78.95%	92.11%	90.04%
E3M4	Dynamic	78.95%	92.11%	90.04%
INT8	Static CV / Dynamic NLP	57.89%	67.65%	65.87%

The headline result: E4M3 static achieves the highest overall pass rate at 92.64%, compared to 65.87% for INT8. This represents a reduction in failure rate from ~34% (INT8) to ~7% (E4M3) — roughly a 5× reduction in the fraction of workloads that cannot be quantized without exceeding 1% accuracy loss.

Domain-specific format preferences emerge clearly:

For NLP, E4M3 static achieves 96.32% pass rate, substantially exceeding E3M4 static at 92.11%. The gap of 4.21 percentage points corresponds to roughly 1.6 additional models failing among the 38 NLP architectures tested.
For CV, the ordering reverses: E3M4 achieves 78.95% while E4M3 achieves 73.68% — a 5.27 percentage point advantage for the higher-precision, lower-range format.
E5M2 underperforms both E4M3 and E3M4 in both domains, confirming that 5 exponent bits with only 2 mantissa bits trades away too much precision for the additional dynamic range to be beneficial for inference.

INT8's per-domain behavior is revealing:

INT8's CV pass rate (57.89%) is only marginally better than E5M2 (55.26%) and substantially below both E4M3 and E3M4.
INT8's NLP pass rate (67.65%) with SmoothQuant enabled is far below E4M3's 96.32% — roughly one in three NLP models fail under INT8 even with the best-available mitigation for activation outliers. This quantifies the magnitude of the outlier problem that SmoothQuant and related techniques only partially address.
INT8's best configuration differs by domain (static for CV, dynamic for NLP), while FP8's best configuration (E4M3 static) achieves higher pass rates in both domains without domain-specific configuration changes — the same standard scheme works broadly.

Dynamic quantization shows mixed value:

For E4M3, dynamic quantization reduces pass rate compared to static (92.64% → 88.74%), meaning that for most models, static calibration works fine and dynamic quantization introduces no benefit that compensates for its variance.
However, Table 6 shows that for specific NLP models (BERT-base on MRPC: +0.87%; BERT-base on COLA: +0.41%; BERT-large on RTE: +0.98%), dynamic quantization provides meaningful accuracy improvements. This supports the paper's claim that dynamic quantization is a useful targeted knob rather than a universal improvement — it helps specific models but hurts none (the pass rate reduction in Table 2 likely reflects different calibration data splits or evaluation noise rather than degradation, since Table 6 shows only improvements in the direct comparison).

Accuracy Variability: FP8 Shows Tighter Distributions

Figure 4 presents box-plot-like visualizations of accuracy loss variability across CV and NLP workloads for each format. The key observations:

INT8 shows higher variability in accuracy loss for CV models than E4M3 and E3M4. The paper attributes this to INT8 "being ineffective on models such as EfficientNet, MobileNetV3, and ViT" (Section 4.2.1 caption). These are modern architectures — EfficientNet uses squeeze-and-excitation and depthwise convolutions, MobileNetV3 uses hard-swish activations and squeeze-and-excitation, ViT is a transformer — where activation distributions deviate from the well-behaved patterns of classic CNNs like ResNet and VGG that INT8 was developed for.
E4M3 and E3M4 show less variability with very few outliers compared to INT8. The whiskers and individual points in Figure 4 suggest that for FP8 formats, most models cluster tightly near 0% loss, with only a handful showing losses exceeding 2–4%. For INT8, the spread is wider and extends to higher losses.
NLP models show smaller variability overall for all formats, likely because the 38 NLP architectures are predominantly transformer variants with similar activation patterns, whereas the 34 CV architectures span a wider range of design choices (plain CNNs, depthwise-separable CNNs, vision transformers, generative diffusion models).

This variability analysis is important because a format with a high average accuracy could still be problematic if it has a fat tail of models with catastrophically bad quantization — those tail failures block deployment in practice. FP8's tighter distribution means the failure cases are fewer and less severe.

Representative Model Accuracy: Per-Model Comparisons

Table 3 presents accuracy for 10 representative models spanning CV, NLP, speech, and recommendation, enabling per-model comparison across formats. Key observations:

Models where FP8 matches or exceeds INT8 while INT8 degrades significantly:

DenseNet-121 (ImageNet): E4M3 achieves 0.7451, E3M4 achieves 0.7459, both close to FP32 (0.7444) and slightly above it (a known phenomenon where mild quantization noise can regularize and modestly improve accuracy). INT8 drops to 0.7253 — a loss of ~1.9 percentage points absolute, or ~2.6% relative.
Wav2Vec2 (LibriSpeech): E4M3 at 0.9661 essentially matches FP32 (0.9660); E3M4 is close at 0.9658. INT8 drops to 0.9552 — a loss of ~1.1 percentage points.
BERT-base (STS-B): E4M3 at 0.8979 matches FP32 (0.8975); INT8 at 0.8809 represents a ~1.9% relative loss.
LLaMA-65B (lambada-openai): E4M3 at 0.7914 slightly exceeds FP32 (0.7908); INT8 at 0.7155 represents a catastrophic ~9.5% relative loss — the model is effectively broken under INT8. This is the most dramatic single-model comparison in the paper and illustrates why INT8 fails on large language models: LLaMA-65B's activation outliers are of sufficient magnitude that even SmoothQuant cannot fully compensate.

Models where INT8 matches or exceeds FP8:

BERT-large (COLA): INT8 at 0.6389 exceeds both FP32 (0.6257) and all FP8 formats. This is an instance of quantization noise acting as beneficial regularization for a relatively small (110M parameter) model on a small dataset (COLA has 8.5K training examples). The improvement is within the expected noise range for this task.
DistilBert (MRPC): INT8 at 0.9042 exceeds FP32 (0.8916) and all FP8 formats. Similar regularization effect.
Bloom-7B1 (lambada-openai): INT8 at 0.5977 exceeds FP32 (0.5764) and all FP8 formats. This model's name suggests Bloom at 7.1B parameters.

Models where all formats perform similarly:

ResNet-50 (ImageNet): All formats cluster within ~0.007 of FP32 (0.7615). This is the archetypal model where INT8 works well — well-behaved BatchNorm-normalized activations, no extreme outliers.
DLRM (Criteo): All formats within ~0.001 of FP32 (0.8027). Recommendation models with embedding-heavy architectures and well-behaved feature interactions are friendly to all quantization formats.
Bloom-176B (lambada-openai): FP32 at 0.6777; E5M2 at 0.6753; E4M3 at 0.6757; E3M4 at 0.6938; INT8 at 0.6899. All formats maintain reasonable accuracy, though E3M4 and INT8 slightly exceed FP32 — possibly reflecting the benefit of reduced precision acting as regularization for the largest model.

Critical observation about Bloom vs. LLaMA: Bloom-176B quantizes reasonably well under INT8 (0.6899 vs. 0.6777 FP32), while LLaMA-65B collapses (0.7155 vs. 0.7908). Both are large decoder-only transformers. The difference likely stems from architectural choices: Bloom uses ALiBi positional encoding and LayerNorm, while LLaMA uses RoPE and RMSNorm — the normalization scheme in particular affects activation outlier magnitude, and RoPE + RMSNorm is known to produce more extreme activation patterns that challenge uniform quantization. This highlights that INT8's viability is architecture-dependent in ways that FP8's inherent dynamic range makes it more robust to.

Accuracy Loss by Model Size: Larger Is Not Necessarily Harder

Figure 5 presents accuracy loss for all workloads sorted by model size, with ball size proportional to log10(model size). The paper defines size bins: tiny (≤32 MB), small (32–384 MB), medium (384–512 MB), and large (>512 MB). The key findings:

No clear monotonic relationship between model size and quantization difficulty. For both CV and NLP, models of all sizes achieve near-zero accuracy loss under E4M3 and E3M4, and models of all sizes show elevated losses under INT8.
The largest NLP models are not the most difficult to quantize under FP8 — Bloom-176B and LLaMA-65B (the large balls in the NLP panel of Figure 5) show losses well within the 1% threshold for E4M3 and E3M4. The INT8 failures (Llama-65B in particular) are outliers at the extreme.
CV model sizes are broadly smaller than NLP model sizes (reflecting the inclusion of massive LLMs in the NLP category), and the FP8 advantage is more consistent across the CV size range — INT8 shows elevated losses across multiple size bins for CV, while FP8 losses cluster near zero regardless of size.
Some points are overlayed due to near-identical accuracy (the paper notes this explicitly for NLP E4M3 and E3M4), meaning the visual density of points somewhat understates FP8's coverage.

This analysis is important for countering the potential hypothesis that FP8's advantages over INT8 are limited to very large models where outliers are extreme. Figure 5 shows INT8 underperforms across the size spectrum, though the most dramatic individual failures do occur at the large end (LLaMA-65B).

Generation Quality: Qualitative and Quantitative Comparison

The paper supplements accuracy metrics with generation quality comparisons for two output modalities — images (Stable Diffusion) and text (Bloom) — where simple accuracy metrics are insufficient.

Image generation (Figure 6): Stable Diffusion with the prompt "A photo of an astronaut riding a horse on Mars" generates 14 images across different format/quantization configurations. FID scores (lower is better) quantify the results:

FP32 (reference): FID = 86 (dynamic) / 58 (static)
E4M3: FID = 57 (static) / 70 (dynamic) — at or near FP32 quality
E3M4: FID = 40 (dynamic) / 43 (static) — notably better than FP32, which the paper does not explain but may reflect denoising diffusion models benefiting from mild quantization noise
E5M2: FID = 40 (static with LayerNorm) / 36 (dynamic) — competitive with E3M4
INT8: FID = 126 (dynamic) / 108 (static) — substantially worse than all FP8 variants

The paper's subjective analysis notes that "FP8 formats achieve superior image quality compared to INT8, as indicated by the green arrow. Additionally, E4M3 and E3M4 produce smoother images and generate more intricate details, particularly on the astronaut." The green arrow in Figure 6 points to specific regions where FP8 images show more detail. The FID scores align with this subjective assessment: E4M3/E3M4 achieve FID scores comparable to or better than FP32, while INT8's FID scores are roughly 2× worse.

Additional Stable Diffusion samples (Appendix A.2, Figures 11 and 12): The paper includes two more prompts — "A delicious ceviche cheesecake slice" and "The spirit of a tamagotchi wandering in the city of Paris" — with similar patterns. FP8 formats consistently achieve FID scores comparable to FP32, while INT8 degrades visibly and quantitatively. The consistency across three prompts with very different visual content (photorealistic astronaut, food, surreal urban scene) provides some evidence against the possibility that the FP8 advantage is prompt-specific.

Text generation (Table 4 and Appendix Table 7): Bloom generates 100 output tokens from a 32-token prompt "Once upon a time, there existed a little girl..." The paper's qualitative assessment:

E3M4 output (Table 4): "One day, she decided to go on an adventure. She packed her suitcase and went to the airport. She boarded a plane and flew to New York City. There, she met a man, and they had a great time together. They went to a restaurant and ate delicious food. Then, they went to..." — coherent narrative with logical progression and no obvious repetition.
INT8 output: "This little girl was very adventurous. One day she decided to go on a trip to a faraway country. When she got there the little girl saw many strange things. She saw many strange people. She saw many strange animals. She saw many strange sights. She saw many strange smells. She saw many strange sounds. She saw many strange sights..." — a repetitive loop ("She saw many strange...") that signifies degeneration, a known failure mode for quantized language models where quantization error accumulates through the autoregressive generation and pushes the model's internal state toward a low-entropy repetitive regime.
FP32 output: "One day, she decided to go on a trip. She packed her suitcase and went to the airport. When she got there, she found out that there was no flight to her destination, so she decided to take a bus. When she got there, she found out that there was no bus to her destination, so she decided to take a train..." — structurally repetitive (the "when she got there..." pattern) but with varied content, unlike INT8's word-level repetition.

The full outputs in Appendix Table 7 provide extended comparisons. E5M2 generates a repetitive story with mild degeneration ("they had a wonderful time" repeated). E4M3 dynamic generates a coherent shopping list narrative. E4M3 static generates a flight cancellation story with some repetition. E3M4 dynamic generates a surreal animal-eating story (degeneration in a different form). E3M4 static matches the Table 4 output. FP8 mixed generates a coherent hotel/restaurant narrative.

The qualitative analysis is necessarily subjective — no automated metric like BLEU or ROUGE is reported for these generations — but the INT8 degeneration pattern (word-level repetition) is a well-documented failure mode that is clearly visible. The paper's claim that E3M4 "shows better response than INT8 with more comprehensive content and few repeated tokens" is supported by the examples shown, though the sample size (one prompt) limits generalizability.

Extended Operator Coverage: FP8 Maintains Accuracy Through Incremental Expansion

Figure 9 presents the central evidence for FP8's ability to quantize a broader set of operators than INT8. For each format and quantization approach, the figure shows accuracy loss as operator coverage expands incrementally:

CV models (Figure 9a):

Starting from "Conv, Linear — excluding first & last ops" (the standard scheme), the operators are successively added: first/last ops included → no further CV-specific expansion (the subsequent additions — BMM, MM, Embedding, LayerNorm — are primarily NLP operators).
E4M3 and E3M4 maintain tight accuracy loss distributions at all expansion levels. INT8 and E5M2 show higher median loss and greater variability, with INT8 exhibiting several extreme outliers.
The inclusion of first/last ops (the first expansion step) notably increases INT8's accuracy loss spread, confirming the paper's rationale for keeping these in FP32 under the standard scheme for CNNs.

NLP models (Figure 9b):

Starting from "Conv, Linear" (standard scheme, but for NLP models which primarily use Linear rather than Conv), operators are added: + BatchMatMul, MatMul → + Embedding, EmbeddingBag → + LayerNorm.
At each expansion step, E4M3 (both static and dynamic) maintains a tight loss distribution centered near zero with very few outliers. E3M4 shows slightly higher median loss but still well-controlled. INT8 shows higher median loss and substantially more variability, with the distribution widening at each expansion step.
The LayerNorm addition (rightmost group) is the most telling: prior work had found LayerNorm quantization to be impossible under INT8 (Bhandare et al., 2019; Kim et al., 2021). FP8 formats quantize it without meaningful accuracy degradation, enabling truly end-to-end quantized models.

The practical significance of these results is that FP8 enables a single, unified quantization configuration — quantize everything — across a wide range of models, without the per-operator exceptions and fallbacks that INT8 requires. This simplifies deployment pipelines and reduces the surface area for format conversion overhead.

Mixed FP8 Formats: Per-Tensor-Type Optimization Outperforms Uniform Format Assignment

Table 5 compares single-format FP8 quantization against mixed FP8 formats (E4M3 for activations, E3M4 for weights) on four NLP models:

Model	Task	FP32	E5M2	E4M3	E3M4	Mixed
BERT-base	MRPC	0.9069	0.9040	0.9050	0.9050	0.9069
BERT-large	RTE	0.7256	0.6968	0.7329	0.6931	0.7365
Funnel	MRPC	0.9225	0.9215	0.9207	0.3704	0.9233
Longformer	MRPC	0.9146	0.8374	0.9113	0.9084	0.9143

The mixed format matches or exceeds all single-format variants for every model. The Funnel result is particularly dramatic: E3M4 alone collapses to 0.3704 (from 0.9225 FP32 — the model is effectively broken), while mixed formats recover to 0.9233 (exceeding FP32, a regularization effect). This illustrates that mixing formats is not merely an incremental improvement but can be the difference between a functional and a broken quantized model.

Figure 8 provides the mechanistic explanation on BERT-base (MRPC). Measuring MSE between FP32 and quantized outputs of a Linear layer:

Input tensor MSE: E4M3 achieves 11,842; E3M4 achieves 30,810; E5M2 achieves 56,132. E4M3 is best because activations are range-bound (outliers need dynamic range).
Weight tensor MSE: E4M3 achieves 0.16; E3M4 achieves 0.13; E5M2 achieves 1.70. E3M4 is best because weights are precision-bound (normal distribution without outliers).
Output tensor MSE: Single-format best is E3M4 at 22,109. Mixed format (E4M3 activations × E3M4 weights) achieves 919.7 — a ~24× reduction.

The pattern is clear: each tensor type has different quantization sensitivity, and assigning the format that matches each tensor's distribution (dynamic range for outlier-heavy activations, precision for well-behaved weights) compound to dramatically reduce the output error, since the errors from weight quantization and activation quantization interact multiplicatively in the matrix multiply.

Dynamic vs. Static Quantization: Targeted Gains for Specific Models

Table 6 isolates the effect of switching from static to dynamic activation quantization for E4M3 and E3M4 on specific NLP models:

Model	Task	Format	Dynamic	Static	Improvement
BERT-base	MRPC	E4M3	0.9151	0.9072	+0.87%
BERT-base	COLA	E4M3	0.6058	0.6033	+0.41%
BERT-large	RTE	E4M3	0.7401	0.7329	+0.98%
XLM-RoBERTa-base	MRPC	E3M4	0.8962	0.8919	+0.48%

All four comparisons show improvements from dynamic quantization, ranging from +0.41% to +0.98%. These are meaningful gains — for a model hovering near the 1% threshold, a +0.87% improvement could push it from "failing" to "passing." However, Table 2 shows that dynamic quantization reduces overall pass rate for E4M3 (96.32% static → 92.11% dynamic). This apparent contradiction likely reflects that the pass rate aggregates across all 38 NLP models, and the dynamic quantization gains in Table 6 are counterbalanced by other models where dynamic quantization performs slightly worse (perhaps due to calibration data mismatch between the calibration set and the evaluation set for static quantization, versus runtime scale computation for dynamic quantization being noisier for some models).

The paper does not resolve this tension explicitly, but the implication is clear: dynamic quantization is a per-model tuning option, not a universal improvement. The tuning workflow's incremental approach — test static first, switch to dynamic only if needed — captures this correctly.

BatchNorm Calibration: Training Transforms Matter More Than Sample Count

Figure 7 evaluates BatchNorm calibration for CV models, sweeping over sample size (300, 3K, 10K) and augmentation strategy (training transforms vs. inference transforms), reported as accuracy loss for 14 CNN architectures:

Training transforms consistently outperform inference transforms at every sample size. With 300 samples + training transforms, accuracy loss is comparable to 10K samples + training transforms for most models, and both are better than 3K samples + inference transforms.
The paper recommends 3K samples with training transforms for "achieving best results across a wide range of networks," though 300 samples + training transforms is nearly as effective.
Model-specific sensitivity varies widely: ResNet-50 and ResNet-18 are insensitive to both sample size and augmentation strategy (loss near 0%). EfficientNet-B0 and ShuffleNetV2 are most sensitive, with losses exceeding 6% under 3K samples + inference transforms but recovering to near-zero under training transforms.
DenseNet-121 benefits notably from larger sample counts even with training transforms — suggesting that models with dense skip connections may need more samples to stabilize BatchNorm recalibration.

The key practical insight: training-time data augmentation (random crops, flips, color jitter) produces more diverse calibration activations that generalize better to the varied inputs seen at deployment. Inference-time transforms (center crop only) produce less diverse activations and require larger sample sizes to compensate. This is a non-obvious finding — one might expect that calibration should match the inference data distribution, but here more augmentation during calibration is better because it produces robust running statistics that cover the range of inputs the model will encounter.

First and Last Operator Quantization: E3M4 Shows Surprising Robustness

The paper reports (Section 4.3.1) that quantizing the first and last operators in CNNs reduces the pass rate for E5M2 by 25% and for E4M3 by 15%, but E3M4 "can maintain a Pass Rate of 70% even with the first and last operators quantized." This is a counterintuitive result: E3M4 has the most restrictive dynamic range (max value 30.0) of all three FP8 formats, yet it is most robust to quantizing the layers that process raw input pixels (first conv) and produce final logits (last FC). The paper does not explain this finding mechanistically, but it may reflect that the first layer's weight distribution is well-behaved (3×3 or 7×7 conv filters with small values) and E3M4's extra mantissa precision better preserves the fine-grained filter patterns. The paper recommends "the enabling of first and last operators for FP8 quantization as a tuning option" rather than mandating the exception.

Ablation Studies and Robustness Checks

The paper's ablation structure is distributed across the main results and appendices, with several important systematic comparisons:

Range calibration method ablation: The paper states (Section 3) that it "examined more sophisticated range-calibration methods such as KL divergence, MSE error and percentile which did not provide any additional benefits" compared to simple max scaling for FP8 formats. This is reported as a finding, not as a table or figure, making it difficult to assess the magnitude of the null result — the paper does not show side-by-side pass rates or accuracy loss distributions for max vs. KL vs. MSE calibration. Given that KL-divergence-based calibration has been a standard component of INT8 quantization pipelines for years, this is a significant claimed result that deserved more detailed presentation. The Appendix A.1 density analysis provides theoretical justification, but the empirical evidence is anecdotal rather than systematic.

PRM vs. ORM: Not applicable — this paper does not use process reward models or outcome reward models. These terms are absent from the FP8 quantization paper.

SmoothQuant enabled on NLP INT8 baseline: The paper mentions that "SmoothQuant is enabled on NLP models with the default smoothing alpha value" (Section 4.2.1 note), and that "alpha tuning is out of scope in this paper." This is an important experimental design choice because SmoothQuant has been shown to significantly improve INT8 quantization for LLMs (Xiao et al., 2022). By keeping the alpha at its default rather than tuning it per-model, the paper's INT8 baseline may be weaker than what a practitioner could achieve with model-specific SmoothQuant tuning. However, the paper argues that this makes the comparison fairer for FP8 because the FP8 standard scheme also uses fixed hyperparameters without per-model tuning — both formats get the same level of automation. The extended scheme's tuning knobs (mixed formats, dynamic quantization) are the FP8 analog of alpha tuning, and the pass rates include both untuned and tuned configurations. The fair comparison would be: tuned INT8 (with per-model alpha and possibly other mitigations) vs. tuned FP8 (with the extended scheme). The paper's default-SmoothQuant INT8 vs. extended-scheme FP8 comparison gives FP8 an advantage because FP8 gets model-specific tuning while INT8 does not. This is a genuine methodological weakness that the paper does not acknowledge.

Format-specific operator coverage expansion (Figure 9): The figure serves as a de facto ablation by showing how accuracy loss changes as operators are added. For E4M3 and E3M4, the accuracy loss distribution remains stable as operators are added; for INT8, it widens. This demonstrates that FP8's operator coverage advantage is not just about initial accuracy but about robustness to coverage expansion — you can add operators without unexpected accuracy cliffs, which is the property that makes a "quantize everything" strategy viable.

Single format vs. mixed formats (Table 5 and Figure 8): The mixed format results (E4M3 + E3M4) are compared against E5M2, E4M3, and E3M4 individually. The improvement is consistent across the four NLP models shown. However, the paper does not report mixed format pass rates or show mixed format results for CV models — the finding that E3M4 is best for CV raises the question of whether E5M2 + E3M4 mixing could further improve CV pass rates, but this configuration is not evaluated.

Static vs. dynamic quantization for E3M4 on CV: Table 2 shows identical pass rates (78.95%) for static and dynamic E3M4 on CV. This suggests that E3M4's limited dynamic range (max value 30.0) means that max-based static scaling is already close to optimal — runtime scale recomputation provides no benefit because the format's range ceiling is the binding constraint, not calibration accuracy.

BatchNorm calibration sample size and augmentation (Figure 7): This is the paper's most systematic single-variable ablation, sweeping two factors (sample size and augmentation type) across 14 architectures. The interaction effect — training transforms are effective even at small sample sizes — is the non-obvious finding. The paper does not test whether this result generalizes to NLP models (which typically don't have BatchNorm anyway) or to other calibration-dependent quantization steps.

Calibration data dependency: All experiments use calibration data from the training distribution for activation range estimation (static quantization) and BatchNorm recalibration. The paper does not test robustness to distribution shift in calibration data — a practical concern for deployed models where calibration data may differ from deployment data — nor does it report results with varying calibration sample counts for the standard scheme (the BatchNorm calibration sweep in Figure 7 is specific to that step).

Missing ablation: Weight-only vs. weight-and-activation quantization. The paper always quantizes both weights and activations. An ablation comparing weight-only FP8 quantization (activations in FP16) against full FP8 quantization would help disentangle whether the accuracy benefits come primarily from weight quantization (where the dynamic range advantage is less relevant) or from activation quantization (where FP8's outlier-handling should matter most). This ablation is not reported.

Missing ablation: Per-channel activation scaling for FP8. The paper uses per-tensor activation scaling exclusively, citing kernel efficiency concerns. An ablation showing how much additional accuracy per-channel activation scaling would provide for FP8 (even if it is currently impractical) would quantify the remaining headroom and help hardware designers decide whether to implement the necessary kernels. This ablation is not reported, though the paper acknowledges per-channel activation scaling has been shown to benefit INT8.

Missing comparison: FP16 baseline. The paper compares FP8 against INT8 and FP32 but does not include an FP16 baseline. Given that FP16 is the standard inference precision for many GPU deployments (and is supported natively on all modern hardware), the question of whether FP8 provides sufficient accuracy at half the memory bandwidth of FP16 is at least as practically relevant as the comparison against INT8. A three-way comparison — FP32 (upper bound), FP16 (current standard), FP8 and INT8 (competing reduced-precision options) — would better contextualize the results. The absence of FP16 is a notable gap in the experimental design.

Critical Assessment

Claim 1: "FP8 formats outperform INT8 in workload coverage (92.64% vs. 65.87%)"

What the experiments demonstrate: Table 2 shows that under the paper's evaluation protocol, E4M3 static passes 92.64% of workloads while INT8 (with SmoothQuant on NLP) passes 65.87%. This is a genuine and substantial difference, not an artifact of comparing apples to oranges — the standard quantization scheme is matched between formats.

What the experiments do NOT demonstrate: Three important caveats apply.

First, the INT8 baseline may be weaker than the state of the art. The paper enables SmoothQuant with the default alpha value and does not perform per-model alpha tuning. It also does not apply other INT8-specific mitigations that have been shown to be effective for specific architectures: Outlier Suppression (Wei et al., 2022), LLM.int8() mixed-precision decomposition (Dettmers et al., 2022), or GPTQ-style weight quantization (Frantar et al., 2022). The paper's claim is therefore that FP8 with the standard scheme outperforms a basic INT8 pipeline, not that it outperforms the best possible INT8 pipeline a dedicated practitioner could construct. The pass rate gap might narrow (though likely not close) if the INT8 baseline incorporated more advanced techniques. The paper's counterargument is that FP8's standard scheme achieves higher pass rates than INT8's standard scheme, and that FP8's extended scheme provides additional headroom — but the extended scheme comparison is asymmetric because INT8 gets only SmoothQuant while FP8 gets mixed formats, dynamic quantization, and expanded operator coverage.

Second, the pass rate threshold of 1% relative accuracy loss is an arbitrary convention. A deployment with a stricter tolerance (0.5%) would show lower pass rates for all formats, potentially changing the relative ordering if FP8 tends to produce small but non-zero losses while INT8 failures are larger but less frequent. The paper's variability analysis (Figure 4) partially addresses this by showing that FP8's loss distribution is not just shifted but also tighter, suggesting the threshold choice does not distort the relative comparison. However, without reporting pass rates at multiple thresholds, this remains an assumption.

Third, the pass rate is computed over the specific set of 75 architectures and 200+ tasks. The paper describes the selection as "randomly selected from a pool of a combination of diversity and popularity," but what constitutes "popularity" is subjective and time-dependent. Architectures that are popular on Hugging Face in 2023–2024 may not be representative of all production models — in particular, mixture-of-experts models, retrieval-augmented models, and multimodal fusion models (beyond the Stable Diffusion text-to-image pipeline studied here) are underrepresented or absent. The 92.64% figure should be interpreted as "on a diverse set of 75 models drawn from mainstream hubs" rather than "on all deep learning models."

Strength of support: The 27-percentage-point gap is large enough that the qualitative conclusion — FP8 achieves higher pass rates than INT8 — is robust to reasonable variations in baseline strength, threshold choice, and model selection. The exact magnitude is less robust.

Claim 2: "E4M3 is better suited for NLP models, whereas E3M4 performs marginally better on computer vision tasks"

What the experiments demonstrate: Table 2 shows E4M3 at 96.32% NLP pass rate vs. E3M4 at 92.11%, and E3M4 at 78.95% CV pass rate vs. E4M3 at 73.68%. The domain-format interaction is clearly visible and consistent with the mechanistic explanation (NLP activations have more extreme outliers needing dynamic range; CV tensors are more compact and benefit from precision).

What the experiments do NOT demonstrate: The "marginal" nature of the CV advantage (5.27 percentage points) is based on 34 CV architectures — a difference of approximately 1.8 models. With such a small absolute difference, the stability of the finding is questionable. If one additional CV model that passes under E3M4 but fails under E4M3 were to be replaced by a model with the opposite pattern, the gap could shrink or reverse. The paper does not report confidence intervals or statistical significance for this domain-level comparison.

Additionally, the domain categories (NLP vs. CV) conflate architecture type with task type. Vision transformers (ViT) are evaluated under CV in this paper, but they share the transformer architecture with NLP models and might be expected to show NLP-like quantization characteristics. The paper does not break out ViT separately to test whether the domain effect is architectural or task-driven. If ViT behaves more like BERT than like ResNet under FP8 quantization, that would refine the claim from "E4M3 for NLP, E3M4 for CV" to "E4M3 for transformers, E3M4 for CNNs" — a more precise and more useful characterization.

The paper also does not report mixed-format results for CV models (e.g., E5M2 activations + E3M4 weights), which might outperform single-format E3M4 and change the domain recommendation. The extended scheme's mixed format capability is demonstrated only for NLP models (Table 5, Figure 8).

Strength of support: The NLP advantage for E4M3 is strongly supported by a clear 4.21-point gap across 38 models. The CV advantage for E3M4 is more tentative — directionally consistent with the mechanistic explanation but weak enough in magnitude that it could be sensitive to model selection.

Claim 3: "FP8 formats can handle operations such as LayerNorm and BatchNorm"

What the experiments demonstrate: Figure 9 shows that adding LayerNorm and BatchNorm to the FP8 quantization set does not increase accuracy loss for E4M3 or E3M4, while INT8 shows elevated loss and variability. The Stable Diffusion results (Figure 6 and Appendix Figures 11–12) provide evidence for LayerNorm specifically — the FP8 configurations with LayerNorm quantized produce images visually comparable to FP32, while INT8 degrades.

What the experiments do NOT demonstrate: The operator coverage expansion in Figure 9 is cumulative — each step adds operators on top of the previous set. This means we cannot disentangle the individual contribution of each new operator type to accuracy loss. If adding LayerNorm alone (without the preceding BatchMatMul and Embedding expansions) causes no degradation, but the paper does not show this per-operator isolation. The finding that FP8 handles these operators is robust at the aggregate level, but the per-operator sensitivity (is LayerNorm harder than Embedding? Is BatchNorm harder than MatMul?) is not quantified.

The BatchNorm claim is further complicated by the fact that the paper's standard scheme for CV models already excludes first/last operators and applies BatchNorm calibration. The BatchNorm quantization benefit is therefore conditional on applying the correction procedure — it's not that FP8 magically makes BatchNorm quantization safe, but rather that FP8 + BatchNorm recalibration works where INT8 + BatchNorm recalibration does not. The paper could have strengthened this claim by showing BatchNorm quantization results without the recalibration step, isolating the format's contribution from the calibration's contribution.

Strength of support: Strong for the aggregate claim that expanded operator coverage is viable under FP8. Weaker for the per-operator claim about which specific operators benefit most. The LayerNorm finding is the most convincing because prior work consistently identified it as unquantizable under INT8.

Claim 4: "Accuracy-driven automatic model tuning for quantization" is a novel contribution

What the experiments demonstrate: The tuning workflow (Figure 2) sequences standard scheme → BatchNorm calibration → expanded operators → mixed formats → dynamic quantization, and the paper's results show that different models require different levels of this sequence to pass the 1% threshold. Table 2's pass rates for different format/approach combinations implicitly show that some models require the extended scheme (otherwise all static pass rates would be identical). Tables 5 and 6 explicitly show models where mixed formats or dynamic quantization provide accuracy gains beyond the standard scheme.

What the experiments do NOT demonstrate: The paper does not report how many models required each level of tuning, or what the marginal contribution of each tuning step is to overall pass rate. For E4M3, the pass rate goes from 92.64% (static) to 88.74% (dynamic) — but these are different subsets of models, not an incremental improvement on the same set. A reader cannot determine from the reported data whether mixed formats rescued 5 specific models that failed under single-format E4M3, or whether it improved accuracy on models that were already passing.

The paper also does not compare its greedy incremental workflow against alternative tuning strategies — exhaustive search, random search, or no tuning at all. The claim that the workflow is "automatic" is true in the sense that it provides a fixed procedure, but whether it is optimal (finding the best configuration) or efficient (finding a good configuration in few steps) is not empirically evaluated. The priority ordering (BatchNorm first, then operators, then mixed formats, then dynamic) is asserted based on "extensive studies" but the evidence for why this ordering is correct is not systematically presented.

Strength of support: The tuning workflow is a genuine contribution in concept and the paper demonstrates its components improve accuracy on specific models. The claim of "automatic model tuning" is somewhat overstated given the lack of systematic tuning trajectory analysis. The workflow is more accurately described as a structured set of incremental configuration options with an empirically-motivated priority ordering, rather than a fully automated optimization procedure.

Overall Assessment of the Experimental Design

Strengths:

Unprecedented breadth. 75 architectures across 200+ tasks is substantially larger than any prior quantization study. This breadth is the paper's primary evidence for generalizability claims, and it is genuinely impressive.
Fair format comparison. Matching the standard quantization schemes between FP8 and INT8 means the accuracy differences are attributable to the format, not to confounding recipe differences.
Domain-conditioned analysis. Breaking out results by CV and NLP separately, rather than aggregating across domains, reveals the format-domain interaction that the paper's central insight depends on.
Generation quality evaluation. Including Stable Diffusion image generation and Bloom text generation provides evidence beyond simple accuracy metrics for output modalities where quantization artifacts manifest differently.
Transparent reporting of failures. The paper does not hide INT8's comparative strengths (INT8 matches or exceeds FP8 on several small models in Table 3) or FP8's limitations (E5M2's poor performance, E3M4's collapse on Funnel).

Weaknesses:

No statistical rigor. No confidence intervals, no significance tests, no cross-validation. With 75 models, per-model accuracy is a single point estimate. The pass rate is sensitive to small per-model variations that could be noise.
INT8 baseline asymmetry. SmoothQuant with default alpha provides a weaker baseline than what a practitioner could achieve. The paper should have either tuned INT8 to the same degree as FP8 (comparing best-possible vs. best-possible) or kept both at standard-scheme-only (comparing out-of-the-box vs. out-of-the-box). The current comparison mixes these and favors FP8.
Missing FP16 baseline. The practical decision for many deployments is FP8 vs. FP16, not FP8 vs. INT8. The paper provides no data for this comparison.
No latency or throughput measurements. The paper measures only accuracy, not computational efficiency. The claim that FP8 is "efficient" is supported only by the qualitative argument that it enables more operators to be quantized, not by timing data.
Calibration data sensitivity untested. The paper's results depend on calibration data from the training distribution. Robustness to calibration set shift, calibration set size, or calibration set composition is not evaluated beyond the BatchNorm-specific sweep in Figure 7.
Limited per-tuning-step analysis. The paper does not report how many models require each tuning step, what the marginal contribution of each step is, or which models are rescued by which step. The tuning workflow's effectiveness is asserted rather than systematically characterized.
Mixed format evaluation limited to NLP. The paper demonstrates mixed formats (E4M3 + E3M4) for NLP models but does not report equivalent experiments for CV models, leaving open whether CV could benefit from format mixing (e.g., E5M2 + E3M4 for CV activations and weights respectively).
No investigation of the LLaMA vs. Bloom INT8 disparity. Table 3 shows LLaMA-65B collapses under INT8 while Bloom-176B does not. This is a striking result that the paper reports but does not analyze. Understanding whether this is due to RMSNorm vs. LayerNorm, RoPE vs. ALiBi, or some other architectural factor would strengthen the paper's domain-specific claims and is within reach given the authors' access to both models.

Experiments that would have strengthened the paper:

Reporting pass rates at multiple accuracy thresholds (0.5%, 1%, 2%, 5%) to show sensitivity to the threshold choice.
An INT8 baseline with per-model SmoothQuant alpha tuning and/or additional mitigations (LLM.int8() for models where it applies).
An FP16 accuracy baseline for all 75 models.
Per-tuning-step marginal contribution analysis: for each format, what fraction of models pass under standard scheme only, what fraction need BatchNorm calibration only, what fraction need mixed formats, etc.
Mixed format evaluation for CV models.
Latency or throughput estimates based on operator counts and format conversion overhead, even if measured on emulated rather than native hardware.
Analysis of why LLaMA-65B fails under INT8 while Bloom-176B does not — a natural experiment that the existing data partially supports but that the paper does not pursue.
A held-out set of architectures not used during workflow development, to test whether the workflow's priority ordering generalizes beyond the development set (though the 75-model set already serves this purpose to some degree, the paper's description of "randomly selected" models makes it unclear whether any were held out).

6. Limitations and Trade-offs

Limitation 1: The INT8 Baseline Is Not Tuned to the Same Degree as FP8, Giving FP8 a Systematic Advantage in the Headline Comparison

The assumption or constraint. The paper's central claim — that FP8 achieves 92.64% pass rate compared to INT8's 65.87% — depends on a comparison where the two formats receive asymmetric levels of optimization. The INT8 baseline uses SmoothQuant with "the default smoothing alpha value" and the paper explicitly states that "alpha tuning is out of scope in this paper" (Section 4.2.1, footnote). In contrast, FP8 benefits from the extended quantization scheme, which provides three tunable knobs — mixed formats, expanded operator coverage, and dynamic quantization — that are applied incrementally to recover accuracy on failing models. The standard scheme alone (the direct INT8 analog) would yield lower FP8 pass rates, and a fully-tuned INT8 pipeline (with per-model SmoothQuant alpha, LLM.int8() decomposition for extreme outliers, or GPTQ-style weight quantization) would likely yield higher INT8 pass rates than the 65.87% reported. The paper does not compare best-tuned-FP8 against best-tuned-INT8, nor does it compare standard-scheme-only-FP8 against standard-scheme-only-INT8. Instead, it compares standard-scheme-INT8-with-default-SmoothQuant against extended-scheme-FP8, conflating format capability with recipe sophistication.

The consequence. The 27-percentage-point headline gap is best understood as an upper bound on FP8's advantage over INT8, not a precise estimate of the difference that a practitioner would experience when deploying both formats with equal engineering investment. A deployment team willing to invest in per-model INT8 tuning (alpha optimization, outlier decomposition, calibration set curation) could close some fraction of the gap. The paper provides no evidence about how much of the gap is format-inherent versus recipe-asymmetry. This matters for organizations deciding whether to invest in FP8 hardware support and tooling: if the real-world gap is 10–15 points rather than 27, the cost-benefit calculation shifts substantially, especially given that INT8 tooling is mature and ubiquitous while FP8 tooling is nascent.

What evidence exists in the paper. Table 2 reports pass rates for FP8 in both static (standard scheme) and dynamic (extended scheme) configurations, but does not break out how many models required the extended scheme's additional knobs to pass. Table 3 shows specific models where INT8 matches or exceeds FP8 (BERT-large on COLA, DistilBert on MRPC, Bloom-7B1 on lambada-openai), demonstrating that INT8 is not universally worse. The SmoothQuant footnote acknowledges the alpha tuning limitation but does not quantify its impact. Figure 4 and Figure 5 show that INT8 failures include multiple models where accuracy loss substantially exceeds the 1% threshold, but these are evaluated with default-SmoothQuant INT8 — it is unknown how many of these failures would recover with alpha tuning.

Mitigation status. The paper does not address this asymmetry. The authors do not report INT8 pass rates with per-model tuning, nor do they report FP8 pass rates under the standard scheme alone (without extended scheme knobs), nor do they perform a head-to-head comparison where both formats receive an equivalent level of tuning. The claim that the standard quantization scheme is "identical to INT8 quantization scheme, allowing a fair accuracy comparison" (Section 3.1) is true for the standard scheme in isolation, but the headline pass rates are drawn from configurations where FP8 additionally benefits from the extended scheme. Future work could resolve this by reporting the full matrix: standard-only and best-tuned pass rates for both formats.

Limitation 2: No Latency, Throughput, or Computational Efficiency Measurements — All Claims of Efficiency Are Inferred, Not Measured

The assumption or constraint. The paper's title and framing emphasize "efficient" post-training quantization, and the abstract claims that FP8 formats "can meet the computational demands of these modern architectures." However, the paper reports zero timing measurements. All experiments use a software emulation framework (FP8 Emulation Toolkit + Neural Compressor) that simulates FP8 arithmetic on FP32 hardware (Section 4.1). This means the paper provides no evidence about actual inference speedup, latency reduction, throughput improvement, energy savings, or memory bandwidth reduction relative to FP32, FP16, or INT8. The efficiency argument is purely structural: FP8 uses 8 bits per value (like INT8), and the paper demonstrates that it can quantize more operators (LayerNorm, BatchNorm, element-wise ops) and a higher fraction of models than INT8. Whether these structural advantages translate into wall-clock efficiency on real FP8 hardware — which must implement the variable-exponent arithmetic, subnormal handling, and format conversion logic that the paper describes — is unmeasured and unverified.

The consequence. A practitioner reading this paper cannot determine whether deploying an FP8-quantized model will actually run faster or use less energy than the same model in FP16 or INT8. Several open questions are critical for deployment decisions but unaddressed:

FP8 vs. FP16 speedup. FP8 halves memory bandwidth and storage relative to FP16, but FP8 arithmetic (particularly with subnormal support and the non-IEEE encoding rules of E4M3/E3M4) may have different throughput characteristics than FP16 on real hardware. The paper's expanded operator coverage (LayerNorm, BatchNorm, element-wise ops) could improve end-to-end latency by eliminating format conversions, but could also add overhead if FP8 implementations of these memory-bound operations are not as optimized as their FP16 counterparts.
Static vs. dynamic quantization overhead. The paper notes that dynamic quantization "offers no additional benefits to E5M2 but observed a noticeable improvement in accuracy for E4M3 and E3M4 formats on selected models" (Section 3.2). However, dynamic quantization requires computing the maximum absolute value of each activation tensor at runtime — a scan operation over the entire tensor for every layer, every inference. For large tensors, this scan could consume non-trivial time relative to the matrix multiplication itself. Table 6 shows accuracy improvements of 0.4–1.0% from dynamic quantization for specific models, but provides no data on the runtime cost of achieving those improvements.
Mixed format overhead. Assigning E4M3 to activations and E3M4 to weights (Table 5, Figure 8) requires the hardware to support both formats simultaneously and the software stack to manage per-tensor format dispatch. This is architecturally feasible (NVIDIA's H100 supports both E4M3 and E5M2), but introduces dispatch logic and potentially separate kernel implementations. The paper provides no analysis of whether mixed-format inference is slower, faster, or equivalent to uniform-format inference on real hardware.
INT8's mature kernel ecosystem. INT8 inference benefits from a decade of kernel optimization on CPUs (VNNI instructions), GPUs (tensor cores), and dedicated accelerators (TPUs, Inferentia). FP8 hardware support is emerging (H100, Gaudi2) but the kernel ecosystem is far less mature. A theoretically superior format with immature kernels may underperform a theoretically inferior format with highly-tuned kernels — a practical reality that the paper's accuracy-only analysis cannot capture.

What evidence exists in the paper. The paper mentions efficiency only in qualitative terms: static quantization is "computationally more efficient" than dynamic (Section 3.2), per-channel activation scaling is avoided because it "may require special kernel implementations that are likely to incur higher compute overheads" (Section 3.1), and the first/last operator exception exists because those layers "typically constitute < 1% of the total computation" (Section 3.1). The paper reports model sizes (in MB, used for the size bins in Figure 5) but no inference time, no FLOP counts, no memory bandwidth estimates, and no operator-level profiling. The Stable Diffusion and Bloom generation quality results (Section 4.2.2) are the closest the paper comes to evaluating practical deployment quality, but they measure output fidelity, not speed.

Mitigation status. The paper does not address this limitation. The authors do not acknowledge that efficiency claims are unmeasured, do not provide theoretical FLOP or bandwidth analysis to partially compensate for the lack of timing data, and do not speculate about expected speedup ranges. The use of software emulation is standard for pre-silicon quantization research and is acknowledged in Section 4.1, but the gap between "accuracy on emulated FP8" and "performance on real FP8 hardware" is not discussed. As real FP8 hardware becomes available, this limitation becomes addressable by the community, but the current paper provides only the accuracy half of the accuracy-efficiency tradeoff.

Limitation 3: Difficulty Estimation and Calibration Data Dependence — the Workflow Assumes Access to Representative In-Distribution Calibration Data

The assumption or constraint. The paper's post-training quantization workflow depends on calibration data — a small set of representative inputs from the training distribution — for two critical steps: static activation range calibration (computing the max absolute value per activation tensor to determine scale factors) and BatchNorm recalibration (recomputing running mean and variance statistics after quantization). Both steps assume that the calibration data faithfully represents the distribution of inputs the model will encounter at deployment. The paper evaluates all models on their standard test sets (Section 4.1), and calibration data is presumably drawn from the corresponding training sets, meaning calibration and evaluation distributions are matched by construction. The paper does not test robustness to distribution shift — what happens when calibration data differs from deployment data — and does not systematically evaluate the sensitivity of pass rates to calibration sample count, calibration set composition, or calibration data quality.

The consequence. For practitioners deploying models in settings where the training distribution does not perfectly match the deployment distribution — which is the norm, not the exception, in production ML — the paper provides no guidance on how much calibration data is needed, how to select it, or what accuracy degradation to expect when calibration and deployment distributions diverge. Specific failure modes include:

Activation scale miscalibration. If the calibration data underestimates the range of activation values that occur in deployment (e.g., calibration was done on shorter text sequences but deployment involves longer documents), the max-based scaling will produce scale factors that are too large, underutilizing the FP8 encoding space and effectively reducing precision. Conversely, if calibration overestimates the range, scale factors will be too small, causing overflow/clipping for values exceeding the format's maximum representable value (30.0 for E3M4, 448.0 for E4M3).
BatchNorm statistic staleness. The BatchNorm recalibration procedure (Section 3, Figure 2) recomputes running statistics from calibration data. If the deployment distribution shifts relative to the calibration data, the recalibrated statistics become stale in a different way than the original FP32 statistics — they are accurate for the calibration distribution under quantized forward passes, but inaccurate for the deployment distribution under quantized forward passes. The paper's Figure 7 shows that BatchNorm calibration is sensitive to the augmentation strategy used during calibration (training transforms outperform inference transforms), indicating that the diversity of calibration data matters as much as its volume. In deployment settings where the input distribution is itself non-stationary (e.g., user-generated content that shifts over time, or seasonal patterns in recommendation data), a one-time calibration may become outdated.
No guidance on calibration sample count for activation scaling. Figure 7 sweeps calibration sample sizes for the BatchNorm step (300, 3K, 10K) but does not report equivalent sweeps for the activation range calibration step. The standard scheme's activation scaling uses an unspecified number of calibration samples — "a small set of representative input data (a few hundred to a few thousand samples)" is inferred from the discussion but never specified. For models with activation distributions that are sensitive to input characteristics (e.g., LLMs where sequence length affects attention pattern magnitudes), the required calibration sample count may vary across architectures, and the paper provides no methodology for determining it.

What evidence exists in the paper. Figure 7 provides the only systematic calibration sensitivity analysis: for BatchNorm recalibration on 14 CV models, training-time augmentation is more effective than inference-time augmentation, and 300 samples with training transforms is nearly as effective as 10K samples. This suggests that calibration data diversity matters more than volume for BatchNorm statistics. However, this result is specific to the BatchNorm step and CV models — the paper does not provide analogous analysis for activation range calibration or for NLP models (which lack BatchNorm but still require activation calibration for static quantization). Section 3 mentions that KL divergence, MSE, and percentile-based calibration methods were tested and "did not provide any additional benefits" compared to max scaling, but this comparison was done under the calibration-distribution-matches-evaluation-distribution assumption and does not address robustness to distribution shift.

Mitigation status. The paper partially addresses the calibration sensitivity problem through the extended quantization scheme's dynamic quantization option (Mechanism 3 in Section 3.2). Dynamic quantization recomputes activation scale factors at runtime for each input, eliminating dependence on calibration data for activation scaling entirely. Table 6 shows that dynamic quantization improves accuracy for specific NLP models, and Table 2 shows that dynamic E4M3 achieves 88.74% pass rate (vs. 92.64% for static). However, the pass rate drops when switching from static to dynamic for E4M3, suggesting that for most models, the calibration-derived static scales are actually more reliable than per-input dynamic scales — possibly because calibration averages over many inputs, smoothing out per-input noise. Dynamic quantization is thus not a universal solution to calibration dependence; it is a targeted knob for models where static calibration fails, not a replacement for the calibration step. The BatchNorm calibration step has no dynamic equivalent — it is inherently an offline procedure that requires a representative calibration set, and the paper provides no fallback for deployment scenarios where such a set is unavailable or becomes stale.

Limitation 4: Verification Metrics Are Limited to Accuracy and FID — No Evaluation of Quantization's Impact on Reliability, Calibration, or Failure Modes

The assumption or constraint. The paper evaluates quantization quality using task-specific accuracy metrics (classification accuracy, F1, Matthews correlation, Pearson/Spearman correlation, BLEU) and image generation quality (FID). For the text generation results (Bloom, Table 4 and Appendix Table 7), the evaluation is purely qualitative — the authors subjectively assess that E3M4 "shows better response than INT8 with more comprehensive content and few repeated tokens." The paper does not evaluate how quantization affects the reliability of model outputs: confidence calibration (do quantized models produce well-calibrated probability estimates?), robustness to adversarial or out-of-distribution inputs, consistency across semantically equivalent inputs, or the distribution of failure modes (does quantization introduce systematic biases toward certain types of errors?).

The consequence. A model that maintains accuracy within 1% of FP32 on a held-out test set may still exhibit degraded behavior in ways that matter for deployment but are invisible to aggregate accuracy metrics. Specific unmeasured risks include:

Confidence miscalibration. Quantization noise can affect the model's confidence estimates (softmax probabilities) differently than its top-1 accuracy. A quantized model might maintain correct classification decisions while becoming overconfident (producing probabilities near 1.0 for correct predictions) or underconfident, which matters for downstream applications that use these confidence scores for decision thresholds, uncertainty estimation, or human-in-the-loop routing. The paper's 1% accuracy loss threshold would not detect this.
Catastrophic degradation on specific input subspaces. The aggregate accuracy metric can mask severe degradation on small but important subpopulations. For example, the text generation results (Table 4, Appendix Table 7) show INT8 producing degenerative repetitive text ("She saw many strange...") while maintaining some surface-level coherence. A reader evaluating only perplexity or BLEU might miss this failure mode. FP8 formats produce more coherent text in the examples shown, but the paper evaluates on a single prompt — there is no evidence that FP8 avoids similar degeneration modes on other prompts, or that there exist prompts where FP8 degrades while INT8 does not.
Bias amplification. Quantization interacts with model parameters in ways that are not uniform across classes, demographic groups, or input types. A 1% aggregate accuracy drop could be composed of 0% drop on majority classes and 5% drop on minority classes — a pattern that would be invisible in aggregate metrics but highly consequential for fairness-sensitive applications. The paper evaluates on standard benchmarks that may not report per-class or per-group accuracy, and it does not perform any disaggregated evaluation.
Generation diversity and mode collapse. For generative models (Stable Diffusion, Bloom), FID and qualitative assessment capture image/text quality but not diversity. A quantized model might produce high-quality outputs for a narrow range of inputs while collapsing to a single mode for others. The paper generates one image per prompt for Stable Diffusion and one text completion for Bloom, providing no evidence about output diversity or mode coverage.

What evidence exists in the paper. The Bloom text generation outputs (Table 4 and Appendix Table 7) reveal format-dependent differences in output quality — coherence, repetitiveness, narrative structure — that would not be captured by a simple accuracy or perplexity metric. The INT8 output exhibits clear degeneration (word-level repetition: "She saw many strange..."), E5M2 exhibits structural repetition ("they had a wonderful time" repeated), and E3M4 static produces the most coherent narrative. These qualitative differences demonstrate that format choice affects output quality in ways beyond simple accuracy, but the paper does not develop this observation into a systematic evaluation. The Stable Diffusion outputs (Figure 6, Appendix Figures 11–12) show FID scores alongside images, providing a quantitative metric that aligns with visual quality — but FID measures distribution-level similarity between generated and real images, not per-sample fidelity, and may be insensitive to rare failure modes.

Mitigation status. The paper does not address this limitation. The authors do not discuss confidence calibration, subpopulation accuracy, bias, or generation diversity as evaluation dimensions. The qualitative text generation analysis is presented as supplementary evidence rather than a systematic evaluation, and the paper does not propose metrics or methodologies for evaluating quantized model reliability beyond accuracy and FID. This is a common limitation in the quantization literature broadly — the field's evaluation conventions are accuracy-centric — but it is particularly relevant for a paper that advocates replacing the industry-standard INT8 format, since practitioners need to understand not just whether the new format is "more accurate on average" but whether it is "at least as reliable in all the ways that matter for deployment."

Limitation 5: Single Hardware Ecosystem and Emulation-Only Validation — No Evidence from Real FP8 Silicon

The assumption or constraint. All experiments in the paper use software emulation of FP8 arithmetic on FP32 hardware (FP8 Emulation Toolkit + Neural Compressor, Section 4.1). The paper does not evaluate on any hardware with native FP8 support (NVIDIA H100, Intel Gaudi2, or other FP8-capable accelerators). Software emulation faithfully reproduces the mathematical rounding behavior of FP8 formats but cannot replicate several properties of real FP8 hardware that affect deployment viability: the actual throughput and latency of FP8 tensor operations, the energy consumption relative to FP16/INT8, the numerical behavior of fused operations (e.g., fused multiply-add with internal higher precision), or the interaction between FP8 arithmetic and memory subsystems (caching, bandwidth, format conversion overhead at operator boundaries).

The consequence. The paper's claim that FP8 is an "efficient and more productive alternative to INT8" (Section 1) extrapolates from accuracy results to efficiency without the intermediate step — real-hardware validation — that would make this claim credible. Specific uncertainties that emulation-only validation cannot resolve:

Fused operation numerics. Real FP8 hardware typically implements matrix multiplication as a fused operation: multiply in FP8, accumulate in FP16 or FP32, then optionally convert back to FP8. The paper's emulation may implement this differently (e.g., performing the multiply in emulated FP8, accumulating in FP32 software, then quantizing), potentially producing different numerical results than hardware that uses a specific internal accumulation width. The paper does not specify its emulation's accumulation precision.
Throughput cliffs from subnormal handling. The paper specifies that all three FP8 formats support subnormals (Table 1). Hardware implementations of subnormal arithmetic are notoriously slow on many architectures — so much so that some FP16 implementations flush subnormals to zero for performance. If FP8 hardware handles subnormals in microcode or with significant pipeline bubbles, the effective throughput for models whose tensors contain many subnormal values could be substantially lower than the peak advertised throughput. The paper provides no data on how frequently subnormals occur in quantized tensors or how hardware should handle them.
Operator coverage benefits vs. kernel maturity. The paper's demonstration that FP8 can quantize LayerNorm, BatchNorm, and element-wise operations (Figure 9) is an accuracy result. Whether hardware vendors have implemented, optimized, and exposed FP8 kernels for these operators — and whether those kernels actually deliver speedup relative to FP16 implementations of the same operators — is unknown. INT8's operator coverage limitations are well-understood and the industry has adapted by optimizing the FP16 paths for the operators INT8 cannot handle. FP8's theoretical operator coverage advantage may not translate to practical speedup if those kernels are not mature or if the memory-bound nature of these operators means that quantization provides minimal speedup regardless.

What evidence exists in the paper. Section 4.1 acknowledges the emulation framework: "For data type emulation, we utilized the FP8 Emulation Toolkit, which provides a reference implementation that runs FP32 hardware." The paper does not discuss the fidelity of this emulation relative to real hardware, does not specify accumulation precision or rounding modes, and does not compare emulation results against any hardware implementation to validate correctness. The paper's efficiency arguments (expanded operator coverage, fewer format conversions) are qualitative architectural claims, not measured performance results.

Mitigation status. The paper does not address this limitation. The use of software emulation for pre-silicon research is standard and appropriate — the paper was published in 2024, and FP8 hardware was only beginning to become widely available at that time. However, the paper's claims about FP8 being "efficient" and suitable for "production" deployment would require hardware validation that the paper does not provide. The authors do not discuss expected speedup ranges, do not provide theoretical FLOP or bandwidth analysis to bound expectations, and do not acknowledge that emulation-only validation is a limitation of the current study. As FP8 hardware becomes more widely available, this limitation can be addressed by follow-up work, but the current paper's efficiency claims remain unsubstantiated.

Limitation 6: The Workflow Operates as a One-Shot Tuning Procedure with No Dynamic Adaptation at Inference Time

The assumption or constraint. The paper's tuning workflow (Figure 2, Section 3.2) is an offline, one-shot procedure: apply the standard scheme, evaluate accuracy, incrementally add extended scheme components, and fix the configuration that first meets the 1% accuracy threshold. Once a configuration is selected, it is frozen and applied uniformly to all future inputs. There is no mechanism for dynamically adjusting the quantization strategy based on the characteristics of individual inputs at inference time — for example, switching between static and dynamic quantization based on whether an input appears "hard" (containing outlier activations) or "easy," or selecting different FP8 formats for different layers based on runtime activation statistics.

The consequence. This one-shot approach leaves efficiency on the table for models and deployment scenarios where inputs are heterogeneous in ways that affect quantization sensitivity. Specific missed opportunities:

Input-dependent activation range variation. For NLP models in particular, activation magnitudes can vary substantially depending on input length, content, and structure — a short factual sentence produces different attention patterns than a long narrative paragraph. The one-shot static calibration computes scale factors that must accommodate the worst-case input in the calibration set (or use max-based scaling that covers the observed maximum), meaning that for typical inputs, the FP8 encoding space may be underutilized. Dynamic quantization addresses this per-input but applies it uniformly — the paper does not explore a hybrid where some layers use static quantization (for activations with stable ranges) and others use dynamic quantization (for activations with input-dependent ranges).
No adaptation to input difficulty or outlier presence. The paper's Figure 3 (tensor distributions) shows that NLP activations contain outliers whose magnitude varies across inputs. A fixed quantization scheme treats all inputs identically: either it provisions enough dynamic range to handle the worst-case outliers (wasting precision on inputs without outliers) or it clips outliers (losing information on outlier-heavy inputs). An adaptive scheme could detect outlier presence from early-layer activation statistics and adjust subsequent layers' quantization accordingly — for example, using E4M3 for layers where outliers are detected and E3M4 where they are not. This is analogous to mixed-precision inference strategies that route "easy" inputs through a smaller model or lower precision, but applied within a single model's quantization scheme.
No feedback from accuracy evaluation to calibration. The tuning workflow evaluates accuracy on a fixed test set and makes a binary pass/fail decision. If a model fails, the workflow applies the next extended scheme component and re-evaluates, but there is no mechanism for using the pattern of errors to inform what went wrong — which layers contributed most to accuracy degradation, which operator types were most sensitive, or whether calibration data insufficiency caused the failure. The tuning is accuracy-driven in the sense that accuracy is the objective function, but it is not diagnostic — it does not analyze why a configuration failed to guide the next tuning step.

What evidence exists in the paper. The paper's entire workflow is presented as an offline procedure. The discussion of static vs. dynamic quantization (Section 3.2, Table 6) shows that dynamic quantization can provide accuracy gains for specific models, but frames it as a per-model configuration choice rather than a per-input adaptation. The extended quantization scheme's incremental approach (Figure 2, Figure 9) demonstrates that different models need different levels of quantization aggressiveness, which implicitly supports the idea that different inputs to the same model might also benefit from different quantization strategies — but this connection is not made or explored.

Mitigation status. The paper does not address input-adaptive quantization, nor does it discuss it as a direction for future work. The one-shot, offline-tuning paradigm is standard in post-training quantization research and appropriate for the paper's scope. The limitation is not that the paper fails to solve input-adaptive quantization — that would be a separate research contribution — but that the paper's claims about FP8's efficiency advantages do not acknowledge that additional gains might be achievable through input-adaptive strategies, and that the existing one-shot workflow may leave performance on the table for heterogeneous deployment distributions. The paper's concluding Section 5 mentions only future work on applying the recipes to "more diverse LLM models" and contributing to open source, without identifying input adaptation as a research direction.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper makes a diagnostic and engineering contribution rather than a theoretical one, but its practical implications are substantial because it challenges a decade-old industrial default. The shift is best characterized as reframing the quantization format decision from a hardware-driven constraint to an empirically-conditioned design choice. Prior to this work, the implicit assumption across most production ML pipelines was that INT8 is the default reduced-precision format — the question was not whether to use INT8 but how to make INT8 work for a given model, through calibration tricks (KL divergence), mathematical transformations (SmoothQuant), or mixed-precision decomposition (LLM.int8()). FP8, if considered at all, was treated as a training format (for gradients) or a hardware curiosity rather than a deployment-ready alternative for inference.

This paper provides the systematic empirical evidence — 75 architectures, 200+ tasks, apples-to-apples recipe matching — to argue that FP8 should be the first-choice format for post-training quantization, with INT8 demoted to a fallback for legacy hardware. The 92.64% vs. 65.87% pass rate difference (Table 2) is large enough, and the breadth of models evaluated is wide enough, that a deployment engineer reading this paper in 2024 or 2025 should seriously question whether investing in INT8-specific mitigations (SmoothQuant tuning, outlier decomposition, per-channel activation scaling) is a better use of engineering time than simply adopting FP8 — if the hardware supports it. The paper does not settle this question definitively (the INT8 baseline is weaker than the state of the art, and no latency data is provided), but it shifts the burden of proof: the default assumption that INT8 is "good enough" for most models is no longer tenable without evidence that the remaining 34% of models can be recovered through more sophisticated INT8 techniques.

Reconciling prior contradictions. The paper resolves a tension that had been building in the quantization literature without being explicitly articulated. On one side, the INT8 community had developed increasingly sophisticated patches — SmoothQuant, Outlier Suppression, LLM.int8(), GPTQ — each extending INT8's reach to cover transformer architectures that naive INT8 quantization broke. On the other side, the FP8 community (Sun et al., 2019; Micikevicius et al., 2022; Kuzmin et al., 2022) had been demonstrating that floating-point formats naturally handle the outlier distributions that INT8 struggles with, but had not performed the large-scale head-to-head comparison that would make the case definitive. This paper's contribution is to unify these narratives under a single empirical framework: it shows that the patches work (SmoothQuant improves INT8 NLP pass rates, which is why it's enabled), but they are compensating for a format-level limitation that FP8 does not have in the first place. A format with 4 exponent bits (E4M3) achieves 96.32% NLP pass rate without per-model alpha tuning, without outlier decomposition, and without per-channel activation scaling — not because the quantization recipe is more sophisticated, but because the representation is more appropriate for the data distribution. This is a format-level insight, not a recipe-level insight, and it has implications for hardware architects deciding which numeric formats to implement in silicon.

Which research directions become more attractive. The paper's findings redirect research effort along several axes:

Less attractive: developing ever-more-complex INT8 calibration and outlier mitigation techniques for transformers. If FP8 hardware is becoming available (H100, Gaudi2, and successors), the marginal value of making INT8 work for the 34% of models it currently fails on diminishes — those models can simply use FP8. The remaining INT8 research frontier shifts to resource-constrained edge devices where FP8 hardware may arrive later or be more expensive.
More attractive: verifier and calibration robustness for FP8. The paper identifies format-specific behaviors (KL divergence is unnecessary for FP8 range calibration, max scaling suffices; dynamic quantization helps specific models but hurts aggregate pass rates) that suggest FP8 calibration has its own design space that is not simply inherited from INT8 best practices. Understanding why KL divergence doesn't help — and whether there are FP8-specific calibration methods that would — is a newly opened question.
More attractive: format co-design for specific architecture families. The domain-format interaction (E4M3 better for NLP, E3M4 better for CV) suggests that format choice should be part of the model architecture design process, not an afterthought applied post-training. If you know you're building a vision transformer, might you train it with E3M4-aware regularization? If you're building an LLM, might you design the normalization scheme (LayerNorm vs. RMSNorm) with E4M3's dynamic range properties in mind? These questions become tractable once the format-domain mapping is empirically established.
More attractive: end-to-end FP8 inference pipelines. The demonstration that LayerNorm, BatchNorm, and element-wise operations can be quantized under FP8 without accuracy loss (Figure 9) enables a single-format model graph — no format conversions at operator boundaries, no mixed-precision dispatch logic. This simplifies compiler design, kernel implementation, and hardware architecture. Research on optimizing FP8 kernels for memory-bound operations (normalization, element-wise) becomes higher-value because those operations can now be included in the quantized graph.

What the paper does NOT change. The paper does not provide evidence that FP8 inference is faster than INT8 inference on real hardware, nor does it demonstrate that FP8's accuracy advantages translate to deployment settings with distribution shift, adversarial inputs, or long-tailed data. It changes the conversation about which format to target for accuracy-preserving quantization, but it does not close the efficiency question — that requires hardware measurements the paper explicitly does not provide. The paper also does not challenge the fundamental limitation of post-training quantization: if a model's weight or activation distributions contain values that fundamentally cannot be represented in 8 bits (of any format) without damaging the computation, no amount of format tuning will help. The hard-problem failure mode (difficulty bin 5 in the example paper's framework) has an analog here: some models or tasks may resist all 8-bit formats, and the paper provides no evidence about what fraction of the 7% of models that fail under E4M3 might be recoverable with 4-bit or 6-bit formats versus requiring FP16.

Follow-Up Research This Work Enables

1. A controlled experiment isolating whether the E4M3-for-NLP / E3M4-for-CV preference is architecture-driven or task-driven. The paper evaluates vision transformers (ViT) under the CV category and BERT-variants under NLP, but both share the transformer architecture. A follow-up study should take a single transformer architecture (e.g., ViT-B/16 pre-trained on ImageNet vs. BERT-base fine-tuned on a classification task) and evaluate both E4M3 and E3M4 under matched conditions, testing whether the format preference follows the architecture (both should prefer E4M3 if transformers inherently produce outlier-heavy activations regardless of modality) or the data domain (ViT should prefer E3M4 if image patches produce more compact activation distributions than text tokens). The experiment requires: (a) measuring activation outlier statistics (99.9th percentile / median ratio) for corresponding layers in both models, (b) evaluating pass rates for both formats, and (c) ablating the effect of LayerNorm placement, patch size, and sequence length on the format preference. The paper's Figure 3 already shows the activation distribution difference at the aggregate level, but cannot disentangle architecture from data effects because the CV and NLP categories confound both.

2. Training a lightweight difficulty predictor that estimates per-layer quantization sensitivity from a single forward pass. The paper's extended quantization scheme is a one-shot offline tuning procedure that evaluates accuracy after each configuration change and stops when the model passes. This is practical but wasteful — it requires multiple evaluation runs, and it applies the same configuration uniformly to all layers even though quantization sensitivity varies across layers (some layers tolerate aggressive quantization, others require higher precision or specific formats). A follow-up could train a quantization sensitivity predictor: using the calibration dataset, run a single FP32 forward pass, extract per-layer activation statistics (mean, variance, outlier ratio, kurtosis) and per-layer weight statistics (norm, sparsity, eigenvalue spectrum), and train a lightweight classifier to predict whether a given layer will pass under E4M3, E3M4, E5M2, or whether mixed formats or dynamic quantization are needed. The training labels would come from the paper's existing 75-architecture results — for each model and each layer within that model, the accuracy impact of quantizing that layer under each format is known (or can be approximated by selectively quantizing individual layers and measuring the output perturbation). A strong result would be a predictor that, given only FP32 statistics from a single forward pass on a new architecture not in the training set, correctly recommends the quantization configuration that achieves within 0.5% of the best possible accuracy. This would transform the paper's offline tuning workflow into a zero-shot configuration step.

3. A systematic evaluation of FP8 robustness to calibration distribution shift, quantifying the failure boundary. The paper evaluates all models under the assumption that calibration data matches the evaluation distribution (both drawn from standard benchmarks with canonical train/test splits). A deployment-critical question is: how much can the calibration distribution diverge from the deployment distribution before FP8 accuracy degrades unacceptably? A follow-up should take a subset of the 75 architectures (covering both CV and NLP) and systematically shift the calibration data: for CV, use calibration images from ImageNet-Sketch or ImageNet-R (rendition shifts) while evaluating on standard ImageNet; for NLP, calibrate on short sequences (e.g., 128 tokens) and evaluate on long sequences (e.g., 512–2048 tokens), or calibrate on formal text (Wikipedia) and evaluate on informal text (social media). The measurement should include: (a) the rate at which pass/fail status flips as calibration distribution diverges, (b) whether dynamic quantization (which recomputes scales per-input) is more robust to distribution shift than static quantization (this is a plausible hypothesis that the paper's matched-distribution results cannot test), and (c) whether E5M2's massive dynamic range makes it more robust to calibration shift than E4M3/E3M4 despite its lower precision — an important practical tradeoff if calibration data quality is uncertain. The experiment would directly inform deployment decisions: if FP8 is robust to moderate distribution shift (e.g., ImageNet calibration works for iNaturalist evaluation), the calibration dependence limitation (Section 6, Limitation 3) is less severe; if pass rates drop sharply under realistic shifts, then calibration robustness becomes a first-order research priority.

4. A FLOPs-matched or bandwidth-matched comparison of FP8 vs. INT8 inference on real hardware with production-quality kernels. The paper's central efficiency claim — that FP8 is "more efficient and productive" than INT8 — is unmeasured. A follow-up must close this gap by benchmarking FP8 and INT8 inference on the same hardware (H100 or Gaudi2, which support both FP8 and INT8 tensor operations) using the same models (or a representative subset of the 75 architectures) with optimized kernels for each format. The comparison should measure: (a) end-to-end inference latency and throughput at batch size 1 and at maximum batch size, (b) the contribution of memory-bound operators (LayerNorm, BatchNorm, element-wise) to total runtime and whether FP8 implementations of these operators are faster than their FP16 counterparts (which INT8 deployments must use for these operators), (c) energy consumption (if measurable), and (d) memory footprint (model size in bytes). The key question is whether FP8's expanded operator coverage (quantizing LayerNorm etc.) translates into a wall-clock speedup that outweighs any per-operation throughput disadvantages relative to INT8's mature kernel ecosystem. A strong negative result — FP8 is more accurate but not faster on current hardware — would still be valuable because it would focus research on kernel optimization rather than format tuning. A strong positive result — FP8 is both more accurate and faster across a range of models — would accelerate hardware vendors' investment in FP8 support and practitioners' migration away from INT8.

5. Investigating whether FP8 enables training-free "self-healing" through dynamic precision allocation during autoregressive generation. The Bloom text generation results (Table 4, Appendix Table 7) reveal that INT8 generation degrades into word-level repetition while FP8 maintains coherence — a failure mode that emerges from error accumulation across autoregressive steps, not from single-step accuracy loss. This suggests a research direction at the intersection of quantization and generation: can the quantization format be dynamically adjusted during generation to prevent error accumulation? Concretely, an autoregressive model could monitor its internal activation statistics (outlier frequency, entropy of the output distribution) as it generates tokens, and switch from E3M4 to E4M3 (increasing dynamic range at the cost of precision) when it detects that the activation distribution is shifting toward a regime where E3M4's limited range (max value 30.0) causes clipping that degrades subsequent tokens. This is a test-time adaptation strategy analogous to the compute-optimal scaling policy in the example paper, but applied to quantization format selection rather than search algorithm selection. The experiment would: (a) measure per-token activation statistics across long generations (e.g., 512 tokens) for Bloom and LLaMA under E3M4, E4M3, and INT8, (b) identify early warning signals (e.g., fraction of activations near the format's max value, sudden increase in outlier magnitude) that predict downstream degeneration, and (c) implement a simple threshold-based switching policy that changes format mid-generation when warning signals fire. Measuring success requires both automated metrics (repetition ratio, self-BLEU, perplexity on a reference corpus) and human evaluation of coherence, since the paper's qualitative analysis (Section 4.2.2) suggests that simple perplexity may not capture the degeneration phenomenon. This direction connects the paper's format comparison to the broader question of test-time adaptation for language models.

6. Stress-testing the claim that "KL divergence provides no benefit for FP8" across a wider range of activation distributions and outlier magnitudes. The paper reports that KL-divergence-based range calibration — the standard INT8 approach — "did not provide any additional benefits" compared to max scaling for FP8 (Section 3), and provides a theoretical justification based on FP8's non-uniform density (Appendix A.1). This is a strong claim that, if true, simplifies FP8 deployment substantially (no need for sophisticated calibration). But the paper tests this on models where FP8's dynamic range is well-matched to the activation distribution. A stress-test should construct adversarial activation distributions — tensors where a small fraction of values are extreme outliers (e.g., 0.1% of activations at 100× the 3σ value, vs. the 1% at 12σ range tested in Figure 1) — and measure whether max scaling (which stretches the grid to cover the absolute maximum) degrades the representation of the dense central mass more than KL-based clipping (which would clip the extreme outliers and devote more encoding space to the central region). The experiment addresses the boundary condition: at what outlier magnitude does max scaling become counterproductive for FP8, and does KL divergence become beneficial beyond that threshold? If the threshold is never crossed in realistic models, the paper's claim is robust. If it is crossed for some architectures (e.g., models with attention patterns that produce extreme activation spikes), then the calibration recommendation needs qualification: "max scaling is sufficient for most models, but KL divergence should be tested if activation outliers exceed X× the median." The experiment requires: (a) constructing synthetic activation tensors with controlled outlier magnitude and frequency, (b) measuring quantization MSE for max scaling vs. KL divergence across formats, and (c) validating on real models that exhibit the identified outlier pattern. The paper's Figure 1 provides the framework for this experiment but tests only one outlier configuration.

Practical Applications and Downstream Use Cases

1. Cloud-based LLM inference serving. For organizations deploying large language models (LLaMA, Bloom, GPT-variants) behind inference APIs, this paper provides direct guidance: use E4M3 static quantization as the default configuration. The paper shows that LLaMA-65B under INT8 suffers catastrophic accuracy degradation (0.7155 accuracy vs. 0.7908 FP32 on lambada-openai, Table 3 — a ~9.5% relative loss), while E4M3 static achieves 0.7914 (slightly above FP32). For a serving infrastructure that must support diverse customer models without per-model engineering, the paper's central finding — E4M3 achieves 96.32% NLP pass rate vs. INT8's 67.65% (Table 2) — translates to roughly 3× fewer models requiring fallback to FP16 or manual intervention. The direct cost implication: if FP8 hardware provides 2× throughput improvement over FP16 (a reasonable expectation for 8-bit vs. 16-bit tensor operations), and FP8 achieves this throughput on 96% of NLP models while INT8 achieves it on only 68%, the expected throughput gain across a diverse model portfolio is substantially higher for FP8. The paper's recommendation to quantize LayerNorm and element-wise operations in FP8 (Figure 9) further improves end-to-end speedup by eliminating format conversion boundaries.

2. On-device and edge deployment of vision models. For mobile and edge deployments running computer vision models — image classification, object detection, segmentation — the paper's finding that E3M4 achieves the highest CV pass rate (78.95%, Table 2) provides a concrete format recommendation. The practical benefit is not just accuracy but operator coverage simplification: edge inference engines (TFLite, ONNX Runtime, OpenVINO) currently must implement both INT8 kernels (for compute-heavy convolutions) and FP32/FP16 kernels (for normalization and element-wise operations that INT8 cannot handle). The paper's demonstration that FP8 can quantize all operator types without accuracy loss implies that a single-format FP8 runtime is viable — one set of kernels for the entire model graph. For edge hardware designers, this simplifies the instruction set architecture (only FP8 tensor instructions needed, no INT8), and for inference engine developers, it eliminates mixed-precision dispatch logic and format conversion overhead. The specific models where E3M4 excels — EfficientNet-B0, MobileNetV3, ShuffleNetV2 (Figure 7, Figure 5) — are exactly the compact architectures designed for edge deployment, making the format recommendation directly applicable to the target hardware class.

3. Image generation services (Stable Diffusion and similar diffusion models). The Stable Diffusion results (Figure 6, Appendix Figures 11–12) demonstrate that FP8 formats — particularly E3M4 and E4M3 with LayerNorm quantized — achieve FID scores comparable to or better than FP32, while INT8 degrades image quality substantially (FID 108–126 for INT8 vs. 40–57 for the best FP8 configurations). For image generation APIs and creative tools, the implication is that FP8 quantization can reduce serving cost (memory bandwidth and compute) without the visible quality degradation that INT8 introduces. The paper's subjective analysis noting that FP8 generates "smoother images and more intricate details, particularly on the astronaut" (Section 4.2.2) suggests that the quality difference is perceptually significant, not just a metric artefact. A practical deployment pipeline would: (a) quantize the Stable Diffusion UNet and text encoder to E3M4 static with LayerNorm included (the best-performing configuration in Figure 6), (b) use the extended scheme's BatchNorm calibration for any residual normalization layers, and (c) validate on a diverse prompt set to ensure generation quality doesn't degrade for specific prompt types (the paper tests only 3 prompts). The FID improvement from FP8 over INT8 — ~2–3× lower FID scores — is large enough that even if FP8 hardware provides no throughput advantage over INT8 (which is unlikely; both are 8-bit formats), the quality advantage alone would justify the format choice for quality-sensitive applications.

4. Automatic speech recognition (ASR) pipelines. The paper includes two speech models — wav2vec 2.0 and HuBERT — showing that wav2vec 2.0 under INT8 degrades from 0.9660 (FP32) to 0.9552, while E4M3 achieves 0.9661 (Table 3). For production ASR systems where word error rate directly impacts user experience, this ~1.1 percentage point accuracy difference is consequential — a 1% relative increase in word error rate at scale translates to thousands of additional transcription errors per day for a large ASR service. The paper's workflow provides a path to deploy FP8-quantized speech models with accuracy indistinguishable from FP32, which INT8 cannot reliably achieve based on the single-model result shown. The practical deployment consideration is that speech models often run on specialized hardware (DSPs, edge AI accelerators) where FP8 support may arrive later than on data-center GPUs; the paper provides the accuracy evidence that makes the case for hardware vendors to prioritize FP8 in speech-oriented silicon.

When to Prefer This Method

The paper does not explicitly articulate a decision framework comparing FP8 against named alternatives (INT8, FP16) with specific trade-off conditions. It presents FP8 as strictly superior to INT8 in accuracy and workload coverage, with efficiency advantages inferred from expanded operator coverage rather than measured. It does not identify scenarios where INT8 is preferable to FP8 (the INT8-wins cases in Table 3 are treated as anomalous regularization effects, not as evidence for a domain where INT8 should be preferred). The paper also does not compare FP8 against FP16 — the FP32 baseline is the only reference point, leaving the practical FP8-vs-FP16 decision (is 8-bit enough, or do I need 16-bit?) unaddressed. A "When to Prefer This Method" section would thus require imposing a decision framework that the paper itself does not provide, and is omitted.

Dependency tree

Papers this one builds on, walking backwards through the same domain and shared topics.

DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models2023· inference-optimization← in-domain predecessor
- LM-Infinite: Simple On-the-Fly Length Generalization for Large Language Models2023· inference-optimization← in-domain predecessor
INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formats2025· inference-optimization← similar topic
- FLIQS: One-Shot Mixed-Precision Floating-Point and Integer Quantization Search2023· inference-optimization← LLM-judged similar
- LLM-FP4: 4-Bit Floating-Point Quantized Transformers2023· inference-optimization← LLM-judged similar
- ZeroQuant(4+2): Redefining LLMs Quantization with a New FP6-Centric Strategy for Diverse Generative Tasks2023· low-precision← LLM-judged similar
- Boundary-Guided Policy Optimization for Memory-efficient RL of Diffusion Large Language Models2025· inference-optimization← in-domain predecessor
- Pretraining Large Language Models with NVFP42025· pretraining← topic thread
ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers2022· inference-optimization← similar topic
- READ MORE2022· inference-optimization← in-domain predecessor
- BinaryBERT: Pushing the Limit of BERT Quantization2020· low-precision← topic thread
- SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models2022· inference-optimization← similar topic

What it influenced

Later papers in the same chain. Direction reversed.

INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formats2025· inference-optimization→ LLM-judged similar
- KLASS: KL-Guided Fast Inference in Masked Diffusion Models2025· inference-optimization→ in-domain predecessor
- QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMs2025· rl-training→ topic thread
EFFICIENT STREAMING LANGUAGE MODELS WITH ATTENTION SINKS2023· inference-optimization→ in-domain predecessor
- Attention Sink in Transformers: A Survey on Utilization, Interpretation, and Mitigation2026· context-optimization→ LLM-judged similar
- LLM-FP4: 4-Bit Floating-Point Quantized Transformers2023· inference-optimization→ in-domain predecessor
- Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps2026· inference-optimization→ similar topic
ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers2022· inference-optimization→ similar topic
- LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale2022· inference-optimization→ in-domain predecessor
- SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models2022· inference-optimization→ similar topic