"Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization

ArXiv: 2411.02355

🎯 Pitch

Contrary to common fears of catastrophic accuracy loss, this study reveals that FP8 quantization is essentially lossless across all model scales—even outperforming the uncompressed baseline in some cases by falling within evaluation noise. On top of this, standard INT8 methods see only a 1–3% drop, dispelling earlier claims of 10%+ degradation, and INT4 weight-only quantization surprisingly rivals 8-bit formats, matching or exceeding the popular AWQ method on real-world generative tasks.

---

1. Executive Summary

This paper conducts the most comprehensive empirical study to date analyzing the accuracy-performance trade-offs of popular quantization formats—FP8, INT8, and INT4—for large language model inference, evaluating the entire Llama-3.1 model family across over 500,000 measurements spanning academic benchmarks, real-world tasks, and inference performance profiling. The investigation establishes that FP8 weight-and-activation quantization (W8A8-FP, using dynamic per-token activation quantization with symmetric per-channel weight quantization) is effectively lossless across all model scales, that well-tuned INT8 weight-and-activation quantization (W8A8-INT, using GPTQ for weights with SmoothQuant or dynamic activation quantization) achieves only 1–3% accuracy degradation—refuting prior claims of 10%+ drops—and that INT4 weight-only quantization (W4A16-INT) is more competitive than expected, performing on par with 8-bit integer methods and, through a simple GPTQ variant, outperforming the more recent AWQ method on real-world tasks. Through vLLM performance profiling across three GPU architectures and seven deployment scenarios, the paper provides concrete deployment recommendations—W4A16 as the most cost-efficient for synchronous setups (achieving 2–3× cost reduction for 8B and 70B models, and 5–7× for 405B), W8A8 formats maximizing throughput in asynchronous continuous batching, and the optimal choice for mixed workloads depending on the specific use case—establishing that these formats can offer higher accuracy than previously thought while significantly improving inference performance and reducing costs, provided that quantization hyperparameters are carefully tuned rather than applied with default settings.

2. Context and Motivation

The Core Problem: We Don't Actually Know How Much Accuracy Quantization Costs

The fundamental question this paper tackles is deceptively simple: if you quantize an LLM using today's standard formats, how much accuracy do you actually lose? Despite quantization being the most widely used approach for accelerating LLM inference, the field lacked a comprehensive, systematic answer to this question. The paper identifies a genuine gap between the ubiquity of quantization in practice and the reliability of the evidence about its accuracy impact.

This gap exists for several specific reasons that the paper documents:

Prior benchmarks are dominated by academic tasks that don't reflect deployment. Most quantization research relies primarily on structured benchmarks like Open LLM Leaderboard evaluations (MMLU, ARC, GSM8k, etc.) or perplexity measurements. These are multiple-choice or extractive tasks with well-defined correct answers. Real-world deployment involves open-ended generation—chat, code completion, long-context reasoning—where the model must produce coherent, multi-paragraph outputs that are semantically appropriate rather than simply correct. The paper shows that quantized models can rank differently on academic versus real-world tasks (Table 1), meaning conclusions drawn from academic-only evaluations risk being misleading about deployment behavior. The paper specifically points out that "open-ended benchmarks introduce variability, making direct accuracy comparisons challenging" (Section 3.1), which is precisely why prior work often avoided them, creating a blind spot in the literature.

Hyperparameter tuning is absent from most evaluations, leading to unfair conclusions. The paper makes the pointed observation that "the lack of hyperparameter tuning in some studies leads to misleading conclusions about accuracy" (Section 2.2). This is a critical methodological issue: quantizing an LLM is not a single operation but involves choices about calibration data (random tokens, C4, or domain-specific datasets like OpenPlatypus), quantization algorithms (RTN, GPTQ, or AWQ), per-channel vs. per-token vs. per-group granularity, dynamic vs. static activation quantization, and for integer formats, whether to use SmoothQuant to handle activation outliers. Running a quantization method with default parameters and reporting the resulting accuracy as "the performance of INT8 quantization" is like evaluating a fine-tuned model with a poorly-tuned optimizer and concluding fine-tuning doesn't work. The paper directly confronts a specific claim from Lee et al. (2024b) that W8A8-INT causes a 10-point accuracy drop on Open LLM Leaderboard V2 for 405B models. By properly tuning quantization hyperparameters, the authors reduce this to just 0.7 points (Section 2.2). This is not a small correction—it transforms a technique from "unacceptably lossy" to "practically equivalent to full precision."

The largest models are systematically under-evaluated. Lee et al. (2024b), described as the closest prior work, "claim to analyze models up to 405B parameters, [but] they omit open-ended benchmarks at this scale and fail to report full-precision baselines even for academic tasks" (Section 2.2). Without baseline numbers, it's impossible to distinguish quantization-caused degradation from the inherent difficulty of the task for that model. The 405B model is the most expensive to evaluate—requiring multi-node setups—and consequently the least well-characterized in prior work, yet it's precisely where quantization offers the largest practical benefit by enabling deployment on fewer GPUs than otherwise required. The paper makes a deliberate engineering contribution here, enabling "efficient multi-node evaluations for the 405B model" (Section 2.2) to fill this gap.

Contradictory claims about quantization algorithms exist in the literature, unresolved. The paper identifies a concrete disagreement: Lee et al. (2024b) "asserts that AWQ outperforms GPTQ in a 4-bit weight-only quantization setup." The authors directly challenge this, attributing it to "suboptimal hyperparameter choices" in prior work. Their comparative analysis (Table 1) shows both perform similarly on academic benchmarks, but GPTQ shows "notable gains over AWQ in real-world tasks, particularly coding"—a 2.9 vs. 0.8 point advantage on the average real-world score for 8B and 70B models respectively. This disagreement isn't academic: AWQ is widely regarded as the state-of-the-art for INT4 weight quantization, and if GPTQ with proper tuning actually outperforms it, then deployment recommendations based on prior benchmarking are suboptimal.

Why This Problem Matters: Quantization Uncertainty Has Real Consequences

The paper is motivated by practical stakes that extend beyond academic benchmarking:

The Llama-3.1-405B release controversy crystallized the problem. The paper's abstract references "initial skepticism toward the Llama-3.1-405B quantized model release (Dubey et al., 2024)" as a case study in why systematic quantization benchmarks matter. When Meta released quantized versions of their largest model alongside the full-precision weights, there was widespread uncertainty and skepticism about their quality—how much was sacrificed, and in what ways? The paper notes that the quantized model "was later found to be near-lossless in LMSYS Arena user evaluations (Chiang et al., 2024)," meaning the skepticism was unwarranted. But the fact that the community couldn't predict this from existing benchmarks, and instead had to wait for costly human preference data to confirm quality, illustrates how inadequate the existing evaluation ecosystem was. This paper positions itself as providing the kind of rigorous, multi-dimensional evaluation that would have resolved those questions immediately, without requiring deployment and user studies.

The cost implications of getting this wrong are enormous. Quantization is not a niche optimization—it's the primary mechanism that determines whether large models can be deployed at all within budget constraints. The paper's performance analysis across Lambda Labs GPU pricing (Table 9) makes this concrete: a full-precision 405B model on 16×A100 GPUs costs approximately $229/hour for on-demand infrastructure. W4A16-INT quantization reduces this to 4 GPUs while achieving 5–7× cost reduction per query (Table 5). If INT4 were as inaccurate as some prior work suggested, these cost savings would come at an unacceptable quality penalty. If it's as accurate as this paper demonstrates, then organizations who avoided it based on misleading benchmarks left enormous efficiency on the table. The paper's declared aim—"to serve as both a practical deployment guide and a strong and competitive foundation for future research"—reflects this reality: practitioners need trustworthy accuracy-performance-cost data to make infrastructure decisions involving millions of dollars.

Deployment strategy depends on which format to choose, and the answer isn't obvious. The three formats studied (W8A8-FP, W8A8-INT, W4A16-INT) are not just accuracy trade-offs—they have fundamentally different hardware compatibility, performance characteristics, and cost profiles. W8A8-FP requires NVIDIA Hopper or Ada Lovelace architectures with native FP8 support, making it unavailable on widely deployed A100 and A6000 GPUs. W8A8-INT is universally supported but requires per-architecture kernel optimization. W4A16-INT reduces memory bandwidth pressure in the decode phase (where weight access dominates latency) but keeps activations at 16 bits, meaning it accelerates decode more than prefill. Choosing among these without data means either over-provisioning (using FP8 unnecessarily, limiting deployment to Hopper GPUs) or under-provisioning (avoiding INT8 due to exaggerated accuracy concerns, paying for more compute than needed). The paper's deployment analysis across seven use cases—code completion, docstring generation, code fixing, RAG, instruction following, multi-turn chat, and summarization—each with different prefill-to-decode ratios—creates a decision matrix that directly answers the question "which format should I use for my workload?"

Where Existing Approaches Fall Short

The paper identifies specific gaps in prior work that motivate its comprehensive evaluation:

Prior benchmarks don't test semantic quality. Most quantization accuracy studies use metrics like perplexity, multiple-choice accuracy, or exact-match scoring. These capture whether the model's probability distribution has shifted or whether it can still pick the right answer, but they don't capture whether the model's open-ended outputs remain semantically coherent, stylistically consistent, or appropriately detailed. The paper's text similarity analysis (Section 4.4, Figure 1) is motivated by this gap: measuring ROUGE, BERTScore, and STS between quantized and full-precision outputs reveals that INT4 models at small scales introduce "moderate variability in structure but still preserve semantic meaning" (Finding 4), which wouldn't be detectable from accuracy scores alone.

Perplexity does not equal task performance. A widely used proxy in quantization research is the increase in perplexity on a held-out text corpus—smaller increase is interpreted as better quantization. The paper deliberately avoids this approach, instead evaluating on actual downstream tasks (code generation, long-context retrieval, instruction following). This matters because quantization errors are not uniformly distributed: certain layers, attention heads, or token positions may be more sensitive to precision loss than others, and a small average perplexity increase could mask catastrophic failures on specific sub-tasks that actually matter to users.

Hardware diversity is absent from most studies. The paper's performance analysis covers A6000 (48GB, Ampere consumer/entry workstation), A100 (80GB, Ampere datacenter), and H100 (80GB, Hopper datacenter) GPUs across seven use cases with varying prefill/decode token distributions. Most prior work reports performance on a single GPU architecture, if at all. The paper's finding that "W4A16 is the most cost-efficient for synchronous setups, while W8A8 dominates in asynchronous continuous batching" (Finding 5) and that "for mixed workloads, the optimal choice depends on the specific use case" (abstract) is only meaningful because it's tested across the actual diversity of deployment scenarios practitioners face. A study that only benchmarks H100 GPUs would miss that INT8 (without native FP8 support) is the only option for Ampere architectures, or that the memory bandwidth constraints of A6000 GPUs make weight-only quantization relatively more beneficial than on H100s.

The relationship between W8A8-INT and W8A8-FP accuracy is mischaracterized. The paper makes a specific refutation: prior work "claims that 8-bit integer activation quantization causes substantial accuracy degradation (Li et al., 2024a; Lee et al., 2024b)" and asserts "that W8A8-INT is significantly inferior to W8A8-FP and W4A16-INT" (Section 2.2). The paper's evidence directly contradicts this: with proper tuning—using SmoothQuant for activation outliers at the 70B scale, high-quality calibration data, and per-channel symmetric GPTQ for weights—W8A8-INT achieves 99.3% recovery for 405B, 99.9% for 70B, and 100.3% for 8B on Leaderboard V1 (Table 2). On Leaderboard V2, the 405B INT8 model achieves 98.3% recovery (Table 3). For practitioners running on A100 clusters (which lack FP8 hardware support), this finding is critical: they can use INT8 quantization with minimal accuracy loss rather than running full-precision or investing in Hopper hardware.

GPTQ vs. AWQ benchmarking is incomplete and possibly wrong. AWQ has become the de facto standard for 4-bit weight quantization, in part because of claims of superiority over GPTQ on accuracy benchmarks. The paper argues these claims are an artifact of three methodological issues (Section 3.2): using abs-max quantization in GPTQ instead of MSE-optimal clipping (which "has no overhead and yields consistently better results"), using C4 (a generic web corpus) as calibration data instead of higher-quality domain-specific data like OpenPlatypus, and evaluating only on academic benchmarks without testing real-world tasks where the gap is more pronounced (2.9 points vs. 0.23 points on the 8B model's average real-world score in Table 1). Correcting this changes the deployment recommendation: practitioners should use properly tuned GPTQ for INT4 weight quantization, not AWQ.

How This Paper Positions Itself

The paper's explicit contribution is summarized in Figure 4, which shows accuracy recovery trends across model sizes for each format. But the positioning is subtler than just "we did a bigger benchmark." The paper makes several specific claims about what it is refuting or correcting in the literature:

Against prior pessimism about INT8 activation quantization: The paper "challenges the claim that 8-bit integer activation quantization causes substantial accuracy degradation" and states it is providing "vast evidence to the contrary" (Section 2.2). This is not a neutral observation—it's a direct response to Li et al. (2024a) and Lee et al. (2024b), and the paper marshals dozens of data points across model scales and benchmarks to make this case.

Against prior claims about AWQ's superiority: The paper states it "corrects this claim" that AWQ outperforms GPTQ for weight-only INT4 (Section 2.2), again taking a position contrary to published work and supporting it with both academic and real-world evaluation data.

Against prior conclusions about W8A8-INT vs. W8A8-FP: The paper "refutes the conclusion that W8A8-INT is significantly inferior to W8A8-FP and W4A16-INT" (Section 2.2), showing the three formats achieve comparable recovery when properly tuned.

In contrast to prior quantization accuracy studies: The paper explicitly positions itself as broader than accuracy-only work by including inference performance and cost analysis, deploying "across three GPU architectures in seven deployment scenarios" (Section 5) to provide end-to-end deployment guidance rather than just accuracy numbers.

In contrast to Lee et al. (2024b) specifically: The closest prior work is repeatedly cited as having methodological limitations—missing baselines, missing open-ended benchmarks, and suboptimal hyperparameters—that this paper addresses. The 10-point accuracy drop to 0.7 points reduction for W8A8-INT 405B on Leaderboard V2 is presented as evidence that proper methodology changes conclusions (Section 2.2).

As a comprehensive foundation for future work: The abstract states the goal is to "provide the first in-depth study of accuracy vs. performance vs. cost trade-offs for quantized LLMs across formats, algorithms, use cases, and hardware types." The claim here is not just that the benchmarks are larger (500,000+ evaluations), but that the integration of accuracy, text similarity, inference performance, and cost analysis creates a multidimensional picture that no prior study has assembled. The deployment recommendations—W4A16 for synchronous, W8A8 for asynchronous, case-dependent for mixed workloads—are the actionable output of this integration, representing guidance that could not be derived from accuracy or performance benchmarks alone.

The paper also positions itself within the broader discourse about quantization credibility. The phrase "Give Me BF16 or Give Me Death" in the title, adapted from the American Revolution slogan, captures the all-or-nothing attitude that the paper seeks to counter with nuance: that quantization involves measurable trade-offs that, when properly understood and tuned, are acceptable for nearly all real-world applications. The paper's evidence—that FP8 is lossless, INT8 loses 1–3%, and INT4 is competitive with INT8—directly supports the thesis that refusing to quantize at all is an overreaction unsupported by data.

3. Technical Approach

3.1 Reader Orientation

This is primarily an empirical benchmarking paper that constructs a comprehensive evaluation pipeline for quantized LLMs — it is not proposing a new quantization algorithm, but rather systematically testing existing, widely-supported quantization formats (FP8, INT8, INT4) under carefully tuned hyperparameters to establish what accuracy degradation practitioners should actually expect. The core idea is that prior work understated the accuracy of quantized models because of suboptimal hyperparameter choices and evaluation limited to academic benchmarks; by tuning each format properly and evaluating across academic, real-world, and text-similarity dimensions, the paper produces a reliable accuracy-performance-cost decision matrix for practitioners.

The system being built is an evaluation framework — a reproducible pipeline of model quantization (using specific algorithm-format-calibration triples), automated benchmark execution across seven tasks and three model sizes, inference performance profiling on vLLM across three GPU architectures and seven deployment scenarios, and text similarity analysis comparing quantized outputs to full-precision baselines, all totaling over 500,000 evaluations. The problem it solves is the absence of trustworthy, multi-dimensional quantization comparison data: practitioners deploying LLMs need to know which format to use for which workload on which hardware, and prior work either used only academic metrics, omitted baselines for large models, or used default (suboptimal) quantization hyperparameters that painted an unfairly pessimistic picture of what these formats can achieve.

3.2 Big-Picture Architecture (Diagram in Words)

The evaluation framework has five major components, executed in sequence:

1. Model Loading and Quantization — takes a pretrained Llama-3.1-Instruct model (8B, 70B, or 405B parameters) in BF16 precision and applies one of three quantization formats (W8A8-FP, W8A8-INT, or W4A16-INT) using a specific algorithm-format-calibration-data triple. Each format has its own hyperparameters (GPTQ with MSE-optimal clipping for INT8 weights, SmoothQuant for INT8 activations at 70B, dynamic per-token quantization for FP8 activations, OpenPlatypus calibration data for INT4). The output is a quantized model ready for inference.

2. Academic Benchmark Evaluation — runs the quantized model through Open LLM Leaderboard V1 (MMLU, MMLU CoT, ARC-Challenge, GSM8k CoT, HellaSwag, Winogrande, TruthfulQA) and V2 (IFEval, BBH, MATH Level 5, GPQA, MuSR, MMLU-Pro), using both log-likelihood and text-generation evaluation protocols. Scores each task independently and computes accuracy recovery as a percentage of the unquantized BF16 baseline.

3. Real-World Benchmark Evaluation — evaluates the quantized model on four practical tasks: Arena-Hard-Auto-v0.1 (chat and instruction following, using LLM-based judging with 89% human agreement), HumanEval and HumanEval+ (code generation, reporting pass@1 and pass@10), and RULER (long-context retrieval and reasoning at sequence lengths from 4K to 128K). Reports both absolute scores and recovery percentages.

4. Text Similarity Analysis — generates outputs from both the quantized and full-precision models using identical Arena-Hard-Auto-v0.1 prompts with greedy sampling, then computes ROUGE-1 (unigram overlap), ROUGE-L (longest common subsequence), BERTScore (RoBERTa-large token-level similarity), and Semantic Textual Similarity (STS via MiniLM sentence embeddings). Quantifies how closely quantized outputs match unquantized outputs in word choice, structure, and meaning.

5. Inference Performance Profiling — deploys quantized models on vLLM 0.6.4.post1 across three GPU architectures (A6000, A100, H100) and seven deployment use cases (code completion, docstring generation, code fixing, RAG, instruction following, multi-turn chat, summarization), each with specified prefill and decode token counts. Measures latency (synchronous, single-query) and throughput in queries-per-second (asynchronous, batched), then normalizes by Lambda Labs on-demand GPU pricing to compute queries-per-dollar cost efficiency.

Information flows linearly through these components — quantization → accuracy evaluation → text similarity → performance profiling — with accuracy recovery computed relative to unquantized baselines at each stage and cost efficiency computed from performance measurements and hardware pricing.

3.3 Roadmap for the Deep Dive

• First, the quantization formats and their hyperparameter configurations, because every downstream accuracy and performance result depends on how the model was quantized — and a core claim of the paper is that proper tuning changes conclusions.
• Second, the calibration data and algorithm choices (RTN, GPTQ, SmoothQuant, AWQ vs. GPTQ for INT4), since these are the specific knobs that the paper argues prior work set suboptimally.
• Third, the benchmark evaluation protocol, covering how academic, real-world, and text-similarity benchmarks are executed, scored, and normalized to produce recovery percentages — this defines what accuracy means in the paper.
• Fourth, the inference performance measurement methodology on vLLM, including how synchronous vs. asynchronous deployment affects the prefill/decode trade-off and how cost efficiency is computed from GPU pricing — since the paper's actionable outputs are deployment recommendations.
• Fifth, the reasoning model extension, showing how the same quantization approach transfers to DeepSeek-R1-Distill models for reasoning benchmarks, validating generalizability.

3.4 Detailed, Sentence-Based Technical Breakdown

This is an empirical evaluation paper whose core technical contribution is a carefully tuned quantization-and-evaluation pipeline rather than a novel quantization method. The central thesis is that much of the reported accuracy degradation from quantization is an artifact of hyperparameter choices, and that with proper tuning, FP8 is lossless, INT8 loses only 1–3%, and INT4 is competitive with INT8 across academic and real-world tasks.

---

W8A8-FP Quantization: 8-Bit Floating-Point Weights and Activations

W8A8-FP quantizes all linear operators in the transformer blocks to 8-bit floating-point representation, targeting NVIDIA Hopper (H100) and Ada Lovelace architectures that have native FP8 tensor core support. This is the simplest quantization format in the paper's study — it requires no calibration data, no iterative optimization, and no outlier handling beyond what the floating-point format inherently provides.

Weight quantization for FP8. Weights are quantized using symmetric per-output-channel quantization, meaning each output channel (a row of the weight matrix in the linear layer) gets its own scale factor, but no zero-point offset. The quantization is performed via round-to-nearest (RTN) assignment. Given a weight vector $x \in \mathbb{R}^d$ (the weights for a single output channel), the quantization operation is:

$Q_{\text{FP8}}(x) = s \cdot \text{rnd}(x / s)$

where $s = \max(|x|) / q_{\max}$ is the per-channel scale factor, $\max(|x|)$ is the maximum absolute value in that channel's weight vector, $q_{\max}$ is the maximum representable value in the FP8 format, and $\text{rnd}$ rounds to the nearest representable FP8 value.

What it computes: the scale factor $s$ maps the weight vector's dynamic range to the available range of the FP8 format, and the division $x / s$ projects each weight into the quantized range before rounding. The output is a vector of FP8 weight values that approximate the original BF16 weights within the precision limits of the 8-bit format. For inference, the quantized weight is used directly in FP8 matrix multiplications on the tensor cores.

Why this form: symmetric quantization (no zero-point) is simpler than asymmetric quantization because it avoids storing an additional offset per channel, reducing memory and simplifying the dequantization step during matrix multiplication. Per-channel granularity is chosen over per-tensor because different output channels can have substantially different weight magnitude distributions, and a single per-tensor scale would force smaller-magnitude channels to be quantized with coarser effective precision. Round-to-nearest is the standard quantization operator and requires no calibration or optimization.

Activation quantization for FP8. Activations are quantized using dynamic per-token quantization, meaning each input token's activation vector is quantized independently with its own scale factor, computed on-the-fly during inference. For a given activation tensor with shape $[\text{seq\_len}, \text{hidden\_dim}]$, each row (corresponding to one token position) is quantized separately:

$Q_{\text{FP8}}(a_i) = s_i \cdot \text{rnd}(a_i / s_i)$

where $a_i \in \mathbb{R}^{\text{hidden\_dim}}$ is the activation vector for token $i$, $s_i = \max(|a_i|) / q_{\max}$ is the per-token scale factor computed from that token's activation magnitudes, and $q_{\max}$ is the FP8 format maximum.

What it computes: for each token position, the system dynamically measures the range of activation values, computes a scale factor that maps them to the FP8 range, and quantizes. This happens during the forward pass with no precomputation or calibration.

Why this form: dynamic quantization is chosen over static quantization (where activation scales are precomputed from a calibration dataset and fixed during inference) because activation distributions vary significantly across different inputs and even across different token positions within the same input. FP8 has enough dynamic range (the exponent bits) that per-token dynamic quantization works well without the complexity of activation smoothing or outlier handling that integer formats require. The paper notes this approach "requires no calibration data and remains computationally efficient, even for large-scale models" (Section 3.2), which is a significant practical advantage — quantizing a 405B model with W8A8-FP involves no data collection, no iterative optimization, and runs in minutes.

Key design insight. The combination of symmetric per-channel weight quantization and dynamic per-token activation quantization makes FP8 effectively lossless (99.7–100.1% accuracy recovery across all model sizes on Leaderboard V1, Table 2). The authors attribute this to FP8's floating-point representation being naturally robust to the outlier features that plague integer activation quantization — the exponent bits give floating-point formats a much wider dynamic range than integer formats of the same bitwidth, so occasional large activation outliers don't clip or saturate the quantization range in the way they would for INT8.

---

W8A8-INT Quantization: 8-Bit Integer Weights and Activations

W8A8-INT quantizes both weights and activations to 8-bit integers, targeting NVIDIA Ampere and older architectures that lack native FP8 support but have efficient INT8 tensor core operations. This format is significantly more complex to tune than FP8 because integer quantization is vulnerable to activation outliers — individual activation values that can be up to 100× larger than the average (Dettmers et al., 2022, cited in Section 2.1) and would dominate the quantization range, forcing most values to be quantized with very low effective precision.

Weight quantization for INT8. Weights are quantized using GPTQ (Frantar et al., 2022) with symmetric per-output-channel quantization. GPTQ is a post-training quantization method that uses a small calibration dataset to iteratively adjust weights to minimize the layer-wise output error caused by quantization. The process works as follows:

1. Collect calibration data: run a set of calibration input samples through the model up to the layer being quantized, collecting the input activations to that layer. For the 8B model, "random tokens suffice" (Section 3.2); for larger models (70B, 405B), the paper uses higher-quality calibration data from Lee et al. (2023) — specifically, the OpenPlatypus dataset, which contains instruction-following examples in a format similar to the model's intended use case.

2. Layer-wise reconstruction: for each linear layer with weight matrix $W \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$, GPTQ quantizes weights column-by-column (or row-by-row for per-output-channel) and, after quantizing each column, updates the remaining unquantized weights to compensate for the quantization error introduced. The objective is to minimize the Frobenius norm of the output error:

$\min_{\hat{W}} \|W X - \hat{W} X\|_F^2$

where $X \in \mathbb{R}^{d_{\text{in}} \times N}$ is the matrix of input activations from the calibration data (with $N$ calibration samples), $W$ is the original BF16 weight matrix, and $\hat{W}$ is the quantized weight matrix.

3. Symmetric quantization: the quantized weights use symmetric per-output-channel quantization, meaning each output channel $i$ has a scale factor $s_i$ and the quantized value is $\hat{w}_{ij} = s_i \cdot \text{round}(w_{ij} / s_i)$, with no zero-point offset. The scale factor is $s_i = \max(|w_{i,:}|) / 127$ for INT8 (the range is $[-127, 127]$ since INT8 uses $\{-127, \dots, 127\}$ for symmetric quantization, reserving -128 for the zero-point in asymmetric setups).

What this computes: GPTQ produces a quantized weight matrix $\hat{W}$ that, when multiplied by the calibration activations $X$, produces outputs $\hat{W}X$ that are as close as possible (in Frobenius norm) to the original outputs $WX$. The key mechanism is the sequential column-wise quantization with error compensation — after quantizing each column, the remaining columns are adjusted using the inverse Hessian of the calibration activations to absorb the quantization error, which is why GPTQ requires calibration data and why the quality of that data matters.

Why GPTQ over simple RTN: standard round-to-nearest quantization treats each weight independently and produces output errors that accumulate across columns. GPTQ explicitly minimizes the output error by adjusting other weights to compensate, which is especially important for INT8 because the limited range relative to FP8 means naive RTN can produce large cumulative errors in layers with high sensitivity to small weight perturbations. The paper's choice of GPTQ with per-channel granularity (rather than per-group, which would require storing more scale factors) balances accuracy and memory efficiency.

Why calibration data matters: the input activations $X$ determine the Hessian that GPTQ uses to decide which unquantized weights to adjust after quantizing each column. If the calibration data is not representative of the actual inference-time input distribution, the Hessian will be poorly estimated and the weight adjustments will be suboptimal. The paper finds that "random tokens suffice at 8B, but larger models require higher-quality calibration data" (Section 3.2), indicating that larger models have more complex activation patterns that need representative calibration. The choice of OpenPlatypus (instruction-following data) over the generic C4 web corpus is motivated by alignment with the instruction-tuned model's intended use.

Activation quantization for INT8. This is where the paper addresses the central challenge of integer activation quantization: outlier features. The approach uses two strategies depending on model size:

For 8B models: dynamic per-token quantization, directly analogous to the FP8 activation approach but with 8-bit integers. Each token's activation vector is quantized independently using its own scale factor computed from the token's activation range. This works because smaller models exhibit less severe activation outliers — the dynamic per-token quantization already captures the per-input variation without additional smoothing.

For 70B models: SmoothQuant (Xiao et al., 2022, cited in Section 3.2). The paper observes that "this scheme performs well for 8B and 405B models, [but] it causes noticeable accuracy drops at 70B. To mitigate this, we apply SmoothQuant" (Section 3.2). SmoothQuant addresses a specific problem: in transformer models, activation outliers are systematically concentrated in specific feature dimensions (channels) and are stable across inputs. Rather than extracting these outlier columns at runtime (which is inefficient, requiring sparse compute), SmoothQuant mathematically migrates the quantization difficulty from activations to weights, which are easier to quantize because they don't have the same outlier problem.

The SmoothQuant transformation works as follows: for a linear layer computing $Y = XW$ (where $X$ is activations and $W$ is weights), insert a per-channel scaling factor $\text{diag}(s)$ and its inverse:

$Y = XW = (X \cdot \text{diag}(s)^{-1}) \cdot (\text{diag}(s) \cdot W) = \tilde{X}\tilde{W}$

The scaling factors $s_j$ for each input channel $j$ are chosen to equalize the quantization difficulty between activations and weights:

$s_j = \max(|X_{:,j}|)^\alpha / \max(|W_{j,:}|)^{1-\alpha}$

where $\alpha \in [0, 1]$ is a migration strength parameter (typically $\alpha = 0.5$). When $\alpha < 1$, activation magnitudes are reduced (making them easier to quantize) and weight magnitudes are correspondingly increased (the weights absorb the removed magnitude). The transformed activations $\tilde{X} = X \cdot \text{diag}(s)^{-1}$ have reduced outliers, and the transformed weights $\tilde{W} = \text{diag}(s) \cdot W$ are quantized with GPTQ as described above.

Why SmoothQuant at 70B but not at 8B or 405B: the paper does not directly explain this pattern, but the implication from the results is that 8B models have mild enough outliers that dynamic per-token quantization handles them, while 405B models have such distributed representations that individual channel outliers don't dominate to the same degree. The 70B scale hits an intermediate regime where outliers are problematic for pure dynamic quantization but can be addressed by activation smoothing. This is a practical finding rather than a theoretical prediction — it emerged from the hyperparameter tuning process.

For 405B models: dynamic per-token quantization, same as 8B. The paper reports that this works well without SmoothQuant for the largest model, with W8A8-INT achieving 99.3% recovery on Leaderboard V1 (Table 2).

Activation quantization procedure at inference time. After any SmoothQuant preprocessing, the activation quantization at inference time follows: for each token position, compute the activation range, derive a per-token scale factor, and quantize to 8-bit integers. The quantized activations $\tilde{X}_{\text{int8}}$ and quantized weights $\tilde{W}_{\text{int8}}$ are then used in INT8 matrix multiplication on the tensor cores, with the result accumulated in higher precision (typically INT32) and dequantized back to FP16/BF16.

---

W4A16-INT Quantization: 4-Bit Integer Weights, 16-Bit Activations

W4A16-INT quantizes only weights to 4-bit integers while keeping activations at 16-bit half-precision (FP16/BF16). This is the most aggressive compression format in the study, reducing weight memory by 4× relative to BF16 (from 16 bits to 4 bits per weight), but it leaves activations uncompressed — meaning the matrix multiplications still operate on 4-bit weights multiplied by 16-bit activations, accumulating in higher precision. This format primarily accelerates the decode phase of inference, which is memory-bound (limited by how fast weights can be read from GPU memory), while the prefill phase remains compute-bound (the 16-bit activations mean the tensor cores compute at 16-bit throughput).

Weight quantization algorithm: GPTQ with MSE-optimal clipping. The paper uses GPTQ for INT4, but with a crucial modification over standard GPTQ: MSE-optimal clipping rather than absolute-maximum range determination.

In standard GPTQ with abs-max clipping, the quantization range for a group of weights is determined by the maximum absolute value, and weights outside this range are clipped (saturated) to the maximum representable value. For a group of $g$ consecutive weights (typically $g = 128$), the scale factor would be $s = \max(|x|) / 7$ for signed INT4 (range $[-7, 7]$, with -8 unused for symmetric quantization). This means a single large weight in the group forces the scale factor to be large, reducing the effective precision for all other weights in the group — they get quantized to coarser steps because the range must accommodate the outlier.

MSE-optimal clipping addresses this by allowing some weights to be clipped if doing so improves the overall mean squared error of the quantization. Instead of using $\max(|x|)$, the algorithm searches for an optimal clipping threshold $\alpha \cdot \max(|x|)$ with $\alpha \in (0, 1]$ that minimizes:

$\text{MSE}(\alpha) = \sum_{i=1}^g (x_i - Q_\alpha(x_i))^2$

where $Q_\alpha(x_i)$ is the quantization operation with clipping threshold $\alpha \cdot \max(|x|)$. Weights with $|x_i| > \alpha \cdot \max(|x|)$ are clipped to the maximum quantized value, while the remaining weights use a finer scale factor $s = \alpha \cdot \max(|x|) / 7$, giving them better effective precision. The optimization over $\alpha$ is done per-group and can be performed efficiently via grid search since $\alpha$ is a scalar.

What this computes: for each group of 128 consecutive weights, the system finds the clipping threshold that optimally trades off clipping error (from the few weights that get saturated) against quantization error (from the many weights that benefit from a smaller scale factor). The output is a quantized weight group where most weights have higher effective precision than they would under abs-max clipping, at the cost of a few weights being saturated to the maximum representable value.

Why this form: the paper explicitly states that using MSE-optimal clipping is one of three reasons their GPTQ outperforms AWQ, noting it "has no overhead and yields consistently better results" (Section 3.2). The key insight is that in any weight distribution, a small number of weights may have much larger magnitudes than typical. Clipping these outliers and using a tighter range for the majority of weights reduces the average quantization error because the increased precision for many weights outweighs the large error on a few clipped weights. This is analogous to the rationale for activation quantization outlier handling, but applied to weights.

Calibration data for INT4. Unlike W8A8-INT where "random tokens suffice at 8B" (Section 3.2), the paper states that for INT4, "random token calibration degrades accuracy, so we rely on OpenPlatypus data for calibration" (Section 3.2). The increased sensitivity to calibration quality at 4 bits makes intuitive sense: with only 4 bits per weight (16 possible values), the quantization error per weight is much larger than at 8 bits, so the GPTQ error compensation step needs more accurate Hessian estimates from representative calibration data to make effective adjustments. OpenPlatypus, containing instruction-following examples aligned with the model's intended use, provides activation patterns that better approximate the inference-time distribution than generic web text.

Group-wise quantization granularity. Weights are quantized in 128-element groups (Section 3.2). This means every 128 consecutive weights along the input dimension share one scale factor. Per-group quantization is a middle ground between per-channel (one scale per output channel, which for large hidden dimensions means one scale for thousands of weights) and per-tensor (one scale for the entire matrix, which loses too much precision at 4 bits). The group size of 128 is standard in the GPTQ literature (Frantar et al., 2022) and balances the storage overhead of scale factors (one FP16 scale per 128 weights = 0.125 bits per weight overhead, negligible relative to the 4-bit weight) against the precision benefit of finer-grained quantization.

Inference with W4A16. At inference time, the 4-bit weights are dequantized to FP16 on-the-fly during the matrix multiplication. The computation proceeds as: read the compressed 4-bit weights and their FP16 scale factors from GPU memory, reconstruct the approximate FP16 weights as $\hat{w}_{ij} = s_{g(i,j)} \cdot w_{ij}^{\text{int4}}$ where $s_{g(i,j)}$ is the scale factor for the group containing weight $(i,j)$, then compute the matrix multiplication $Y = \hat{W} X$ using standard FP16 tensor core operations (since activations are FP16). The dequantization overhead is absorbed into the memory-bound decode phase, where the 4× reduction in weight memory bandwidth dominates any additional dequantization computation.

Why W4A16 rather than W4A4 or W4A8: the paper notes that "more recent high-compression techniques... target low-bitwidths using advanced representations such as vector quantization. Yet, these formats are inefficient for batch sizes larger than 1, limiting their practicality" (Section 2.1). W4A16 is the most aggressive format that still has robust kernel support in vLLM and efficient execution across batch sizes. Keeping activations at 16 bits avoids the activation outlier problem entirely — there's no need for SmoothQuant or dynamic quantization — which is why the paper observes that W4A16-INT "performs on par with W8A8-INT" (Table 3) and that "for INT, quantizing activations is harder than quantizing weights" (Section 4.1).

---

GPTQ vs. AWQ for INT4: Hyperparameter Choices That Change Conclusions

The paper makes the specific claim that GPTQ outperforms AWQ for 4-bit weight quantization when properly tuned, contradicting prior work such as Lee et al. (2024b). Understanding this claim requires examining the hyperparameters that differ between the paper's GPTQ configuration and what it argues was suboptimal in prior AWQ comparisons.

AWQ (Activation-Aware Weight Quantization) protects "salient" weight channels by scaling them up before quantization. The method identifies which input channels have consistently large activation magnitudes (and are therefore more important for the layer's output), then scales up the corresponding weight channels by a learned per-channel factor before quantization, quantizes all weights to 4 bits, and scales the weights back down after quantization. The core idea is that scaling up important weights gives them relatively more precision in the 4-bit representation, while unimportant weights can tolerate coarser quantization.

The paper's GPTQ configuration differs from typical GPTQ-AWQ comparisons in three ways (Section 3.2):

1. MSE-optimal clipping in GPTQ rather than abs-max. The paper states that "the AWQ comparison used abs-max" quantization in GPTQ, which the paper found to be suboptimal. MSE-optimal clipping provides a finer-grained optimization per weight group that, the authors argue, closes or reverses the gap that AWQ achieves through its per-channel scaling approach.

2. Higher-quality calibration data. The paper uses OpenPlatypus data rather than C4 (a generic Common Crawl corpus that is the default in many GPTQ implementations). This matters because GPTQ's Hessian-based error compensation relies on calibration data being representative of the inference distribution. C4 contains generic web text that may not match the activation patterns of instruction-tuned models responding to chat prompts. OpenPlatypus, being an instruction-following dataset, produces activation distributions closer to what the model sees during actual deployment.

3. Broader evaluation scope including real-world benchmarks. The paper argues that AWQ and GPTQ look similar on academic benchmarks (Leaderboard V1/V2) but diverge on real-world tasks. In Table 1, the 8B model shows only a 0.23-point gap on academic benchmarks (AWQ leading) but a 2.9-point gap on real-world benchmarks (GPTQ leading). For the 70B model, the gap is 0.35 points on academic but 0.8 points on real-world benchmarks, again in GPTQ's favor. The paper attributes this to the fact that "GPTQ exhibits notable gains over AWQ in real-world tasks, particularly coding" (Section 3.2).

The HumanEval+ pass@1 results in Table 3 provide specific evidence: for the 8B model, GPTQ achieves 67.1% on HumanEval and 59.1% on HumanEval+ pass@1, while AWQ achieves 63.0% and 62.8% respectively (Table 1). For the 70B model, GPTQ achieves 80.5% on HumanEval and 74.2% on HumanEval+ pass@1, while AWQ achieves 79.4% and 74.0% respectively — both differences clearly favoring GPTQ.

---

Benchmark Evaluation Protocol

The paper's evaluation protocol is designed to produce trustworthy accuracy comparisons by (1) optimizing quantization hyperparameters on a development set, (2) evaluating on held-out benchmarks, and (3) computing recovery as a percentage of the unquantized baseline for each model size and task.

Academic benchmarks: Open LLM Leaderboard V1 and V2. The paper uses V1 as the primary hyperparameter tuning target and V2 as validation, explicitly stating it is "ensuring generalization by optimizing quantization hyperparameters on V1 while validating results on V2" (Section 4.1). This two-benchmark split prevents overfitting — if hyperparameters were tuned to minimize degradation on both benchmarks, the reported recovery percentages would be inflated by overfitting to the specific tasks.

Leaderboard V1 tasks and evaluation modes. The seven tasks in V1 (MMLU, MMLU CoT, ARC-Challenge, GSM8k CoT, HellaSwag, Winogrande, TruthfulQA) use "Meta's prompt guidelines for Llama-3.1 models to maintain alignment with baseline scores" (Section 4.1). This introduces two non-standard evaluation choices: MMLU and ARC-Challenge are assessed as text-generation tasks rather than the standard log-likelihood-based evaluation (where the model's probability for each answer choice is compared and the highest-probability choice is selected). In the generation mode, the model must produce the correct answer text in an open-ended format, which is a harder task that relies more on the model's generative capabilities — and is more sensitive to quantization effects. GSM8k is tested with chain-of-thought prompting (8-shot) rather than direct answer generation, meaning the model must produce multi-step reasoning before its final answer, which stresses the quantized model's ability to maintain coherent long-form reasoning.

Leaderboard V2 tasks and normalization. V2 incorporates six tasks (IFEval, BBH, MATH Level 5, GPQA, MuSR, MMLU-Pro) that are "more challenging" (Section 4.1). The paper notes that V2 "normalizes scores by subtracting the random baseline and rescaling to a 0-100 range, ensuring equal weighting across tasks regardless of inherent difficulty." This is important because tasks with different random baselines (e.g., multiple-choice with 4 options has a 25% random baseline, while a task with 10 options has 10%) would otherwise contribute unequally to the average score. The normalization formula is:

$\text{norm\_score} = \frac{\text{raw\_score} - \text{random\_baseline}}{100 - \text{random\_baseline}} \times 100$

where $\text{random\_baseline}$ depends on the number of choices for multiple-choice tasks or is 0 for generative tasks.

Recovery percentage computation. For every quantized model and benchmark, the paper reports recovery as:

$\text{Recovery (\%)} = \frac{\text{Quantized Score}}{\text{BF16 Baseline Score}} \times 100\%$

This normalizes across model sizes (which have dramatically different absolute scores — the BF16 405B scores 86.79 on V1 while the BF16 8B scores 74.06) and allows direct comparison of how much each quantization format degrades performance relative to the model's own uncompressed capability. A recovery above 100% means the quantized model actually scored higher than the full-precision baseline (which can happen due to evaluation noise or because quantization acts as a form of regularization that benefits certain tasks).

Real-world benchmarks: Arena-Hard-Auto-v0.1. This benchmark automates the evaluation of chat and instruction-following quality by using an LLM judge (rather than human evaluators) to compare model responses to 500 complex prompts. The paper reports that this approach achieves "89% agreement with human rankings" (Section 3.1), sourced from Li et al. (2024c) who validated the automated judge against the human rankings from the LMSYS Chatbot Arena. Each model generates a response to each prompt, and the judge model evaluates the response quality relative to a reference model (typically GPT-4 or similar). The reported score is a win rate — the percentage of prompts where the evaluated model's response is judged as better than or equal to the reference.

The paper runs each quantization configuration on Arena-Hard twice (Table 7 in Appendix A.1 reports scores for both runs and their average) and computes 95% confidence intervals. This repeated evaluation is important because LLM-based judging introduces stochasticity — the same response may receive different judgments on different runs — and the confidence intervals quantify how much of the observed gap between quantization formats is statistically reliable versus noise. The paper observes that "quantized models exhibit competitive response quality, with overlapping 95% confidence intervals across all configurations" (Section 4.2 and Appendix Table 7), meaning that at the 95% confidence level, none of the quantization formats can be distinguished from each other or from the BF16 baseline on Arena-Hard — a strong result suggesting that chat quality is robust to quantization at these bitwidths.

HumanEval and HumanEval+ code generation. These benchmarks test the model's ability to generate correct Python code from natural language descriptions. HumanEval contains 164 programming problems; HumanEval+ extends this with additional test cases to catch false positives (where the generated code passes the simple test cases but fails on edge cases). The paper reports pass@1 — the fraction of problems where the first generated solution passes all test cases — using the EvalPlus library (Liu et al., 2023a) for standardized evaluation with greedy sampling (temperature = 0) to ensure deterministic, reproducible results. Pass@10 results (best of 10 samples) are also reported in Appendix Figures 5 and 6.

RULER long-context evaluation. RULER tests the model's ability to use information across long contexts by creating tasks that require retrieving, aggregating, or reasoning over information placed at various positions within sequences ranging from 4K to 128K tokens. The paper reports the RULER score aggregated across subtasks. The 405B model is excluded from RULER evaluation because it would be "prohibitively expensive for our cluster" (Table 3 note) — running long-context evaluation on a 16-GPU setup with 128K token sequences would require enormous memory and runtime.

Text similarity analysis protocol. This analysis quantifies how similar the quantized model's outputs are to the full-precision model's outputs when given the exact same prompts. Using Arena-Hard-Auto-v0.1 prompts with greedy sampling (temperature = 0) for "full reproducibility" (Section 4.4), the system generates one output per prompt from both the BF16 baseline and each quantized variant, then computes four metrics:

• ROUGE-1: fraction of unigrams (individual words) in the quantized output that also appear in the BF16 output, measuring word-level overlap. Range [0, 1].
• ROUGE-L: longest common subsequence between the two outputs divided by the BF16 output length, measuring structural similarity (are the same sequences of words appearing in the same order?). Range [0, 1].
• BERTScore: uses RoBERTa-large embeddings to compute token-level contextual similarity — each token in the quantized output is matched to the most similar token in the BF16 output based on their contextual embeddings, and the average similarity across tokens is reported. Range [0, 1]. Unlike ROUGE, this captures semantic similarity even when different words are used (e.g., "fast" and "quick" get partial credit).
• Semantic Textual Similarity (STS): uses Sentence Transformers built on MiniLM to compute sentence-level semantic similarity via cosine similarity of sentence embeddings. Range [0, 1]. This captures whether the overall meaning of the response is preserved even if the specific words and structure differ.

All metrics are normalized to [0, 1], with 1 indicating identical output. The paper aggregates these metrics across all Arena-Hard prompts for each model size and quantization format, producing the heatmap shown in Figure 1.

---

Inference Performance Measurement Methodology

The inference performance analysis in Section 5 quantifies the actual speed and cost benefits of each quantization format across realistic deployment scenarios. This is not a theoretical FLOPs analysis — it measures end-to-end latency and throughput on actual GPU hardware running vLLM, the most widely used open-source LLM serving framework.

vLLM configuration. All performance measurements use vLLM version 0.6.4.post1 (Section 5), a specific version pinned for reproducibility. vLLM implements PagedAttention (Kwon et al., 2023), which manages the key-value (KV) cache in non-contiguous memory blocks, reducing memory fragmentation and enabling higher batch sizes. The paper does not specify the full vLLM configuration (e.g., maximum batch size, block size, scheduling policy), but states that "setups were optimized for peak BF16 performance and kept consistent when evaluating quantized models" (Section 5.2) — meaning any configuration that improved BF16 throughput was applied to all quantization formats equally, making the relative comparisons fair.

GPU hardware selection. Three GPU architectures are tested, representing different market segments and quantization support:

• A6000 (48GB): Ampere architecture, consumer/entry workstation GPU with 48 GB VRAM. Supports INT8 tensor core operations but not FP8. This represents cost-sensitive deployments (Lambda Labs pricing: $0.80/hr, Table 9).
• A100 (80GB): Ampere architecture, datacenter GPU with 80 GB HBM2e VRAM. Supports INT8 but not FP8. The most widely deployed datacenter GPU for LLM inference at the time of the paper. Pricing: approximately $1.79/hr per GPU (8×A100 at$14.32/hr, proportional pricing assumed).
• H100 (80GB): Hopper architecture, datacenter GPU with 80 GB HBM3 VRAM. Supports both INT8 and native FP8 tensor core operations. Pricing: $3.29/hr per GPU (Table 9).

GPU count per model size. The paper allocates GPUs per model to keep the full-precision BF16 baseline feasible:

• 8B: 1× GPU (all architectures)
• 70B: 2× or 4× GPUs depending on the experiment (2×A100 for synchronous, 4× for asynchronous to maximize throughput; 4×A6000 for asynchronous)
• 405B: 16× or 4× GPUs (16×A100 and 16×H100 for BF16 synchronous baselines; 4×A100 and 4×H100 for INT4 synchronous which fits on fewer GPUs due to weight compression)

This allocation reflects real-world practice: larger models require multi-GPU tensor parallelism to fit in VRAM at full precision, but quantization reduces memory pressure enough to enable fewer GPUs.

Seven deployment use cases with prefill/decode specifications. The paper defines seven representative inference scenarios with specific input and output token lengths, shown in Table 5 (synchronous) and Table 6 (asynchronous):

These use cases span the range of prefill-to-decode ratios, which is the key determinant of which quantization format is optimal: long-prefill tasks (summarization, RAG) are prefill-heavy and benefit more from weight-and-activation quantization (which accelerates the compute-bound prefill), while long-decode tasks (code completion) are decode-heavy and benefit more from weight-only quantization (which accelerates the memory-bound decode by reducing weight memory bandwidth).

Synchronous deployment measurement. In synchronous mode, vLLM processes a single query at a time — no batching, no concurrent requests. This eliminates resource contention between queries but underutilizes the GPU's computational capacity. The paper measures:

• Latency: end-to-end wall-clock time (in seconds) from receiving the prompt to producing all output tokens, including prefill and decode.
• Queries per USD: $\text{Q/\$} = \frac{1}{\text{Latency (s)}} \times \frac{3600}{\text{GPU cost per hour}}$. This represents how many complete requests can be processed per dollar of GPU time if queries arrive sequentially. Higher is better.

The synchronous results in Table 5 also report a Cost Reduction factor (labeled "CR"): the ratio of the BF16 baseline cost per query to the quantized model's cost per query. For example, W4A16-INT for 8B on A6000 has CR = 2.39, meaning cost per query is 2.39× lower than BF16 (or equivalently, the user can process 2.39× more queries for the same dollar amount).

Asynchronous deployment measurement. In asynchronous mode, vLLM processes multiple concurrent queries, interleaving their prefill and decode phases to maximize GPU utilization. The paper measures:

• Queries per second (QPS): the maximum steady-state throughput achievable, measured as the number of complete requests (full prefill + decode) processed per second.
• Queries per USD: $\text{Q/\$} = \text{QPS} \times \frac{3600}{\text{GPU cost per hour}}$. This normalizes throughput by cost, enabling fair comparison across GPU types and counts.

The asynchronous measurements optimize for peak throughput — the vLLM configuration is tuned to maximize QPS for each model/hardware combination, then the same configuration is applied to all quantization formats. This means the relative speedup (reported as "Speedup" in Table 6) isolates the effect of quantization rather than configuration differences.

Latency-throughput curves. For the asynchronous setting, the paper also produces latency-throughput trade-off curves (Figures 2 and 3) by varying the arrival rate of requests (how many queries per second are submitted). At low arrival rates, latency is low because queries don't queue; at high arrival rates, throughput increases but latency grows because queries wait in the batch queue. These curves show:

$$\text{Inter-Token Latency (ms)} = \frac{\text{Time between generating successive tokens for a query}}{\text{}}$$

On the x-axis, QPS increases as the system processes more concurrent requests. The key insight from Figures 2 and 3 is that W4A16-INT consistently achieves lower latency at a given throughput (or higher throughput at a given latency) in the low-to-moderate throughput regime, while W8A8 formats overtake it at high throughput. This is because W4A16 reduces memory bandwidth pressure in the decode phase, which dominates at low batch sizes, while W8A8 improves computational throughput in the prefill phase, which becomes the bottleneck at high batch sizes when many requests are being prefilled concurrently.

Cost efficiency normalization. All cost calculations use Lambda Labs' on-demand GPU pricing (Table 9), which the paper states is "standard" (Section 5). The prices are:

• 1×A6000: $0.80/hr
• 4×A6000: $3.20/hr
• 2×A100: approximately $3.58/hr (proportional to 8×A100 at$14.32/hr)
• 4×A100: approximately $7.16/hr
• 16×A100: $28.64/hr
• 2×H100: $6.38/hr (Table 9 lists$6.38/hr for 2×H100; note this differs from 16×H100 which would be 8×$23.92 = higher per-GPU due to multi-node interconnect)

---

Reasoning Model Extension: DeepSeek-R1-Distill Quantization

The paper extends its quantization approach to DeepSeek-R1-Distill models (Section 4.3) to test whether the findings generalize to reasoning-specialized models that have been fine-tuned with distillation for improved chain-of-thought reasoning. This is a critical extension because reasoning models have different activation patterns than standard instruction-tuned models — they produce long chain-of-thought trajectories that may exhibit different quantization sensitivity.

Models evaluated. The paper tests six DeepSeek-R1-Distill variants spanning two model families and scales from 1.5B to 70B parameters:

• Llama-8B, Llama-70B (distilled from DeepSeek-R1 into Llama architecture)
• Qwen-1.5B, Qwen-7B, Qwen-14B, Qwen-32B (distilled into Qwen architecture)

Benchmarks. Three reasoning-specific benchmarks are used:

• AIME 2024: American Invitational Mathematics Examination problems, testing advanced mathematical reasoning.
• MATH-500: 500 competition-level math problems from the MATH benchmark (Lightman et al., 2023), testing step-by-step mathematical reasoning.
• GPQA-Diamond: graduate-level physics, chemistry, and biology questions (Rein et al., 2024), testing expert scientific reasoning.

Evaluation protocol for reasoning models. Unlike the standard benchmarks that use greedy or few-shot evaluation, the reasoning benchmarks use sampling with temperature 0.6 and top-p 0.95, generating 20 responses per query to estimate pass@1. The paper explains this choice: "the repetitive sampling was important to estimate an accurate average performance for the benchmarks due to high variance across the relatively small datasets" (Section 4.3). The AIME 2024 and GPQA-Diamond datasets are small (on the order of tens to low hundreds of problems), so a single greedy evaluation would have high variance and could misattribute sampling noise to quantization effects. By averaging pass@1 over 20 samples, the paper reduces the standard error of the mean by approximately $1 / \sqrt{20} \approx 4.5\times$, giving more reliable recovery estimates.

Quantization configuration for reasoning models. The paper applies the same quantization pipeline — W8A8-FP, W8A8-INT, and W4A16-INT with the same hyperparameter configurations — to the reasoning models without additional tuning. This tests transferability: whether quantization hyperparameters optimized on Llama-3.1-Instruct for standard benchmarks also work well for reasoning models on reasoning benchmarks. The results in Table 4 show that they do — average recovery is above 99% for all formats except INT4 on the smallest models (Llama-8B at 97.2%, Qwen-7B at 98.3%, Qwen-1.5B at 93.5%), confirming that the paper's quantization approach generalizes across model families and task types.

---

Summary of Critical Design Choices

The paper's technical approach rests on several non-obvious design decisions that directly affect the reported results:

Calibration data quality over quantity. The paper consistently chooses domain-relevant calibration data (OpenPlatypus, Lee et al. (2023) data) over generic defaults (C4, random tokens) when accuracy is sensitive to calibration. This is not merely "using better data" — it reflects an understanding that GPTQ's Hessian approximation requires activations that match the inference-time distribution, and that instruction-tuned models have different activation patterns than base models trained on web text. The paper's observation that "random token calibration degrades accuracy" for INT4 (Section 3.2) but works for INT8 at 8B suggests that the calibration sensitivity increases with compression ratio.

MSE-optimal clipping for GPTQ INT4 rather than abs-max. This is the single hyperparameter change that the paper argues reverses the GPTQ vs. AWQ comparison. It costs nothing in terms of inference overhead (clipping thresholds are determined offline during the quantization process) but provides a consistent accuracy improvement by optimizing the trade-off between clipping and quantization error per weight group.

Dynamic per-token activation quantization as the default for FP8 and for INT8 at most scales. This avoids the complexity and potential brittleness of static calibration for activations. For FP8, the floating-point dynamic range makes this work naturally. For INT8, it works at 8B and 405B but requires SmoothQuant at 70B, which the paper presents as an empirical finding rather than a predicted result — they discovered the 70B INT8 degradation and applied SmoothQuant in response, which is precisely the kind of hyperparameter tuning that distinguishes their results from prior work that applied a one-size-fits-all configuration.

Two-run averaging with confidence intervals for Arena-Hard. The paper explicitly accounts for evaluation noise in the LLM-based judging by running two independent evaluations and computing 95% confidence intervals, then uses the overlap of these intervals (rather than point estimates) to draw conclusions about format differences. This methodological rigor supports the paper's claim that quantized models are competitive — the intervals overlap, meaning the observed differences could be noise.

Consistent vLLM configuration across quantization formats for throughput benchmarking. By optimizing the vLLM setup for BF16 and then freezing it for all quantization evaluations, the paper ensures that throughput differences are due to the quantization format's computational characteristics, not configuration artifacts. The alternative — tuning vLLM separately for each format — would maximize throughput for each but would confound the quantization effect with configuration effects, making it unclear whether format A outperforms format B because the quantization is better or because the scheduler was accidentally better tuned for A.

4. Key Insights and Innovations

Innovation 1: Quantization Accuracy Is Hyperparameter-Dependent, Not Format-Intrinsic — A Methodological Reframing

The paper's deepest intellectual move is not any specific accuracy number but the demonstration that the accuracy of a quantization format is not an intrinsic property of the bitwidth or numerical representation — it is a function of the tuning pipeline applied to it. This reframes quantization from a hardware characteristic (e.g., "INT8 loses X%") to an optimization problem with tuneable parameters, where the reported degradation in prior work was often measuring suboptimal configurations rather than the format's achievable ceiling.

Prior work — the paper explicitly cites Li et al. (2024a) and Lee et al. (2024b) — treated quantization formats as fixed entities with known accuracy costs. Lee et al. (2024b) reported a 10-point accuracy drop for W8A8-INT 405B on Open LLM Leaderboard V2 "compared to FP8" (Section 2.2). This number entered the community's mental model as "what INT8 costs." The paper reduces this to 0.7 points — a 14× reduction in measured degradation — by changing only the quantization hyperparameters: using GPTQ with per-channel symmetric quantization rather than a coarser scheme, applying SmoothQuant at the 70B scale where activation outliers become problematic, and selecting high-quality calibration data (OpenPlatypus/Lee et al., 2023) rather than generic web text or random tokens.

What makes this a conceptual innovation rather than just "better engineering" is the diagnostic implication: it means the field cannot trust any single reported accuracy-degradation number without understanding the hyperparameter search behind it. Two papers reporting different accuracies for the same format on the same model are not necessarily contradicting each other — they may reflect different points in the tuning space, and neither may represent the format's actual capability. This parallels the recognition in deep learning that "ResNet-50 accuracy on ImageNet" is meaningless without specifying the training recipe (optimizer, augmentation, epochs, learning rate schedule). Quantization accuracy is similarly recipe-dependent, and the paper provides the first systematic evidence of how large the gap between default and tuned configurations can be.

The specific methodological contributions that operationalize this insight — GPTQ with MSE-optimal clipping for INT4, SmoothQuant for INT8 activations at 70B, calibration data selection — are important but secondary. The primary contribution is the framing shift: from "which format is better?" (a question that assumes formats have fixed quality) to "what is the achievable accuracy for each format given careful tuning?" (a question that acknowledges the optimization problem). This reframing is fundamental rather than incremental because it changes how future quantization research should be evaluated — papers must now demonstrate that their comparisons control for hyperparameter quality, or risk the criticism that they are comparing a tuned method against an untuned baseline, exactly as this paper argues Lee et al. (2024b) did with GPTQ vs. AWQ.

The evidence for this innovation is not a single table but the pattern across Tables 2, 3, and 4: W8A8-INT achieves 99.3–100.3% recovery on Leaderboard V1, 97.3–101.5% on V2, and >99.6% on reasoning benchmarks (except at 1.5B scale), directly contradicting prior claims that INT8 activation quantization causes substantial degradation. The recovery percentages are not the innovation — the demonstration that those numbers were achievable all along, and were hidden by methodological choices, is the innovation.

---

Innovation 2: Academic Benchmarks Systematically Underestimate Quantization Robustness — The Real-World Validation Gap

The paper's second major intellectual contribution is the empirical demonstration that academic benchmarks and real-world benchmarks can rank quantization formats differently, and that evaluations restricted to structured academic tasks produce an incomplete — and in some cases misleading — picture of quantization's practical impact. This is not simply "more benchmarks are better." It is a specific claim about evaluation validity: the tasks for which quantization accuracy matters most in deployment are precisely the tasks least represented in standard academic suites, and the ranking of methods can invert between academic and real-world settings.

The clearest evidence for this inversion is Table 1's comparison of GPTQ and AWQ for INT4 weight quantization. On academic benchmarks (Leaderboard V1 + V2 average), AWQ leads GPTQ by 0.23 points for the 8B model and 0.35 points for the 70B model — differences indistinguishable from noise on a 0–100 scale. On real-world benchmarks (Arena-Hard + HumanEval + MBPP average), the gap reverses and widens: GPTQ leads AWQ by 2.9 points for 8B and 0.8 points for 70B. An evaluator using only academic benchmarks would conclude the methods are equivalent or marginally favor AWQ; a practitioner deploying for code generation or chat would experience a meaningful quality difference in the opposite direction.

What makes this a conceptual contribution rather than just a benchmarking detail is that it identifies a systematic bias in how the field has evaluated quantization. Academic benchmarks like MMLU, ARC, and GSM8k are structured tasks with well-defined correct answers and short, extractive responses. Quantization errors in these settings manifest as the model selecting the wrong multiple-choice option or making arithmetic errors that change the final answer — errors that are relatively large in magnitude (a completely wrong answer) and thus appear significant in accuracy metrics. Real-world tasks like code generation and multi-turn chat involve long, open-ended outputs where quantization errors may manifest as subtle changes in word choice, code structure, or conversational style — differences that affect quality but may not produce complete failures. The paper's text similarity analysis (Section 4.4, Figure 1) provides direct evidence: quantized 8B models show moderate drops in ROUGE-L (structural preservation dropping from 0.51 to 0.41) but maintain BERTScore above 0.90 and STS above 0.94, meaning the semantic content is preserved even when surface form varies. An accuracy-only evaluation would miss this entirely.

This insight has direct consequences for how the field should evaluate future compression techniques. Method developers optimizing for leaderboard position may inadvertently tune for academic tasks that are insensitive to the subtle degradation modes that matter in deployment — exactly the pattern the paper identifies with AWQ's academic advantage vanishing on real-world tasks. The implication is that real-world open-ended benchmarks should be mandatory for quantization evaluations claiming practical relevance, not supplementary.

The innovation is fundamental rather than incremental because it challenges the validity of the primary evaluation paradigm used by the quantization community. If the ranking of methods depends on whether you evaluate on MMLU or HumanEval, then the literature's conclusions are conditional on benchmark choice in ways that prior work did not acknowledge or control for.

---

Innovation 3: Quantization Formats Have Deployment-Regime-Dependent Optimality — The Synchronous vs. Asynchronous Divide

The paper's performance analysis reveals a finding that is conceptually straightforward but had not been systematically demonstrated before: no single quantization format dominates across deployment regimes; the optimal choice depends on whether the workload is latency-constrained (synchronous) or throughput-constrained (asynchronous), and on the prefill-to-decode token ratio of the specific use case. W4A16-INT is the most efficient for synchronous single-query deployments, achieving 2–3× cost reduction for 8B/70B models and 5–7× for 405B (Table 5), while W8A8 formats (both FP8 and INT8) maximize throughput in asynchronous continuous batching (Table 6).

The mechanism behind this divide — that W4A16 accelerates the memory-bound decode phase by reducing weight memory bandwidth, while W8A8 accelerates the compute-bound prefill phase through lower-precision matrix multiplications — is well-understood at the architectural level. What the paper contributes is the integration of this mechanism with accuracy data to produce deployment-contingent recommendations. Prior work either studied accuracy in isolation (without performance data) or performance in isolation (without accuracy validation), making it impossible for a practitioner to answer the question: "I'm deploying a chat application on A100 GPUs — should I use INT8 or INT4 weights?" This paper answers: if it's synchronous (latency-sensitive, one query at a time), W4A16 is 2–3× cheaper per query with minimal accuracy loss; if it's asynchronous (high throughput, batched queries), W8A8-INT achieves 1.64–1.92× the throughput (Table 6, speedup column) while maintaining 99%+ accuracy recovery.

The latency-throughput curves in Figures 2 and 3 add a second dimension to this insight: the crossover point between formats depends on the operating point on the latency-throughput curve. At low latency (low throughput), W4A16-INT is more efficient; at high latency (high throughput), W8A8 overtakes it. This means there is no single "best" format even for a fixed model and hardware — the choice depends on the service-level objective (SLO) the deployment must meet. A code completion system with a 200ms latency target might prefer W4A16; a batch summarization pipeline with no latency constraint might prefer W8A8.

This insight is incremental rather than fundamental in terms of theoretical novelty — the prefill/compute-bound vs. decode/memory-bound distinction is well-known — but it is practically fundamental because it converts a qualitative understanding into a quantitative decision framework with specific numbers (Table 5, Table 6, Figures 2–3) across seven use cases, three GPU architectures, and three model sizes. The granularity of the analysis makes this more than a general guideline; it is a lookup table for deployment decisions.

---

Innovation 4: W4A16-INT Rivals W8A8-INT in Accuracy — Challenging the Assumption That Activating Quantization Is Always Worth the Weight Trade-Off

A finding that emerges across the paper's results but is not explicitly foregrounded as a single claim is this: INT4 weight-only quantization (W4A16-INT) performs on par with INT8 weight-and-activation quantization (W8A8-INT) across most benchmarks and model scales, despite quantizing weights to half the bitwidth and leaving activations entirely uncompressed. On Leaderboard V1, the 405B model achieves 99.98% recovery with W4A16-INT versus 99.32% with W8A8-INT — W4A16 actually recovers slightly better (Table 2). On Leaderboard V2, the 8B W8A8-INT achieves 101.5% recovery while W4A16-INT achieves 96.1% — the only case where INT8 meaningfully leads. On real-world benchmarks (Table 3), the two formats are essentially indistinguishable at 70B and 405B scales, with overlapping Arena-Hard confidence intervals and similar HumanEval scores.

This finding challenges a tacit assumption in the quantization literature: that quantizing both weights and activations (W8A8) should be more accurate than quantizing weights more aggressively but leaving activations at full precision (W4A16), because the former distributes the compression burden across both tensors while the latter concentrates all compression on weights. The paper's observation that "for INT, quantizing activations is harder than quantizing weights" (Section 4.1) inverts this logic: activation quantization introduces its own errors that can outweigh the benefit of keeping weights at higher precision. By avoiding activation quantization entirely, W4A16-INT sidesteps the outlier feature problem that requires SmoothQuant or dynamic quantization for INT8 activations, and this avoidance compensates for the more aggressive weight compression.

The conceptual significance of this finding is that it decouples the accuracy and performance dimensions of quantization format choice. Prior to this work, a practitioner might assume that higher compression ratio (4-bit weights) necessarily implies lower accuracy, and therefore W8A8-INT must be more accurate than W4A16-INT. The paper shows this is false: W4A16-INT achieves comparable accuracy while providing different (and often greater) performance benefits in synchronous deployments. This means the format choice can be driven primarily by deployment characteristics (synchronous vs. asynchronous, prefill/decode ratio) rather than by a perceived accuracy hierarchy — exactly the decision framework the paper constructs in its deployment recommendations.

The paper attributes W4A16-INT's competitiveness to three specific factors: (1) GPTQ with MSE-optimal clipping (rather than abs-max in prior comparisons), (2) high-quality calibration data (OpenPlatypus, not C4), and (3) avoiding activation quantization entirely, which eliminates the need for SmoothQuant and removes a source of error that affects W8A8-INT at the 70B scale. The finding that the tuned GPTQ variant outperforms AWQ on real-world tasks (Table 1) is part of this same insight: concentration of compression on weights, when done with proper tuning, is more robust than prior work suggested.

This innovation is incremental in that it emerges from systematic benchmarking rather than a new algorithmic idea, but it is consequential for practice because it reshuffles the default assumptions about which format to try first for a given deployment. The paper's explicit recommendation — W4A16 for synchronous, W8A8 for asynchronous — flows directly from this finding.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper uses multiple benchmark suites spanning academic and real-world tasks. The academic benchmarks are the Open LLM Leaderboard V1 (7 tasks: MMLU, MMLU CoT, ARC-Challenge, GSM8k CoT, HellaSwag, Winogrande, TruthfulQA; Beeching et al., 2023) and V2 (6 tasks: IFEval, BBH, MATH Level 5, GPQA, MuSR, MMLU-Pro; Fourrier et al., 2024), each using the standard test splits. The real-world benchmarks comprise Arena-Hard-Auto-v0.1 (500 complex prompts evaluating chat and instruction following; Li et al., 2024b), HumanEval (164 programming problems) and HumanEval+ (extended test cases; Chen et al., 2021; Liu et al., 2023a), and RULER (long-context retrieval and reasoning at 4K–128K sequence lengths; Hsieh et al., 2024). For reasoning models, three additional benchmarks are used: AIME 2024, MATH-500, and GPQA-Diamond (Section 4.3). The total evaluation corpus spans over 500,000 measurements (abstract).

• Base model(s). The primary evaluation uses the Llama-3.1-Instruct model family (Dubey et al., 2024) at all three available scales: 8B, 70B, and 405B parameters. These are instruction-tuned conversational models trained for chat, code, and general reasoning. The choice is motivated by their widespread adoption and the availability of a quantized release whose quality sparked community skepticism (abstract). For reasoning benchmarks, the paper extends to six DeepSeek-R1-Distill models (DeepSeek-AI, 2025), including Llama-8B, Llama-70B, Qwen-1.5B, Qwen-7B, Qwen-14B, and Qwen-32B variants, to test generalizability across model families and reasoning capabilities.

• Metrics. The primary metric is accuracy recovery, computed as the ratio of quantized model score to BF16 baseline score, expressed as a percentage (Recovery % = Quantized Score / BF16 Score × 100%). For Leaderboard V1, raw task scores are reported on a 0–100 scale. For Leaderboard V2, scores are normalized by subtracting the random baseline and rescaling: norm_score = (raw_score − random_baseline) / (100 − random_baseline) × 100, ensuring equal weighting across tasks (Section 4.1). Arena-Hard uses a win rate (percentage of prompts where the model's response is judged better than or equal to a reference), averaged over two independent runs with 95% confidence intervals (Appendix Table 7). HumanEval and HumanEval+ use pass@1 (fraction of problems where the first generated solution passes all test cases, with greedy sampling) and pass@10 (best of 10 samples; Appendix Figures 5–6). RULER reports an aggregate score across subtasks. Text similarity metrics (Section 4.4, Figure 1) use ROUGE-1, ROUGE-L, BERTScore, and Semantic Textual Similarity (STS), all normalized to [0, 1].

• Baselines. The primary baseline for accuracy comparisons is the unquantized BF16 model for each model size, evaluated on the same benchmarks under identical protocols. For inference performance comparisons, the baseline is BF16 inference on the same GPU configuration. The paper also compares against prior quantization studies: Lee et al. (2024b) is the most directly compared prior work, cited as claiming a 10-point accuracy drop for W8A8-INT 405B on Leaderboard V2 (Section 2.2). Li et al. (2024a) is cited as another source of pessimism about INT8 activation quantization. For the INT4 weight quantization algorithm comparison, AWQ (Lin et al., 2024a) serves as the baseline against which the paper's tuned GPTQ variant is compared (Table 1, Tables 8, 12, 13).

• Generation budget / compute accounting. For accuracy evaluations, generation hyperparameters are task-specific: academic benchmarks use Meta's prompt guidelines with text-generation mode for MMLU and ARC-Challenge, and 8-shot chain-of-thought for GSM8k (Section 4.1). Long-context RULER evaluations span 4K to 128K tokens (Table 3). Text similarity analysis uses greedy sampling (temperature = 0) for full reproducibility (Section 4.4). For reasoning benchmarks, the paper uses sampling with temperature 0.6, top-p 0.95, and generates 20 responses per query to estimate pass@1, with the multiple samples "important to estimate an accurate average performance... due to high variance across the relatively small datasets" (Section 4.3). For inference performance, compute is measured as end-to-end latency (seconds, synchronous) or throughput in queries per second (QPS, asynchronous), with cost efficiency computed as queries per USD using Lambda Labs on-demand GPU pricing (Tables 5, 6, 9). The FLOPs cost of quantization itself — running GPTQ, SmoothQuant preprocessing, or calibration data collection — is not accounted for since quantization is a one-time offline cost.

• Cross-validation / statistical protocol. The paper does not use formal k-fold cross-validation for accuracy benchmarks. Instead, it uses a two-way split: quantization hyperparameters are optimized on the Open LLM Leaderboard V1, and results are validated on the separate Leaderboard V2, which the paper describes as "ensuring generalization by optimizing quantization hyperparameters on V1 while validating results on V2" (Section 4.1). For Arena-Hard-Auto-v0.1, each quantization configuration is evaluated twice independently, with both run scores, the average, and 95% confidence intervals reported (Appendix Table 7). The paper explicitly notes that conclusions about format differences on Arena-Hard are drawn from overlapping confidence intervals rather than point estimates (Section 4.2). For reasoning benchmarks (AIME 2024, MATH-500, GPQA-Diamond), the 20-sample pass@1 estimation provides standard error bars reported with each score (Table 4). No statistical significance tests (t-tests, bootstrap) are reported for the main comparison tables (Tables 2, 3).

---

Main Quantitative Results

W8A8-FP Is Effectively Lossless Across All Model Scales and Benchmark Types

The headline finding for FP8 is that W8A8-FP quantization preserves accuracy within the evaluation's margin of error across all model sizes and benchmarks. On the Open LLM Leaderboard V1, W8A8-FP achieves 99.31% recovery for 8B, 99.72% for 70B, and 100.12% for 405B — the 405B quantized model actually scores slightly above the BF16 baseline (Table 2). On Leaderboard V2, the corresponding recoveries are 101.2%, 100.0%, and 99.9% (Table 3). On real-world benchmarks, the 405B quantized model achieves 66.9% Arena-Hard win rate vs. 67.4% for BF16, with overlapping 95% confidence intervals of (−2.6, 2.3) and (−2.6, 1.9) respectively (Appendix Table 7). HumanEval pass@1 is 87.0% vs. 86.8% for the 405B BF16 baseline, with HumanEval+ at 81.0% vs. 80.1% — the quantized model actually scores slightly higher in both cases (Table 3). For reasoning benchmarks, the W8A8-FP DeepSeek-R1-Distill models achieve 99.0–101.0% recovery across all six model variants (Table 4), with the Qwen-14B model actually improving from 73.6 to 74.3 average score.

The critical methodological detail enabling this lossless performance is that W8A8-FP uses dynamic per-token activation quantization rather than static calibration (Section 3.2), which the paper argues avoids the activation outlier problem entirely through FP8's floating-point dynamic range. The format also requires no calibration data, making it computationally simpler than INT8 while achieving better accuracy. However, this format is only available on NVIDIA Hopper (H100) and Ada Lovelace architectures with native FP8 tensor core support, which limits its applicability to the newest GPU hardware.

W8A8-INT Achieves Only 1–3% Degradation, Refuting Prior Claims of 10%+ Drops

The most provocative accuracy finding is that W8A8-INT quantization, when properly tuned, achieves dramatically better accuracy than prior work reported. On Leaderboard V1, W8A8-INT recovers 100.31% for 8B, 99.87% for 70B, and 99.32% for 405B (Table 2) — all within 1% of the BF16 baseline. On the more challenging Leaderboard V2, recoveries are 101.5%, 97.3%, and 98.3% respectively (Table 3). The lowest per-task recovery on V2 for a task where the BF16 baseline scores above 40% is 97.8% on MMLU-Pro for the 405B model (Section 4.1, Appendix Table 11).

These numbers directly contradict Lee et al. (2024b), which the paper states reported "a 10-point accuracy drop for W8A8-INT quantized 405B models on the Open LLM Leaderboard V2 compared to FP8" (Section 2.2). The paper's tuned approach reduces this to 0.7 points: the 405B BF16 scores 48.7 on V2 average, W8A8-FP scores 48.7, and W8A8-INT scores 47.9 — a 0.8-point gap from BF16 (Table 3). The paper attributes prior pessimism to suboptimal hyperparameters: Lee et al. (2024b) likely used coarser quantization schemes without SmoothQuant for activation outliers and with lower-quality calibration data.

The key hyperparameter enabling this result at the 70B scale is SmoothQuant (Xiao et al., 2022). The paper notes that dynamic per-token activation quantization "performs well for 8B and 405B models, [but] it causes noticeable accuracy drops at 70B. To mitigate this, we apply SmoothQuant" (Section 3.2). This size-specific tuning — applying SmoothQuant at exactly the scale where activation outliers become problematic — is precisely the kind of hyperparameter optimization that the paper argues distinguishes its results from prior work. On real-world benchmarks, the 70B W8A8-INT achieves 57.0% Arena-Hard vs. 57.0% for BF16 (identical), and 78.7% HumanEval vs. 79.7% for BF16 — a 1.0-point drop (Table 3). For the 405B, the Arena-Hard drops from 67.4% to 64.6% with overlapping confidence intervals (−2.4, 2.8) vs. (−2.6, 1.9), while HumanEval remains essentially unchanged (86.9% vs. 86.8%). On reasoning benchmarks, W8A8-INT recovers 99.4–100.7% across the DeepSeek-R1-Distill models except for the smallest Qwen-1.5B (96.9%) (Table 4).

W4A16-INT Is Competitive With W8A8-INT, and Properly Tuned GPTQ Outperforms AWQ on Real-World Tasks

The paper demonstrates that INT4 weight-only quantization achieves accuracy comparable to INT8 weight-and-activation quantization, challenging the assumption that more aggressive weight compression necessarily implies higher accuracy loss. On Leaderboard V1, W4A16-INT recovers 98.72% for 8B, 99.53% for 70B, and 99.98% for 405B (Table 2) — at 405B, the recovery is actually higher than W8A8-INT's 99.32%. The lowest per-task recovery on V1 is 96.88% on TruthfulQA for the 8B model (Appendix Table 10). On Leaderboard V2, recoveries are 96.1% (8B), 97.4% (70B), and 98.9% (405B) (Table 3), with the 8B model showing the largest gap — primarily on IFEval (98.0% recovery) and MuSR (83.18% recovery, though the absolute BF16 score is only 7.61, making percentage recovery unreliable at near-random baselines; Appendix Table 11). The paper notes that "W4A16-INT models demonstrate superior recovery over W8A8-INT" for the 70B model on V2 (97.4% vs. 97.3%), concluding that "for INT, quantizing activations is harder than quantizing weights" (Section 4.1).

On real-world benchmarks, W4A16-INT is highly competitive: the 405B model achieves 66.5% Arena-Hard vs. 67.4% BF16 (overlapping CIs), 85.1% HumanEval vs. 86.8% BF16 (a 1.7-point drop), and 78.9% HumanEval+ vs. 80.1% BF16 (a 1.2-point drop) (Table 3). On RULER long-context, quantized models achieve ≥98% average score recovery across formats (Table 3), with W4A16-INT at 81.1% for 8B vs. 82.8% BF16, and 82.2% for 70B vs. 83.3% BF16. For reasoning models, W4A16-INT achieves 97.2–99.5% recovery for Llama-8B through Qwen-32B, dropping to 93.5% only for the smallest Qwen-1.5B model (Table 4).

The GPTQ vs. AWQ comparison (Table 1) reveals that the two methods perform similarly on academic benchmarks — AWQ leads by 0.23 points on the 8B academic average and 0.35 points on the 70B — but GPTQ opens a meaningful gap on real-world tasks: 2.9 points on the 8B real-world average (52.3 vs. 49.4) and 0.8 points on the 70B (73.1 vs. 72.3). The coding benchmarks drive this gap: on HumanEval, GPTQ achieves 67.1% vs. AWQ's 63.0% for 8B (a 4.1-point gap), and 80.5% vs. 79.4% for 70B; on MBPP, GPTQ achieves 65.8% vs. 62.8% for 8B. Arena-Hard shows a smaller but consistent advantage: 24.0% vs. 22.3% for 8B, 57.0% vs. 56.7% for 70B (Appendix Table 8). The paper attributes this to three factors: GPTQ with MSE-optimal clipping rather than abs-max, higher-quality calibration data (OpenPlatypus vs. C4), and the inclusion of real-world benchmarks that "provide a broader evaluation scope" (Section 3.2).

Text Similarity: Quantized Models Preserve Semantic Content Even When Surface Form Varies

The text similarity analysis (Section 4.4, Figure 1) reveals a scale-dependent pattern: larger quantized models (70B and 405B) closely track their BF16 counterparts in both word choice and structure, while smaller quantized models (8B) show moderate surface-form variability but maintain semantic fidelity.

For the 405B model, all three quantization formats achieve ROUGE-1 between 0.67 and 0.75 (unigram overlap), ROUGE-L between 0.51 and 0.63 (structural similarity), BERTScore between 0.92 and 0.95 (token-level contextual similarity), and STS between 0.95 and 0.97 (sentence-level semantic similarity). W8A8-FP achieves the best scores across all metrics (ROUGE-1 0.75, ROUGE-L 0.63, BERTScore 0.95, STS 0.97), while W4A16-INT shows slightly lower structural preservation (ROUGE-L 0.51) but identical semantic scores (BERTScore 0.92, STS 0.95). The 70B model shows a similar pattern: W8A8-FP leads on structural metrics, while all formats converge on semantic metrics (BERTScore 0.92–0.95, STS 0.95–0.97). For the 8B model, structural scores drop more noticeably — ROUGE-1 from 0.75 (405B FP8) to 0.66 (8B FP8), ROUGE-L from 0.63 to 0.51 — but BERTScore remains at 0.93 and STS at 0.96 for W8A8-FP, confirming that "quantized models generate high-quality outputs across all sizes and schemes" (Section 4.4). W4A16-INT for 8B shows the largest surface-form divergence (ROUGE-1 0.58, ROUGE-L 0.41) but still maintains BERTScore at 0.90 and STS at 0.94.

The paper interprets this as evidence that larger quantized models "closely adhere to the word choices and sentence structures of their uncompressed counterparts" while smaller quantized models "introduce moderate variability in structure but still preserve semantic meaning" (Finding 4, Section 1). This has practical implications: for applications where output consistency with a known reference is important (e.g., deterministic API responses), larger models with FP8 quantization are preferable; for applications where semantic quality matters more than exact wording, even INT4 on smaller models is adequate.

Reasoning Models Robustness: Quantization Findings Generalize Across Model Families and Task Types

The reasoning model experiments (Table 4) serve as a cross-validation of the paper's main findings across different model architectures (Llama and Qwen), training paradigms (distillation from a reasoning model), and task types (advanced math, graduate-level science). The core pattern from the Llama-3.1 evaluation replicates: 8-bit formats (both FP8 and INT8) are near-lossless, while INT4 shows slightly larger but manageable degradation, especially at smaller scales.

For the Llama-70B R1-Distill model, recoveries are 100.3% (FP8), 99.7% (INT8), and 98.3% (INT4). The Llama-8B R1-Distill shows slightly more degradation: 100.6% (FP8), 99.6% (INT8), and 97.2% (INT4). The Qwen family shows similar patterns, with the Qwen-32B achieving 99.0% (FP8), 99.6% (INT8), and 99.5% (INT4) — essentially lossless across all formats. The Qwen-1.5B, the smallest model tested, shows the largest degradation: 100.3% (FP8), 96.9% (INT8), and 93.5% (INT4). On specific benchmarks, the pass@1 scores reveal substantial variance: for Llama-8B on AIME 2024, the BF16 model scores 49.3 ± 6.4, while W8A8-FP scores 50.8 ± 9.0 — the quantized model actually scores higher but with wider error bars, illustrating why the paper uses 20-sample averaging. The MATH-500 results are tighter: Llama-8B BF16 at 90.2 ± 1.2, W8A8-FP at 90.2 ± 1.1, W8A8-INT at 90.0 ± 1.0, and W4A16-INT at 89.9 ± 1.1 — all within measurement error. GPQA-Diamond shows more separation: Llama-8B BF16 at 49.3 ± 3.1, W4A16-INT at 47.1 ± 2.6, a 2.2-point drop that is within the overlapping standard errors.

The paper summarizes these findings as confirming that "when quantization is properly tuned and configured, quantized models perform very competitively with their unquantized (BF16) baselines, recovering on average >99% accuracy except for the smallest models at INT4 which exhibit a bit larger but reasonable drops" (Section 4.3).

Synchronous Deployment: W4A16-INT Achieves Largest Cost Reductions Across All Scales

The synchronous inference performance results (Table 5) establish W4A16-INT as the most cost-efficient format for latency-sensitive single-query deployments, with cost reduction factors increasing with model size.

For the 8B model on a single A6000 GPU, W4A16-INT achieves a 2.39× cost reduction over BF16 (from 183 to 462 queries per USD for code completion, and from 335 to 736 queries per USD for summarization). W8A8-INT achieves a 1.54× reduction, while W8A8-FP is not available on this GPU. The latency improvements mirror the cost: code completion drops from 24.5 seconds (BF16) to 9.7 seconds (INT4), a 2.5× speedup.

For the 70B model on 4×A6000 GPUs, the gap widens: W4A16-INT achieves 2.96× cost reduction (from 18 to 57 queries per USD for code completion), while W8A8-INT achieves 1.94×. This means INT4 not only reduces cost more than INT8, but the relative advantage grows with model size — INT4 is 1.53× more cost-efficient than INT8 (2.96/1.94) for the 70B on 4×A6000, compared to 1.55× for the 8B (2.39/1.54). On 2×A100, W4A16-INT achieves 2.67× reduction and W8A8-INT achieves 1.81×. On 2×H100, W4A16-INT achieves 2.11× and W8A8-FP achieves 1.84× — the gap narrows on Hopper hardware because FP8 operations are computationally more efficient than INT4 dequantization for certain tensor shapes.

The most dramatic result is at the 405B scale: W4A16-INT achieves 5–7× cost reduction (from 2 to 10 queries per USD for code completion on 16×A100 using the 4-GPU INT4 configuration vs. the 16-GPU BF16 baseline; Table 5) and critically enables deployment on 4 GPUs rather than 16, reducing inter-GPU communication overhead. The cost reduction for 405B INT4 on 16×H100 reaches 5.15×, while FP8 achieves 3.17×. The paper notes this as a major practical advantage: "deploying the 405B model on 4× A100 or H100 GPUs with W4A16-INT meets performance thresholds that previously required 16 GPUs in BF16, reducing inter-GPU communication and latency" (Section 5.1).

Asynchronous Deployment: W8A8 Formats Maximize Throughput, but W4A16 Wins at Low Latency

The asynchronous throughput results (Table 6) reveal a deployment-regime-dependent trade-off with no universally dominant format. W8A8 formats (INT8 and FP8) achieve the highest throughput speedups — 1.38× to 3.09× over BF16 depending on model size, GPU architecture, and use case — while W4A16-INT achieves 1.08× to 3.09×.

For the 8B model on 1×A6000, W8A8-INT achieves 1.38× throughput speedup over BF16 (from 1.5 to 2.2 QPS for code completion), while W4A16-INT achieves only 1.08× — INT8 is 27% faster than INT4 in this throughput-maximizing regime. The cost efficiency follows: W8A8-INT delivers 9.8k queries per USD vs. 9.8k for W4A16-INT and 6.8k for BF16. For the 70B on 4×A6000, the gap is similar: W8A8-INT achieves 1.91× speedup, W4A16-INT achieves 1.92× — nearly tied overall, but with use-case-dependent variation. On 4×A100, W8A8-INT achieves 1.87×, W4A16-INT achieves 1.64×, with W8A8-INT pulling ahead due to A100's efficient INT8 tensor core operations. On 4×H100, W8A8-FP achieves 1.77×, W4A16-INT achieves 1.55× — FP8's native hardware support gives it a clear throughput advantage.

The 405B results on 16×GPUs are striking: W8A8-FP on H100 achieves 3.04× speedup over BF16 (from 0.7 to 4.4 QPS for code completion), while W4A16-INT achieves 3.09× — they are essentially tied. But the per-dollar efficiency differs: W8A8-FP delivers 329 queries per USD, while W4A16-INT requires 4 GPUs instead of 16 (Table 5), dramatically reducing infrastructure cost at the expense of lower peak throughput on a per-GPU basis.

The latency-throughput trade-off curves (Figures 2 and 3) provide the key insight: at low latency (low throughput, left side of each curve), W4A16-INT achieves higher throughput at a given latency than W8A8-INT. At high latency (high throughput, right side of each curve), W8A8 overtakes W4A16. The paper describes this as: "W4A16-INT is more efficient at lower latencies, making it ideal for applications requiring rapid response times. In contrast, W8A8 formats maximize throughput at the cost of higher latency, making them better suited for batch processing." The crossover point — where W8A8 becomes more efficient — "depends on factors such as model size, hardware, and task requirements" (Section 5.2). For the 8B model on A6000 with docstring generation (Figure 2), W4A16-INT dominates up to approximately 3.5 QPS, after which W8A8-INT surpasses it. For the 70B model on 2×A100 with code fixing (Figure 3), W4A16-INT leads below approximately 0.35 QPS (inter-token latency below ~45ms), with W8A8-INT overtaking at higher throughput (lower latency sensitivity).

Accuracy Recovery Trends: FP8 > INT4 ≈ INT8, with Model Size and Format Interactions

Figure 4 provides the aggregated accuracy recovery trends across all three model sizes for the academic benchmarks. The visualization shows W8A8-FP as the clear leader (≈99.5–100% recovery across all sizes), with W8A8-INT and W4A16-INT closely clustered (both ≈98–99.5% recovery, with INT8 slightly ahead at 8B and 70B, and INT4 slightly ahead at 405B). The 8B model shows the largest spread between formats: FP8 ≈ 99.5%, INT8 ≈ 100%, INT4 ≈ 98.5%. At 70B, the formats converge: FP8 ≈ 99.7%, INT8 ≈ 99.9%, INT4 ≈ 99.5%. At 405B, the ordering inverts: INT4 ≈ 100%, FP8 ≈ 100.1%, INT8 ≈ 99.3% — INT8 is actually the lowest recovery at the largest scale. The paper attributes this to "the challenges of integer activation quantization, particularly at larger model sizes" (Figure 4 caption), consistent with the earlier observation that for INT, quantizing activations (as in W8A8-INT) is harder than quantizing weights (as in W4A16-INT).

---

Ablation Studies and Robustness Checks

GPTQ vs. AWQ for INT4 weight-only quantization (Table 1, Appendix Tables 8, 12, 13): The two methods perform near-identically on academic benchmarks (AWQ leads by 0.23 points for 8B, 0.35 points for 70B on the V1+V2 average), but GPTQ opens a measurable gap on real-world tasks (2.9 points for 8B, 0.8 points for 70B on the Arena-Hard+HumanEval+MBPP average). The gap is driven primarily by coding benchmarks: GPTQ achieves 67.1% vs. AWQ's 63.0% on HumanEval for 8B, and 65.8% vs. 62.8% on MBPP. Arena-Hard shows consistent but smaller gaps (24.0% vs. 22.3% for 8B, 57.0% vs. 56.7% for 70B). The paper identifies three hyperparameters responsible for the difference: (1) MSE-optimal clipping in GPTQ rather than abs-max, (2) high-quality calibration data (OpenPlatypus vs. C4 default), and (3) evaluation on real-world benchmarks that reveal gaps invisible in academic metrics.

SmoothQuant for INT8 activation quantization at 70B scale (Section 3.2): The paper reports that W8A8-INT with dynamic per-token quantization alone "causes noticeable accuracy drops at 70B" but works well at 8B and 405B. Applying SmoothQuant (Xiao et al., 2022) at the 70B scale restores accuracy to 99.87% recovery on V1 (Table 2) and 97.3% on V2 (Table 3). The paper does not provide a direct dynamic-only vs. dynamic+SmoothQuant ablation table for the 70B model, but the claim is integral to the reported W8A8-INT results at 70B.

Calibration data quality and source (Section 3.2, Tables 2, 3): The paper uses task-specific calibration data selection: random tokens for W8A8-INT at 8B (which "suffice"), Lee et al. (2023) / OpenPlatypus data for W8A8-INT at 70B and 405B ("larger models require higher-quality calibration data"), and OpenPlatypus exclusively for W4A16-INT at all scales (since "random token calibration degrades accuracy"). The specific ablation of random tokens vs. OpenPlatypus for INT4 is not quantified in a dedicated table, but the claim that calibration data quality affects accuracy differentially by format and scale is central to the paper's methodology.

Two-run reliability for Arena-Hard-Auto-v0.1 (Appendix Table 7): The paper evaluates each quantization configuration on Arena-Hard twice and reports both run scores, their average, and 95% confidence intervals. The confidence intervals overlap across all formats for each model size, confirming that quantization format differences on this benchmark are not statistically distinguishable at the 95% level. For example, the 405B BF16 has CI (−2.6, 1.9), W8A8-FP has (−2.6, 2.3), W8A8-INT has (−2.4, 2.8), and W4A16-INT has (−2.6, 2.3) — all overlapping substantially. This rules out claims that any format is "clearly better" for chat quality on this benchmark.

Pass@10 for code generation (Appendix Figures 5 and 6): The paper reports pass@10 scores for HumanEval and HumanEval+, showing that the relative patterns from pass@1 are preserved when the model gets 10 attempts per problem. All formats maintain high pass@10 scores, reducing the gap to BF16 further.

Reasoning model family and scale generalizability (Table 4): The quantization approach generalizes across six reasoning model variants spanning Llama and Qwen architectures and scales from 1.5B to 70B, with recovery consistently ≥99% for 8-bit formats and ≥97% for INT4 (except Qwen-1.5B at 93.5%). This serves as a cross-validation that the hyperparameter configuration from Llama-3.1-Instruct transfers without modification to different model families and training paradigms.

30B-scale DeepSeek-R1-Distill reasoning models (Table 4): The Qwen-32B results show virtually identical recovery across formats: 99.0% (W8A8-FP), 99.6% (W8A8-INT), and 99.5% (W4A16-INT), with average scores of 75.6, 76.0, and 75.9 respectively on a 0–100 scale. The Qwen-14B shows minor format gaps: 101.0% (FP8), 99.4% (INT8), and 99.0% (INT4). The smallest models show the largest sensitivity. This scale-dependent pattern — quantization robustness improves with model size — aligns with the Llama-3.1 findings and is a robustness confirmation across model families.

70B model on 2×A100 code fixing latency-throughput (Figure 3): The crossover point between W4A16-INT and W8A8-INT is shown for a specific hardware + model + use case combination. W4A16-INT achieves higher QPS at inter-token latencies below approximately 45ms; above this latency (higher throughput, more batching), W8A8-INT overtakes. This figure serves as an existence proof that the format trade-off depends on the operating point, even if the exact crossover varies by configuration.

Negative result: W4A16-INT can underperform W8A8-INT in throughput at high batch sizes (Table 6, Figures 2, 3): While INT4 is superior for synchronous single-query deployments, its asynchronous throughput advantage over BF16 is smaller (1.08× for 8B on A6000 vs. 1.38× for INT8) or absent at high batch sizes. This is because INT4 reduces memory bandwidth pressure in decode but the dequantization overhead and 16-bit activation computation in prefill limit scaling when many queries are batched. The paper presents this as a deployment-dependent trade-off rather than a failure of INT4 — the format is optimal for some regimes and suboptimal for others, which is precisely why the study's multi-regime analysis is necessary.

---

Critical Assessment

Claim: "W8A8-FP quantization is essentially lossless" (Finding 1)

This claim is well-supported but with a hardware availability caveat. The evidence across all benchmarks consistently shows W8A8-FP recovery ≥ 99.3%, often exceeding 100% (Tables 2, 3, 4, Appendix Tables 10, 11). The 95% confidence intervals on Arena-Hard overlap with BF16 for all model sizes (Appendix Table 7), meaning the format is not statistically distinguishable from full precision on the most practically relevant chat quality benchmark. However, the claim applies only to hardware with native FP8 support (NVIDIA Hopper, Ada Lovelace). The paper does not evaluate FP8 emulated on Ampere GPUs, which would be slower than INT8 and defeat the purpose. This means "essentially lossless" is true in a conditional sense: it is lossless on the hardware where it is relevant, but that hardware is a subset of deployed infrastructure. The paper would have been strengthened by explicitly quantifying what fraction of current LLM-serving GPU hours are on FP8-capable hardware versus Ampere-class hardware, to contextualize how widely applicable this finding is.

Additionally, the paper shows W8A8-FP achieving slightly above 100% recovery in several cases (100.12% at 405B on V1, 101.2% at 8B on V2, 100.6% for Llama-8B R1-Distill). These are likely evaluation noise — quantization cannot actually improve a model's knowledge. Reporting these without hypothesis tests or explicit acknowledgment that they reflect measurement variance rather than true improvement slightly overstates the precision of the measurements.

Claim: "W8A8-INT quantization exhibits only a modest accuracy degradation of 1–3% per task on average, far lower than the 10%+ drops reported in prior work" (Finding 2)

This claim is strongly supported, but the paper does not reproduce the prior work's exact configuration to isolate which hyperparameter changes cause the difference. The paper states that Lee et al. (2024b) reported a 10-point drop on V2 for W8A8-INT 405B versus FP8, and the paper reduces this to 0.7 points. This is the comparison that carries the paper's refutation. However, the paper does not run an ablation showing what happens if its own pipeline uses the Lee et al. (2024b) configuration — no SmoothQuant, no per-channel GPTQ, generic calibration data. Without that ablation, it is possible that other differences (vLLM version, evaluation protocol details, or even random seed effects on the V2 normalization) contribute to the gap beyond the claimed hyperparameter improvements. The paper's methodological argument — that proper tuning matters enormously — is persuasive, but the exact magnitude of improvement attributable to tuning vs. uncontrolled variables is not isolated.

The 70B results also raise a question about scale-specific brittleness. The paper states that W8A8-INT "causes noticeable accuracy drops at 70B" without SmoothQuant, and applies SmoothQuant only at that scale. But why does 70B specifically suffer from activation outliers that neither 8B nor 405B exhibit? The paper does not provide diagnostic evidence (e.g., activation magnitude distributions, per-channel variance) to explain this non-monotonic pattern. It is plausible that the 70B model represents a particular architectural regime where the ratio of outlier magnitude to typical activation magnitude peaks, but without evidence this remains an empirical observation rather than an explained phenomenon.

Claim: "W4A16-INT quantization maintains consistently low accuracy loss, performing on par with W8A8-INT" (Finding 3) and "GPTQ outperforms the more recent AWQ method on real-world tasks" (Finding 3)

The equivalence of W4A16-INT and W8A8-INT is supported at 70B and 405B scales, but less clearly at 8B. At 8B on V2, W4A16-INT recovery is 96.1% vs. 101.5% for W8A8-INT (Table 3); on Arena-Hard, W4A16-INT scores 24.0% vs. 27.2% for W8A8-INT; on HumanEval+, 59.1% vs. 60.0%. These differences are small in absolute terms (1–3 points) but consistent in direction — W8A8-INT edges out W4A16-INT on the smallest model. The paper emphasizes the equivalence at larger scales but the 8B model shows a clear, if modest, accuracy hierarchy: FP8 > INT8 > INT4. Practitioners deploying 8B models (which are popular for edge and consumer applications) may care about these differences.

The GPTQ-vs-AWQ finding is the most novel empirical claim in the paper and the one most vulnerable to reproducibility concerns. The paper attributes the difference to three hyperparameter choices (MSE-optimal clipping, better calibration data, broader benchmarks). The gap on academic benchmarks is negligible (0.23–0.35 points in AWQ's favor), so an evaluator using only academic metrics would conclude equivalence. But the real-world gap (2.9 points for 8B) depends heavily on two coding benchmarks (HumanEval and MBPP) — datasets with 164 and ~500 problems respectively. The paper does not report 95% confidence intervals or statistical tests on these differences, making it unclear whether a 2.9-point gap on the average of three real-world benchmarks is statistically reliable or could be attributable to the specific problem sample. A bootstrap confidence interval on the difference would substantially strengthen this claim.

Additionally, the paper's AWQ configuration is described as using "abs-max" quantization rather than MSE-optimal clipping, but AWQ's own methodology involves per-channel scaling, not abs-max clipping per se. The paper would benefit from specifying exactly which AWQ implementation and settings were used, and whether the AWQ authors' recommended hyperparameters were followed. If the AWQ configuration is suboptimal (just as the paper argues prior GPTQ configurations were suboptimal), then the comparison is GPTQ-tuned vs. AWQ-untuned, which would be a symmetric version of the very methodological criticism the paper levels at others.

Claim: "Larger quantized models closely adhere to the word choices and sentence structures of their uncompressed counterparts... smaller quantized models introduce moderate variability in structure but still preserve semantic meaning" (Finding 4)

This claim is descriptively accurate but its practical significance is not established. Figure 1 shows that ROUGE-L for the 8B model drops from ~0.51 (INT8/FP8) to ~0.41 (INT4), while BERTScore remains at 0.90 and STS at 0.94. This confirms the pattern, but the paper does not connect these numbers to any downstream consequence. Is a ROUGE-L of 0.41 vs. 0.51 noticeable to users? Does it affect task success rates in multi-turn settings where consistency matters? Without a user study or task-based evaluation of consistency importance, the similarity metrics remain intermediate representations whose practical interpretation is unclear. The paper could have strengthened this analysis by showing examples of outputs where structural similarity diverged and discussing whether the differences were substantive or cosmetic.

Claim: "W4A16-INT is the most cost-efficient for synchronous setups, while W8A8 formats maximize throughput in asynchronous settings" (Finding 5)

This claim is well-supported by the measured data but the performance measurements have an important scope limitation: they benchmark only vLLM 0.6.4.post1, a specific version of a single serving framework. Different serving frameworks (TensorRT-LLM, SGLang, TGI) may have different kernel implementations for INT4 dequantization or INT8 matrix multiplication, potentially changing the relative performance of quantization formats. The paper's recommendations are most reliable for vLLM deployments specifically. The paper acknowledges this implicitly by specifying the vLLM version, but does not discuss whether results might transfer to other frameworks.

The cost modeling also has a simplifying assumption: it uses on-demand pricing from a single cloud provider (Lambda Labs) and assumes linear cost scaling with GPU count. In practice, reserved instances, spot pricing, and multi-node interconnect costs create non-linear pricing that could change the cost-optimal format. The paper's cost numbers should be interpreted as indicative rather than precise — the relative ordering (W4A16 best for sync, W8A8 best for async throughput) is likely robust, but the exact cost reduction factors may vary by deployment.

The throughput measurements also do not model queueing effects or service-level objective (SLO) violations. Figures 2 and 3 show throughput vs. latency trade-offs, but real deployments have SLOs (e.g., "99th percentile latency must be under 500ms"). A format that achieves higher peak throughput but violates latency SLOs at high load would be inferior in practice to one with lower peak throughput that stays within SLO. The paper's latency-throughput curves enable practitioners to make these trade-offs, but the recommendation "W4A16 for synchronous, W8A8 for asynchronous" simplifies a continuous spectrum into a binary choice.

Claim: "W4A16-INT is more competitive than expected, rivaling 8-bit quantization" (abstract)

This claim holds for 70B and 405B models but is overstated at 8B, where W4A16-INT consistently shows the largest accuracy degradation across all benchmark categories: 98.72% V1 recovery vs. 100.31% for INT8 (Table 2), 96.1% V2 recovery vs. 101.5% for INT8 (Table 3), and the largest drops on coding (HumanEval+ pass@1 59.1% vs. 60.7% BF16). These differences are small (1–3 absolute points) but systematic. The paper's aggregated messaging — Figure 4 showing all formats clustered at 98–100% — can obscure that at the 8B scale specifically, there is a clear ordering: FP8 > INT8 > INT4. The claim "rivaling 8-bit quantization" is more precise when bounded to "at 70B and 405B scales" and qualified at 8B.

Missing Experiments That Would Strengthen the Paper

1. Direct ablation of hyperparameter contributions to accuracy recovery. The paper attributes the gap between its results and prior work to hyperparameter choices (GPTQ vs. RTN, SmoothQuant at 70B, calibration data quality, MSE-optimal clipping). Running a factorial ablation — systematically varying each hyperparameter and measuring its marginal contribution to accuracy recovery — would quantify which choices matter most and by how much. This would convert the paper's methodological argument (tuning matters) into a quantitative one (e.g., "calibration data quality accounts for 60% of the recovery gap, SmoothQuant for 25%, and MSE clipping for 15%"). Without this, the paper demonstrates that tuning matters but not which tuning decisions are most consequential.

2. Perplexity evaluation alongside task benchmarks. The paper deliberately avoids perplexity, arguing that real-world tasks matter more. But perplexity is the standard metric in most prior quantization work, and reporting it would (a) enable direct comparison to the literature the paper is refuting, and (b) test whether perplexity changes track task performance changes — if they do not, that strengthens the paper's argument for task-based evaluation; if they do, it weakens the novelty of choosing real-world benchmarks.

3. KV-cache quantization interaction. The paper notes in its limitations that "we have primarily focused on weight and activation quantization, leaving open questions about the impact of compressing other model components such as the KV-cache" (Limitations). This is a significant omission for the throughput analysis, where KV-cache memory is often the binding constraint on batch size for long-context workloads. A quantized model that compresses weights but leaves the KV-cache at full precision may be memory-limited by the KV-cache rather than by weights, meaning the practical throughput benefit is smaller than the quantization speedup alone suggests. The paper's RULER evaluation at long contexts (up to 128K tokens) would particularly benefit from KV-cache quantization analysis.

4. Distillation or fine-tuning interaction. The paper shows that post-training quantization works well for instruction-tuned models and distilled reasoning models. But it does not test whether quantization-aware fine-tuning (where the model is fine-tuned with quantization in the loop) recovers the small remaining accuracy gap, or whether fine-tuning a quantized model (e.g., for domain adaptation) preserves the quantization benefits. Given that many practitioners fine-tune models after quantization, this is a practical gap.

5. Activation distribution diagnostics. The paper's central explanation for INT8 accuracy degradation — activation outliers — is supported by citations to prior work (Dettmers et al., 2022; Xiao et al., 2022) but not by direct measurements on the Llama-3.1 models. Showing activation magnitude distributions, per-channel variance, and the effect of SmoothQuant on these distributions for the 70B model would strengthen the mechanistic explanation and help practitioners diagnose future models.

6. Multi-turn consistency evaluation. The text similarity analysis measures single-turn output similarity, but multi-turn chat (one of the seven profiled use cases) involves accumulated context across turns. Quantization errors in early turns could compound across the conversation in ways not captured by single-turn similarity metrics. A multi-turn consistency benchmark — such as measuring whether the quantized model's response in turn 3 matches the BF16 model's response in turn 3 given identical conversation histories — would test this.

7. vLLM version sensitivity and framework comparison. The paper benchmarks vLLM 0.6.4.post1 exclusively. Testing at least one other framework (e.g., TensorRT-LLM) or two vLLM versions would indicate whether the performance ordering of formats is robust to implementation details or specific to this version's kernel implementations.

6. Limitations and Trade-offs

6.1 The $4\times$–$7\times$ Cost Reduction Figures in Synchronous Deployments Ignore the One-Time Quantization Cost

The assumption or constraint. The inference performance analysis in Section 5 measures cost efficiency as queries per dollar for serving already-quantized models. The cost of performing the quantization itself — running GPTQ with MSE-optimal clipping on calibration data, applying SmoothQuant preprocessing, and validating accuracy — is excluded from every reported cost reduction factor. The paper acknowledges this implicitly by describing quantization as a one-time offline step ("Weights are compressed using GPTQ with MSE-optimal clipping," Section 3.2), but never quantifies the calibration and quantization runtime or amortizes it into the cost-per-query figures. For the 405B model specifically, collecting calibration data of sufficient quality (OpenPlatypus / Lee et al., 2023), running GPTQ across all transformer layers, and then validating accuracy on benchmarks before deployment represents a non-trivial engineering effort whose compute cost could equal many hours of inference on the quantized model. For a deployment processing millions of queries, this one-time cost amortizes to near zero; for a researcher or small team quantizing a model for a short-term project or evaluation, the overhead could dominate the total cost.

The consequence. The headline cost reductions — "2–3× for 8B and 70B models, and 5–7× for 405B" (Section 5.1, Finding 5) — underestimate the total cost of obtaining a quantized model. A deployment that processes relatively few queries after quantization may find that the quantization process itself consumed more GPU-hours than the inference savings recoup. This is particularly relevant for the INT4 format, which requires OpenPlatypus calibration data (random tokens "degrade accuracy," Section 3.2), and for the W8A8-INT format at 70B, which requires SmoothQuant preprocessing and per-channel GPTQ. FP8 quantization is the exception — it "requires no calibration data and remains computationally efficient, even for large-scale models" (Section 3.2) — so its cost reduction factors are more accurate as stated. The paper's omission also makes it impossible to compute a break-even query volume: how many queries must be served before the inference savings exceed the quantization cost?

What evidence exists in the paper. The paper provides no measurements of quantization runtime, calibration data collection cost, or accuracy validation overhead. Table 5 and Table 6 report only inference-time cost efficiency (queries per USD computed from GPU pricing and measured throughput), with no line item for the quantization pipeline. Section 3.2 describes the algorithmic steps — GPTQ layer-wise reconstruction, SmoothQuant per-channel scaling, MSE-optimal clipping grid search per weight group — but does not estimate their FLOPs or wall-clock time relative to inference. The lack of break-even analysis is particularly notable given the paper's framing as a "practical deployment guide" (abstract) where total cost of ownership matters.

Mitigation status. Not addressed. The limitation section (Section 6) does not mention this. The paper's focus on "one-time offline cost" as separable from "inference cost" is standard in the quantization literature, but for a paper whose explicit goal is to compute cost-per-query trade-offs for practitioners, the omission is a genuine gap.

---

6.2 The W8A8-FP "Lossless" Finding Applies Only to Hopper/Ada Lovelace GPUs, Excluding the Most Widely Deployed Inference Hardware

The assumption or constraint. The paper's most confident finding — that "W8A8-FP quantization is essentially lossless" (Finding 1, Section 4.1) — applies exclusively to NVIDIA Hopper (H100) and Ada Lovelace architectures with native FP8 tensor core support, as the paper states in Section 1: "using floating-point (FP) precision for NVIDIA Hopper and Ada Lovelace." The performance analysis confirms this: W8A8-FP does not appear in Tables 5 and 6 for A6000 or A100 GPUs because those Ampere-architecture GPUs lack FP8 hardware. The paper evaluates W8A8-FP inference performance only on H100 GPUs (Tables 5 and 6), meaning the accuracy findings for FP8 — 99.3–100.1% recovery across all model sizes on Leaderboard V1 (Table 2) — are paired with performance data only for the most expensive and newest GPU hardware. On A100 GPUs, the dominant datacenter inference GPU as of the paper's writing (and likely for years to come given installed base and availability constraints), the practitioner cannot use this format at all — they must choose between W8A8-INT and W4A16-INT.

The consequence. A practitioner reading the abstract's claim that FP8 is "essentially lossless" and W8A8-INT achieves "only 1–3% accuracy degradation" might conclude that FP8 is the preferred format for any accuracy-sensitive deployment. But if their infrastructure runs on A100 GPUs — as is the case for most cloud GPU instances and on-premise clusters deployed before Hopper became widely available — FP8 is not an option, and they must accept the accuracy trade-offs of INT8 or INT4. The paper's aggregated deployment recommendations (Section 5) do account for this by showing INT8 and INT4 performance on A100s, but the narrative framing around FP8 as the gold standard is only actionable for a subset of practitioners. The paper does not provide quantitative data on what fraction of LLM inference GPU-hours are served on FP8-capable hardware vs. Ampere, so the practical reach of its strongest result is unknown.

What evidence exists in the paper. The hardware dependency is explicitly stated in the format descriptions (Section 3.2: "W8A8-FP quantizes all linear operators... using round-to-nearest quantization"). The performance tables confirm W8A8-FP appears only in H100 rows (Tables 5 and 6). However, the accuracy recovery figure (Figure 4) and the abstract's claim do not carry hardware caveats, potentially misleading readers who do not check the hardware compatibility constraints buried in the methods section. The paper's Finding 1 in the abstract does not mention the hardware restriction.

Mitigation status. The paper partially addresses this by providing W8A8-INT results (the format available on Ampere GPUs) and demonstrating it achieves 99%+ recovery when properly tuned, making it a viable alternative. Section 5.1 explicitly evaluates W8A8-INT on A6000 and A100 GPUs. However, the paper does not systematically compare FP8 on H100 vs. INT8 on A100 for the same deployment scenario — such a comparison would help practitioners decide whether upgrading to Hopper hardware is justified by the accuracy + performance improvement.

---

6.3 The Difficulty Estimation Cost (2048 Samples per Question) Is Unaccounted For in the Headline Efficiency Gains

The assumption or constraint. While not directly stated as a limitation, the paper's central accuracy conclusions — that FP8 is lossless, INT8 loses only 1–3%, INT4 is competitive — depend on selecting the correct hyperparameters for each format-model pair. The paper optimized these hyperparameters on the Open LLM Leaderboard V1 and validated on V2 (Section 4.1), but does not report the computational cost of this hyperparameter search. The specific costs include: running multiple GPTQ configurations with different calibration datasets to determine that OpenPlatypus works for INT4 while random tokens suffice for INT8 at 8B; testing whether SmoothQuant is needed at each model scale (finding it necessary at 70B but not 8B or 405B); evaluating both AWQ and GPTQ for INT4 to determine which is superior; and running the entire V1 benchmark suite for each candidate configuration to select the best one. This hyperparameter optimization is a form of "difficulty estimation" for the quantization problem — determining which configuration works for which model — and its cost is not amortized into any reported efficiency metric.

The consequence. A practitioner attempting to replicate the paper's results on a new model (not in the Llama-3.1 family) cannot simply apply the paper's recommended configuration and expect the same recovery percentages. The paper's hyperparameter findings — SmoothQuant at 70B, OpenPlatypus calibration for INT4, MSE-optimal clipping for GPTQ — may be specific to the Llama-3.1 architecture, scale, and training distribution. The cost of re-discovering the optimal configuration for a new model through benchmark-driven hyperparameter search could be substantial, involving multiple quantization runs and full benchmark evaluation at each candidate configuration. The paper does not provide a "recipe" that guarantees accuracy recovery without per-model tuning, nor does it quantify how sensitive the recovery percentages are to hyperparameter perturbations (e.g., does switching from OpenPlatypus to C4 cause a 1% drop or a 10% drop?).

What evidence exists in the paper. The paper demonstrates scale-specific tuning requirements explicitly: "this scheme performs well for 8B and 405B models, [but] it causes noticeable accuracy drops at 70B. To mitigate this, we apply SmoothQuant" (Section 3.2). The finding that "random token calibration degrades accuracy" for INT4 (Section 3.2) is also a per-format discovery. The paper reports the final configurations that worked, not the search process or cost that led to them. The DeepSeek-R1-Distill results (Table 4) provide suggestive evidence that the Llama-3.1 hyperparameters transfer to reasoning models, but this is a single cross-validation point — it does not guarantee transfer to other architectures (e.g., Mistral, Gemma, or non-distilled reasoning models).

Mitigation status. The paper partially addresses this by testing across two model families (Llama-3.1-Instruct and DeepSeek-R1-Distill) and showing the same quantization configurations work on both. The reasoning model results in Section 4.3 (Table 4) serve as a transferability test, with recovery >99% for most models. However, the paper does not test on non-Llama base architectures, leaving open the question of whether the hyperparameter recommendations generalize beyond the Llama family. The limitations section does not mention this.

---

6.4 Single Serving Framework (vLLM 0.6.4.post1) and Cloud Provider (Lambda Labs) Limit the Generality of Performance Recommendations

The assumption or constraint. All inference performance measurements (Tables 5, 6; Figures 2, 3) use vLLM version 0.6.4.post1 exclusively, as stated in the abstract and Section 5. The paper makes deployment recommendations — "W4A16 is the most cost-efficient for synchronous setups, while W8A8 dominates in asynchronous continuous batching" (Finding 5, Section 5.2) — based on a single serving framework, a single version, and a single cloud provider's on-demand pricing (Lambda Labs, Table 9). Different serving frameworks (TensorRT-LLM, SGLang, HuggingFace TGI) may implement INT4 dequantization, INT8 matrix multiplication, or FP8 tensor core operations with different kernel efficiency, potentially changing the relative throughput and latency ordering of quantization formats. Similarly, different cloud providers (AWS, GCP, Azure) or on-premise hardware amortization schedules produce different cost-per-GPU-hour figures, shifting the absolute cost efficiency numbers even if relative ordering is preserved.

The consequence. The paper's deployment recommendations are most reliable for practitioners using vLLM on Lambda Labs infrastructure. For a practitioner using TensorRT-LLM on AWS, the relative throughput of W8A8-FP vs. W4A16-INT might differ if TensorRT-LLM has better-optimized INT4 kernels or if AWS's H100 pricing differs from Lambda Labs' markup. The paper's claim that "W4A16-INT is the most efficient choice for synchronous deployments" could invert on a framework with particularly well-optimized INT8 decode kernels or on hardware where INT8 tensor core utilization is higher than vLLM achieves. The paper does not discuss vLLM-specific bottlenecks (e.g., whether INT4 dequantization is fused into the attention or MLP kernels, whether the PagedAttention implementation has INT4-specific memory access patterns) that might not transfer to other frameworks.

What evidence exists in the paper. The paper specifies vLLM 0.6.4.post1 exactly (Section 5) but provides no alternative framework benchmarks, no sensitivity analysis to vLLM configuration parameters (batch size, block size, scheduling policy), and no cross-cloud pricing comparison. Table 9 lists Lambda Labs pricing as the sole cost basis. The latency-throughput curves (Figures 2 and 3) are produced from vLLM measurements, and the paper does not verify that the crossover point between W4A16 and W8A8 is robust to framework implementation details. The paper's finding that W4A16-INT throughput speedup over BF16 is only 1.08× for 8B on A6000 in asynchronous mode (Table 6) while INT8 achieves 1.38× suggests that vLLM's INT4 dequantization path may be suboptimally implemented for high-throughput scenarios — a framework-specific effect that could change with a different serving system.

Mitigation status. The paper does not address this limitation. The text mentions vLLM as "the popular vLLM framework" (abstract) and "the most widely used open-source LLM serving framework" (implied by context), but does not acknowledge that framework choice could affect relative performance conclusions. The limitation is not mentioned in Section 6 (Conclusion / Limitations). The paper's recommendation that "the optimal quantization scheme depends on model size, hardware, and deployment needs" (Finding 5) could be extended to include "and serving framework," but this is not done.

---

6.5 The 8B Model Shows a Clear Accuracy Hierarchy (FP8 > INT8 > INT4) That the Aggregated Messaging Obscures

The assumption or constraint. The paper's abstract and conclusion aggregate results across model scales to produce general claims: W8A8-FP is "essentially lossless across all model scales," W8A8-INT achieves "only 1–3% accuracy degradation," and W4A16-INT "is more competitive than expected, rivaling 8-bit quantization" (abstract, Findings 1–3). These claims implicitly assume that quantization formats have approximately constant relative accuracy across model sizes. However, the detailed results reveal a monotonic accuracy hierarchy at the 8B scale that differs from the near-equivalence observed at 70B and 405B. On Leaderboard V2, the 8B model recovers 101.2% (FP8), 101.5% (INT8), and 96.1% (INT4) — a 5.4-point gap between INT8 and INT4, and the only case where W4A16 recovery drops below 97% (Table 3). On Arena-Hard, the 8B scores are 26.8% (FP8), 27.2% (INT8), and 24.0% (INT4) — a 3.2-point gap with non-overlapping confidence intervals between INT8 and INT4 (Appendix Table 7). On HumanEval+, the 8B model scores 61.3% (FP8), 60.0% (INT8), and 59.1% (INT4) — INT4 is consistently lowest. The 8B is also the only scale where the paper's text similarity analysis shows a meaningful structural degradation for INT4 (ROUGE-L drops from 0.51 for INT8 to 0.41 for INT4; Figure 1).

The consequence. A practitioner deploying an 8B model — which is the most popular scale for consumer, edge, and cost-sensitive applications — may be misled by the paper's aggregated claim that INT4 "rivals" 8-bit quantization. At 8B, INT4 is consistently, if modestly, worse across all benchmark categories: academic, real-world, and structural text similarity. The paper's recommendation to prefer W4A16-INT for synchronous deployment (Section 5.1) based on cost efficiency should be weighed against this consistent accuracy deficit at the 8B scale — a trade-off the paper does not explicitly quantify per model size. The $2.39\times$ cost reduction for 8B INT4 (Table 5) comes with a 1–3% accuracy penalty across tasks and a measurable degradation in output structural similarity. Whether this trade-off is acceptable depends entirely on the application's accuracy requirements, which the paper does not model or discuss per scale.

What evidence exists in the paper. The scale-dependent accuracy differences are visible in the per-model-size breakdowns (Tables 2, 3, Appendix Tables 10, 11), but the paper's narrative and Figure 4 emphasize the clustering of recovery percentages. The abstract's claims are not conditioned on model size. Finding 3 states that W4A16 "performs on par with W8A8-INT" without noting the 8B exception. Section 4.1 notes that "W4A16-INT models demonstrate superior recovery over W8A8-INT" at 70B, but it does not symmetrically note W4A16-INT's inferior recovery at 8B. The text similarity analysis (Figure 1) shows the 8B INT4 model has the lowest ROUGE-L (0.41) of any configuration studied, but the paper interprets this as "moderate variability in structure" (Finding 4) rather than as a scale-dependent limitation of INT4.

Mitigation status. Not addressed. The paper does not provide per-scale deployment recommendations that account for the accuracy-performance trade-off. For example, a table showing "at 8B, INT4 provides 2.39× cost reduction but trades 1–3% accuracy and reduced structural similarity; at 70B, INT4 provides 2.96× cost reduction with no measurable accuracy loss" would frame the trade-off honestly. The conclusions and abstract present INT4 as uniformly competitive, which is accurate at larger scales but overstates the case at 8B.

---

6.6 No Evaluation of Quantization's Impact on Multi-Turn Conversation Consistency or Cumulative Error Propagation

The assumption or constraint. The paper evaluates quantized model accuracy on single-turn benchmarks: multiple-choice questions (MMLU, ARC), single-prompt code generation (HumanEval), single-turn chat (Arena-Hard), and single-document long-context tasks (RULER). It does not evaluate multi-turn conversation scenarios where quantization errors in early turns could propagate through the accumulated context and affect later turn quality. The text similarity analysis (Section 4.4, Figure 1) measures single-turn output similarity, not multi-turn trajectory divergence. The inference performance analysis profiles multi-turn chat (Tables 5, 6) as a deployment use case with specified prefill/decode token counts, but the accuracy impact of quantization specifically in the multi-turn setting is never measured — only the throughput and latency are profiled.

The consequence. Multi-turn chat is one of the seven deployment use cases the paper explicitly profiles for performance (Tables 5 and 6: "Multi-Turn Chat" with 512 prefill tokens and 256 decode tokens per turn). For a production chatbot, quantization errors that produce subtly different responses in early turns — different word choices, slightly different factual emphasis, or minor conversational tone shifts — will be incorporated into the conversation history. In subsequent turns, the BF16 and quantized models are effectively responding to different conversation histories, and the divergence may compound. A 1–3% per-turn accuracy degradation could amplify across a 10-turn conversation into a meaningfully different user experience, but single-turn benchmarks cannot detect this. The text similarity metrics confirm that 8B INT4 models already show structural divergence (ROUGE-L 0.41) in a single turn; over multiple turns, this divergence could produce qualitatively different conversations.

What evidence exists in the paper. No multi-turn consistency evaluation is reported. The paper's limitations section does mention that "our analysis does not fully explore the effects of quantization across specialized use cases," but it cites "multi-lingual tasks" as the example, not multi-turn conversations. The performance tables include multi-turn chat as a column — showing latency and throughput — but there is no corresponding accuracy column for multi-turn consistency.

Mitigation status. Not addressed. The paper describes the text similarity analysis as assessing "how closely quantized models' outputs align with their full-precision counterparts," but this is only done for single-turn Arena-Hard prompts (Section 4.4). The paper does not suggest multi-turn evaluation as future work, despite having the infrastructure to generate multi-turn conversations and apply the same ROUGE/BERTScore/STS metrics to conversation-level trajectories.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper achieves something methodologically important but conceptually modest: it does not propose a new quantization algorithm or discover a new principle. Instead, it corrects the field's empirical understanding of what existing quantization formats can achieve when properly tuned, and in doing so, shifts the burden of proof for future quantization research. The shift is not a paradigm change — it is a recalibration of baselines that has consequences for how the field evaluates compression techniques going forward.

The primary reframing: quantization accuracy is a hyperparameter optimization problem, not a hardware characteristic. Prior to this work, it was common in the literature to report a single accuracy-degradation number for a quantization format as if it were an intrinsic property. Papers would state "INT8 causes a 10-point accuracy drop" (as Lee et al., 2024b did) without specifying the quantization algorithm, calibration data, clipping strategy, or outlier handling technique. This paper demonstrates that the same format can produce a 10-point drop or a 0.7-point drop (Section 2.2) depending entirely on these hyperparameters. The implication is not merely that earlier studies were suboptimally tuned — it is that format-vs-format comparisons are meaningless without controlling for hyperparameter quality on both sides. A paper claiming "method X outperforms method Y for 4-bit quantization" must now demonstrate that Y was tuned to its achievable ceiling, not its default configuration. The paper's GPTQ-vs-AWQ comparison (Table 1) is the canonical example: AWQ appeared superior when GPTQ was evaluated with abs-max clipping and C4 calibration data, but the ranking reversed when GPTQ used MSE-optimal clipping and OpenPlatypus calibration. Future quantization papers that do not control for this risk the same methodological criticism this paper levels at Lee et al. (2024b).

Resolution of contradictory prior findings about INT8 activation quantization. The paper directly reconciles the conflict between work showing INT8 weight-and-activation quantization causes substantial degradation (Li et al., 2024a; Lee et al., 2024b) and the hardware community's assumption that INT8 should work well given its widespread tensor core support. The reconciliation is not that one side was wrong about their measurements — it is that they were measuring different things under the same format name. The "INT8 quantization" tested in prior work was likely RTN with static activation calibration on generic data; the "INT8 quantization" this paper tests is GPTQ for weights with dynamic per-token or SmoothQuant-adjusted activations on instruction-tuning data. Both are W8A8-INT, but they differ in the specific hyperparameter choices that determine accuracy. This reconciles the contradiction by revealing that the format label was insufficiently specific to characterize the method. The field now has clarity that W8A8-INT can achieve 99%+ recovery, but that getting there requires careful per-scale tuning (SmoothQuant at 70B, dynamic quantization at 8B and 405B, calibration data selection). The practical question shifts from "should I use INT8?" to "do I have the engineering resources to tune INT8 properly, or should I use FP8 which works out of the box?"

Redirection of research investment away from novel quantization algorithms and toward hyperparameter optimization and evaluation methodology. For practitioners deploying models today, this paper's findings suggest that improving the tuning of existing formats yields larger gains than adopting newer, more complex quantization methods that lack robust kernel support. The paper explicitly notes that "more recent high-compression techniques... are inefficient for batch sizes larger than 1, limiting their practicality" (Section 2.1). W4A16-INT with tuned GPTQ achieves accuracy competitive with W8A8-INT while providing the largest cost reductions for synchronous deployment (2–7×, Table 5). This implies that the marginal benefit of techniques like vector quantization or 2-bit representations — which require custom kernels and struggle with batching — is limited until the deployment regime (batch size > 1, standard serving frameworks) can efficiently support them. The research community's attention might be better spent on improving verifier robustness, calibration data curation, and automated hyperparameter search for existing formats rather than pursuing lower bitwidths that sacrifice practical deployability.

The paper also makes academic benchmarks less credible as the sole evaluation for quantization research. Table 1 shows that academic benchmarks (Leaderboard V1+V2) fail to distinguish GPTQ and AWQ (0.23–0.35 point differences), while real-world benchmarks reveal a 2.9-point gap favoring GPTQ. This is not a small methodological footnote — it means that a quantization researcher optimizing for leaderboard position would make the wrong choice for deployment. The paper's text similarity analysis (Figure 1) provides a second dimension that academic benchmarks miss: structural and semantic output fidelity. Together, these findings create pressure for the quantization evaluation community to adopt real-world open-ended benchmarks (Arena-Hard, HumanEval, RULER) and text similarity metrics as mandatory evaluation components, not optional supplements.

The synchronous vs. asynchronous deployment framework as a decision tool. By measuring across seven use cases with specified prefill/decode ratios and producing latency-throughput curves (Figures 2, 3), the paper operationalizes a previously qualitative insight — that weight-only quantization helps decode-bound workloads more, while weight-and-activation quantization helps prefill-bound workloads more — into a quantitative lookup framework. This is an incremental contribution (the prefill-compute-bound / decode-memory-bound distinction is well-known) but a practically important one: practitioners now have specific numbers on which format to choose for their workload, rather than heuristics. The finding that "the crossover point depends on factors such as model size, hardware, and task requirements" (Section 5.2) means the deployment recommendation is not universal, but the paper provides the data to make per-workload decisions.

---

Follow-Up Research This Work Enables

Automated hyperparameter search for quantization pipelines across model architectures. The paper demonstrates that the optimal quantization configuration depends on model size (SmoothQuant at 70B but not 8B or 405B), format (OpenPlatypus calibration for INT4 but random tokens suffice for INT8 at 8B), and algorithm (MSE-optimal clipping for GPTQ INT4). These discoveries were made through manual trial-and-error on a single model family. A natural follow-up is building an automated pipeline that, given a new model architecture, efficiently searches the space of {calibration dataset, clipping strategy, outlier handling method, per-channel vs. per-group granularity} and produces a Pareto frontier of accuracy vs. compression ratio, analogous to how AutoML systems search neural architecture and hyperparameter spaces. This paper makes such a system newly tractable because it establishes that the search space is small enough to be enumerable (the key knobs are calibration data, clipping, and SmoothQuant — fewer than 10 binary/choice variables) and that the accuracy differences are large enough to be detected with standard benchmarks. A strong follow-up would demonstrate such a system on 3–5 model architectures (Llama, Mistral, Gemma, Qwen, Falcon) and measure whether the discovered configurations generalize or are architecture-specific. The negative result — that per-architecture tuning is always necessary — would be equally valuable, as it would establish the cost of quantization as a deployment tax.

KV-cache quantization interaction with weight and activation quantization in long-context serving. The paper explicitly identifies this gap: "we have primarily focused on weight and activation quantization, leaving open questions about the impact of compressing other model components such as the KV-cache" (Limitations). This is not a minor omission — for the long-context RULER benchmark and summarization use cases the paper profiles (Table 5, 6), the KV-cache memory footprint can exceed the weight memory footprint, making KV-cache compression the binding constraint on batch size. A direct follow-up would evaluate W4A16-INT weights with KV-cache quantization at 4-bit or 8-bit precision on RULER tasks at 32K–128K sequence lengths, measuring both accuracy recovery and throughput improvement over weight-only quantization. The paper's existing RULER infrastructure (Section 3.1, Table 3) could be extended with KV-cache compression, and the throughput benefit could be quantified using the same vLLM-based performance profiling methodology (Section 5). The hypothesis — that KV-cache compression's throughput benefit is larger than weight compression's for long-context workloads — would either complement or complicate the paper's synchronous-deployment recommendation of W4A16, since W8A8-INT (which quantizes activations and thus affects KV-cache representation) might interact differently with KV-cache compression than W4A16-INT.

Multi-turn conversation divergence measurement for quantized models. The paper's text similarity analysis (Section 4.4, Figure 1) establishes that quantized 8B models show structural divergence (ROUGE-L dropping from ~0.51 to ~0.41 for INT4) even in single-turn outputs. A natural extension is measuring whether this divergence compounds across multiple conversation turns. A concrete experiment: take the BF16 model and a quantized variant, run both through a 10-turn conversation with the same initial prompt and user turns, and at each turn measure the ROUGE-L, BERTScore, and STS between the two models' responses. Additionally, measure whether the models eventually produce factually divergent responses (e.g., disagreeing on a fact established in turn 2) as opposed to merely stylistically divergent ones. This would quantify the "cumulative error propagation" concern and establish whether the single-turn similarity metrics are conservative or optimistic estimates of practical divergence. The paper's Arena-Hard infrastructure could be adapted by prompting the judge model to generate follow-up questions based on the model's previous answers, creating a multi-turn trajectory without human involvement. A negative result — that divergence saturates after 2–3 turns with no further compounding — would significantly increase confidence in quantized models for chatbot deployment; a positive result — that 8B INT4 trajectories diverge qualitatively from BF16 after 5+ turns — would establish an important boundary condition on the paper's claim that INT4 "rivals" 8-bit quantization.

Benchmarking the optimization floor: how low can INT8 accuracy go with deliberately bad hyperparameters, and which bad choices dominate? The paper demonstrates that proper tuning recovers ~14× less degradation than prior work reported (10 points vs. 0.7 points for 405B W8A8-INT). But it does not isolate which hyperparameter choices drive this gap. A valuable follow-up would be a factorial ablation on the 405B model: starting from the tuned configuration, individually regress each choice to its "default" or "prior work" equivalent and measure the marginal accuracy cost. For instance: (1) replace OpenPlatypus calibration with C4, keeping all else equal; (2) replace GPTQ with round-to-nearest, keeping all else equal; (3) remove SmoothQuant at 70B, keeping all else equal; (4) replace MSE-optimal clipping with abs-max in GPTQ for INT4, keeping all else equal. This would produce a ranked list of hyperparameter importance, answering the question "if I have limited time to tune my quantization, which knob should I turn first?" The paper's current framing — "tuning matters" — is correct but vague; a factorial ablation would make it precise and actionable. The methodology is straightforward given the paper's existing evaluation pipeline.

Quantization-aware fine-tuning as a recovery mechanism for the remaining INT4 accuracy gap at small scales. The paper shows that 8B INT4 has a consistent 1–3% accuracy deficit relative to INT8 and FP8 (Tables 2, 3), and that smaller reasoning models (Qwen-1.5B) show larger gaps (93.5% recovery for INT4, Table 4). A targeted follow-up would test whether lightweight quantization-aware fine-tuning (QAT) — fine-tuning the quantized model on a small amount of instruction data with the quantization operators in the forward pass — can close this gap. The experiment: take the 8B W4A16-INT model, fine-tune on 1,000–10,000 OpenPlatypus examples with the INT4 weights frozen in their quantized representation but with trainable scale factors (or with straight-through estimator for the weights themselves), and measure whether the V2 recovery improves from 96.1% (Table 3) to above 99%. The paper's own finding that calibration data quality matters for INT4 (Section 3.2) suggests that the model's weights could benefit from adaptation to the quantization-induced error — GPTQ minimizes layer-wise output error, but fine-tuning could minimize task-specific loss with the quantization in place. If QAT closes the gap, it would resolve the main remaining weakness of INT4; if it does not, it would suggest the 4-bit weight precision itself is the bottleneck, not the interaction between quantization and task-specific weights.

---

Practical Applications and Downstream Use Cases

Cost-efficient LLM serving on fixed hardware budgets. Organizations with an existing GPU cluster (or a fixed cloud budget) that need to serve Llama-3.1-class models can directly apply this paper's recommendations to maximize throughput per dollar. The paper provides a specific decision procedure: (1) identify whether the workload is synchronous (latency-sensitive, one query at a time) or asynchronous (throughput-oriented, batched); (2) identify the GPU architecture (Ampere → INT8 or INT4 only; Hopper → FP8, INT8, or INT4); (3) look up the cost efficiency numbers in Tables 5 and 6 for the closest use case (prefill/decode token ratio). For a concrete scenario: a startup running a code completion service (256 prefill, 1024 decode) on 2×A100 GPUs with the 70B model should use W4A16-INT in synchronous mode (2.67× cost reduction, from 20 to 57 queries per USD; Table 5) if serving individual requests, or W8A8-INT in asynchronous mode (1.87× throughput speedup, from 0.7k to 1.2k queries per USD; Table 6) if batching multiple users. The accuracy cost is negligible at 70B (99.5% V1 recovery, 97.4% V2 recovery; Tables 2, 3), making the trade-off almost purely about cost. This is actionable today — the paper tested these exact configurations on vLLM, the framework most practitioners use.

Enabling deployment of 405B models on resource-constrained infrastructure. The paper's finding that W4A16-INT allows the 405B model to run on 4 GPUs instead of 16 (Table 5) changes the accessibility calculus for the largest models. A research lab, university, or small company with a 4×A100 or 4×H100 node — a common configuration in academic GPU clusters — can now serve the 405B model at acceptable latency (10 queries per USD on A100 for code completion, vs. 2 for BF16 on 16 GPUs; Table 5). Without quantization, the 405B requires 16 GPUs with tensor parallelism and inter-GPU communication overhead, which is both expensive and technically complex. The accuracy recovery is 99.98% on V1 and 98.9% on V2 (Tables 2, 3), meaning the model's capabilities are essentially fully preserved. This democratizes access to the largest models for organizations that cannot afford or provision 16-GPU instances. The paper notes that for the H100 at 405B, INT4 even outperforms FP8 on some cost metrics (5.15× reduction vs. 3.17×; Table 5) despite FP8 being theoretically better supported on Hopper — a non-obvious finding that changes the default recommendation for 405B deployments.

Batch inference pipelines with adaptive format selection. The paper's latency-throughput curves (Figures 2, 3) enable a deployment architecture where the serving system dynamically selects the quantization format based on current load. At low load (few concurrent requests, latency-sensitive), the system serves queries from a W4A16-INT replica; at high load (many queued requests, throughput-sensitive), it routes to a W8A8 replica. This is analogous to autoscaling but at the format level rather than instance count. The paper provides the data to set the crossover threshold: for the 8B model on A6000 with docstring generation (Figure 2), switch from INT4 to INT8 when QPS exceeds ~3.5; for the 70B model on 2×A100 with code fixing (Figure 3), switch when inter-token latency exceeds ~45ms. A production system could implement this by running both quantized model variants on the same GPU pool (if memory allows) or by maintaining separate hot-replica pools, with a load-based router. The accuracy equivalence at 70B and 405B scales (Tables 2, 3) means the routing decision is purely performance-based, with no accuracy penalty for format switching.

Quantization validation pipeline for new model releases. When a new open-weight model is released (e.g., a future Llama, Mistral, or Gemma version), the engineering effort to determine the best quantization configuration is currently ad-hoc — practitioners try various combinations and hope for the best. This paper provides a reproducible validation recipe: (1) run W8A8-FP with dynamic per-token quantization (trivial to implement, no calibration needed); (2) for W8A8-INT, try dynamic per-token activation quantization first, evaluate on a held-out benchmark (e.g., Leaderboard V1), and if degradation exceeds 2%, apply SmoothQuant with migration strength α=0.5 and re-evaluate; (3) for W4A16-INT, use GPTQ with 128-element groups, MSE-optimal clipping, and OpenPlatypus (or equivalent instruction-tuning data) for calibration; (4) evaluate accuracy recovery on both academic and real-world benchmarks, and if degradation is below 3%, proceed to inference profiling using the paper's seven use cases as a template. This recipe is not guaranteed to transfer to all architectures — the paper's extended evaluation on DeepSeek-R1-Distill models (Table 4) provides some transferability evidence, but it is limited to Llama and Qwen architectures. A team adopting this pipeline for a new model would contribute a data point on whether the Llama-3.1 optimal configurations generalize, advancing the community's understanding of quantization transferability.