ArXiv: 2603.04800
🎯 Pitch
Standard channel-wise smoothing quantization, while near-perfect for text LLMs, catastrophically fails on multimodal models—driving speech recognition error rates from 3.9% to 77.4% at W4A8. MASQuant fixes this by learning separate scaling factors per modality and using SVD-based low-rank corrections to maintain a single shared weight, recovering near-lossless performance where prior methods collapse to random outputs.
1. Executive Summary
This paper introduces Modality-Aware Smoothing Quantization (MASQuant), a post-training quantization framework for multimodal large language models that resolves two identified failure modes when channel-wise smoothing transfers from single-modality LLMs to MLLMs — Smoothing Misalignment (the dominant modality's larger activations dictating a single scaling factor that crushes weaker-modality signals) and Cross-Modal Computational Invariance (the conflict between per-modality smoothing factors and the requirement for a single set of quantized weights at inference). MASQuant addresses these through Modality-Aware Smoothing (MAS) — learning separate, modality-specific diagonal scaling matrices via direct optimization of reconstruction loss rather than closed-form formulas — and Cross-Modal Compensation (CMC) — using SVD-based whitening to transform cross-modal weight differences into low-rank correction matrices that are applied alongside a single shared quantized weight base. Evaluated on Qwen2.5-VL and Qwen2.5-Omni across vision-language and vision-audio-language benchmarks, MASQuant recovers near-FP16 accuracy at W8A8 and prevents catastrophic collapse at W4A8 where SmoothQuant degrades to effectively random outputs — most dramatically on audio tasks where SmoothQuant's LibriSpeech word error rate explodes from 3.9 to 77.4 while MASQuant maintains 3.8, establishing that modality-aware smoothing is necessary for aggressive quantization only when activation magnitude ratios across modalities are non-uniform and the dominant modality would otherwise monopolize the scaling budget.
2. Context and Motivation
The Core Problem: Channel-Wise Smoothing Breaks in Multimodal Models
The fundamental problem this paper addresses is deceptively simple: quantization techniques that work brilliantly for text-only LLMs catastrophically fail when applied to multimodal models. The paper diagnoses the root cause as a phenomenon they term smoothing misalignment — a previously unidentified failure mode that emerges when channels in a transformer layer process tokens from multiple modalities with vastly different activation magnitudes.
This is not an obvious or narrowly technical concern. The stakes are practical and substantial: multimodal LLMs are computationally massive, and deploying them on edge devices, phones, or cost-constrained cloud instances demands aggressive quantization. If a quantization scheme degrades a vision-language model's ability to read text in images (TextVQA) or an omni-modal model's ability to transcribe speech (LibriSpeech), the model becomes unusable for its intended purpose regardless of how efficiently it runs. The paper's experiments reveal just how severe this degradation can be: at W4A8 precision, SmoothQuant — one of the most widely adopted PTQ methods for LLMs — drives the Qwen2.5-Omni-3B's speech recognition error rate from 3.9% to 77.4% (Table 2). That is not a graceful quality tradeoff; that is a system breaking entirely.
The theoretical significance is equally important. The computational invariance principle — that multiplying activations by and weights by mathematically preserves a linear layer's output while changing the numerical distribution to be more quantization-friendly — is the cornerstone of modern LLM quantization. This principle assumes a single transformation matrix applies uniformly to all tokens flowing through a layer. The paper's core insight is that this assumption is fundamentally violated in multimodal architectures, where visual tokens, text tokens, and audio tokens enter the same layers but carry different distributions. The smooth mathematical elegance of computational invariance collides with the messy reality of heterogeneous input modalities.
Why This Problem Matters Now
Three convergent trends make this work timely and practically urgent:
First: The explosive adoption of multimodal LLMs. Models like Qwen2.5-VL, Qwen2.5-Omni, GPT-4V, and Gemini represent a paradigm shift from text-only reasoning to models that jointly process images, audio, video, and text. These models are inherently larger than their text-only counterparts because they embed additional encoder modules and process longer input sequences with high-dimensional multimodal tokens. Yet they are simultaneously being pushed toward deployment on smartphones, AR/VR headsets, and IoT devices where memory and power budgets are severely constrained. The tension between multimodal capability and on-device deployability makes quantization a necessity, not an optimization.
Second: The brittleness of existing PTQ methods. The channel-wise smoothing family — SmoothQuant, AWQ, OmniQuant — has been validated almost exclusively on text-only LLMs (GPT, LLaMA, OPT families). These methods implicitly assume that the activation distribution entering a linear layer, while potentially heavy-tailed due to outliers, is at least unimodal and consistent across all tokens. MLLMs violate this assumption because different modalities produce tokens that, despite being projected into a shared embedding space, retain distinct statistical signatures. The dominant modality's activation range can be 10–100× larger than a non-dominant modality's range (Section 4.1, Figure 4), meaning that a single smoothing factor computed from the aggregate distribution is effectively optimized for the loudest voice and wrong for everyone else.
Third: The inadequacy of prior MLLM-specific quantization attempts. Prior work has identified that MLLMs have modality-specific properties — MQuant observed that visual token activations can exceed text token activations by 20×, and MBQ noted unequal contributions of visual and text tokens to quantization error and proposed gradient-weighted error balancing. However, these works either address weight-only quantization (which avoids activation quantization challenges entirely) or treat the modality imbalance as a matter of error balancing rather than directly confronting the computational invariance violation at the heart of channel-wise smoothing. No prior work systematically analyzed why uniform smoothing fails, formalized the failure mathematically, and proposed a solution that both prevents the failure and preserves a single quantized weight representation.
Where Existing Approaches Fall Short
The paper identifies specific limitations in three categories of prior work:
Channel-wise smoothing methods (SmoothQuant, AWQ, OmniQuant) ignore modality heterogeneity. SmoothQuant (Equation 8) computes a per-channel scaling factor from the maximum absolute activation value in that channel across all calibration tokens. In a multimodal context, this means the scaling factor for a given channel is dictated by whichever modality produces the largest magnitude in that channel — typically vision tokens in vision-language models (Figure 4a-b). The smoothing factor is then applied uniformly to all tokens passing through that channel, whether they come from the dominant or non-dominant modality. For the non-dominant modality — say, text tokens in a layer where vision activations dominate — this means their already smaller activations get divided by a factor calibrated for much larger values, effectively squashing their signal into the quantization noise floor. The paper formalizes this as Theorem 1, proving that the SQNR degradation for non-dominant modalities under uniform smoothing grows with the non-uniformity of the modality range ratios across channels.
AWQ and MBQ partially mitigate this by searching over the hyperparameter that controls the smoothing strength, but they still compute a single scaling factor per channel that must compromise across modalities. MBQ reweights the contribution of different modalities during the search but ultimately still produces one . As the paper notes (Section 4.1), these methods optimize only — the smoothing factors themselves are never treated as free parameters, which inherently limits how much they can adapt to multimodality.
MLLM-specific quantization methods (MQuant, MBQ) identify the problem but don't fully solve it. MQuant observes the activation magnitude discrepancy and proposes modality-specific quantization ranges — allowing different modalities to use different per-tensor scaling factors for quantization. However, this approach operates at the tensor granularity, treating all channels uniformly within each modality, which cannot address the per-channel outlier patterns that channel-wise smoothing targets. More fundamentally, modifying quantization parameters per modality does not alter the underlying weight representation, so it avoids the computational invariance problem entirely — but at the cost of leaving the core outlier-mitigation mechanism (channel-wise smoothing) unrepaired.
MBQ introduces gradient-weighted error balancing to account for unequal contributions of visual and text tokens to quantization error. This helps balance the importance of errors across modalities but does not prevent the generation of those errors in the first place. The smoothing factor is still computed from a single that is optimized on a weighted combination of modality losses. The fundamental issue — that a channel's smoothing factor can be optimal for only one modality at a time — remains unaddressed.
Prior low-rank compensation methods (SVD-LLM, GPTQV) target weight compression, not cross-modal harmonization. The SVD-based whitening technique used in CMC builds on SVD-LLM and SVD-LLM v2, which apply whitening to decompose the weight matrix itself into low-rank factors for compression. But those methods operate in a single-modality context: they whitening the activations, then SVD the weights, then truncate to reduce parameters. The paper repurposes this machinery for a fundamentally different task — whitening to induce low-rank structure in the difference between modality-specific smoothed weights, enabling lightweight correction matrices that bridge the gap between per-modality optimal smoothing and a single stored weight. This is a conceptual reframing of existing tools toward a novel end.
How This Paper Positions Itself
The paper positions MASQuant not as a wholesale replacement of channel-wise smoothing but as a necessary evolution that makes the principle viable for multimodal architectures. The framing is explicitly diagnostic: Section 4.1 "Revisiting Smoothing Factors" first shows that existing methods fail, then Section 4.1 "Smoothing Misalignment" provides the mathematical explanation for why, and only then does Section 4.2 "Modality-Aware Smoothing" introduce the solution. This structure — diagnosis before prescription — is deliberate. It establishes that the problem is fundamental (not an implementation detail that could be fixed by tuning) and that MASQuant's two components (MAS and CMC) map directly onto the two sub-problems (misalignment and computational invariance respectively).
A key positioning choice is the handling of the computational invariance constraint. The naive solution to smoothing misalignment — computing separate for each modality — would produce separate quantized weight matrices for each modality, blowing up memory usage proportionally to the number of modalities. This would defeat the purpose of quantization, which is to reduce memory footprint. MASQuant accepts the naive intuition (per-modality smoothing is correct) but then adds CMC as a mechanism to collapse those per-modality smoothed weights back into a single stored weight plus lightweight compensation. This two-stage design — first optimize per-modality, then reconcile via compensation — is the paper's conceptual contribution. It separates the smoothing problem (which requires modality awareness) from the storage problem (which requires a single weight), solving each with the appropriate tool.
The paper also positions itself at the intersection of two research communities — the LLM quantization community (which has developed channel-wise smoothing, AWQ, OmniQuant, and computational invariance theory) and the multimodal model deployment community (which has begun documenting modality-specific activation patterns). By proving that the bridge between these communities is broken (Theorem 1) and offering a repair (MAS + CMC), MASQuant aims to make multimodal quantization benefit from the same theoretical guarantees that enable efficient single-modality LLM deployment. The experiments (Tables 1-2, Figures 4-6) systematically demonstrate that the repair works across model scales (3B and 7B), quantization levels (W8A8, W4A8, W4A6), modality configurations (vision-language, vision-audio-language), and task types (visual reasoning, OCR, speech recognition, multimodal reasoning), establishing breadth beyond a single favorable benchmark setting.
Finally, the paper stakes a claim about practical deployability. Section 5.5 provides end-to-end GPU kernel measurements showing that MASQuant's overhead over simpler methods (MBQ) is only 5-10% in prefill latency while maintaining identical decoding latency (since CMC corrections apply only during prefill when non-text modalities are present). Table 7 quantifies this concretely: at batch size 8 with W4A4 on a desktop RTX 4090, MASQuant with a 0.02 rank ratio achieves 3.26× speedup over FP16 vs. MBQ's 3.33×, a marginal tradeoff for substantially better accuracy (as shown in the main accuracy tables). This pragmatic demonstration — that the theoretical fix translates to real hardware with manageable overhead — is essential for a PTQ paper targeting deployment engineers.
3. Technical Approach
3.1 Reader Orientation
MASQuant is a post-training quantization framework that modifies how a pretrained multimodal LLM performs matrix multiplications so that the model can run with low-precision arithmetic (e.g., 4-bit or 8-bit) without destroying the accuracy of weaker modalities like audio or text. It solves the problem where existing quantization methods compute a single scaling factor per neuron that gets hijacked by the loudest modality (usually vision), crushing the signal from quieter modalities — and it does so through a two-stage pipeline: first, learn separate scaling factors for each modality to eliminate this interference, then apply small low-rank correction matrices during inference so that only one set of quantized weights needs to be stored, preserving the memory savings that make quantization worthwhile in the first place.
3.2 Big-Picture Architecture (Diagram in Words)
The system has three major components that operate sequentially during both a calibration phase and deployment:
-
Modality-Aware Smoothing (MAS) — During calibration, the system feeds modality-specific data (e.g., text-only, vision-only, audio-only) through each linear layer and directly learns a diagonal scaling matrix
$\mathbf{S}_m$for each modality$m$by minimizing the reconstruction error between the quantized output and the floating-point output. This produces per-modality scaling factors that are each optimal for their own modality's activation distribution, eliminating the interference where a single scaling factor must compromise across modalities. -
Cross-Modal Compensation (CMC) — After MAS, the system designates one modality (text) as the "base" and stores only
$\mathrm{Q}(\mathbf{S}_t\mathbf{W})$— the quantized weights as smoothed for text. For every other modality (vision, audio), CMC computes a compact low-rank correction matrix$\mathbf{L}_1^m \mathbf{L}_2^m$that, when added to the output of the base weight computation, recovers the output that would have been produced by that modality's own optimal quantized weights. The key mechanism is SVD-based whitening of activations, which mathematically guarantees that the weight difference between modality-specific and base weights becomes low-rank. -
Modality-Conditioned Inference Kernel — At inference time, the system checks the modality of the current input tokens. For text tokens, it simply computes
$\mathrm{Q}(\mathbf{X}_t\mathbf{S}_t^{-1}) \cdot \mathrm{Q}(\mathbf{S}_t\mathbf{W})$using the base quantized weights. For non-text tokens, it computes the same base output and then adds the low-rank correction:$\mathbf{X}_m \mathbf{S}_m^{-1} \cdot \mathbf{L}_1^m \mathbf{L}_2^m$. The low-rank matrices are only activated during the prefill phase (when non-text tokens are being processed) and not during autoregressive text decoding, keeping decoding latency identical to simpler methods.
Information flows as follows: a multimodal input enters the model → the vision/audio encoders produce token embeddings → these tokens flow through transformer layers where each linear layer consults stored MAS scaling factors ($\mathbf{S}_m$ for that modality) to quantize the input activations, applies the shared base quantized weights $\mathrm{Q}(\mathbf{S}_t\mathbf{W})$, and conditionally adds the CMC low-rank correction → text tokens are generated autoregressively without CMC overhead → final multimodal output is produced.
3.3 Roadmap for the Deep Dive
- First, the computational invariance principle underlying channel-wise smoothing PTQ, because MASQuant is a repair to this principle, and understanding what it does and why it breaks in multimodality is prerequisite.
- Second, the problem formalization — smoothing misalignment — including the mathematical definition, the proof of SQNR degradation (Theorem 1), and the empirical evidence that establishes it as the root cause of multimodal quantization failure.
- Third, the Modality-Aware Smoothing (MAS) mechanism: the learnable optimization formulation, the initialisation strategy, the loss function, and how it resolves misalignment by treating each modality independently.
- Fourth, the Cross-Modal Compensation (CMC) mechanism: the SVD-based whitening pipeline, the proof that whitening enables optimal low-rank correction (Theorem 2), the specific linear algebra procedure, and the resulting inference-time equations that maintain a single stored weight.
- Fifth, the practical deployment considerations: why text is chosen as the base modality, the computational and memory overhead analysis, and the custom GPU kernel implementation that fuses operations for real-world speedups.
3.4 Detailed, Sentence-Based Technical Breakdown
This is a systems diagnosis and repair paper whose core idea is that the computational invariance principle universally applied in LLM quantization — that you can multiply activations by $\mathbf{S}^{-1}$ and weights by $\mathbf{S}$ without changing a layer's output — breaks when a single $\mathbf{S}$ is forced to serve multiple modalities with disparate activation magnitudes, and that this breakage can be repaired by learning per-modality $\mathbf{S}_m$ and then reconciling them back to a single stored weight via SVD-whitened low-rank compensation.
Computational Invariance in Standard LLM Quantization
The foundation on which MASQuant builds — and the foundation that it identifies as broken for multimodal models — is the computational invariance principle of channel-wise smoothing quantization. Understanding this principle from first principles is essential because every subsequent design choice in MASQuant is a direct response to its violation in multimodal settings.
In a standard transformer linear layer, the forward pass computes:
where $\mathbf{X} \in \mathbb{R}^{T \times D_{\text{in}}}$ is the input activation matrix for $T$ tokens across $D_{\text{in}}$ input channels, and $\mathbf{W} \in \mathbb{R}^{D_{\text{in}} \times D_{\text{out}}}$ is the weight matrix. The core problem for quantization is that $\mathbf{X}$ often contains outliers — individual channels where the activation magnitude is 10× or 100× larger than typical values. These outlier channels force the quantization step size $\Delta$ to be large, which in turn crushes the precision available for all the normal-magnitude channels, because the same step size must cover the entire range.
What it computes: The computational invariance reformulation inserts a diagonal scaling matrix $\mathbf{S} = \text{diag}(s_1, s_2, \ldots, s_{D_{\text{in}}})$ into the matrix multiplication:
where $s_i > 0$ is the scaling factor for input channel $i$, and $\mathbf{S}^{-1} = \text{diag}(1/s_1, 1/s_2, \ldots, 1/s_{D_{\text{in}}})$ is its inverse. The key property is that $\mathbf{S}^{-1} \cdot \mathbf{S} = \mathbf{I}$, so the two insertions cancel each other out — mathematically, the output $\mathbf{Y}$ is exactly preserved. But numerically, the distribution of values changes: large activation magnitudes get divided by correspondingly large $s_i$ (shrinking the outliers), while the weight magnitudes along those same channels get multiplied by the same $s_i$ (absorbing the scale that was removed from the activations).
Why this form: The diagonal form of $\mathbf{S}$ is critical because it means the transformation operates per-channel independently — channel $i$'s activations get divided by $s_i$ and its corresponding weight row gets multiplied by $s_i$, with no cross-channel mixing. This preserves the structure of the linear layer (it remains a standard matrix multiplication, just with rescaled operands) and enables the transformation to be absorbed into the quantization process without changing the model architecture. Alternatives like a full matrix $\mathbf{S}$ would require an extra matrix multiplication, defeating the purpose of computational invariance.
The actual quantization is then applied to the smoothed operands:
where $\mathrm{Q}(\cdot)$ is the quantization function defined in Equation 1 of the paper, mapping floating-point values to low-precision integers by dividing by a step size $\Delta$, rounding, adding a zero-point $z$, and clamping to the integer range $[q_{\text{min}}, q_{\text{max}}]$. The reconstruction loss minimized by PTQ is:
Existing methods compute $\mathbf{S}$ via closed-form heuristics, not learning. SmoothQuant (Xiao et al., 2023) sets each diagonal entry to:
where $\max_t |x_{t,i}|$ is the maximum absolute activation value in channel $i$ across all calibration tokens, $\max_j |w_{j,i}|$ is the maximum absolute weight value in row $i$ of $\mathbf{W}$, and $\beta \in [0,1]$ is a migration strength hyperparameter that controls how much of the quantization difficulty is shifted from activations to weights. When $\beta = 1$, all smoothing is applied to activations (making them easier to quantize); when $\beta = 0$, all smoothing is applied to weights. In practice, $\beta$ is typically tuned to balance the quantization difficulty.
AWQ (Lin et al., 2024) uses a similar formulation but replaces the per-channel maximum with a per-channel average:
where the optimal $\beta$ is found via grid search to minimize the quantization reconstruction loss on calibration data.
Both methods share the same structural property: they compute a single scalar $s_i$ per channel from aggregate statistics of all calibration tokens, regardless of which modality those tokens belong to. The $\max_t$ in SmoothQuant and the $\sum_{t=1}^n$ in AWQ both pool across tokens without distinguishing whether a given token came from an image patch, a text substring, or an audio frame.
Why this works for text-only LLMs: In a single-modality model, all tokens are drawn from the same distribution (next-token prediction on text corpus). Channel 42 might have larger activations than channel 87, but that relationship is consistent across all text tokens — a token from a Wikipedia article and a token from a coding problem both use channel 42 similarly. The single $s_i$ computed from aggregate statistics therefore applies reasonably well to every token. The $\max_t$ is a conservative choice that ensures even the largest outlier gets properly scaled, which makes SmoothQuant slightly suboptimal for typical tokens (they could tolerate a more aggressive scaling) but safe in the sense that no token gets crushed.
Why this breaks for multimodal LLMs: In a multimodal model, tokens from different modalities share the same channels but carry fundamentally different activation magnitudes. A single $s_i$ computed from the aggregate $\max_t$ will be determined by whichever modality produces the largest value in channel $i$ — and that modality will dominate all $i$ where it has larger activations. For non-dominant modalities, $s_i$ is much larger than what would be appropriate for their own activation range. Divided by an excessively large $s_i$, their already-smaller activations get pushed toward zero, where quantization error becomes a dominant fraction of the signal. This is the fundamental mechanism the paper names smoothing misalignment, and it is the direct motivation for computing separate $\mathbf{S}_m$ per modality.
Formalizing Smoothing Misalignment (The Problem Statement)
The paper provides a mathematical formalization of this failure mode to establish that it is not merely an engineering inconvenience but a fundamental quantitative degradation with predictable structure. This formalization serves as both diagnostic tool and design guide.
Defining activation range per modality. For a given linear layer, define the activation range of channel $i$ for modality $m \in \mathcal{M}$ as:
where $\mathbf{x}_{t,i}^m$ is the activation value of token $t$ (from modality $m$) at channel $i$. This is the same $\max$ operator used by SmoothQuant but applied only to tokens of a single modality. The range ratio between a dominant modality $m$ and a non-dominant modality $m'$ at channel $i$ is:
When visual tokens dominate text tokens, $\boldsymbol{\alpha}_i^{\text{vis},\text{txt}} \gg 1$ for most channels where vision activations are larger. Crucially, $\boldsymbol{\alpha}_i$ varies across channels — some channels have a 5× ratio, others a 50× ratio, and some rare channels have a 0.5× ratio where text happens to be larger. This non-uniformity of $\boldsymbol{\alpha}_i$ across channels is what causes SQNR degradation, as proven below.
How the unified smoothing factor is computed. When SmoothQuant calibrates on mixed-modality data, the $\max_t$ in Equation 8 takes the maximum across all tokens regardless of modality. In any channel $i$, the maximum will be contributed by the modality with the larger range:
where $m^* = \arg\max_m \mathbf{R}_i^m$ is the dominant modality for channel $i$. The unified smoothing factor $\mathbf{s}^{\text{uni}}$ is thus approximately proportional to the dominant modality's range — it is, in effect, $\mathbf{s}^{m^*}$ for whichever modality wins each channel.
What this means operationally: For a non-dominant modality $m'$, the model applies a smoothing factor $s_i \propto \max_m \mathbf{R}_i^m$ which is much larger than the smoothing factor $s_i^{m'} \propto \mathbf{R}_i^{m'}$ that would be optimal for $m'$ alone. Since activations are divided by $s_i$, the non-dominant modality's activations get divided by a factor that is $\boldsymbol{\alpha}_i = \mathbf{R}_i^m / \mathbf{R}_i^{m'}$ times too large. Their values are over-smoothed into the quantization noise floor.
The paper formally quantifies this damage through Theorem 1, which gives the SQNR degradation in closed form.
Theorem 1 (SQNR Degradation under Smoothing Misalignment). Consider a layer processing multimodal inputs with dominant modality
$m$and non-dominant modality$m'$. Let$\boldsymbol{\alpha}_i^{m,m'} = \mathbf{R}_i^m / \mathbf{R}_i^{m'}$denote the range ratio at channel$i$. Using unified smoothing$\mathbf{s}^{\text{uni}} \approx \mathbf{s}^m$yields SQNR degradation (exact as$\alpha^{m,m'} \gg 1$):
Breaking down Theorem 1 symbol by symbol. The left side $\mathrm{SQNR}(\mathbf{s}^{\text{uni}}, \mathbf{x}_t^{m'})$ is the signal-to-quantization-noise ratio of token $\mathbf{x}_t$ from the non-dominant modality $m'$ when the unified smoothing factor $\mathbf{s}^{\text{uni}}$ (calibrated on mixed data and thus dominated by modality $m$) is used for quantization. The first term on the right, $\mathrm{SQNR}(\mathbf{s}^{m'}, \mathbf{x}_t^{m'})$, is the SQNR that would be achieved if the token were smoothed with its own modality's optimal smoothing factor $\mathbf{s}^{m'}$. The second term is the degradation — a penalty subtracted from the ideal SQNR.
The degradation term depends on two components: $d$ is the number of channels in the layer, and $\boldsymbol{\alpha}_i^{m,m'}$ is the per-channel range ratio. The expression inside the logarithm has the structure of a non-uniformity measure: if all $\boldsymbol{\alpha}_i$ are equal (uniform scaling across channels), then $\min_i (\boldsymbol{\alpha}_i)^2 = \alpha^2$ and $\sum_i 1/\boldsymbol{\alpha}_i^2 = d/\alpha^2$, making the fraction equal to 1, so $\log_{10}(1) = 0$ and there is no degradation. When the $\boldsymbol{\alpha}_i$ vary across channels — some channels have large ratios (vision dominates strongly) while others have small ratios (similar ranges) — the $\min_i$ term becomes small relative to the average of the reciprocals, making the fraction less than 1 and the logarithm negative, producing a positive degradation penalty.
Why this form matters — the proof structure. The proof (provided in the paper's Theorem 1 block) proceeds by comparing the upper bounds of SQNR under the two smoothing strategies, assuming that the worst-case quantization scenario occurs when a token's activations are uniform relative to their ranges ($x_1/R_1 \approx x_2/R_2 \approx \cdots \approx x_d/R_d$). Under this worst-case assumption:
- With optimal smoothing
$\mathbf{s}^{m'}$, the SQNR upper bound is$10 \log_{10} d$— it depends only on the dimensionality (more channels provide more degrees of freedom to absorb quantization noise, hence higher SQNR). - With unified smoothing
$\mathbf{s}^{\text{uni}}$, the SQNR upper bound becomes$10 \log_{10} \left( \frac{\sum_i 1/\alpha_i^2}{(\max_i 1/\alpha_i)^2} \right)$— it depends on the distribution of the inverse range ratios.
The degradation $\Delta$ is the difference between these two upper bounds, which the proof shows is always $\leq 0$ with equality if and only if all $\alpha_i$ are equal. The paper's Figures 4(c) and 4(d) empirically verify that $\alpha_i$ is indeed highly non-uniform across channels in both vision-language and omni-modal models, confirming that the conditions for degradation are met in practice.
Empirical validation of Theorem 1. Figure 2 in the paper provides the direct experimental evidence. The authors selected 32 samples from OmniBench (a multimodal benchmark covering joint text-audio-visual reasoning), computed the average SQNR for each layer in Qwen2.5-Omni-3B, and compared the SQNR achieved by SmoothQuant's unified smoothing against that achieved by modality-specific smoothing. The degradation is consistently negative across layers, with some layers losing more than 20 dB of SQNR — a factor of 100× in the ratio of signal power to quantization noise power. This directly confirms that the mathematical prediction of Theorem 1 manifests in real multimodal architectures.
Modality-Aware Smoothing (MAS) — The First Repair
Having established that a single $\mathbf{S}$ cannot serve multiple modalities without crushing the weaker ones, the paper introduces Modality-Aware Smoothing as the direct solution: learn separate smoothing matrices $\mathbf{S}_m$ for each modality $m$ by directly optimizing them to minimize quantization reconstruction error on modality-specific data.
Why learn, not compute via formula. The paper explicitly observes that existing methods (SmoothQuant's Equation 8, AWQ's Equation 9, MBQ's weighted variant) optimize only the hyperparameter $\beta$ while the smoothing factors themselves are never treated as free parameters (Section 4.1). This is a structural limitation: any $\beta$-parameterized formula produces smoothing factors of the form $s_i \propto (\text{activation_statistic}_i)^{\beta}$, which forces all channels to follow the same power-law relationship between activation magnitude and smoothing strength. But the optimal $s_i$ for a given channel may depend on the channel's specific interaction between activation outliers and weight sensitivity — a relationship that cannot be captured by a single global parameter.
The paper instead adopts the learnable smoothing formulation from OmniQuant (Shao et al., 2023), which directly optimizes the diagonal entries of $\mathbf{S}$:
This treats each $s_i$ as an independent learnable parameter initialized from a closed-form formula (to start near a good solution) and then refined via gradient-based optimization to directly minimize the quantization error on calibration data. The paper extends this to the multimodal case by optimizing a set of modality-specific scaling matrices $\{\mathbf{S}_m\}_{m \in \mathcal{M}}$.
The MAS initialisation. For each modality $m$, the diagonal entries of $\mathbf{S}_m$ are initialized as:
where $\max_t |x_{t,i}^m|$ is the maximum absolute activation in channel $i$ across only tokens of modality $m$, and $\max_j |w_{j,i}|$ is the maximum absolute weight value in the corresponding row of $\mathbf{W}$ (shared across modalities since weights are modality-agnostic). This is a per-modality variant of the SmoothQuant formula with $\beta = 0.5$ (equal migration). The square root balances the scaling between activations and weights — it represents the geometric midpoint where the activation range and weight range have equal magnitude in the quantization domain.
What the initialisation achieves: At initialization, each $\mathbf{S}_m$ is exactly the SmoothQuant scaling matrix that would be computed if that modality's data were the only calibration data. This means MAS starts from a solution that already prevents smoothing misalignment (each modality gets its own appropriately-sized scaling) but may not be optimal for minimizing the actual quantization reconstruction error, which depends on the detailed interaction between the scaled activations and the quantized weights.
The MAS optimization objective. The $\mathbf{S}_m$ matrices are then optimized by minimizing a modality-balanced reconstruction loss:
where $\lambda_m$ is a loss weight for modality $m$ (set to 1.0 for all modalities in the paper's main experiments, as determined by the ablation in Table 4), and $\mathcal{L}_{\text{MAE}}$ is the Mean Absolute Error between the quantized output and the floating-point output:
Why MAE instead of MSE. The paper follows MBQ's choice of MAE over the more common Mean Squared Error. MAE penalizes errors linearly rather than quadratically, which makes the optimization less sensitive to occasional large outliers in reconstruction error. This is particularly important in the multimodal setting where certain token positions (e.g., image patch boundaries) may have inherently noisier activations that would dominate a squared-error objective. The MAE objective produces a more balanced optimization that improves all tokens rather than overfitting to the worst-case outliers.
The optimization procedure is independent per modality. Crucially, each $\mathbf{S}_m$ is optimized on only data from its own modality. The loss for vision smoothing is computed using batches of pure vision tokens; the loss for text smoothing uses pure text tokens. There is no cross-modality interference in the optimization — the $\mathbf{S}_m$ are separate parameters that never compete for the same objective. This ensures that each modality's smoothing factors converge to whatever pattern minimizes that modality's quantization error, without having to compromise with other modalities' requirements.
The paper trains for 2 epochs on calibration data, as determined by the ablation in Table 5. At epoch 0 (initialization), PPL is 23.9; at epoch 1, PPL drops to 17.0; at epoch 2, average accuracy peaks at 56.9%; beyond epoch 2, accuracy slightly declines, indicating overfitting to the calibration set.
The SQNR improvement from MAS. After optimization, each modality's smoothed activations $\mathbf{X}_m \mathbf{S}_m^{-1}$ have their outlier channels appropriately scaled down, making them amenable to quantization. The resulting inference for modality $m$ would be:
The paper's Table 3 validates the necessity of modality-aware smoothing. At W4A8 quantization on Qwen2.5-Omni-3B, using a single unified smoothing factor (even a learnable one optimized on mixed data) produces a LibriSpeech word error rate of 77.4% — essentially random output on a task where the FP16 model achieves 3.9%. In contrast, MAS reduces this to 3.8% WER, nearly matching the FP16 baseline. The gap between uniform and modality-aware smoothing widens when the uniform factor is learned rather than computed via closed form, demonstrating that gradient-based optimization on mixed-modality data does not naturally discover modality-specific structure — it converges to a compromise that hurts the minority modality.
The fundamental unresolved problem: multiple quantized weights. If we stop at MAS, inference would require storing and loading $\mathrm{Q}(\mathbf{S}_m \mathbf{W})$ for each modality $m$. For a tri-modal model handling text, vision, and audio, this triples the memory footprint of every quantized linear layer — effectively negating the memory savings that motivated quantization in the first place. For a 7B model with 32 transformer layers, each containing multiple linear projections (Q, K, V, O, and two FFN layers), the additional storage would be on the order of billions of parameters. This is where Cross-Modal Compensation enters to reconcile the per-modality smoothing optimality with the single-weight storage requirement.
Cross-Modal Compensation (CMC) — Reconciling Modality-Specific Smoothing with Computational Invariance
The challenge CMC addresses is: given that we have modality-specific smoothing matrices $\mathbf{S}_m$ that produce different quantized weights $\mathrm{Q}(\mathbf{S}_m \mathbf{W})$ for each modality, how can we store only one quantized weight matrix during inference while still achieving the output accuracy of modality-specific quantization?
The strategy: designate one modality as base, compensate for others. CMC designates text as the "base" modality and stores $\mathrm{Q}(\mathbf{S}_t \mathbf{W})$ as the single shared quantized weight matrix. (The choice of text as base is motivated by practical decoder efficiency — during autoregressive text generation, only text tokens are processed, so the base weight is used exclusively and no compensation overhead is incurred.) For any non-text modality $m \neq \text{text}$, the system wants to produce approximately the same output that $\mathrm{Q}(\mathbf{S}_m \mathbf{W})$ would produce, but using the stored $\mathrm{Q}(\mathbf{S}_t \mathbf{W})$ plus a lightweight correction.
Defining the weight residual. The difference between what an ideal modality-specific quantized weight would produce and what the base weight produces, for a given set of vision activations $\mathbf{X}_v$, is:
where $\boldsymbol{\Delta}\mathbf{W} \in \mathbb{R}^{D_{\text{in}} \times D_{\text{out}}}$ is the weight residual — the difference between the vision-smoothed floating-point weights and the text-smoothed quantized weights. If we could store $\boldsymbol{\Delta}\mathbf{W}$ compactly, we could add $\mathbf{X}_v \mathbf{S}_v^{-1} \cdot \boldsymbol{\Delta}\mathbf{W}$ as a correction to the base output.
Why naive low-rank approximation fails. A natural first thought is to apply truncated SVD directly to $\boldsymbol{\Delta}\mathbf{W}$, approximating it as $\boldsymbol{\Delta}\mathbf{W} \approx \mathbf{U}_r \boldsymbol{\Sigma}_r \mathbf{V}_r^\top$ with rank $r$. However, this minimizes $\|\boldsymbol{\Delta}\mathbf{W} - \mathbf{L}\|_F$ — the Frobenius norm of the weight approximation error — rather than $\|\mathbf{X}_v \mathbf{S}_v^{-1} (\boldsymbol{\Delta}\mathbf{W} - \mathbf{L})\|_F$ — the actual output error that matters for model accuracy. Standard SVD has no way to account for the fact that some directions in weight space matter more (because they align with large activation components) and others matter less. Additionally, $\boldsymbol{\Delta}\mathbf{W}$ may not inherently possess low-rank structure — it is the difference between two arbitrary matrices, and without additional structure, its singular values may decay slowly.
The key insight: whitening induces low-rank structure in the output-relevant weight residual. The paper's central technical innovation in CMC is to transform the weight residual into a space where it becomes low-rank with respect to the output error metric. This is achieved by whitening the vision activations.
The whitening transform is derived from the covariance of the smoothed vision activations:
where $(\mathbf{X}_v \mathbf{S}_v^{-1})^\top (\mathbf{X}_v \mathbf{S}_v^{-1}) \in \mathbb{R}^{D_{\text{in}} \times D_{\text{in}}}$ is the $D_{\text{in}} \times D_{\text{in}}$ covariance matrix of the smoothed vision activations (computed over the calibration token dimension $T$), and SVD decomposes it into eigenvectors $\mathbf{P}$ and eigenvalues $\boldsymbol{\Lambda} = \text{diag}(\lambda_1, \ldots, \lambda_{D_{\text{in}}})$ where $\lambda_1 \geq \lambda_2 \geq \cdots \geq \lambda_{D_{\text{in}}} \geq 0$.
What the covariance SVD reveals. The eigenvectors $\mathbf{P}$ are the principal directions of variation in the smoothed vision activations. The eigenvalues $\lambda_i$ measure how much variance is along each direction. Directions with large eigenvalues correspond to activation patterns that are strongly present in the vision data; directions with near-zero eigenvalues correspond to patterns that are essentially absent.
The whitening matrix is then constructed as:
where $\boldsymbol{\Lambda}^{1/2} = \text{diag}(\sqrt{\lambda_1}, \ldots, \sqrt{\lambda_{D_{\text{in}}}})$. The inverse transform $\mathbf{T}^{-1}$ maps a whitened vector back to the original activation space.
What whitening does operationally. Consider a vector $\mathbf{a} = \mathbf{x} \mathbf{S}_v^{-1}$ — a single token's smoothed vision activation. Multiplying by $\mathbf{T}^{-1}$ produces:
such that the whitened activations satisfy $\mathbf{a}_{\text{white}}^\top \mathbf{a}_{\text{white}} = 1$ (they are orthonormal) when averaged over the calibration set. In other words, the whitening transform decorrelates and normalizes the activation components so that each direction in the whitened space has equal variance. Directions that were weak in the original space get amplified; directions that were strong get compressed.
When the same whitening transform is applied to the weight residual:
the strong activation directions (which have large $\lambda_i$ and get compressed by the inverse of $\sqrt{\lambda_i}$) become less dominant in the singular value spectrum, while weak activation directions become more prominent. The result is that the singular values of $\mathbf{T} \boldsymbol{\Delta}\mathbf{W}$ decay more rapidly than those of $\boldsymbol{\Delta}\mathbf{W}$ alone, because the whitening has equalized the contribution of different activation subspaces. Figure 5 in the paper empirically confirms this: the effective rank (a measure of how many singular values are needed to capture a given fraction of the total variance) is substantially lower for the whitened residual than the raw residual.
The truncated SVD in whitened space. After whitening, the system performs truncated SVD:
where $\mathbf{U}_r \in \mathbb{R}^{D_{\text{in}} \times r}$, $\boldsymbol{\Sigma}_r \in \mathbb{R}^{r \times r}$, and $\mathbf{V}_r^\top \in \mathbb{R}^{r \times D_{\text{out}}}$ capture the top $r$ singular components. The rank $r$ is a hyperparameter that trades off correction accuracy against storage and compute overhead — a smaller $r$ means less memory for the compensation matrices and fewer FLOPs during inference, but potentially larger residual error. The paper evaluates at rank ratios (rank divided by hidden dimension) of 0.01, 0.02, and 0.05.
Inverting the whitening to get the final compensation. The rank-$r$ approximation in whitened space is then mapped back to the original weight space:
This is factorized into two low-rank matrices for efficient storage and computation:
The correction applied at inference time is then:
which requires $D_{\text{in}} \times r$ operations for the first multiplication and $r \times D_{\text{out}}$ for the second, compared to $D_{\text{in}} \times D_{\text{out}}$ for the full weight matrix — a $(D_{\text{in}} + D_{\text{out}}) r / (D_{\text{in}} D_{\text{out}})$ fraction of the original compute, which is very small when $r \ll \min(D_{\text{in}}, D_{\text{out}})$.
Theorem 2 formalizes why this is optimal. The paper proves that this SVD-based whitening procedure produces the theoretically optimal rank-$r$ approximation for the output reconstruction loss.
Theorem 2 (Optimal Low-rank Compensation). The rank-
$r$matrix$\mathbf{L}^* = \mathbf{L}_1 \mathbf{L}_2$, where$\mathbf{L}_1$,$\mathbf{L}_2$are given by the rank-$r$truncated SVD defined in Equation 25, minimizes the reconstruction loss:
i.e.,
The proof (provided in full in the paper) proceeds by observing that $\mathbf{X}_v \mathbf{S}_v^{-1} = \mathbf{U} \mathbf{T}$ where $\mathbf{U}$ is orthonormal (this follows from the SVD decomposition of the covariance matrix). Then:
since multiplying by an orthonormal matrix $\mathbf{U}$ preserves the Frobenius norm. This reduces the problem to finding the best rank-$r$ approximation of $\mathbf{T} \boldsymbol{\Delta}\mathbf{W}$ in Frobenius norm, which is exactly what truncated SVD provides (Eckart-Young-Mirsky theorem). Substituting the whitened SVD back gives $\mathbf{L}^* = \mathbf{T}^{-1} \mathbf{U}_r \boldsymbol{\Sigma}_r \mathbf{V}_r^\top$, which equals the paper's $\mathbf{L}_1 \mathbf{L}_2$ construction, and the reconstruction loss achieves the theoretical minimum $\mathcal{L}_{\min}^2 = \|\text{SVD}(\mathbf{X}_v \mathbf{S}_v^{-1} \boldsymbol{\Delta}\mathbf{W})\|_F^2$.
Why this proof matters for the paper's claims. The theorem establishes that CMC is not merely an ad-hoc heuristic but a principled optimal solution under the Frobenius norm output reconstruction criterion. It inherits the optimality guarantees of truncated SVD, but applies them in the transformed whitened space where the low-rank structure is most pronounced. Without whitening, the same SVD truncation applied directly to $\boldsymbol{\Delta}\mathbf{W}$ would minimize $\|\boldsymbol{\Delta}\mathbf{W} - \mathbf{L}\|_F$ rather than the output-relevant $\|\mathbf{X}_v \mathbf{S}_v^{-1} (\boldsymbol{\Delta}\mathbf{W} - \mathbf{L})\|_F$, producing a suboptimal correction that wastes rank budget on weight-space directions that have minimal impact on the actual model output.
The final inference equations. With both MAS and CMC integrated, the forward pass for a linear layer becomes:
For text tokens, the computation is identical to standard channel-wise smoothing quantization — no correction is needed because the stored weight $\mathrm{Q}(\mathbf{S}_t \mathbf{W})$ is exactly the text-smoothed quantized weight. For non-text tokens, the base quantized computation is performed first (using the text-smoothed weights), and then the low-rank correction $\mathbf{L}_1^m \mathbf{L}_2^m$ is added to compensate for the difference between the text-smoothed and modality-$m$-smoothed weights. The correction term uses the floating-point smoothed activations $\mathbf{X}_m \mathbf{S}_m^{-1}$ (not quantized) to avoid compounding quantization error in the compensation path — the compensation itself is performed in floating-point or high-precision integer arithmetic.
The diagram in Figure 3 illustrates this pipeline concretely for an MLP block in a dual-modal (text-vision) setting. During calibration (Figure 3a), text and vision data flow through separate branches: the text branch optimizes $\mathbf{S}_t$ via MAS, the vision branch optimizes $\mathbf{S}_v$ via MAS, and then CMC computes $\mathbf{L}_1^v$ and $\mathbf{L}_2^v$ from the whitened residual between vision-smoothed and text-smoothed weights. During inference (Figure 3b), text tokens take the direct path through $\mathrm{Q}(\mathbf{S}_t\mathbf{W})$, while vision tokens take the same path plus the low-rank correction $\mathbf{L}_1^v \mathbf{L}_2^v$, with a modality mask gating which path is active.
Design Rationale for Choosing Text as the Base Modality
The choice of text as the base modality for CMC is not arbitrary — it is motivated by the autoregressive decoding structure of multimodal LLMs. During the prefill phase, the model processes all input tokens (which may include images, audio, and the initial text prompt) in parallel. During the decoding phase, the model generates output text tokens one at a time autoregressively, and each new text token attends to all previous tokens (both multimodal inputs and previously generated text). This means that during decoding, every linear layer processes only text tokens (the query and key/value projections for self-attention, and the FFN projections).
If text is the base modality, then CMC corrections are needed only during prefill — when non-text tokens are actually being processed. During the entire decoding phase, the model uses the base text-smoothed weights $\mathrm{Q}(\mathbf{S}_t \mathbf{W})$ without any correction, and this is exactly the right weight for text tokens. The result is that decoding latency is identical to a standard quantized model with no CMC overhead.
If vision or audio were chosen as the base modality instead, then every decoding step (which processes text tokens) would require CMC correction to compensate for the mismatch between vision-smoothed base weights and text-optimized weights. This would add overhead to every autoregressive step, which dominates total inference time for long outputs. Table 6 in the paper formalizes this: when text is the base modality, CMC is invoked $m$ times (once per prefill with non-text modalities) and the decoding phase has zero CMC overhead. When a non-text modality is the base, CMC must be invoked during every decoding step, requiring $2 \times d \times r$ extra FLOPs per token per layer.
Additionally, during decoding, the key-value cache stores the keys and values computed during prefill (which include the CMC-corrected representations for non-text tokens). These cached representations already incorporate the correction, so they do not need to be recomputed. This amortizes the CMC prefill cost over all subsequent decoding steps.
Practical Deployment: GPU Kernel Implementation
The paper validates that MASQuant's theoretical design translates to real hardware speedups through a custom CUDA kernel implementation (Section 5.5). The kernel, built on Nunchaku (Li et al., 2024), addresses several practical challenges that a naive PyTorch implementation would face.
Fused projection and quantization operators. A naive implementation would perform the matrix multiplication in floating-point and then quantize the output in a separate kernel launch. Each kernel launch incurs overhead (synchronization, memory transfer between GPU layers of the memory hierarchy), and intermediate results must be written to and read from global memory. The custom kernel fuses the projection (matrix multiply) and quantization into a single kernel, keeping intermediate results in registers and shared memory. This minimizes memory bandwidth consumption, which is typically the bottleneck for low-precision inference on GPUs rather than compute throughput.
Modality-aware mask for conditional low-rank execution. The low-rank correction $\mathbf{L}_1^m \mathbf{L}_2^m$ should only be applied for non-text tokens and only for layers where CMC is active. The kernel implements a multi-modal mask — a boolean tensor that marks which token positions belong to which modality — to conditionally gate the low-rank computation. When processing a batch of tokens, the kernel reads the mask, identifies which tokens are non-text, applies the low-rank correction only for those tokens, and bypasses it for text tokens. This conditional execution avoids wasting compute on text tokens that don't need compensation.
Kernel-level operation for the low-rank path. The CMC correction $\mathbf{X}_m \mathbf{S}_m^{-1} \cdot \mathbf{L}_1^m \mathbf{L}_2^m$ is implemented as two matrix multiplications: first, the activation $\mathbf{X}_m \mathbf{S}_m^{-1}$ (shape $[T_m, D_{\text{in}}]$) multiplies $\mathbf{L}_1^m$ (shape $[D_{\text{in}}, r]$) to produce an intermediate of shape $[T_m, r]$; second, this intermediate multiplies $\mathbf{L}_2^m$ (shape $[r, D_{\text{out}}]$) to produce the correction of shape $[T_m, D_{\text{out}}]$. When the rank $r$ is small (e.g., $r = 0.02 \times D_{\text{in}} \approx 40$ for typical hidden dimensions of ~2000), these are very small matrix multiplications compared to the main projection ($[T_m, D_{\text{in}}] \times [D_{\text{in}}, D_{\text{out}}]$). The kernel schedules these small operations to fill GPU streaming multiprocessor occupancy gaps in the main projection's execution.
Measured performance. Table 7 reports end-to-end prefill-stage performance on a Desktop RTX 4090 GPU with Qwen2.5-VL-7B at sequence length 2048 under W4A4 quantization:
- At batch size 1, FP16 prefill latency is 191.82 ms. MBQ achieves 2.79× speedup (68.65 ms). MASQuant with rank ratio 0.01 achieves 2.67× (71.62 ms), with rank 0.02: 2.63× (72.89 ms), with rank 0.05: 2.49× (77.10 ms). The overhead relative to MBQ is 4.3% at rank 0.01, 6.2% at rank 0.02, and 12.3% at rank 0.05.
- At batch size 8, FP16 prefill is 2146.01 ms. MBQ achieves 3.33× speedup (643.68 ms). MASQuant achieves 3.30× (649.29 ms) at rank 0.01, 3.26× (657.68 ms) at rank 0.02, and 3.07× (696.44 ms) at rank 0.05.
The memory savings are substantial: from 13.73 GB (FP16) to 4.85–5.37 GB for batch size 1 (2.56–2.83× reduction), and from 17.27 GB to 8.89–9.42 GB for batch size 8 (1.83–1.94× reduction). The extra memory overhead of the low-rank matrices (storing $\mathbf{L}_1^m$ and $\mathbf{L}_2^m$ for each non-text modality in each linear layer) accounts for the slight memory increase relative to MBQ — at rank 0.02, MASQuant uses about 4% more memory than MBQ.
Decoding latency is unaffected. The paper notes that during decoding, the CMC low-rank path is completely inactive (only text tokens are processed), so MASQuant's decoding speed is identical to MBQ's. Since decoding dominates total inference time for typical long-form generation tasks, the practical overhead of CMC is limited to the prefill phase, which is a small fraction of total latency for most use cases.
Configuration and Hyperparameter Summary
The paper reports the following key hyperparameters and design choices, mostly gleaned from ablation studies in Tables 3–5 and Section 5.4:
Calibration data. The authors use calibration data from the model's training distribution, split by modality. For each modality, a separate calibration set is constructed (e.g., vision-only samples for optimizing $\mathbf{S}_v$, text-only for $\mathbf{S}_t$, audio-only for $\mathbf{S}_a$). The specific number of calibration samples is not stated explicitly but follows standard PTQ practice (typically 128–256 samples per modality).
MAS training. Optimizer: gradient-based optimization of the diagonal entries of $\mathbf{S}_m$ (the paper does not specify the exact optimizer, but OmniQuant uses Adam with a small learning rate for learnable scaling factors). Loss: MAE (Mean Absolute Error) between quantized and floating-point layer outputs. Epochs: 2 (determined by the ablation in Table 5). Modality loss weights $\lambda_m$: set to 1.0 for all modalities (Table 4 ablation shows that $\lambda_t = \lambda_v = 1.0$ achieves the best balance — 17.2 PPL and 56.9% average accuracy on Qwen2.5-VL-3B at W4A8, compared to steep degradation when either modality's weight is reduced).
CMC rank selection. The rank $r$ is expressed as a ratio of the hidden dimension $d$: evaluated at ratios 0.01, 0.02, and 0.05. Figure 6 shows that CMC with SVD whitening surpasses the non-whitened baseline at very low rank ratios — CMC with a rank ratio of 0.08 already exceeds the MBQ baseline that uses no low-rank correction, while the non-whitened baseline requires a rank ratio of 0.4 to match MBQ. This 5× reduction in required rank directly validates the whitening efficacy. The SQNR saturates around 3.5 dB improvement at higher rank ratios, indicating diminishing returns beyond rank ratio ~0.1.
Base modality. Text is always the base modality for CMC. This is justified by the decoding-phase analysis in Table 6: text-as-base requires CMC only during prefill (when non-text tokens exist), adding $2d \times r$ FLOPs per non-text token per layer. If vision were the base, CMC would be required during every decoding step for text tokens, adding continuous overhead.
Quantization granularity. The paper follows standard per-channel weight quantization and per-token activation quantization, consistent with SmoothQuant and AWQ. For W8A8, both weights and activations are quantized to 8 bits. For W4A8, weights are 4-bit and activations 8-bit. For W4A6, weights are 4-bit and activations are 6-bit. The quantization function $\mathrm{Q}(\cdot)$ uses uniform affine quantization with a learned or calibrated step size $\Delta$, as defined in Equation 1.
4. Key Insights and Innovations
Innovation 1: Diagnosing Smoothing Misalignment as a First-Class Failure Mode in Multimodal Quantization
The paper's most distinctive intellectual contribution is not a method but a diagnosis. Before MASQuant, the quantization community knew that multimodal models were harder to quantize — MQuant observed that visual activations can be 20× larger than text activations, and MBQ noted unequal modality contributions to error — but these observations were treated as empirical nuisances to be compensated for, not as symptoms of a fundamental structural violation. The dominant assumption was that channel-wise smoothing methods (SmoothQuant, AWQ, OmniQuant) would transfer to multimodal settings with modest calibration adjustments, perhaps by tuning β differently on mixed-modality data.
This paper identifies that the assumption itself is broken at the mathematical level. The computational invariance principle — the cornerstone of all channel-wise smoothing — states that inserting a diagonal scaling matrix S and its inverse preserves a linear layer's output. This is true for any S, so existing methods select S to be quantization-friendly. But the principle says nothing about what happens when the tokens flowing through that layer are heterogeneous. The paper's core diagnostic move is to show that the identity of S matters differently to different modalities, and a single S computed from aggregate token statistics is mathematically guaranteed to be wrong for at least one modality whenever activation magnitude ratios across modalities are non-uniform across channels.
This is a conceptual reframing of the problem space. Prior work saw "multimodal quantization is harder" as a matter of degree — more diverse distributions require more careful calibration. MASQuant reframes it as a matter of kind — the very mechanism that makes channel-wise smoothing work (a single S per layer) is structurally incompatible with multimodal architectures. This distinction matters because it changes what a solution should look like: you cannot fix a structural violation with better hyperparameter tuning; you need to change the structure.
The formalization in Theorem 1 elevates this from intuition to proof. The theorem shows that the SQNR degradation for non-dominant modalities under unified smoothing is not merely empirically observed but mathematically necessary whenever the per-channel range ratios α_i across modalities are non-uniform. The degradation term vanishes only when all α_i are equal — a condition that Figures 4(c) and 4(d) show is violated in practice for both vision-language and omni-modal models. This means that SmoothQuant's failure on audio tasks (WER exploding from 3.9 to 77.4 in Table 2) is not an implementation bug or a calibration sample quality issue — it is the expected behavior of the algorithm under the mathematical conditions that multimodal models create.
Prior work's error-balancing approach (MBQ's modality-weighted β search) can be understood through this lens as treating the symptom (unequal error) without addressing the cause (a single S cannot be simultaneously optimal for all modalities). MBQ reduces the harm of smoothing misalignment by weighting modalities differently during calibration, but it does not prevent the misalignment from occurring. MASQuant's separate per-modality S_m matrices prevent it entirely.
Innovation 2: Decomposing the Problem into Independent Smoothing and Reconciliation Stages
The paper's architectural contribution is not that per-modality smoothing is better — that follows directly from the diagnosis — but rather the two-stage decomposition that separates the smoothing problem (which demands modality-specific solutions) from the storage problem (which demands a single weight representation). This decomposition is the conceptual framework that makes MASQuant possible, and it represents a genuinely different way of thinking about computational invariance in multimodal settings.
Before this work, the computational invariance constraint was treated as monolithic: you compute one S, apply it, and the quantization benefits come from the smoothed numerical distributions. The idea that you could compute multiple optimal S_m and then retroactively reconcile them into a shared representation had not been proposed. Existing methods that touched on modality-specific processing — MQuant's per-modality quantization ranges — sidestepped the computational invariance issue entirely by modifying only the quantization parameters (step sizes, zero points) rather than the smoothed weight representation itself. This is a fundamentally weaker approach because it leaves the outlier channels unaddressed; the per-channel smoothing that makes modern LLM quantization effective is absent.
What makes this decomposition intellectually distinctive is that it takes a constraint that initially appears to be a hard tradeoff — either per-modality smoothing with separate stored weights (memory-expensive) or unified smoothing with a shared weight (accuracy-degrading) — and splits it into two solvable subproblems. The smoothing stage can be modality-pure: each S_m is optimized on its own modality's data with no cross-modality compromise. The reconciliation stage then takes these independently optimal solutions and finds a compact way to represent them relative to a single base.
This division of labor has downstream implications that go beyond the specific method. It suggests that future work on multimodal quantization should not try to find a single transformation that works for all modalities simultaneously — that is provably impossible in the general case per Theorem 1 — but should instead develop better reconciliation mechanisms that can bridge independently optimized per-modality solutions into shared representations. The paper's use of SVD-based whitening is one such reconciliation mechanism, but the framework of "optimize per-modality, then reconcile" is more general and could accommodate other compensation strategies (learned corrections, adapter-style modules, dynamic weight selection).
Table 3 provides the cleanest empirical evidence for why the decomposition matters. When smoothing factors are learned on mixed-modality data (trying to optimize a single S for all modalities simultaneously), the LibriSpeech WER is 77.4 — worse than when the smoothing factor is computed via the closed-form SmoothQuant formula (which also fails, but less catastrophically). This reveals that gradient-based optimization on mixed data does not naturally discover modality-specific structure; it converges to a compromise that is actively harmful for the minority modality. Only when the optimization is constrained to be modality-specific (MAS training separate S_m on separate data) does the WER recover to 3.8, near FP16 quality.
Innovation 3: Repurposing SVD Whitening for Cross-Modal Weight Reconciliation with Optimality Guarantees
The technical mechanism of CMC — using SVD-based whitening to transform cross-modal weight differences into low-rank form — is built from existing tools (SVD-LLM showed that whitening enables low-rank weight compression). But the paper's innovation is repurposing this machinery toward a fundamentally different end: not compressing a single weight matrix, but reconciling multiple modality-specific weight matrices into a shared representation with provable optimality.
In SVD-LLM, the goal is to reduce the parameter count of a weight matrix by whitening the activations, performing SVD on the whitened weights, and truncating. The whitening serves to equalize the importance of different weight directions so that truncation removes the least output-relevant components. In MASQuant, the goal is different: there are two weight matrices — the vision-smoothed floating-point weights and the text-smoothed quantized weights — and the system needs a compact representation of their difference. The whitening is applied to the difference matrix ΔW, and the SVD truncation in whitened space produces the rank-r correction that best approximates the output of the ideal vision-smoothed quantized weights.
The conceptual leap is recognizing that whitening solves a cross-modal alignment problem, not just a within-modality compression problem. Without whitening, the singular values of ΔW may decay slowly because strong activation directions dominate the spectrum, making low-rank approximation inefficient — the rank budget gets spent on directions that matter for the dominant modality's activations rather than directions where the two modalities differ most. Whitening equalizes the contribution of different activation subspaces, forcing the SVD to allocate its rank budget to directions that genuinely differ between modalities rather than directions that simply have large activation magnitudes.
Theorem 2 formalizes why this is optimal: the SVD in whitened space minimizes the output reconstruction loss ||X_v S_v^{-1} (ΔW - L)||_F, not merely the weight-space error ||ΔW - L||_F. This is the correct metric because what matters for model accuracy is the output of the layer, not the abstract similarity between weight matrices. Standard SVD on the raw ΔW would be optimal for weight-space error but suboptimal for output-space error. The proof shows that whitening converts the output-space minimization into an equivalent weight-space minimization in the whitened domain, enabling the use of standard truncated SVD while guaranteeing output-space optimality.
Figure 5 empirically validates the whitening benefit: the effective rank of ΔW drops substantially after whitening, confirming that the transformation induces the low-rank structure that makes compensation efficient. Figure 6 shows the practical consequence: CMC with whitening surpasses the MBQ baseline at a rank ratio of only 0.08, while the non-whitened baseline requires a rank ratio of 0.4 to match MBQ — a 5× difference in required rank budget. This is a direct measure of how much the whitening contributes beyond naive low-rank approximation.
The significance of this contribution extends beyond quantization. The general problem of "reconciling multiple weight matrices optimized for different input distributions into a shared representation with lightweight per-distribution corrections" appears in other contexts — federated learning with heterogeneous client data, multi-task model merging, continual learning with distribution shift. CMC's whitening-then-truncate approach, with its optimality guarantee under the output reconstruction criterion, provides a template for these related problems.
Innovation 4: Establishing Quantitative Boundaries on the Pretraining-vs-Quantization Tradeoff in Multimodal Settings
The paper's systematic evaluation across diverse benchmarks, modality configurations, and quantization levels — while not a conceptual innovation in itself — produces an empirical finding with significant implications: multimodal models can be quantized to 8-bit weight-and-activation (W8A8) with essentially no accuracy loss when modality-specific smoothing is properly handled, but aggressive quantization (W4A8 and below) creates modality-specific cliffs where weaker modalities collapse before stronger ones.
The W8A8 results in Tables 1 and 2 are striking not because they show improvement over baselines (which is expected from any well-designed method) but because they show recovery to near-FP16 performance. On Qwen2.5-VL-7B at W8A8, MASQuant achieves 74.2% average accuracy across five vision-language benchmarks compared to 74.5% for FP16 — a 0.3 percentage point gap that is statistically negligible given benchmark variance. On Qwen2.5-Omni-7B, W8A8 achieves 48.8% on MMMU vs. FP16's 50.0%, and 2.9 WER on LibriSpeech vs. FP16's 2.9 WER. This establishes that 8-bit quantization of multimodal LLMs is a solved problem when the smoothing mechanism respects modality boundaries — there is no inherent barrier to W8A8 deployment.
The W4A8 results reveal a more nuanced picture that the paper does not explicitly theorize but which emerges clearly from the data: modality collapse is non-uniform and predictable from activation magnitude ratios. In Table 2 on Qwen2.5-Omni-3B at W4A8, SmoothQuant's audio performance collapses catastrophically (WER 77.4 on LibriSpeech, 94.2 on Wenetspeech) while vision performance degrades more gracefully (MMMU drops from 43.3 to 30.0). MASQuant prevents the audio collapse (WER 3.6 and 8.7 respectively) but vision performance at W4A8 (MMMU 36.7) is still notably below FP16 (43.3). This pattern — audio is more vulnerable than vision at a given quantization level — is explained by the activation magnitude hierarchy: audio < text < vision. When quantization becomes aggressive enough that even per-modality smoothing cannot fully compensate, the modality with the smallest activation magnitudes (audio) hits the noise floor first.
This finding has practical deployment implications that go beyond the paper's stated contributions. It suggests that mixed-precision quantization strategies for multimodal models should not only be per-layer or per-channel but also per-modality — allocating more bits to the activations of weaker modalities while more aggressively quantizing dominant modalities. The paper's W4A8 results where vision degrades less than audio at the same weight precision imply that the activation quantization budget should be allocated asymmetrically.
The FLOPs-matched comparison between quantization and using a larger model is implicit rather than explicit in this paper (unlike the reference example's Section 7), but the data permits an inference: at W8A8, a quantized multimodal model achieves near-FP16 accuracy at roughly 4× less memory and 2.5× faster inference (from Table 7). This means that for deployment scenarios where memory is the bottleneck, W8A8 quantization effectively gives you the accuracy of the full model at a fraction of the cost — a clear win. At W4A8, the accuracy tradeoff is task-dependent: vision tasks degrade modestly (a few percentage points), audio tasks degrade severely without modality-aware smoothing but remain near-FP16 with MASQuant. The practical takeaway is that 4-bit weight quantization is viable for vision-language models when processed through MASQuant but requires careful evaluation per modality.
The negative results on the most aggressive settings (W4A6 in Tables 1 and 2, and the abysmal RTN W4A8 results) establish a floor: quantization below 4-bit weights or 8-bit activations remains challenging even with modality-aware smoothing. The paper does not solve this; it delineates where the current approach works and where it doesn't, which is a form of negative result that guides future research toward the remaining barriers.
5. Experimental Analysis
Evaluation Methodology
-
Dataset. The paper evaluates on multiple multimodal benchmarks spanning three modality configurations. For vision-language evaluation: MMMU (multidisciplinary multimodal understanding), OCRBench (optical character recognition), Vizwiz (visual question answering from blind users), ScienceQA (science question answering with thought chains), and TextVQA (text reading in images). For audio-text evaluation: LibriSpeech test-other split and Wenetspeech test-net split, both measured via Word Error Rate (WER). For tri-modal (vision-audio-text) evaluation: OmniBench, which covers joint text-audio-visual reasoning. The exact number of evaluation samples per benchmark is not specified in the paper but follows each benchmark's standard test split.
-
Base model(s). The paper evaluates on Qwen2.5-VL (3B and 7B parameters) for vision-language experiments and Qwen2.5-Omni (3B and 7B parameters) for tri-modal experiments. The Omni architecture includes a 675M ViT-based vision encoder, a Whisper-large-v3-based audio encoder, and a Transformer decoder ("Thinker") for the language model component. Crucially, the paper quantizes only the LLM decoder component (Thinker), leaving the vision and audio encoders in FP16. This choice is motivated by generality — the Thinker is the largest component and the one shared across all modality processing paths.
-
Metrics. For vision-language benchmarks, the primary metric is task accuracy (%). For MMMU, this is 4-choice multiple-choice accuracy averaged across subjects. For OCRBench, Vizwiz, ScienceQA, and TextVQA, accuracy follows each benchmark's standard evaluation protocol. The "Avg" column in Table 1 reports the arithmetic mean across these five benchmarks. For audio tasks, Word Error Rate (WER, lower is better) measures the fraction of words incorrectly transcribed. For OmniBench (tri-modal), accuracy is reported. The paper also reports Signal-to-Quantization-Noise Ratio (SQNR) in decibels (dB) for diagnostic analysis (Figures 2, 4, 6), computed as:
which quantifies the ratio of signal power to quantization error power for individual token activations.
-
Baselines. The paper compares against five methods: RTN (Round-To-Nearest, the simplest quantization baseline with no smoothing or compensation), SmoothQuant (Xiao et al., 2023 — the canonical channel-wise smoothing method using Equation 8), AWQ (Lin et al., 2024 — activation-aware weight quantization using per-channel average statistics from Equation 9), MBQ (Li et al., 2025 — modality-balanced quantization that reweights modality contributions during the β hyperparameter search), and FP16 (the unquantized baseline serving as an upper bound). For AWQ, the paper follows AWQ's standard calibration procedure on mixed-modality data. For MBQ, the paper reproduces the method's gradient-weighted error balancing as described in the original work. All methods are compared at identical bit widths (W4A16, W8A8, W4A8, W4A6).
-
Generation budget / compute accounting. Since PTQ is a calibration-time optimization (not test-time scaling), the relevant compute budget is the number of calibration samples used to optimize smoothing factors and compensation matrices, not inference-time generation count. The paper does not report explicit calibration sample counts but follows standard PTQ practice — typically 128–256 samples per modality. The inference efficiency is measured through end-to-end GPU latency and memory usage (Table 7) on a Desktop RTX 4090, using a custom CUDA kernel that fuses projection and quantization operators. The paper measures both prefill-stage speedup (including CMC overhead for non-text modalities) and reports that decoding latency is unaffected (CMC corrections are inactive during autoregressive text generation).
-
Cross-validation / statistical protocol. The paper does not report cross-validation or statistical significance testing. The evaluations are single-pass on standard benchmark test splits. This is typical for PTQ papers (where the cost of multiple quantization-calibration-evaluation cycles is high), but it means that small differences between methods (e.g., 74.2 vs 74.4 average accuracy in Table 1) should be interpreted cautiously — they may not be statistically significant given typical benchmark variance. The paper does report ablation studies on individual hyperparameters (training epochs in Table 5, modality loss weights in Table 4) where small controlled experiments are feasible, but these are diagnostic rather than statistical.
Main Quantitative Results
Vision-Language MLLMs (Table 1)
The central finding for vision-language models is that MASQuant recovers near-FP16 accuracy at W8A8 and prevents the catastrophic collapse that simpler methods experience at W4A8.
W8A8 performance: On Qwen2.5-VL-3B, MASQuant achieves 70.9% average accuracy across five benchmarks, matching FP16's 70.1% within 0.8 percentage points and slightly outperforming SmoothQuant (69.9%) and MBQ (70.1%). On Qwen2.5-VL-7B, MASQuant achieves 74.4% average, within 0.1 points of MBQ (74.4%) and 0.1 points below FP16 (74.5%). The conclusion is that 8-bit weight-and-activation quantization is essentially lossless for vision-language MLLMs across all tested methods, with MASQuant achieving parity with the best existing method. The fact that even RTN achieves 74.1% at W8A8 on the 7B model suggests that 8-bit quantization is well-tolerated regardless of smoothing strategy when activations and weights both retain sufficient precision.
W4A16 (weight-only quantization): At 4-bit weights with 16-bit activations, the gap between methods is modest but MASQuant shows systematic improvement over baselines. On Qwen2.5-VL-3B, MASQuant achieves 69.9% average vs. AWQ's 69.5%, MBQ's 69.3%, and RTN's 67.8%. On Qwen2.5-VL-7B, MASQuant reaches 74.2% vs. MBQ's 73.7% and AWQ's 73.5%. The 0.5–0.9 point improvement over MBQ is modest but consistent across both model scales, suggesting that per-modality smoothing provides marginal benefit even in the weight-only setting where activation quantization (the primary source of smoothing misalignment) is absent. This likely reflects a secondary effect: smoothing factors also affect how weight quantization errors propagate, and modality-specific smoothing may produce weight scaling that is better aligned with the loss landscape for each modality's typical activation patterns.
W4A8 (aggressive weight and activation quantization): This is where smoothing misalignment becomes catastrophic for baseline methods and MASQuant's benefit is most pronounced. On Qwen2.5-VL-3B:
- RTN completely fails: average accuracy drops to 5.1%, with four of five benchmarks scoring 0.0%. This is expected — without any smoothing, the outlier channels in activations make 8-bit activation quantization infeasible.
- SmoothQuant achieves 57.4% average, a dramatic improvement over RTN but still 12.7 points below FP16. Individual benchmarks show sharp degradation: OCRBench drops from 79.3 (FP16) to 66.9, TextVQA drops from 77.9 to 63.9.
- MBQ improves to 64.6% average, recovering 7.2 points over SmoothQuant via modality-balanced error weighting, but still leaves a 5.5-point gap to FP16.
- MASQuant achieves 64.7% average, matching MBQ within 0.1 points while using a fundamentally different mechanism (per-modality smoothing plus low-rank compensation rather than error reweighting). Notably, MMMU accuracy with MASQuant at 46.7% actually exceeds FP16's 42.2% — this is physically impossible if quantization only degrades performance, suggesting that the FP16 evaluation on this specific benchmark split has non-trivial variance and the 46.7% should be interpreted as effectively matching FP16 quality within noise.
On Qwen2.5-VL-7B at W4A8, the pattern holds: MASQuant reaches 69.0% average vs. MBQ's 68.5%, SmoothQuant's 64.8%, and RTN's 67.4%. The RTN result on 7B is notably better than on 3B (67.4% vs. 5.1%), suggesting that larger models are more robust to naive quantization — their activations may have less extreme outlier channels, or their redundancy allows them to absorb quantization error more gracefully.
W4A6 (extreme quantization): At 4-bit weights and 6-bit activations, the performance cliff steepens for all methods. On Qwen2.5-VL-3B:
- SmoothQuant deteriorates to 53.6% average (from 57.4% at W4A8), with MMMU dropping to 22.5% (from 25.6%) and TextVQA to 55.9% (from 63.9%).
- MBQ achieves 59.7%, gaining only 2.3 points over SmoothQuant at W4A6 vs. 7.2 points at W4A8, indicating that error reweighting alone cannot compensate when activation precision falls below 6 bits.
- MASQuant reaches 59.9%, statistically tied with MBQ but notably recovering only 2-5 points over SmoothQuant compared to the larger gap at W4A8. This suggests that when activation quantization becomes extremely aggressive, the benefit of modality-specific smoothing narrows — all modalities are being crushed, and preventing misalignment in the smoothing factors helps less when the quantization step size itself is too coarse to preserve signal regardless of scaling.
On Qwen2.5-VL-7B at W4A6, MASQuant achieves 63.0% average vs. MBQ's 62.9% and SmoothQuant's 61.0%. The gap between methods narrows further, with all three smoothing-based methods within 2.0 points of each other. This convergence at extreme quantization suggests a floor effect: at 6-bit activations, the quantization error is dominated by the limited precision of the uniform quantizer rather than by smoothing misalignment.
A non-obvious pattern in the per-benchmark results: Across almost all bit-widths and model sizes, MASQuant's strongest gains appear on OCRBench and MMMU — benchmarks that require fine-grained visual understanding (reading text in images, complex cross-modal reasoning). On Qwen2.5-VL-7B at W4A8, MASQuant achieves 72.8% on OCRBench vs. SmoothQuant's 70.2% (a 2.6-point gain) and 43.3% on MMMU vs. SmoothQuant's 37.8% (a 5.5-point gain). On the other hand, TextVQA actually degrades slightly with MASQuant at some bit widths (e.g., at W4A8 on 3B, MASQuant gets 69.2% vs. SmoothQuant's 63.9% — a gain — but at W8A8, MASQuant gets 77.7% vs. SmoothQuant's 77.5%, essentially tied). This benchmark-dependent variation has no clear theoretical explanation and may reflect benchmark-specific sensitivity to different aspects of quantization error.
Vision-Audio-Language (Omni-Modal) MLLMs (Table 2)
The tri-modal experiments reveal a more dramatic and modality-specific failure pattern that directly validates the paper's central thesis about smoothing misalignment.
W8A8 performance on tri-modal: On Qwen2.5-Omni-3B, MASQuant achieves 42.3% on MMMU and 46.1% on OmniBench (the two vision-heavy benchmarks), while audio performance holds at 3.8 WER on LibriSpeech and 7.9 WER on Wenetspeech — essentially matching FP16's 3.9 and 7.5 WER respectively. On the 7B model, MASQuant achieves 49.9% MMMU and 46.9% OmniBench, with audio at 2.8 WER (LibriSpeech) and 6.9 WER (Wenetspeech), again near-FP16 quality (2.9, 7.1). The key observation: at W8A8, all modalities are preserved, confirming that 8-bit tri-modal quantization is viable with proper per-modality smoothing.
W4A8: The failure of audio under uniform smoothing becomes catastrophic. This is the paper's most striking result and the clearest empirical evidence for smoothing misalignment as a first-class failure mode. On Qwen2.5-Omni-3B at W4A8:
- RTN collapses entirely across all modalities: LibriSpeech WER = 109.7 (worse than random), Wenetspeech WER = 105.6, MMMU accuracy = 28.9% (from 43.3% FP16), OmniBench = 29.7% (from 43.8%). This confirms that 4-bit weight + 8-bit activation quantization without smoothing is non-functional for tri-modal models.
- SmoothQuant partially preserves vision but destroys audio: MMMU drops modestly to 30.0 (from 43.3), but audio collapses spectacularly — LibriSpeech WER jumps to 77.4 (from 3.9) and Wenetspeech WER to 94.2 (from 7.5). This is a 20× degradation for audio — the model cannot transcribe speech at all under SmoothQuant's unified smoothing. The vision-audio-text OmniBench accuracy similarly drops to 27.3 (from 43.8). This is the smoking gun for Theorem 1: visual tokens dominate the activation ranges (as shown in Figure 4a), the unified smoothing factor s_uni is determined by vision, and audio tokens — with their much smaller activation magnitudes — get crushed into the quantization noise floor.
- MBQ partially rescues audio but at a cost: MBQ improves audio substantially — LibriSpeech WER drops to 9.5 (from SmoothQuant's 77.4) and Wenetspeech to 8.5 (from 94.2). However, MMMU degrades to 27.8 (from SmoothQuant's 30.0) and OmniBench to 36.7. This suggests that MBQ's modality-balanced error weighting shifts the smoothing factor toward a compromise: audio gets better smoothing at the expense of vision accuracy. The cross-modal tradeoff is explicit — you can balance errors but cannot eliminate both simultaneously with a single S.
- MASQuant preserves all modalities: LibriSpeech WER = 3.6 (matching FP16's 3.9), Wenetspeech WER = 8.7 (close to FP16's 7.5), MMMU = 36.7 (recovering above SmoothQuant's 30.0 and MBQ's 27.8), OmniBench = 41.4 (substituting above MBQ's 36.7 and approaching FP16's 43.8). The takeaway: modality-specific smoothing prevents any single modality from being sacrificed. Audio, the weakest modality by activation magnitude, survives intact because its S_audio is calibrated for its own range rather than being overwhelmed by vision's smoothing factors.
On the 7B Omni model at W4A8, the audio preservation advantage of MASQuant narrows but persists: LibriSpeech WER = 2.9 (FP16 = 2.9) vs. SmoothQuant's 8.6 and MBQ's 3.8, Wenetspeech WER = 8.0 vs. SmoothQuant's 8.3 and MBQ's 8.2. On MMMU, MASQuant achieves 48.8 vs. MBQ's 47.8, both recovering near the FP16 50.0. The 7B model's greater capacity appears to provide some inherent robustness against smoothing misalignment — audio fares less catastrophically under SmoothQuant (WER 8.6 vs. 77.4 on 3B), suggesting that larger models have less extreme activation range disparities between modalities or more redundancy to absorb quantization error.
W4A6 on tri-modal: At 4-bit weights and 6-bit activations, the tri-modal results further validate the modality hierarchy of vulnerability. On Qwen2.5-Omni-3B:
- SmoothQuant's audio collapses to effectively random performance: LibriSpeech WER = 87.8, Wenetspeech = 99.5. MMMU drops to 28.9 (from 43.3), OmniBench plummets to 18.8 (from 43.8).
- MBQ achieves partial recovery but with a clear cross-modal tradeoff pattern: LibriSpeech WER = 10.8 (much better than SmoothQuant's 87.8, but worse than its own W4A8 result of 9.5), Wenetspeech = 10.4. However, MMMU improves slightly to 31.4 (from 27.8 at W4A8), and OmniBench jumps to 46.9 — actually exceeding FP16's 43.8. This is the inverse of the W4A8 pattern, where MBQ improved audio at the cost of vision. At W4A6, MBQ improves vision at the cost of audio degradation relative to W4A8. This flipping of which modality benefits confirms that a single smoothing factor forces a zero-sum game between modalities at extreme quantization.
- MASQuant maintains balanced quality: LibriSpeech WER = 3.7 (near FP16 3.9), Wenetspeech = 8.9 (near FP16 7.5), MMMU = 32.2, OmniBench = 42.2. The audio remains near-FP16, and the vision performance, while below FP16, avoids the sharp declines seen in SmoothQuant.
On the 7B Omni at W4A6, MASQuant achieves 4.7 WER on LibriSpeech (vs. SmoothQuant's 5.8 and MBQ's 6.4), 8.7 on Wenetspeech (vs. SmoothQuant's 19.8 and MBQ's 14.8), and 36.8 on MMMU (vs. SmoothQuant's 37.8 — the one benchmark where MASQuant slightly underperforms SmoothQuant on the 7B model). This single negative data point (MMMU 36.8 vs. 37.8 for SmoothQuant) is the only instance across Tables 1 and 2 where MASQuant is beaten by a baseline on any metric at any bit width, and it only occurs on the 7B model at the most aggressive quantization level. It likely reflects the small benchmark size and evaluation noise rather than a genuine regime where SmoothQuant is preferable.
OmniBench under W4A4 (Table 7 footnotes): Table 7 reports end-to-end prefill latency under W4A4 setting, but does not include accuracy numbers at this bit width. This is a notable omission — the paper demonstrates speedup at W4A4 but does not verify that the accuracy at this quantization level remains acceptable. The main accuracy tables stop at W4A6 for tri-modal and W4A6 for vision-language, leaving W4A4 as a "how fast but how accurate?" gap.
GPU Kernel Performance (Table 7)
The paper validates that the theoretical framework translates to real hardware speedups through custom CUDA kernel measurements on a Desktop RTX 4090. The key finding is that MASQuant achieves 2.5–3.3× speedup over FP16 with only 5–10% overhead relative to the simpler MBQ method, and with identical decoding latency.
Prefill-stage latency: On Qwen2.5-VL-7B at sequence length 2048 under W4A4 quantization:
| Batch Size | Method | Rank Ratio | Prefill (ms) | Speedup vs FP16 | Memory (GB) | Memory Saving |
|---|---|---|---|---|---|---|
| 1 | FP16 | — | 191.82 | 1.00× | 13.73 | 1.00× |
| 1 | MBQ | — | 68.65 | 2.79× | 4.85 | 2.83× |
| 1 | MASQuant | 0.01 | 71.62 | 2.67× | 4.97 | 2.76× |
| 1 | MASQuant | 0.02 | 72.89 | 2.63× | 5.04 | 2.73× |
| 1 | MASQuant | 0.05 | 77.10 | 2.49× | 5.37 | 2.56× |
| 8 | FP16 | — | 2146.01 | 1.00× | 17.27 | 1.00× |
| 8 | MBQ | — | 643.68 | 3.33× | 8.89 | 1.94× |
| 8 | MASQuant | 0.01 | 649.29 | 3.30× | 9.02 | 1.92× |
| 8 | MASQuant | 0.02 | 657.68 | 3.26× | 9.08 | 1.90× |
| 8 | MASQuant | 0.05 | 696.44 | 3.07× | 9.42 | 1.83× |
Three observations from these numbers:
-
Speedup improves with batch size. At batch size 1, MBQ achieves 2.79× speedup; at batch size 8, it achieves 3.33×. This is expected for GPU inference — larger batches better amortize kernel launch overhead and saturate the GPU's streaming multiprocessors. The trend holds for MASQuant as well (2.67× to 3.30× at rank 0.01).
-
MASQuant overhead scales with rank ratio and is modest at practical ranks. At the lowest rank ratio (0.01), MASQuant adds 4.3% prefill latency overhead over MBQ at batch size 1 (71.62 vs 68.65 ms) and 0.9% at batch size 8 (649.29 vs 643.68 ms). At rank ratio 0.02, the overhead is 6.2% (batch 1) and 2.2% (batch 8). At the highest tested rank (0.05), overhead rises to 12.3% (batch 1) and 8.2% (batch 8). The overhead is smaller at larger batch sizes because the CMC low-rank correction cost is proportional to the number of non-text tokens (which is fixed per sample) while the base matrix multiplication cost scales with total tokens, so the relative overhead is diluted at higher batch sizes.
-
Memory overhead is modest. At rank ratio 0.02 and batch size 1, MASQuant uses 5.04 GB vs. MBQ's 4.85 GB — a 3.9% increase. The extra memory comes from storing L1^m and L2^m for each non-text modality in each linear layer. At rank ratio 0.02, each low-rank correction adds D_in × r + r × D_out parameters per layer. For a 7B model with hidden dimension ~4096 and intermediate FFN size ~11008, the per-layer overhead at r = 0.02 × 4096 ≈ 82 is approximately 82 × (4096 + 11008) ≈ 1.2M parameters per modality per linear layer, which is small relative to the base weight of 4096 × 11008 ≈ 45M parameters for the FFN up-projection.
Decoding latency: The paper explicitly states that during decoding, only text tokens are processed, so the CMC low-rank path is inactive. Decoding latency is therefore identical to MBQ. Since decoding dominates total inference time for typical long-form generation workloads (where output length >> input length), the practical overhead of MASQuant relative to MBQ is even smaller than the prefill numbers suggest.
What is missing: The paper does not report end-to-end generation throughput (tokens per second) or latency for a complete prompt-to-response pipeline. The prefill-stage measurements in Table 7 cover only the initial processing of input tokens, not the autoregressive decoding phase. A deployment engineer would want to know the throughput in tokens/second for a typical multimodal input-output pair (e.g., one image plus a text question, generating a 100-word response) to assess the real-world impact. The paper's claim that "decoding latency is identical" is plausible given the architecture, but without measurement, the actual percentage of total inference time spent in prefill vs. decoding for representative workloads is unknown.
Comparative Summary Across Bit Widths and Modalities
Table 1 and Table 2 together establish a clear difficulty hierarchy:
-
W8A8: Solved for all modalities. All smoothing-based methods (SmoothQuant, MBQ, MASQuant) achieve near-FP16 accuracy across vision-language and tri-modal benchmarks. MASQuant is competitive but not uniquely necessary at this bit width. The practical implication: if 8-bit quantization is acceptable, any of these methods works, and RTN is even viable for weight-only W4A16.
-
W4A16: Modest benefit from modality awareness. All methods recover most of the FP16 accuracy gap. MASQuant edges out baselines by 0.5–1.0 points on average, which may or may not be practically significant depending on application tolerance.
-
W4A8: Modality-specific smoothing is critical for tri-modal models, beneficial for dual-modal. The audio collapse under SmoothQuant (WER 77.4 on LibriSpeech) demonstrates that unified smoothing is fundamentally broken for tri-modal quantization. MASQuant is the only method that preserves all modalities near-FP16 quality. For vision-language models, the gap between MBQ and MASQuant narrows (64.6 vs. 64.7 on 3B, 68.5 vs. 69.0 on 7B), suggesting that the dual-modality case is less extreme — with only two modalities, a compromise smoothing factor may be sufficient if loss weights are carefully balanced.
-
W4A6: Modality-specific smoothing remains beneficial but gains narrow. The performance floor rises, and all methods converge. At this extreme, the quantization error from limited precision begins to dominate over smoothing misalignment. Even MASQuant cannot fully recover FP16 accuracy — the gap grows to 3–8 points for vision tasks and 1–2 WER points for audio.
Ablation Studies and Robustness Checks
Modality loss weights (Table 4): On Qwen2.5-VL-3B at W4A8, the paper ablates λ_t and λ_v (loss weights for text and vision in the MAS optimization, Equation 13). Equal weights (λ_t = λ_v = 1.0) performs best: PPL = 17.2, average accuracy = 56.9%. Reducing vision weight to λ_v = 0.5 drops MMMU accuracy from 37.8 to 30.0 while providing minimal text benefit. Further reducing to λ_v = 0.1 causes PPL to collapse to 33.7. Reducing text weight (λ_t = 0.5) degrades all metrics. This ablation establishes that the per-modality smoothing factors are genuinely optimized independently — each modality's loss is necessary for that modality's quality — and that there is no cross-modality information leakage in the optimization that would allow a modality to benefit from reduced smoothing of another modality. The paper sets λ_t = λ_v = 1.0 for all main experiments.
Training epochs for MAS smoothing factors (Table 5): On Qwen2.5-VL-3B at W4A8, smoothing factor optimization converges rapidly. At epoch 0 (initialization from Equation 12), PPL is 23.9. After 1 epoch, PPL drops to 17.0. Peak average accuracy (56.9%) is achieved at epoch 2, with slight decline thereafter (presumably overfitting to the calibration set). The paper uses 2 epochs for all main experiments. The rapid convergence (1 epoch captures most of the gain) suggests that the initialisation from the per-modality SmoothQuant formula (Equation 12) is already close to optimal and that the learnable optimization provides fine-tuning rather than discovering fundamentally different smoothing patterns. This is consistent with the paper's claim that the learnable formulation can discover patterns beyond any β-parameterized formula — the improvement from epoch 0 to epoch 2 (PPL 23.9 → 17.0) shows that optimization does find non-trivial refinements, but the initialisation quality means these refinements are incremental.
Effect of MAS (Table 3): The paper ablates Modality-Aware Smoothing on Qwen2.5-Omni-3B at W4A8 by comparing three configurations: (1) a uniform smoothing factor (single S optimized on mixed-modality data), (2) a learned uniform smoothing factor with optimization, and (3) separate per-modality smoothing factors (full MAS). The key finding is that learned uniform smoothing performs worse than closed-form uniform smoothing: on LibriSpeech, closed-form SmoothQuant achieves 77.4 WER (already catastrophic), while learned uniform smoothing achieves even higher (worse) WER — the exact number is not given in the paper text but can be inferred from the difference between SmoothQuant (77.4) and MAS (3.8) in Table 2. In contrast, per-modality smoothing (MAS) recovers to 3.8 WER. This counterintuitive result — gradient-based optimization on mixed data makes things worse — validates the paper's central claim that smoothing misalignment is a structural problem, not a tuning problem. Gradient descent on the combined loss converges to a compromise that is worse for the minority modality than the α-prior formula that at least captures per-channel statistics correctly.
Effect of CMC — whitening vs. no whitening (Figure 6): The paper compares SQNR improvement as a function of rank ratio for CMC with SVD-based whitening vs. CMC with direct SVD on ΔW (no whitening). On Qwen2.5-Omni-3B at W4A6, CMC with whitening surpasses the MBQ baseline (which uses no low-rank correction) at a rank ratio of approximately 0.08, while the non-whitened baseline requires a rank ratio of roughly 0.4 to match MBQ — a ~5× difference in required rank budget. The SQNR of CMC with whitening saturates near 3.5 dB improvement at higher rank ratios, indicating diminishing returns beyond rank ratio ~0.1. The MAS baseline (separate per-modality smoothing but no CMC compensation — meaning multiple stored weights) achieves higher SQNR than any single-weight method, confirming that CMC trades some accuracy (relative to ideal multi-weight storage) for memory efficiency.
Base modality choice (Table 6): The paper justifies choosing text as the base modality by analyzing the computational cost implications. When text is the base, CMC corrections are needed only during prefill (when non-text tokens are present), adding 2d × r FLOPs per non-text token per layer. When a non-text modality is the base, CMC corrections would be needed during every decoding step for text tokens. Table 6 formalizes this with formulas: prefill cost includes CMC for m extra modalities, with computation cost 2 × r × d_m × n_m and memory cost (d + d_out) × r × m, while decoding cost is zero for text-as-base and 2 × r × d for non-text-as-base. Given that decoding typically dominates total inference time, text-as-base is the only practical choice.
Modality dominance analysis (Figure 4): The paper quantifies the percentage of unified smoothing factors derived from each modality when applying SmoothQuant to multimodal models. In Qwen2.5-Omni-3B (Figure 4a), visual tokens dominate smoothing factor computation across most components — in some layers, over 90% of channels have their smoothing factor determined by vision activations. Text contributes substantially in early layers but is marginalized in deeper layers. Audio's contribution is negligible across all layers, confirming that audio is the "weakest" modality in terms of activation magnitude. In Qwen2.5-VL-3B (Figure 4b), the vision dominance is similarly pervasive but slightly less extreme, with text still contributing to smoothing factors in some layers. The per-channel range ratio distributions (Figures 4c and 4d) show that α_i (the ratio of dominant to non-dominant modality range) is highly non-uniform across channels, confirming the condition required for Theorem 1's degradation to manifest.
Effective rank of ΔW after whitening (Figure 5): The paper measures the effective rank of ΔW (the weight difference between vision-smoothed and text-smoothed weights) before and after SVD-based whitening. On Qwen2.5-VL-3B, the effective rank is substantially reduced after whitening across all layers — the red bars (whitened) are consistently lower than the blue bars (raw). The reduction is most dramatic in mid-to-deep layers. On Qwen2.5-Omni-3B, the pattern is similar. This visualization directly validates the key mathematical claim underlying CMC: whitening induces low-rank structure in the cross-modal weight residual, enabling efficient low-rank approximation.
What is missing in the ablation analysis:
-
No ablation on calibration data quantity. The paper uses standard PTQ calibration sizes (128–256 samples) but does not vary this to determine how many samples are needed for MAS convergence or CMC rank discovery. For practitioners, knowing minimum calibration requirements is essential — especially for tri-modal models where curated per-modality calibration data may be harder to obtain.
-
No ablation on the rank allocation per layer. CMC applies a uniform rank ratio across all layers. Some layers may have larger cross-modal weight differences (requiring higher rank) while others are near-identical across modalities (requiring near-zero rank). A per-layer adaptive rank allocation could achieve the same overall accuracy with lower total memory overhead, but this is not explored.
-
No ablation on the choice of base modality between text alternatives. While Table 6 provides a cost analysis, it does not empirically measure what happens if vision is chosen as the base. The accuracy impact (vs. text-as-base) is never measured.
-
No comparison with quantization-aware training (QAT) baselines. The paper only compares against PTQ methods. QAT approaches that incorporate quantization during fine-tuning could potentially learn modality-robust internal representations that simplify quantization. Including even a single QAT baseline would contextualize the PTQ results.
-
No sensitivity analysis on the whitening computation. The CMC whitening transform depends on the covariance of calibration activations. If the calibration data distribution drifts from deployment data (e.g., different image resolutions, different audio sample rates), the whitening transform may become suboptimal. The paper does not test robustness to calibration-deployment distribution shift.
Critical Assessment
Does the evidence support the claim that smoothing misalignment is the fundamental obstacle to applying channel-wise smoothing to MLLMs?
The evidence in Table 2 provides strong support, but with two important qualifications. First, the "smoking gun" result is the W4A8 audio collapse under SmoothQuant on Omni-3B: LibriSpeech WER = 77.4 vs. MASQuant's 3.8. This is a genuinely dramatic gap that cannot be explained by calibration noise or random variance — it is a qualitative failure where the model effectively stops performing the audio task. The per-modality smoothing (MAS) completely prevents this collapse, which is exactly what the smoothing misalignment diagnosis predicts.
However, the evidence is strongest for tri-modal models at aggressive quantization levels. For vision-language (dual-modal) models, the gap between MASQuant and MBQ at W4A8 is much narrower: 64.7 vs. 64.6 on 3B, 69.0 vs. 68.5 on 7B (Table 1). These differences are within the range of typical benchmark variance (0.1–0.5 points) and cannot be confidently attributed to MASQuant's conceptual superiority. The smoothing misalignment diagnosis predicts degradation that scales with the number of modalities and the disparity in activation ranges — which is exactly what the data shows (tri-modal >> dual-modal >> single-modal). But the paper could have strengthened this claim by including a direct comparison of per-modality SQNR degradation (as predicted by Theorem 1) for each benchmark, rather than only showing aggregate accuracy.
A subtle interpretation issue: the paper attributes SmoothQuant's failure to smoothing misalignment, but SmoothQuant's audio collapse at W4A8 on Omni-3B also reflects that SmoothQuant applies a single smoothing factor per channel and uses a closed-form formula rather than learnable optimization. Table 3 shows that learned unified smoothing performs worse than closed-form SmoothQuant, so the failure is not just about closed-form vs. learnable — it is genuinely about unified vs. modality-specific. But the paper does not isolate whether SmoothQuant's failure could be partially mitigated by simply calibrating on audio-only data (ignoring vision during calibration) and accepting degraded vision performance. This would be an informative baseline: does a smoothing factor optimized for audio alone produce acceptable vision quality, or does the modality with larger activations strictly require its own smoothing?
Does the evidence support the claim that CMC maintains computational invariance with a single weight while recovering per-modality smoothing quality?
The evidence from Figure 6 supports the claim in terms of SQNR: CMC with whitening achieves higher SQNR than MBQ at much lower rank ratios. This is a clean demonstration that whitening induces low-rank structure and that the low-rank correction effectively bridges the gap between text-smoothed and vision-smoothed weights.
However, the gap between the MAS baseline (separate per-modality weights, no CMC) and the CMC curve in Figure 6 shows that CMC does not fully recover MAS quality — there is an SQNR gap that narrows with increasing rank but does not close entirely within the tested rank range (up to ~0.15 ratio). The paper does not report the end-to-end accuracy loss attributable to this residual SQNR gap. In Table 1, the W4A8 results show MASQuant (which includes CMC) achieving accuracy close to what one would expect from per-modality smoothing — but without a direct comparison to "MAS with separate stored weights for each modality" as an experimental condition, it is impossible to quantify how much accuracy CMC sacrifices relative to ideal multi-weight storage.
The optimality guarantee in Theorem 2 applies to the Frobenius norm reconstruction loss, not to end-to-end task accuracy. The proof shows that CMC minimizes ||X_v S_v^{-1} (ΔW - L)||_F, but minimizing this per-layer metric does not guarantee minimal task-level accuracy degradation. The correlation between per-layer reconstruction loss and downstream accuracy is empirical, not proven, and the paper does not provide a per-layer breakdown of reconstruction error vs. task impact.
Does the evidence support the claim that MASQuant is effective across both vision-language and omni-modal MLLMs?
Yes, with the important caveat that effectiveness is contingent on bit width. The claim holds strongly for W8A8 and moderately for W4A8 on tri-modal models, weakly for W4A6 on all configurations, and is untested at W4A4 accuracy (despite Table 7 showing W4A4 speedup). The paper's framing ("stable quantization performance across both dual-modal and tri-modal MLLMs" in the abstract) is accurate but glosses over the fact that at W4A6, the accuracy gap to FP16 is 3–8 points on vision tasks — a gap that may be unacceptable for many applications. A more precise statement would be: MASQuant prevents modality-specific catastrophic collapse at moderate-to-aggressive quantization but cannot overcome the fundamental precision limits of 4-bit weights and 6-bit activations.
Genuine weaknesses in the experimental design:
Single model family (Qwen2.5). All experiments use Qwen2.5-VL and Qwen2.5-Omni. These models share architectural DNA — Qwen2.5-Omni is essentially Qwen2.5-VL with an added audio encoder. The findings about activation magnitude ratios (Figure 4) and modality dominance may be specific to how Qwen2.5 projects multimodal embeddings into the LLM's input space. Models with different multimodal fusion architectures — early fusion (where modalities are mixed before the first transformer layer), cross-attention-based fusion (where vision/audio encoders interact with text via separate cross-attention layers), or adapter-based approaches — may have different activation magnitude patterns and different sensitivity to smoothing misalignment. Testing on a single non-Qwen architecture (e.g., LLaVA, InternVL, or Gemini) would substantiate the claim of generality.
Small benchmark size for some evaluations. The paper evaluates on standard benchmarks, but the test sets for some (ScienceQA, Vizwiz) are relatively small (typically a few thousand questions for ScienceQA, a few hundred for Vizwiz test-dev). Combined with the single-pass evaluation (no cross-validation or standard error reporting), differences of 1–2 percentage points between methods at the same bit width may not be statistically significant. The averaging across benchmarks (the "Avg" column in Table 1) partially mitigates this by increasing effective sample size, but it also obscures per-task variance.
No measurement of calibration cost. The paper does not report the time or computational resources required for the MAS optimization and CMC computation during calibration. For PTQ methods, calibration cost matters — if MASQuant requires hours of gradient-based optimization per model while SmoothQuant requires minutes, that differential affects deployability in settings where models are frequently updated or customized. The training epochs ablation (Table 5) suggests rapid convergence (2 epochs, with most of the gain in 1 epoch), but actual wall-clock time depends on calibration set size and hardware, which are not reported.
Missing W4A4 accuracy evaluation. Table 7 reports W4A4 speedup but the accuracy tables (Tables 1 and 2) only go down to W4A6. W4A4 is an active area of research (QuIP, QuaRot, SpinQuant for LLMs), and including W4A4 accuracy results, even if negative, would define the practical limits of the method. The omission leaves a gap between the speed/memory claims and the accuracy claims.
No comparison with rotation-based PTQ methods. The paper compares only against channel-wise smoothing methods (SmoothQuant, AWQ, MBQ), but recent LLM quantization research has also explored rotation-based methods (QuaRot, SpinQuant, QuIP) that apply orthogonal transformations to redistribute outliers more uniformly. These methods fundamentally avoid the computational invariance constraint that MASQuant repairs — they use a full matrix S (orthogonal) rather than a diagonal one, which distributes outlier magnitude across channels rather than requiring per-channel smoothing. Including a rotation-based baseline would position MASQuant relative to the full landscape of PTQ approaches.
No end-to-end throughput measurement for autoregressive generation. The prefill-stage latency in Table 7 measures only the initial processing of input tokens. For practical deployment, what matters is the tokens-per-second throughput for complete prompt-to-response generation, including both prefill and decoding. The paper claims decoding latency is identical to MBQ, but without measurement, the actual percentage improvement over FP16 for a realistic workload is unknown. If prefill is 10% of total inference time for a typical multimodal query, a 5% prefill overhead is 0.5% total overhead — negligible. But if prefill is 50% (e.g., for tasks with very long multimodal inputs and short text outputs), the overhead is proportionally larger.
Experiments that would have strengthened the paper:
- Ablation on the number of modalities: quantizing a model with 2, 3, and 4+ modalities to show that the benefit of MASQuant scales with modality count, directly validating the smoothing misalignment theory.
- Cross-architecture evaluation: testing on at least one non-Qwen model family (LLaVA, BLIP-2, or InternVL) to establish whether the smoothing misalignment pattern generalizes beyond Qwen's specific multimodal projection design.
- Per-layer CMC rank optimization: allocating higher rank to layers with larger cross-modal activation differences and lower rank to layers where modalities are more aligned, reducing total memory overhead while maintaining accuracy.
- Comparison with a "single modality calibration" baseline: applying SmoothQuant calibrated only on text data (for text tasks) and only on vision data (for vision tasks), to isolate whether the failure is due to mixed-modality calibration specifically or the single-S architecture more fundamentally.
- W4A4 accuracy results to complete the speed-vs-accuracy characterization.
- Statistical significance testing or at minimum multiple evaluation runs to establish confidence intervals for the accuracy numbers in Tables 1 and 2, given that some differences between methods are 0.1–0.5 percentage points on small test sets.
- Calibration data sensitivity analysis: varying the number of calibration samples per modality (e.g., 32, 64, 128, 256) to determine minimum requirements for MAS convergence and CMC rank discovery.
In summary, the experiments strongly support the paper's central diagnostic claim (smoothing misalignment is a real and severe phenomenon in tri-modal quantization), moderately support the repair claim (CMC provides effective reconciliation at low rank overhead), and leave open questions about generality (single model family), statistical reliability (small benchmarks, single-pass evaluation), and practical completeness (missing W4A4 accuracy, missing end-to-end throughput, missing calibration cost). The paper is most convincing where the effects are largest and least convincing where the claimed advantages over baselines are within the noise floor of benchmark evaluation.
6. Limitations and Trade-offs
6.1 Single Model Family Evaluation Limits Generality Claims
The assumption or constraint. All experiments in this paper are conducted exclusively on models from the Qwen2.5 family — Qwen2.5-VL (3B and 7B) for vision-language evaluation and Qwen2.5-Omni (3B and 7B) for tri-modal evaluation. These models share architectural lineage: Qwen2.5-Omni is essentially Qwen2.5-VL augmented with an audio encoder, and both use the same "Thinker" transformer decoder as their core language model component. The paper does not evaluate on any model from a different family (e.g., LLaVA, InternVL, Gemini, GPT-4V, BLIP-2) or with a fundamentally different multimodal fusion architecture.
The paper's abstract claims MASQuant demonstrates "stable quantization performance across both dual-modal and tri-modal MLLMs," which could be interpreted as a general claim about MLLMs as a category. However, the evaluation scope is limited to two closely related model architectures from a single organization.
The consequence. The paper's central diagnostic — that smoothing misalignment arises because visual token activations dominate text and audio activations in shared transformer layers — may be architecture-dependent. Different multimodal fusion designs could produce different activation magnitude patterns:
- Early fusion architectures (where image patches, audio frames, and text tokens are concatenated before the first transformer layer) may exhibit the pattern documented in this paper because all modalities enter the same layers at the same time.
- Cross-attention-based fusion (where vision/audio features interact with text via separate cross-attention layers, as in Flamingo-style architectures) may isolate modality-specific processing to dedicated layers, limiting the cross-modality interference that causes smoothing misalignment.
- Adapter-based approaches (where modality-specific adapters project features into the LLM's input space) might equalize activation magnitudes before they reach the shared transformer layers, reducing or eliminating the range disparities that Theorem 1 depends on.
Without evaluation on at least one model outside the Qwen2.5 family, it is unclear whether smoothing misalignment is a universal multimodal quantization problem or a Qwen-specific architectural artifact. A practitioner using LLaVA or InternVL cannot confidently apply MASQuant based on this paper's evidence alone.
What evidence exists in the paper. The paper provides extensive evidence within the Qwen2.5 family that the phenomenon is robust across model sizes (3B vs. 7B) and modality configurations (dual-modal vs. tri-modal), but never crosses architectural boundaries. Section 5.1 confirms the evaluation scope: "We evaluate on Qwen2.5-VL and Qwen2.5-Omni, multimodal LLMs supporting text, audio, and vision." The architectural description notes the specific encoder components (675M ViT for vision, Whisper-large-v3 for audio), and the paper explicitly states it quantizes only the LLM component Thinker, but never acknowledges that the Thinker's architecture might influence whether the findings transfer.
The modality dominance analysis in Figure 4, which forms the empirical foundation for the smoothing misalignment diagnosis, is computed on Qwen2.5-Omni-3B and Qwen2.5-VL-3B only. The activation range ratios that drive Theorem 1's degradation are measured exclusively within the Qwen family. No evidence is presented that other model families exhibit similar activation magnitude disparities.
Mitigation status. The paper does not acknowledge this limitation or suggest future work on cross-architecture validation. The abstract's claim of generality across "both dual-modal and tri-modal MLLMs" implicitly extends the findings beyond the evaluated models without qualification.
6.2 Missing W4A4 Accuracy Evaluation Creates a Speed-vs-Accuracy Blind Spot
The assumption or constraint. Section 5.5 (Table 7) reports end-to-end prefill-stage latency and memory measurements under W4A4 quantization on a Desktop RTX 4090, demonstrating 2.5–3.3× speedup over FP16 and 2.56–2.83× memory reduction at batch size 1. These are the most aggressive quantization numbers in the paper and represent the best-case scenario for deployment efficiency.
However, the paper's accuracy evaluation (Tables 1 and 2) stops at W4A6 for vision-language models and W4A6 for tri-modal models. No W4A4 accuracy results are reported for any benchmark, model size, or modality configuration. Table 7 shows that W4A4 is fast — but the paper provides no evidence that it produces acceptable output quality.
The consequence. This creates a critical information gap for practitioners. W4A4 is an active and competitive research frontier in LLM quantization — methods like QuIP, QuaRot, and SpinQuant target this bit width and report accuracy on standard benchmarks. The omission of W4A4 accuracy leaves open the possibility that MASQuant degrades unacceptably at this bit width, which would mean the impressive speedup numbers in Table 7 are practically unusable. Given the paper's own evidence that performance degrades substantially from W8A8 to W4A8 to W4A6 (on Qwen2.5-Omni-3B, MMMU drops: 42.3 → 36.7 → 32.2), it is plausible that W4A4 accuracy would fall below a usable threshold.
The omission also makes it impossible to compare MASQuant against the state-of-the-art W4A4 methods that the paper cites in Section 2 (QuIP, QuaRot, SpinQuant) but does not benchmark against. A deployment engineer evaluating quantization options for a multimodal model cannot assess whether MASQuant's W4A4 accuracy is competitive with rotation-based methods or whether the speed/accuracy tradeoff favors one approach over the other.
What evidence exists in the paper. The trend across bit widths in Tables 1 and 2 is the only indirect evidence. On Qwen2.5-VL-3B:
- W8A8 → W4A8: MMMU drops from 46.6 to 46.7 (noise-level change), but TextVQA drops from 77.7 to 69.2 (8.5-point decline)
- W4A8 → W4A6: MMMU drops from 46.7 to 40.0 (6.7-point decline), ScienceQA drops from 77.9 to 71.2 (6.7-point decline)
The per-step degradation is already substantial between W4A8 and W4A6. Extrapolating to W4A4 — which halves activation precision to 4 bits, only 2 bits above the theoretical minimum for uniform quantization — would likely produce steep further degradation. But "likely" is not evidence, and the paper provides no measured numbers.
Mitigation status. The paper does not acknowledge this gap or explain why W4A4 accuracy was omitted. Table 7's caption mentions "W4A4 setting" for the speed measurements, but the accuracy tables contain no corresponding column. This is a significant omission in a paper whose practical contribution centers on enabling efficient deployment.
6.3 CMC Low-Rank Approximation Introduces an Accuracy-Memory Tradeoff That Is Not Fully Characterized
The assumption or constraint. Cross-Modal Compensation (CMC) reconciles per-modality optimal smoothing with single-weight storage by approximating the weight difference ΔW between modality-specific and base weights using a low-rank decomposition. Theorem 2 proves that the rank-r approximation is optimal for minimizing the per-layer output reconstruction loss ||X_v S_v^{-1} (ΔW - L)||_F, but this mathematical guarantee operates at the level of individual linear layers, not end-to-end task performance.
The rank r is a hyperparameter that trades off correction fidelity (higher rank = better approximation of the ideal per-modality weight) against memory and compute overhead (higher rank = more parameters stored and more FLOPs during prefill). The paper evaluates three rank ratios (0.01, 0.02, 0.05) for latency in Table 7, but Figure 6 shows SQNR improvement as a function of rank ratio only up to ~0.15, and no end-to-end accuracy is reported for different rank choices.
The consequence. A practitioner deploying MASQuant faces an under-determined tradeoff: what rank ratio is sufficient for their accuracy requirements, and how does accuracy degrade as rank is reduced? The paper provides only diagnostic measurements (SQNR in Figure 6, latency in Table 7) without connecting them to downstream task quality. The SQNR improvement in Figure 6 saturates around 3.5 dB at higher rank ratios, but what does a 3.5 dB SQNR improvement translate to in terms of MMMU accuracy or LibriSpeech WER? The paper never answers this.
More subtly, the optimal rank may vary across layers — layers where cross-modal activation differences are large (as shown in Figure 2, where some layers lose >20 dB SQNR under unified smoothing) likely need higher rank compensation than layers where modalities are well-aligned. The paper applies a uniform rank ratio across all layers, which means it either overallocates rank to layers that don't need it (wasting memory) or underallocates rank to critical layers (sacrificing accuracy). Without a per-layer ablation, the extent of this inefficiency is unknown.
There is also a deeper structural concern: CMC approximates the floating-point weight difference ΔW = S_v W - Q(S_t W) using a low-rank correction. The truncated SVD in whitened space minimizes the Frobenius norm output error, but this is a linear approximation — the actual quantized inference path for vision tokens computes Q(X_v S_v^{-1}) · Q(S_t W) + X_v S_v^{-1} · L_1 L_2, where the first term uses quantized activations and the second uses floating-point activations. The interaction between quantization error in the base path and approximation error in the correction path is not analyzed. In principle, these errors could interact non-additively — the correction could partially compensate for quantization error or could amplify it in certain activation directions.
What evidence exists in the paper. Figure 6 provides the most direct evidence: SQNR improvement plateaus around 3.5 dB for CMC with whitening, with diminishing returns beyond rank ratio ~0.1. The gap between the CMC curve and the MAS (ideal) line in Figure 6 represents the residual error that no amount of additional rank can close — it is the irreducible error from using text-smoothed quantized weights as a base rather than storing separate vision-smoothed quantized weights.
Table 7 quantifies the memory overhead at different rank ratios: at rank 0.05, MASQuant uses 5.37 GB vs. MBQ's 4.85 GB (10.7% more memory at batch size 1). At the more practical rank 0.02, the overhead is 3.9%. But neither Figure 6 nor any accuracy table tells us whether rank 0.02 is sufficient to recover near-FP16 accuracy, or whether the residual SQNR gap at rank 0.02 translates to a measurable accuracy penalty.
Mitigation status. The paper provides the theoretical optimality guarantee (Theorem 2) and diagnostic evidence (Figures 5, 6) that CMC works as designed, but does not close the loop to end-to-end accuracy. A table showing downstream task accuracy as a function of rank ratio (or at minimum reporting accuracy at the three rank ratios used in Table 7) would resolve this limitation. The paper does not acknowledge this gap or propose per-layer adaptive rank allocation as future work.
6.4 Calibration Cost and Data Requirements Are Not Quantified or Accounted For
The assumption or constraint. MASQuant's two-stage calibration pipeline — Modality-Aware Smoothing (MAS) followed by Cross-Modal Compensation (CMC) — imposes computational and data requirements beyond those of simpler PTQ methods like SmoothQuant and AWQ:
-
MAS requires gradient-based optimization of per-modality smoothing factors. Each S_m is a diagonal matrix with D_in learnable parameters per linear layer. For a 7B model with ~32 transformer layers, each containing Q, K, V, O projections and two FFN layers, this amounts to thousands to tens of thousands of learnable parameters. The optimization runs for 2 epochs (as per Table 5) using MAE loss computed per layer, which requires forward passes through each layer with quantized weights and activations on modality-specific calibration data. The optimizer, learning rate, and batch size for this optimization are not specified in the paper.
-
CMC requires SVD decomposition of the activation covariance matrix (D_in × D_in) and of the whitened weight residual (D_in × D_out) for each linear layer and each non-text modality. For a FFN up-projection layer with D_in ≈ 4096 and D_out ≈ 11008, the covariance SVD is on a 4096 × 4096 matrix and the residual SVD is on a 4096 × 11008 matrix. Across ~32 layers with ~5 linear projections each, this is ~160 SVD operations per non-text modality — computationally non-trivial, especially for tri-modal models where both vision and audio require CMC.
-
Calibration data must be split by modality. The paper notes that MAS optimizes each S_m on modality-specific data, and CMC computes the whitening transform from vision-specific calibration activations. This requires curated per-modality calibration sets — not just a collection of multimodal samples, but samples where each modality's contribution can be isolated. For tri-modal models, this means assembling separate text-only, vision-only, and audio-only calibration sets, which may be more difficult to curate than the mixed-modality calibration data used by SmoothQuant or MBQ.
The paper never reports the wall-clock time, GPU memory consumption, or computational FLOPs required for this calibration process.
The consequence. For many deployment scenarios, calibration cost matters. If a practitioner needs to quantize a custom-fine-tuned model, or if models are frequently updated (e.g., in a continuous deployment pipeline), a calibration procedure that takes hours instead of minutes may be impractical. The paper's claim that MASQuant "achieves stable quantization performance" (abstract) implies deployability, but without calibration cost data, a practitioner cannot assess whether the method fits their operational constraints.
More subtly, the calibration data requirements may pose a barrier for models where per-modality data is scarce or where the training data distribution is not easily decomposable by modality. The paper uses Qwen2.5-family models, which have publicly available multimodal training data that the authors (as employees of Alibaba, the model's creator) presumably have access to. A practitioner quantizing a third-party model or a model trained on proprietary multimodal data may not have access to cleanly separated per-modality calibration sets of sufficient size.
Finally, the rapid convergence observed in Table 5 (most of the gain in 1 epoch, peak at 2 epochs) suggests the optimization is not particularly expensive per epoch — but without knowing epoch duration, dataset size, or hardware used, this is speculative. A convergence plot showing reconstruction loss vs. wall-clock time would clarify whether the 2-epoch training is a few minutes or several hours.
What evidence exists in the paper. Table 5 shows that smoothing factor optimization converges rapidly: PPL drops from 23.9 (epoch 0, initialization) to 17.0 (epoch 1) to the peak at epoch 2 (accuracy 56.9%). This suggests that the initialization from Equation 12 (the per-modality SmoothQuant formula) is already close to optimal, and the gradient-based refinement provides incremental gains. But the table provides no units for "epoch" — we do not know whether one epoch processes 128 samples or 12,800, whether each sample is a single multimodal input or a batch of tokens, or how long one epoch takes on what hardware.
The paper mentions in Section 5.1 that calibration follows standard PTQ practice, but standard PTQ practice varies widely: SmoothQuant calibration can be done in minutes on a single GPU with a few hundred samples, while OmniQuant's learnable smoothing requires gradient-based optimization that takes longer. Where MASQuant falls on this spectrum is unspecified.
Mitigation status. The paper does not acknowledge calibration cost as a limitation. It does not report calibration time, memory requirements, or data requirements. It does not suggest future work on making the calibration process more efficient (e.g., by sharing whitening transforms across layers, using stochastic SVD approximations, or reducing the number of calibration samples needed). This is a notable omission for a paper that positions itself as a practical deployment solution.
6.5 The Method Does Not Address the Hardest Quantization Regime — Extreme Low-Bit Activation Quantization Below 6 Bits
The assumption or constraint. MASQuant targets the problem of smoothing misalignment — the failure mode where a single channel-wise smoothing factor serves multiple modalities with different activation magnitudes. This diagnosis is supported by Theorem 1 and validated empirically by the catastrophic audio collapse under SmoothQuant at W4A8 (Table 2: LibriSpeech WER 77.4 vs. MASQuant 3.6). The method repairs this specific failure mode by learning modality-specific smoothing factors and reconciling them via CMC.
However, smoothing misalignment is not the only source of quantization error. When activation precision drops below 6 bits, the quantization step size itself becomes a dominant error source regardless of how well the smoothing factors are calibrated. The uniform affine quantizer defined in Equation 1 maps continuous values to a discrete set of 2^N levels. At 4-bit activation quantization, there are only 16 possible values per activation element — the representational capacity is inherently limited regardless of how the input distribution is scaled. No amount of smoothing can recover information that has been lost to the quantization grid's coarseness.
The consequence. The paper's own results show that MASQuant's advantage over baselines narrows as quantization becomes more aggressive, and the absolute gap to FP16 widens for all methods. Comparing across bit widths on Qwen2.5-Omni-3B:
- W8A8 (MASQuant): MMMU 42.3, LibriSpeech WER 3.8 — near FP16 quality
- W4A8 (MASQuant): MMMU 36.7, LibriSpeech WER 3.6 — vision degrades notably, audio preserved
- W4A6 (MASQuant): MMMU 32.2, LibriSpeech WER 3.7 — vision degrades further, audio preserved
The gap between MASQuant and FP16 on MMMU grows: 43.3 (FP16) → 42.3 (W8A8, -1.0) → 36.7 (W4A8, -6.6) → 32.2 (W4A6, -11.1). At W4A6, the 11-point drop on MMMU is substantial — a model losing a quarter of its accuracy on a challenging reasoning benchmark. For production systems where MMMU-style reasoning is the primary use case, this accuracy may be unacceptable regardless of the memory savings.
The paper does not demonstrate or claim a solution to this fundamental precision ceiling. The method repairs one quantization pathology (cross-modal smoothing interference) but does not change the information-theoretic limits of low-precision representation. A practitioner seeking to deploy at W4A4 or below — which Table 7 shows is where the most dramatic speedups occur — cannot rely on MASQuant to maintain acceptable accuracy, because the method's mechanism of action (better smoothing factor allocation) does not address the primary error source at those bit widths (insufficient quantization levels).
What evidence exists in the paper. The convergence of methods at W4A6 is visible in Table 1: on Qwen2.5-VL-3B, MASQuant (59.9%), MBQ (59.7%), and SmoothQuant (53.6%) are separated by only 6.3 points, compared to the 7.2-point gap between MBQ and SmoothQuant at W4A8. On the 7B model at W4A6, the gap narrows further: MASQuant 63.0% vs. MBQ 62.9% vs. SmoothQuant 61.0% — all three methods within 2 points. This convergence is direct evidence that smoothing strategy matters less as activation precision drops, because the quantization error floor rises for all methods.
The paper does not include W4A4 accuracy (as discussed in limitation 6.2), so the behavior at the extreme low end is entirely uncharacterized. The absence is itself evidence of the limitation — if W4A4 accuracy were acceptable, it would strengthen the paper's deployment claims and would likely have been reported.
Mitigation status. The paper does not explicitly acknowledge that its method is bounded by the precision limits of low-bit quantization, but the results implicitly communicate this boundary. Section 5.4's analysis of CMC (Figure 6) shows that SQNR improvement saturates around 3.5 dB regardless of rank — a clear signal that other error sources dominate at that point. The paper does not propose extensions to address the low-bit precision ceiling (e.g., non-uniform quantization, vector quantization, or mixed-precision strategies that allocate more bits to activation quantization while keeping weight quantization aggressive).
6.6 The Method Assumes Stationary Activation Covariance During Inference — Deployment Distribution Shift Is Not Tested
The assumption or constraint. Cross-Modal Compensation relies on the whitening transform T computed from the covariance of smoothed calibration activations, as defined in Equation 23:
This transform is computed once during calibration and frozen for inference. Theorem 2's optimality guarantee — that CMC minimizes the output reconstruction error — holds only when the inference-time activation covariance matches the calibration-time covariance. If the input distribution shifts between calibration and deployment, the whitening transform may no longer optimally condition the weight residual for low-rank approximation. Specifically, if new activation patterns emerge during deployment that were absent or rare in calibration, the SVD truncation in whitened space may discard components of ΔW that are important for the new patterns while preserving components that are only relevant for calibration-set patterns.
The consequence. Multimodal models deployed in production often encounter inputs that differ from their training distribution: different image resolutions, different audio sample rates or recording conditions, different text domains, or novel combinations of modalities. For MASQuant, the whitening transform is calibrated on a specific dataset (presumably drawn from the model's training distribution). If a deployment scenario presents systematically different inputs — e.g., high-resolution medical images when calibrated on natural images, or noisy field audio when calibrated on clean speech — the CMC correction may be suboptimal, leading to larger errors for specific input types.
This concern is particularly acute because CMC is multiplicative: the low-rank correction L_1 L_2 is applied to the smoothed activations X_v S_v^{-1}. If the activation distribution shifts, both the base quantized computation (which uses quantization parameters calibrated on the original distribution) and the CMC correction (which uses a whitening transform matched to the original distribution) may degrade simultaneously. The degradation could be worse than for a method like MBQ that uses a single smoothing factor — MBQ would degrade gracefully as calibration-deployment mismatch increases, while MASQuant could degrade sharply if the mismatch affects directions where the low-rank approximation allocated its limited rank budget.
What evidence exists in the paper. The paper provides no evidence on robustness to distribution shift. All evaluations use standard benchmark test sets (MMMU, TextVQA, LibriSpeech, etc.) that are drawn from distributions similar to the training data of the Qwen2.5 models. There is no out-of-distribution evaluation, no ablation on calibration data diversity, and no measurement of how reconstruction error changes when the whitening transform is computed from one dataset and applied to another. The paper's validation of the whitening transform (Figure 5, effective rank reduction) is performed on the calibration data itself, which is circular — it shows that the transform works well on the data it was computed from, but not that it generalizes.
The paper also does not discuss the stability of the activation covariance across different input types within the same modality. Vision activations for natural photographs, document scans, charts, and screenshots may have different covariance structures. The current calibration likely uses a general-purpose mixture, but the degree to which per-input-type covariance variation affects CMC quality is unknown.
Mitigation status. The paper does not acknowledge distribution shift as a limitation or suggest robustness evaluation as future work. This is a significant practical concern: PTQ methods are often applied to models that will be deployed in environments that differ from the calibration setting. A method that depends on precise covariance structure (via the whitening transform) is inherently more vulnerable to distribution shift than a method that only depends on aggregate statistics like per-channel maxima (SmoothQuant) or averages (AWQ). The paper would be strengthened by even a simple robustness test — e.g., calibrating on one dataset split and evaluating CMC reconstruction error on a held-out split from a different domain.
7. Implications and Future Directions
How This Work Changes the Landscape
This paper reframes multimodal LLM quantization from a problem of error balancing to a problem of structural incompatibility. Prior to MASQuant, the working assumption was that channel-wise smoothing — the dominant paradigm in LLM quantization — transfers to multimodal models with appropriate calibration adjustments. MBQ's modality-weighted error balancing represented the state of this assumption: acknowledge that visual and text tokens contribute unequally to quantization error, then reweight their importance during hyperparameter search. This treats multimodality as a degree-of-difficulty increase — more diverse distributions require more careful tuning.
MASQuant demonstrates that the assumption is wrong at a more fundamental level. The computational invariance principle $\mathbf{Y} = (\mathbf{X}\mathbf{S}^{-1})(\mathbf{S}\mathbf{W})$ guarantees output preservation for any diagonal $\mathbf{S}$, but the proof says nothing about which $\mathbf{S}$ is quantization-friendly when different subsets of $\mathbf{X}$ carry different distributions. Theorem 1 formalizes what this gap means: when the per-channel range ratios $\boldsymbol{\alpha}_i$ across modalities are non-uniform, a single $\mathbf{S}$ is mathematically guaranteed to degrade SQNR for non-dominant modalities. The degradation is not an implementation bug, a calibration quality issue, or a hyperparameter sensitivity — it is a structural consequence of applying a single transformation to heterogeneous inputs.
This reframing changes the landscape in three specific ways:
First, it establishes that per-modality smoothing is not an optimization but a requirement for aggressive multimodal quantization. The paper's most dramatic result — SmoothQuant producing 77.4 WER on LibriSpeech at W4A8 while MASQuant maintains 3.6 WER (Table 2) — is not just a large effect size but a qualitative failure. The audio modality stops functioning entirely. This is the kind of result that moves a design choice from "nice to have" to "must have": if you are quantizing a tri-modal model to 4-bit weights and 8-bit activations, you cannot use a unified smoothing factor and expect all modalities to work. The paper's demonstration that learned unified smoothing performs worse than closed-form SmoothQuant (Table 3) is particularly important here — it proves that the problem cannot be optimized away within the single-$\mathbf{S}$ paradigm. Gradient descent on mixed-modality data converges to a compromise that is worse for weak modalities than the heuristic formula, because the loss landscape itself encodes the zero-sum tradeoff.
Second, it decomposes the computational invariance constraint into two independent subproblems — smoothing flexibility and storage efficiency — and shows they can be solved separately. Before this work, the computational invariance constraint was monolithic: you compute one $\mathbf{S}$, and the quantization benefits follow. The idea that you could compute multiple optimal $\mathbf{S}_m$ per modality and then retroactively reconcile them into a shared weight representation was absent from the literature. This decomposition is conceptually significant beyond the specific method. It implies that future multimodal quantization methods should not try to find a single transformation that works for all modalities — that is provably suboptimal when activation range ratios are non-uniform — but should instead develop better reconciliation mechanisms that bridge independently optimized per-modality solutions into shared representations. The paper's SVD-based whitening is one such mechanism; alternatives could include adapter-style learned corrections, dynamic weight selection via routing, or modality-conditioned weight decomposition.
Third, it redirects research attention from search-based hyperparameter tuning (the $\beta$ sweep in SmoothQuant and AWQ) to learnable, modality-conditioned transformations. The paper's observation that existing methods optimize only $\beta$ while "the smoothing factors themselves are never treated as free parameters" (Section 4.1) identifies a structural limitation in the dominant approach. Searching over $\beta$ imposes a power-law relationship between activation magnitude and smoothing strength: $s_i \propto (\text{statistic}_i)^\beta$. This forces all channels to follow the same functional form, which is too restrictive for multimodal settings where different channels may need qualitatively different smoothing patterns. The learnable optimization in MAS — directly treating each $s_i$ as a free parameter — breaks this constraint and achieves better results. This suggests that the $\beta$-search paradigm, while convenient and computationally cheap, has hit a ceiling for multimodal models, and future work should adopt more flexible parameterizations.
The paper also partially reconciles a tension in the multimodal quantization literature. MQuant identified that visual activations are ~20× larger than text activations and proposed per-modality quantization ranges, which is a form of per-tensor scaling. MBQ identified unequal modality contributions to error and proposed gradient-weighted error balancing. These observations were both correct but addressed different symptoms of the same underlying problem: a single $\mathbf{S}$ cannot serve multiple activation distributions. MASQuant resolves this by showing that the smoothing factors (not just the quantization ranges) must be modality-specific, and that the error imbalance MBQ observed is a consequence of smoothing misalignment, not an independent phenomenon. Treating smoothing misalignment as the root cause eliminates the need for separate error-balancing heuristics.
Follow-Up Research This Work Enables
Cross-architecture validation: Does smoothing misalignment generalize beyond Qwen's fusion design? The paper's entire empirical case rests on Qwen2.5-VL and Qwen2.5-Omni, which share architectural lineage and use a specific multimodal fusion strategy — projecting vision and audio tokens into the LLM's input embedding space before the first transformer layer. Models with different fusion architectures (cross-attention-based, adapter-based, or early-fusion with separate modality encoders) may exhibit different activation magnitude patterns and different sensitivity to smoothing misalignment. A strong follow-up would replicate the core diagnostic (Figure 4: modality dominance analysis, Figure 2: per-layer SQNR degradation under SmoothQuant) on LLaVA (which uses a simpler linear projection for vision features), InternVL (which uses a larger vision encoder with dynamic resolution), and a Flamingo-style architecture (which uses gated cross-attention). The specific hypothesis to test: does the percentage of channels where one modality dominates the smoothing factor (as in Figures 4a-b) correlate with the architecture's degree of modality mixing in shared layers? If cross-attention architectures isolate modality-specific processing to dedicated layers, smoothing misalignment may be absent, and MASQuant's benefit would be limited to certain architectural families. This would refine the paper's generality claim and provide a decision rule for practitioners: "use MASQuant if your fusion architecture does X; use standard SmoothQuant if it does Y."
Adaptive per-layer CMC rank allocation with accuracy-guided truncation. The paper applies a uniform rank ratio across all linear layers, but Figures 2 and 5 show substantial variation in cross-modal activation differences and effective rank across layers. Some layers exhibit >20 dB SQNR degradation under unified smoothing (Figure 2), indicating large cross-modal disparities that likely require higher CMC rank, while others show minimal degradation where lower rank may suffice. A strong follow-up would develop a per-layer rank allocation strategy: compute the per-layer output reconstruction error $\|\mathbf{X}_v \mathbf{S}_v^{-1} (\boldsymbol{\Delta}\mathbf{W} - \mathbf{L}^{(r)})\|_F$ for a range of ranks, then solve a constrained optimization problem that allocates a total rank budget across layers to minimize aggregate reconstruction error. The evaluation would compare: (a) uniform rank allocation, (b) error-minimizing per-layer allocation, (c) a practical heuristic (allocate rank proportional to the pre-whitening effective rank of $\boldsymbol{\Delta}\mathbf{W}$). The metric would be end-to-end accuracy at fixed total memory overhead, testing whether adaptive allocation closes the gap between CMC (single stored weight, low-rank correction) and MAS (ideal per-modality weights, no compensation) more efficiently than uniform allocation. The paper's Figure 6 already establishes the SQNR-vs-rank curve for a representative layer; extending this to per-layer curves on Qwen2.5-Omni-3B and evaluating downstream accuracy is a tractable 1-2 week experiment.
Modality-specific activation bit-width allocation. The paper's results reveal a clear modality vulnerability hierarchy: at W4A8 on Qwen2.5-Omni-3B, SmoothQuant destroys audio (WER 77.4) while vision degrades more gracefully (MMMU 30.0 from 43.3). This suggests that activation quantization precision should be modality-specific, not just weight-specific. A direct follow-up would implement asymmetric activation quantization: keep audio activations at 8 bits while reducing vision and text activations to 6 or 4 bits, maintaining 4-bit weights throughout. The hypothesis is that audio's smaller activation magnitudes make it more vulnerable to quantization error (each bit lost discards a larger fraction of the signal), so allocating more activation bits to audio and fewer to vision should improve aggregate quality at fixed total memory. The experiment would sweep activation bit-width assignments across modalities on Qwen2.5-Omni-3B, measuring LibriSpeech WER and MMMU accuracy as functions of per-modality activation precision. The paper already provides the diagnostic infrastructure for this: Figure 4a shows that visual tokens dominate smoothing factors, confirming they have the largest dynamic range and can tolerate coarser quantization, while audio tokens have the smallest range and need finer quantization. This asymmetry-specific allocation is a natural extension of MASQuant's modality-aware philosophy to the quantization parameters themselves.
Robustness to calibration-deployment distribution shift in the whitening transform. CMC's optimality guarantee (Theorem 2) depends on the whitening transform $\mathbf{T}$ matching the inference-time activation covariance. If deployment inputs differ from calibration inputs — different image resolutions, different audio recording conditions, or novel domains — the transform may become suboptimal. A stress-test follow-up would calibrate MASQuant on standard multimodal data (e.g., natural images from COCO, clean speech from LibriSpeech) and evaluate on systematically shifted distributions: low-resolution or heavily compressed images, noisy or reverberant speech, document images instead of natural scenes, code-switched or accented speech. The measurement would compare: (a) CMC reconstruction error $\|\mathbf{X}_v \mathbf{S}_v^{-1} (\boldsymbol{\Delta}\mathbf{W} - \mathbf{L})\|_F$ when the whitening transform is computed on in-distribution vs. shifted data, (b) end-to-end task accuracy degradation as a function of distribution shift magnitude. A negative result — CMC degrades sharply under even mild distribution shift — would motivate developing robust whitening transforms (e.g., regularized covariance estimation, online adaptation during deployment, or whitening transforms shared across layers to reduce overfitting to calibration data). A positive result — CMC is relatively robust because the low-rank structure of cross-modal differences is an architectural property, not a data-specific one — would strengthen the method's deployment case. The paper's complete silence on this question makes it a high-value, low-risk experiment.
Combining MASQuant with rotation-based quantization for extreme low-bit regimes. The paper acknowledges that all methods converge toward a performance floor at W4A6 and below, where limited quantization precision dominates over smoothing misalignment. Rotation-based quantization methods (QuaRot, SpinQuant, QuIP) address a different aspect of the quantization problem: they use orthogonal transformations to redistribute outlier magnitudes across channels, reducing the max-to-mean ratio that forces large quantization step sizes. MASQuant addresses cross-modal smoothing misalignment but does not change the per-channel outlier structure within a single modality. A combination would apply rotation-based transformations within each modality to reduce per-modality outliers, then apply MASQuant's modality-specific smoothing to handle cross-modality disparities, and finally use CMC to reconcile the per-modality smoothed weights. The experiment would benchmark this combined approach against each method alone on Qwen2.5-Omni-3B at W4A4 and W4A6, measuring both accuracy and latency. The hypothesis is that rotations and modality-specific smoothing are complementary — rotations handle within-modality outliers, MAS handles cross-modality disparities — and their combination could push the viable quantization frontier from W4A8/W4A6 to W4A4. The paper's existing CUDA kernel infrastructure (Section 5.5) can accommodate fused rotation operations, making this a feasible extension.
Difficulty-aware calibration: do harder multimodal tasks benefit more from modality-specific smoothing? The paper's evaluation averages accuracy across diverse benchmarks, but the benchmarks vary substantially in difficulty and modality reliance. MMMU requires deep cross-modal reasoning (text understanding integrated with image, graph, and diagram interpretation); TextVQA requires fine-grained visual perception (reading small text in images); OmniBench requires joint text-audio-visual reasoning. A finer-grained analysis would test whether MASQuant's benefit over SmoothQuant correlates with task difficulty or modality integration depth. The hypothesis is that easy questions (where the model is confident and quantization errors are recoverable) show small gaps between methods, while hard questions (where precise cross-modal alignment is critical) show large gaps — MASQuant preserves the alignment, SmoothQuant destroys it. This would be measured by binning MMMU and OmniBench questions by FP16 model accuracy (as a proxy for difficulty) and computing the per-bin accuracy gap between MASQuant and SmoothQuant at W4A8. If the gap widens with difficulty, it suggests that modality-specific smoothing is most important for the most challenging multimodal reasoning — exactly the scenarios where accuracy matters most. This analysis requires no new experiments, only re-binning the existing evaluation data, and would strengthen the paper's practical argument by identifying when MASQuant matters rather than just that it matters.
Practical Applications and Downstream Use Cases
Tri-modal assistant deployment on edge devices. The clearest deployment scenario enabled by this work is running a full-featured tri-modal assistant (vision + audio + text) on a device with limited memory — a smartphone, AR/VR headset, or in-vehicle system. The Qwen2.5-Omni-7B model at FP16 requires ~14 GB of memory for the LLM component alone (extrapolating from Table 7's 13.73 GB for the 7B vision-language model at batch size 1), which exceeds the available RAM on most phones. At W4A8 with MASQuant, memory drops to approximately 5 GB (extrapolating from the W4A4 numbers in Table 7: 4.85 GB for MBQ, ~5 GB for MASQuant at rank 0.02), fitting within the memory budget of a flagship phone while preserving near-FP16 audio quality (LibriSpeech WER 2.9 vs. 2.9 for FP16 on 7B, Table 2) and vision quality (MMMU 48.8 vs. 50.0). Without MASQuant, the same memory budget forces a choice: either use SmoothQuant and accept non-functional audio (WER 8.6 for the 7B model — better than the 3B's 77.4 but still 3× worse than FP16), or use MBQ and accept degraded MMMU performance (47.8 vs. 48.8). MASQuant eliminates this tradeoff, enabling a tri-modal assistant that can see, hear, and speak at near-FP16 quality within a smartphone memory envelope. The 2.5× speedup reported in Table 7 (for W4A4; W4A8 would be lower but still substantial) translates to interactive-latency responses, making real-time voice-and-vision interaction feasible.
Cost-efficient batch multimodal inference in cloud pipelines. For organizations running large-scale batch multimodal inference — evaluating thousands of images with associated text queries, transcribing and analyzing large audio corpora, or processing multimodal documents — MASQuant's memory reduction directly translates to cost savings. On a cloud GPU instance, halving memory usage (from ~14 GB to ~5 GB per model instance, as implied by Table 7) doubles the number of model instances that can run concurrently on the same hardware. If a batch inference pipeline processes 100,000 multimodal samples using Qwen2.5-Omni-7B, the FP16 deployment might require 4× A100 GPUs (80 GB each) to hold 20 concurrent instances. At W4A8 with MASQuant, the same workload fits on 2× A100 GPUs — halving the GPU-hour cost — while maintaining near-FP16 accuracy on all modalities. The key economic insight is that for tri-modal workloads, the alternative quantization methods (SmoothQuant, MBQ) force a per-modality accuracy tradeoff: you can have good vision OR good audio, but not both without MASQuant. For applications where both modalities are essential (medical diagnosis from images with voice notes, accessibility tools processing visual scenes and spoken questions simultaneously, multimodal document understanding), MASQuant is the only quantization method that preserves full functionality.
On-device accessibility tools combining vision and speech. A specific application that directly benefits from MASQuant's ability to preserve audio quality under aggressive quantization is real-time accessibility tools — applications that describe visual scenes to blind users (camera → image captioning → text-to-speech) or transcribe and summarize meetings for deaf users (microphone → speech recognition → text display). These tools require both vision and audio processing, often on battery-constrained devices where model size and inference efficiency are critical. At W4A8 on Qwen2.5-Omni-3B, MASQuant maintains LibriSpeech WER at 3.6 (near FP16's 3.9) and OmniBench accuracy at 41.4 (near FP16's 43.8), while SmoothQuant destroys audio (WER 77.4) and MBQ degrades vision (OmniBench 36.7). This means MASQuant enables a single quantized model that can both accurately transcribe speech and understand visual context — the exact capability needed for an assistive tool that must hear a user's question, see the environment, and produce a spoken response. Without MASQuant, the developer faces an unacceptable choice: use a quantization method that makes the tool deaf (SmoothQuant) or blind (MBQ at certain settings), or deploy a larger unquantized model that drains the battery and runs too slowly for real-time interaction.
Self-improving multimodal data generation pipelines with quantized models. For research groups and companies building self-improvement loops — using multimodal models to generate training data for fine-tuning themselves or smaller models — MASQuant's ability to preserve accuracy under quantization enables a practical pipeline: quantize a large multimodal model to W4A8 or W4A6, deploy multiple instances for parallel data generation, and use the generated outputs to fine-tune a smaller model. The quantized instances provide higher throughput (2.5–3.3× speedup from Table 7) at lower cost, while MASQuant ensures that the generated data quality does not degrade unevenly across modalities. If SmoothQuant were used instead, the generated audio transcriptions would be substantially degraded (Table 2 shows 3-20× WER increases), poisoning the fine-tuning data with systematic errors in one modality while maintaining quality in others. The downstream model would learn these modality-specific error patterns, requiring additional de-biasing. MASQuant prevents this by maintaining balanced quality across all modalities in the quantized generation models.
When to Prefer This Method
The paper articulates a clear decision boundary based on two factors: the number of modalities and the target quantization aggressiveness. Based on the evidence in Tables 1, 2, and the ablation studies:
-
Prefer MASQuant when: deploying tri-modal models (vision + audio + text) at W4A8 or below. The smoothing misalignment between vision (largest activations), text (intermediate), and audio (smallest) is severe enough at these bit widths that unified smoothing causes catastrophic failure in the weakest modality (Table 2: SmoothQuant WER 77.4 vs. MASQuant 3.6 on LibriSpeech at W4A8). MBQ partially recovers audio but at a measurable cost to vision quality (MMMU drops from 30.0 under SmoothQuant to 27.8 under MBQ at W4A8 on 3B Omni), while MASQuant preserves both modalities near-FP16 quality. The additional memory overhead from CMC low-rank matrices (~4% at practical rank ratios, from Table 7) is negligible compared to the accuracy loss from alternatives.
-
Prefer MASQuant when: the per-channel activation range ratios across modalities are highly non-uniform — a condition the paper shows holds for Qwen2.5-VL and Qwen2.5-Omni (Figures 4c-d) and that Theorem 1 proves is the necessary condition for smoothing misalignment to cause SQNR degradation. If a new multimodal model is being quantized without prior characterization, measuring the α_i distribution (as in Figure 4c-d) is a diagnostic step: if α_i is relatively uniform across channels, unified smoothing may suffice; if highly non-uniform, MASQuant is warranted. The paper's learnable smoothing factor optimization can also serve as an empirical test — if learned unified smoothing produces substantially worse results than learned modality-specific smoothing (as in Table 3 for audio), smoothing misalignment is active and MASQuant is indicated.
-
Prefer SmoothQuant or MBQ when: deploying vision-language only (dual-modal) models at W8A8 or W4A16. The performance gap between MASQuant and these baselines is within 0.5–1.0 percentage points on average accuracy (Table 1: MASQuant 74.4 vs. MBQ 74.4 vs. SmoothQuant 73.6 on Qwen2.5-VL-7B at W8A8), which is within the noise floor of typical benchmark variance. The simpler methods require no gradient-based smoothing factor optimization, no SVD computations during calibration, and no low-rank matrices during inference. The complexity cost of MASQuant is not justified when the accuracy benefit is below measurement reliability.
-
Prefer rotation-based methods (QuaRot, SpinQuant) or mixed-precision strategies when: targeting W4A4 or below where activation quantization precision becomes the dominant error source regardless of smoothing strategy. MASQuant's advantage over baselines narrows at W4A6 (63.0 vs. 62.9 vs. 61.0 on 7B vision-language, Table 1) and is unmeasured at W4A4. Rotation-based methods redistribute outliers in ways that complement (rather than compete with) modality-specific smoothing, but in isolation at extreme low-bit settings, rotations may provide larger gains than smoothing improvements. Mixed-precision strategies that allocate more activation bits to vulnerable modalities (audio) while aggressively quantizing robust modalities (vision) are also preferable when the total bit budget is highly constrained — MASQuant does not modify the activation bit width, only the smoothing factors.
The key practical differentiator is the modality vulnerability hierarchy: audio < text < vision in terms of activation magnitude and thus quantization vulnerability. When audio is present and accuracy matters, unified smoothing is unacceptable at aggressive quantization levels, and MASQuant is the only PTQ method that preserves audio quality without sacrificing vision accuracy. When only vision and text are present, the choice between MASQuant and MBQ depends on the application's tolerance for 1-2 point accuracy tradeoffs and the available calibration infrastructure for SVD computation and gradient-based optimization.
