QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models

🎯 Pitch

Trillion-parameter MoE models can be compressed to under 1 bit per parameter—a 20x reduction—and run on a single commodity GPU server with negligible accuracy loss. This shatters the assumption that massive sparse models require massive hardware, as QMoE exploits the 89% natural sparsity from ternary quantization to fit 1.6 trillion parameters into just 160GB.

1. Executive Summary

This paper introduces QMoE, a compression and execution framework that accurately compresses trillion-parameter Mixture-of-Experts models to less than 1 bit per parameter via a custom compression format co-designed with bespoke GPU decoding kernels, enabling efficient end-to-end compressed inference on commodity hardware. The system targets the 1.6 trillion parameter SwitchTransformer-c2048 model, applying a scalable data-dependent quantization algorithm (GPTQ adapted for MoEs) followed by a dictionary-based entropy encoding scheme that exploits the high natural sparsity (~89% zeros) emerging from ternary quantization. QMoE achieves a 20× compression rate — reducing the model from 3.2TB (bfloat16) to less than 160GB (0.8 bits per parameter) — while incurring only a 6.7% relative increase in pretraining validation loss and under 5% runtime overhead relative to idealized uncompressed execution, enabling for the first time the execution of a trillion-parameter model on a single server with 4× NVIDIA A6000 or 8× NVIDIA 3090 GPUs. The work establishes that massive MoEs are significantly more compressible than equivalently-sized dense models, though the approach is confined to models where the base quantizer already produces correct solutions at a non-trivial rate and relies on a static dictionary optimized for a specific zero-probability distribution.

2. Context and Motivation

The Core Problem: Massive MoE Models Are Too Large to Deploy

Mixture-of-Experts architectures represent a fundamental tension in modern deep learning: they exploit conditional computation to decouple model capacity from inference cost, but in doing so they create an entirely new problem — the number of stored parameters becomes enormous, even though only a fraction are active at any given moment. A standard Transformer routes every token through every parameter; an MoE routes each token through only a small subset of experts, keeping the per-token FLOP budget manageable while scaling the total parameter count into the trillions. The SwitchTransformer-c2048 model, the paper's primary case study, has 1.6 trillion parameters. Stored in standard bfloat16 precision, this consumes 3.2TB of accelerator memory. That is not a figure that fits on a single GPU, a single server, or even a modest cluster — it requires a hundred or more high-end accelerators just to hold the weights, before any computation happens.

This is the specific, concrete gap the paper addresses: there exists no practical way to run trillion-parameter MoE models on anything other than large-scale, expensive, accelerator-rich infrastructure. The consequence is that these models — which the paper notes "significantly outperform standard dense T5 models in terms of inference and training costs, at equivalent model accuracy" — remain effectively inaccessible to most researchers and practitioners. The paper states this bluntly in Section 1: "This not only makes practical deployment costly and challenging, but also strongly limits research on such models."

Why This Problem Matters: The Economics of Model Scale

The importance of this problem manifests along three dimensions, all of which the paper touches on either explicitly or implicitly.

First, the deployment cost barrier is absolute, not marginal. A model requiring 3.2TB of memory cannot be incrementally scaled down by shaving off a few GPUs. As the paper notes in Section 5.3, running the uncompressed c2048 model in bfloat16 would require more than 65 NVIDIA A6000 GPUs (48GB each) or more than 130 NVIDIA RTX 3090 GPUs (24GB each). These are not numbers that a university lab, a startup, or even a well-funded research group can typically access. The gap between what is theoretically possible (the model exists and is publicly available) and what is practically feasible (running it) is a chasm. Bridging it would democratize access to the largest available language models.

Second, MoEs represent a Pareto-optimal point in the accuracy-vs-inference-cost tradeoff that is currently inaccessible for memory reasons. The paper notes that SwitchTransformers use between 128 and 2048 experts to "significantly outperform standard dense T5 models in terms of inference and training costs, at equivalent model accuracy." This means that for a fixed inference latency budget, MoEs produce higher-quality outputs than dense models. The memory problem breaks this logic: you get the computational benefits of sparsity during inference, but you pay the storage cost of density. The model sits idle in memory, all 1.6 trillion parameters of it, even though any individual token touches only a handful of experts. This is fundamentally wasteful, and solving it would unlock the full promise of conditional computation.

Third, the largest and best-performing MoEs are also the most compressed in this paper's sense. The finding in Table 4 — that natural sparsity after ternary quantization increases with model size, from 85.7% for base128 to 88.6% for c2048 — means that the very models that are most desperately in need of compression are also the ones that compress best. This creates a virtuous cycle where the QMoE approach becomes more effective precisely where it is most needed.

Where Prior Compression Approaches Fall Short

The paper identifies a specific set of limitations in existing model compression techniques when applied to trillion-parameter MoEs at the target compression rates (Section 1, "Challenges"). These are not generic complaints about compression being hard — they are precise, technically grounded barriers.

Challenge 1: Post-training compression methods cannot reach sub-1-bit precision at acceptable accuracy on their own. The paper acknowledges that existing post-training quantization methods like GPTQ (Frantar et al., 2022), ZeroQuant (Yao et al., 2022; Wu et al., 2023), and LLM.int8() (Dettmers et al., 2022) work well for reducing precision to 3 or 4 bits per parameter on large dense models. But making trillion-parameter MoEs practical — the paper's stated goal — requires compression rates between 10× and 20× relative to 16-bit, which translates to less than 1 bit per parameter on average. No existing post-training method reaches this regime without catastrophic accuracy loss. The gap between what existing quantizers can do (3–4 bits) and what is needed (<1 bit) is roughly a factor of 3–4× in bitwidth, which is enormous in compression terms.

The paper's own experiments in Table 5 demonstrate this clearly. Round-to-nearest (RTN) quantization — the simplest baseline — produces validation losses of 2.15 for ternary c2048 versus 1.18 for the bfloat16 baseline, a relative degradation of roughly 82%. This is not usable. Even the more sophisticated GPTQ-based approach closes most of this gap (achieving 1.26 loss, or 6.7% relative degradation), but critically, GPTQ alone cannot deliver sub-1-bit storage — it produces ternary weights that still require a naive storage format consuming more than 1 bit per parameter once metadata is accounted for. The paper states this explicitly: "we find that compression rates can be pushed significantly further by taking advantage of the low entropy in the quantized weights" (Section 4). In other words, quantization is necessary but not sufficient — the sub-1-bit target requires a second stage of compression that exploits the statistical structure of the quantized weights.

Challenge 2: Scaling compression algorithms to trillion-parameter models breaks existing implementations along multiple axes. This is discussed in detail in Section 3.1, and the challenges are worth enumerating because they explain why nobody had done this before despite the obvious motivation:

Memory costs for calibration data balloon. Data-dependent quantization methods like GPTQ require passing calibration data through the model to collect layer-wise input statistics (the Hessian matrices $X_\ell X_\ell^\top$ from Equation 1). For dense models, a few hundred thousand tokens suffice. But in an MoE with 2048 experts, each expert sees only a tiny fraction of all tokens — the paper notes that a single expert processes only the tokens routed to it, and in encoder-decoder architectures like SwitchTransformers, each token is processed by only half the model. This means achieving good coverage of all experts requires much more calibration data overall — the paper uses 160K samples for c2048 versus 10K for base128. Maintaining intermediate activations for this many samples during compression consumes "100s of GBs of memory for the largest models" (Section 3.1).
GPU utilization collapses under many small layers. Existing quantization implementations (the paper specifically cites GPTQ and related methods) are optimized for the "massive individual layers occurring in dense models." MoEs invert this: they have many small layers (each expert is a modestly-sized feedforward block) rather than a few large ones. Running thousands of small quantization operations sequentially leads to poor GPU utilization because each individual operation cannot saturate the GPU's compute capacity. The paper quantifies this in Table 1: compressing a sparse encoder layer takes 174.1 seconds with per-expert processing (|E| = 1) versus 28.8 seconds with expert grouping (|E| = 16), a ~6× speedup from batching alone.
Memory transfers dominate if not carefully managed. If activations and weights must be repeatedly transferred between CPU and GPU to cope with the massive memory requirements, performance degrades severely. The 3.2TB model cannot even fit in CPU RAM, let alone GPU memory. The paper notes that "simply (iteratively) loading the original 1.6T model into RAM takes close to 5 hours on our slow disk storage" (Section 5.2, Table 8 caption). Any compression pipeline must be designed from the ground up to minimize data movement.
Reliability becomes a first-order concern. With tens of thousands of layers, "running into rare edge cases, which may break the process, is highly likely" (Section 3.1). These include numerical issues like non-invertible Hessians (the layer-wise second-order matrices used by GPTQ) and model-specific edge cases like extreme routing patterns where certain experts receive anomalously large numbers of tokens.

Challenge 3: Achieving sub-1-bit storage requires a non-trivial compression format, which must also support fast GPU decoding for inference. This is the most technically distinctive challenge. It is not enough to represent weights in less than 1 bit — that representation must be decodable on-the-fly during inference without creating a new bottleneck. The paper breaks this down into four specific sub-challenges in Section 4.2.1:

Entropy-based codes (like Huffman coding) have sequential decoding dependencies — you cannot determine where symbol i begins until you know the variable lengths of all previous symbols. This prevents parallel decoding within a GPU warp.
Binary storage words may contain different numbers of decoded symbols, meaning threads in a warp would diverge in their work — some would be decoding the "next" symbol while others are still on the "current" one. This breaks GPU's SIMT execution model.
Variable-length decoding involves many bit-level operations (shifts, masks) that GPUs are not optimized for.
MoE weight matrices are individually small (each expert is a modest feedforward layer), making it hard to split them into enough independently-decodable segments to keep the GPU busy without additional metadata that would harm compression rates.

The paper frames the competing baseline clearly: "uncompressed half-precision matrix-vector products, which are the primary operation underlying generative inference, easily achieve close to ideal memory-bandwidth utilization" (Section 4.2.1). In other words, the uncompressed baseline is already very efficient in hardware terms. Any compressed format that adds decoding overhead risks being slower than simply storing and reading the uncompressed weights, defeating the purpose.

Prior MoE Compression Work

The paper positions itself relative to several threads of existing research on MoE compression (Section 6, "MoE Compression"):

Task-specific pruning (Chen et al., 2022; Koishekenov et al., 2022). These works compress MoEs by pruning experts or components that are not relevant to a specific downstream task after fine-tuning. This is "downstream" compression — it reduces the model for a particular use case. QMoE, in contrast, performs "upstream" compression of the pretrained base model, preserving generality. The paper explicitly states this distinction, positioning QMoE as solving a different problem: general-purpose compression rather than task specialization.

Higher-bitwidth MoE quantization (Kim et al., 2022b; Yi et al., 2023; Kim et al., 2023). These works apply quantization to MoEs but stay at 8 or 4 bits per weight, primarily using simple rounding. The paper's own experiments (Table 5) show that round-to-nearest at ternary precision produces unacceptably high loss (2.15 for c2048 versus 1.18 baseline), establishing that these simpler methods cannot reach the sub-1-bit regime. The gap is not incremental — it is qualitative. Moving from 4-bit to ternary requires fundamentally different techniques.

Quantization-aware training for MoEs (Kim et al., 2022a). This work achieves 2-bit quantization on a 5-billion-parameter MoE by fine-tuning the model with quantization simulated during training. The paper notes that "applying such an approach for trillion-scale models would be extremely resource intensive" (Section 6). This is a key differentiator for QMoE: it is a post-training method that requires no access to the original training pipeline, no gradient updates to the model weights, and can be run on a single GPU in less than a day. The practical barrier to quantization-aware training at trillion-parameter scale is not just cost — it is the fact that most researchers simply cannot run the training infrastructure required.

No existing work provides a mechanism for exploiting low-bit quantization in actual inference. The paper notes that even the works that do achieve low-bitwidth quantization for MoEs "do not provide any mechanisms for exploiting low-bit quantization and its corresponding natural sparsity in practice, which is challenging and constitutes a key contribution of our work" (Section 6). This is a crucial observation: getting a model into a low-bitwidth representation is only half the problem. Actually running that model efficiently — decoding the compressed weights on-the-fly during inference without creating a new bottleneck — is the other half, and it requires co-designing the compression format with GPU kernels. Prior work treats compression as a storage problem; QMoE treats it as a storage and execution problem simultaneously.

How This Paper Positions Itself

QMoE is not positioned as an incremental improvement over existing quantization methods, but rather as an end-to-end system that solves the full stack of challenges — from scalable compression of trillion-parameter models to efficient compressed inference on commodity hardware — that together make massive MoEs practical for the first time. The paper draws a clear line between what existed before and what QMoE provides:

"Our work enables, for the first time, the performant execution of massive-scale MoE models on commodity hardware. This is illustrated by the fact that we are able to efficiently run the trillion-parameter SwitchTransformer-c2048 model on a single commodity GPU server, with minor accuracy loss. This addresses one of the key limitations behind MoE architectures, and should improve their practical adoption as well as facilitate further research on understanding and improving such models." (Section 1)

The paper's contribution is not a novel quantization algorithm per se — it builds on GPTQ (Frantar et al., 2022) — but rather the system-level engineering and the entropy-coding stage that together push compression into the sub-1-bit regime while maintaining fast GPU decoding. The paper emphasizes this by devoting substantial space to the system design (Section 3.2), the robustness modifications (Section 3.2.5), the dictionary-based encoding scheme (Section 4.3), and the GPU kernel co-design (Section 4.3.3). These are not afterthoughts — they are the core of the contribution.

Finally, the paper positions itself as enabling further research on massive MoEs by dramatically lowering the barrier to entry. When the largest publicly-available model requires a hundred GPUs to run, the number of groups that can study it is vanishingly small. By bringing that requirement down to a single commodity server, QMoE opens the door to broader experimentation — fine-tuning compressed models on downstream tasks (Section 7), studying expert specialization patterns at scale, or using the compressed model as a starting point for further compression research.

3. Technical Approach

3.1 Reader Orientation

This paper describes QMoE, an end-to-end compression and inference framework that takes a massive Mixture-of-Experts language model (specifically the 1.6 trillion parameter SwitchTransformer-c2048, stored in 3.2TB of bfloat16 weights) and produces a compressed version that occupies less than 160GB — under 0.8 bits per parameter — while still being executable for inference on a single commodity GPU server with minimal accuracy loss and runtime overhead. The system solves the problem that trillion-parameter MoEs are too large to fit in the memory of any affordable hardware by combining two stages: first, a scalable data-dependent quantization algorithm that reduces each weight to ternary values (three possible numbers: a learned minimum, zero, and a learned maximum) while preserving model accuracy, and second, a dictionary-based entropy encoding scheme co-designed with custom GPU kernels that exploits the high proportion of zeros in the ternary weights (nearly 89% for the largest model) to achieve sub-1-bit storage and fast on-the-fly decoding during inference.

3.2 Big-Picture Architecture (Diagram in Words)

The QMoE system has four major components that operate in sequence, plus an execution component that runs at inference time:

Calibration Data Pipeline (Section 3.2.1–3.2.3): A memory-efficient system for passing large volumes of text data through the uncompressed model to collect the per-layer activation statistics needed for data-dependent quantization. It uses a list-buffer data structure in CPU memory with optimized activation offloading, lazy weight fetching from disk, and expert grouping to keep the GPU fed with work while the CPU and disk handle the bulk storage.
Scalable Quantization Engine (Section 3.2.4–3.3): A modified version of the GPTQ algorithm that compresses all expert weight matrices to ternary precision simultaneously using expert grouping (batching multiple experts together for GPU utilization), augmented with robustness modifications (increased Hessian dampening, token caps, fallback to rounding for degenerate layers) and accuracy improvements (premasking special tokens in the calibration data). This outputs ternary weight matrices — each weight is one of three values — but stored in a naive 2-bits-per-weight format at this stage.
Entropy Encoding Scheme (Section 4.1–4.3.2): A lossless compression stage that takes the ternary weight matrices and encodes them using a fixed dictionary of variable-length symbol sequences, where frequently-occurring patterns (especially runs of zeros) are assigned short 16-bit codewords and rare patterns are assigned longer codewords. The dictionary is generated once from the statistical distribution of zeros in the ternary weights and is shared across all experts. This reduces storage from ~2 bits per ternary weight to well under 1 bit per weight (achieving 20.07× compression relative to bfloat16 for c2048).
GPU Decoding Kernels (Section 4.3.3): Custom CUDA kernels that decompress the dictionary-encoded weights on-the-fly during inference, fused with the matrix-vector multiplication operation that is the core computation in Transformer inference. The kernel assigns one GPU warp (32 threads) per row of each weight matrix, uses shared memory for the input vector and a ternary dequantization lookup table, and processes one 16-bit codeword at a time — decoding it into up to 28 ternary weights simultaneously — before accumulating the dot product via a warp reduction.
Inference Execution (Section 5.3): At runtime, the compressed model is loaded into GPU memory (now fitting on 4–8 consumer GPUs instead of 65–130). Text input is processed through the standard Transformer forward pass, but whenever a weight-matrix-vector-product operation is required for a quantized expert, the fused decompress-and-multiply kernel is invoked instead of a standard bfloat16 matrix multiply. The dense (non-expert) layers remain uncompressed.

Information flows as follows: raw text calibration data → calibration data pipeline (CPU/GPU coordinated offloading, token routing through uncompressed model) → per-expert activation statistics (Hessian matrices) → GPTQ quantization engine (produces ternary weights in 2-bit format) → dictionary-based entropy encoder (produces the final sub-1-bit compressed representation, stored with a shared dictionary, per-row offsets, and per-row min/max values) → at inference time, the GPU kernels read the compressed format, decompress on-the-fly, and perform the matrix-vector product.

3.3 Roadmap for the Deep Dive

First, the scalable compression system (Section 3.2): how QMoE overcomes the memory, GPU utilization, and reliability barriers to apply data-dependent quantization to a model that is too large to fit in RAM. This is the prerequisite for everything else — you cannot compress what you cannot load.
Second, the accuracy improvements (Section 3.3): two specific discoveries about applying GPTQ to masked-language-modeling MoEs that improve compression quality at zero cost. These are small but practically important optimizations.
Third, the entropy encoding problem (Section 4.1–4.2): why ternary quantization alone does not achieve sub-1-bit storage, where the natural sparsity comes from, and why standard sparse matrix formats are inadequate for exploiting it.
Fourth, the dictionary-based encoding scheme and GPU kernel co-design (Section 4.3): the core technical innovation — a compression format specifically designed to enable fast parallel decoding on GPU hardware, and the CUDA kernel that implements it. This is where the sub-1-bit target is actually achieved in a practically useful way.
Fifth, the end-to-end compression pipeline (Section 5.2): concrete numbers on accuracy, compression rates, and compression time that validate the approach, including scaling behavior across model sizes.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems paper whose core idea is that massive MoE models can be compressed to sub-1-bit precision by combining data-dependent ternary quantization (adapted for MoE scale) with a co-designed entropy-coding scheme and GPU kernel that exploits the emergent sparsity of ternary weights to achieve fast compressed inference on commodity hardware.

The Data-Dependent Quantization Objective

The quantization engine that QMoE builds on solves a layer-wise optimization problem: given the original weight matrix $W_\ell$ of layer $\ell$ and a set of calibration inputs $X_\ell$ observed at that layer when running real data through the model, find quantized weights $Q_\ell$ that minimize the error in the layer's output:

$\arg\min_{Q_\ell} ||Q_\ell X_\ell - W_\ell X_\ell||$

where $W_\ell$ is the original floating-point weight matrix for layer $\ell$ , $Q_\ell$ is the quantized weight matrix (with each entry restricted to a small set of allowed values), and $X_\ell$ is the matrix of input activations observed at this layer from running calibration data through the model.

What it computes: a set of quantized weights $Q_\ell$ whose output — when multiplied by the calibration inputs $X_\ell$ — approximates the output of the original weights $W_\ell$ multiplied by those same inputs, measured by the squared $L_2$ norm of the difference. This is a per-layer, data-dependent objective: "data-dependent" because the optimization targets depend on $X_\ell$ (the actual activations this layer sees in practice), not just on $W_\ell$ in isolation. The objective is solved independently for each layer, using the activations from the partially-quantized model up to layer $\ell-1$ — this sequential execution reduces the accumulation of quantization error across layers.

Why this form: a naive approach would round each weight to its nearest allowed quantized value independently (round-to-nearest, or RTN). This minimizes the $L_2$ error in weight space, $||Q_\ell - W_\ell||$ , but ignores how those weight errors interact with the actual input distribution. Weights that have large magnitude but are always multiplied by near-zero inputs contribute little to output error and don't need precise quantization; weights with small magnitude but that are always multiplied by large inputs need more care. The $Q_\ell X_\ell - W_\ell X_\ell$ formulation automatically captures this: it weights each weight's quantization error by the variance of the corresponding input feature. The paper uses the GPTQ solver for this objective, which leverages the layer-wise Hessian matrix $X_\ell X_\ell^\top$ — representing the second-order correlations between input features — to make coordinated quantization decisions that compensate for errors across different weights in the same row.

Quantization grid: For ternary quantization specifically, the allowed values per row of $W_\ell$ are $\{w_{\text{min}}, 0, w_{\text{max}}\}$ , where $w_{\text{min}}$ and $w_{\text{max}}$ are the minimum and maximum values in that row. This row-wise grid (rather than a global grid shared across the entire matrix) is important because different rows can have different dynamic ranges, and tying them to a single min/max would force some rows to use a badly-calibrated zero point. The zero value is always explicitly included because weights are "typically close to normally distributed" (Section 4.1), meaning most weights cluster near zero and can be rounded to it with minimal individual error.

Memory-Efficient Calibration Data Handling

The first practical barrier to applying data-dependent quantization to a 1.6 trillion parameter model is that you cannot even hold the model in RAM, let alone GPU memory, and the calibration data (the activations $X_\ell$ ) needed for quantization is also massive. The paper addresses this with a carefully orchestrated offloading strategy described in Section 3.2.1 and illustrated in Figure 2.

The core idea: maintain a single large buffer $B$ in CPU memory that holds the hidden states for all tokens currently being processed. Only a small subset of this buffer is ever transferred to the GPU at any moment — specifically, the tokens that are about to be processed by a particular layer or expert. The rest stays in CPU RAM (or on disk for the weights). This is possible because the computation proceeds in a disciplined layer-by-layer, expert-by-expert order where you always know exactly which subset of data is needed next.

For the dense part of a Transformer block (attention, layer norm, the router), the procedure is:

Fetch one "sample" $X$ — containing a few hundred tokens — from CPU to GPU.
Pass it through the dense layers to obtain the output $Y$ .
Calculate and store the expert assignment for each token in $Y$ (the router decides which expert each token goes to).
Send $Y$ back to CPU and overwrite $X$ in the buffer $B$ .

For the sparse (MoE) part, the procedure loops over experts:

Fetch all individual tokens in $B$ that have been assigned to expert $E$ — call this subset $X_E$ — from CPU to GPU.
Use these tokens to compress expert $E$ : compute the Hessian $H_E = X_E X_E^\top$ and run GPTQ to produce the quantized expert $E'$ .
Run $X_E$ through the newly quantized expert $E'$ to get the output activations $Y_{E'}$ .
Send $Y_{E'}$ back to CPU and overwrite $X_E$ in $B$ .

Key advantage: each token's hidden state is read from CPU and written back to CPU exactly twice per Transformer block — once for the dense part and once for its assigned expert. There is no redundant copying, and the CPU buffer $B$ remains the single source of truth. The buffer capacity, not GPU memory, determines how many calibration samples can be used.

The list buffer data structure (Section 3.2.2, Figure 3): Storing $B$ as a simple 2D array of shape [num_tokens, hidden_dim] would make it difficult to efficiently extract tokens belonging to a specific expert (which requires gathering scattered indices) or to process individual samples for the dense layers (which requires slicing along the token dimension). The paper solves this with a list buffer: all token hidden states are packed contiguously into one massive flat buffer, with delimiter indices marking the boundaries between samples. This enables both operations: per-sample access for dense layers (by reading the contiguous segment between two delimiters) and fully-vectorized gathering of expert tokens across all samples simultaneously (by indexing into the flat buffer with pre-computed expert assignment lists). The paper notes that "naively iterating over samples and fetching relevant tokens via masking is unusably slow for large sample counts."

Lazy weight fetching (Section 3.2.3): Since the 1.6T model's weights consume over 3TB — too large even for CPU RAM — they are fetched directly from disk storage as needed. Under the sequential processing order described above, each weight matrix is needed exactly once: when computing $H_E = X_E X_E^\top$ for its corresponding expert. Immediately after use, the weight matrix's memory is released. The paper notes that simply "loading the original 1.6T model into RAM takes close to 5 hours on our slow disk storage" (Table 8 caption), so this lazy strategy avoids both memory pressure and wasted I/O.

Expert Grouping for GPU Utilization

The second practical barrier is that MoEs contain thousands of small weight matrices (one per expert) rather than a few large ones. Running GPTQ on each expert sequentially would leave the GPU severely underutilized because each individual quantization operation is too small to saturate the GPU's compute units. The paper introduces expert grouping (Section 3.2.4): instead of quantizing one expert at a time, group $|\mathcal{E}|$ experts together and quantize them jointly.

Concrete procedure: For a group $\mathcal{E}$ of experts (default size $|\mathcal{E}| = 16$ ), the system extracts the calibration inputs $X_E$ for each expert $E \in \mathcal{E}$ . These inputs will generally have different numbers of tokens (different experts receive different numbers of assignments), but their hidden dimension is the same. The corresponding Hessian matrices $H_E = X_E X_E^\top$ are then computed. The weight matrices $W_E$ and Hessians $H_E$ for all experts in the group are stacked into 3-dimensional tensors, and a modified batched version of the GPTQ algorithm operates on this stacked representation, compressing all experts simultaneously.

Efficiency gain: Table 1 shows the impact. Compressing a single sparse encoder layer of switch-base-128 with 10K samples takes 174.1 seconds when processing experts one-by-one ( $|\mathcal{E}| = 1$ ). Increasing the group size to 4 reduces this to 54.4 seconds (~3.2× speedup). A group size of 16 yields 28.8 seconds (~6× speedup). The trade-off is GPU memory: larger groups require storing more experts' weights and Hessians simultaneously on the GPU. The default of $|\mathcal{E}| = 16$ is selected as the sweet spot between utilization and memory consumption.

Direct Hessian computation: The paper also notes that because the expert input matrices $X_E$ are generally small enough (each expert receives a modest number of tokens), the Hessian can be computed directly as $H_E = X_E X_E^\top$ with a single matrix multiplication, rather than using the slow per-sample accumulation strategy employed by prior GPTQ implementations. This is another efficiency win that the expert-grouped approach enables.

Robustness Modifications for Trillion-Parameter Scale

When compressing a model with tens of thousands of layers, rare edge cases that would be negligible in a smaller model become near-certainties. The paper describes three specific robustness modifications (Section 3.2.5) that were necessary to successfully complete compression of c2048:

Increased Hessian dampening ( $\delta = 0.1$ ): The GPTQ algorithm requires inverting the Hessian matrix $H_E$ (or a regularized version of it) to make optimal quantization decisions. In standard usage on dense models, a small dampening constant $\delta$ is added to the diagonal ( $H_E + \delta I$ ) to ensure numerical stability. For the trillion-parameter MoE setting, the paper increases this dampening by a factor of 10× (to $\delta = 0.1$ ) to "avoid breakdowns with inf-values." The higher dampening makes the Hessian more diagonally dominant, which reduces the influence of second-order corrections but prevents catastrophic numerical failures. The accuracy impact of this higher dampening is implicitly tolerated because the alternative — the compression process crashing — is worse.

Fallback to vanilla rounding for degenerate layers: Even with high dampening, some layers produce Hessians that are still not invertible (singular or near-singular). For these rare cases, GPTQ is skipped entirely and the weights are quantized using simple round-to-nearest (RTN). The paper states this applies to "very few" layers, so the overall accuracy impact is minimal. This is a pragmatic engineering decision: in a model with ~10,000+ expert layers, it is better to have a handful of suboptimally-quantized experts than to have the entire compression pipeline fail.

Token capping for outlier experts: Occasionally, an expert receives a number of tokens that is much larger than the average — an extreme routing pattern where many tokens converge on a single expert. When these tokens are fetched to GPU for compression, the memory required for the per-token activations can cause out-of-memory errors. The paper caps the maximum number of tokens used for compressing any single expert at $4\times$ the mean token count across experts. If an expert exceeds this cap, its compression is split across multiple iterations: the available tokens are processed in chunks, and the output activations $Y_E$ are computed and written back in multiple passes.

Accuracy Improvement: Premasking Special Tokens

The paper discovers a surprising accuracy improvement specific to the masked-language-modeling (MLM) pretraining objective used by SwitchTransformers (Section 3.3). MLM training inserts special "mask" tokens into the input text, which the model must predict. These tokens are extremely common during pretraining, appearing in a large fraction of training examples. The paper observes that the model becomes so robust at predicting them that "any error compensation on them during quantization is unnecessary, while worsening correction for other tokens."

Concrete implementation: In the encoder, the special mask tokens are excluded from the Hessian computation — they are simply not used to inform the quantization decisions. In the decoder, the token immediately before a special separator token is skipped, since this is the position used to predict that token. The masking is applied during the calibration data pass, before the Hessians are computed.

Empirical validation: Table 2 shows the effect for ternary quantization of switch-base-128 with 10K calibration samples. Without premasking, validation loss is 2.16 (versus 1.73 for the uncompressed bfloat16 baseline). With premasking, loss drops to 1.99 — a substantial improvement at zero additional computational cost. The effect is also visible but less pronounced at 2-bit precision (1.86 vs. 1.76).

Ineffective Heuristics for This Setting

The paper also evaluates two optimization heuristics that were recently proposed for improving GPTQ (Frantar et al., 2023) but finds them unhelpful or harmful in the MoE ternary quantization setting (Section 3.3, Table 3).

Activation reordering: This heuristic sorts the columns of the weight matrix by activation magnitude before quantization, quantizing the "most sensitive" columns (those with the largest input activations) first. The idea is that quantizing the most important weights early allows subsequent columns to compensate for the error. The paper finds that, for ternary quantization of switch-base-128, activation reordering increases validation loss from 1.99 (GPTQ baseline) to 2.23 — a substantial degradation. The hypothesized explanation: "in this highly aggressive setting, quantizing all the most sensitive columns first, leads to large changes of the entire weight matrix, and thus to overfitting." In other words, the procedure over-optimizes for the calibration data's activation pattern and loses generalization.

True sequential execution: Standard GPTQ already processes columns sequentially, but "true sequential" refers to updating the remaining unquantized weights after each column is quantized, using the full Hessian information. The paper finds this to be "more or less quality neutral" in their setting (loss remains at 1.99). This is notable because it means the simpler, faster version of GPTQ works equally well here — the extra bookkeeping isn't justified.

Combined (act + seq): Applying both heuristics together further degrades performance to 2.28 loss. This negative result is important: it shows that techniques that help for dense model quantization at moderate bitwidths do not necessarily transfer to extreme compression of MoEs.

Why Ternary Quantization Alone Does Not Achieve Sub-1-Bit Storage

After the GPTQ quantization stage, each expert weight matrix is in ternary format — every weight is one of three values. Naively, storing three values requires $\lceil\log_2(3)\rceil = 2$ bits per weight (with some wasted encoding space). This gives a compression rate of 8× relative to 16-bit bfloat16 (from 16 to 2 bits per weight), which is substantial, but the paper's target is 10–20× — meaning under 1 bit per parameter on average, accounting for all metadata. Section 4.1 establishes the key observation that makes further compression possible.

Natural sparsity: When weights are quantized to the ternary grid $\{w_{\text{min}}, 0, w_{\text{max}}\}$ around a near-normal distribution, a large fraction of weights become exactly zero. Table 4 reports the sparsity levels for different models: base128 has 85.7% zeros, large128 has 86.4%, and c2048 has 88.6%. The standard deviation is under 5%, meaning this is consistent across layers. The sparsity increases with model size, which the paper attributes in part to the weight distributions becoming closer to independent for larger layer sizes.

Why standard sparse formats don't work well enough: The paper considers two standard approaches for exploiting sparsity and explains why they are inadequate at these sparsity levels and base bitwidths:

Bitmask format: A bitmap indicating which weights are non-zero requires 1 bit per weight — just for the mask itself, before storing any values. At ~88.5% sparsity, this means you burn 1 bit on zeros and then need additional bits for the ~11.5% non-zero values. Even if you stored those non-zeros compactly, the bitmask overhead prevents reaching sub-1-bit overall.
Column index format: Store the column indices of non-zero weights, using 10–13 bits per index (depending on layer width), plus the value bits. At 88.5% sparsity, you have ~11.5% non-zeros, each requiring 10+ bits just for its position — this is even less memory-efficient than the bitmask approach for these sparsity levels.

The conclusion: "standard sparsity metadata formats would only allow limited additional compression." The high sparsity is real and exploitable, but not through sparse matrix representations designed for the regime where the base precision is high (e.g., 16-bit or 8-bit values) and the sparsity eliminates most of the storage. Here, the base precision is already only 2 bits, and the metadata overhead dominates.

The entropy perspective: The paper reframes the problem: the ternary weights have low entropy because one value (zero) occurs with probability $p_0 \approx 0.886$ . This means that, on average, fewer than $\log_2(3) \approx 1.58$ bits should be needed per weight if we use a code that assigns shorter representations to more probable symbols. The Shannon entropy for the distribution $P(0) = p_0, P(1) = P(2) = (1-p_0)/2$ with $p_0 = 0.885$ is approximately $-p_0\log_2(p_0) - (1-p_0)\log_2((1-p_0)/2) \approx 0.63$ bits per weight — corresponding to a theoretical compression limit of $16 / 0.63 \approx 25.4\times$ relative to bfloat16. The remaining challenge is: how to build a practical code that approaches this entropy bound while supporting fast GPU decoding.

The Decoding Challenge on GPUs

Section 4.2.1 formalizes why standard entropy-coding approaches (like Huffman coding) are not suitable for the inference use case, even though they would achieve good compression rates. The paper identifies four specific challenges, all stemming from GPU hardware constraints:

Challenge 1 — Sequential decoding dependencies: Entropy codes (Huffman, arithmetic coding) produce variable-length codewords. To decode the $i$ -th symbol in a sequence, you must know where it starts in the bitstream, which depends on the cumulative length of all $(i-1)$ previous codewords. This is an inherently sequential process — you cannot decode symbols 5 and 10 in parallel because you don't know where symbol 10 begins until you've decoded symbols 1 through 9. GPU warps execute 32 threads in lockstep; a sequential dependency forces all threads to wait while one thread decodes its next symbol.

Challenge 2 — Non-uniform decoding work: Even if you break the sequence into independently-decodable blocks (e.g., one block per row), the blocks will contain different numbers of symbols, requiring different amounts of decoding work. On a GPU, threads within a warp must all execute the same instruction simultaneously (SIMT execution model). If thread 0 is still decoding a block with many symbols while thread 1 has finished its shorter block, thread 1 must idle — wasting compute. This is "thread divergence," and it destroys throughput.

Challenge 3 — Operations mismatch: Variable-length decoding involves many bit-level operations: shifts, masks, bit-extractions to pull out individual codewords from packed bitstreams. GPUs are optimized for floating-point and integer arithmetic on 32-bit words, not for bit twiddling on arbitrary bit boundaries. A kernel dominated by shift-and-mask operations will underutilize the GPU's compute units.

Challenge 4 — Small matrix sizes: MoE expert weight matrices are individually small. To achieve good GPU utilization, you need to split work into many independently-decodable segments that can be processed in parallel. But the more segments you create, the more metadata you need to store — offsets, lengths, or dictionary indices for each segment — which eats into your compression budget. There's a direct tension between parallelism (GPU efficiency) and compression rate (storage efficiency).

The uncompressed baseline — a simple bfloat16 matrix-vector product — has none of these problems: the data is uniformly sized, requires no decoding, and maps directly to highly-optimized cuBLAS kernels that achieve near-peak memory bandwidth. Any compressed format that adds 2× overhead in decoding time would be slower than simply storing and reading the uncompressed weights, defeating the purpose entirely. The design challenge is to create a format that stays within ~5% of uncompressed runtime.

Co-Designing the Compression Format and GPU Kernel

The paper's central technical innovation is a compression scheme specifically designed to be decodable fast on GPUs, trading off some compression rate for decoding efficiency. The design is described in Section 4.3 and represents a co-design process where the encoding format and the GPU kernel were developed simultaneously, with each constraining the other.

The Fixed-to-Variable Dictionary Code

Reversing the standard approach: Instead of a variable-length-to-fixed code (like Huffman, where variable-length bit sequences map to single symbols), the paper uses a fixed-length-to-variable code: each codeword has exactly the same bitwidth (16 bits, stored as a UINT16), but maps to a sequence of multiple ternary weights of varying lengths. This completely eliminates Challenge 1: since every codeword is exactly 16 bits, the start of the $i$ -th codeword is trivially at position $16 \times i$ — no sequential decoding dependency exists. The decoder can process codeword $i$ and codeword $j$ independently and in parallel.

This is a dictionary-based code (similar in spirit to LZW compression, popular in ZIP), but applied in the opposite direction from typical text compression. In text compression, LZW maps variable-length repeated strings to fixed-length dictionary indices; here, QMoE maps fixed-length dictionary indices to variable-length ternary weight sequences. The dictionary is pre-computed and shared across all experts, so its storage cost (described below) is amortized.

Dictionary generation (Algorithm 1): The dictionary is built to contain sequences that are likely to occur frequently in the quantized weight matrices, under the simplifying assumption that ternary values are drawn independently from the distribution $P(0) = p_0$ and $P(1) = P(2) = (1-p_0)/2$ . The independence assumption is an approximation — real weight matrices have correlations — but the paper validates empirically that it works well (Section 4.3.2).

Algorithm 1 generates the $2^{16} = 65,536$ most probable sequences:

Start with an empty dictionary and a max-priority queue seeded with the empty sequence $()$ with probability 1.0.
Repeatedly pop the highest-probability sequence $s$ from the queue.
If $s$ has between 1 and 28 ternary values (inclusive), add it to the dictionary.
Expand $s$ by appending each possible pair of ternary values $(t_1, t_2)$ where $t_1, t_2 \in \{0, 1, 2\}$ , and push the resulting sequences back into the queue with probabilities multiplied by $P(t_1) P(t_2)$ .

The procedure uses pairs of ternary values rather than individual values because the dictionary stores up to 14 pairs (28 individual weights) per entry, and the pair count must fit into 4 bits (see the data format below). Working with pairs halves the count and naturally fits this constraint.

The final dictionary contains exactly $2^{16}$ sequences, ordered from highest to lowest probability. The most common sequences (dominated by long runs of zeros) get the same 16-bit codeword length as rare sequences (dominated by non-zero values with many 1s and 2s) — the compression comes from the fact that common sequences are longer (encoding many weights in 16 bits, yielding <1 bit per weight) while rare sequences are shorter (encoding few weights in 16 bits, yielding >1 bit per weight). The weighted average across the actual distribution achieves the sub-1-bit target.

Dictionary storage format (Figure 4): Each of the 65,536 dictionary entries maps a 16-bit codeword to a 64-bit data blob (two consecutive UINT32 values). The format is:

Each ternary weight is stored using 2 bits (values 0, 1, 2 map to binary 00, 01, 10), rather than a more compact base-3 encoding. The reason is practical: 2-bit extraction is fast (a shift and mask), while base-3 encoding would require slow integer modulo and division on the GPU. Since the 64-bit payload has room for up to 28 weights × 2 bits = 56 bits, plus 8 bits of metadata, the 2-bit-per-weight format fits without wasting space.
The 64 bits are organized as two 32-bit halves, each containing up to 7 pairs (14 weights, 28 bits of weight data, leaving 4 bits for the pair count). The pair count is stored twice — once in each half — so that each half can be processed independently by different threads without coordination.
The total number of weights in the full 64-bit entry is 2 × pair_count, stored in 4 bits, giving a maximum of 14 pairs = 28 weights.

The total dictionary storage is $2^{16} \times 8$ bytes = 512KB, which fits comfortably in the GPU's L2 cache (typically several MB on modern GPUs). The paper explicitly states that "keeping the dictionary in the L2-cache of the GPU is critical for good decoding performance."

Validation of the independence assumption: The paper computes achieved compression rates on the real ternary-quantized c2048 model (20.07× compression relative to bfloat16) and on synthetic weight matrices sampled directly from the independent distribution of Equation 2 (21.11×). The gap is only ~5%, confirming that "our simplifying independence assumption is indeed quite close for large models." The achieved rate is ~20% away from the theoretical Shannon limit of 25.40×, which the paper considers "a reasonable trade-off for enabling fast GPU decoding."

The GPU Decoding Kernel

The kernel design (Section 4.3.3, Listing 1) is what makes the dictionary-based approach practically fast. It fuses decompression with the matrix-vector multiplication operation, avoiding any intermediate uncompressed weight storage.

Parallelization strategy: The kernel uses one warp (32 consecutive threads) per row of the weight matrix, with each threadblock handling multiple rows. Specifically, the kernel launches with $\min(\text{num\_rows\_in\_block}, 32)$ warps per threadblock, and one threadblock per Streaming Multiprocessor (SM). If a block has more than 32 rows, warps process multiple rows sequentially. This is described as "an effective heuristic that yields good performance for all matrix shapes we consider" and avoids bad wave quantization effects (where available warps don't cleanly divide the number of SMs).

Rows are encoded independently in the compressed format, separated by row offsets stored in an auxiliary array row_off. This means each warp can decode its assigned row without any coordination with other warps — addressing Challenge 4 by providing fine-grained parallelism with negligible metadata overhead (one integer per row).

Step-by-step execution:

Load input vector to shared memory (lines 7–9): All warps in the threadblock cooperatively load the entire input vector $x$ (of width w_width) into x_shared. This is done once per threadblock, amortizing the global memory read across all rows in the block. Using shared memory for subsequent accesses is critical because each weight in the row will need to multiply with the corresponding input element — shared memory is ~20–100× faster than global memory for these repeated reads.
Initialize dequantization lookup table (lines 11–14): A shared memory table deq[3][32 * num_warps] maps the ternary codes 0, 1, 2 to their actual floating-point values: 0 for code 0, w_min for code 1, and w_max for code 2. The w_min and w_max values are per-row and stored in the ter_minmax array. Crucially, the table is replicated 32 times across the column dimension (one copy per thread) to avoid bank conflicts: when the 28 active threads simultaneously dequantize their decoded ternary values (line 30), they may access different entries. Without replication, multiple threads accessing the same bank but different rows would serialize. With replication, each thread's column of the table is in a different bank, and all accesses hit different banks.
Main decoding loop (lines 22–35): Each warp processes its row in chunks of up to 32 codewords at a time:
- Fetch 32 codewords from global memory into w_comp_block using a single coalesced transaction (line 23). Coalescing is important for memory bandwidth: all 32 threads in the warp read consecutive 16-bit values, which the GPU's memory controller can combine into a single wide memory request.
- Loop over these 32 codewords (lines 26–33), processing one codeword per iteration:
  - Each codeword enc (a UINT16) serves as an index into the dictionary. The dictionary lookup produces two UINT32 values, stored in dec[2 * enc] and dec[2 * enc + 1].
  - The 32 threads in the warp are partitioned: 28 threads (lanes 0–27) do useful work, while 4 threads (lanes 28–31) are idle. The 28 active threads handle the 28 possible weight positions in a decoded sequence.
  - For each active thread, the first UINT32 of the dictionary entry goes to threads 0–13, and the second UINT32 goes to threads 14–27 (line 28: lane / 14 determines which half).
  - The specific ternary value for that thread is extracted from its UINT32 by shifting right by 4 + 2 * (lane % 14) bits and masking with 0x3 (line 29). The +4 skips the 4-bit pair-count field at the start of each UINT32.
  - The ternary code is translated to a float via the dequantization table (line 30), and the thread accumulates w * x_shared[idx + lane] into its private res register (line 31).
  - After multiplication, each thread advances its index idx by 2 * (wx14 & 0xf) — twice the pair-count, since the pair-count is in the low 4 bits of wx14 (line 32). This ensures that across the 28 threads, the index advances by the total number of weights decoded from this codeword, keeping all threads synchronized for the next codeword.
Warp reduction (lines 37–38): After all codewords for the row are processed, the 28 partial dot products in the thread-local res registers are summed using a warp shuffle reduction (__shfl_down_sync). This is a tree reduction: first threads 0–13 add the values from threads 14–27, then 0–6 add from 7–13, and so on, halving each time until thread 0 holds the full sum. This takes $\log_2(32) = 5$ rounds of shuffle-and-add.
Write output (lines 39–40): Thread 0 writes the accumulated sum to the output buffer y[row], converted back to bfloat16.

Key design choices explained:

Fixed-to-variable with large dictionary: This allows the entire warp to cooperatively process one codeword at a time — all threads decode from the same codeword simultaneously, extracting different weight positions. This eliminates thread divergence (Challenge 2): all 28 active threads do exactly the same operations on each codeword, just with different lane-dependent shifts.
2-bit-per-weight in dictionary entries, not compact ternary: Extracting a value from a 2-bit field is a simple shift-and-mask; extracting from a base-3 packed representation would require division and modulo by 3, which GPUs handle poorly. The slight storage overhead (2 bits vs. $\log_2(3) \approx 1.58$ bits) is paid only within the 512KB dictionary, not in the compressed model itself, so it has negligible impact on overall compression.
Exact 16-bit codewords: Using UINT16 native type avoids any bit-level extraction at the codeword level — the hardware loads 16-bit values directly. The only bit operations are within the 64-bit dictionary entries (line 29), and these are amortized across 28 threads.
Shared memory dequantization table with column replication: The deq[3][32 * num_warps] table is replicated 32 times across columns to avoid bank conflicts. Without replication, when different threads dequantize different ternary values (0, 1, or 2), they would access different rows of the same column, hitting the same bank and serializing. With replication, each thread accesses its own column, eliminating conflicts. The 3-row × 32-column table is small enough to fit in shared memory alongside the input vector.
Caching behavior: The dictionary is sorted from highest to lowest probability, so the most frequent codewords (which tend to be long zero-runs) are at the beginning of the dictionary array. On GPU, each cache line fetched from L2 into L1 typically contains multiple consecutive dictionary entries. Since the most common codeword indices are clustered at the start, the lookups for those entries will frequently hit in L1 cache after the first access, further reducing latency.

Encoding (compression time): The paper briefly describes the encoding procedure (Section 4.3.3, "Encoding"): a trie is built from the dictionary, mapping sequences of ternary values to their codewords. The input weight matrix is scanned sequentially while traversing the trie to find the longest prefix match, producing the corresponding codeword. Rows are encoded independently, and the resulting variable-length rows (different numbers of codewords depending on the natural sparsity pattern) are packed densely into a contiguous buffer, with row offsets recorded. Encoding is less latency-critical than decoding (it happens once offline during compression, not during every inference call), so a straightforward GPU kernel with one thread per row suffices.

Compression Time and Resource Usage

The paper reports compression times in Table 8 for different model sizes and calibration data amounts:

Model	5K/80K samples	10K/160K samples	20K/320K samples
base128	8.4 min	14.0 min	21.6 min
large128	22.0 min	30.2 min	45.2 min
c2048	13.3 hr	16.0 hr	20.8 hr

All experiments run on a single NVIDIA A6000 GPU with 48GB of memory, though the c2048 compression requires "a few 100GBs of (CPU) RAM" and over 3TB of disk storage. The runtime scaling from large128 to c2048 is roughly proportional to the increase in total parameters (26B to 1.6T is ~60×, while 30 minutes to 16 hours is ~32×), which is faster than linear scaling because the number of calibration samples per expert stays constant (the 16× increase in total samples for c2048 compensates for the much larger number of experts, keeping the per-expert data volume roughly constant) and the expert size increases only slightly. The ~5 hours required simply to load the model from disk is not counted in the compression time.

Summary of Design Choices and Their Justifications

Two-stage compression (GPTQ quantization + entropy encoding): GPTQ alone cannot reach sub-1-bit storage because it outputs ternary weights that naively require 2 bits each; the entropy encoding stage exploits the statistical structure (high zero frequency) that GPTQ creates. Neither stage alone is sufficient — GPTQ provides the accuracy-preserving quantization, and the dictionary code provides the sub-1-bit representation.
Expert grouping with $|\mathcal{E}| = 16$ over per-expert quantization: increases GPU utilization by 6× (Table 1), making trillion-parameter compression feasible in under a day. The batch size of 16 is an empirical sweet spot between GPU memory consumption and utilization.
Fixed-to-variable dictionary code with 16-bit codewords over variable-length entropy codes (Huffman, arithmetic): eliminates sequential decoding dependencies, enables warp-level parallelism (all threads process the same codeword simultaneously), and uses native hardware-supported data types. The ~20% gap from the theoretical entropy limit is a deliberate trade-off for GPU efficiency.
2-bit-per-weight internal storage in dictionary entries over compact base-3 encoding: avoids slow integer modulo and division operations on the GPU during decoding. The extra storage is confined to the 512KB dictionary, not the model weights.
Row-wise independent encoding and one-warp-per-row kernel over larger decoding blocks: provides sufficient parallelism for all relevant matrix sizes with negligible metadata overhead (one row offset per row), addressed Challenge 4.
Dictionary sorted by probability over random ordering: exploits L1 cache locality — frequent codewords (at the start of the array) benefit from cache line prefetching of their neighbors.
Premasking special tokens over standard calibration data usage: leverages the observation that the model is over-robust to errors on MLM-specific tokens; removing them from the Hessian computation improves quantization of normal tokens at zero cost.
Robustness modifications ( $\delta = 0.1$ , RTN fallback, token capping) over standard GPTQ defaults: these are not theoretically justified improvements but practical engineering necessities for handling 10,000+ layers without process failures. They represent a pragmatic acceptance that at trillion-parameter scale, rare events become certainties.

4. Key Insights and Innovations

Innovation 1: Massive MoEs are fundamentally more compressible than equivalently-sized dense models, and the compressibility increases with scale

This is the paper's most consequential empirical discovery, and it is not an incremental refinement of existing quantization knowledge — it is a property of the model class that had not been previously characterized. Before this work, the dominant assumption in the compression literature was that compressibility is primarily a function of parameter count: larger models are more robust to quantization noise than smaller ones (Frantar et al., 2022; Chee et al., 2023), but the relationship was understood as a monotonic function of model size, not architecture. The paper's finding that MoEs are qualitatively different in their compressibility — sustaining ternary quantization at 6.7% relative loss increase on a 1.6T parameter model, a regime where dense models would collapse — reorients the compression challenge from "how do we push post-training quantization below 2 bits for general LLMs?" to "how do we exploit the specific statistical structure of MoE weights?"

The evidence for this claim is in Table 5 and Table 4, but the insight goes deeper than the numbers. Table 4 shows that natural sparsity after ternary quantization increases with model size (85.7% → 86.4% → 88.6% for base128 → large128 → c2048), which is the opposite of what one might expect if larger models simply had more diverse weight distributions. The paper attributes this partly to weight distributions becoming "closer to independent for larger layer sizes" (Section 5.2), which is not an obvious consequence of scaling and represents a structural property of how MoE training interacts with extreme quantization. This matters because it means the method becomes more effective at precisely the scale where it is most needed — a rare and practically important alignment between technique and target.

The paper's framing of this finding is also distinctive: rather than treating the MoE architecture as an obstacle to compression (many small layers, routing stochasticity), it argues that MoE-specific properties — the stochasticity from token dropping and routing instability during training, the isolation of parameters in experts that are only occasionally activated — make these models inherently more noise-resistant. This is not obvious from first principles, and the paper provides evidence for it by showing that even vanilla round-to-nearest (RTN) quantization at ternary precision does not cause complete model collapse (Table 5: RTN c2048 ternary loss is 2.15 versus 1.18 baseline), whereas one would expect catastrophic degradation for a dense model at comparable precision. The conceptual move is reframing MoEs from "hard to compress because they're too big to handle" to "uniquely compressible because of how they're trained and structured."

Innovation 2: The two-stage compressibility decomposition — quantization creates the statistical structure, entropy coding exploits it

This is the paper's central architectural insight, and it represents a conceptual separation that has implications beyond this specific system. Prior work on extreme quantization (2–3 bits) treated the quantization step as the entire compression pipeline: you choose a bitwidth, you quantize to that bitwidth, and you store at that bitwidth. The paper demonstrates that this collapses two distinct problems into one. Quantization is an accuracy-preserving transformation: it maps weights to a small discrete set while minimizing the impact on model outputs, but the resulting representation is still stored at ceil(log2(K)) bits per weight for K quantization levels. Entropy coding is a statistical compression applied to the already-quantized representation: it exploits the non-uniform distribution of the quantized values to reduce the average bitwidth below the naive storage cost.

The innovation is not that these two stages exist — both quantization and entropy coding are individually well-known — but the recognition that for MoEs at ternary precision, the gap between the naive bitwidth (2 bits for ternary) and the achievable bitwidth via entropy coding (under 1 bit) is enormous and worth exploiting, and that the quantization stage creates the statistical structure (high zero frequency) that makes entropy coding effective. This is a different division of labor than in prior work: standard post-training quantization treats the quantizer as the primary compression mechanism and the storage format as an implementation detail; QMoE treats the quantizer as a structure generator and the entropy codec as an equal partner in achieving the final compression rate.

The paper quantifies this gap precisely: the theoretical Shannon limit for the ternary weight distribution with p₀ = 0.885 is 25.40× compression relative to bfloat16, while naive 2-bit storage achieves only 8×. The practical dictionary-based code achieves 20.07× — capturing roughly 70% of the gap between naive storage and the entropy limit. This decomposition also explains why prior MoE quantization work at 8 or 4 bits (Kim et al., 2022b; Yi et al., 2023) did not attempt sub-1-bit compression: the entropy gap between 4-bit naive storage (4 bits) and the entropy limit for 4-bit quantized weights is much smaller, making the second stage less impactful. It is specifically the combination of ternary quantization (which creates high sparsity) and large model scale (which increases sparsity further) that makes the two-stage approach compelling.

Innovation 3: Warp-level fixed-to-variable dictionary decoding as a GPU-friendly alternative to variable-length entropy codes

This is the paper's core systems contribution, and what makes it an innovation rather than just engineering is the inversion of the standard entropy-coding paradigm specifically for the GPU execution model. The natural approach to exploiting low entropy is a variable-length code (Huffman, arithmetic coding) that assigns short bit sequences to common symbols and long bit sequences to rare ones. This is optimal in the information-theoretic sense but creates exactly the sequential dependencies that break GPU parallelism. The paper's alternative — a fixed-length-to-variable dictionary code — is not novel in isolation (LZW has existed since 1984), but its application to GPU weight decoding is, and the specific design choices (16-bit codewords mapping to 64-bit data blobs, decoded cooperatively by a 28-thread subset of a warp) are the result of first-principles reasoning about what GPU hardware actually executes efficiently.

The innovation can be stated as a design principle: rather than optimizing a compression format for the rate-distortion tradeoff and then attempting to write a fast decoder for it, one should co-design the format and the kernel from the start, accepting a controlled sacrifice in compression rate (the ~20% gap from the Shannon limit) in exchange for eliminating the GPU execution pathologies (sequential dependencies, thread divergence, bit-level extraction overhead) that would otherwise dominate runtime. The paper articulates this through the four specific challenges in Section 4.2.1, but the conceptual contribution is the process of diagnosing these challenges as fundamental incompatibilities between standard entropy codes and GPU SIMT execution, and then building a format that circumvents all of them simultaneously.

What elevates this beyond a straightforward engineering choice is the validation that the independence assumption underlying the dictionary generation is "indeed quite close for large models" (Section 4.3.2), with only ~5% gap between the compression rate achieved on real ternary c2048 weights (20.07×) and on synthetic weights sampled from the independent distribution (21.11×). This means the fixed-to-variable dictionary approach is not a hack that happens to work for one model — it exploits a genuine statistical property of large MoE weight distributions (near-independence of ternary values) that makes it broadly applicable. The paper does not overclaim this as a theoretical result, but the empirical validation that the independence approximation holds at scale makes the approach principled rather than heuristic.

Innovation 4: The diagnosis that MoE-specific calibration data challenges require system-level solutions, not just better quantization algorithms

This is a subtle but important conceptual contribution buried in the systems engineering of Section 3. The dominant approach in post-training quantization research is to improve the optimization algorithm — better solvers for Equation 1, better rounding strategies, better handling of outliers (Dettmers et al., 2023b; Chee et al., 2023). The paper implicitly argues that for trillion-parameter MoEs, the bottleneck is not algorithmic but logistical: the calibration data volume needed for good expert coverage is so large that data movement and memory management dominate, and the number of layers is so large that reliability (numerical edge cases, outlier routing patterns) becomes a first-class concern.

The innovation is in identifying and systematically addressing a set of problems — activation offloading with a list buffer, lazy disk-based weight fetching, expert grouping for GPU utilization, robustness modifications for edge cases — that are not specific to the GPTQ algorithm and would apply to any data-dependent compression method operating at this scale. The paper explicitly notes this: "most techniques described below will generalize to other data-dependent quantization approaches, like ZeroQuant, as well" (Section 3.2). This is a systems architecture for scalable MoE compression that is algorithm-agnostic, and it represents a shift in how to think about the compression problem: from "design a better quantizer" to "design a compression pipeline that can physically handle the model."

The significance of this insight is amplified by the fact that it enables compression on modest hardware — a single A6000 GPU and a server with a few hundred GB of RAM, not a cluster. The paper's framing in Section 3.2 emphasizes that preserving "the key feature of post-training compression techniques: the ability to perform effective compression using only modest computational resources" required solving these system-level challenges. Without the activation offloading, the list buffer, the expert grouping, and the lazy weight fetching, the compression simply could not be completed on the available hardware, regardless of how good the quantization algorithm was. This reframes scalability as a systems property rather than an algorithmic one — a perspective that is underappreciated in the quantization literature, which tends to report algorithmic improvements measured on models small enough to fit comfortably in GPU memory.

The robustness modifications (Section 3.2.5) are particularly revealing as a diagnostic contribution. The fact that 10× higher Hessian dampening, fallback to RTN for degenerate layers, and token capping were necessary to complete compression of c2048 suggests that there is a qualitative difference between compressing a model with hundreds of layers (where edge cases are rare) and tens of thousands of layers (where edge cases are certain). This has implications for future work on even larger models: reliability engineering, not just algorithmic accuracy, will become an increasingly central concern.

5. Experimental Analysis

Evaluation Methodology

Dataset. All compression and evaluation uses the C4 dataset (Raffel et al., 2020a), the same corpus on which SwitchTransformers were originally pretrained for a masked-language-modeling (MLM) objective. Calibration data for compression is drawn from the first two shards of the training set, while evaluation uses 128 samples from the first shard of the validation set — these 128 samples correspond to over 10K tokens, which the paper notes "is quite stable." The authors use the HuggingFace reproduction of C4 and their replication of the original masking procedure. For out-of-distribution evaluation, additional data is sampled from RedPajama (Computer, 2023), covering Arxiv, GitHub, StackExchange, and Wikipedia subsets.
Base models. The paper focuses on the SwitchTransformer family (Fedus et al., 2022), specifically three variants: base128 (7B parameters, 128 experts), large128 (26B parameters, 128 experts), and c2048 (1.6T parameters, 2048 experts) — the largest publicly-available model. The paper explicitly justifies this choice: SwitchTransformers are "among the most popular massive MoEs, with several implementations across frameworks," and they feature "a similar or higher number of training tokens to parameters ratio than potential alternatives like Artetxe et al. (2022)." The c2048 variant is the central case study for demonstrating trillion-parameter compression.
Metrics. The primary metric is C4 validation loss under the masked-language-modeling objective — the same loss function the models were trained on. This is reported as the average loss over 128 validation samples (>10K tokens). The paper argues this is a well-established evaluation protocol in LLM quantization research (Yao et al., 2022; Frantar et al., 2022; Dettmers & Zettlemoyer, 2022), chosen because it tests "general upstream compression directly on this pretraining task/dataset combination." For the compressed representation, additional metrics are compression rate (ratio of original bfloat16 size to compressed size, both MoE-only and including all model layers and metadata) and checkpoint size in GB. For runtime evaluation, the paper reports per-layer matrix-vector product time relative to bfloat16 (measured in isolation on individual weight matrices) and end-to-end generation time for producing 128 tokens from a single C4 prompt.
Baselines. The paper uses two distinct baselines. First, for quantization accuracy, the baseline is round-to-nearest (RTN) quantization (Dettmers et al., 2022) — a data-free method that simply rounds each weight to its nearest allowed quantized value on the row-wise min-max grid. The paper simulates RTN within the QMoE framework by "fixing Hessians to the identity matrix, thus applying precisely the same quantization settings and evaluation protocol." This ensures a clean comparison controlling for all factors except the data-dependent correction. Second, for runtime evaluation, the baseline is uncompressed bfloat16 matrix-vector products using PyTorch's standard cuBLAS kernels — described as achieving "close to ideal memory-bandwidth utilization." The bfloat16 baseline for c2048 cannot be run directly (it would require 65+ GPUs), so the paper estimates it by having all experts in a layer point to the same weight data, collecting timings "with precisely the same overheads as for our compressed models."
Generation budget / compute accounting. The paper does not use a conventional "budget" framework like sampling N generations. Instead, compression budget is measured along three axes: (a) calibration data volume — the number of text samples used for the data-dependent quantization stage, with default settings of 10K samples for 128-expert models and 160K for 2048 experts, and sweeps at 0.5× and 2× these defaults; (b) compression time — absolute wall-clock time on a single NVIDIA A6000 GPU (48GB), reported in Table 8; (c) hardware resources — all compression runs on one GPU, with c2048 requiring "a few 100GBs of (CPU) RAM" and >3TB disk storage. For inference, the budget is measured as end-to-end wall-clock time for generating 128 tokens on 4× A6000 or 8× RTX 3090 GPUs.
Cross-validation / statistical protocol. The paper does not employ cross-validation in the conventional sense because the evaluation is on pretraining validation loss (a fixed test set), not on a downstream task requiring hyperparameter selection. The calibration data is drawn deterministically from the first two shards of the training set, and evaluation is on the first shard of the validation set — both are fixed, not split or rotated. The paper does not report confidence intervals or statistical significance tests. For the robustness of the loss metric, the paper notes that evaluation on 128 samples (>10K tokens) is "quite stable" but does not quantify variance. The ablation of calibration data quantity (0.5×, 1.0×, 2.0× defaults in Table 5) implicitly serves as a sensitivity analysis, showing that results are stable with respect to this parameter.

Main Quantitative Results

Quantization Accuracy: 2-Bit and Ternary Compression Across Model Scales

The headline result from Table 5 is that data-dependent quantization via QMoE achieves substantial accuracy preservation at bitwidths where simpler methods fail:

Method	base128 (tern)	large128 (tern)	c2048 (tern)
BF16 (uncompressed)	1.73	1.55	1.18
RTN	4.54	2.79	2.15
QMoE (1.0× calib)	1.99	1.69	1.26
Relative degradation (QMoE)	+15.0%	+9.0%	+6.7%

Several patterns emerge from this table:

First, RTN does not cause complete model collapse even at ternary precision. The paper describes this as "perhaps surprising" (Section 5.2). For c2048, RTN ternary loss is 2.15 — substantially worse than the 1.18 baseline, but the model still produces meaningful predictions rather than random outputs. This is cited as evidence for the "high robustness of large MoEs to quantization." The degradation is much worse for smaller models: base128 RTN ternary loss is 4.54 versus 1.73 baseline, an increase of 162%. This suggests that MoE robustness to quantization noise scales strongly with model size.

Second, data-dependent quantization (QMoE/GPTQ) closes most of the accuracy gap. At 2-bit precision, QMoE achieves near-lossless compression: c2048 2-bit loss is 1.20 versus 1.18 baseline (1.7% relative increase). At ternary precision, the gap is larger but still modest: c2048 ternary loss is 1.26 (6.7% relative). The improvement over RTN is dramatic — for c2048 ternary, QMoE reduces the loss degradation from +82% (RTN) to +6.7%.

Third, compressibility improves with model scale. The relative loss increase from ternary quantization decreases monotonically as the model grows: +15.0% for base128, +9.0% for large128, +6.7% for c2048. This is the inverse of what would be expected if larger models simply magnified quantization error. The paper attributes this to both "higher resistance to noise" and the specific distributional properties that make large MoE weights "closer to independent" (Section 5.2).

Fourth, additional calibration data provides diminishing returns. Doubling calibration samples from 1.0× to 2.0× the default improves ternary loss only slightly: base128 goes from 1.99 to 1.93, large128 from 1.69 to 1.64, and c2048 from 1.26 to 1.26 (no change). Halving the data (0.5×) causes noticeable degradation for base128 ternary (1.99 → 2.11) but minimal impact for c2048 ternary (1.26 → 1.27). This suggests the largest models are more robust to limited calibration data, consistent with their overall higher noise tolerance.

Out-of-Distribution Robustness

Table 6 evaluates the compressed c2048 model on four domains drawn from RedPajama that are not part of the C4 calibration distribution. The paper notes that "only < 0.01% of our C4 calibration data originates from those websites":

Domain	BF16 Loss	2-bit Loss	Ternary Loss
Arxiv	1.31	1.34	1.42
GitHub	0.99	1.05	1.13
StackExchange	1.15	1.17	1.22
Wikipedia	1.20	1.24	1.32

The key observation: the relative degradation patterns are similar to those on the in-distribution C4 data. Ternary quantization increases loss by 8.4% on Arxiv (compared to 6.7% on C4), 14.1% on GitHub, 6.1% on StackExchange, and 10.0% on Wikipedia. There is no catastrophic failure on any out-of-distribution domain, suggesting the compression does not overfit to the calibration distribution. The paper does not report RTN results on these domains, so the contribution of data-dependent correction specifically for OOD robustness cannot be isolated.

Compression Rates: Achieving Sub-1-Bit Storage

Table 7 reports the actual compression rates achieved by the full QMoE pipeline (ternary quantization + dictionary-based entropy encoding):

Model	MoE-only compression	Full model compression	Compressed size
base128	17.06×	11.76×	1.27 GB
large128	18.34×	13.32×	3.96 GB
c2048	20.07×	19.81×	158.6 GB

The MoE-only rate measures compression relative to only the expert layers (which are quantized), while the full model rate includes the uncompressed dense layers (attention, layer norm, embeddings, router). For c2048, the nearly identical MoE-only (20.07×) and full-model (19.81×) rates reflect the fact that "almost all parameters are located in the experts" — the uncompressed dense components contribute negligibly to total model size. The full-model rate of 19.81× corresponds to 0.807 bits per parameter, satisfying the paper's stated goal of sub-1-bit compression.

The compression rate increases with model size along both metrics. The paper attributes this to two factors: (a) natural sparsity after ternary quantization increases (Table 4: 85.7% → 86.4% → 88.6%), providing more exploitable zero patterns; (b) the encoding dictionary is optimized for c2048's specific zero probability distribution, so smaller models with different sparsity patterns use a slightly suboptimal dictionary.

The absolute sizes tell a compelling story: the 1.6T parameter model shrinks from 3.2TB (bfloat16, tabulated as 3142GB due to overhead) to 158.6GB — a size that fits within the combined GPU memory of 4× A6000 (4 × 48GB = 192GB) or 8× RTX 3090 (8 × 24GB = 192GB).

Compression Runtime Efficiency

Table 8 reports wall-clock compression times on a single A6000 GPU:

Model	5K/80K calib	10K/160K calib	20K/320K calib
base128	8.4 min	14.0 min	21.6 min
large128	22.0 min	30.2 min	45.2 min
c2048	13.3 hr	16.0 hr	20.8 hr

The scaling properties are notable: from large128 (26B parameters) to c2048 (1.6T parameters), the parameter count increases by ~60×, but the compression time at default calibration settings increases from 30.2 minutes to 16.0 hours — approximately 32×. This sub-linear scaling (in terms of time-to-parameter ratio) is attributed to the fact that "the number of samples per expert stays constant and the expert size increases only slightly" (Section 5.2). In other words, c2048 has more experts (2048 vs. 128), but each expert is only modestly larger, and each expert receives a similar number of calibration tokens.

The largest configuration (c2048 at 20K/320K) compresses in under 21 hours — less than one day. The paper emphasizes this as demonstrating the "high efficiency of QMoE" and contrasts it with quantization-aware training approaches that "would be extremely resource intensive" at this scale. The 5 hours required simply to load the original model from disk is not counted in these times, however.

Per-Layer Kernel Performance

Figure 5 (Left) shows the runtime of QMoE's compressed matrix-vector product kernels relative to PyTorch's uncompressed bfloat16 cuBLAS kernels, measured on individual weight matrices of the shapes found in the SwitchTransformer MoE layers. The matrix shapes evaluated are the standard feedforward dimensions: 768 × 3072, 3072 × 768 (base128), 1024 × 4096, 4096 × 1024 (large128), and 2080 × 6144, 6144 × 2080 (c2048). Results are shown for both NVIDIA RTX 3090 and A6000 GPUs.

The headline finding: the compressed kernels are faster than uncompressed bfloat16 on all tested matrix shapes, achieving relative times between ~0.70 and ~0.98 on the A6000, and between ~0.60 and ~0.95 on the RTX 3090. The speedup is most pronounced on larger matrices (the c2048 shapes show the largest margins) and on the RTX 3090 (which has higher memory bandwidth relative to compute throughput, making the memory-bandwidth savings from compression more impactful).

The paper explains this counterintuitive result — compressed execution being faster than uncompressed, despite the additional decoding computation — as a consequence of reduced global memory traffic. The compressed format reads far fewer bytes from GPU DRAM than the bfloat16 baseline (0.8 bits vs. 16 bits per weight), and the decoding overhead (dictionary lookups, bit extraction) is amortized across 28 threads and uses fast shared memory and L1 cache. The memory-bandwidth savings outweigh the extra compute. The paper notes that these are "very low-latency operations, with the smallest matrix taking <0.02 milliseconds and the largest <0.05" — meaning they operate in a regime where kernel launch overhead and memory latency dominate, making bandwidth reduction particularly effective.

The practical significance: the per-layer speedup means that the end-to-end overhead of compressed inference should be small or negative, a point validated by the next result.

End-to-End Inference Runtime

Figure 5 (Right) shows end-to-end generation time for producing 128 tokens from a single C4 prompt, running the full compressed model within HuggingFace. Three model sizes are shown (base128, large128, c2048), each on two GPU configurations (A6000 and RTX 3090). The bfloat16 baseline for c2048 cannot be run directly, so the paper estimates it by having all experts point to the same weight data — a configuration that eliminates memory pressure and is described as "a highly optimistic estimate since real execution would require close to 20× more GPUs, with corresponding communication overheads."

The central result: end-to-end execution of compressed models is only <5% slower than standard (uncompressed) execution. For base128 on A6000, the compressed model generates 128 tokens in approximately 3.5 seconds versus approximately 3.3 seconds for uncompressed. For c2048 on 4× A6000, compressed execution takes approximately 5.5 seconds versus an estimated approximately 5.3 seconds for the (idealized) uncompressed baseline.

The paper notes that "this slight slow-down despite faster per-layer timings is due to the fact that the encoder may sometimes route multiple tokens to the same expert." In the uncompressed baseline, multiple tokens assigned to the same expert are processed via a batched matrix multiplication, which is highly efficient. QMoE's current implementation "naively executes a separate matrix-vector product for each token," which loses the batching benefit. The paper suggests this could be addressed by adding an inner loop over tokens to the kernel or by fully decompressing first followed by a standard batched matmul when token counts are large.

The resource requirements tell the main story: c2048 runs on 4× A6000 GPUs for the compressed version versus an estimated 65+ A6000 GPUs for the uncompressed version — a ~16× reduction in GPU count. On RTX 3090s, the reduction is even starker: 8 GPUs for compressed versus 130+ for uncompressed.

Ablation Studies and Robustness Checks

Premasking special tokens (Table 2): For ternary quantization of switch-base-128 with 10K calibration samples, excluding MLM special tokens from the Hessian computation reduces validation loss from 2.16 to 1.99, with the uncompressed baseline at 1.73. The effect at 2-bit is smaller but still present: 1.86 vs. 1.76. The paper characterizes this as "noticeably lower loss at no additional compute cost" and hypothesizes that the model is over-robust to errors on frequently-occurring mask tokens, so compensating for quantization error on them is unnecessary while degrading correction for other tokens. This is a practically significant finding because it is zero-cost and specific to MLM-pretrained models — it would not apply to autoregressive (causal) LLMs.

Activation reordering (Table 3): Applying activation reordering to GPTQ for ternary quantization of switch-base-128 increases validation loss from 1.99 (baseline GPTQ) to 2.23. This is a negative result with a substantive explanation: "in this highly aggressive setting, quantizing all the most sensitive columns first, leads to large changes of the entire weight matrix, and thus to overfitting." The paper suggests that the extreme compression regime (ternary, with only three possible weight values) amplifies the error from early quantization decisions in a way that the activation reordering heuristic — designed for moderate bitwidths — does not account for. This is an important caution for practitioners: techniques validated at 3–4 bits may not transfer to 2-bit or ternary settings.

True sequential execution (Table 3): Using the full sequential variant of GPTQ (updating remaining weights after each column quantization) yields validation loss of 1.99 — identical to the standard GPTQ baseline. The paper describes this as "more or less quality neutral." This is a useful finding because the standard variant is faster and simpler, so there is no accuracy reason to adopt the more complex sequential approach in this setting.

Combined heuristics (Table 3): Applying both activation reordering and true sequential execution together yields loss of 2.28 — worse than either alone. The paper does not elaborate on the mechanism, but this reinforces the pattern that optimizations designed for moderate compression regimes can interact negatively under extreme quantization.

Natural sparsity scaling (Table 4): The proportion of weights that become exactly zero after ternary quantization increases with model size: 85.7% (base128), 86.4% (large128), 88.6% (c2048). Standard deviation is "<5%," indicating consistency across layers. The paper uses this to motivate the entropy-coding stage and to explain why compression rates improve with scale. This is not strictly an ablation, but it establishes the relationship between model size and the statistical property that the compression scheme exploits.

Calibration data quantity (Table 5, QMoE 0.5×, 1.0×, 2.0× rows): For c2048 ternary, halving the calibration data (80K samples instead of 160K) increases loss from 1.26 to 1.27 — a negligible change. Doubling calibration data (320K) leaves loss unchanged at 1.26. For base128 ternary, the sensitivity is higher: 2.11 (0.5×) vs. 1.99 (1.0×) vs. 1.93 (2.0×). This suggests that larger models are more robust to limited calibration data quantity, consistent with their overall higher noise tolerance. The paper uses the 1.0× default as the primary reported setting, which appears to be near or at the point of diminishing returns for all model sizes.

Dictionary compression validation (Section 4.3.2, "Validation"): The achieved compression rate on real ternary-quantized c2048 weights (20.07×) is compared against the rate achieved on synthetic weights drawn from the independent distribution of Equation 2 with p₀ = 0.885 (21.11×). The gap of ~5% validates the independence assumption used in dictionary generation. The achieved rate is ~20% below the Shannon-theoretic limit (25.40×), which the paper frames as a "reasonable trade-off for enabling fast GPU decoding." This is an internal consistency check, not a comparison against an alternative encoding scheme — the paper does not implement a Huffman or arithmetic code for comparison.

Expert grouping size (Table 1): The compression time for a sparse encoder layer of switch-base-128 with 10K samples scales with expert group size: 174.1s for |E| = 1, 54.4s for |E| = 4 (~3.2× speedup), and 28.8s for |E| = 16 (~6× speedup). The paper selects |E| = 16 as the default, describing it as "a good trade-off between GPU memory consumption and utilization." The marginal benefit of further increases is implied to not justify the additional memory cost, though this is not explicitly quantified.

Comparison against prior MoE compression (Section 6, discussed but not ablated directly): The paper does not implement and compare against prior MoE compression methods (Kim et al., 2022a; Kim et al., 2022b; Yi et al., 2023) within its own experimental framework. Instead, it references their reported results and notes that they operate at higher bitwidths (8 or 4 bits) and/or on much smaller models (5B parameters). The RTN baseline in Table 5 serves as a proxy for the simpler rounding-based quantization used in those works, and the large gap between RTN ternary and QMoE ternary (e.g., 2.15 vs. 1.26 for c2048) argues for the necessity of data-dependent methods at these bitwidths. However, the absence of a direct comparison at a shared bitwidth (e.g., 4-bit) on the same models makes it difficult to assess whether QMoE's compression algorithm itself is better than prior methods, or whether the gains come primarily from the two-stage architecture (quantization + entropy coding) that enables the lower bitwidth.

Critical Assessment

Claim 1: QMoE achieves accurate sub-1-bit compression of trillion-parameter MoEs.

The evidence for this claim is strong but carries an important caveat about what "accurate" means in context. Table 5 demonstrates that c2048 ternary quantization achieves a 6.7% relative increase in C4 validation loss (1.18 → 1.26), and Table 7 confirms that the full compressed model achieves 19.81× compression, corresponding to 0.807 bits per parameter. These numbers are unambiguous and represent the first demonstration of sub-1-bit post-training compression at this scale.

However, the paper evaluates accuracy only through the lens of pretraining validation loss — not on any downstream task. This is a deliberate choice, framed as being "similar to most works in the area of LLM quantization" (Section 5.1), and the out-of-distribution evaluations in Table 6 provide some reassurance that the loss degradation is not catastrophically worse on unseen domains. But a 6.7% loss increase on C4 does not directly translate to a known impact on, say, question answering, summarization, or reasoning tasks. The paper acknowledges this scope limitation in Section 7, noting that "it would also be interesting to further finetune a compressed model for specialized downstream tasks." From a practical deployment perspective, a user who wants to run c2048 for a specific application needs to know whether the compressed model still works for that application, and the paper provides only the pretraining loss signal — which is a reasonable proxy but not a substitute.

Additionally, the paper never characterizes where the accuracy degradation concentrates. Are certain experts more affected? Certain token positions? Certain input types? The aggregate loss number conceals potentially important heterogeneity. If the degradation is concentrated on rare but important linguistic phenomena (e.g., numerical reasoning, factual recall), the practical impact could be larger than the 6.7% loss increase suggests. The paper provides no layer-wise or expert-wise analysis of quantization error.

Claim 2: The compression can be performed on modest hardware (single GPU, under one day).

Table 8 directly supports this: c2048 compresses in 13.3–20.8 hours on a single A6000 GPU. This is genuinely impressive — compressing a 1.6T parameter model on a single consumer-grade GPU in less than a day removes a major barrier to entry. However, two qualifications are necessary.

First, the quoted times exclude the approximately 5 hours required to load the original model from disk (noted in the Table 8 caption: "simply (iteratively) loading the original 1.6T model into RAM takes close to 5 hours on our slow disk storage"). The total wall-clock time, including disk loading, is closer to 18–26 hours — still under a day and a half, but the "less than a day" framing in the paper's claim is slightly optimistic.

Second, the compression requires "a few 100GBs of (CPU) RAM" and over 3TB of disk storage. While a single server with 512GB or 1TB of RAM is far more accessible than a cluster of 65 GPUs, it is not trivial commodity hardware in the sense that a typical researcher's workstation might have 32–64GB of RAM. The paper is transparent about these requirements, but the "modest hardware" framing should be understood as "modest relative to what was previously required" (a hundred GPUs) rather than "modest in absolute terms."

Claim 3: The compressed model can be executed on affordable commodity GPUs (4× A6000 or 8× RTX 3090) with under 5% runtime overhead.

Figure 5 (Right) supports the <5% overhead claim for the measured configurations. The compressed c2048 runs on 4× A6000 or 8× RTX 3090 — both configurations that fit within a single server — whereas the uncompressed version would require 65+ or 130+ GPUs respectively.

The critical weakness in this evaluation is that the uncompressed baseline for c2048 is estimated, not measured. The paper states: "As actually running the bfloat16 version of the c2048 model would require >65 A6000 and >130 3090 GPUs (versus 4 and 8, respectively, for sub-1-bit compressed weights) we have to estimate its runtime. We do this by having all experts in a layer point to the same weight data (completely resolving memory issues), which allows us to collect timings with precisely the same overheads as for our compressed models." This estimation procedure eliminates the inter-GPU communication overhead that would exist in a real 65-GPU deployment — all-to-all data exchanges during expert routing, synchronization barriers, and network latency. The paper acknowledges this: "this is a highly optimistic estimate since real execution would require close to 20× more GPUs, with corresponding communication overheads, and our numbers should thus be viewed only as a lower bound."

This means the <5% overhead claim should be interpreted carefully. What the paper actually demonstrates is that the per-operator cost of decompressing and multiplying with compressed weights is comparable to or faster than the same operation with uncompressed weights on a single GPU — Figure 5 (Left) convincingly shows this. But the end-to-end system overhead relative to a deployable uncompressed baseline (which would require multi-node communication) is not measured and could be substantially lower (or even negative, if communication overhead dominates). The paper is transparent about this limitation, but it means Claim 3 is partially supported by measurement and partially by extrapolation.

A second concern: the evaluation uses a single prompt generating 128 tokens — an "individual user application" scenario. This is reasonable for interactive use, but does not characterize batch inference throughput, which is more relevant for many production deployments. The paper notes that the current kernel "naively executes a separate matrix-vector product for each token" when multiple tokens route to the same expert, and this could be optimized. In high-batch scenarios where many tokens map to the same expert, this inefficiency would compound, potentially pushing the overhead beyond 5%.

Claim 4: Compressibility increases with model scale (larger MoEs are more compressible).

The evidence in Table 4 (natural sparsity increases: 85.7% → 86.4% → 88.6%), Table 5 (relative loss increase decreases: 15.0% → 9.0% → 6.7% for ternary), and Table 7 (compression rate increases: 17.06× → 18.34× → 20.07× for MoE-only) consistently supports this claim across three different metrics. The trend is monotonic across all three model sizes and all three metrics. This is the most robustly supported claim in the paper.

However, the claim is only demonstrated for three data points within a single model family (SwitchTransformer). Whether the trend generalizes to other MoE architectures (e.g., GLaM, ST-MoE, SoftMoE) or to different expert counts at the same total parameter count is unknown. The paper's attribution of the trend to "weight distributions becoming closer to independent for larger layer sizes" is a post-hoc explanation, not a tested hypothesis — there is no experiment that isolates layer size from total parameter count or number of experts.

Missing Experiments That Would Strengthen the Paper

Comparison with 4-bit and 8-bit baselines on the same models. The paper's RTN baseline effectively simulates the simpler rounding methods used in prior MoE quantization work, but a direct measurement of 8-bit and 4-bit QMoE (with and without entropy coding) would establish the full compression-accuracy Pareto frontier. This would allow practitioners to make informed tradeoffs: is the jump from 4-bit to ternary worth the additional loss? The current results only show 2-bit and ternary.

Per-expert or per-layer analysis of quantization error. The paper reports aggregate loss but not how error distributes across experts. Are certain experts more sensitive to quantization? Do the most frequently activated experts degrade more or less than rarely-used ones? This would inform whether expert-specific compression rates (e.g., higher precision for important experts) could further improve the accuracy-compression tradeoff.

Downstream task evaluation beyond pretraining loss. Even a single representative downstream task (e.g., SuperGLUE, SQuAD, summarization) on the compressed c2048 would substantially strengthen the practical value proposition. The paper acknowledges this as future work (Section 7), but its absence limits confidence that the compressed model retains the capabilities that make the original valuable.

Comparison with an alternative entropy coding scheme. The paper argues that standard entropy codes (Huffman, arithmetic) are unsuitable for GPU decoding, but does not implement one to quantify the gap. A Huffman-coded baseline — even if slow — would provide an empirical upper bound on compression rate and quantify the actual compression-rate sacrifice made for GPU efficiency. The current comparison is only against the Shannon-theoretical limit, which no practical scheme can achieve.

Ablation of dictionary size. The dictionary has exactly 2^16 = 65,536 entries, selected to fit codewords in UINT16. The paper does not evaluate whether a smaller or larger dictionary would be better. A larger dictionary (e.g., 2^18 entries) might improve compression rate at the cost of larger storage and slower lookup; a smaller one might reduce cache pressure. This choice is presented as a design decision without empirical justification.

Impact of the fused kernel design. The paper does not compare the fused decompress-and-multiply kernel against an alternative where weights are fully decompressed first (e.g., into a bfloat16 buffer) and then multiplied using standard cuBLAS. This comparison would quantify the benefit of fusion specifically versus the simpler approach of decompressing once and reusing.

Conditional Boundaries of the Claims

The paper's claims hold most strongly under conditions that the paper itself identifies:

The model must have a high expert-to-dense parameter ratio. The 19.81× full-model compression rate for c2048 is achievable because "almost all parameters are located in the experts." For MoEs with fewer experts, larger dense components, or shared parameters across experts, the full-model compression rate would be lower because the dense parts remain uncompressed.

The base model must achieve non-trivial accuracy on the target task. The paper's compression preserves accuracy that already exists; it does not improve a broken model. If an MoE is poorly trained or fundamentally inadequate for a task, compression will not help — though this is less of a concern for the well-trained SwitchTransformer family.

The hardware must support the dictionary in L2 cache. The 512KB dictionary fits in the L2 cache of modern GPUs (A6000: 6MB L2; RTX 3090: 6MB L2). On GPUs with smaller L2 caches or when running many operations concurrently that compete for cache, dictionary lookup latency could increase.

The inference workload should not be dominated by batched expert execution. The current kernel's inefficiency when multiple tokens route to the same expert (separate matrix-vector products instead of batched matmul) means the <5% overhead claim is most applicable to single-token or small-batch inference. High-throughput batch processing could see larger overheads, though the paper suggests this is fixable.

The model must be available in a framework that supports the necessary routing interception. The paper's HuggingFace integration required "a handful of bugfixes" for the largest models and a custom modification to skip empty CUDA calls for experts receiving zero tokens. Applying QMoE to models in other frameworks (MeshTensorflow, T5X) would require reimplementing the routing and execution pipeline, which is non-trivial engineering.

6. Limitations and Trade-offs

The Difficulty Estimation Cost is Unaccounted For in the Headline Compression Pipeline—and It Dominates at Deployment

The assumption or constraint. The QMoE compression pipeline requires passing calibration data through the full uncompressed model to collect per-layer activation statistics (the Hessian matrices $X_\ell X_\ell^\top$ needed by GPTQ). For a 1.6T parameter model, this means the entire uncompressed weight set—all 3.2TB of bfloat16 parameters—must be loaded and processed, if only in a streaming, offloaded fashion. The system's memory-efficient design (Sections 3.2.1–3.2.3) makes this feasible on a single server, but it does not make it cheap. The model must be read from disk in its entirety (the paper notes that simply iteratively loading the 1.6T model takes close to 5 hours on their storage), and activations for 160K calibration tokens must be computed through every layer. This cost is incurred before any compression takes place and is entirely separate from the quantization and encoding stages that Table 8 times.

The consequence. In any deployment where the model is compressed once and then served many times (the standard production scenario), this one-time cost is amortized and becomes negligible. But for research workflows where models are frequently updated, fine-tuned, or experimented with, the calibration-data pass represents a substantial barrier: you must have the hardware to run the uncompressed model (even if slowly, via offloading) every time you want to produce a compressed version. More critically, for the scenario the paper envisions—democratizing access to large MoEs—the calibration pass requires already having access to the uncompressed model and enough CPU RAM plus disk bandwidth to process it. A researcher who receives only the compressed 160GB checkpoint cannot re-compress it with different quantization settings or calibration data; the compression process is not self-contained within the compressed artifact. This limits the downstream flexibility that the paper's "open-source and accessible" framing implies.

What evidence exists in the paper. The paper is transparent about the hardware requirements for compression (Section 5.1: "a few 100GBs of (CPU) RAM" and ">3 TB disk storage"), and Table 8 reports compression times of 13.3–20.8 hours for c2048. The paper also notes the ~5 hour disk-loading time in the Table 8 caption. However, the calibration-data forward pass is not broken out as a separate cost—it is bundled into the total compression time—and the paper does not quantify how much of the 16-hour default c2048 compression time is spent on the forward pass through the uncompressed model versus the actual quantization and encoding.

Mitigation status. Not addressed. The paper frames the compression time as acceptable ("less than a day on a single GPU") and does not discuss the cost of the calibration pass as a distinct limitation. A partial mitigation would be to reuse cached activation statistics if the same model is being recompressed with different quantization parameters or bitwidths, but this is not explored. The calibration-data volume itself is studied via the 0.5×/1.0×/2.0× ablation in Table 5, but only for its impact on final accuracy, not on the cost of data collection. Future work on predicting calibration statistics from a smaller proxy model, or on methods that require less calibration data for MoEs, would directly address this limitation.

The Approach Is Demonstrated on a Single Model Family with a Single Pretraining Objective and Evaluated Only on Pretraining Loss

The assumption or constraint. Every experiment in the paper uses the SwitchTransformer family (Fedus et al., 2022), pretrained on C4 with a masked-language-modeling (MLM) objective, and evaluated using C4 MLM validation loss. The paper explicitly acknowledges this scope constraint in Section 7: "Our study is confined to a limited set of models, as only very few massive and accurate MoEs are available publicy." The out-of-distribution evaluation in Table 6 (Arxiv, GitHub, StackExchange, Wikipedia) extends the evaluation beyond C4, but these are still text corpora evaluated under the MLM loss—not downstream task benchmarks like question answering, summarization, or reasoning tasks that would validate whether the compressed model retains the capabilities that make SwitchTransformer valuable.

The consequence. Three distinct generalization gaps remain unaddressed. First, the architectural gap: SwitchTransformers use a specific MoE design (top-1 routing, encoder-decoder architecture, ReLU-based experts). Whether QMoE's compressibility findings transfer to decoder-only MoEs (e.g., Mixtral-style models), to models with top-k routing (k > 1), to models with different expert architectures, or to Soft-MoEs with continuous routing (Puigcerver et al., 2023) is unknown. Second, the objective gap: MLM training involves predicting masked tokens, which the paper shows creates over-robustness to errors on mask tokens (Section 3.3, premasking experiment) and may produce weight distributions that are particularly amenable to the ternary sparsity pattern QMoE exploits. Autoregressive language models, which dominate current LLM deployment, do not have this property and may exhibit different compressibility characteristics. Third, the evaluation gap: a 6.7% increase in MLM validation loss (c2048 ternary, Table 5) does not directly map to a known degradation on practical tasks. The paper's finding that RTN ternary quantization does not cause "complete model collapse" (Section 5.2) sets only a floor on acceptable performance, not a ceiling on usable quality.

What evidence exists in the paper. The paper provides modest evidence for cross-domain robustness via Table 6, which shows that the loss degradation on Arxiv, GitHub, StackExchange, and Wikipedia is qualitatively similar to C4 (ternary losses of 1.42, 1.13, 1.22, 1.32 vs. 1.26 on C4). This suggests the compression does not catastrophically overfit to the calibration distribution. However, this is still an MLM loss evaluation, not a capabilities evaluation. The paper provides no evidence about whether the compressed c2048, when fine-tuned on a downstream task, would match the uncompressed model's performance. The authors acknowledge this gap in Section 7: "it would also be interesting to further finetune a compressed model for specialized downstream tasks, similar to QLoRA (Dettmers et al., 2023a)."

Mitigation status. Partially acknowledged, not addressed experimentally. The paper is transparent about its scope and names extension to other models and downstream evaluation as future work. The inclusion of out-of-distribution MLM loss in Table 6 is a step toward addressing the generalization concern but does not replace actual task evaluation. A practitioner considering QMoE for a specific application (say, using SwitchTransformer-c2048 as a base model for fine-tuning on a QA task) receives no direct evidence about expected performance.

The Sub-1-Bit Encoding Dictionary is Static and Optimized for a Specific Sparsity Distribution—It Does Not Adapt to Different Models or Compression Settings

The assumption or constraint. The dictionary-based entropy encoding scheme (Section 4.3, Algorithm 1) generates a single fixed dictionary of 65,536 codewords based on the probability distribution in Equation 2, with $p_0$ set to match the natural sparsity of a specific model after ternary quantization. The paper states that the dictionary is "optimized for c2048" (Section 5.2, Table 7 discussion) and that a "static dictionary works well enough, while simplifying memory efficient compression (see Section 3.2) as we do not have to collect statistics over many yet uncompressed experts." The compression procedure does not adapt the dictionary to each expert's individual sparsity pattern, to each layer's weight distribution, or even to each model's overall zero frequency—one dictionary serves all experts in all models.

The consequence. The compression rate achieved by QMoE is sensitive to the match between the dictionary's target distribution and the actual weight statistics. For c2048, with $p_0 = 0.886$ , the achieved compression rate is 20.07× (Table 7). For base128, with $p_0 = 0.857$ , the rate drops to 17.06×—a roughly 15% relative reduction in compression efficiency. If QMoE were applied to a model with substantially lower natural sparsity (e.g., a 50% zero rate rather than 88%), the dictionary would become mismatched: it would still contain many long zero-run sequences that rarely occur, while frequently-occurring patterns with more non-zeros would map to suboptimally short dictionary entries. In the limit, if the actual distribution diverges far enough from the dictionary's design point, the achieved bits-per-weight could exceed 1 bit, violating the paper's headline sub-1-bit claim. More practically, this means QMoE's compression performance is not portable across MoE architectures without some degradation, and the degradation is not characterized.

Additionally, the dictionary size (65,536 entries, 512KB) is fixed by the choice of 16-bit codewords. The paper does not explore whether this is the optimal trade-off point—a larger dictionary (requiring wider codewords and thus slower decoding or more cache pressure) might capture longer patterns and improve compression rate, while a smaller dictionary might reduce cache footprint and potentially improve decoding speed for models with simpler weight distributions.

What evidence exists in the paper. The compression rate degradation across model sizes in Table 7 (17.06× → 18.34× → 20.07× for base128 → large128 → c2048) directly demonstrates the sensitivity to sparsity distribution. The paper acknowledges that the dictionary is optimized for c2048 and that "compression rates increase with model size, which is for two reasons: (a) natural sparsity increases while our encoding dictionary is also optimized for c2048, and (b) weight distributions become closer to independent for larger layer sizes." This acknowledgment implicitly concedes that the dictionary is suboptimal for smaller models. The validation experiment comparing real c2048 compression (20.07×) against synthetic weights from the independent distribution (21.11×) quantifies the penalty for violating the independence assumption (~5%) but does not quantify the penalty for mismatch between the dictionary's $p_0$ and the model's actual sparsity.

Mitigation status. Not addressed. The paper does not discuss the possibility of model-specific or layer-specific dictionaries, which could improve compression rates for smaller models or for models with heterogeneous sparsity across layers. The design choice favoring a single static dictionary is motivated by simplicity and memory efficiency during compression—not having to aggregate statistics over many experts—but the trade-off is not analyzed. A simple mitigation would be to generate a per-model dictionary during compression (the encoding stage is offline and cost-insensitive), which would at least eliminate the model-scale mismatch visible in Table 7.

The Uncompressed Baseline for End-to-End Runtime Evaluation Is Idealized and Not Measured—Making the "Under 5% Overhead" Claim Partially Extrapolated

The assumption or constraint. The paper's central runtime claim—that compressed inference incurs under 5% overhead relative to uncompressed execution—relies on an idealized baseline for the largest and most important model (c2048). Section 5.3 states: "As actually running the bfloat16 version of the c2048 model would require >65 A6000 and >130 3090 GPUs (versus 4 and 8, respectively, for sub-1-bit compressed weights) we have to estimate its runtime. We do this by having all experts in a layer point to the same weight data (completely resolving memory issues), which allows us to collect timings with precisely the same overheads as for our compressed models." This estimation procedure eliminates all inter-GPU communication—the all-to-all token exchanges during expert routing that would occur in a real 65-GPU deployment, plus any synchronization overhead and network latency—and runs what is effectively a single-GPU execution (with duplicated weights) rather than a distributed one.

The consequence. The 5% overhead number is not directly falsified by the paper's evidence—the per-layer kernel measurements in Figure 5 (Left) convincingly show that the compressed matrix-vector product is faster than the uncompressed bfloat16 version on a single GPU, with speedups up to 35%. The question is whether this per-operator advantage translates to real distributed inference. In a 65-GPU deployment of the uncompressed model, the dominant cost might be communication (all-to-all token routing between devices, gradient-like exchanges during the forward pass), not computation. If communication dominates, then reducing per-operator computation time via compression might yield less than 5% end-to-end improvement—the bottleneck is elsewhere. Conversely, if communication is relatively light (because expert routing sends only small token batches between devices), the per-operator speedup might translate directly to end-to-end gains, and the compressed model could be faster than the 65-GPU uncompressed deployment, not just 5% slower.

The problem is that neither scenario is measured. The paper acknowledges this: "this is a highly optimistic estimate since real execution would require close to 20× more GPUs, with corresponding communication overheads, and our numbers should thus be viewed only as a lower bound." But a lower bound on the uncompressed runtime means the 5% overhead is an upper bound on the compressed model's advantage—the real gap could be much larger in the compressed model's favor. The claim "under 5% runtime overhead relative to ideal uncompressed inference" is therefore true relative to the idealized single-GPU baseline, but a practitioner deploying the uncompressed model would experience worse performance than this baseline, making the compressed model's relative advantage understated.

What evidence exists in the paper. Figure 5 (Right) shows the end-to-end measurements, with c2048 generating 128 tokens in approximately 5.5 seconds (compressed, 4× A6000) versus an estimated 5.3 seconds (idealized uncompressed). The per-layer measurements in Figure 5 (Left) provide the stronger, more direct evidence: compressed kernels achieve 0.70–0.98× the time of uncompressed bfloat16 across all matrix shapes on A6000, and 0.60–0.95× on RTX 3090. These per-layer numbers are measured, not estimated.

Mitigation status. Partially acknowledged. The paper is transparent about the estimation procedure and its limitations. However, the headline "less than 5% runtime overhead" framing does not carry this caveat visibly—it appears in the abstract and Section 1 without qualification. A more conservative claim would be: "compressed per-operator performance matches or exceeds uncompressed; end-to-end overhead relative to a single-GPU idealized baseline is <5%; overhead relative to a real multi-GPU deployment is likely negative (i.e., compressed execution is faster) but cannot be measured." The paper's discussion in Section 5.3 does separate the per-layer and end-to-end results and acknowledges the estimation issue, but the high-level claims elide the distinction.

The Revision/Verifier Model—and Therefore Any Accuracy Recovery—Is Not Part of the Compression Pipeline; What You Compress Is What You Get

The assumption or constraint. QMoE is a pure post-training compression framework: it takes a pretrained model, compresses it, and outputs a smaller model with (ideally minimal) accuracy degradation. There is no mechanism within the QMoE system for recovering accuracy lost during compression—no fine-tuning of the compressed model, no distillation from the original, no iterative refinement of the quantization decisions based on feedback from the compressed model's outputs. The paper evaluates accuracy purely as the loss increase between the uncompressed and compressed checkpoints (Table 5). Section 7 mentions that "it would also be interesting to further finetune a compressed model for specialized downstream tasks, similar to QLoRA," but this is positioned as future work, not as part of the QMoE framework.

The consequence. The 6.7% relative loss increase for c2048 ternary (Table 5) is a floor on achievable quality, not a ceiling—additional fine-tuning or distillation could potentially recover some of this degradation. But the paper provides no evidence about how much recovery is possible, how it would interact with the compressed format (can you fine-tune the dictionary-encoded weights, or must you decompress first?), or whether the compressed model's representational capacity (ternary weights with ~89% sparsity) is sufficient to match the original model's performance even with additional training. A practitioner who needs the compressed model to match the original's accuracy on a specific task receives no guidance about whether fine-tuning is sufficient to close the gap.

Moreover, the compression process is destructive: once weights are quantized to ternary values and encoded via the fixed dictionary, the original weight values are lost. There is no "decompressed bfloat16" version of the model that could be used as a starting point for standard fine-tuning; the compressed weights are the model. Any accuracy recovery would need to work directly with the ternary, dictionary-encoded representation—a constraint that standard fine-tuning methods (LoRA, full fine-tuning, distillation) are not designed to handle without modification.

What evidence exists in the paper. The paper provides one relevant negative result: the ReST[superscript EM] experiment mentioned in the prior sections (Appendix K, Figure 16) attempted to optimize a revision model and found that "additional sequential revisions substantially hurt performance," suggesting that naive fine-tuning of aggressively quantized models can backfire. This is a small piece of evidence that accuracy recovery is non-trivial, but it is specific to a different setting (revision models, not the SwitchTransformers compressed by QMoE). The paper provides no direct evidence about fine-tuning or distillation of QMoE-compressed SwitchTransformers.

Mitigation status. Acknowledged as future work (Section 7), not addressed experimentally. The paper's contribution is explicitly the compression pipeline and inference system, not a complete accuracy-preservation solution. This is a legitimate scoping decision—a paper can introduce a compression method without also solving fine-tuning—but it means the accuracy numbers in Table 5 represent a permanent quality reduction for the compressed model in the current system, not a starting point for recovery. Practitioners evaluating QMoE against alternatives that include accuracy-recovery mechanisms (e.g., quantization-aware training, which bakes compression into the training process) need to account for this gap.

The Current Kernel Falls Back to Inefficient Execution When Multiple Tokens Route to the Same Expert—Limiting Batch Inference Throughput

The assumption or constraint. The GPU decoding kernel described in Section 4.3.3 and Listing 1 is designed for the matrix-vector product case: one input vector (representing the hidden state of a single token) multiplied by one weight matrix (an expert's feedforward layer). This is the natural operation when each expert receives exactly one token—the standard case in top-1 routing during single-example inference. However, during batched inference or when the encoder routes multiple tokens to the same expert (which the paper notes does occur: "the encoder may sometimes route multiple tokens to the same expert"), the uncompressed baseline can perform a batched matrix-matrix multiplication, which is substantially more efficient than separate matrix-vector products due to better data reuse and higher arithmetic intensity.

The paper acknowledges this: "Our current implementation naively executes a separate matrix-vector product for each token, while the baseline performs a much more efficient joint matrix multiplication" (Section 5.3).

The consequence. The under-5% runtime overhead reported in Figure 5 (Right) applies to the single-token, single-prompt, generate-128-tokens scenario—a reasonable model of interactive use but not of batch inference. In a high-throughput serving scenario where many sequences are processed simultaneously, the probability of multiple tokens routing to the same expert increases, and the inefficiency of separate matrix-vector products would compound. The per-layer speedup shown in Figure 5 (Left) for single matrix-vector products might reverse: an uncompressed batched matmul can achieve much higher throughput than N separate matrix-vector products, and the compressed kernel's advantage from reduced memory traffic would diminish relative to the computational inefficiency of unbatched execution.

Additionally, because the kernel processes one weight-matrix row per warp, its utilization depends on the matrix having enough rows to fill the GPU's SMs. For the c2048 expert shapes (2080 × 6144 and 6144 × 2080), this is adequate. But for smaller experts or models with narrower feedforward dimensions, the number of rows might not provide enough parallelism, leaving SMs underutilized.

What evidence exists in the paper. The paper explicitly identifies this bottleneck in Section 5.3: "This slight slow-down despite faster per-layer timings is due to the fact that the encoder may sometimes route multiple tokens to the same expert." This is presented as an explanation for why the end-to-end overhead is <5% (slightly positive) rather than negative (faster) despite the per-layer speedups in Figure 5 (Left). The paper does not quantify how often multiple tokens route to the same expert, what batch sizes were used in the end-to-end measurements, or how the overhead scales with batch size.

Mitigation status. Partially addressed via suggested improvements, not implemented. The paper proposes two potential fixes: "one could easily introduce an inner loop over tokens into our kernel (Listing 1, line 30), or fully decompress first, followed by a standard matmul, for large token counts." The first approach would fuse the per-token loop into the kernel, allowing the weight matrix to be decoded once and multiplied against multiple input vectors, recovering the batched execution benefit. The second would trade off memory (decompressing the expert weights to a bfloat16 buffer) for computational efficiency (using standard batched cuBLAS). Neither is implemented or evaluated, so the current system's batch inference performance remains an open question.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a new point on the accuracy-compression Pareto frontier for large models — sub-1-bit post-training compression at scale — and demonstrates that it is achievable not through a breakthrough in quantization theory, but through a systems-level co-design of compression format and execution infrastructure. This represents a shift in how the field should think about extreme compression: the bottleneck is not algorithmic (the GPTQ quantizer was already known) but architectural — how you represent and decode the quantized weights determines whether extreme compression is practically useful.

The finding that massive MoEs are fundamentally more compressible than equivalently-sized dense models, and that this compressibility increases with scale (Table 4: 85.7% → 86.4% → 88.6% natural sparsity; Table 5: +15.0% → +9.0% → +6.7% relative loss increase for ternary), reframes MoEs from "powerful but hopelessly large" to "uniquely suited for extreme compression." This is a counterintuitive discovery that reverses the standard narrative: MoEs were seen as trading memory for compute efficiency, but QMoE demonstrates that the memory cost can be nearly eliminated through compression that exploits the very statistical structure that large-scale expert training produces. The implication is that MoE architectures should be evaluated not on their uncompressed memory footprint but on their compressibility — a model with 2048 experts may actually require less storage than a dense model of equivalent quality once both are compressed, because the MoE's weight distribution (high zero frequency from ternary quantization, near-independent values across large layers) is more amenable to entropy coding.

The paper also introduces a new design methodology for compressed inference that extends beyond this specific system. The co-design process — identifying GPU execution pathologies (sequential decoding dependencies, thread divergence, bit-level operation inefficiency, insufficient parallelism from small matrices) and then building a compression format that eliminates all of them simultaneously, accepting a controlled sacrifice in compression rate (~20% below the Shannon limit) — is a template for future compressed inference systems. The insight that a fixed-to-variable dictionary code can invert the standard entropy-coding paradigm to match GPU SIMT execution is specific to this paper, but the underlying principle — that compression formats must be designed backward from the hardware's parallel execution model, not forward from information theory — is broadly applicable. Prior work treated fast decoding as an optimization step applied after format design; this paper shows that format design is decoding optimization.

A less obvious but important reframing: the paper demonstrates that post-training compression can achieve rates previously associated only with training-intensive methods (quantization-aware training, pruning during training) when the target architecture has the right statistical properties. Kim et al. (2022a) required quantization-aware training to reach 2-bit on a 5B-parameter MoE; QMoE reaches sub-1-bit on a 1.6T-parameter MoE without touching the training pipeline. This opens the possibility that for sufficiently large and well-trained models, the line between post-training and training-aware compression blurs — the model's natural robustness to noise, combined with data-dependent correction, can substitute for explicit compression-aware optimization.

The work also reconciles a latent tension in the MoE deployment literature. Prior work on MoE systems (Barham et al., 2022; Gale et al., 2023; Hwang et al., 2023) focused on the communication bottleneck — how to efficiently route tokens between devices — under the implicit assumption that the memory bottleneck was unsolvable (you need enough total accelerator memory to hold all experts). QMoE eliminates the memory bottleneck for the weights themselves, which means future MoE systems research should focus on the routing and load-balancing aspects of distributed inference, not on the weight storage problem. This reorients the systems challenge: instead of "how do we shard 3.2TB of weights across 65 GPUs?", the question becomes "how do we efficiently route tokens through 160GB of weights that fit on 4 GPUs?" — a qualitatively different and simpler distributed systems problem.

Finally, by making the 1.6T-parameter SwitchTransformer-c2048 runnable on a single commodity server, the paper dramatically lowers the barrier to entry for MoE research. Before QMoE, studying the largest publicly-available MoE required access to infrastructure available to perhaps a dozen organizations worldwide. After QMoE, any research group with a server containing 4× A6000 or 8× RTX 3090 GPUs can load the compressed model and experiment — fine-tuning on downstream tasks, analyzing expert specialization, probing internal representations, or using the model as a baseline for new architectures. This is not a conceptual contribution but a practical one with significant implications for the rate of progress in MoE research.

Follow-Up Research This Work Enables

Replicating the compressibility-vs-scale relationship on other MoE families. The paper demonstrates that natural sparsity after ternary quantization increases monotonically with model size within the SwitchTransformer family, but this is three data points on one architecture. A strong follow-up would measure the same relationship on GLaM (Du et al., 2022), ST-MoE (Zoph et al., 2022), and decoder-only MoEs like Mixtral, testing whether the trend is universal or specific to SwitchTransformer's training recipe (token dropping, high dropout, ReLU experts). The concrete experiment: take a series of MoE models at different scales within a single family, apply the QMoE pipeline (ternary GPTQ + dictionary encoding), and measure natural sparsity, compression rate, and relative loss increase as a function of parameter count and expert count independently. The key question is whether compressibility is driven by total parameters, by expert count, by expert size, or by some interaction — the paper attributes it partly to "larger layer sizes" making weight distributions closer to independent, but this has never been isolated. If the trend generalizes, it becomes a scaling law for MoE compressibility; if it doesn't, it reveals something specific about SwitchTransformer training.

Adaptive per-expert or per-layer dictionaries instead of one static dictionary. Table 7 shows a clear compression rate penalty for smaller models (17.06× for base128 vs. 20.07× for c2048) because the dictionary is optimized for c2048's sparsity distribution. A direct extension would generate a per-model dictionary during compression (trivially cheap since encoding is offline) and measure the compression rate gain for base128 and large128. Beyond per-model, one could generate per-layer dictionaries or even per-expert dictionaries: do different experts have meaningfully different sparsity patterns that would justify the additional dictionary storage? A researcher would: (1) compute the zero frequency for each expert in a compressed model, (2) cluster experts by their sparsity distribution, (3) generate cluster-specific dictionaries, (4) measure the compression rate improvement vs. the overhead of storing multiple dictionaries. The hypothesis (suggested by the paper's observation that MoE training involves "routing instabilities" and "token dropping") is that expert sparsity may be heterogeneous, with frequently-activated experts developing different weight distributions than rarely-activated ones. If the gain is substantial, adaptive dictionaries become a standard component of the pipeline; if negligible, the static dictionary is validated as sufficient.

Fine-tuning compressed models with LoRA or QLoRA on downstream tasks. The paper explicitly flags this as future work (Section 7) and cites the finding from Zoph et al. (2022) that "finetuning only non-expert layers" can be effective. A natural experiment: take QMoE-compressed c2048 (ternary, 158.6GB), attach LoRA adapters to the non-expert (dense) layers, and fine-tune on a standard benchmark suite (SuperGLUE, SQuAD, CNN/DailyMail summarization). Compare against: (a) the uncompressed model fine-tuned with the same LoRA setup (requiring 65+ GPUs, so only feasible for well-resourced groups), (b) the compressed model without fine-tuning, and (c) a 4-bit QMoE-compressed version without entropy coding (to isolate whether the additional loss from ternary vs. 4-bit can be recovered by fine-tuning). The key measurement is whether fine-tuning closes the 6.7% loss gap between compressed and uncompressed — and if so, whether the ternary weight representation (only three values per weight, ~89% zeros) has sufficient capacity to match the full-precision model's downstream performance, or whether it creates a ceiling that no amount of adapter training can surpass.

Combining QMoE-style entropy coding with other quantizers and other model types. The two-stage architecture (quantization creates structure, entropy coding exploits it) is independent of the specific quantizer. A researcher could pair QMoE's dictionary-based encoding with a different quantization method — for instance, QuIP (Chee et al., 2023) with its incoherence preprocessing, or SpQR (Dettmers et al., 2023b) with its outlier handling — and measure whether the combination achieves sub-1-bit on dense models. The paper demonstrates that the gap between naive storage and entropy-coded storage is largest when the quantization creates high sparsity (ternary yields ~89% zeros; 4-bit quantization yields much lower zero frequency), so the two-stage approach is most impactful at extreme bitwidths. But even at 3-bit or 4-bit, there may be exploitable structure (e.g., some quantized values occurring much more frequently than others) that the dictionary code could capture, yielding incremental compression beyond the quantizer alone. The concrete experiment: quantize a dense 70B-parameter model (e.g., Llama-2) to 3-bit and 4-bit with GPTQ, then apply QMoE's entropy coding stage (with a dictionary generated from the empirical distribution of the quantized values), and measure the additional compression. The negative result — "entropy coding provides negligible gains above the naive bitwidth for non-ternary quantization" — would establish the boundary condition for when the two-stage approach is worthwhile.

Extending the kernel to batched matrix-matrix products for high-throughput inference. Section 5.3 acknowledges that the current kernel "naively executes a separate matrix-vector product for each token" when multiple tokens route to the same expert, losing the batching benefit of standard matmul. A systems follow-up would implement one of the two suggested fixes: either (a) add an inner loop over tokens to the fused decompress-and-multiply kernel, allowing the weight matrix to be decoded once and applied to a batch of input vectors, or (b) implement a two-pass approach where weights are first decompressed into a bfloat16 buffer (amortized over the batch), then multiplied via standard batched cuBLAS. The evaluation would sweep batch sizes (1, 4, 16, 64) and measure throughput (tokens/second) for the compressed c2048 model versus an idealized uncompressed baseline at each batch size. The hypothesis is that the batched kernel would eliminate or reverse the <5% overhead seen at batch size 1, potentially making compressed inference faster than uncompressed at realistic serving batch sizes. This would shift QMoE from "compression with minimal overhead" to "compression with throughput gains" — a much stronger value proposition for production deployment.

Measuring expert-level sensitivity to quantization as a guide for mixed-precision compression. The paper reports only aggregate loss and does not analyze which experts or layers suffer most from quantization. A researcher could: (1) after QMoE ternary compression of c2048, measure the per-expert output error (difference between compressed expert output and original expert output on a held-out set of tokens), (2) correlate this error with expert properties — activation frequency, layer depth, weight norm, sparsity level — and (3) test whether a mixed-precision scheme (ternary for most experts, 2-bit or 3-bit for the most sensitive ones) improves the accuracy-compression tradeoff. The paper's expert grouping infrastructure (Section 3.2.4) already supports heterogeneous compression parameters across groups, making this experiment mechanically straightforward. The finding could establish a "sensitivity hierarchy" for MoE experts analogous to what the quantization literature has established for dense model layers (where earlier and later layers are typically more sensitive), potentially yielding further accuracy gains at equivalent or better compression rates.

Practical Applications and Downstream Use Cases

Democratized research on the largest publicly-available models. Before QMoE, running the 1.6T-parameter SwitchTransformer-c2048 required an estimated 65+ NVIDIA A6000 GPUs — infrastructure accessible to perhaps a dozen industry labs and well-funded academic groups worldwide. QMoE reduces this to a single server with 4× A6000 or 8× RTX 3090 GPUs and the compressed 158.6GB checkpoint. This is not an incremental improvement in accessibility; it is a qualitative threshold crossing. A university lab with a ~$30,000 GPU server can now load, study, and experiment with a trillion-parameter model — probing expert specialization, analyzing routing patterns, using it as a feature extractor, or fine-tuning on domain-specific data. For the many research groups whose work on large language models has been constrained to studying models small enough to fit on their hardware, QMoE removes the memory barrier for the largest openly-available model, enabling comparative studies across the full scale range from 7B to 1.6T parameters on a single machine.

Cost-effective deployment of MoE-based services where model quality is paramount and the problem distribution includes many easy-to-medium queries. The paper's runtime results show that compressed c2048 runs on 4× A6000 GPUs with under 5% overhead relative to idealized uncompressed execution, and the per-layer kernels are actually faster than bfloat16 on all measured matrix shapes (Figure 5, Left). For a small company or research deployment serving an MoE-based application — code completion, document analysis, scientific QA — this means the hardware cost is reduced from a cluster of 65+ datacenter GPUs (hundreds of thousands of dollars, plus operational costs) to a single server with 4 consumer or prosumer GPUs (~ $20,000–$ 40,000). The 6.7% validation loss increase from ternary compression (Table 5) represents a quality trade-off, but one that may be acceptable for many applications — and for use cases where the original bfloat16 model was simply impossible to deploy, the compressed model provides access to a quality tier that was previously unreachable at any budget. The paper does not evaluate downstream task performance, so practitioners would need to validate on their specific task, but the pretraining loss signal suggests the degradation is modest and uniform across domains (Table 6).

Enabling MoE-based self-improvement pipelines on modest hardware. The paper's finding that QMoE compression runs on a single GPU in under a day opens the possibility of iterative workflows: compress a large MoE, fine-tune the compressed model on a specific task (Section 7's proposed future work), and then re-compress or further optimize. This is infeasible with the uncompressed model because even loading it requires 3.2TB of storage and hours of disk I/O. With the compressed 158.6GB checkpoint, the model can be loaded, modified, and saved on commodity hardware, making iterative experimentation practical. For research on self-improving systems (where a model generates training data, is fine-tuned on that data, and then re-evaluated), the QMoE-compressed model provides a starting point that is large enough to be interesting (1.6T parameters) but small enough to be manipulated in a standard research environment.

When to Prefer This Method

The paper does not explicitly position QMoE against a named alternative with a clear set of decision criteria — it is presented as the first system to achieve sub-1-bit compressed inference for trillion-parameter MoEs on commodity hardware, not as one option among several in a mature trade-off space. The method it implicitly competes against is "deploy the uncompressed model on a large GPU cluster." The decision rule that follows from the paper's evidence is:

Prefer QMoE compression when the uncompressed model's memory requirements exceed available hardware (as is true for c2048 on any single server), and a modest accuracy degradation (6.7% relative validation loss increase for ternary c2048, less for 2-bit) is acceptable for the target application. This covers essentially all non-industrial deployment scenarios for trillion-parameter models and many industrial ones where hardware cost reduction outweighs the accuracy trade-off.
Prefer uncompressed execution (if hardware allows) when the task demands the highest possible accuracy and any loss degradation is unacceptable, or when the problem distribution consists almost entirely of queries that the uncompressed model gets right but the compressed model gets wrong — though the paper provides no task-level evaluation to identify whether such queries exist. The out-of-distribution evaluations in Table 6 suggest the degradation is uniform rather than concentrated on specific input types, but this has not been verified on downstream tasks.
The paper does not compare against alternative compression methods at the same bitwidth (e.g., QuIP-based ternary quantization, or a Huffman-coded version of the same ternary weights), so there is no empirical basis for choosing QMoE's specific encoding scheme over another sub-1-bit format. The paper's contribution is demonstrating that sub-1-bit is achievable at all with acceptable accuracy and fast decoding, not that its particular dictionary design is Pareto-optimal. A practitioner considering QMoE today would adopt the entire pipeline as described, with the dictionary optimized for c2048-like models, until follow-up work establishes whether adaptive or alternative encoding schemes improve the trade-off.