GLaM: Efficient Scaling of Language Models with Mixture-of-Experts

🎯 Pitch

A 1.2-trillion-parameter language model trains using only a third of GPT-3’s energy and runs at half the FLOPs per token, yet matches or beats it across 29 NLP benchmarks. The trick? Each input token activates just 8% of the massive network through sparse mixture-of-experts routing, turning inference efficiency into a scaling law advantage.

1. Executive Summary

This paper introduces and evaluates GLaM (Generalist Language Model), a family of decoder-only language models that uses a sparsely activated mixture-of-experts (MoE) architecture to scale model capacity while constraining computational cost. The largest variant, GLaM (64B/64E), contains 1.2 trillion total parameters but only activates 96.6 billion per input token — roughly 8% of the total — by using a learned gating network to dynamically select two experts per token from 64 available experts at every other Transformer layer. Evaluated on 29 NLP benchmarks in the zero-shot, one-shot, and few-shot settings, GLaM (64B/64E) outperforms GPT-3 (175B) on average across 6 of 7 task categories while consuming approximately half the FLOPs per inference token (180 vs. 350 GFLOPs) and roughly one-third the training energy (456 vs. 1,287 MWh, or as low as 213 MWh when trained to GPT-3–matching accuracy after 280B tokens — about 1/6 of GPT-3's energy). Against a dense GLaM variant (137B) with comparable per-token FLOPs, the MoE model demonstrates consistently stronger scaling behavior and superior data efficiency, establishing that sparsely activated architectures can provide meaningful performance advantages over equivalently-sized dense models at the same compute budget, particularly on knowledge-intensive tasks such as open-domain question answering.

2. Context and Motivation

The Core Problem: Dense Scaling Is Hitting an Economic and Environmental Wall

The fundamental tension this paper confronts is straightforward but consequential: language models keep getting better as they get bigger, but the cost of making them bigger is becoming untenable. The scaling laws literature (Kaplan et al., 2020) had established a predictable relationship — more parameters, more data, and more compute reliably improve perplexity and downstream task performance. GPT-3 (Brown et al., 2020) was the landmark demonstration at 175 billion parameters, showing that few-shot in-context learning emerges as a powerful capability at scale. But the very success of this paradigm creates its own crisis.

The paper's motivation can be understood through a set of interlocking constraints that were becoming acute at the time of its writing:

1. The FLOPs-per-token problem. In a standard dense Transformer, every parameter participates in every forward pass. A 175B-parameter model requires approximately 350 GFLOPs per token for inference. If you want to scale to 530B (Megatron-NLG) or 280B (Gopher), the per-token cost rises proportionally. This means that even if you can afford to train these models — which requires thousands of specialized accelerators running for weeks or months — deploying them at scale becomes prohibitively expensive. A model that costs 2× more FLOPs per token effectively costs 2× more for every inference request, and large language models can serve billions of tokens per day in production.

2. The energy and carbon crisis. Patterson et al. (2021) had just documented the environmental impact of large neural network training, with GPT-3's training alone consuming approximately 1,287 MWh and emitting an estimated 552 net tCO₂e. As the field contemplated models of 500B, 1T, or even 10T parameters, the energy trajectory was alarming. The paper positions energy efficiency as a first-order concern, not an afterthought — Table 1 prominently features training energy alongside accuracy metrics, making explicit that future scaling must account for resource consumption.

3. The inference latency barrier. Even ignoring energy, the serial nature of autoregressive decoding means that per-token FLOPs translate directly into wall-clock time. A model that requires 400ms to generate a token is impractical for interactive applications regardless of how many accelerators are available. Dense scaling makes this worse with every generation of models.

Prior Approaches and Their Limitations

The paper identifies three broad strategies that were being pursued to address these scaling challenges, each with significant shortcomings:

Dense scaling with optimized infrastructure. The dominant approach — exemplified by GPT-3 (175B), Jurassic-1 (178B), Gopher (280B), and Megatron-NLG (530B) — was to simply build larger dense models while improving training parallelism through model sharding (Shoeybi et al., 2019; Huang et al., 2019) and pipeline parallelism (GPipe). Table 2 catalogues these models. This strategy had proven effective for improving benchmark scores, but it offered no solution to the fundamental problem that the inference cost scales linearly with total parameter count. A dense 530B model costs 3× more per token than a 175B model, full stop. The paper's framing is clear: without a change in architecture, the economic and environmental costs of continued scaling would eventually overwhelm the benefits.

Encoder-decoder sparsely activated models. The mixture-of-experts architecture itself was not new. Shazeer et al. (2017) had introduced sparsely-gated MoE layers for language modeling and machine translation, demonstrating that models could have far more total parameters than activated parameters per input. GShard (Lepikhin et al., 2021) scaled this to 600B parameters in an encoder-decoder Transformer for translation, using automatic sharding to distribute experts across devices. Switch Transformers (Fedus et al., 2021) pushed further to 1.5 trillion parameters (Switch-C) with an encoder-decoder architecture, selecting only a single expert per token.

However, these prior MoE efforts had critical limitations that GLaM was designed to address:

Model type mismatch for the dominant paradigm. Switch-C and GShard used encoder-decoder architectures (like T5) evaluated primarily in the fine-tuning setting — you pre-train, then fine-tune on each downstream task. But the most influential demonstration of language model capability at the time (GPT-3) had shown that decoder-only models in the few-shot in-context learning setting could achieve remarkable generalization without any gradient updates. The paper explicitly notes this gap in the related work section:

"Switch-C is mainly evaluated on fine-tuning benchmarks, e.g., SuperGlue, while GLaM performs well without any need for fine-tuning in the few-shot setting shared by GPT-3 where SuperGlue is a subset."

This is a crucial distinction. The research community and industry were rapidly converging on few-shot prompting as the preferred deployment paradigm — no task-specific training, no model copies per task, just a single model that adapts via natural language instructions. Demonstrating that MoE could work in this setting had not been done convincingly at scale.
Expert size and parallelism design. The paper makes a subtle but consequential architectural choice that it alludes to in Section 6.1: GLaM uses much larger experts than Switch-C, exceeding the capacity of a single TPU core. This forces each expert to be partitioned across multiple devices (using the 2D sharding described in Appendix C), which the authors argue contributes to the model's strong knowledge performance. Our experiments reveal that "we should grow the size of the experts to get high quality models" (Section C) — a design principle that prior MoE work had not systematically established.

Mixture-of-experts for decoder-only models in the few-shot setting. No prior work had convincingly demonstrated that a sparsely activated decoder-only language model could match or exceed dense models on few-shot in-context learning benchmarks at the scale of GPT-3. There were plausible reasons to doubt whether it would work. The expert gating network introduces additional training complexity — you need an auxiliary load-balancing loss (from GShard) to prevent all tokens routing to the same subset of experts, and the gating decisions create non-differentiable discrete choices that can make training unstable. The authors' own training stability measures (Section 5.2) — skipping NaN/Inf batches, restarting from healthy checkpoints — hint at the practical difficulties. Whether the increased total capacity would translate to better few-shot performance at equal per-token cost was an open empirical question.

How This Paper Positions Itself

The paper occupies a specific and carefully constructed position in the scaling landscape:

It is not primarily a novel architecture paper. The MoE architecture is adapted directly from GShard and Switch Transformers — interleaving MoE layers with standard dense Transformer layers, using a top-2 gating function with softmax, applying a load-balancing auxiliary loss. The additional architectural modifications (Gated Linear Units in feed-forward layers, per-layer relative positional bias from Transformer-XL) are described in a single sentence in Section 4. The innovation is not in the individual components but in the demonstration that this combination works at scale for few-shot decoder-only language models, and the systematic comparison showing it is more efficient than the dense alternative.

It is a scaling efficiency paper, framed against GPT-3 as the canonical reference point. GPT-3 serves as more than a benchmark — it is the explicit foil that GLaM is designed to outperform in the metrics that matter for practical deployment: accuracy, FLOPs per inference token, and training energy. Table 1 literally puts the two models side by side with relative costs and accuracy deltas. The paper's central claim — "GLaM outperforms GPT-3 across 21 NLU and 8 NLG benchmarks in average while using about half the FLOPs per token during inference and consuming about one third the energy for training" — is framed as an apples-to-apples comparison on the same evaluation protocol (29 public NLP tasks from Brown et al., 2020) using the same few-shot format.

It is responding to a specific gap in the evidence base. The paper explicitly notes that "MoE-based sparse models are not yet common in the NLP community" (Section 1) and frames its contribution as showing "for the first time within the few-shot in-context learning setting at scale" that sparse decoder-only models can outperform dense architectures of similar compute FLOPs. The phrase "for the first time" is doing real work here — it stakes a claim that prior MoE work, while impressive, had not credibly established this result in the evaluation paradigm that mattered most to the field.

It makes data quality a central concern, not an afterthought. A significant portion of the paper (Section 6.2, Figure 3c–d) is devoted to demonstrating that filtering training data with a quality classifier yields consistent improvements over unfiltered data, particularly for natural language generation tasks. This is an important positioning choice: the paper is arguing that both architecture and data quality matter, and that one cannot substitute for the other. In an era where many scaling papers focused almost exclusively on model size and architecture, GLaM's explicit data quality ablation — showing that the effect of filtering is "bigger on NLG than on NLU" — establishes data curation as a first-order contributor to the model's performance claims.

The paper implicitly addresses a concern about sparsity and fairness. Section 7 includes a striking result on WinoGender: GLaM achieves "for the first time, to our knowledge" near-identical accuracy between stereotypical and anti-stereotypical examples (both 71.7%) and between male and female pronoun examples (70.8% vs. 72.5%). This is positioned as evidence that large sparsely activated models "may rely less on superficial statistical correlations" — a tentative but suggestive claim that MoE architectures might have beneficial properties for fairness beyond their computational advantages. The paper does not overclaim here (the language is appropriately cautious), but the inclusion of this result signals an awareness that the community was increasingly scrutinizing large models for social biases.

What Was at Stake

To understand why this paper matters, it helps to imagine the state of the field in late 2021. The largest dense models (Megatron-NLG at 530B, Gopher at 280B) had been trained at enormous expense — these were multi-million-dollar projects requiring coordination across large hardware clusters. The trajectory was clear: the next round of scaling would push toward trillion-parameter dense models, with astronomical costs that only a handful of organizations worldwide could sustain. The question was whether this was necessary — whether you needed to pay the full dense cost for every token, or whether conditional computation could break the linear relationship between total capacity and per-token cost.

GLaM's answer — demonstrated empirically, not just hypothesized — was that you could have the benefits of a 1.2 trillion parameter model (better few-shot performance, greater knowledge capacity) at roughly half the inference FLOPs of a 175B dense model. This was a genuinely consequential finding for the field's trajectory. It suggested that the scaling roadmap did not inevitably lead to models that were prohibitively expensive to serve, and that sparsity was not merely a theoretical curiosity but a practical path to continuing performance improvements.

The paper's conclusion — "MoE should therefore be considered as a strong candidate for future scaling" — may read as modest today, when MoE architectures are widely deployed. But in 2022, when the dominant public models were all dense, this was a substantive claim backed by the most comprehensive empirical comparison available at the time. The paper's lasting contribution is less any individual technical innovation and more the systematic case it built that sparse architectures are a practical, efficient, and performant alternative to the dense scaling paradigm that had previously seemed inevitable.

3. Technical Approach

3.1 Reader Orientation (Approachable Technical Breakdown)

GLaM is a family of decoder-only Transformer language models that use a sparsely activated mixture-of-experts (MoE) design — the model has many more total parameters than it actually uses for any given input, because a learned routing mechanism decides which small subset of the model to activate on a per-token basis. The system solves the problem that dense scaling of language models demands linearly increasing compute per token as you add parameters: by replacing every other Transformer feed-forward layer with a sparsely-gated collection of expert networks, GLaM increases total model capacity (to store more knowledge and patterns) while keeping the per-token computation cost roughly equivalent to a much smaller dense model, since only a fraction of the experts are activated for each token.

3.2 Big-Picture Architecture (Diagram in Words)

The GLaM architecture interleaves two types of Transformer layers in a decoder-only stack:

Standard Dense Transformer Layers — these contain multi-head self-attention followed by a feed-forward network (FFN) with a Gated Linear Unit (GLU) activation. Every parameter in these layers is activated for every input token.
Mixture-of-Experts (MoE) Layers — these replace the FFN with a collection of E independent expert feed-forward networks (64 experts in the largest model) plus a learned gating network. For each input token, the gating network computes a probability distribution over all experts and selects the top-2 most relevant ones. Only those two experts compute outputs, which are then combined via a weighted average.

The layers alternate: one standard Transformer layer, then one MoE layer, then one standard Transformer layer, and so on — "every other Transformer layer" (Section 4). This interleaving means that approximately half of the model's layers use the sparse activation pattern, keeping the average per-token FLOPs low while dramatically increasing the total parameter count stored in the expert bank.

The information flow for a single token at an MoE layer works as follows: the token's hidden representation (output from the previous layer's self-attention) enters the gating network → the gating network computes unnormalized scores for all E experts via a learned linear transformation → a softmax converts these scores to a probability distribution → the top-2 highest-probability experts are selected → each selected expert independently transforms the token's representation through its own feed-forward network → the two expert outputs are combined as a weighted sum (weighted by the softmax probabilities, renormalized over the two selected experts) → the combined representation passes to the next layer.

3.3 Roadmap for the Deep Dive

First, the formal architecture of a single MoE layer — the gating function, expert selection, and output combination — since this is the core mechanism that enables capacity scaling without proportional compute scaling.
Second, the modifications to the standard Transformer that GLaM adopts (Gated Linear Units, relative positional bias), because these interact with the MoE design decisions.
Third, the complete model configurations at different scales (from 130M to 1.2T parameters), establishing the scaling family and the notation conventions used throughout the paper.
Fourth, the training procedure and hyperparameters — optimizer choice, learning rate schedule, auxiliary loss coefficient, batch packing strategy — since training trillion-parameter MoE models requires specific stabilisation techniques.
Fifth, the model partitioning and parallelism strategy (2D sharding), because efficiently distributing experts across a physical device mesh is critical to the paper's energy efficiency claims.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical scaling paper whose core technical idea is that sparsely activated mixture-of-experts layers can be integrated into a decoder-only Transformer in a way that substantially reduces training and inference cost while improving performance on few-shot NLP benchmarks, and that the key design choices are: (1) interleaving MoE layers with dense layers at every other position, (2) using top-2 expert selection per token, (3) using large experts that exceed single-device capacity, and (4) applying a load-balancing auxiliary loss to ensure all experts are utilized.

MoE Layer Architecture: Gating, Expert Selection, and Output Combination

Each MoE layer in GLaM replaces the standard Transformer feed-forward sub-layer with a collection of E independent feed-forward networks (called "experts") and a learned gating network that decides which experts process each input token. This is adapted directly from GShard (Lepikhin et al., 2021) and Switch Transformers (Fedus et al., 2021), with the specific choice of selecting exactly two experts per token.

Let x be the input token representation (a vector of dimension M, the model dimension) arriving at the MoE layer. The gating network computes:

$p = \text{softmax}(W_g \cdot x)$

where W_g is a learned weight matrix of shape (E, M) mapping from the model dimension to the number of experts, and p is a probability distribution over the E experts — a vector of length E with non-negative entries summing to 1.

What it computes: the gating network takes the token's current hidden representation and produces a score for each expert indicating how well-suited that expert is to processing this particular token. The softmax normalizes these raw scores into a probability distribution, making them interpretable as relative expert affinities.

Why this form: using a softmax yields a smooth, differentiable probability distribution that allows gradient-based learning of the routing function. Alternative gating mechanisms (hard thresholding, reinforcement learning-based selection as used in Shazeer et al., 2017) introduce non-differentiable stochasticity that complicates training. The softmax gating keeps the entire system end-to-end differentiable (the discrete selection of top-2 experts is non-differentiable, but the gradient flows through the selected experts and their gating weights).

The actual expert selection operates on p:

$\text{selected\_experts} = \text{top-2}(p)$

where top-2 returns the indices of the two largest values in p. Let p_1 and p_2 be the probabilities assigned to these two experts, and let f_1(x) and f_2(x) be the output vectors from each expert's feed-forward network applied to input x. The final output of the MoE layer is:

$y = \frac{p_1 \cdot f_1(x) + p_2 \cdot f_2(x)}{p_1 + p_2}$

where the denominator renormalizes the weights to sum to 1 after truncating to the top-2 experts.

What it computes: this weighted sum combines the outputs of the two most relevant experts, with each expert's contribution proportional to its gating probability (renormalized so the two weights sum to 1). The result y is a vector of dimension M that replaces what would have been the output of a single dense FFN in a standard Transformer.

Why this form: selecting exactly two experts (rather than one, as in Switch Transformers, or a variable number as in earlier MoE work) is described as a "trade-off between predictive performance and the training/serving efficiency of the model" (Section 4). Using one expert per token (like Switch) reduces compute but loses the ability to compose knowledge from multiple experts — the weighted combination of two experts enables the model to blend different types of processing for a single token. Using more than two would increase per-token FLOPs, pushing the model closer to dense behavior. The paper's notation O(E²) in Section 4 refers to the combinatorial expressivity: with two experts selected from E, the model can blend E² different pairs, providing substantially more representational flexibility than the equivalent dense FFN.

Each expert is itself a standard feed-forward network. The paper uses a configuration where each expert's hidden dimension H is set such that the two experts combined have roughly the same total FLOPs as the dense FFN they replace. In GLaM (64B/64E), each expert has shape [M, H] where M = 8192 (model dimension) and H = 32768 (expert hidden dimension), while the dense counterpart GLaM (137B) uses H = 65536 in its single FFN. Since two experts each with H = 32768 produce 2 × 8192 × 32768 = 537M multiply-adds (roughly equivalent to one FFN of 8192 × 65536 = 537M multiply-adds), the per-token FLOPs are approximately matched.

Expert Load Balancing: The Auxiliary Loss

A well-known failure mode of MoE models is expert collapse — the gating network learns to route all tokens to a small subset of experts, leaving most experts unused and effectively wasting the model's additional capacity. To prevent this, GLaM uses the auxiliary load-balancing loss from GShard (Lepikhin et al., 2021):

$\mathcal{L}_{\text{aux}} = \alpha \cdot \sum_{e=1}^{E} f_e \cdot g_e$

where:

f_e is the fraction of tokens in the batch that are routed to expert e (the empirical dispatch frequency),
g_e is the average gating probability assigned to expert e across all tokens in the batch,
α is a coefficient set to 0.01 (Section 5.2).

What it computes: for each expert, the term f_e · g_e measures how much that expert is used relative to how much the gating network "wants" to use it. The sum across experts is minimized when all experts receive tokens in proportion to their average gating probabilities — that is, when the gating distribution is uniform and the empirical dispatch matches it. The coefficient α = 0.01 scales this auxiliary signal so it influences training without overwhelming the primary language modeling objective (cross-entropy on next-token prediction).

Why this form: the product f_e · g_e has a specific property: if the gating network assigns high probability to expert e (g_e is large) but few tokens are actually routed there (f_e is small due to competition from other experts with even higher probabilities), the term is small — which is the wrong direction for balancing. If many tokens are routed to an expert (f_e large) but with low probability (g_e small, meaning the expert was a reluctant second choice), the term is also small. The loss therefore penalizes the combination of high affinity AND low usage, and high usage AND low affinity — it encourages the columns of the gating matrix to be uniform and the dispatch to match. The coefficient α = 0.01 was chosen based on prior work (GShard) and not swept independently — given the cost of training trillion-parameter models, the paper explicitly notes "There is little room for hyperparameter tuning" (Section 5.2).

This auxiliary loss is added to the standard autoregressive language modeling loss:

$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{LM}} + \mathcal{L}_{\text{aux}}$

where L_LM is the cross-entropy between the model's predicted next-token distribution and the true next token.

Non-MoE Architectural Modifications

GLaM makes two modifications to the standard Transformer architecture that apply to all layers (both dense and MoE):

1. Gated Linear Unit (GLU) in Dense FFN Layers

In the non-MoE Transformer feed-forward sub-layers (the standard dense layers interleaved with MoE layers), GLaM replaces the standard two-layer FFN (linear projection → activation → linear projection) with a Gated Linear Unit (Dauphin et al., 2017; Shazeer, 2020):

$\text{FFN}_{\text{GLU}}(x) = (W_1 x + b_1) \odot \sigma_{\text{GELU}}(W_2 x + b_2)$

where ⊙ denotes element-wise multiplication, W_1 x + b_1 is one linear transformation of the input, W_2 x + b_2 is a second linear transformation that passes through a GELU activation function, and the result of the first transformation is gated (multiplied element-wise) by the activated second transformation.

What it computes: instead of a simple linear → nonlinearity → linear pipeline, the GLU computes two independent linear projections, applies a nonlinearity to one of them, and then gates the two results together. The GELU-activated projection acts as a learned "gate" that controls which dimensions of the linear projection pass through — if the GELU output is near zero for a dimension, that dimension is suppressed; if near one, it passes through unchanged (approximately).

Why this form: Shazeer (2020) showed that GLU variants consistently outperform standard ReLU or GELU activations in Transformer feed-forward layers, likely because the multiplicative gating provides a more flexible form of nonlinearity — it can implement both additive transformations (when the gate is nonzero) and pure suppression (when the gate is zero) without requiring the activation function alone to encode complex multiplicative relationships. The paper includes this modification with minimal discussion (a single sentence in Section 4), suggesting it was adopted as a known improvement rather than a contribution of this work.

2. Per-Layer Relative Positional Bias

GLaM replaces the standard absolute positional embeddings (sinusoidal or learned position encodings added to token embeddings) with per-layer relative positional bias from Dai et al. (2019) (Transformer-XL):

Instead of adding a position-dependent vector to each token's embedding at the input layer, the relative positional bias modifies the attention computation at every layer:

$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_{\text{head}}}} + B\right)V$

where B is a learned bias matrix whose entry B[i, j] depends only on the relative distance i - j between positions i and j, not on their absolute positions. This matrix is learned per attention head per layer.

What it computes: for each pair of positions in the attention window, the model adds a learned scalar bias that depends on how far apart the two positions are. A token attending to an immediately adjacent token gets one bias; attending to a token 100 positions away gets a different bias. This is separate from the content-based dot-product attention QK^T/√(d_head) — it purely captures position-based priors about which relative distances are most relevant.

Why this form: relative positional encoding has two advantages over absolute embeddings. First, it provides a natural inductive bias that translation-invariant relationships matter — the model learns that "the word two positions ago" is a meaningful concept regardless of absolute position in the sequence. Second, it enables the model to generalize to sequence lengths beyond those seen during training, since relative distances up to the maximum encountered continue to have learned biases (and distances beyond the maximum can be clamped). For a model trained on 1024-token sequences that may need to handle variable-length inputs at inference time, this is a practical robustness consideration. The paper states that this replaces the standard positional embedding (Section 4) without further elaboration, treating it as a known architectural improvement.

Model Scaling Configurations

Table 4 defines the complete family of GLaM models trained for the paper, organized by activated parameter count (which determines inference-time FLOPs). The key axis of variation is between MoE models (which scale by adding more experts and/or larger expert hidden dimensions) and dense models (which scale by increasing layer count, model dimension, and FFN hidden dimension in the conventional way).

The smallest model pair has approximately 1.7B activated parameters:

Model	Type	Total params	Activated params	Layers (L)	Model dim (M)	FFN hidden (H)	Experts (E)
1.7B	Dense	1.7B	1.7B	24	2048	8192	–
1.7B/64E	MoE	27B	1.879B	24	2048	8192	64

The notation 1.7B/64E means: a model whose dense backbone has approximately 1.7B activated parameters, with every other layer replaced by a 64-expert MoE layer. The total parameter count (27B) is much larger because the 12 MoE layers each store 64 experts, each with its own [M, H] weight matrices, but only 2 are activated per token.

The largest model pair has approximately 96.6–137B activated parameters:

Model	Type	Total params	Activated params	Layers (L)	Model dim (M)	FFN hidden (H)	Experts (E)
137B	Dense	137B	137B	64	8192	65536	–
64B/64E	MoE	1.2T	96.6B	64	8192	32768	64

Note the critical relationship: in the dense 137B model, the FFN hidden dimension H = 65536 in every layer. In the MoE 64B/64E model, each expert has H = 32768 — exactly half — but there are 64 experts in each MoE layer, so the total expert parameters per MoE layer are 64 × 8192 × 32768 × 2 (for the two linear projections in each expert's FFN), dramatically exceeding the dense layer's total. However, per-token inference only activates 2 × 8192 × 32768 × 2 ≈ 8192 × 65536 × 2, approximately matching the dense model's per-token FLOPs.

The paper also trains a sweep of MoE models at the ~1.7B activated parameter scale with increasing numbers of experts: 1.7B/32E, 1.7B/64E, 1.7B/128E, and 1.7B/256E (with total parameters of 20B, 27B, 53B, and 105B respectively) to study the effect of expert count at fixed per-token compute (Appendix B, Figure 6).

Other configurations include a tiny model pair (0.1B dense, 0.1B/64E MoE at 1.9B total parameters) and an intermediate pair (8B dense, 8B/64E MoE at 143B total parameters).

Across all models, the number of attention heads (nheads) and head dimension (dhead) are set so that nheads × dhead = M (the model dimension). For the largest models, this is 128 heads × 128 dimensions per head = 8192. The sequence length S is fixed at 1024 for all models.

Training Procedure and Hyperparameters

The paper uses a single training recipe across all model scales (Section 5.2), with the key elements:

Opimizer: Adafactor (Shazeer & Stern, 2018), configured with:

First-moment decay β₁ = 0 (no momentum on the first moment — effectively using RMSProp-style scaling),
Second-moment decay β₂ = 0.99,
A decay schedule of β₂ = 1 - t^{-0.8} where t is the training step (this decays the second-moment half-life over the course of training, making the optimizer more responsive to recent gradient statistics later in training),
Update clipping threshold of 1.0 (the maximum ratio of the update norm to the parameter norm, preventing individual steps from being excessively large),
Factored second-moment estimation (the default Adafactor parameterization that uses a low-rank factorization of the second-moment accumulator to save memory — critical for trillion-parameter models where a full M×M accumulator per parameter matrix would be prohibitive).

Learning rate schedule:

Initial learning rate: 0.01,
Kept constant for the first 10,000 training steps,
Then decayed with an inverse square root schedule: lr(t) ∝ 1/√t for all subsequent steps.

This warmup-hold-then-decay pattern is a common recipe for large Transformer training — the initial constant phase allows the model to find a good region of the loss landscape before the learning rate begins to decay, preventing early instability from aggressive learning rate reduction when the model hasn't yet converged to a stable parameterization.

Batch construction:

Maximum sequence length: 1024 tokens,
Each input example is packed into super-batches of up to 1 million tokens per batch — this means multiple shorter sequences are concatenated together (with appropriate attention masking) to fill the batch budget,
The paper does not specify gradient accumulation steps, but the 1M token batch size effectively determines the number of tokens processed per optimizer step.

Dropout: Set to 0 — no dropout is used anywhere in the network. The paper's rationale is that "the number of available tokens in the training corpus is much greater than the number of processed tokens during training" (Section 5.2), meaning the model never sees the same data enough times to require regularization through dropout. This is typical for large language models trained for less than one epoch on massive datasets.

Precision: Model weights are stored in float32, while activations use bfloat16. This mixed-precision strategy saves memory for activations (which dominate memory consumption in large Transformers) while maintaining full precision for weight updates to avoid numerical instability.

Tokenization: A SentencePiece (Kudo & Richardson, 2018) subword tokenizer with a vocabulary size of 256K tokens. This is a relatively large vocabulary — GPT-3 used approximately 50K tokens — which means each token represents less text on average, potentially reducing the number of tokens needed for a given corpus and improving throughput (fewer tokens → fewer model forward passes for the same text).

Hardware: The largest model, GLaM (64B/64E), was trained on 1,024 Cloud TPU-V4 chips. The paper notes that models are trained for a total of 600B tokens, with the key comparison point against GPT-3 being at 280B processed tokens (where GLaM already matches or exceeds GPT-3's accuracy after training on 300B tokens).

Stability measures: Training trillion-parameter MoE models introduces specific failure modes. The paper describes three practical interventions (Section 5.2):

Skip weight updates if NaNs or Infs appear in gradients. If any gradient value in a batch is NaN or infinite, the entire weight update for that batch is discarded. This prevents corrupted gradients from damaging model parameters. However, the paper notes that "NaN/Inf could still occur during the applying gradient step" — meaning the gradient itself can be clean, but the parameter update can still produce invalid values (e.g., if a large gradient pushes a weight into overflow). In that case, they use the next intervention.
Restart from an early healthy checkpoint. When encountering "rare large fluctuations or even NaN/Inf during training," training is restarted from a previously saved checkpoint. The paper suggests that "randomness of the sequentially loaded batches might help escape from previous failed states in the training after restart" — meaning the specific sequence of mini-batches that triggered the instability is unlikely to repeat exactly.
Train smaller-scale models to convergence first. This is described as an infrastructure and dataset validation strategy: "This allows us to expose potential issues in the dataset and infrastructure as early as possible" (Section 5.2). Issues that would be catastrophic at trillion-parameter scale (bugs in data filtering, memory leaks, sharding configuration errors) can be caught and fixed on cheaper smaller-scale runs.

Training Data Construction and Filtering

The training dataset (Section 3) is a 1.6 trillion token corpus constructed from multiple sources with explicit quality filtering. This is not merely an implementation detail — the paper devotes a full section and significant experimental analysis (Section 6.2) to demonstrating that data quality matters for downstream performance.

Data sources and proportions:

Source	Tokens (B)	Weight in mixture
Filtered Webpages	143	0.42
Conversations	174	0.28
Books	390	0.20
Wikipedia	3	0.06
News	650	0.02
Forums	247	0.02

What the mixture weights mean: these are not proportions of total tokens in the corpus — the "Tokens" column shows the absolute number of tokens available from each source after filtering. The "Weight" column shows the probability that a training batch samples from that source. For example, Wikipedia has only 3B tokens (0.2% of the total 1.6T), but is sampled with 6% probability — this oversampling prevents the model from underfitting small, high-quality sources. Conversely, Filtered Webpages (143B tokens, 9% of the corpus) are sampled at 42% weight, and Books (390B, 24% of corpus) at 20% — these reflect the paper's judgment about the relative value of each data type.

The mixture weights are described as being "chosen based on the performance of the component in a small model and to prevent small datasets such as Wikipedia from being over-sampled" (Section 3). The first criterion — performance of each component measured on a smaller model — means the authors trained a proxy model on each data source individually, evaluated its downstream task performance, and weighted sources proportionally to their quality contribution. The second criterion addresses the opposite problem: without deliberate oversampling, a source like Wikipedia would appear so rarely that the model would effectively ignore it.

Webpage filtering procedure:

The web corpus starts as a raw collection of pages (approximately 7T tokens before filtering, per Section 6.2). A quality classifier is trained to distinguish between "curated text" (Wikipedia, books, and a few selected websites — treated as positive examples) and "other webpages" (treated as negative examples). The classifier uses feature hashing (a technique that maps arbitrary text features into a fixed-size vector using hash functions, enabling fast linear classification on high-dimensional sparse features) combined with a linear classifier — the paper explicitly chooses this for "inference speed," since the classifier must process 7T tokens of webpages.

After scoring each webpage, the filtering step uses a Pareto distribution to sample webpages according to their score. This is a deliberately chosen non-uniform sampling strategy: high-quality pages are sampled with high probability, but some lower-quality pages are included as well. The paper's rationale: "to prevent systematic biases in the classifier" — if only the highest-scoring pages were kept, the classifier's own biases (toward specific writing styles, topics, or formats) would be amplified in the training data. Including some lower-scoring pages provides diversity that might capture language patterns the classifier undervalues.

The final filtered web corpus contains 143B tokens, reduced from approximately 7T — a roughly 98% reduction, indicating aggressive filtering. Section 6.2 demonstrates that this filtering produces consistent improvements over the unfiltered alternative, with the effect being "bigger on NLG than on NLU" — natural language generation tasks, which require producing fluent, coherent text, benefit more from clean training data than understanding tasks that primarily require extracting information.

Model Partitioning (2D Sharding)

Efficiently distributing a model with 64 experts per layer across 1,024 TPU chips requires careful parallelism design. GLaM uses the 2D sharding algorithm from GSPMD (Xu et al., 2021), which generalizes earlier model parallelism approaches.

Expert placement strategy:

The key constraint is that each expert should reside on a contiguous set of devices to minimize communication. GLaM places experts with the same index across different MoE layers on the same device. For example, expert 0 in MoE layer 2 and expert 0 in MoE layer 4 are physically colocated on the same TPU core. This has two benefits:

Identical computation graphs across layers: since every MoE layer has the same expert-to-device mapping, the compiler can treat each MoE layer as an identical computational module, wrapped in a while-loop control flow (Abadi et al., 2016a; Yu et al., 2018). This dramatically reduces compilation time for models with 32 MoE layers.
Reduced communication for expert parameter access: if a token is routed to expert 5 in layer 2 and also to expert 5 in layer 4, the relevant parameters are on the same device in both cases — no cross-device transfer is needed except for the token's hidden state itself.

Tensor sharding details:

For a weight tensor of shape [E, M, H] in an MoE layer (the first linear projection of each expert), the 2D sharding partitions along two dimensions simultaneously:

The expert dimension E is partitioned across N/E devices (where N is the total number of devices), so each device stores a subset of the experts.
The hidden dimension H is also partitioned, so individual devices store only a slice of each expert's weight matrix.

Correspondingly, activation tensors of shape [B, S, M] (batch size × sequence length × model dimension) are partitioned along the batch dimension B and the model dimension M.

Why this design: partitioning along two dimensions simultaneously (rather than one, as in earlier mesh-TensorFlow approaches from Shazeer et al., 2018) ensures that no device stores a full copy of any large tensor. The paper states: "we are then able to fully divide those large weight and activation tensors into smaller pieces such that there is no redundancy in data or compute across all devices." This complete elimination of data redundancy is what enables training a 1.2T parameter model on 1,024 chips without exhausting memory.

The expert size constraint is important here. The paper notes: "our experiments reveal that we should grow the size of the experts to get high quality models. Therefore, when each expert gets sufficiently large, we have to allocate each expert across a set of N/E devices" (Section C). This means individual experts are too large to fit on a single TPU core — the 2D sharding distributes even a single expert across multiple devices, requiring all-to-all communication during the expert computation. The GSPMD compiler (Xu et al., 2021) automatically determines sharding for intermediate tensors not explicitly specified.

Evaluation Protocol

The evaluation setup (Section 5.3) is designed for direct comparison with GPT-3, using the same 29 public NLP benchmarks (excluding arithmetic, word unscramble, and machine translation tasks from the original GPT-3 evaluation suite). The remaining tasks are grouped into:

8 Natural Language Generative (NLG) tasks: TriviaQA, NQS, WebQS, SQuADv2, LAMBADA, DROP, QuAC, CoQA — these require the model to generate free-form text answers, evaluated via exact match (EM) and F1 score.
21 Natural Language Understanding (NLU) tasks: 7 categories including open-domain QA, cloze/completion, Winograd-style, commonsense reasoning, reading comprehension, SuperGLUE, and natural language inference — these are predominantly multiple-choice, evaluated by comparing log-likelihoods of candidate answers.

Zero-shot evaluation: the model is given a task description and input example with no demonstrations — it must infer the task format purely from the prompt structure.

One/few-shot evaluation: 1 or few randomly selected examples from the training set are concatenated before the evaluation example (separated by two newlines), serving as in-context demonstrations. The model never receives gradient updates on these examples.

NLG decoding: generative tasks use beam search with width 4 to produce output sequences. The choice of beam search over greedy decoding or temperature sampling reflects the goal of maximizing exact-match accuracy on tasks with relatively constrained answer formats (a few words to a sentence).

NLU scoring: for multiple-choice tasks, the model computes the log-likelihood of each option given the context: log P(option | context), normalized by the token length of the option (to prevent bias toward shorter options). The prediction is the option with the highest normalized log-likelihood. For some tasks (ReCoRD, COPA), the paper notes that "non-normalized loss can yield better results" — meaning raw log-likelihood without length normalization is used when empirically superior.

Aggregation: the paper reports the arithmetic mean of scores across all datasets within the NLG and NLU groups, as well as the 7 category averages shown in Figure 1. Both accuracy (EM) and F1 metrics are "normalized to lie between 0 and 100" for consistent averaging.

4. Key Insights and Innovations

Innovation 1: Sparse Architectures Are Not Just Efficient — They Are More Capable at Equal Compute, Reframing the Scaling Conversation from "Cost-Saving" to "Performance-Advantaged"

The dominant framing of mixture-of-experts models prior to GLaM was that they offered a compromise: accept some architectural complexity and potential training instability in exchange for reduced computational cost. Shazeer et al. (2017) demonstrated large speedups for language modeling and translation; GShard (Lepikhin et al., 2021) showed that MoE could scale to 600B parameters with manageable overhead; Switch Transformers (Fedus et al., 2021) pushed to 1.5T parameters while keeping per-token FLOPs constant. But the implicit assumption across this lineage was that MoE models would, at best, match the performance of equivalently-sized dense models while costing less to train and run. The value proposition was economic and environmental — you could get the same model quality for fewer FLOPs.

GLaM's central empirical finding overturns this assumption. Figure 3(a–b) and Figure 4(a–h) demonstrate that GLaM MoE models do not merely match their dense counterparts at equal per-token FLOPs — they consistently outperform them, and the performance gap widens at larger scales. The dense 137B model and the MoE 64B/64E model have approximately equal activated parameters (137B vs. 96.6B) and therefore similar per-token inference cost. Yet the MoE model achieves higher average scores across all three evaluation settings (zero-shot, one-shot, few-shot) on both NLG and NLU benchmarks (Figure 3a–b, Table 4). At smaller scales (1.7B dense vs. 1.7B/64E MoE), the performance gap is narrower or even reversed in some settings — the MoE advantage emerges with scale.

This is a fundamentally different argument than "MoE is more efficient." It says that sparse parameterization itself provides a representational advantage — that having 1.2T total parameters organized into specialized experts, even when only 96.6B are activated per token, yields strictly better models than having 137B parameters all always active. The information stored in the 1.2T parameters cannot be compressed into the 137B parameters without loss, even though the model uses only a fraction of its total capacity for any given token. The paper suggests a mechanism for this in Section 4: with E experts and top-2 selection, the model can compose O(E²) different combinations of feed-forward subnetworks per token, providing combinatorial expressivity that a single dense FFN cannot match. But the argument is primarily empirical — the scaling curves in Figure 3 show MoE models pulling away from dense models as activated parameter count increases.

The significance of this reframing for the field's trajectory is hard to overstate. Before GLaM, organizations choosing between dense and sparse architectures faced a tradeoff: pay more engineering complexity to save money, with uncertain quality implications. After GLaM, the choice looked different: for a given inference budget, sparse models offer better capability, and the capability gap grows with scale. The paper's conclusion that "MoE should therefore be considered as a strong candidate for future scaling" is modestly stated but represents a genuine shift in the evidence base — it converts MoE from a cost-saving technique to a capability-enhancing technique at fixed cost, which changes the calculus for resource allocation in model development.

The paper acknowledges the resource tradeoff in Section 8: "the sparsely activated models consist of a higher number of parameters and thus require a larger number of devices. This limits the resource accessibility and increases the serving cost especially when the serving traffic is low." This is not a minor caveat — a 1.2T parameter model requires substantial memory to store all experts even if they aren't all accessed simultaneously, and low-traffic serving means the hardware idles with expensive parameters sitting in memory. But for the dominant use case the paper addresses (large-scale inference where per-token cost dominates), the finding that sparse models outperform dense ones at equal per-token FLOPs rather than just matching them makes the architectural choice considerably more compelling.

Innovation 2: Data Quality and Architecture Are Independent, Non-Substitutable Levers for Scaling — and Data Quality Matters More for Generative Tasks

A temptation in scaling work is to treat data as an afterthought — assemble the largest corpus possible, assume scale compensates for noise, and focus the intellectual contribution on the model architecture. The Chinchilla scaling laws (Hoffmann et al., 2022, contemporaneous but separate) would soon formalize the importance of training data quantity, but data quality was less systematically studied. Brown et al. (2020) used a quality classifier to filter Common Crawl for GPT-3, but the effect of that filtering was not isolated — it was part of the data processing pipeline, not an experimental variable.

GLaM elevates data quality to a first-class experimental axis and, crucially, demonstrates that it is not redundant with architectural improvements. Section 6.2 and Figure 3(c–d) show a clean ablation: train the same GLaM (1.7B/64E) model on the same data mixture, once with the quality-classifier-filtered web corpus (143B tokens from the filtered subset) and once with the unfiltered raw web corpus (approximately 7T tokens, subsampled to match the mixture proportions). Everything else — architecture, optimizer, learning rate schedule, auxiliary loss coefficient, training duration — is held constant.

The result is that the filtered-data model consistently outperforms the unfiltered-data model across NLG and NLU tasks, in all three evaluation settings. The paper notes that "the effect of filtering is bigger on NLG than that on NLU" and hypothesizes: "Perhaps this is because NLG often requires generating high-quality language and filtered pretraining corpora is crucial to the generation capability of language models." This is an intuitively plausible but empirically demonstrated claim — generating fluent, coherent text appears to benefit more from clean training data than answering multiple-choice questions, where the model can extract factual knowledge even from noisy sources so long as the signal is present.

The significance of this finding is that it rejects a specific form of scaling maximalism: the idea that more data always compensates for lower quality, and that the optimal strategy is simply to train on everything available. The unfiltered web corpus has approximately 50 times more tokens than the filtered subset (7T vs. 143B), but the model trained on the larger, noisier corpus performs worse. The paper does not argue that data quantity is irrelevant — the total corpus is still 1.6T tokens after filtering and combining with other sources — but it demonstrates that quality selection within a data type can have a larger impact than simply including more data of that type. This is a practical insight for data curation strategies: invest in filtering before investing in collection.

The mechanism for quality filtering — a linear classifier with feature hashing, deliberately retaining some lower-quality pages via Pareto sampling to avoid systematic classifier bias — is described in Section 3 but the conceptual contribution here is the demonstration that this mechanism matters for downstream results. The paper explicitly connects data quality to architectural scaling: "Our study highlights the fact that the quality of the pretrained data also plays a critical role, specifically, in the performance of downstream tasks" (Section 6.2). The word "also" is important — data quality is presented as an independent lever alongside model architecture, not subordinate to it.

Section 7 presents a result that the paper itself describes as "the first, to our knowledge" — GLaM (64B/64E) achieves near-identical accuracy between stereotypical and anti-stereotypical examples on the WinoGender benchmark (both 71.7%), and near-identical accuracy between male and female pronoun examples (70.8% vs. 72.5%). This is a departure from prior dense models, where coreference resolution accuracy was substantially lower on anti-stereotypical examples (where the gender-occupation pairing contradicts social statistics) than on stereotypical ones, and often lower on female pronoun examples than male ones.

The paper's language is appropriately tentative about the mechanism: "suggesting that large, sparsely activated models may rely less on superficial statistical correlations." There is no causal experiment isolating why sparsity would reduce reliance on stereotypical associations — it could be a property of scale rather than architecture, or a property of the training data, or an interaction effect. But the result is noteworthy because it contradicts a plausible null hypothesis: that adding more capacity via experts would amplify statistical associations in the training data, including undesirable ones, and that the sparsity mechanism (which dynamically selects subsets of the model) would have no particular effect either way on fairness metrics.

The WinoGender result sits alongside the paper's broader bias analysis (co-occurrence prompts, toxicity degeneration) which generally shows patterns similar to other large language models — the model reproduces gender, racial, and religious associations present in its training data (Tables 8–10), and its toxicity follows the prompt distribution (Figure 5). In these analyses, GLaM behaves similarly to GPT-3 and Gopher — large language models are influenced by prompt content, and toxic prompts beget toxic continuations. The WinoGender result is the exception, not the rule, and the paper does not overstate it.

What makes this an innovation rather than merely an interesting datapoint is the conceptual possibility it opens: that architectural choices in model design (dense vs. sparse, expert count, gating mechanism) may interact with social bias properties in ways that are not predictable from scale alone. The paper does not develop this into a theory or a prescription, but it plants a flag — MoE architectures deserve study not just for their computational properties but for their behavioral properties. Given the intense and ongoing community focus on language model fairness, this is a suggestive observation that has implications for how we evaluate architectural decisions: cost and accuracy are not the only relevant axes.

Innovation 4: Training Energy Can Be Reduced by 3–6× Without Sacrificing (and While Improving) Model Quality — Making the Environmental Argument Concrete and Quantified

Prior to GLaM, the energy consumption of large language models had been documented (Patterson et al., 2021, reporting GPT-3 at 1,287 MWh / 552 tCO₂e) and criticized (Bender et al., 2021; Strubell et al., 2019), but there was limited evidence that architectural choices could substantially change the energy-accuracy tradeoff. The dominant response to energy concerns was either mitigation (use renewable energy, improve datacenter efficiency) or reduction (train smaller models). GLaM presents a third option: change the architecture to achieve strictly better energy-accuracy Pareto efficiency.

The numbers in Table 1 and Section F are specific and auditable:

GLaM (64B/64E) trained on 600B tokens consumes 456 MWh, approximately 1/3 of GPT-3's 1,287 MWh.
To reach GPT-3–matching accuracy, GLaM requires only 280B tokens of training (vs. GPT-3's 300B), consuming 213 MWh — approximately 1/6 of GPT-3's energy cost.
The corresponding carbon emissions at the datacenter's 0.088 tCO₂e/MWh rate are 18.7 tCO₂e for the 280B-token training and 40.2 tCO₂e for the full 600B-token training, compared to GPT-3's estimated 552 tCO₂e.

What distinguishes this from prior energy reporting is that GLaM does not merely document its consumption — it demonstrates that the energy reduction is accompanied by accuracy improvement. GLaM at 280B tokens matches or exceeds GPT-3's accuracy on 4 out of 6 evaluation settings and matches on the remaining 2, as shown in Figure 4. This is not a case of "spend less energy and accept lower quality" — it is "spend dramatically less energy and get better quality." The paper explicitly frames this as a joint optimization: "our work shows that sparse decoder-only language models can be more performant than the dense architectures of similar compute FLOPs ... suggesting that sparsity is one of the most promising directions to achieve high-quality NLP models while saving energy costs" (Section 1).

The energy efficiency gains come from two sources whose relative contributions are not disentangled: the MoE architecture (which reduces total FLOPs for a given level of performance) and the hardware-software optimizations (TPU-v4's improved energy efficiency over the hardware used for GPT-3, and GSPMD's efficient sharding). The paper explicitly credits both: "The reduced energy consumption of GLaM is due to the MoE architecture and computation efficiency optimizations from TPU-v4 hardware and GSPMD software" (Section F). This makes the comparison to GPT-3 not strictly apples-to-apples in hardware — GPT-3 was trained on V100 GPUs with a different generation of accelerator — but the 1/6 energy figure remains informative for practitioners choosing an architecture for their next training run on current-generation hardware.

The significance of this quantified energy-accuracy result is that it moves the MoE efficiency argument from theoretical ("in principle, conditional computation saves FLOPs") to empirical and benchmark-grounded. A decision-maker evaluating whether to invest in MoE infrastructure can point to specific, auditable numbers: at the scale of GPT-3–class models, sparsity delivers a 3–6× energy reduction while improving benchmark scores. This is the kind of evidence that shifts organizational priorities and research investment — it makes the environmental case for sparsity concrete rather than aspirational.

5. Experimental Analysis

Evaluation Methodology

Dataset. The evaluation suite consists of 29 public NLP benchmarks, selected as a subset of the 42 tasks used in GPT-3 (Brown et al., 2020) after excluding 7 synthetic tasks (arithmetic and word unscramble) and 6 machine translation datasets (Section 5.3). The remaining tasks comprise 8 natural language generative (NLG) tasks — including TriviaQA, Natural Questions (NQS), Web Questions (WebQS), SQuADv2, LAMBADA, DROP, QuAC, and CoQA — and 21 natural language understanding (NLU) tasks spanning open-domain QA, cloze/completion, Winograd-style, commonsense reasoning, reading comprehension, SuperGLUE, and natural language inference. All evaluations are conducted on standard development or test splits as specified in Brown et al. (2020), with dataset sizes and exact splits listed in Table 11 and detailed in Appendix A.
Base model(s). The primary evaluation target is GLaM (64B/64E), the largest MoE variant with 1.2 trillion total parameters and 96.6B activated parameters per token (Table 4). This is compared against the dense GLaM (137B) with comparable per-token FLOPs, as well as a family of smaller GLaM MoE and dense models spanning from 130M to 1.7B activated parameters to characterize scaling behavior. External comparisons are made against GPT-3 (175B) — the canonical dense few-shot language model — and, where available, against Gopher (280B) and Megatron-NLG (530B) as reference points for dense scaling beyond GPT-3's scale (Table 11). All internal GLaM variants are trained on the identical 1.6 trillion token dataset described in Section 3, ensuring that architecture (MoE vs. dense) and scale are the only variables in internal comparisons.
Metrics. For NLG tasks, performance is measured via exact match (EM) accuracy and F1 score, following the standard per-task protocol from Brown et al. (2020) — for instance, TriviaQA uses EM on the development set, SQuADv2 uses both EM and F1, and DROP uses F1. For NLU tasks, which are predominantly multiple-choice, the prediction is the option that maximizes length-normalized log-likelihood given the context: log P(option | context) / |option|, with the exception of ReCoRD and COPA where non-normalized loss empirically yields better results (Section 5.3). MultiRC reports F1 over answer sets (F1a). All accuracy and F1 scores are normalized to a 0–100 scale. Aggregate performance is reported as the arithmetic mean across all 8 NLG tasks and separately across all 21 NLU tasks, as well as means within each of the 7 benchmark categories shown in Figure 1.
Baselines. The paper's primary external baseline is GPT-3 (175B) (Brown et al., 2020), evaluated on the identical 29-task suite using the same zero-shot, one-shot, and few-shot protocols with scores taken directly from the GPT-3 paper where available. Additional external reference points include Gopher (280B) (Rae et al., 2021) and Megatron-NLG (530B) (Shoeybi et al., 2019), though these are included for context only in Table 11 since they were not evaluated on the full suite or with the identical protocol. Internally, the primary baseline is GLaM (137B), a dense model with 137B activated parameters that serves as the dense counterpart to GLaM (64B/64E) at similar per-token FLOPs (Figure 3a–b, Tables 11–14). Smaller-scale internal baselines include dense models at 0.1B, 1.7B, and 8B activated parameters, each paired with an MoE variant at the same activated parameter count. An ablation using unfiltered training data is compared against the standard filtered-data model for the GLaM (1.7B/64E) configuration (Section 6.2, Figure 3c–d).
Generation budget / compute accounting. Compute is measured in GFLOPs per token prediction, derived from the number of activated parameters per token — since each activated parameter contributes approximately 2 FLOPs (one multiply and one add) per forward pass, and the per-token FLOPs scale linearly with n_act-params. Table 1 reports 180 GFLOPs/token for GLaM (64B/64E) versus 350 GFLOPs/token for GPT-3 (175B), representing a 48.6% reduction. For training, compute is measured in total TPU years (Figure 4d, h) and training energy in MWh (Table 1, Section F), accounting for the full training run including all model parameters, auxiliary losses, and data loading. Inference comparisons control for per-token cost by comparing models at similar n_act-params (e.g., 64B/64E MoE at 96.6B activated vs. 137B dense at 137B activated), while noting that the MoE model has substantially more total parameters stored in device memory. The paper explicitly accounts for the difference between total parameters (which determine memory requirements and device count) and activated parameters (which determine per-token FLOPs) when comparing efficiency.
Cross-validation / statistical protocol. The paper does not employ cross-validation or statistical significance testing — all evaluations are single-run results on fixed development or test splits. For few-shot evaluation, demonstrations are "randomly selected from the training set" (Section 5.3), but the paper does not report multiple random seeds or confidence intervals. The data contamination analysis in Appendix D (Table 6) reports per-dataset overlap between training and evaluation data using n-gram collision detection, finding that most datasets have contamination rates broadly consistent with prior work (Brown et al., 2020; Wei et al., 2021), though several datasets show high overlap: for example, only 0.01% of QuAC examples are clean, and StoryCloze shows 0% clean due to construction methodology. However, the paper does not use this contamination analysis to adjust reported scores or to filter overlapping examples — it is purely a transparency measure — so all accuracy numbers should be interpreted with the caveat that some evaluation examples may have been seen during training.

Main Quantitative Results

Headline Comparison: GLaM vs. GPT-3 Across 29 Benchmarks

The paper's central empirical claim — summarized in Table 1 and Figure 1 — is that GLaM (64B/64E) outperforms GPT-3 (175B) on average across zero-shot, one-shot, and few-shot settings on 29 NLP benchmarks while using approximately half the inference FLOPs per token and one-third (to one-sixth) the training energy.

Zero-shot (Table 11, Figure 1a): GLaM achieves an average NLG score of 54.6 versus GPT-3's 47.6 (a relative improvement of 14.7%), and an average NLU score of 66.2 versus 60.8 (a relative improvement of 8.9%). The overall average across all 29 tasks is 62.7 for GLaM versus 56.9 for GPT-3, representing a 10.2% relative improvement (Table 1). GLaM outperforms GPT-3 in 6 out of 7 benchmark categories, with the largest absolute gains in in-context reading comprehension and open-domain question answering (Figure 1a). The one category where GLaM underperforms appears to be Cloze and Completion Tasks, visible as the sole negative bar in Figure 1a.

One-shot (Table 11, Figure 1b): GLaM achieves NLG 58.4 versus GPT-3 52.9 (+10.4%) and NLU 68.6 versus 65.4 (+4.9%), with an overall average of 65.5 versus 61.6 (+6.3%). The pattern of 6-out-of-7 category wins persists, with consistent but slightly smaller gains than zero-shot.

Few-shot (Table 11, Figure 1c): GLaM achieves NLG 61.6 versus GPT-3 58.8 (+4.8%) and NLU 71.4 versus 68.4 (+4.4%), with an overall average of 68.1 versus 65.2 (+4.4%). The relative advantage narrows further as more in-context examples are provided, but GLaM maintains a positive delta across 6 of 7 categories.

A notable pattern in these aggregate numbers: the performance gap between GLaM and GPT-3 shrinks as the number of shots increases (10.2% relative improvement in zero-shot → 6.3% in one-shot → 4.4% in few-shot). This is consistent with the interpretation that the additional capacity of the MoE model (storing more knowledge in its 1.2T total parameters even though only 96.6B are activated per token) provides the greatest advantage when the model must rely entirely on its internal knowledge (zero-shot), and that in-context examples partially compensate for lower capacity in the dense model. The paper does not explicitly discuss this trend, but it is visible across all aggregate metrics in Table 1.

Per-task breakdown (Table 11): The aggregate numbers obscure substantial task-level variation. GLaM dramatically outperforms GPT-3 on several benchmarks: SQuADv2 F1 improves from 59.5 to 71.1 in zero-shot; DROP F1 from 23.6 to 57.3 in zero-shot — a more than 2× improvement; and TriviaQA from 64.3 to 71.3 in zero-shot. On other tasks, GLaM underperforms: LAMBADA accuracy drops from 76.2 (GPT-3) to 64.2 (GLaM) in zero-shot, though this reverses in one-shot (80.9 vs. 72.5) and few-shot (86.6 vs. 86.4). The CoQA and QuAC benchmarks show consistent gaps favoring GPT-3 (e.g., CoQA zero-shot: 78.8 GLaM vs. 81.5 GPT-3; QuAC few-shot: 42.8 vs. 44.3). These per-task fluctuations are aggregated away in the averages but are important context: GLaM's advantage is broad but not uniform.

Reference to larger dense models (Table 11): The paper includes Gopher (280B) and Megatron-NLG (530B) scores where available, though with inconsistent shot counts and incomplete coverage. On the tasks where comparisons can be made, GLaM generally performs competitively: on LAMBADA few-shot, GLaM achieves 86.6 versus Gopher's 74.5 and Megatron-NLG's 87.2; on WinoGrande few-shot, GLaM's 79.2 compares to Gopher's 70.1 and Megatron-NLG's 78.9. However, these comparisons are provided for context only — the evaluation protocols differ, and the paper does not claim systematic superiority over these models.

MoE vs. Dense Scaling Behavior at Fixed Per-Token Compute

The paper's second major empirical axis is the comparison between MoE and dense GLaM variants at equivalent activated parameter counts (and therefore equivalent per-token FLOPs). The key results appear in Figure 3(a–b) and Figures 4(a–h).

NLG scaling (Figure 3a): Across the range from 0.1 to 1,000 GFLOPs per token (corresponding to models from 130M to 137B activated parameters), MoE models achieve consistently higher average NLG scores than their dense counterparts at the same per-token compute. At the smallest scale (0.1B/64E vs. 0.1B dense), both model types perform similarly in few-shot (~27 vs. ~20 for NLG), but the gap expands with scale: at approximately 100+ GFLOPs/token, the MoE models' few-shot NLG scores pull substantially ahead of the dense models' (reaching 61.6 for 64B/64E vs. 57.1 for 137B dense at similar FLOPs). The GPT-3 reference points (shown as separate markers at the right edge) sit near but below the MoE scaling trend in few-shot and one-shot, and below both MoE and dense trends in zero-shot, reflecting that GPT-3 was trained on different data and with different hyperparameters.

NLU scaling (Figure 3b): The pattern is qualitatively similar but less dramatic. At the smallest scales, MoE and dense models are nearly indistinguishable in all three evaluation settings. As activated parameters increase, the MoE models begin to outpace dense models, with the gap most visible in few-shot NLU at the largest scales (~71.4 for MoE vs. ~66.8 for dense at similar FLOPs). GPT-3 sits at roughly the same NLU level as the largest GLaM dense model in zero-shot and one-shot, and slightly below both GLaM variants in few-shot.

Interpretation: These scaling curves demonstrate that the MoE advantage is not merely a property of the largest model — it is a systematic trend that emerges with scale and holds across evaluation settings. The crossover point where MoE definitively pulls ahead appears to be in the 10–100 GFLOPs/token range, corresponding roughly to the 8B/64E configuration. Below this, the benefits of combinatorial expert selection may not outweigh the optimization difficulties of the gating mechanism. The paper does not explicitly analyze the crossover point, but the visual evidence in Figure 3 suggests it.

Scaling the number of experts at fixed compute (Figure 6, Appendix B): Using the ~1.7B activated parameter base model, the paper sweeps the expert count from 1 (equivalent to dense, since 1 expert with top-2 selection reduces to dense behavior) to 256 while holding per-token FLOPs constant. On both NLG and NLU tasks, more experts yield monotonic improvements in zero-shot, one-shot, and few-shot performance. On NLG (Figure 6, left panel), the few-shot score rises from approximately 35 at 1 expert to approximately 45 at 256 experts; on NLU (right panel), the few-shot score rises from approximately 56 to 60 over the same range. This demonstrates that, for a fixed per-token compute budget, increasing the number of experts (and thus the total parameter count) reliably improves performance — the capacity of the model is genuinely increased by adding more specialized subnetworks, even though only two are ever activated simultaneously.

Data Efficiency: MoE Models Require Less Data to Match Dense Performance

Figures 4(a–c) and 4(e–g) plot average scores against the number of training tokens processed, comparing MoE and dense models at comparable FLOPs scales. The x-axis spans from approximately 100B to 631B tokens, with GPT-3's 300B-token training point marked for reference (Brown et al., 2020 trained GPT-3 on approximately 300B tokens).

NLG data efficiency (Figures 4a–c): At every training token count, the MoE models (dashed lines) achieve higher scores than their dense counterparts (solid lines). The gap is most pronounced in one-shot and few-shot settings. For the largest model pair (64B/64E MoE vs. 137B dense): at 100B tokens, the MoE model achieves approximately 40 in few-shot NLG versus the dense model's 25 — a gap of roughly 15 points. By 630B tokens, both models have improved, but the MoE advantage remains (61.6 vs. 57.1). Importantly, the MoE model trained on only 158.5B tokens matches or exceeds the dense model trained on the full 630B tokens in few-shot NLG.

NLU data efficiency (Figures 4e–g): The data efficiency advantage is smaller on NLU tasks. MoE models lead dense models at most training token budgets, but the gap is narrower — at 630B tokens, 64B/64E achieves approximately 71.4 few-shot NLU versus 66.8 for 137B dense. GPT-3's NLU scores (trained on ~300B tokens) are roughly matched by the GLaM MoE models at similar or lower token counts.

Comparison to GPT-3 at matched training tokens (Section 6.4): The paper explicitly states that GLaM (64B/64E) trained on 280B tokens outperforms GPT-3 trained on 300B tokens "by large margins on 4 out of the 6 learning settings (zero-shot/one-shot NLU and one-shot/few-shot NLG), and matches GPT-3 scores for the remaining setting, i.e., zero-shot NLG tasks" (Section 6.4). This is visible in Figures 4a–c and 4e–g where the GLaM curves at approximately 280B tokens sit above the GPT-3 markers at 300B tokens for most settings.

Compute Efficiency: MoE Models Require Less TPU Time to Match Dense Performance

Figures 4(d) and 4(h) replot the scaling data against TPU years (log scale) rather than training tokens, directly comparing the computational cost of achieving a given accuracy level.

NLG (Figure 4d): The MoE model (64B/64E) achieves higher NLG scores at every TPU-year budget after approximately 30 TPU-years. At approximately 300 TPU-years, the MoE model reaches few-shot NLG of ~61.6, while the dense model at similar or slightly higher TPU-years reaches ~57.1. The gap in one-shot and zero-shot is of similar magnitude. The curves suggest that to match the MoE model's performance, the dense model would need substantially more than the maximum compute budget shown.

NLU (Figure 4h): The TPU-year advantage on NLU is less dramatic — both models follow similar trajectories, with the MoE model achieving slightly higher scores (~71.4 vs. ~66.8 few-shot) at similar training budgets. The zero-shot and one-shot curves for MoE and dense largely overlap at smaller TPU-year budgets and only diverge at the largest scale.

Training energy (Table 1, Section F): Translating TPU-years into absolute energy, the paper reports that GLaM (64B/64E) training on 600B tokens consumes 456 MWh total, compared to GPT-3's estimated 1,287 MWh — approximately 1/3 the energy. Training to 280B tokens (the point of GPT-3–matching accuracy) consumes only 213 MWh, approximately 1/6 of GPT-3's training energy. The corresponding carbon emissions at the datacenter's 0.088 tCO₂e/MWh rate are 40.2 tCO₂e and 18.7 tCO₂e, respectively, compared to GPT-3's 552 tCO₂e.

Effect of Data Quality on Downstream Performance

Section 6.2 and Figures 3(c–d) isolate the impact of the web corpus filtering procedure described in Section 3. Using the modest-sized GLaM (1.7B/64E) model, the authors train two versions: one on the standard dataset with the quality-classifier-filtered web corpus (143B tokens) and one with the filtered webpages replaced by unfiltered raw webpages (approximately 7T tokens, subsampled to match the mixture proportions in Table 3).

NLG tasks (Figure 3c): The filtered-data model consistently outperforms the unfiltered-data model across all training token counts and all evaluation settings. At 630B training tokens, the few-shot NLG gap is approximately 45.5 (filtered) versus 42.0 (unfiltered) — a difference of roughly 3.5 points out of a maximum of 100. The gap is larger in one-shot (~42 vs. ~37) and zero-shot (~35 vs. ~28). The performance curves for the filtered model are shifted upward by approximately 5–7 points across the full training range.

NLU tasks (Figure 3d): The same qualitative pattern holds, but the magnitude is smaller. At 630B tokens, few-shot NLU scores are approximately 59.5 (filtered) versus 58.0 (unfiltered) — a gap of roughly 1.5 points. The zero-shot gap is slightly larger (~56 vs. ~54). The paper explicitly notes: "the effect of filtering is bigger on NLG than that on NLU" (Section 6.2) and hypothesizes that NLG tasks, which require generating fluent, high-quality language, benefit more from clean pretraining corpora than NLU tasks that primarily require extracting factual knowledge.

What this demonstrates: The filtering procedure does not merely remove noise — it improves the model's ability to perform downstream tasks, particularly generation. The finding that a model trained on 143B filtered tokens outperforms one trained on approximately 7T unfiltered tokens (a 50:1 ratio in available web data) strongly rejects the hypothesis that more data always compensates for lower quality. It also validates the specific filtering approach described in Section 3 (quality classifier with Pareto sampling) as a meaningful contributor to GLaM's overall performance.

Open-Domain Question Answering: TriviaQA as a Highlight Case

The paper singles out TriviaQA as a particularly challenging open-domain QA benchmark that measures a model's knowledge capacity (Section 6.1). The results appear in Table 5 and Table 11.

GLaM (64B/64E) one-shot on the TriviaQA development set achieves 75.8% exact match, compared to GPT-3 one-shot at 68.0% and GPT-3 64-shot at 71.2% on the test set. GLaM's one-shot test server submission achieves 75.0% — this exceeds the prior fine-tuned state-of-the-art of 69.8% (KG-FiD, Yu et al., 2022, which infuses knowledge graph information) by 5.2 points, and exceeds GPT-3's 64-shot test result of 71.2% by 3.8 points. The paper also notes that Switch-C (Fedus et al., 2021), which has a similar total parameter count but smaller experts, achieves only 47.5% on the development set when fine-tuned — though this comparison crosses evaluation paradigms (fine-tuning vs. few-shot) and should be interpreted cautiously.

The significance of this result for the paper's narrative is that it demonstrates the MoE model's superior knowledge capacity — the ability to store and retrieve factual information — despite having fewer activated parameters per token than GPT-3 (96.6B vs. 175B). The paper attributes this to the larger total parameter count (1.2T) and the larger expert size: "GLaM (64B/64E) uses much larger experts (beyond one TPU core) than Switch-C" (Section 6.1), suggesting that expert capacity per se matters for knowledge storage.

Ablation Studies and Robustness Checks

Expert count scaling at fixed compute (Figure 6, Appendix B): For the ~1.7B activated parameter base configuration, increasing the number of experts from 1 to 256 while holding per-token FLOPs constant yields monotonically improving performance on both NLG and NLU tasks across all three evaluation settings. On NLG, few-shot scores rise from approximately 35 (1 expert, effectively dense) to 45 (256 experts). On NLU, the improvement is from approximately 56 to 60. This confirms that the performance gains are driven by expert count (and thus total parameter capacity) rather than merely by the presence of the MoE mechanism, and that the benefits of additional capacity have not saturated at 256 experts. The paper does not ablate expert size separately from expert count, so the effect of making experts larger versus making more experts of fixed size cannot be disentangled from this experiment.

Model scale sweep (Tables 12–14, Figures 3–4): The paper evaluates the full family of GLaM models — from 0.1B to 64B/64E — on all 29 benchmarks in all three evaluation settings. The complete per-task scores appear in Tables 12 (zero-shot), 13 (one-shot), and 14 (few-shot). This sweep demonstrates that the MoE advantage over dense models at equal activated parameters is not idiosyncratic to the largest configuration: it is visible in the 1.7B and 8B activated parameter comparisons as well, though the gap is smaller at smaller scales. The sweep also enables the scaling analysis in Figure 3, establishing that the MoE advantage is a systematic trend rather than a point estimate.

Data contamination analysis (Table 6, Appendix D): To assess whether evaluation performance is inflated by training data leakage, the paper quantifies n-gram overlap between the training corpus and each evaluation dataset. The contamination rates vary widely: some datasets are almost entirely clean (WinoGrande: 99.66% clean; WiC: 92.79%), while others show near-total overlap (QuAC: 0.01% clean; StoryCloze: 0.0% clean due to the dataset construction methodology). The paper notes that these rates "roughly match those of prior work" (Brown et al., 2020) but does not adjust reported scores based on contamination or exclude contaminated examples. The presence of datasets with near-total overlap is acknowledged transparently, but the paper treats this as an inherent limitation of large-scale pretraining rather than a confound to be corrected.

Training data quality ablation (Figures 3c–d, Section 6.2): As described in the main results, this ablation demonstrates that filtering the web corpus with a quality classifier yields 3–7 point improvements on NLG and 1–2 point improvements on NLU compared to using unfiltered data, with the effect most pronounced on NLG tasks. This is a relatively clean ablation since both models share the same architecture, optimizer, learning schedule, and mixture weights — only the web corpus quality varies.

Gating function configuration: The paper uses top-2 expert selection (gating network selects the two highest-probability experts and renormalizes their probabilities for the output combination) without ablating this choice. The rationale given is that it represents "the trade-off between predictive performance and the training/serving efficiency of the model" (Section 4). Using top-1 expert (as in Switch Transformers, Fedus et al., 2021) would reduce per-token FLOPs further but sacrifice the combinatorial expressivity of blending two experts; using top-k for k > 2 would increase FLOPs. The paper does not empirically compare these alternatives, so the claim that top-2 is optimal for the performance-efficiency tradeoff is based on design reasoning rather than experimental evidence from this model family.

Auxiliary loss coefficient: The load-balancing auxiliary loss coefficient is set to 0.01 following GShard (Lepikhin et al., 2021) and is not swept. Given the paper's acknowledgment that "there is little room for hyperparameter tuning" (Section 5.2) when training trillion-parameter models, this is a practical necessity rather than a methodological choice, but it means the sensitivity of MoE performance to this coefficient is unknown for these model configurations.

Effect of relative positional bias and GLU activations: The paper adopts per-layer relative positional bias (from Dai et al., 2019) and Gated Linear Units (from Shazeer, 2020) in all GLaM models without ablating their contribution. Since these modifications are applied uniformly to both MoE and dense variants, they do not confound the MoE vs. dense comparison, but the paper cannot attribute how much of the absolute performance (as opposed to the relative MoE advantage) derives from these choices versus the base Transformer architecture.

Training stability interventions: The paper describes three stability procedures — skipping NaN/Inf gradient batches, restarting from healthy checkpoints, and training smaller models first to validate infrastructure — but does not quantify their impact on training success rates or final model quality. These are presented as implementation practices rather than experimental variables, and given that training runs at this scale are unrepeatable as controlled experiments, their necessity cannot be empirically isolated.

Critical Assessment

Claim 1: "GLaM outperforms GPT-3 across 29 NLP tasks while using half the FLOPs per token and one-third the training energy"

This claim, which anchors the paper's abstract and conclusion, is partially supported but requires careful qualification.

What the experiments demonstrate: On the specific 29-task suite evaluated, GLaM (64B/64E) achieves higher arithmetic mean scores than GPT-3 (175B) in zero-shot, one-shot, and few-shot settings (Table 1). The FLOPs comparison (180 vs. 350 GFLOPs/token) is based on activated parameter counts and is internally consistent. The training energy comparison (456 vs. 1,287 MWh) is based on auditable measurements with specified PUE and hardware power characteristics (Section F).

What the experiments do NOT demonstrate: The claim of "outperformance" is based on arithmetic means across tasks of varying type, difficulty, and metric — a model that is better on average can be worse on many individual tasks, and the paper's per-task table (Table 11) shows that GLaM indeed underperforms GPT-3 on several benchmarks (LAMBADA zero-shot, CoQA, QuAC, several Winograd-style tasks in one-shot). The energy comparison is not hardware-controlled: GPT-3 was trained on V100 GPUs, while GLaM was trained on TPU-v4 chips, which are a later-generation accelerator with different power characteristics. The 1/3 energy figure conflates architectural efficiency improvements with hardware generation improvements in an unquantified ratio. A more conservative interpretation is that GLaM demonstrates it is possible to train a model with better benchmark scores than GPT-3 at substantially lower energy cost using current-generation hardware and a sparse architecture; the relative contribution of each factor is unknown.

Additionally, the GPT-3 comparison is to a model trained with different data, different hyperparameters, and a different training procedure — GPT-3 is referenced as a fixed external benchmark, not as a controlled experimental condition. The comparison therefore demonstrates that GLaM, as a whole system (architecture + data + training recipe), achieves better results than GPT-3, as a whole system, at lower cost. It does not isolate the contribution of the MoE architecture to this advantage versus contributions from data quality, training stability, hardware efficiency, or any of the other differences between the two systems.

Claim 2: "MoE models consistently outperform dense models at similar per-token FLOPs, and the gap widens with scale"

This claim is well-supported within the paper's experimental framework but bounded by the specific dense baselines used.

What the experiments demonstrate: Across the model family from 0.1B to 1.2T total parameters, MoE variants achieve higher or equal scores compared to dense variants at matched n_act-params on the aggregate NLG and NLU metrics (Figures 3a–b). The expert count sweep at fixed compute (Figure 6) provides additional evidence that increasing expert count while holding per-token FLOPs constant improves performance, confirming that the MoE advantage is not simply an artifact of the specific expert count configurations chosen. The data efficiency curves (Figure 4) further show that MoE models reach any given accuracy threshold with fewer training tokens than dense models.

What is less clear: The dense baselines (0.1B, 1.7B, 8B, 137B) represent one particular scaling strategy for dense models — the paper scales layers, model dimension, and FFN hidden dimension according to the configuration in Table 4, but does not establish that this is the optimal dense configuration for each compute budget. A dense model with a different depth-width tradeoff, a different activation function, or different training hyperparameters might close some of the gap to the MoE variant. The paper's claim of "consistent outperformance" is therefore relative to the specific dense baselines evaluated, not relative to the best possible dense model at each scale. This is a standard limitation in scaling studies (you cannot optimize every configuration), but it means the MoE advantage should be interpreted as "MoE scaling with our specific configuration outperforms dense scaling with our specific configuration" rather than "MoE is universally superior to dense at equal FLOPs."

The finding that the MoE-dense gap widens with scale (Figures 3a–b) is visually apparent but not quantified — the paper does not report the slope of the performance gap as a function of log-FLOPs, nor does it establish that the trend continues beyond the largest scale tested (96.6B activated parameters). Extrapolating this trend to even larger models would be speculative.

Claim 3: "Data quality filtering provides consistent improvements, particularly for NLG tasks"

This claim is well-supported by the clean ablation in Figure 3(c–d) using the GLaM (1.7B/64E) model.

Strengths of the ablation: The comparison holds architecture, optimizer, learning schedule, and data mixture weights constant — only the web corpus component varies between filtered (143B tokens of quality-classified webpages) and unfiltered (roughly 7T tokens of raw webpages). This is a clean test of the filtering procedure's contribution.

Limitations: The ablation is performed at a single model scale (1.7B/64E). It is possible that the benefit of data filtering changes with model size — larger models might be more robust to noise (because they have more capacity to separate signal from noise) or more sensitive to it (because they memorize more low-quality patterns). The paper extrapolates the filtering benefit to the largest model without direct evidence that it persists at the 64B/64E scale. Additionally, the filtering procedure itself involves a classifier trained on curated text — the ablation tests filtered vs. unfiltered, but does not test alternative filtering strategies, so the claim that Pareto sampling prevents systematic classifier bias (Section 3) is a design rationale, not an experimentally validated property.

Claim 4: "GLaM closes the gap between stereotypical and anti-stereotypical examples on WinoGender"

This claim — that GLaM achieves near-identical accuracy on stereotypical and anti-stereotypical examples (both 71.7%) and on male and female pronoun examples (70.8% vs. 72.5%) — is intriguing but weakly supported as a property of the architecture.

What the experiments demonstrate: A single model (GLaM 64B/64E) evaluated on the WinoGender benchmark shows balanced accuracy across these demographic axes, representing a new state-of-the-art on the full dataset (71.7% vs. GPT-3's 64.2%) with improved demographic balance.

What is missing: The paper does not evaluate the dense GLaM (137B) on WinoGender, so there is no controlled comparison to determine whether the demographic balance is a property of scale (both models are large), architecture (MoE vs. dense), training data (the GLaM data mixture vs. GPT-3's data), or some interaction. The claim that sparsely activated models "may rely less on superficial statistical correlations" (Section 7.2) is speculative — the paper does not propose or test a mechanism by which sparsity would reduce reliance on statistical correlations, nor does it evaluate smaller MoE models to see if the balance property emerges with expert count as would be expected if sparsity were causal. The WinoGender result is a single datapoint from which no general conclusion about sparsity and bias can be drawn. The paper appropriately hedges this claim ("suggesting," "may rely less"), but the hedging reflects the absence of causal evidence rather than mere authorial caution.

Claim 5: "Training energy can be reduced by 3–6× while improving accuracy"

This claim quantifies an important practical outcome, but the 6× figure requires careful reading.

The 3× figure (456 MWh vs. 1,287 MWh) compares GLaM's full 600B-token training to GPT-3's training, at which point GLaM substantially outperforms GPT-3 on the aggregate benchmarks. The 6× figure (213 MWh vs. 1,287 MWh) compares the point where GLaM reaches approximately GPT-3–matching accuracy (after 280B tokens) to GPT-3's full training cost. Both figures are subject to the hardware-generation confound discussed under Claim 1 — the TPU-v4 vs. V100 difference contributes an unquantified portion of the energy savings. The paper acknowledges this implicitly by listing both hardware and architecture as contributors: "The reduced energy consumption of GLaM is due to the MoE architecture and computation efficiency optimizations from TPU-v4 hardware and GSPMD software" (Section F), but the relative weighting cannot be extracted from the available data.

Missing Experiments That Would Strengthen the Paper

Inference latency, not just FLOPs: The paper measures inference cost in GFLOPs/token, which maps linearly to total energy and compute time on a perfectly parallelized system. In practice, MoE models introduce all-to-all communication between devices during expert routing, and the 2D sharding strategy adds synchronization overhead that does not exist in dense model inference. Wall-clock latency per token — the metric that matters for interactive applications — is not reported. A comparison of tokens-per-second on equivalent hardware between GLaM (64B/64E) and a dense model of similar per-token FLOPs (e.g., 137B dense or a hypothetical ~100B dense model) would substantially strengthen the practical efficiency claims.

Load balancing quality: The paper uses the GShard auxiliary loss with coefficient 0.01 to encourage expert load balancing, but never reports the resulting expert utilization — what fraction of experts are used for a typical batch, whether some experts receive near-zero tokens, or how the gating distribution evolves during training. Load balancing is a well-known challenge in MoE models (Fedus et al., 2021; Lepikhin et al., 2021), and demonstrating that the auxiliary loss actually produces balanced routing at this scale would validate a critical implementation assumption. The paper's silence on expert utilization statistics is a notable gap.

Scaling behavior on individual tasks, not just averages: The aggregate NLG and NLU averages in Figures 3–4 obscure task-level heterogeneity that might be informative. For instance, does the MoE advantage concentrate in knowledge-intensive tasks (TriviaQA, NQS) and disappear for reasoning-heavy tasks (ANLI, RTE)? Per-task scaling curves would reveal whether the MoE architecture improves performance uniformly or whether its benefits are domain-specific, which would be practically important for deciding when to deploy MoE versus dense models.

Comparison to a compute-matched dense model with more training tokens: The MoE models are more data-efficient than dense models (Figure 4). A fair comparison that gives the dense model additional training tokens to compensate — i.e., match total training FLOPs rather than training tokens — would test whether the MoE advantage persists when the dense model is allowed to use the compute savings for additional training rather than additional parameters. This is the training-side analog of the inference FLOPs matching that the paper already does, and its absence means the data efficiency advantage is demonstrated but not fully characterized.

Ablation of the auxiliary loss coefficient: With only a single value (0.01) tested, the sensitivity of MoE performance to load-balancing strength is unknown. An auxiliary loss that is too weak leads to expert collapse; too strong forces the model to use experts that are genuinely irrelevant, potentially harming performance. The paper does not establish that 0.01 is near the optimum for these model configurations, which matters for practitioners seeking to reproduce the results.

Summary Assessment

The paper's experimental evidence convincingly demonstrates that, for this specific model family, training data, and evaluation suite, the sparsely activated MoE architecture achieves better benchmark scores than equivalently-sized dense models at lower per-token FLOPs, with substantially reduced training energy. The scaling trends (Figures 3–4) suggest these benefits are systematic rather than idiosyncratic to a particular scale or configuration. The data quality ablation (Figure 3c–d) shows that training data curation is an independent and important lever. The WinoGender result is suggestive but experimentally isolated — the paper lacks the controlled comparisons needed to attribute the fairness improvement to sparsity rather than scale, data, or chance.

The experiments are weakest where they conflate multiple causal factors: architecture + hardware generation + training data + hyperparameters all differ between the GLaM and GPT-3 comparisons, so the specific contribution of the MoE architecture to the headline efficiency and accuracy numbers cannot be isolated from these confounds. The internal MoE vs. dense comparisons within the GLaM family control for data and hyperparameters, providing much cleaner evidence for the architectural advantage, but these comparisons are limited to the specific dense scaling strategy adopted. The paper's central claims are generally consistent with the evidence presented, but the strength of the evidence varies substantially — the claim of superior scaling at equal per-token FLOPs is on much firmer ground than the claim about sparsity reducing reliance on statistical correlations.

6. Limitations and Trade-offs

Limitation 1: Inference Per-Token FLOPs Do Not Capture the Full Serving Cost of MoE Models

The assumption or constraint. The paper's efficiency argument rests on a single metric: per-token FLOPs (180 GFLOPs/token for GLaM 64B/64E vs. 350 GFLOPs/token for GPT-3, a 48.6% reduction — Table 1). This metric captures the arithmetic operations performed during a forward pass but explicitly excludes two critical real-world costs: device memory requirements and cross-device communication overhead. The paper acknowledges this distinction in Section 8:

"the sparsely activated models consist of a higher number of parameters and thus require a larger number of devices. This limits the resource accessibility and increases the serving cost especially when the serving traffic is low."

The consequence. The FLOPs-per-token figure tells an incomplete story about what it costs to actually serve GLaM in production. A 1.2T parameter model requires storing all 64 experts per MoE layer across the device mesh even though only two are activated per token. For GLaM (64B/64E), this means the system must provision enough total device memory to hold 1.2T parameters — roughly 2.4 TB in float32/bf16 — even though per-token computation is equivalent to a ~97B parameter dense model. This has three specific consequences:

Minimum device count: You cannot serve GLaM (64B/64E) on hardware that cannot collectively hold 1.2T parameters in memory. A dense model with equivalent per-token FLOPs (~100B parameters) might fit on 4–8 high-memory accelerators, while GLaM requires substantially more devices simply for storage — the paper trained on 1,024 TPU-v4 chips (Section 5.2). At low serving traffic, these devices sit mostly idle, paying a hardware cost that per-token FLOPs do not reflect. The paper's own caveat about "serving cost especially when the serving traffic is low" directly acknowledges this failure mode.
All-to-all communication latency: Each MoE layer requires routing tokens from their current device to the devices hosting their selected experts. This cross-device communication — an all-to-all scatter/gather pattern — adds latency that does not exist in dense model inference. For the 2D sharding strategy described in Appendix C, where individual experts are themselves partitioned across multiple devices, the communication pattern is even more complex. A token routed to expert 17 must have its hidden state sent to all devices that hold a shard of expert 17, the expert computation performed, and the results gathered back. This communication latency is not captured by FLOPs counting, which only measures the arithmetic once data is in place. For interactive applications where per-token latency matters (chatbots, code completion), the communication overhead could make GLaM slower in wall-clock time than a dense model with equivalent theoretical FLOPs.
Batch size sensitivity: The efficiency of MoE models depends on having enough tokens per batch to amortize the gating overhead and keep all experts utilized. At low batch sizes — common in interactive serving where requests arrive one-at-a-time — many experts may sit idle or receive too few tokens to be efficiently parallelized, while the dense model's cost scales linearly and predictably with batch size. The paper does not report performance at batch size 1 or study the throughput-latency tradeoff for online inference, so this regime is entirely uncharacterized.

What evidence exists in the paper. None — the paper does not report inference latency, memory footprint per request, minimum device requirements for serving, or batch-size-dependent throughput. The efficiency argument is based entirely on FLOPs and training energy. The acknowledgment in Section 8 is qualitative and does not quantify the magnitude of the issue.

Mitigation status. The paper does not attempt to mitigate this limitation. It flags the concern in Section 8 and moves on. There is no suggestion of future work to characterize or reduce serving overhead beyond the architectural choices already made. The efficiency claims in the abstract and Table 1 should therefore be read as training-time and theoretical inference-FLOPS efficiency, not as end-to-end serving cost efficiency. For a practitioner deciding whether to deploy a MoE model, the FLOPs-per-token number is a lower bound on serving cost, and the true cost — especially at low traffic — may be substantially higher than the 48.6% reduction claimed.

Limitation 2: The GPT-3 Comparison Conflates Architecture, Data, Hardware, and Hyperparameters — the MoE Advantage Cannot Be Isolated

The assumption or constraint. The paper's headline comparison — "GLaM outperforms GPT-3 on 29 NLP benchmarks while using 1/3 the training energy" — treats GPT-3 as a fixed reference point. But the two models differ in architecture (sparse MoE decoder-only vs. dense decoder-only), training data (GLaM's 1.6T-token curated corpus with quality filtering vs. GPT-3's corpus), hardware generation (TPU-v4 vs. V100 GPUs), hyperparameters (Adafactor with specific schedule vs. Adam), vocabulary size (256K vs. ~50K tokens), and additional architectural components (GLU activations, relative positional bias). The paper acknowledges the hardware difference in Section F:

"The reduced energy consumption of GLaM is due to the MoE architecture and computation efficiency optimizations from TPU-v4 hardware and GSPMD software."

But the magnitude of each factor's contribution cannot be extracted from the available data.

The consequence. A decision-maker reading this paper might conclude that MoE architectures are inherently 3–6× more energy-efficient than dense architectures for a given accuracy level. But the evidence does not support isolating the architecture's contribution from the other confounding factors. Specifically:

Hardware generation confound: TPU-v4 chips are a newer generation than the V100 GPUs used for GPT-3, with different process technology, memory bandwidth, and FLOPs-per-watt characteristics. The paper reports that TPU-v4 system power is 326W per chip (Section F), but does not provide a like-for-like comparison — what would GPT-3's energy consumption have been if trained on TPU-v4 with equivalent software optimizations? Without this, the 1/3 to 1/6 energy figure conflates architectural efficiency with hardware progress. A model trained on TPU-v4 with a dense architecture might close some fraction of the gap. The paper cannot tell us how much.
Data quality confound: Section 6.2 and Figure 3(c–d) demonstrate that data filtering produces a 3–7 point NLG improvement at the 1.7B/64E scale. Since GPT-3 used a different filtering pipeline (also a quality classifier, but with different training data and thresholds), any performance difference attributable to data cannot be separated from the architectural difference. GLaM's superior performance on TriviaQA (+7 points zero-shot over GPT-3) might be driven by better knowledge coverage in its training corpus rather than by expert capacity.
Training duration confound: GLaM was trained on 600B tokens (or 280B for the GPT-3–matching comparison), while GPT-3 was trained on approximately 300B tokens. The models were trained for different numbers of optimizer steps with different learning rate schedules. The paper's internal comparisons (Figures 4a–c, e–g) show that GLaM MoE models at 280B tokens match or exceed GPT-3 at 300B tokens, but this is at 280B tokens of GLaM's specific data distribution, not a controlled experiment where data and training budget are held constant.

What evidence exists in the paper. The paper provides internal MoE vs. dense comparisons within the GLaM family (Figures 3–4, Tables 12–14) that control for data, hardware, hyperparameters, and auxiliary architectural components. These comparisons are much cleaner than the GPT-3 comparison and constitute the paper's strongest evidence for MoE superiority. But the paper's framing — and its abstract — emphasizes the GPT-3 comparison as the primary result, and the internal comparisons receive less prominence. A reader who only absorbs the headline comparison to GPT-3 will overestimate the certainty with which the architectural contribution is known.

Mitigation status. The paper partially mitigates this by providing the internal MoE vs. dense comparisons (Figures 3–4, Tables 12–14), which isolate architecture as the sole variable. These show a consistent but smaller advantage for MoE over dense at equal activated parameters. The paper does not attempt to ablate the hardware, data, and hyperparameter contributions to the GPT-3 comparison individually, nor does it acknowledge the confound explicitly in the main text. The hardware caveat is in Section F (appendix), separate from where the headline comparison is made (abstract, Section 1, Table 1). A thorough treatment would have been to train a dense GLaM variant of comparable total FLOPs and compare it to GPT-3 as a baseline to bound the non-architectural contribution, but this was likely computationally prohibitive.

Limitation 3: Inference Efficiency Claims Are Based Solely on Activated Parameters — Wall-Clock Latency, Communication Overhead, and Batch-Size Effects Are Never Measured

The assumption or constraint. The paper measures inference efficiency exclusively in GFLOPs per token, derived from the number of activated parameters per forward pass (180 GFLOPs/token for 64B/64E vs. 350 for GPT-3 — Table 1). This metric assumes that FLOPs are the bottleneck and that all FLOPs can be executed at peak hardware utilization. The paper does not report any empirical inference timing measurement — no tokens-per-second throughput, no per-token latency, no batch-size scaling curves, no communication-to-computation ratio analysis.

The consequence. MoE architectures introduce several overheads that do not exist in dense models and are invisible to FLOPs counting:

Gating computation: For every token at every MoE layer, the gating network must compute softmax probabilities over all 64 experts, select the top-2, and dispatch the token to the appropriate devices. This is a small fraction of total FLOPs but involves a top-k operation (which requires sorting or selection) and device-level routing decisions — operations that are not pure matrix multiplies and may have different throughput characteristics on accelerators.
All-to-all communication: Between the gating decision and the expert computation, tokens must be physically moved across the device mesh. The 2D sharding described in Appendix C distributes each expert's weight matrix across multiple devices, meaning a token routed to expert 17 must communicate with every device holding a shard of expert 17. This communication is not optional — it is on the critical path of every MoE layer — and its latency depends on the device interconnect bandwidth, topology, and contention from other tokens' routing. For the interleaved architecture (dense layer → MoE layer → dense layer → MoE layer), this communication happens at every other layer, potentially doubling the communication frequency compared to a pure dense model.
Load imbalance: The gating network routes tokens to experts based on their hidden representations, which are data-dependent and not known in advance. If a batch routes 40% of tokens to expert 3 and 1% to expert 27, the device hosting expert 3 will be heavily loaded while the device hosting expert 27 idles. The GShard auxiliary loss (coefficient 0.01, Section 5.2) encourages balanced routing, but: (a) the paper never reports achieved load balance, (b) even with auxiliary loss, some imbalance persists in practice, and (c) at small batch sizes (common in interactive serving), statistical imbalance becomes unavoidable regardless of the loss. The straggler effect — where overall batch latency is determined by the slowest (most overloaded) expert — is a well-documented challenge in MoE systems that FLOPs counting entirely ignores.
Memory bandwidth: The expert parameters not used for a given token still occupy device memory. In the 64B/64E configuration, each MoE layer stores 64 expert FFNs, each with weight matrices of shape [8192, 32768] and [32768, 8192] (roughly 1B parameters per expert, ~2 GB). All 64 experts must be resident in device memory across the pod, consuming aggregate memory proportional to the total parameter count (1.2T) even though only ~8% is accessed per token. This means the model's memory footprint is ~12× larger than a dense model with equivalent per-token FLOPs, requiring more devices and more total memory bandwidth simply to hold the model.

What evidence exists in the paper. None. The paper does not report any empirical timing or throughput measurement, does not analyze communication patterns or overhead, does not report load balancing statistics, and does not characterize batch-size sensitivity. The inference efficiency claim is purely theoretical, based on the activated parameter count.

Mitigation status. The paper does not address this limitation. The efficiency argument as presented invites the reader to equate "half the FLOPs per token" with "half the inference cost," which is only true under the assumption that FLOPs are the binding constraint and all FLOPs are executed at equal efficiency. For practitioners, the absence of any latency or throughput measurement means the inference efficiency claims cannot be validated without reproducing the system — and the paper's acknowledgment in Section 8 that serving cost is higher "especially when the serving traffic is low" suggests the authors are aware the story is more complicated than FLOPs alone.

Limitation 4: The WinoGender Fairness Result Lacks Controlled Comparisons — the Claim That Sparsity Reduces Reliance on Stereotypes Is Speculative

The assumption or constraint. Section 7.2 reports that GLaM (64B/64E) achieves near-identical accuracy on stereotypical and anti-stereotypical WinoGender examples (both 71.7%) and on male and female pronoun examples (70.8% vs. 72.5%), representing a new state-of-the-art (71.7% overall vs. GPT-3's 64.2%). The paper speculates:

"suggesting that large, sparsely activated models may rely less on superficial statistical correlations."

This causal attribution — that sparsity causes reduced stereotype reliance — lacks any controlled comparison. The paper evaluates exactly one model on this benchmark and does not report WinoGender results for the dense GLaM (137B), smaller GLaM MoE variants, or any model trained with identical data but a dense architecture.

The consequence. The WinoGender result is consistent with multiple explanations, only one of which involves sparsity:

Scale, not architecture: The result could be driven by model scale — larger models in general may learn more abstract representations that are less stereotype-reliant. Without evaluating the 137B dense model, this cannot be ruled out.
Data, not architecture: The GLaM training corpus, with its specific mixture weights and quality filtering, may simply contain fewer gender-occupation stereotypical associations than GPT-3's corpus. The data quality ablation (Section 6.2) shows that data filtering has substantive effects on downstream performance — if the filtered corpus is less stereotypically skewed, the fairness improvement would be a data effect, not an architectural one.
Idiosyncrasy, not systematic effect: With a single model evaluated on a single benchmark, the result could be noise. The WinoGender test set has 120 examples total (60 stereotypical, 60 anti-stereotypical), and a 71.7% accuracy means the model gets approximately 86 out of 120 correct. A swing of 2–3 examples changes the stereotypical/anti-stereotypical balance noticeably. Without multiple random seeds, multiple models, or confidence intervals, statistical robustness cannot be assessed.
Architecture: Sparsity could genuinely reduce reliance on superficial correlations — perhaps because expert specialization forces the model to route gender-related and occupation-related processing through different experts, preventing entanglement. This hypothesis is plausible but untested.

The consequence for scholarship is that the paper's WinoGender result is cited in its abstract and Section 1 as a contribution ("our results are also the first, to our knowledge, to close the performance gap between stereotypical and anti-stereotypical examples on the WinoGender benchmark"), but the evidence cannot distinguish between the architectural explanation and simpler alternatives (scale, data, noise). A reader might take away that MoE architectures have been shown to reduce bias, when the paper has only observed that one MoE model performs well on one bias benchmark without controlled comparisons.

What evidence exists in the paper. Exactly one datapoint: GLaM (64B/64E) evaluated on WinoGender (Section 7.2). No dense baseline, no scale sweep, no data ablation, no multiple random seeds, no confidence intervals. The co-occurrence analysis (Tables 8–10) and toxicity analysis (Figure 5) — which do show patterns similar to other large language models — are separate evaluations using different methodologies and are not compared to the WinoGender result. The paper's broader bias analysis does not demonstrate a systematic MoE advantage; it shows GLaM behaving similarly to dense models on most bias metrics.

Mitigation status. The paper uses appropriately cautious language ("suggesting," "may rely less") and does not claim to have proven causality. But the hedging does not compensate for the absence of experimental controls. The paper does not propose a mechanism by which sparsity would reduce bias, does not evaluate alternative explanations, and does not suggest follow-up experiments to test the hypothesis. The WinoGender result is presented as evidence for a claim that the experimental design cannot support. A fairer framing would treat it as an intriguing observation requiring controlled study rather than a demonstrated property of sparse architectures.

Limitation 5: Difficulty Estimation — the Compute-Optimal Framework's Central Mechanism — Requires Prohibitively Expensive Pre-Computation

The assumption or constraint. This is not a limitation of the current GLaM paper (which does not use a compute-optimal allocation framework). I note that this limitation appears to have been carried over from the reference example and does not apply to the GLaM paper. Let me replace it with an actual limitation of the GLaM paper.

Limitation 5: Expert Count Scaling Is Studied at Only One Model Size — the Interaction Between Expert Count and Base Model Scale Is Uncharacterized

The assumption or constraint. The paper's expert count ablation (Figure 6, Appendix B) sweeps the number of experts from 1 to 256 using the 1.7B activated parameter base configuration while keeping per-token FLOPs constant. This demonstrates that, at this specific scale, more experts yield monotonically improving performance. But the paper does not perform this sweep at other scales — we do not know how the benefit of additional experts changes as the base model grows. The largest model (64B/64E) uses 64 experts, but there is no evidence that 64 is optimal at this scale, or that 128 or 256 experts would provide further gains comparable to those observed at the 1.7B scale.

The consequence. A practitioner designing a MoE model at a different scale cannot use this paper to determine the optimal expert count. If the benefit of additional experts saturates at larger base model sizes (because the larger dense backbone already captures more of the representational diversity that experts provide), then using 256 experts at 64B activated parameters might yield minimal improvement over 64 experts while substantially increasing memory footprint and device count. Conversely, if the benefit accelerates with scale (because larger experts can specialize more effectively), then 64 experts might be substantially suboptimal at the 64B scale, leaving performance on the table. The paper's recommendation to "grow the size of the experts to get high quality models" (Section C) addresses expert size but not expert count scaling at different base model capacities.

What evidence exists in the paper. Figure 6 shows the expert count sweep at a single scale (1.7B/64E). Figure 3(a–b) shows performance at different scales, but with expert count co-varying with other architectural parameters — the 0.1B/64E, 1.7B/64E, 8B/64E, and 64B/64E models all use exactly 64 experts, so the effect of using different expert counts at different scales cannot be extracted. The paper does not ablate expert count at the 8B or 64B scale, and does not report results for configurations like 64B/128E or 8B/256E.

Mitigation status. The paper does not address this limitation. The choice of 64 experts for all MoE variants (except the 1.7B sweep in Appendix B) is not justified with scaling evidence — it appears to be a fixed design choice carried across scales, possibly constrained by hardware (device count, memory) rather than optimized for model quality. The expert count sweep at 1.7B provides suggestive evidence that more experts help, but the interaction with base model scale remains uncharacterized. A systematic study varying expert count and base model scale jointly would be expensive but necessary to establish a MoE scaling law analogous to Kaplan et al. (2020) for dense models.

Limitation 6: Single Benchmark Domain and Model Family — All Results Are on English NLP Benchmarks with One Model Architecture Trained on One Data Corpus

The assumption or constraint. All experiments use the MATH — correction, all experiments use 29 English NLP benchmarks (Section 5.3, Appendix A) with a single model family (GLaM, based on the PaLM 2 — correction, based on a custom Transformer decoder with GLU and relative positional bias, Section 4) trained on a single dataset (the 1.6T token corpus described in Section 3). The paper does not evaluate on code generation, multilingual tasks, mathematical reasoning, or any domain outside English natural language understanding and generation. The authors state a belief about broader applicability but do not test it:

"We believe this model is representative of the capabilities of many contemporary LLMs." (No such statement appears — the paper does not claim generalizability beyond its evaluated scope, which is appropriate but means the scope is limited.)

The consequence. The paper's conclusions — that MoE architectures outperform dense models at equal per-token FLOPs, that data quality filtering matters more for NLG, that sparsity may reduce stereotype reliance — are strictly bounded by the evaluated domain (English NLP) and model family (GLaM's specific architecture). Several aspects of the findings could be domain- or architecture-specific:

Knowledge-intensive vs. reasoning-intensive tasks: The MoE advantage is most dramatic on open-domain QA tasks like TriviaQA (+7 points zero-shot over GPT-3, Table 11), which test knowledge recall. On reasoning-heavy tasks, the advantage is smaller or negative — on ANLI R1 zero-shot, GLaM scores 39.2 vs. GPT-3's 34.6, a modest gain; on ANLI R3 zero-shot, GLaM's 41.3 vs. GPT-3's 34.5 is larger but still far from the TriviaQA gap. This pattern is consistent with the hypothesis that MoE's primary benefit is increased knowledge storage capacity (more total parameters = more facts memorizable) rather than improved reasoning capability. Without evaluating on benchmarks that more cleanly separate knowledge from reasoning (e.g., synthetic reasoning tasks, knowledge-free logical puzzles), this hypothesis cannot be tested.
Language coverage: All benchmarks are in English. The gating network's language-conditional routing behavior — whether experts specialize by language in multilingual settings — is a well-documented phenomenon in MoE models (e.g., GShard's original motivation was multilingual translation) that the paper cannot speak to.
Modality: The architecture is text-only. Whether the MoE advantage persists in vision-language models, speech models, or multimodal settings — where different modalities might naturally route to different experts — is unknown.
Architecture specificity: GLaM uses specific design choices (interleaved MoE/dense layers rather than all-MoE, GLU activations, per-layer relative positional bias, top-2 gating, specific expert size) that may interact with the scaling behavior. A different MoE configuration (e.g., all layers with MoE, top-1 gating like Switch, different expert sizes) might show different scaling trends. The paper establishes that this particular MoE recipe works well; it does not establish which elements of the recipe are necessary or how sensitive the results are to architectural variations.

What evidence exists in the paper. All evidence is from 29 English NLP benchmarks evaluated on the GLaM model family. The paper does not claim generalizability beyond this scope, but the abstract and conclusion language ("generalist language model," "strong candidate for future scaling") implicitly invites broader interpretation. Table 2 situates GLaM among other large language models (BERT, T5, GPT-3, Gopher, Megatron-NLG, Switch-C) that span different architectures and training paradigms, but the comparison is taxonomic rather than experimental — no cross-model controlled experiments are performed.

Mitigation status. The paper does not address the domain/model specificity limitation. This is standard for scaling papers (evaluating on additional domains is expensive and the paper's contribution is primarily the demonstration that MoE works at all in the few-shot setting), but it means a practitioner cannot assume the findings transfer to their domain without additional validation. The paper's contribution is best understood as establishing that sparse decoder-only models can be competitive in the specific evaluation paradigm where GPT-3 had set the standard, not as establishing universal properties of MoE architectures.

7. Implications and Future Directions

How This Work Changes the Landscape

The landscape change this paper produces is best characterized as a reframing with concrete empirical backing rather than a paradigm shift. The core idea — that conditional computation can decouple model capacity from per-token inference cost — was not new in 2022. Shazeer et al. (2017) had demonstrated the principle, GShard (Lepikhin et al., 2021) had scaled it, and Switch Transformers (Fedus et al., 2021) had pushed it to a trillion parameters. What GLaM changes is the credible default architecture for few-shot language models.

Before GLaM, the dominant public models in the in-context learning paradigm were all dense: GPT-3 (175B), Jurassic-1 (178B), Gopher (280B), Megatron-NLG (530B). The trajectory was linear — if you wanted better few-shot performance, you built a bigger dense model and paid the proportional inference cost. Switch-C (Fedus et al., 2021) was the closest MoE competitor in scale, but it used an encoder-decoder architecture evaluated primarily in the fine-tuning setting. A practitioner deciding how to build the next generation of few-shot language models in early 2022 could reasonably look at the evidence and conclude: MoE is promising but unproven; the safe bet is dense scaling.

GLaM demolishes that uncertainty with a systematic, apples-to-apples comparison that the field had been missing. The combination of evidence — MoE outperforms dense at equal per-token FLOPs within a controlled model family (Figures 3–4), the gap widens with scale (so the finding is not just about one favorable configuration), and the result holds across zero-shot, one-shot, and few-shot evaluation on 29 benchmarks — converts MoE from an experimental curiosity into a defensible engineering choice backed by head-to-head comparisons. The paper's explicit conclusion that "MoE should therefore be considered as a strong candidate for future scaling" (Section 1) reads as deliberately measured, but within the context of the evidence assembled, it represents a genuine shift in the default position.

The reframing is subtle but consequential. Before GLaM, the MoE conversation was about efficiency at the cost of complexity — you added gating networks, load-balancing losses, and cross-device communication to save FLOPs, with the implicit assumption that a dense model of equivalent total compute would achieve similar or better accuracy if you could afford to train it. After GLaM, the conversation is about capability advantage at equal cost — you get strictly better models for the same inference FLOPs, and the advantage grows with scale. Figure 3(a–b) is the critical evidence here: at larger activated parameter counts, the MoE scaling curves sit definitively above the dense curves in both NLG and NLU, in all three evaluation settings. The scaling trend means this is not just a favorable datapoint for one model size — it suggests a structural advantage of combinatorial expert selection over monolithic feed-forward layers.

This reframing makes certain research directions more attractive and others less so:

More attractive: improving gating mechanisms to enable finer-grained expert specialization; developing training recipes that eliminate the need for auxiliary load-balancing losses (since these impose a distortion on the model's natural routing preferences); studying what experts actually learn (do they specialize by topic, by linguistic phenomenon, by reasoning pattern?) to inform better architecture design; exploring whether expert count should scale as a function of base model size along a predictable law analogous to the Chinchilla scaling laws for dense models; investigating whether sparsity interacts with fairness properties in a systematic way (the WinoGender result, while experimentally uncontrolled, is genuinely intriguing as a direction).

Less attractive: building ever-larger dense models as the default strategy for improving few-shot performance. The paper does not argue that dense scaling is obsolete — it acknowledges that the 137B dense model is still highly capable — but it raises the bar for what a new dense model must demonstrate to justify its cost. If a 1.2T-parameter MoE model can match or exceed a 175B dense model at half the inference FLOPs, a new 500B dense model must show that it can outperform MoE models at the same inference budget, not just that it improves on previous dense models.

The paper also resolves a tension that had been building in the scaling literature. The Kaplan et al. (2020) scaling laws established that model quality improves as a power law with parameter count, implying that the path to better performance was straightforward: add parameters. But Patterson et al. (2021) had just documented that this path led to GPT-3 consuming 1,287 MWh and 552 tCO₂e for training — numbers that made environmental sustainability a first-order concern. These two findings pulled in opposite directions: better models meant bigger models, but bigger models meant unsustainable energy consumption. GLaM's contribution is to demonstrate empirically that this tension is not fundamental — you can scale total parameters (for knowledge capacity) while keeping per-token FLOPs (and thus inference energy) roughly constant, and you can do so while actually improving accuracy. The energy-accuracy Pareto frontier is not fixed; it moves with architecture choice. This is not a theoretical insight — the paper's specific, auditable numbers (456 MWh for full training, 213 MWh to match GPT-3, vs. 1,287 MWh) provide the quantitative evidence that the tension can be substantially reduced.

Follow-Up Research This Work Enables

Characterizing expert specialization in large decoder-only MoE language models. The paper demonstrates that MoE works at scale for few-shot tasks, but provides no analysis of what the 64 experts in each MoE layer actually learn. Do experts specialize by topic (one expert handles mathematical content, another handles biographical text)? By linguistic phenomenon (one expert processes relative clauses, another handles coordination)? By position in the sequence? Or is the routing behavior more diffuse, with experts learning complementary transformations that don't map cleanly onto human-interpretable categories? GShard (Lepikhin et al., 2021) found that experts in multilingual translation models specialized by language, but a decoder-only language model trained on English text has no analogous structural partition. A follow-up study could analyze the routing patterns in GLaM (64B/64E) on curated evaluation sets — for example, feeding in sentences from different domains (scientific abstracts, dialogue, legal text, code) and measuring whether the gating distribution shifts systematically, or analyzing whether specific experts are consistently activated for tokens at particular syntactic positions (e.g., expert 17 always fires on the first token after a comma). The practical implication of such analysis would be to inform expert count design: if experts specialize by coarse domain, you might want many small experts; if they learn diffuse complementary features, fewer large experts might suffice.

Establishing a MoE scaling law that relates expert count, expert size, and base model scale to downstream performance. The paper provides scaling curves for MoE vs. dense models (Figures 3–4) at fixed expert counts (64), and an expert count sweep at a single base model size (Figure 6). What is missing is a systematic study that varies both base model scale and expert configuration jointly to produce a predictive relationship analogous to the Kaplan et al. (2020) or Hoffmann et al. (2022) scaling laws. Concretely: for a given per-token FLOP budget and total parameter budget, what is the optimal allocation between dense layers and MoE layers? Between expert count and expert hidden dimension? Does the optimal expert count scale with base model dimension? The paper's observation that "we should grow the size of the experts to get high quality models" (Section C) is a qualitative finding; a scaling law would quantify the tradeoff (e.g., expert size should grow as M^α for some exponent α). A strong follow-up would train several model families varying base dimension (e.g., 1B, 4B, 16B activated parameters) and expert configuration (32, 64, 128, 256 experts with varying hidden dimensions), fit power-law relationships predicting downstream accuracy from these architectural parameters, and validate on a held-out scale. The computational cost would be substantial but the resulting design principles would replace the current practice of "pick 64 experts because it worked well in GLaM" with a principled optimization.

Testing whether the WinoGender fairness result is architectural, scale-dependent, or idiosyncratic. The paper reports that GLaM (64B/64E) achieves near-identical accuracy on stereotypical and anti-stereotypical WinoGender examples (both 71.7%), speculating that sparsity "may rely less on superficial statistical correlations" (Section 7.2). This hypothesis is testable with a clean experimental design: evaluate GLaM (137B) dense — which shares training data, vocabulary, and auxiliary architectural components (GLU, relative positional bias) — on WinoGender. If the dense model shows a substantial stereotypical/anti-stereotypical gap while the MoE model does not, sparsity becomes a plausible causal factor. If both models show balanced accuracy, the effect is likely driven by scale or data quality rather than architecture. A stronger version of this experiment would evaluate the full GLaM model family (0.1B, 1.7B, 8B, 64B activated parameters, both MoE and dense variants) on WinoGender to trace when and whether the balance property emerges. If the MoE models show consistently lower bias than dense models at matched scale, the case for an architectural mechanism strengthens; if the effect only appears at the largest scale regardless of architecture, scale (or training data interactions at scale) is the more likely driver. A negative result — the dense 137B model also shows balanced WinoGender performance — would be valuable because it would reframe the WinoGender finding as a general property of large, well-trained models rather than a sparsity-specific benefit, which is itself an important clarification for the fairness literature.

Developing verifier-free difficulty estimation for adaptive compute allocation in MoE models. This direction applies the compute-optimal test-time scaling framework from the reference paper (which is not part of GLaM but represents a natural connection) to MoE architectures. The paper demonstrates that MoE models achieve strong performance substantially earlier in training than dense models (the GLaM 64B/64E at 280B tokens matches GPT-3's accuracy, Figure 4). This data efficiency property suggests that MoE models might be particularly well-suited for inference-time compute adaptation: if a MoE model already "knows" the answer (stored somewhere in its 1.2T parameters) but doesn't reliably activate the right experts to retrieve it, additional inference-time computation (more samples, verifier-guided search) might unlock that knowledge at lower marginal cost than for a dense model. A concrete experiment: take GLaM (64B/64E), apply best-of-N sampling with a verifier on an open-domain QA task like TriviaQA, and measure whether the accuracy improvement from additional samples scales differently (e.g., steeper slope) than for a dense model of comparable per-token FLOPs. The hypothesis would be that the larger total parameter count of the MoE model gives it a richer set of possible outputs to explore through sampling, making verifier-guided search more effective per additional sample. This would connect the MoE scaling efficiency results to the growing literature on inference-time compute optimization.

Ablating the interaction between data filtering quality and MoE expert count. The paper's data quality ablation (Figure 3c–d) demonstrates that filtering matters more for NLG than NLU at a fixed model scale (1.7B/64E), but does not explore whether the benefit of filtering depends on expert count. A plausible hypothesis: with more experts (each with more specialized capacity), the model can better separate high-quality and low-quality patterns in the training data, making filtering less necessary — the model itself learns to route low-quality patterns to "noise experts" that get low gating probability for high-quality evaluation examples. Alternatively, more experts might amplify noise because more specialized subnetworks more faithfully memorize low-quality patterns. A concrete experiment would train a set of models at the 1.7B/64E scale but with varying expert counts (32, 64, 128, 256) on both filtered and unfiltered data, measuring the performance gap between filtered and unfiltered at each expert count. If the gap shrinks with expert count, MoE architectures provide some inherent robustness to noisy data; if it widens, data quality becomes even more important for MoE models, and practitioners should invest more heavily in filtering when deploying MoE architectures. Either outcome would refine the paper's data quality findings and provide actionable guidance for training data curation.

Stress-testing the MoE advantage on tasks that require multi-step reasoning rather than knowledge recall. The paper's strongest results are on knowledge-intensive benchmarks — TriviaQA (+7 points zero-shot over GPT-3, Table 11), SQuADv2 (+11.6 points zero-shot F1, Table 11), DROP (+33.7 points zero-shot F1). On reasoning-intensive tasks, the advantage is more modest or occasionally negative — ANLI R2 one-shot: GLaM 40.0 vs. GPT-3 33.9 (+6.1 points, solid but not dramatic); LAMBADA zero-shot: GLaM 64.2 vs. GPT-3 76.2 (−12 points, a substantial loss). A systematic follow-up would construct or select a benchmark suite that explicitly separates knowledge recall (e.g., "What year did X happen?") from multi-step reasoning (e.g., "If A implies B and B implies not C, and C is true, what can we conclude about A?") and evaluate the full GLaM model family against equivalently-sized dense baselines. If the MoE advantage concentrates entirely in knowledge-intensive tasks — which is consistent with the interpretation that the additional expert parameters store more facts but don't improve reasoning — then the design implications are significant: for applications requiring factual breadth (open-domain QA, encyclopedic chatbots), invest in MoE; for applications requiring logical depth (theorem proving, multi-hop reasoning, complex code generation), dense models with deeper reasoning capacity per parameter might be preferable. This would carve the generalization space at its joints rather than relying on aggregate benchmark averages that obscure the knowledge-vs-reasoning distinction.

Practical Applications and Downstream Use Cases

Cost-efficient deployment of large language models for knowledge-intensive applications. The paper's finding that GLaM achieves GPT-3–exceeding accuracy at roughly half the per-token FLOPs (180 vs. 350 GFLOPs/token, Table 1) has direct implications for any organization serving language model inference at scale. Consider an open-domain question answering product (similar to TriviaQA's task) that serves millions of queries per day. At GPT-3's inference cost, each query requires 350 GFLOPs per generated token; with a GLaM-style MoE model of comparable accuracy, the cost drops to approximately 180 GFLOPs per token — a 48.6% reduction. For a product generating an average of 100 output tokens per query across millions of daily queries, this translates to a near-halving of inference compute costs, which at scale represents substantial operational savings. The tradeoff is the higher memory footprint (1.2T parameters requiring more devices), but for high-traffic applications where per-token throughput dominates, the FLOPs advantage translates directly to cost reduction. The paper's TriviaQA result — 75.8% one-shot accuracy vs. GPT-3's 68.0% (Table 5) — additionally shows that the cost savings come with accuracy improvement rather than sacrifice, meaning organizations don't face a cost-vs-quality tradeoff.

Training budget allocation for organizations building custom large language models. The training energy comparison in Table 1 (456 MWh for full GLaM training vs. 1,287 MWh for GPT-3, or 213 MWh to reach GPT-3–matching accuracy — roughly 1/6 the energy) provides a concrete planning number for teams deciding whether to invest in MoE infrastructure or to scale a dense model. A team with a fixed compute budget — say, 500 MWh of training energy available — could train a GLaM-scale MoE model that outperforms GPT-3, or a dense model of comparable quality that costs 1,287 MWh (exceeding their budget), or a smaller dense model that fits the budget but underperforms. The paper's data efficiency results (Figure 4) further show that the MoE model reaches any given accuracy threshold with fewer training tokens, meaning the training run can be shorter and cheaper. For organizations where training cost is the binding constraint (academic labs, startups, companies entering NLP from adjacent fields), the MoE architecture represents a path to competitive model quality at a trainable budget. The caveat is device count: the paper used 1,024 TPU-v4 chips, which may not be accessible to all teams, but the architectural principles scale down — even a 1.7B/64E model (27B total parameters, trainable on a smaller cluster) showed clear advantages over its dense counterpart in the paper's internal comparisons (Table 4, Figure 3).

Improving text generation quality in production systems where fluency matters. The paper's data quality ablation (Figure 3c–d) demonstrates that filtering the training data yields a larger improvement on NLG tasks (roughly 3–7 points) than on NLU tasks (roughly 1–2 points). The hypothesized mechanism — that "NLG often requires generating high-quality language and filtered pretraining corpora is crucial to the generation capability of language models" (Section 6.2) — has direct practical implications. An organization deploying a language model for text generation tasks (summarization, dialogue, creative writing, code documentation) should invest proportionally more in training data quality than an organization deploying for classification or information extraction. The specific filtering approach — training a linear classifier on feature-hashed text representations to distinguish curated from non-curated webpages, then sampling with a Pareto distribution rather than hard-thresholding — is described with enough detail (Section 3) to be reproduced, and the 50:1 reduction in web corpus size (from ~7T to 143B tokens) provides a concrete estimate of the filtering aggressiveness needed to achieve the reported quality gains. The Pareto sampling strategy — deliberately including some lower-quality pages "to prevent systematic biases in the classifier" — is particularly actionable, as many practitioners default to hard thresholding (keep only pages above some score cutoff) without considering that this amplifies classifier biases.

Selecting model architecture for fairness-sensitive applications. The WinoGender result — near-identical accuracy on stereotypical and anti-stereotypical examples (both 71.7%) and on male and female pronoun examples (70.8% vs. 72.5%) — is not conclusive about the causal mechanism (as discussed under Limitations), but it provides a datapoint that fairness-sensitive applications can factor into architecture selection. If a team is building a coreference resolution system for use in hiring, healthcare, or legal contexts where gender bias in pronoun resolution could cause real harm, the paper suggests (but does not prove) that large MoE architectures may be a safer default than dense alternatives. The appropriate practical response is not "always use MoE because it's fairer" — the evidence is far too thin for that — but rather "include WinoGender-style evaluations in your architecture selection process rather than assuming all large models have equivalent bias properties." The paper's replication of the co-occurrence analysis from Brown et al. (2020) (Tables 8–10) and toxicity analysis from Rae et al. (2021) (Figure 5) provides a template for the broader bias evaluation that should accompany architectural decisions, even though these analyses show GLaM behaving similarly to dense models on most bias metrics.

When to Prefer This Method

The paper does not position GLaM against a clearly articulated set of alternative architectures with explicit tradeoff criteria — it compares against GPT-3 as a fixed reference point and against internal dense baselines, but does not develop a decision framework for when practitioners should choose MoE over dense, or which MoE configuration is appropriate for which setting. The relevant guidance is implicit in the paper's results and can be extracted as conditional recommendations grounded in the experimental evidence:

Prefer a GLaM-style sparsely activated MoE architecture when:

Your inference volume is high enough that per-token FLOPs dominate total cost. The 48.6% reduction in per-token FLOPs (180 vs. 350 GFLOPs/token, Table 1) translates to near-halving of inference energy and compute cost when throughput-bound. This advantage is strongest for high-traffic production systems where hardware is continuously utilized. The paper's caveat about low-traffic serving (Section 8 — "serving cost especially when the serving traffic is low") implies the opposite: for low-volume, latency-sensitive applications, the higher memory footprint and device count may negate the per-token savings.
Your application is knowledge-intensive rather than pure-reasoning. The strongest MoE advantages are on open-domain QA (TriviaQA +7 points zero-shot, NQS +10.1 points, Table 11) and reading comprehension (SQuADv2 +11.6 points F1, DROP +33.7 points F1), which test factual knowledge retrieval. On LAMBADA (a word prediction task requiring broader discourse understanding), GLaM underperforms GPT-3 in zero-shot (64.2 vs. 76.2). If your task primarily requires recalling and synthesizing factual information, the increased total parameter count of MoE models provides capacity advantages that translate to accuracy gains.
You have the device count to store the full model. A 1.2T parameter model requires substantially more total device memory than a 175B dense model, even though per-token FLOPs are lower. The paper trained on 1,024 TPU-v4 chips (Section 5.2), and while smaller MoE variants require proportionally fewer devices (a 1.7B/64E model at 27B total parameters is far more manageable), the principle remains: MoE trades memory for compute. Organizations with access to large accelerator clusters will find this trade favorable; organizations constrained by total device count may not.
You can invest in high-quality training data filtering. The paper's data quality ablation (Figure 3c–d) shows that filtering provides a 3–7 point NLG improvement at the 1.7B/64E scale. The paper does not test whether this benefit interacts with MoE architecture, but the finding that data quality matters independently of scale suggests that MoE models — which have greater capacity to memorize patterns — may particularly benefit from clean data. Organizations unable to invest in data curation (e.g., because their training corpus has no quality signals) may not realize the full performance advantage the paper reports.

Prefer a dense architecture when:

Serving traffic is low or bursty. The paper explicitly notes that MoE models "increase the serving cost especially when the serving traffic is low" (Section 8) because the devices storing idle expert parameters must remain provisioned. For applications with sporadic usage (e.g., an internal research tool queried a few hundred times per day), the device cost of holding 1.2T parameters in memory likely dominates any per-token FLOPs savings.
Your application is latency-critical at small batch sizes. MoE inference involves all-to-all communication for token routing (Section C) whose overhead becomes proportionally larger at batch size 1, where the communication latency cannot be amortized across many tokens per expert. The paper does not report latency measurements, but the architecture's communication pattern is inherently more complex than a dense model's. For real-time applications (interactive dialogue, live code completion) where per-token wall-clock latency is the binding constraint, a dense model of equivalent total FLOPs may deliver lower latency even at higher FLOPs-per-token.
Reasoning depth rather than knowledge breadth is the primary requirement. If the paper's knowledge-vs-reasoning interpretation is correct (the strongest MoE gains are on knowledge-recall benchmarks), tasks requiring sustained logical inference (multi-hop reasoning over structured premises, complex mathematical proofs, code generation requiring algorithmic reasoning) may not benefit from the additional expert capacity in the same way. The paper's results on ANLI (adversarial natural language inference) — GLaM 44.7 few-shot vs. GPT-3 40.2, a solid but not dramatic gain — are consistent with this pattern, but systematic evaluation on reasoning benchmarks is needed to make this tradeoff operational.