Memory Augmented Language Models through Mixture of Word Experts

ArXiv: 2311.10768

🎯 Pitch

Language models can store massive world knowledge by giving each word its own tiny, dedicated neural network—activated only when that word appears—so that a 45B-parameter sparse model matches the performance of a 13B-parameter dense model on TriviaQA while using 6.6× fewer FLOPs. Fixed vocabulary-based routing turns word identity into a simple hash lookup, eliminating the need for learned gating or complex memory retrieval mechanisms.

---

1. Executive Summary

This paper proposes Mixture of Word Experts (MoWE), a novel neural architecture that combines the FLOPs efficiency of Mixture-of-Experts with the knowledge-storage capacity of memory-augmented models by routing tokens to tens of thousands of small, word-specific experts through a fixed hash function over a large knowledge-rich vocabulary (e.g., the word "Turing" always activates the same expert). Evaluated against the T5.1.1 family on knowledge-intensive tasks like TriviaQA, WebQuestions, and FEVER, MoWE-Base outperforms T5.1.1-XL while achieving a ~4.3× training speedup, and MoWE-Large matches or exceeds T5.1.1-XXL with a ~6.6× training speedup — all while using an order of magnitude fewer FLOPs per token. On SuperGLUE, MoWE matches comparably sized regular MoE models, establishing that the fixed lexical routing and extremely sparse expert activation confer disproportionate benefits on tasks requiring memorization and retrieval of world knowledge, while remaining competitive on general language understanding.

2.

3. Technical Approach

3.1 Reader Orientation (Approachable Technical Breakdown)

MoWE is a Transformer-based language model where a small subset of the feed-forward network layers are replaced by sparse layers containing tens of thousands of tiny, word-specific sub-networks, with each input token routed to exactly one sub-network based on a fixed, vocabulary-driven hash function rather than a learned router. The system solves the problem of increasing a model's knowledge capacity without proportionally increasing its computational cost: by giving each word its own dedicated "expert" that activates only when that word appears, the model can store far more world knowledge (31B–45.5B parameters) while performing inference with roughly the same FLOPs as a small dense model, yielding dramatic improvements on knowledge-intensive tasks like TriviaQA at a fraction of the training and inference cost of equivalently performing dense models.

3.2 Big-Picture Architecture (Diagram in Words)

The MoWE architecture consists of five major components integrated into a standard T5.1.1 encoder-decoder Transformer:

1. Default Tokenizer and Embedding Layer — the standard T5 SentencePiece tokenizer (32K vocabulary) that converts input text into token IDs and corresponding dense embeddings, defining the sequence length and the base representations that flow through the Transformer stack.

2. Routing Tokenizer — a separate, large auxiliary vocabulary (~1M entries derived from Wikidata entity/relation names) that tokenizes the same input text independently, producing a "routing id" for each input position. This routing id determines which expert will process that position's representation in the MoWE layers.

3. MoWE Layers — four sparse layers (two in the encoder, two in the decoder) inserted at specific Transformer block positions. Each MoWE layer contains 32K small feed-forward networks (experts), grouped into blocks and frequency buckets. A token's representation enters the MoWE layer, gets dispatched via all-to-all communication to the single expert corresponding to its routing id, is transformed by that expert, and is then returned to its original sequence position.

4. Standard Dense Transformer Blocks — the remaining T5.1.1 blocks (self-attention, cross-attention, dense FFNs) that provide contextualization, interaction between positions, and standard language modeling capability. The MoWE layers are embedded within this stack, with dense blocks both before and after them.

5. Frequency Bucketing and Expert Blocks Infrastructure — the implementation substrate that makes 32K–1M experts feasible on TPU hardware. It pre-sorts routing vocabulary entries into frequency buckets (so frequently occurring words don't overwhelm the buffers of rare-word experts), groups experts into blocks for efficient all-to-all communication, and uses a hierarchical dispatch that resolves token→expert mapping statically at compilation time.

Information flows as follows: input text → dual tokenization (default + routing) → embedding lookup → first few dense Transformer blocks → MoWE layer in encoder (tokens dispatched to word-specific experts, transformed, returned) → more dense encoder blocks → second encoder MoWE layer → encoder output → decoder with its own MoWE layers at parallel positions → output projection to vocabulary.

3.3 Roadmap for the Deep Dive

• First, the core MoWE layer mechanism — what happens when a token hits a MoWE layer, how routing differs from learned MoE routers, and why fixed lexical routing is the central design choice.
• Second, the knowledge-rich routing vocabulary construction — because the entire system's effectiveness hinges on having a vocabulary where entries correspond to meaningful, knowledge-bearing words rather than subword fragments.
• Third, the infrastructure for scaling to 32K–1M experts — frequency bucketing, expert blocks, hierarchical routing, and how these overcome the communication and load-balancing challenges that make large-scale MoE training difficult.
• Fourth, the pretraining and finetuning protocol — what data, what objective, what hyperparameters, and the critical decision to freeze experts during finetuning.
• Fifth, the MoWE-Base and MoWE-Large configurations — the specific numbers of layers, experts, blocks, bucket sizes, and MLP dimensions that define the two main model scales.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an architecture design and empirical validation paper whose core idea is that replacing a small number of FFN layers with extremely sparse, vocabulary-routed word experts creates a model that functions as a memory-augmented architecture — storing factual knowledge in the experts — while preserving the FLOPs efficiency of sparse MoE models and avoiding the training complexity of learned routers or external memory retrieval.

---

The MoWE Layer: Token-Level Expert Selection via Fixed Lexical Routing

What a MoWE layer replaces. In a standard Transformer block, the feed-forward network (FFN) takes each position's representation (output of the self-attention or cross-attention sublayer) and applies the same two-layer MLP to every position independently. In MoWE, a subset of these FFN layers — specifically, four of them across the full encoder-decoder stack — are replaced by a MoWE layer, which contains a pool of many small FFNs (the "experts") and a routing mechanism that selects exactly one expert per token position.

How routing works — the defining mechanism. Unlike standard MoE models (Shazeer et al., 2017; Lepikhin et al., 2020), where a learned routing function computes dot-product scores between each token representation and each expert embedding and then selects the top-k experts, MoWE uses a fixed, vocabulary-determined routing:

1. Dual tokenization of the input. The input text is tokenized twice: once by the default T5 SentencePiece tokenizer (32K vocabulary), producing the sequence of token IDs that define the input embeddings and sequence length; and once by a separate routing tokenizer with a much larger vocabulary (~1M entries), producing a parallel sequence of "routing ids" — one per input position.

2. Mapping routing ids to expert ids. Each routing id is mapped to a specific expert through a hash function. In the extreme case (used for frequency buckets 1–3 in the main configuration), this is a one-to-one mapping: routing id i is always handled by expert i. In bucket 4 (low-frequency words), multiple routing ids share the same expert, so the mapping is many-to-one — but still fixed and pre-determined by the vocabulary construction and bucketing.

3. Static dispatch. Because the routing function is purely a function of the input token identities (not their contextualized representations), the token→expert assignment is known the moment the input is tokenized — before any Transformer computation begins. This is a fundamental departure from learned routers, where the assignment changes dynamically during training and inference based on the evolving token representations.

What the expert does. An expert is a small two-layer MLP (feed-forward network) with its own weight matrix. When a token is routed to expert e, that token's representation (a dense vector of dimension d_model) is multiplied by the expert's weight matrices, producing a transformed vector of the same dimension. The transformation is the standard FFN computation (linear projection → activation → linear projection), just with a much smaller hidden dimension than a dense FFN. Specifically, in MoWE-Base:

• Bucket 1 experts (handling the 128 most frequent knowledge-bearing routing tokens): MLP hidden dimension = 2048, which matches the dense FFN size but serves only a single token.
• Bucket 2 experts (handling 896 tokens): MLP hidden dimension = 2048.
• Bucket 3 experts (handling 1024 tokens): MLP hidden dimension = 1024.
• Bucket 4 experts (handling ~220K tokens, shared across 30,080 experts): MLP hidden dimension = 512.

The hidden dimension determines the expert's capacity to store information: larger hidden dimensions can memorize more patterns associated with that word. The paper's use of decreasing expert sizes with decreasing token frequency (from 2048 down to 512) reflects an intuitive principle — frequent words appear in more diverse contexts and therefore need more capacity to capture their varied associations, while rare words appear in fewer, more specific contexts and can be adequately served by smaller networks.

How tokens pass through the MoWE layer — the dispatch-compute-collect cycle. For a batch of input sequences, the MoWE layer performs these steps:

1. Identify assignments: Using the pre-computed routing ids for each token position, determine which expert each token belongs to.

2. Dispatch via all-to-all communication: Tokens whose assigned experts reside on different devices (TPU chips) are sent to their target device. This is the same all-to-all collective used in standard MoE models, but the communication pattern is fixed and known at compilation time because the routing is static — enabling optimizations that dynamic routers cannot use.

3. Expert computation: On each device, the tokens assigned to the experts on that device are processed by their respective experts. Tokens assigned to the same expert are batched together and processed as a single matrix multiplication (since the expert is just an MLP, multiple tokens can be processed by the same expert in parallel).

4. Collect via inverse all-to-all: The transformed token representations are sent back to their original devices and positions.

5. Residual connection: As with a standard FFN, the output of the MoWE layer is added to its input via a residual connection.

The result is that each token position has been transformed by a word-specific sub-network, with different words pulling in different stored knowledge. The token "Turing" activates the expert that has been trained to associate "Turing" with relevant facts (computer science, Alan Turing, Turing Award, etc.), while the token "mathematician" activates a different expert trained on associations specific to that word.

Why fixed lexical routing instead of learned routing? The paper's design choice is motivated by several considerations:

1. Encouraging expert specialization on specific words. If the router is learned, there is no guarantee that a particular word always routes to the same expert — the routing could depend on context, position, or other factors. Fixed lexical routing forces each expert to specialize on the content that co-occurs with its assigned word(s). As the paper states:

"We conjecture that the large routing vocabulary and associated large number of experts further encourage the MoWE layer to function as a sparse memory. We find that using complete words instead of word pieces to perform routing is a strong inductive bias that makes it easier for the experts to specialize on specific words. For example, the expert for the word 'Turing' will be activated only when that word appears in the input, and therefore will be specialized on content that co-occur with that word."

2. Eliminating the need for auxiliary load-balancing losses. Standard MoE models require an additional loss term to encourage balanced expert utilization, because learned routers tend to collapse to a small number of experts. MoWE's fixed routing means the load on each expert is determined purely by token frequency in the data, which is known in advance and can be accommodated through the frequency bucketing infrastructure.

3. Compilation-time optimization of communication. Because the routing pattern is static, the compiler can pre-allocate buffers, pre-schedule communication, and avoid the runtime dispatch overhead that dynamic routers incur.

4. Scaling to extreme numbers of experts. Previous work (Du et al., 2022) found diminishing returns beyond ~64–128 experts with learned routing, likely because the router's scoring matrix (which must compute num_tokens × num_experts dot products) becomes a bottleneck and the training signal for any individual expert becomes too sparse. MoWE sidesteps both problems: there is no router scoring matrix (routing is a O(1) table lookup), and the training signal for each expert is proportional to the frequency of its assigned word, which is non-zero by construction.

Parameter sharing across MoWE layers. A distinctive design choice: the expert parameters are shared across all four MoWE layers (two in the encoder, two in the decoder). The paper gives two reasons:

"(1) it makes the MoWE layer even more similar to a memory that is accessed at different points of the network; (2) we can keep the overall number of sparse parameters relatively low without the need to decrease the total and the size of experts."

Additionally, "empirical results indicated that sharing parameters across the MoWE layers leads to better performance." This sharing means the total expert parameter count is not 4 × 32K × expert_size but just 1 × 32K × expert_size, yielding 31B sparse parameters for MoWE-Base rather than ~124B. The expert memory is accessed at multiple depths in the network — early encoder (block 5), late encoder (block 10), early decoder (block 5 of encoder layers, corresponding position), late decoder (block 10) — with each access drawing from the same knowledge store but integrating the expert output into a different level of contextualization.

Positioning within the Transformer stack. MoWE layers are "placed near the middle of the encoder (decoder)" to satisfy two design goals:

"(1) the MoWE layers receive a representation of the token that is already somewhat contextualized; (2) after the MoWE layer, there are still multiple Transformer Blocks that can benefit from the output of that layer."

In MoWE-Base (12 encoder blocks, 12 decoder blocks), the MoWE layers are at encoder blocks 5 and 10, and decoder blocks 5 and 10. This means tokens hitting the first MoWE layer have already undergone 4 blocks of self-attention, giving them a contextualized representation that includes information from surrounding tokens — the expert for "Turing" sees not just the token embedding but a representation that already incorporates "Alan Turing was a..." from surrounding tokens. After the second MoWE layer in the encoder, there are 2 more encoder blocks to further integrate the expert's output with the full sequence context.

---

The Knowledge-Rich Routing Vocabulary

Why the routing vocabulary matters. The entire MoWE mechanism depends on the quality of the routing vocabulary. If the vocabulary consisted primarily of subword fragments (like "tion" or "pre"), routing decisions would be essentially arbitrary — no meaningful word-level knowledge could be associated with such fragments. If the vocabulary contained only very frequent words, most tokens would fall back to shared experts, diluting the specialization the architecture aims to achieve. The paper therefore invests significant design effort in constructing a vocabulary that maximizes coverage of knowledge-bearing terms.

Construction procedure — a four-step pipeline. The routing vocabulary is built as follows:

Step 1: Seed from Wikidata. The authors "start with the set of all entity and relation names that appears in a Wikidata dump." This ensures the vocabulary is seeded with proper names, technical terms, and domain-specific concepts — exactly the kinds of words for which factual knowledge needs to be stored. Crucially, this step biases the vocabulary toward single-token coverage of named entities. For example, "mathematician" appears as a single token in the routing vocabulary, whereas the default T5 32K tokenizer breaks it into 5 subword tokens ("math", "e", "m", "a", "tician").

Step 2: Normalize and split. Each name is lowercased and split using whitespace and a regex to remove punctuation. The paper notes that "languages that do not use white space for word splitting will require slightly modified processing."

Step 3: Frequency-based ordering on C4. The resulting tokens are ordered by their frequency in the C4 dataset (version 2.2.0), which is the pretraining corpus. This step ensures that the most useful entries — those that actually appear in the training data — are prioritized.

Step 4: Select top 1M. The top 1 million tokens by C4 frequency form the routing vocabulary. The paper reports that "this strategy increases the likelihood that the majority of entries in the vocabulary are (single word) names — i.e., terms that we want to store knowledge about."

What the routing vocabulary looks like in practice. Appendix D provides samples:

• Top 50 words by C4 frequency: "isn", "aren", "...", "3d", "1st", "whilst", "copyright", "creates", "2nd", etc. These are mostly common words and their variants.
• After position 6000: "consignment", "billboards", "primal", "discrepancy", "callback", "freeware", "horticulture", "jb", "s8", "aspirants", "commemorative", "brisk", "arched", "pondering", "fluff", "diwali", "landline", etc.

The vocabulary quality improves further into the tail, where entries become more specialized and knowledge-bearing. The paper acknowledges that "more work can definitely be done to improve the routing vocabulary, but we wanted to keep it simple for our experiments."

How routing tokenization works at inference time — the hash-based alignment problem. Because the routing vocabulary uses whole words while the default tokenizer produces subword pieces, there is no guarantee that token boundaries align. The paper solves this with an offline-online hash lookup:

• Offline: The auxiliary vocabulary is extended by concatenating the default T5 32K vocabulary to it (ensuring every default token has some routing id). Each entry in the auxiliary vocabulary is tokenized using the default tokenizer, and a hash table is built where the key is the sequence of default token ids and the value is the routing id.

• Online: Given a tokenized input sequence s of n token ids {t₁, t₂, ..., tₙ}, the routing id for token tᵢ is determined by looking up all sub-sequences {tᵢ₋ₖ, ..., tᵢ} for k ∈ [0, 8] in the hash table, and adopting the routing id of the largest sub-sequence found. This greedy longest-match approach means that when the default tokenizer produces multiple tokens for a single word (e.g., "mathematician" → "math" + "e" + "m" + "a" + "tician"), the routing id for the final token "tician" will match the longest sequence (the full "mathematician") and route accordingly, while the earlier sub-tokens may match shorter entries.

The autoregessive generation challenge. The paper notes that this approximate routing is also necessary "to allow... the use of the MoWE layer in auto-regressive scenarios where normally only the initial part of the word is known." During autoregressive decoding, at the moment the model generates the token "math" (the first part of "mathematician"), it doesn't yet know that the full word is "mathematician" — so the routing id for that position must be based on the prefix seen so far, which will map to a shorter vocabulary entry. This introduces some routing noise during generation, but the paper's strong overall results suggest the approximation is adequate.

---

Infrastructure for Scaling to 32K–1M Experts: Frequency Bucketing, Expert Blocks, and Hierarchical Routing

The three challenges of large-scale MoE with fixed routing. The paper identifies three key obstacles to implementing MoWE efficiently on TPU hardware using the standard single-program-multiple-data (SPMD) parallelism strategy that underlies most MoE implementations:

"(1) The sheer number of experts brings an unpractical overhead in terms of all-to-all communication. (2) Word frequency follows a Zipfian-like distribution. This unbalanced nature of vocabulary-driven routing requires different word experts to process orders of magnitude more tokens than others."

To these we can add a third: (3) extremely small experts (MLP hidden dimension as low as 8 or 96) would be inefficient on matrix-multiplication-optimized accelerators if each expert were computed individually without batching.

Solution component 1: Expert Blocks. Rather than sharding individual experts across devices — which would require num_experts all-to-all communication groups — experts are grouped into blocks, and all-to-all communication is performed only between blocks. The paper explains:

"All-to-all communication is only performed between blocks instead of between experts. Provided we keep the number of expert blocks small enough, we can increase the number of experts without increasing all-to-all communication costs. For example, if we use 128 blocks with 256 experts each, we end up with 32768 experts."

The key insight: the number of communication groups scales with the number of blocks (128), not the number of experts (32K+). Inside each block, routing to the specific expert is a local operation with no cross-device communication. This makes the communication cost of a 1M-expert MoWE layer identical to that of a 128-expert MoE layer, provided both use 128 blocks.

Solution component 2: Frequency Bucketing. Because token frequency follows a Zipfian distribution, a tiny number of tokens (punctuation, "the", "of") appear millions of times, while most tokens appear only a handful of times. If all experts shared the same capacity buffer (the maximum number of tokens they can process per batch), either rare-word experts would have mostly empty buffers (wasting compute) or frequent-word experts would overflow (dropping tokens). Standard MoE models solve this with a load-balancing loss that encourages uniform assignment, but MoWE's fixed routing prevents this.

Instead, the paper computes the frequency of each routing vocabulary entry on a sample of 2B pretraining tokens, then splits the vocabulary into k frequency buckets where each bucket contains words of approximately equal frequency. In the main MoWE-Base configuration, k = 4:

Each bucket is handled by a separate set of expert blocks, conceptually executing "k MoWE layers... in parallel." Buckets can have different expert sizes, different numbers of experts per block, and different token capacities. Bucket 1, containing the most frequent knowledge-bearing tokens (roughly the top 128 tokens by frequency after excluding the top 16 most frequent tokens which are never routed — these are "punctuation marks and other non-content words" that "can represent up to 28% of the tokens in a batch," and skipping them "speeds up the training time and does not hurt downstream performance"), uses 2048-dimensional experts — as large as a dense FFN, but dedicated to single tokens. Bucket 4, handling the long tail of ~220K rare tokens, uses 512-dimensional experts and shares 30,080 experts across all those tokens, with an average of ~7.3 tokens per expert.

Why different expert sizes per bucket? The paper does not explicitly justify this choice in detail, but the logic is clear from the design: frequent words appear in many different contexts and need more capacity to store their diverse associations (e.g., "president" appears with many different people, countries, policies), while rare words appear in fewer, more specific contexts and can be adequately served by smaller networks. The ratio of expert size across buckets (2048:1024:512 ≈ 4:2:1 for buckets 1-2, 3, 4) roughly mirrors the ratio of expected diversity of usage.

Solution component 3: Hierarchical Routing. The full dispatch path for a token in a batch is:

1. Route to frequency bucket: Based on the token's routing id, determine which of the four buckets it belongs to. Tokens in bucket 1 go to the bucket-1 expert blocks, tokens in bucket 2 to bucket-2 blocks, etc. Tokens with routing id among the top 16 most frequent are skipped entirely.

2. Route to expert block within bucket: Within each bucket, tokens are assigned to the specific block (out of 128 blocks per bucket) that contains their expert. For buckets 1-3, the mapping is one-to-one (one expert per routing id) and pre-computed; for bucket 4, the mapping is a hash of the routing id modulo the number of bucket-4 experts.

3. Route to expert within block: Inside the block, each token is routed to and processed by its specific expert. For bucket 4, where blocks contain 235 experts each, this is a local lookup within the 128-expert block.

The paper emphasizes that "since routing decisions are based purely on (static) routing ids, token-to-expert assignments are known beforehand and the full path through the hierarchical routing tree becomes trivial." This is a key advantage: the entire dispatch pattern can be computed at compilation time, allowing the XLA compiler to optimize memory allocation, communication scheduling, and kernel fusion in ways that dynamic routers cannot.

Implementation detail: Bucket 4 uses lookup tables. Because bucket 4 contains 30,080 experts that are extremely small (MLP dim 512 for Base, or as small as 96 for the 2B parameter variant), the expert blocks in this bucket are implemented as lookup tables rather than as individual weight matrices. This means the computation for an entire block of 235 experts is performed as a single batched matrix operation, which is far more efficient on TPUs than 235 separate small matrix multiplications. The paper notes that "using a larger number of smaller experts is preferable because it is more memory efficient and also speeds up our lookup table implementation of Expert Blocks in frequency bucket 4."

Scaling to 1 million experts. To demonstrate the robustness of their infrastructure, the paper trained MoWE-Base variants "with up to 1 million (small) experts using 16 v3 TPUs." In the 1M-expert configuration, the experts are extremely small (MLP hidden dimension of 8) to keep the total parameter count fixed, and the bucket-4 blocks contain many more experts per block. The paper reports that "we did not observe any training instability (e.g. gradient blowup) that are often reported in the pretraining of regular MoE models" and hypothesizes that "this is a helpful artifact of our fixed routing scheme."

---

Pretraining and Finetuning Protocol

Pretraining objective and data. MoWE models are pretrained using the same span masking (denoising) objective as T5 (Raffel et al., 2020). The pretraining corpus is C4 version 2.2.0. The paper uses "the same pretraining hyperparameters of T5.1.1" — meaning the AdamW optimizer with the T5 learning rate schedule (inverse square root), though the exact learning rate values are not restated in the MoWE paper (they reference the T5.1.1 defaults).

Pretraining scale. The main results use models pretrained for "roughly 1 trillion tokens — 1M steps, with batch size 2048 and input sequence length of 512 tokens; the target sequence length is 114." Computing the total tokens: 1M steps × 2048 batch size × (512 input + 114 target) = 1M × 2048 × 626 ≈ 1.28 trillion tokens, consistent with "roughly 1 trillion tokens" after accounting for padding and variable-length sequences. Pretraining uses 64 TPUs v3.

No auxiliary losses. A notable distinction from standard MoE training: "we only use cross-entropy loss; no additional auxiliary losses are used." This is possible because the fixed routing eliminates the need for the load-balancing loss that standard MoE models require to prevent router collapse. The cross-entropy loss is the standard language modeling objective: given the corrupted input (with spans masked out), predict the original uncorrupted tokens.

Finetuning with frozen experts — a critical design decision. During finetuning on downstream tasks, all MoWE expert parameters are frozen. The paper states the rationale:

"We freeze all MoWE experts to avoid both overfitting and catastrophic forgetting of knowledge acquired during pretraining."

The ablation in Appendix B.0.3 supports this: MoWE-Base on TriviaQA achieves an exact match of 37.7 with frozen experts, which drops by 5 points to 33.5 when experts are allowed to update during finetuning. This is consistent with the interpretation of experts as a memory: if the knowledge is stored in the pretrained expert weights, finetuning on a narrow downstream dataset risks overwriting that knowledge (catastrophic forgetting) or overfitting to the limited finetuning examples.

The only part of the model trained during finetuning is the dense Transformer parameters (attention weights, layer norms, dense FFN weights in non-MoWE layers, embedding, and output projection). The main hyperparameter tuned during finetuning is the learning rate. For most QA datasets (TriviaQA, WebQuestions, Natural Questions), a learning rate of 1e-4 and dropout rate of 0.05 gave the best results; for SuperGLUE and FEVER, higher learning rates between 1e-3 and 5e-4 worked better. Batch sizes are 256 or 512.

Fine-grained evaluation protocol for QA tasks. Following Roberts et al. (2020), the model has "no access to external knowledge/text during finetuning and inference" — all knowledge must come from parameters. For TriviaQA, WebQuestions, and Natural Questions, 10% of the training set is held out as validation, and models are finetuned on the remaining 90%. Evaluation metrics are exact match (EM), with F1 used for some ablation experiments because it "is slightly less noisy and highlights the trends more clearly." For FEVER, accuracy is reported on both validation and test sets. For SuperGLUE, following T5 convention, models are finetuned on a mixture of all SuperGLUE tasks, the best checkpoint per task is selected, and the average validation score over all tasks is reported.

---

MoWE-Base and MoWE-Large Configurations

MoWE-Base (31B sparse parameters). The base configuration uses T5.1.1-Base as the backbone:

• Encoder: 12 Transformer blocks, d_model = 768, 12 attention heads.
• Decoder: 12 Transformer blocks, d_model = 768, 12 attention heads.
• MoWE layers at encoder blocks 5 and 10 and decoder blocks 5 and 10 (4 layers total, parameters shared).
• 32K experts total, configured as in Table 6: Bucket 1 (128 experts, MLP dim 2048), Bucket 2 (896 experts, MLP dim 2048), Bucket 3 (1024 experts, MLP dim 1024), Bucket 4 (30,080 experts, MLP dim 512).
• Expert blocks per bucket: 128.
• Total sparse parameters: ~31B.

MoWE-Large (45.5B sparse parameters). The large configuration uses T5.1.1-Large as the backbone:

• Encoder: 24 Transformer blocks, d_model = 1024, 16 attention heads.
• Decoder: 24 Transformer blocks, d_model = 1024, 16 attention heads.
• MoWE layers at encoder blocks 9 and 17 and decoder blocks 9 and 17 (4 layers total, parameters shared).
• Same number of experts (32K) and same bucketing structure as Base.
• Expert MLP dimensions scaled up: Bucket 1 (MLP dim 2816), Bucket 2 (MLP dim 2816), Bucket 3 (MLP dim 1536), Bucket 4 (MLP dim 512 — the same as Base, because these are extreme-tail experts and adding capacity to 30K small networks would explode the parameter count).
• Total sparse parameters: ~45.5B.

The ratio of expert dimensions from Base to Large (2048→2816 for buckets 1-2, 1024→1536 for bucket 3) roughly mirrors the d_model scaling ratio (768→1024), keeping the experts' capacity proportional to the model's representational dimension.

How FLOPs compare to dense models. A MoWE layer processes each token with exactly one expert — a small MLP with hidden dimension 512–2048 (Base) or 512–2816 (Large). A dense FFN in T5 processes each token with a single large MLP with hidden dimension d_ff (2048 for Base, 2816 for Large). Since most MoWE experts are smaller than the dense FFN they replace (bucket 4 experts are 512 vs. 2048/2816), and since only 4 out of 24 Transformer blocks use MoWE layers (the other 20 use standard dense FFNs), the total FLOPs per token for MoWE is only slightly higher than for the corresponding T5 dense model — despite having 31B or 45.5B additional parameters. The paper states that MoWE-Base and MoWE-Large have "similar number of FLOPs" to T5-Base and T5-Large respectively, which is what enables the dramatic training speedups (2.0× the training time of T5-Base for MoWE-Base, vs. 8.6× for T5-XL and 26.4× for T5-XXL).

---

Summary of Design Choices and Their Justifications

• Fixed lexical routing over learned routing: eliminates load-balancing losses, enables compilation-time communication optimization, encourages word-level expert specialization, and scales to tens of thousands of experts without router computation overhead.
• Wikidata-seeded routing vocabulary over SentencePiece-trained vocabulary: ensures vocabulary entries are knowledge-bearing single words rather than subword fragments, providing the inductive bias for expert specialization on named entities and concepts.
• Four frequency buckets with decreasing expert sizes: the Zipfian distribution of word frequencies means different experts need different capacities and different load-handling strategies; bucketing allows this while keeping the communication pattern manageable.
• 128 expert blocks per bucket: caps all-to-all communication overhead at 128 communication groups regardless of total expert count (32K, 64K, or 1M).
• Parameter sharing across MoWE layers: reduces total parameter count by 4×, reinforces the "sparse memory" interpretation (same knowledge accessed at different depths), and empirically improves performance over non-shared variants.
• MoWE layers at mid-network positions: early enough that there are dense blocks after them to integrate expert outputs into deeper representations; late enough that tokens have some contextualization before expert lookup.
• Freezing experts during finetuning: prevents catastrophic forgetting of pretrained knowledge and overfitting to narrow downstream datasets, supported by a 5-point exact-match drop in the ablation.
• Skipping top-16 most frequent tokens: these are non-content words (punctuation, "the", etc.) that would consume ~28% of MoWE computation without contributing to knowledge retrieval.

4. Key Insights and Innovations

Innovation 1: Fixed Lexical Routing as a Sufficient and Advantageous Alternative to Learned Routing for Extreme-Scale MoE

The dominant assumption in mixture-of-experts modeling, from Shazeer et al. (2017) through GShard (Lepikhin et al., 2020), Switch Transformer (Fedus et al., 2022), and GLaM (Du et al., 2022), has been that routing must be learned jointly with the model parameters — the router function computes token-expert affinity scores, selects the top-k experts, and is trained via gradient descent alongside everything else. This learned routing is treated as essential because the alternative seems untenable: how could a fixed, vocabulary-based assignment possibly be optimal when the same word can mean different things in different contexts?

MoWE challenges this assumption with a finding that is both counterintuitive and practically powerful: fixed lexical routing not only works, but works better than learned routing on knowledge-intensive tasks, and enables scaling to expert counts (32K–1M) that learned routers cannot reach. The paper demonstrates this directly in Table 2: a MoWE-Base model with 32K fixed-routed experts achieves 39.4 EM on TriviaQA versus 36.2 for an MoE-Top2 model with 512 learned-routed experts per layer, despite both having ~30B parameters. On SuperGLUE, they match (83.5 each), confirming that the fixed routing is not universally superior but is specifically advantageous for factual recall.

What makes this a genuine conceptual advance rather than a mere implementation trick is the identification of routing as a design choice with fundamentally different scaling properties depending on the task family. Learned routing optimizes for representational flexibility — the ability to route the same word to different experts depending on context, which is valuable for tasks requiring nuanced language understanding (SuperGLUE). Fixed lexical routing optimizes for storage locality — forcing each word's knowledge to be stored in a dedicated, always-accessed location, which is valuable for tasks requiring factual retrieval (TriviaQA, FEVER). Prior MoE work conflated these two goals; MoWE separates them and shows they trade off against each other.

This framing also explains why Du et al. (2022) found diminishing returns beyond ~64–128 experts with learned routing: the learned router's scoring matrix must compute num_tokens × num_experts dot products, and the training signal per expert becomes vanishingly sparse as expert count grows. Fixed routing has neither bottleneck — the router is a table lookup, and each expert receives training signal proportional to its word's frequency, which is non-zero by construction. The paper demonstrates this scalability concretely by training models with up to 1 million experts, a regime that is essentially inaccessible to learned-routing MoE.

The significance beyond raw performance: MoWE reframes routing not as a learning problem to be solved but as an architectural inductive bias to be deliberately chosen based on what you want the experts to specialize in. If you want experts to represent context-dependent subfunctions (the standard MoE goal), learn the router. If you want experts to function as a sparse memory for word-associated facts (the knowledge-intensive goal), fix the router to vocabulary entries. This is a conceptual move from "we must learn how to route" to "routing is a design dimension with task-dependent optimal settings."

The supporting evidence is in Table 2 (MoWE vs. MoE-Top2 at matched parameter counts on both knowledge and general NLP tasks) and the routing vocabulary ablation in Figures 4–5, which show monotonic improvement as the vocabulary grows from 32K to 1M entries — the larger the knowledge-rich vocabulary, the better the fixed routing performs, consistent with the storage locality interpretation.

Innovation 2: Decoupling Knowledge Capacity from Compute via Word-Level Expert Specialization

The field has long understood that larger models store more world knowledge, as evidenced by the scaling laws (Kaplan et al., 2020) and the strong performance of massive dense models on closed-book QA (Roberts et al., 2020; Chowdhery et al., 2022). The standard approach to increasing knowledge capacity is to scale the entire model — more layers, wider FFNs, more attention heads — which proportionally increases both parameter count and FLOPs. Recent work has attempted to decouple these through retrieval augmentation (Guu et al., 2020; Borgeaud et al., 2022), where knowledge is stored in an external corpus and retrieved at inference time, but this introduces a separate retrieval subsystem, a non-differentiable search step, and dependence on a curated knowledge corpus.

MoWE introduces a third pathway: store knowledge in sparsely-activated, word-specific parameters that are part of the model itself, accessed through the normal forward pass with no retrieval step, but incurring almost no additional FLOPs because each expert activates only for its specific word. The conceptual move is subtle but significant: rather than asking "how do we make the model bigger to store more knowledge?" or "how do we retrieve knowledge from an external source?", MoWE asks "how do we co-locate knowledge with the words that name it?" This reframes the knowledge storage problem as one of architectural sparsity rather than model scale or retrieval mechanics.

What distinguishes this from standard MoE sparsity: in a Switch Transformer, any token can route to any expert, and the experts learn distributed, compositional representations that aren't tied to specific lexical items. In MoWE, the expert for "Turing" is only activated when the word "Turing" appears, and thus specializes on facts that co-occur with that word — Turing's biography, the Turing Award, the Turing test. The paper makes this interpretation explicit:

"By using word-specific key-value memories (word experts), our hope is that MoWE can make it easier for the model to store and retrieve information about those words."

The significance beyond performance: this provides a new conceptual model for where factual knowledge lives in language models. Prior work (Geva et al., 2021; Dai et al., 2022; Meng et al., 2022) established that Transformer FFNs function as key-value memories and that factual associations can be localized to specific neurons or layers. MoWE takes this observation and turns it into a design principle: if knowledge is stored in FFN weights and if it's associated with specific lexical triggers, then giving each trigger word its own dedicated FFN should produce cleaner, less-interfering knowledge storage. The empirical evidence in Section 4.3 supports this interpretation dramatically: deactivating the expert for a single key word in a TriviaQA question (e.g., the expert for "Neptune" in "What is Neptune's main satellite?") causes the model to produce a completely wrong answer (Table 4), with a 9-point drop in exact match when experts for content words are disabled (Table 3). A single expert representing only 0.33% of activated parameters controls a substantial fraction of the model's ability to answer certain factoid questions — strong evidence for distributed-but-word-localized knowledge storage.

The insight's relationship to memory-augmented models (EaE, TOME, FILM) is also significant. These models inject external memory by embedding entity mentions, performing k-NN retrieval over a memory bank, and integrating the retrieved embeddings into the model's computation. This requires custom training procedures (entity-aware losses), domain-specific memory curation (Wikipedia entities), and separate retrieval infrastructure. MoWE achieves comparable or superior results (Table 5: MoWE-Large matches TOME on TriviaQA, outperforms on FEVER) while using no retrieval mechanism at all — the "memory" is just the expert weights, accessed through the standard forward pass. This demonstrates that for knowledge stored in pretraining and triggered by lexical cues, explicit retrieval is unnecessary; co-location of knowledge with trigger words is sufficient.

The FLOPs-matched comparison with T5 in Figure 1 and Table 1 provides the quantitative anchor: MoWE-Base outperforms T5-XL on TriviaQA (39.4 vs. 36.0 EM) while being 4.3× faster to train; MoWE-Large outperforms T5-XXL (44.8 vs. 42.9 EM) while being 6.6× faster. This is not an incremental improvement — it is a regime change in the cost of knowledge-intensive capability, made possible by the architectural insight that knowledge capacity and compute capacity can be almost fully decoupled through word-level sparsity.

Innovation 3: The Zipfian Nature of Language as an Architectural Opportunity Rather Than a Load-Balancing Problem

Standard MoE models treat the uneven distribution of tokens across experts as a problem to be solved, typically through auxiliary load-balancing losses that penalize unbalanced assignment (Shazeer et al., 2017; Fedus et al., 2022) or through expert-choice routing that lets each expert pick its top-k tokens to enforce perfect balance (Zhou et al., 2022). These solutions implicitly assume that the ideal state is uniform expert utilization — every expert processing the same number of tokens.

MoWE takes the opposite stance: the Zipfian distribution of word frequencies is not a bug to be corrected but a signal to be exploited. Words that appear frequently (thousands or millions of times in the pretraining corpus) genuinely have more varied associations and need more capacity; words that appear rarely need less. The frequency bucketing strategy (Section 2.3, Table 6) embraces this by giving frequent-word experts larger MLPs (hidden dimension 2048 for the top ~1000 tokens) and rare-word experts smaller ones (hidden dimension 512 for the ~30K long-tail experts), with bucket-4 experts shared across hundreds of routing vocabulary entries to prevent the long tail from consuming disproportionate memory.

This is a fundamental conceptual shift from the load-balancing paradigm to a capacity-adaptive paradigm. It recognizes that in a vocabulary-routed system, load imbalance is a feature of the data distribution, not a failure of the routing mechanism. The engineering challenge shifts from "how do we force even assignment?" to "how do we provision capacity efficiently given the known assignment distribution?" — which is a much more tractable problem because the distribution is static and can be measured once on a sample of pretraining data.

The innovation's significance extends beyond MoWE: it suggests that any system with fixed, data-driven routing (hash layers, domain-specific expert partitioning, language-specific routing in multilingual models) should similarly embrace rather than fight the underlying distribution, provisioning capacity proportionally to expected load rather than enforcing artificial balance. The paper's demonstration that this works at scale (32K–1M experts on 16–64 TPUs) with no training instability — "we did not observe any training instability (e.g. gradient blowup) that are often reported in the pretraining of regular MoE models" — is a strong practical validation of the approach. Training instability in MoE models (Zoph et al., 2022) is often attributed to the dynamic interaction between the learned router and the load-balancing loss; MoWE sidesteps both, and the resulting training stability is a significant practical advantage even if it wasn't the primary motivation.

The expert-sharing strategy for bucket 4 (30,080 experts for ~220K routing vocabulary entries, averaging ~7.3 vocabulary entries per expert) is a pragmatic compromise that acknowledges the limits of specialization in the long tail: tokens that appear only a few hundred or few thousand times in the entire pretraining dataset don't provide enough training signal for truly dedicated experts. By sharing experts among multiple low-frequency words, the model gets the benefit of expert specialization where it has sufficient data (buckets 1–3) without wasting parameters on experts that would be undertrained. This is an implicit form of statistical capacity allocation — matching the expert granularity to the expected training signal — that is novel in the MoE literature.

The evidence supporting this as a genuine innovation rather than an engineering convenience is the ablations in Figures 6 and 7. Figure 6 shows that 32K experts is a sweet spot for a 1M routing vocabulary, with both 16K (too few, under-specialization) and 64K (too many, under-training per expert) performing worse on TriviaQA. Figure 7 shows that matching the number of experts 1:1 to the vocabulary size (1M experts for 1M vocabulary entries) produces progressively worse results as vocabulary grows, confirming that expert sharing in the long tail is necessary, not just convenient. These ablation curves are the empirical manifestation of the capacity-adaptive principle.

Innovation 4: Expert Freezing During Finetuning as a Strategy for Knowledge Retention in Sparse Models

The standard practice in MoE models is to finetune all parameters — including expert weights — on downstream tasks (Fedus et al., 2022; Zoph et al., 2022; Lepikhin et al., 2020). This makes intuitive sense: if you've trained a large model on a general corpus, finetuning all parameters should allow the model to adapt its knowledge to the specific task distribution.

MoWE deliberately breaks with this practice by freezing the expert parameters during finetuning, training only the dense backbone (attention, layer norm, non-MoWE FFNs, embedding, and output projection). The paper reports a stark ablation (Appendix B.0.3): MoWE-Base on TriviaQA drops from 37.7 EM with frozen experts to 33.5 EM when experts are unfrozen and updated during finetuning — a loss of more than 4 points. This is a negative result with positive implications: it reveals that pretrained experts are fragile knowledge stores that are easily corrupted by task-specific gradient updates, and that the separation of "knowledge parameters" (frozen experts) from "task adaptation parameters" (trainable backbone) is not just convenient but necessary for preserving the benefits of the pretraining.

The conceptual advance here is the recognition that in a model where parameters are explicitly designed to store factual knowledge (through word-specific routing), those parameters should be treated differently from the parameters that perform general language processing. This is a form of procedural knowledge separation: the "what" (factual associations stored in experts) is frozen, while the "how" (language understanding and task adaptation in the backbone) is finetuned. This mirrors ideas from the continual learning literature about catastrophic forgetting as a result of distributed representations (French, 1999; Kirkpatrick et al., 2017), but applies them in a novel architectural context where the separation is enforced by the routing scheme itself rather than by regularization terms.

The significance goes beyond MoWE. If the finding generalizes — that sparsely-activated, content-specialized parameters are particularly vulnerable to catastrophic forgetting during finetuning — it suggests a general design principle for memory-augmented architectures: the memory should be read-only after pretraining, and task adaptation should be performed by the processor that reads the memory, not by modifying the memory itself. This is analogous to how retrieval-augmented models keep the external knowledge store fixed during finetuning, but it applies the same principle to internal, parameterized memories.

The 4-point drop is also diagnostically useful: it suggests that MoWE experts store knowledge that is specifically useful for the downstream task (otherwise freezing them wouldn't matter), but that this knowledge is concentrated in a way that makes it susceptible to being overwritten by task-specific gradients. This is consistent with the single-expert deactivation experiments (Section 4.3) showing that individual experts can control substantial fractions of task performance.

A limitation to note: the paper only tests freezing versus unfreezing on TriviaQA, and only for one model scale. Whether the optimal strategy varies by task type (knowledge-intensive vs. general NLU) or model scale is not explored, leaving open the question of whether this is a universal principle or a task-specific observation.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on a wide range of NLP tasks, with primary focus on knowledge-intensive closed-book question answering: TriviaQA (Joshi et al., 2017), WebQuestions (Berant et al., 2013), Natural Questions (Kwiatkowski et al., 2019), and the FEVER claim verification dataset (Thorne et al., 2018). Additionally, the SuperGLUE benchmark (Wang et al., 2019b) is used to compare with standard MoE models. For QA tasks, 10% of the training set is held out as validation, and models are finetuned on the remaining 90%. FEVER has separate standard validation and test splits. All models are pretrained on C4 version 2.2.0 (Raffel et al., 2020).

• Base model(s). All MoWE models use T5.1.1 as the backbone architecture. MoWE-Base is built on T5.1.1-Base (12 encoder blocks, 12 decoder blocks, d_model = 768, 12 attention heads), while MoWE-Large uses T5.1.1-Large (24 encoder blocks, 24 decoder blocks, d_model = 1024, 16 attention heads). The T5.1.1 dense baselines (Base, Large, XL, XXL) are drawn from Roberts et al. (2020) for the QA tasks. MoE baselines are based on the canonical GShard Top-2 MoE Transformer (Lepikhin et al., 2020), also implemented on T5-Base to ensure fair FLOPs comparison.

• Metrics. For TriviaQA, WebQuestions, and Natural Questions, the primary metric is Exact Match (EM), with F1 used in some ablation experiments because it is "slightly less noisy and highlights the trends more clearly." For FEVER, accuracy is reported on both validation and test sets. For SuperGLUE, following Raffel et al. (2020) and Xue et al. (2022), models are finetuned on a mixture of all SuperGLUE tasks, the best checkpoint per task is selected, and the average validation set score over all tasks is reported as a blended average of accuracy and F1.

• Baselines. The paper compares against four families of baselines: (1) T5.1.1 dense models at scales Base, Large, XL, and XXL, with results from Roberts et al. (2020) for the QA tasks; (2) Regular MoE models using the GShard Top-2 MoE Transformer architecture (Lepikhin et al., 2020) with 32 and 512 experts per sparse layer, implemented on T5-Base backbones; (3) Memory-augmented models including Entities as Experts (EaE) (Févry et al., 2020) and Transformer Over Mention Encodings (TOME) (de Jong et al., 2022), both of which are specialized entity-centric architectures with Wikipedia-pretrained memories; (4) Majority voting and ORM best-of-N weighted baselines from the MoE and dense T5 literature, though these are primarily referenced for context rather than directly replicated.

• Generation budget / compute accounting. The primary compute metric for comparing efficiency is training time relative to T5.1.1-Base, estimated by running each model with a batch size of 256, input sequence length 512, and output sequence length 62 on 64 TPUs v3. The paper explicitly notes that "this likely underestimates the speed of the smaller models, which would enjoy better utilization on fewer devices." For FLOPs comparisons, the paper states that MoWE models have "similar number of FLOPs" to their corresponding T5 backbone models (Base or Large), since only 4 out of 24 FFN layers are replaced with sparse MoWE layers containing small experts (MLP hidden dimensions 512–2048 for Base), and most of those experts are smaller than the dense FFN they replace.

• Cross-validation / statistical protocol. For the compute-optimal strategy selection used in the MoE comparison experiments and ablation studies, the paper does not describe a cross-validation protocol in detail for MoWE evaluation. The QA tasks use a fixed 90/10 train/validation split, with validation used for hyperparameter selection (primarily learning rate) and reported results on the test set. For SuperGLUE, the standard practice of selecting the best per-task checkpoint on validation data is followed. The routing vocabulary size ablation (Figures 4, 5) and number of experts ablation (Figures 6, 7) use models pretrained for 200K steps except where longer pretraining (1M steps) is specified.

---

Main Quantitative Results

Comparison with T5.1.1 Dense Models

The headline finding appears in Table 1 and Figure 1: MoWE-Base achieves 39.4 EM on TriviaQA, outperforming T5.1.1-Base (24.2 EM) by 15.2 points — a 62.8% relative improvement — while also surpassing T5.1.1-Large (28.2) and T5.1.1-XL (36.0). MoWE-Large reaches 44.8 EM on TriviaQA, outperforming T5.1.1-XXL (42.9) by 1.9 points. On WebQuestions, MoWE-Base achieves 35.7 EM vs. T5.1.1-Base at 28.2 and T5.1.1-XL at 32.4; MoWE-Large reaches 38.8 EM vs. T5.1.1-XXL at 35.6. On Natural Questions, the margins are narrower but still significant: MoWE-Base at 29.6 EM vs. T5.1.1-Base at 25.7 and T5.1.1-XL at 29.5; MoWE-Large at 31.9 EM vs. T5.1.1-XXL at 32.8 (here MoWE-Large slightly underperforms the largest dense model). On FEVER, MoWE-Base achieves 66.3 accuracy vs. T5.1.1-Base at 61.3 and T5.1.1-XL at 65.9; MoWE-Large reaches 68.5 vs. T5.1.1-XXL at 67.5. On SuperGLUE, MoWE-Base achieves 83.5 vs. T5.1.1-Base at 77.2, and MoWE-Large reaches 87.4 vs. T5.1.1-Large at 85.1, though here both MoWE models trail the larger dense models (T5.1.1-XL at 88.5 and T5.1.1-XXL at 89.9).

The cost-to-quality tradeoff is the crucial story: MoWE-Base (2.0× T5.1.1-Base training time) matches or beats T5.1.1-XL (8.6×) on all four knowledge-intensive tasks, a 4.3× relative training speedup at matched or superior accuracy. MoWE-Large (4.0×) matches or beats T5.1.1-XXL (26.4×), a 6.6× speedup. On SuperGLUE, the efficiency advantage is present but smaller: MoWE-Base at 2.0× training time outperforms T5.1.1-Base (1.0×) by 6.3 points but trails T5.1.1-Large (3.1×) by 1.6 points. This pattern — larger gains on knowledge-intensive tasks, more modest gains on general NLU — is consistent across all comparisons.

Comparison with Regular MoE Models

Table 2 presents the direct MoE comparison at matched FLOPs (all models use T5-Base backbones, hence similar per-token compute). At the 2B sparse parameter scale (top section), MoWE (8K experts, expert parameters shared across 4 layers, 141 MLP dim) achieves 29.8 EM on TriviaQA vs. 26.5 for MoE-Top2 (32 experts per layer × 12 sparse layers, no parameter sharing), with similar advantages on WebQuestions (31.6 vs. 27.7) and Natural Questions (26.0 vs. 25.8), and a smaller lead on SuperGLUE (81.2 vs. 80.2). At the ~30B sparse parameter scale (bottom section), MoWE with 32K experts achieves 39.4 EM on TriviaQA vs. 36.2 for MoE-Top2 with 512 experts per layer — a 3.2 point advantage. On WebQuestions, the gap is 35.7 vs. 31.6 (4.1 points), while on Natural Questions it narrows to 29.6 vs. 28.5 (1.1 points). On SuperGLUE, the two architectures match exactly at 83.5.

The key takeaway from Table 2 is that MoWE's advantage is concentrated on knowledge-intensive tasks, with the gap widening as expert count increases and as task knowledge requirements grow. MoWE uses 4 sparse layers with 32K total experts (shared) vs. MoE-Top2's 12 sparse layers with 512 experts each (unshared), representing fundamentally different approaches to allocating sparse capacity: MoWE invests in expert breadth (many small experts, each specialized to specific vocabulary entries) while MoE-Top2 invests in expert depth (fewer larger experts per layer, each processing a broader distribution of tokens).

The MoWE Layer as a Sparse Memory: Single-Expert Deactivation Experiments

Section 4.3 and Tables 3–4 provide the most vivid evidence that MoWE experts function as localized knowledge stores. The experimental design: take a pretrained MoWE-Base model (single MoWE layer in the encoder, 32K experts), finetune it on TriviaQA in two modes — (1) all experts activated (normal operation), and (2) experts deactivated for tokens with routing ids above 32K (i.e., all experts derived from the knowledge-rich Wikidata vocabulary are disabled, leaving only experts corresponding to the frequent, non-knowledge tokens from the default T5 vocabulary).

The result (Table 3): exact match drops from 35.1 to 25.6, a 9.5-point decline attributable entirely to deactivating the knowledge-bearing experts. Table 4 provides qualitative examples showing that deactivating the expert for a single key word in the question changes the answer completely — and incorrectly:

• "What is the name of Adele's first album?" → "19" (correct) vs. "Addiction" (expert deactivated)
• "Who followed William Taft as US President?" → "Woodrow Wilson" (correct) vs. "James Garfield" (expert deactivated)
• "What is Neptune's main satellite?" → "Triton" (correct) vs. "Uranus" (expert deactivated)

In each case, the expert for the highlighted word represents only 0.33% of the estimated total activated parameters, yet its deactivation causes a complete (and incorrect) change in the model's factual recall. The paper argues that because "the MoWE layer is frozen during finetuning, all the knowledge that is being leveraged in the downstream task comes from the pretraining corpus" — this is not finetuning-time adaptation but pretraining-time memorization.

Comparison with Memory Augmented Models

Table 5 compares MoWE with EaE and TOME on TriviaQA (dev set, since the test server was inactive) and FEVER (dev/test). The comparison requires a small methodological adjustment: to make MoWE more competitive on Wikipedia-centric tasks (EaE and TOME are pretrained on Wikipedia with entity-aware losses), the MoWE models receive an additional 40K pretraining steps on the Salient Span Masking (SSM) data from Guu et al. (2020), following Roberts et al. (2020).

On TriviaQA: MoWE-Base + SSM achieves 44.9 EM vs. EaE at 43.2, but trails TOME 1 (50.8) and TOME 2 (54.6). MoWE-Large + SSM reaches 50.2 EM, nearly matching TOME 1 at 50.8 but remaining behind TOME 2 at 54.6. On FEVER: MoWE-Base + SSM achieves 69.1 / 66.9 accuracy (dev/test) vs. EaE at 66.1 / 63.6 and TOME 1 at 70.5 / 67.8. MoWE-Large + SSM reaches 70.5 / 68.7, matching TOME 1 on dev and exceeding it on test (68.7 vs. 67.8), and slightly trailing TOME 2 (71.1 / 68.1).

The paper emphasizes that EaE and TOME are "arguably more customized solutions" — they treat TriviaQA as entity linking over a closed set of 1M Wikipedia entities, use specialized training objectives, and require k-NN search tools to access their memory. MoWE performs open-ended answer generation without any such special mechanisms, using the same pretraining-finetuning pipeline as a standard T5 model.

Routing Vocabulary Size Scaling

Figures 4 and 5 show the effect of increasing routing vocabulary size from 32K to 1M entries, holding the number of experts fixed at 32K (15.5B sparse parameters, T5.1.1-Base backbone). In Figure 4 (200K pretraining steps), TriviaQA F1 improves from 34.3 at 32K to 35.6 at 262K to 35.3 at 1M — the bulk of the gain comes from expanding to 262K, with diminishing returns beyond that. The improvement is ~2 points F1 from 32K to 524K+. In Figure 5 (1M pretraining steps), the gap is larger: TriviaQA F1 improves from 37.2 at 32K to 39.1 at 262K to 39.0 at 1M — nearly 2 points gained, and the 262K vocabulary essentially saturates performance. On Natural Questions, the corresponding improvement is from 32.8 at 32K to 33.9 at 262K to 34.3 at 1M — a more modest 1.5-point improvement, with the 1M vocabulary providing a small additional gain over 262K.

The pattern — gains from vocabulary expansion are larger on TriviaQA than Natural Questions, and they grow with pretraining length — is consistent with the interpretation that larger routing vocabularies increase the lexical inductive bias and allow more fine-grained expert specialization. Both curves (Figures 4 and 5) flatten after 262K, suggesting that the top quarter of the 1M vocabulary contains the most useful knowledge-bearing terms.

---

Ablation Studies and Robustness Checks

Number of experts with fixed routing vocabulary (Figure 6): Varying the number of experts (16K, 32K, 64K) while keeping the routing vocabulary fixed at 1M and the total sparse parameters fixed at 15.5B, TriviaQA F1 is 35.2 for 16K, 35.3 for 32K, and 34.7 for 64K (200K pretraining steps). The drop at 64K suggests that experts become too small (the fixed parameter budget forces shrinking individual expert capacity) and/or receive too few training updates, identifying 32K as a "sweet spot."

Matching number of experts to vocabulary size (Figure 7): When the number of experts is set equal to the routing vocabulary size (32K, 65K, 131K, 262K, 524K, 1M), with total sparse parameters held fixed, TriviaQA F1 degrades progressively from 34.5 at 32K to 32.6 at 1M. The paper attributes this to two factors: "(1) the number of training updates that each expert receives becomes increasingly sparse" at higher expert counts, and "(2) the size of the experts are decreased." The 1M-expert configuration uses MLP hidden dimension of only 8 per expert, which is likely too small to store meaningful knowledge. This result is important because it demonstrates that one-to-one word-to-expert mapping (the extreme of specialization) is counterproductive without sufficient per-expert capacity and training signal.

Number of MoWE layers (Table 8): For MoWE-Base on TriviaQA (200K steps, expert parameters shared across encoder layers), increasing encoder MoWE layers from 1 to 2 improves EM from 31.0 to 31.6, but a third encoder layer yields no further gain (31.5). Adding MoWE layers in the decoder provides additional benefit: 2 encoder + 1 decoder reaches 32.4 EM, and 2 encoder + 2 decoder reaches 33.1 EM, the best configuration tested at this scale. The marginal gain from the second decoder layer (0.7 EM) is smaller than the first (0.8 EM over encoder-only), suggesting diminishing returns.

Expert size scaling (Table 9): Using a single encoder MoWE layer and sequentially doubling the expert sizes in all four frequency buckets (keeping the bucket size hierarchy intact), TriviaQA EM improves roughly linearly with parameter count: from 28.5 EM at 3.9B sparse params (bucket expert dims: 512/256/128/64) to 29.6 at 7.8B (1024/512/256/128) to 30.0 at 15.5B (2048/1024/512/256) to 31.0 at 31.0B (2048/2048/1024/512). Each doubling of expert capacity yields ~1 point of EM improvement. The paper speculates that "the increase would be larger if we pretrained the model for 1M steps instead of 200K steps."

Freezing vs. unfreezing experts during finetuning (Appendix B.0.3): This critical ablation shows that MoWE-Base on TriviaQA achieves 37.7 EM with frozen experts vs. 33.5 EM when experts are unfrozen — a 4.2-point penalty for allowing expert updates during finetuning. This result underpins the paper's strategy of treating experts as a read-only memory after pretraining.

Expert deactivation by vocabulary range (Section 4.3, Table 3): Deactivating experts for tokens with routing id above 32K (i.e., all knowledge-rich vocabulary experts) drops TriviaQA EM from 35.1 to 25.6 (9.5 points). The top 32K routing ids roughly correspond to the default T5 vocabulary appended to the auxiliary vocabulary, meaning the remaining experts are primarily handling non-knowledge-bearing tokens.

---

Critical Assessment

Do the Experiments Support the Claim That MoWE Outperforms T5 Models with Comparable FLOPs?

The claim is that MoWE-Base and MoWE-Large "perform significantly better than the T5 family of models with similar number of FLOPs in a variety of NLP tasks" (Abstract), with Figure 1 showing MoWE-Base outperforming T5-XL and MoWE-Large outperforming T5-XXL on TriviaQA while using far fewer FLOPs. Table 1 supports this for knowledge-intensive tasks: the margins are large and consistent. On TriviaQA, WebQuestions, and FEVER, MoWE-Base beats T5-XL and MoWE-Large matches or beats T5-XXL. The training speedup numbers (4.3× and 6.6×) are derived from measured training times on the same hardware and are convincing.

However, the claim weakens for SuperGLUE: MoWE-Base (83.5) outperforms T5.1.1-Base (77.2) but trails T5.1.1-Large (85.1), and MoWE-Large (87.4) trails T5.1.1-XL (88.5) and T5.1.1-XXL (89.9). The paper does not claim dominance here, but the "variety of NLP tasks" framing is misleading if readers expect universal gains. The gains are genuinely task-dependent, concentrated on knowledge-intensive benchmarks, which is fully consistent with the architecture's design rationale.

A significant limitation: the T5 baseline results for QA tasks come from Roberts et al. (2020), not from the authors' own reproduction. This means potential confounds from differences in pretraining data version, hyperparameter tuning, or evaluation protocol cannot be ruled out, though the T5.1.1 models and C4 dataset are well-standardized.

A missing experiment that would strengthen the claim: head-to-head comparison of MoWE-Base vs. T5-XL at matched inference FLOPs (not just matched training time). The paper reports training speedups, but inference efficiency matters equally for deployment, and the all-to-all communication in MoWE layers may have different latency characteristics than dense FFNs, especially at small batch sizes.

Do the Experiments Support the Claim That MoWE Outperforms Regular MoE on Knowledge-Intensive Tasks?

Table 2 shows MoWE beating MoE-Top2 on TriviaQA (39.4 vs. 36.2 at ~30B scale) and WebQuestions (35.7 vs. 31.6), and matching on SuperGLUE (83.5 vs. 83.5). The evidence is solid but limited: only one MoE variant is tested (GShard Top-2), and only at two scales (2B and ~30B). Other MoE configurations — Top-1 routing (Switch Transformer), different expert counts, different expert sizes, different sparse layer counts — are not explored. The claim "outperforms vanilla MoE models on knowledge intensive tasks" is accurate for the tested configurations but the generalizability to other MoE designs is untested.

A missing experiment: an MoE model where the router is initialized from the fixed MoWE routing pattern and then allowed to learn. This would test whether the fixed routing is genuinely better or merely a good initialization that could be improved upon — a key question for understanding whether MoWE's advantage is architectural (fixed routing is inherently better for knowledge) or procedural (fixed routing is a good starting point that a learned router might drift from).

Do the Experiments Support the Claim That MoWE Experts Function as a Sparse Memory?

The single-expert deactivation experiments in Section 4.3 (Tables 3 and 4) are the most compelling evidence in the paper. The 9.5-point drop from deactivating knowledge-bearing experts (Table 3) and the qualitative examples showing individual experts controlling factual answers (Table 4) strongly suggest that specific factual knowledge is localized in specific experts. The fact that experts are frozen during finetuning and the knowledge thus comes from pretraining is correctly noted.

However, the experiments have a limitation: they show that deactivating experts changes answers, but not that the experts store the knowledge in a causal sense. It is possible that the experts compute transformations that are necessary but not sufficient for correct fact retrieval — for example, an expert might be encoding syntactic or semantic features that are prerequisites for downstream fact lookup in the dense layers, rather than storing the fact itself. The paper interprets the results as evidence of memory storage, which is plausible and consistent with prior work (Geva et al., 2021; Dai et al., 2022; Meng et al., 2022), but not definitively proven.

A follow-up experiment that would strengthen the claim: for the single-expert deactivation examples in Table 4, check whether the correct answer degrades to a plausible-but-wrong answer (suggesting knowledge corruption) or to a random/unrelated answer (suggesting a broader representational failure). The examples shown — "Woodrow Wilson" → "James Garfield" (both US presidents), "Triton" → "Uranus" (both celestial bodies), "19" → "Addiction" (unrelated) — are mixed, suggesting the effect may be knowledge-specific in some cases and more general in others.

Does the Comparison with Memory Augmented Models Fairly Represent MoWE's Capabilities?

Table 5 shows MoWE-Large + SSM trailing TOME 2 on TriviaQA (50.2 vs. 54.6) while being competitive on FEVER. The paper frames this positively — MoWE "matches or outperforms" while "avoiding invoking any custom mechanism to search the sparse memory" — which is a fair characterization given MoWE's simpler architecture. However, the comparison is not entirely clean: MoWE receives 40K additional pretraining steps on SSM data to better match the Wikipedia domain that EaE and TOME are specifically designed for. Without these steps, the gap would likely be larger. This domain-adaptation step is reasonable but should be flagged as making the comparison somewhat favorable to MoWE.

A missing comparison: MoWE against a standard retrieval-augmented model (e.g., REALM, RAG, or a T5 + DPR pipeline) on TriviaQA. The paper positions MoWE as an alternative to retrieval, but never directly compares against the dominant retrieval paradigm for knowledge-intensive tasks. If a T5-Base + retriever achieves significantly better results than MoWE-Base at similar or lower cost, the case for parameterized memory over retrieval would be weakened.

Limitations in Experimental Scope

The paper's experimental design has several genuine weaknesses that temper its claims:

• Single pretraining corpus (C4) and single model family (T5.1.1). All results are on one dataset, one architecture family, and one pretraining paradigm (span masking). Whether the findings transfer to autoregressive models (GPT-style), other pretraining objectives, or other corpora is unexplored.

• No inference latency measurements. The paper reports training speedups but not inference wall-clock time. The MoWE layer's all-to-all communication (even with expert blocks) may have different latency characteristics than dense FFNs, especially at low batch sizes typical of online inference.

• Test set sizes are small for some results. The QA test sets are standard, but the ablation experiments use TriviaQA F1 on what appears to be the validation set (10% of training), which is a relatively small number of questions for distinguishing ~0.5-point differences. The error bars on the routing vocabulary and expert count ablations (Figures 4–7) are not reported.

• No evaluation of expert utilization or load in practice. The paper describes the frequency bucketing design but never reports actual expert utilization statistics during pretraining or inference — what fraction of experts are used, whether the capacity buffers are correctly sized, whether any experts are "dead" (never activated).

• The top-16 token exclusion is not ablated. The decision to skip routing for the 16 most frequent tokens (which "can represent up to 28% of the tokens in a batch") is stated as not hurting performance, but no ablation supports this. Given that these tokens account for more than a quarter of all tokens, their exclusion represents a significant design choice that could meaningfully affect both efficiency and quality.

• No exploration of alternative routing vocabulary sources. The Wikidata-seeded vocabulary is used exclusively; no comparison against vocabulary derived from other knowledge bases (Wikipedia titles, Freebase, ConceptNet), from frequency-filtered SentencePiece tokens, or from purely frequency-based selection of C4 words is presented. The claim that Wikidata provides better routing would be stronger with such a comparison.

• Parameter sharing across MoWE layers is not ablated. The paper states that "empirical results indicated that sharing parameters across the MoWE layers leads to better performance," but no Table or Figure presents this comparison. Given that parameter sharing is a key design choice (reducing total sparse parameters by 4×), its absence from the ablations is a notable gap.

6. Limitations and Trade-offs

6.1 The Routing Vocabulary Construction Is a Critical, Underexplored Design Bottleneck with No Universal Recipe

The assumption or constraint. The entire MoWE architecture depends on the quality of the routing vocabulary — which words get their own experts, which words share experts, and how the vocabulary is constructed determines which knowledge can be stored and retrieved. The paper's vocabulary construction method (Section 2.4) is a specific four-step pipeline: seed from Wikidata entity and relation names → normalize and split on whitespace → order by C4 frequency → select top 1M entries. This is described as a "straightforward strategy," and the paper acknowledges that the approach is intentionally simple:

"More work can definitely be done to improve the routing vocabulary, but we wanted to keep it simple for our experiments."

The assumption is that this particular vocabulary source and selection method provides adequate coverage of knowledge-bearing terms for the evaluated tasks. However, there is no principled justification for why Wikidata entities (rather than, say, Wikipedia article titles, Freebase concepts, or purely frequency-filtered C4 n-grams) should be optimal, nor is there any comparison against alternative vocabulary sources.

The consequence. A practitioner attempting to apply MoWE to a new domain — legal documents, medical texts, code, a non-English language — has no guidance on how to construct their routing vocabulary. The Wikidata seed is English-centric and entity-focused; for domains where knowledge is not organized around named entities (e.g., procedural knowledge about how to perform tasks, mathematical theorems, or domain-specific jargon not captured in Wikidata), the vocabulary construction strategy may be suboptimal or entirely inappropriate. The results on Natural Questions (where MoWE-Large slightly underperforms T5.1.1-XXL: 31.9 vs. 32.8 EM, Table 1) hint that the routing vocabulary may be better matched to entity-heavy tasks like TriviaQA than to the more diverse question types in Natural Questions. More broadly, the paper provides no diagnostic for determining whether a given routing vocabulary is adequate — no metric for vocabulary quality, no analysis of how vocabulary coverage correlates with downstream performance, and no fallback strategy if the vocabulary proves insufficient.

What evidence exists in the paper. The routing vocabulary size ablation (Figures 4 and 5) shows that expanding vocabulary from 32K to 262K improves TriviaQA F1 by ~2 points, with diminishing returns beyond that. Appendix D provides sample entries from different frequency ranges, revealing that the top 50 entries are mostly common word variants ("isn", "aren", "whilst", "3d", "1st") while entries after position 6000 become more knowledge-bearing ("consignment", "discrepancy", "horticulture", "diwali"). However, there is no ablation comparing the Wikidata-seeded vocabulary against alternatives (e.g., a vocabulary derived from the most frequent C4 whitespace-delimited tokens, or a vocabulary extracted from Wikipedia rather than Wikidata), and no measurement of what fraction of test-set answers involve entities or terms that are well-covered vs. poorly-covered by the routing vocabulary.

Mitigation status. Not addressed. The paper specifies a single vocabulary construction method, tests it on a single family of tasks (English knowledge-intensive QA and NLU), and leaves vocabulary design as an open problem. The authors flag this as an area for future work only implicitly, through the acknowledgment that the vocabulary construction was intentionally simple. No ablation or analysis explores how sensitive results are to the vocabulary source, the frequency filtering threshold, or the whitespace-splitting heuristic (which "languages that do not use white space for word splitting will require slightly modified processing," as the paper notes, suggesting cross-lingual extension would require non-trivial redesign).

---

6.2 Expert Freezing Is Essential but Eliminates Adaptability to New Knowledge or Domains

The assumption or constraint. MoWE's strong performance on knowledge-intensive tasks depends critically on freezing the expert parameters during finetuning. The paper states:

"We freeze all MoWE experts to avoid both overfitting and catastrophic forgetting of knowledge acquired during pretraining."

The ablation in Appendix B.0.3 quantifies the cost of unfreezing: MoWE-Base on TriviaQA drops from 37.7 EM to 33.5 EM — a 4.2-point penalty when experts are allowed to update. The underlying assumption is that all the knowledge needed for downstream tasks was already acquired during pretraining and can be effectively leveraged through the frozen experts, with only the dense backbone adapting to the specific task format.

The consequence. This design creates a fundamental tension: the same mechanism that makes MoWE effective at knowledge retention (frozen, word-localized experts) makes it incapable of learning new factual knowledge after pretraining. If a MoWE model is deployed and subsequently needs to incorporate new factual information (e.g., a new president takes office, a new scientific discovery is made, a new product is released), the architecture provides no mechanism to update the relevant word experts without risking the catastrophic forgetting that the freezing strategy is designed to prevent. The alternative — unfreezing and finetuning on new data — is explicitly shown to degrade performance substantially. Similarly, if a MoWE model pretrained on general web text (C4) needs to be adapted to a specialized domain with its own factual knowledge (e.g., internal corporate documents, legal precedents, medical literature), the frozen experts will be locked into the pretraining distribution's facts, and the dense backbone alone cannot compensate, since the architecture deliberately routes knowledge storage through the experts.

This limitation is particularly consequential because it undermines one of the primary use cases for efficient knowledge-intensive models: continuous updating as world knowledge evolves. Standard dense models can be finetuned on new data to incorporate new facts (though they also suffer from catastrophic forgetting to some degree). Retrieval-augmented models handle this naturally by updating the external knowledge corpus. MoWE, by design, has no update mechanism that doesn't destroy the very knowledge it was built to store.

What evidence exists in the paper. The freezing-vs-unfreezing ablation (Appendix B.0.3) provides the direct evidence: unfreezing hurts. However, this ablation only tests the scenario where experts are unfrozen on the same task they were pretrained for (or a closely related QA task), without new knowledge being introduced. There is no experiment that tests whether MoWE can learn new facts after pretraining (e.g., by finetuning on a temporally shifted dataset or a new domain), so the claim about inability to adapt is a logical consequence of the design and the freezing ablation, but is not directly measured. The paper's framing of experts as a "sparse memory" (Section 4.3) reinforces this interpretation — memories are written during pretraining and read during finetuning/inference, with no write capability thereafter.

Mitigation status. Not addressed. The paper treats expert freezing as a feature (preventing catastrophic forgetting) rather than a limitation, and does not propose or test any mechanism for incremental knowledge updating. Potential mitigations — selective unfreezing of only a small subset of experts, elastic weight consolidation to protect pretrained knowledge, or a separate set of adaptor parameters for new knowledge — are not discussed. This is a significant gap for anyone considering MoWE for production deployment where the knowledge base must evolve over time.

---

6.3 The Architecture Is Evaluated on a Single Model Family, Single Pretraining Objective, and Single Language

The assumption or constraint. All experiments in the paper use the T5.1.1 encoder-decoder architecture with span-masking pretraining on the English C4 corpus. The paper states that "we use T5.1.1 as the backbone of our MoWE models," and all comparisons — against dense T5 models, MoE-T5 models, and memory-augmented models — are within this ecosystem. The implicit assumption is that the findings about fixed lexical routing, frequency bucketing, and expert freezing generalize across model architectures, pretraining paradigms, and languages.

The consequence. A practitioner cannot determine from this paper whether MoWE would be effective in an autoregressive (decoder-only) setting (GPT-style models, which dominate the current LLM landscape), in a model pretrained with a next-token prediction objective rather than span masking, or in a multilingual or non-English setting. These are not hypothetical concerns: the routing vocabulary construction method explicitly relies on whitespace-split tokens, which the paper acknowledges "languages that do not use white space for word splitting will require slightly modified processing." For Chinese, Japanese, Thai, or other languages without whitespace delimiters, the entire vocabulary pipeline would need to be redesigned. Similarly, the decision to place MoWE layers at specific encoder and decoder positions (blocks 5 and 10 for Base) is tuned to the T5.1.1 encoder-decoder architecture; the optimal positioning in a decoder-only model with 32, 70, or 96 layers is unknown.

The limitation is compounded by the fact that the paper's headline claims — 4.3× training speedup over T5-XL, expert localization of factual knowledge, superiority over learned-routing MoE — are presented as properties of the MoWE approach in general, not as properties of MoWE-on-T5.1.1 specifically. If a different backbone architecture interacts differently with the MoWE layer (e.g., if autoregressive attention patterns dilute the expert's influence, or if the lack of an encoder changes how contextualization before expert lookup works), the claimed benefits may not transfer.

What evidence exists in the paper. None. The paper does not include any experiment with a different backbone architecture, pretraining objective, or language. The entire evaluation is on English T5.1.1 models. The paper's related work section mentions connections to decoder-only MoE models (GLaM, Du et al., 2022) and vision models (Riquelme et al., 2021), but no cross-architecture comparison is performed. The memory-augmented model comparisons (EaE, TOME) use different architectures and training objectives entirely, but these are competitors rather than alternative MoWE backbones.

Mitigation status. Not addressed. The paper does not claim generalizability beyond T5.1.1, but it also does not acknowledge this as a limitation. A single sentence noting that "future work should validate MoWE on other model families and languages" would clarify the scope. The paper's contributions are framed architecturally ("a novel neural net architecture") rather than as a T5-specific variant, which implies broader applicability that remains unverified.

---

6.4 The Cost of the Routing Vocabulary Tokenizer and Hash Lookup Is Not Accounted for in Speedup Claims

The assumption or constraint. The paper reports training speedups of 4.3× (MoWE-Base vs. T5-XL) and 6.6× (MoWE-Large vs. T5-XXL) based on measured training time at fixed batch size, input length, and output length on 64 TPUs v3 (Table 1). These speedups compare the full MoWE training pipeline to the dense T5 baselines. However, the speedup calculation does not account for the offline preprocessing cost of constructing the routing vocabulary and the hash table that maps default tokenizer sub-sequences to routing ids, nor does it account for the online cost of the hash lookup that determines each token's routing id during training and inference. The paper assumes these costs are negligible relative to the Transformer computation.

The consequence. The hash lookup described in Section 2.4 performs, for each token in the input sequence, up to 9 hash-table probes (sub-sequences of length 1 through 9 tokens ending at the current position) and selects the longest match. For an input sequence of 512 tokens, this is up to ~4,600 hash lookups per sequence. While hash lookups are fast relative to matrix multiplications, they are not free, and they become more expensive at larger batch sizes and longer sequences. The paper does not provide any profiling data on what fraction of total training or inference time is consumed by the routing tokenization step. In deployment scenarios with very large batch sizes (where FLOPs from matrix multiplications are well-utilized), the hash lookups could become a non-trivial overhead; in latency-sensitive online inference (batch size 1), the overhead relative to the Transformer computation is likely negligible but still unmeasured.

More significantly, the offline preprocessing to build the routing vocabulary — extracting entity and relation names from a Wikidata dump, computing token frequencies on a 2B-token sample of C4, building the hash table of default-tokenizer sequences to routing ids — is a non-trivial engineering effort that a practitioner must replicate for every new domain, language, or data distribution. This is not a one-time cost if the pretraining data or domain changes. The paper treats this as straightforward, but it represents a barrier to adoption that dense models and standard MoE models (which require no auxiliary vocabulary) do not face.

What evidence exists in the paper. None. The paper reports total training time (Table 1) and the number of TPUs used (64 v3), but does not break down the time by component (dense layers, MoWE dispatch, hash lookup, etc.). The top-16 token exclusion is mentioned as an optimization that "speeds up the training time" because it skips routing for tokens representing up to 28% of the batch, but the absolute time saved is not quantified. There is no ablation comparing end-to-end training throughput with and without the hash-lookup step (which could be approximated by using identity routing, though this would change the semantics).

Mitigation status. Partially acknowledged but not measured. The paper notes that the routing tokenization uses a hash operation to "allow (a) efficient lookup of routing ids and (b) the use of the MoWE layer in auto-regressive scenarios where normally only the initial part of the word is known." The word "efficient" appears without quantitative backing. The top-16 token exclusion is a partial mitigation for the most frequent tokens, but the remaining tokens still require hash lookups. The paper does not propose or evaluate alternative, lower-overhead routing mechanisms (e.g., a smaller hash table keyed only on individual default tokens, a Bloom filter, or a learned mapping).

---

6.5 The Difficult Tradeoff Between Expert Count and Per-Expert Training Signal Is Not Resolved

The assumption or constraint. MoWE's design embodies a tension: more experts means finer-grained word specialization (potentially better knowledge storage), but also means fewer training examples per expert (since each expert activates only when its assigned word appears) and smaller per-expert capacity (if total parameter budget is fixed). The paper assumes that 32K experts strikes a workable balance, but the optimal point is likely task-, data-, and scale-dependent.

The consequence. Figure 6 shows that increasing experts from 16K to 32K yields minimal F1 improvement on TriviaQA (35.2 → 35.3), while 64K actually degrades performance (34.7). Figure 7 shows that matching experts 1:1 to the routing vocabulary (up to 1M experts) causes a progressive and substantial degradation from 34.5 F1 at 32K experts to 32.6 F1 at 1M experts. The 1M-expert configuration uses MLP hidden dimension of only 8 per expert, which is almost certainly too small to store any meaningful factual associations. These results indicate that MoWE's performance is sensitive to a hyperparameter (expert count relative to vocabulary size and total parameter budget) that has no clear selection criterion — the "sweet spot" at 32K was found empirically for this specific combination of backbone, vocabulary, pretraining data scale, and task suite, and it is not obvious how it would change if any of these factors varied.

The deeper issue is that MoWE's design couples expert count to vocabulary size and frequency distribution, but the optimal coupling depends on the amount of pretraining data. With 1T pretraining tokens, a word appearing 100 times in the corpus gets 100 training updates for its expert, which may be insufficient for learning robust associations. With 10T tokens, that same word might get 1,000 updates, and the optimal expert count might shift upward. The paper's pretraining scale (1T tokens) is fixed, so it cannot measure how the expert-count sweet spot scales with data. A practitioner pretraining on more or less data has no guidance on how to adjust expert count accordingly.

What evidence exists in the paper. Figure 6 (expert count ablation at fixed vocabulary size) and Figure 7 (expert count matching vocabulary size) provide clear evidence of this tradeoff, but both are at a single pretraining scale (200K steps, ~200B tokens). Table 2 shows that MoWE with 32K experts outperforms MoE-Top2 with 512 experts on knowledge tasks, but this comparison confounds several variables (fixed vs. learned routing, shared vs. unshared parameters, different number of sparse layers), making it impossible to isolate the effect of expert count alone. The frequency bucketing strategy (Table 6) implicitly acknowledges the training signal problem by sharing experts in bucket 4 (30,080 experts for ~220K vocabulary entries, or ~7.3 entries per expert), but the sharing ratio is chosen heuristically without an ablation exploring alternative ratios.

Mitigation status. Not addressed. The paper identifies 32K as a sweet spot for their configuration but provides no systematic method for determining the optimal expert count given a vocabulary, data scale, and parameter budget. The degradation at 1M experts (Figure 7) is presented as an interesting result rather than as a problem to be solved. The paper hypothesizes that the degradation is due to sparse training updates and decreasing expert size, but does not propose solutions such as expert warm-starting, multi-task pretraining to increase per-expert signal, or dynamic expert capacity allocation during training.

---

6.6 The FLOPs Advantage Claim Depends on a Baseline That May Not Be Optimally Configured

The assumption or constraint. The paper's central efficiency claim — that MoWE models achieve comparable or superior accuracy to much larger dense models while using far fewer FLOPs — rests on comparisons against T5.1.1 baselines drawn from Roberts et al. (2020) for the QA tasks, and against a specific MoE-Top2 configuration for the MoE comparison (Table 2). The assumption is that these baselines represent the best achievable performance at their respective FLOPs budgets.

The consequence. For the dense T5 comparison, the T5.1.1 models from Roberts et al. (2020) were pretrained with the same span-masking objective and same C4 dataset, but the exact pretraining details (number of steps, learning rate schedule, batch size) may differ from the MoWE pretraining run. If MoWE received more careful hyperparameter tuning or a slightly different pretraining recipe, some fraction of the reported gains may be attributable to training procedure differences rather than architecture. This is a standard concern when comparing against published baselines rather than reproducing them under identical conditions. The paper attempts to control for this by using "the same pretraining hyperparameters of T5.1.1," but there are inevitably implementation-level differences between the T5X framework used for MoWE and the original T5 codebase.

For the MoE comparison, the paper uses a single MoE architecture (GShard Top-2 with specific expert counts, expert sizes, and sparse layer configurations) and does not explore whether a differently configured MoE — with, for example, a larger number of experts, smaller expert sizes, or expert parameter sharing like MoWE — would close the gap. The finding from Du et al. (2022) that MoE performance diminishes beyond ~64–128 experts with learned routing is cited as motivation for MoWE's fixed routing, but this finding comes from a different model family (GLaM, decoder-only) and may not apply to the T5 backbone and task suite used here. The paper does not reproduce the diminishing-returns experiment for MoE-Top2 on their own setup.

A more subtle issue: the MoE-Top2 baseline in Table 2 uses a different structure of sparse layers than MoWE (12 sparse layers vs. 4) and no parameter sharing. These differences make it impossible to attribute MoWE's advantage specifically to fixed lexical routing — it could equally be attributed to the parameter sharing across layers (which MoE-Top2 doesn't use), the different number of sparse layers, or the different expert sizing strategy. A fairer ablation would be an MoE-Top2 model with 4 sparse layers and shared expert parameters, isolating the routing mechanism as the only variable.

What evidence exists in the paper. The comparison data is in Table 1 (dense baselines) and Table 2 (MoE baselines). The paper does not report any experiments that control for all variables except the routing function, which is the minimum necessary to attribute the advantage to fixed lexical routing rather than other architectural differences. The paper also does not report confidence intervals or statistical tests for any of the pairwise comparisons between MoWE and baselines, so it is unclear whether, for example, the 39.4 vs. 36.2 TriviaQA EM difference (MoWE-Base vs. MoE-Top2 at ~30B) is statistically significant given the test set sizes.

Mitigation status. Not addressed. The paper treats the baseline comparisons as sufficient to establish MoWE's advantage, and the margins on knowledge-intensive tasks are large enough that statistical significance is plausible. However, the absence of a controlled ablation that isolates routing mechanism from other architectural choices, and the reliance on external baselines for the dense comparison, means that the headline efficiency gains may overstate the benefit attributable specifically to the MoWE design rather than to a combination of favorable design choices (parameter sharing, carefully tuned expert sizes, the particular MoE-Top2 configuration chosen as baseline). This does not invalidate the practical claim that MoWE-as-a-whole outperforms T5, but it weakens the scientific claim that fixed lexical routing is the causal driver of the improvement.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a third architectural paradigm for scaling knowledge capacity in language models, distinct from both "make the dense model bigger" and "add an external retrieval system." Before MoWE, the field understood two ways to store more world knowledge in a model: scale up the parameter count of a dense Transformer (which proportionally increases FLOPs) or augment the model with an external retrieval corpus accessed via nearest-neighbor search at inference time (which introduces a separate subsystem, a non-differentiable retrieval step, and dependence on a curated knowledge store). MoWE demonstrates that there is a third option — co-locate knowledge with the words that trigger it, using extremely sparse, vocabulary-routed parameters that are part of the model itself but only activate for their specific lexical cues — and that this option achieves the knowledge capacity of much larger dense models while preserving the FLOPs efficiency of small ones.

This is not an incremental improvement in the efficiency-accuracy tradeoff curve. The magnitude of the gain — MoWE-Base outperforming T5-XL on TriviaQA by 3.4 points exact match while being 4.3× faster to train (Table 1, Figure 1) — represents a regime shift in what is possible at a given compute budget for knowledge-intensive tasks. It is not a 10% or 20% improvement; it is a model with 2.0× the training time of T5-Base matching a model with 8.6× the training time. For practitioners whose primary bottleneck is storing and retrieving factual knowledge, MoWE essentially drops the cost of that capability by a factor of 4–6×. This is large enough to change deployment economics: tasks that previously required a datacenter-scale model can now plausibly run on a much smaller, cheaper model with MoWE layers.

The paper also forces a rethinking of what routing is for in mixture-of-experts models. The dominant MoE paradigm since Shazeer et al. (2017) has treated routing as a learning problem — the router must be trained to discover which experts should handle which tokens, with load-balancing losses to prevent collapse, auxiliary losses to encourage diversity, and capacity factors to manage over-subscription. MoWE shows that when the goal is knowledge storage rather than representational flexibility, routing does not need to be learned at all. A fixed, vocabulary-driven routing function works better on knowledge-intensive tasks (Table 2: 39.4 vs. 36.2 TriviaQA EM for MoWE vs. MoE-Top2 at ~30B scale) while matching learned routing on general NLU (83.5 vs. 83.5 on SuperGLUE). This reframes routing as a design dimension with task-dependent optimal settings — you choose your routing mechanism based on what you want the experts to do, not based on some universal notion of "good routing." If you want experts to store word-associated facts, fix the router to a knowledge-rich vocabulary. If you want experts to represent context-dependent subfunctions, learn the router. This conceptual separation unblocks scaling to expert counts (32K–1M) that learned routers cannot reach, and it invites future work to explore other fixed routing functions tailored to other goals (e.g., syntactic routing for structured prediction, entity-type routing for relation extraction).

The single-expert deactivation experiments in Section 4.3 (Tables 3 and 4) provide a different kind of contribution: they offer the most direct evidence to date that factual knowledge can be causally localized to individual, interpretable parameters in a language model. When deactivating the expert for a single word (e.g., the expert for "Neptune") causes the model to switch from the correct answer ("Triton") to an incorrect one ("Uranus"), and when a single expert representing 0.33% of activated parameters controls a substantial fraction of the model's ability to answer related questions, it validates and extends the "knowledge neuron" hypothesis (Dai et al., 2022; Meng et al., 2022) with a stronger causal intervention than prior work. This matters not just for MoWE but for the broader interpretability and model-editing communities: it provides an existence proof that factual associations can be architecturally localized and that this localization can be achieved by design (through fixed lexical routing) rather than discovered post-hoc through probing. The examples in Table 4 — "Woodrow Wilson" → "James Garfield," "Triton" → "Uranus," "19" → "Addiction" — are vivid, concrete demonstrations that the knowledge labeled by a specific word lives (at least in part) in that word's expert.

The paper also reconciles conflicting signals in the MoE scaling literature. Du et al. (2022) found diminishing returns beyond ~64–128 experts in GLaM with learned routing, which might have been interpreted as a fundamental limit on how many experts are useful in any MoE architecture. MoWE demonstrates that this limit is specific to learned routing — the bottleneck is in the router's ability to assign tokens meaningfully when expert count is high, not in the model's ability to use many experts. With fixed routing, 32K experts outperform 16K, and the paper successfully trains models with up to 1M experts (the largest expert count reported for any Transformer MoE, to my knowledge). This clarifies that the diminishing returns were a property of the learning dynamics, not of expert sparsity itself, and opens the door to architectures with vastly more experts than previously thought practical.

Finally, the paper's expert freezing result (Appendix B.0.3: unfreezing experts during finetuning drops TriviaQA EM from 37.7 to 33.5) adds a new data point to the catastrophic forgetting literature that is architecturally specific. It suggests that in models with explicitly knowledge-localized parameters, those parameters are particularly vulnerable to being overwritten by task-specific gradients, and that freezing them is not just a convenience but a necessity for preserving pretrained knowledge. This has implications for any architecture that attempts to separate "knowledge storage" from "task processing" — if you successfully build knowledge-specific compartments, you must protect them from gradient updates during adaptation, or the compartmentalization unravels.

The research directions that become more attractive as a result of this work: (1) fixed, task-specific routing functions for MoE models (the paper opens the door beyond learned routing); (2) architectural knowledge localization as a design principle (if you want facts to be editable or interpretable, route them to dedicated experts); (3) extremely sparse models with expert counts in the tens or hundreds of thousands, which MoWE shows are trainable and beneficial; (4) read-only memory in neural architectures, where knowledge-bearing parameters are frozen after pretraining and only processing parameters adapt to tasks. Directions that become less attractive: (1) the assumption that learned routing is always superior and that fixed routing is a primitive fallback — MoWE shows the opposite can be true for knowledge tasks; (2) the implicit belief that external retrieval is necessary for efficient knowledge storage — MoWE provides a competitive parameterized alternative that avoids the retrieval infrastructure entirely.

---

Follow-Up Research This Work Enables

Isolating the causal contribution of fixed lexical routing vs. parameter sharing vs. sparse layer count. The MoWE vs. MoE-Top2 comparison in Table 2 confounds three variables: the routing mechanism (fixed lexical vs. learned Top-2), parameter sharing across sparse layers (present in MoWE, absent in MoE-Top2), and the number and placement of sparse layers (4 in MoWE, 12 in MoE-Top2). A clean ablation would be: take the MoWE architecture and replace the fixed lexical router with a learned Top-2 router, keeping everything else (4 sparse layers, shared expert parameters, 32K experts, frequency bucketing) identical. This would isolate whether the performance advantage on TriviaQA (39.4 vs. 36.2) comes from the fixed routing itself, from the parameter sharing (which MoE-Top2 lacks), or from the different sparse layer structure. The experiment is straightforward to run on the existing MoWE codebase and would transform the paper's claim from "MoWE outperforms MoE" (a system-level comparison) to "fixed lexical routing outperforms learned routing for knowledge-intensive tasks" (a mechanism-level finding). The negative result — if the learned-router variant matches MoWE's TriviaQA performance — would be equally informative, suggesting that the architectural innovations (sharing, bucketing, expert count) matter more than the routing mechanism.

Measuring expert utilization and knowledge localization at scale. The paper provides qualitative evidence that individual experts store factual knowledge (Table 4) but no systematic analysis of how many experts are "knowledge-bearing" vs. "dead" or "syntactic," what fraction of total knowledge is localized in experts vs. distributed in the dense backbone, or how expert utilization varies by frequency bucket. A follow-up study could: (1) measure the activation frequency of every expert across the full TriviaQA test set and correlate it with downstream accuracy; (2) perform systematic causal interventions (deactivating each expert one at a time) to build a map of which facts live where; (3) compare expert specialization across buckets — do bucket-1 experts (frequent words, large MLPs) store more diverse knowledge than bucket-4 experts (rare words, small MLPs, shared across many vocabulary entries)? (4) test whether the knowledge stored in an expert is truly word-specific or whether it captures broader entity-type or topic-level associations. This would validate or refine the paper's "sparse memory" interpretation and provide actionable guidance for future vocabulary construction — if the top 1,000 experts store 80% of the factual knowledge, the routing vocabulary could be sharply pruned, simplifying the architecture.

Testing MoWE in a decoder-only (GPT-style) architecture. The paper evaluates solely on T5.1.1 encoder-decoder models with span-masking pretraining. The current LLM landscape is dominated by decoder-only autoregressive models (GPT-4, LLaMA, PaLM). A critical follow-up would implement MoWE layers in a decoder-only Transformer and evaluate on the same knowledge-intensive tasks, asking: (1) Does the optimal placement of MoWE layers change when there is no separate encoder? (2) Does autoregressive next-token pretraining (vs. span masking) affect how experts specialize — since span masking gives the model bidirectional context for expert lookup, while causal masking restricts it to left context? (3) Does the 38% correct-to-incorrect reversion problem seen in the revision model experiments transfer — i.e., during autoregressive generation, does the partial-word routing approximation (using prefixes before the full word is known) cause degradation compared to oracle full-word routing? The experiment would use a LLaMA-style architecture with MoWE layers inserted at positions analogous to the T5 setup, pretrained on the same C4 data, and evaluated on TriviaQA and Natural Questions. A positive result (decoder-only MoWE matches or exceeds encoder-decoder MoWE on knowledge tasks) would dramatically expand the method's applicability; a negative result would reveal that MoWE's benefits depend on architectural features specific to encoder-decoder models.

Domain adaptation of the routing vocabulary without catastrophic forgetting. MoWE's expert freezing strategy prevents incorporating new factual knowledge after pretraining. A concrete research question: can a small set of new experts be added (or a small subset of existing experts be selectively unfrozen) to adapt MoWE to a new domain without degrading performance on the original domain? The experimental design: pretrain MoWE-Base on C4 as in the paper, then add new experts for 1,000 domain-specific terms (e.g., from a biomedical corpus) in a new frequency bucket, pretrain only those new experts on the domain corpus, freeze all original experts, and evaluate on both TriviaQA (to measure forgetting) and a biomedical QA task (to measure adaptation). Compare against (a) finetuning all experts on the domain data (expected to cause forgetting, per Appendix B.0.3) and (b) keeping all experts frozen and training only the dense backbone on the domain data (the paper's current approach, which can't add completely new factual knowledge). This would test whether incremental knowledge addition is possible in the MoWE framework and, if successful, would address the most significant deployment limitation identified in Section 6.2.

Comparing MoWE against retrieval-augmented generation at matched parameter and FLOPs budgets. The paper compares MoWE against memory-augmented models (EaE, TOME) that use specialized entity-centric training and k-NN retrieval, but never against a standard retrieval-augmented generation (RAG) pipeline — a T5-Base model with a DPR retriever over Wikipedia, which is the dominant paradigm for knowledge-intensive open-domain QA. A head-to-head comparison on TriviaQA, WebQuestions, and Natural Questions, with both total parameters (model + retriever index) and inference FLOPs matched, would clarify whether MoWE's parameterized memory is genuinely competitive with explicit retrieval. The experiment would need to account for: the FLOPs cost of DPR query encoding and k-NN search (which MoWE avoids), the storage cost of the Wikipedia index versus MoWE's expert parameters, and the retrieval latency versus MoWE's all-to-all communication overhead. If MoWE matches or exceeds RAG at lower total cost, it strengthens the argument that lexical co-location can replace retrieval for pretraining-distribution knowledge; if RAG significantly outperforms MoWE, the paper's framing of MoWE as an alternative to retrieval would be weakened, and MoWE would be better understood as a complement to (or component within) retrieval-augmented systems.

Stress-testing the routing vocabulary: how does MoWE perform when the vocabulary deliberately excludes key task entities? The paper's routing vocabulary is derived from Wikidata entity and relation names, which means it covers precisely the kinds of named entities that appear in TriviaQA and WebQuestions questions. This creates a potential circularity: MoWE performs well on these tasks partly because its vocabulary was designed to cover the entity space those tasks sample from. A stress test would construct routing vocabularies that deliberately exclude named entities — e.g., a vocabulary built from the most frequent C4 whitespace-delimited tokens with all Named Entity Recognition (NER) spans removed, or a vocabulary built from a completely different knowledge domain (e.g., medical terms only) — and measure how much of MoWE's TriviaQA advantage is attributable to vocabulary coverage versus the architecture itself. If performance degrades to near the T5-Base baseline, it would show that MoWE's gains are almost entirely vocabulary-driven and that the architecture serves mainly as an efficient lookup mechanism for a pre-specified set of terms. If substantial gains remain even with a deliberately mismatched vocabulary, it would suggest that the architecture encourages knowledge localization even without vocabulary guidance. This experiment would clarify whether MoWE is fundamentally a knowledge storage architecture or primarily a mechanism for leveraging a high-quality vocabulary definition.

---

Practical Applications and Downstream Use Cases

Cost-efficient closed-book QA in production assistants. A MoWE-Base model (31B parameters, training cost 2.0× T5-Base) matches or exceeds T5-XXL (training cost 26.4× T5-Base) on TriviaQA, WebQuestions, and FEVER (Table 1). For a production question-answering system that must answer factoid questions without access to a retrieval corpus — for instance, a voice assistant that needs to answer "Who wrote The Great Gatsby?" without making a web request, either because of latency constraints or because the device is offline — MoWE offers the accuracy of an enormous model at the inference cost of a small one. The 6.6× training speedup (MoWE-Large vs. T5-XXL) also reduces the cost of periodic model retraining as the knowledge base evolves (new facts enter the pretraining corpus). Combined with the frozen-expert finetuning strategy, deploying a MoWE model in this setting requires only: (1) pretrain a single MoWE model on a general corpus with the appropriate routing vocabulary; (2) finetune a lightweight task-specific head and dense backbone on the QA dataset, keeping the knowledge-bearing experts unchanged; (3) serve with inference FLOPs comparable to T5-Large but accuracy comparable to T5-XXL.

Domain-specific encyclopedic knowledge in resource-constrained deployments. The paper demonstrates that MoWE's advantage is largest on tasks requiring memorization and retrieval of world knowledge (TriviaQA: +15.2 points over T5-Base), while gains on general NLU are more modest (SuperGLUE: +6.3 points over T5-Base). This makes MoWE specifically valuable for applications where a model needs to serve as a compact encyclopedia — a legal research tool that must recall case names and statutes, a medical coding system that maps symptoms to ICD codes, an educational app that answers factual questions about history or science, or an internal corporate knowledge base that must recall product specifications and internal acronyms. In each case, the routing vocabulary can be tailored to the domain's key terms (medical jargon, legal terminology, product names) using the same Wikidata-seeded pipeline, and the resulting MoWE model will store domain knowledge in the experts for those terms. The key advantage over dense models is that MoWE's knowledge capacity does not require scaling the entire model — a small backbone (T5-Base) with domain-specific experts can approach the accuracy of a much larger dense model, making on-device or edge deployment of encyclopedic models feasible where it wouldn't be otherwise. The paper's results on SSM-pretrained MoWE (Table 5: 44.9 TriviaQA EM for Base + SSM) suggest that additional domain-adaptive pretraining on a targeted corpus further improves knowledge retrieval without requiring architectural changes.

Knowledge editing and model patching via expert replacement. The single-expert deactivation experiments (Section 4.3, Tables 3 and 4) demonstrate that specific factual associations are concentrated in specific, identifiable experts. This opens a practical pathway for model editing — correcting factual errors or updating outdated knowledge without retraining the entire model. If the expert for "current US President" stores information about the presidency, and that information becomes outdated, that expert could be retrained on corrected data (or replaced with a newly trained expert for the same routing id) without affecting the rest of the model's knowledge. This is a much more targeted intervention than existing model editing techniques (Meng et al., 2022; Mitchell et al., 2022), which must identify specific MLP weights to modify through gradient-based searches and which risk collateral damage to unrelated facts stored in nearby parameters. In MoWE, expert replacement is architecturally trivial — substitute the weight matrix for the expert corresponding to the term you want to update — and the only question is whether the new expert generalizes correctly. A knowledge management pipeline could monitor for factual drift (e.g., a new president takes office), identify the routing vocabulary entry for the affected concept, collect a small dataset of correct associations from recent data, retrain only that expert, and hot-swap it into the deployed model. The paper doesn't demonstrate this, but the architecture makes it natural in a way that dense models and learned-routing MoE models do not.

Pretraining data augmentation via expert-guided vocabulary expansion. The routing vocabulary size ablation (Figures 4 and 5) shows monotonic improvement as the vocabulary expands from 32K to 262K, with those gains growing when pretraining is extended from 200K to 1M steps. For teams preparing to pretrain a large knowledge-intensive model, this suggests a concrete data engineering step: before pretraining begins, invest in building the largest, highest-quality routing vocabulary feasible from available knowledge bases (Wikidata, Wikipedia, domain-specific entity lists, etc.), because the pretraining compute will be used more efficiently — the model will allocate its sparse parameters to precisely the terms that benefit from dedicated expert capacity. The vocabulary construction pipeline described in Section 2.4 (seed from Wikidata, normalize, order by frequency, select top N) is simple enough to be replicated for any domain or language with a suitable knowledge base, and the results suggest that a vocabulary in the range of 250K–1M entries provides most of the benefit. This is a practical, actionable step that goes beyond "use more compute" or "collect more data" — it is a data curation strategy specifically enabled by the MoWE architecture.

---

When to Prefer This Method

The paper explicitly frames MoWE against two alternatives — dense models (T5.1.1 family) and learned-routing MoE models (GShard Top-2) — and provides sufficient comparative data to extract decision rules, though the authors do not state these as an explicit decision framework:

• Prefer MoWE over dense models (e.g., T5) when the primary task requires memorization and retrieval of factual world knowledge (closed-book QA, claim verification, entity-centric tasks), the inference compute budget is tight (you can't afford T5-XL or T5-XXL at inference time even though their accuracy would be desirable), and you have access to a knowledge base (Wikidata, Wikipedia, domain entity list) from which to construct a large routing vocabulary. The efficiency gains are largest on knowledge-intensive tasks: MoWE-Base matches T5-XL on TriviaQA at 4.3× less training time (Table 1). If your task is general NLU (SuperGLUE-style) rather than knowledge retrieval, the advantage is present but smaller — MoWE-Base outperforms T5-Base by 6.3 points but trails T5-Large by 1.6 points (Table 1).

• Prefer MoWE over learned-routing MoE when the task is knowledge-intensive (TriviaQA, WebQuestions, FEVER) and you want to scale to very large numbers of experts (32K+) without dealing with load-balancing losses, router collapse, or the training instability commonly reported in MoE pretraining (Zoph et al., 2022). The paper shows that MoWE matches MoE-Top2 on SuperGLUE (83.5 vs. 83.5 at ~30B scale, Table 2) while significantly outperforming it on knowledge tasks (39.4 vs. 36.2 on TriviaQA). If your task is primarily general NLU with less emphasis on factual knowledge, learned-routing MoE and MoWE are roughly equivalent, and the choice may depend on implementation convenience or the availability of a suitable routing vocabulary.

• Prefer dense models (or learned-routing MoE) over MoWE when the task requires learning new factual knowledge after pretraining (since MoWE's expert freezing prevents this — Appendix B.0.3 shows a 4.2-point drop when experts are unfrozen during finetuning), when you cannot construct a suitable routing vocabulary (e.g., for a language without whitespace delimiters, or for a domain without an entity knowledge base to seed the vocabulary), or when the primary inference cost is dominated by factors other than FLOPs (e.g., memory bandwidth in low-batch-size deployment) where MoWE's all-to-all communication overhead might negate the FLOPs advantage. The paper also does not evaluate MoWE in latency-critical online settings, so dense models remain the safer choice when inference latency is the binding constraint.

The paper does not provide direct comparisons against retrieval-augmented models (RAG, REALM), so no decision rule for MoWE vs. retrieval can be derived from its data. The comparison with memory-augmented models (EaE, TOME) in Table 5 suggests MoWE is competitive but not clearly superior — MoWE-Large trails TOME 2 on TriviaQA (50.2 vs. 54.6) while being simpler to implement — so the choice between parameterized memory and explicit retrieval remains task- and infrastructure-dependent based on factors (retrieval corpus availability, latency tolerance, maintenance burden) that the paper does not evaluate.