EMO: Pretraining Mixture of Experts for Emergent Modularity

🎯 Pitch

Standard mixture-of-experts models fall apart when you try to use only a subset of experts, but EMO trained with a simple document-level routing constraint allows you to drop 75% of experts with just a 1% accuracy loss—because experts spontaneously organize by semantic domains like math or code instead of low-level syntax.

1. Executive Summary

This paper introduces EMO, a Mixture-of-Experts language model designed so that modularity—the ability to independently use and compose expert subsets for specific domains—emerges during pretraining without human-defined domain labels. The core mechanism is a document-level expert pool constraint: all tokens within a given document must select their active experts from a shared subset chosen by the router, enforcing consistent expert usage across tokens that likely share a domain (e.g., code documents route through a code-relevant subset while biomedical documents use a different pool), with the pool size sampled randomly per document during training to support flexible subset sizes at inference. Evaluated on a 1B-active, 14B-total parameter architecture trained on 1 trillion tokens, EMO matches standard MoE performance as a full model while enabling selective expert use across MMLU categories: retaining only 25% of experts incurs a 1% absolute performance drop, and retaining 12.5% incurs only a 3% drop, compared to 10% and 15% drops respectively for standard MoEs that break under the same constraint. Unlike standard MoEs whose experts specialize in low-level syntactic patterns, EMO produces expert clusters aligned with semantic domains (e.g., math, code, biomedical), establishing that modular structure can emerge from document boundaries alone as a weak supervisory signal.

2. Context and Motivation

The Core Problem: Language Models Are Monolithic Black Boxes

The fundamental problem this paper addresses is that current large language models—including Mixture-of-Experts models—are deployed as indivisible monoliths, even when applications require only a narrow slice of their capabilities. If a user needs only code generation, mathematical reasoning, or domain-specific biomedical knowledge, they must still load and execute the complete model with all its parameters. The paper articulates this clearly in Section 1:

"In many deployment settings, applications require only a narrow subset of capabilities—such as code generation, mathematical reasoning, or domain-specific knowledge—but must still serve the full model, incurring unnecessary computational cost and memory use."

This is not merely an efficiency concern. The monolithic paradigm creates a cascade of practical problems:

Memory bottlenecks at deployment: Large Mixture-of-Experts models are growing increasingly sparse—DeepSeek-V3 employs hundreds of experts per layer while activating only a handful per token—but even inactive experts must reside in VRAM at inference time. The paper references this directly (Section 2): "As MoEs grow larger and sparser, memory bottlenecks become a central challenge: even inactive experts need to reside in VRAM at inference time." This means that the memory cost of serving an MoE is proportional to its total parameter count, not its active parameter count, eroding one of the primary motivations for sparsity.
Inability to isolate or update capabilities: A monolithic model cannot have its mathematical reasoning improved without retraining and redeploying the entire system, nor can problematic capabilities (e.g., generating harmful content in specific domains) be surgically removed without affecting others. This prevents the kind of modular maintenance—patching, upgrading, or deprecating individual components—that is standard practice in software engineering.
Deployment inequity: As models scale to hundreds of billions or trillions of parameters, the hardware requirements for serving them become prohibitive for all but the most well-resourced organizations. A model that could be deployed in pieces—loading only the subsets relevant to a given task—would dramatically lower the barrier to using state-of-the-art models in resource-constrained settings.

Why MoEs Seem Like They Should Solve This—And Why They Don't

Mixture-of-Experts architectures appear, on the surface, to offer a natural path toward modularity. An MoE consists of many small feedforward networks ("experts"), with only a small subset activated for any given input token. The intuition is compelling: if experts specialize by domain—some handling code, others handling biomedical text, others handling mathematical notation—then for a coding task, one could simply activate only the code-specialized experts and leave the rest dormant, achieving substantial memory savings.

The paper's Section 2 systematically dismantles this intuition by reviewing empirical findings on expert specialization in standard MoEs:

"Prior work finds that specialization is often driven by surface-level patterns (e.g., token ID that is context-independent) or low-level lexical cues (e.g., prepositions, punctuations) [16, 17], while other works find that specialization is confined to only a tiny subset of experts [18]."

This is a crucial observation: experts in standard MoEs do not learn the kind of high-level, domain-coherent groupings that would enable selective deployment. Instead, they learn to handle specific surface-level features—one expert might specialize in prepositions, another in punctuation, another in proper names. This means that within a single document, tokens activate a diverse set of experts across many different surface-level categories, causing most or all experts to be engaged over the course of processing even a narrowly-domain task. As the paper states:

"This behavior... prevents subsets of the model from being usable independently, limiting the deployability of MoEs in memory-constrained settings."

The paper's own experiments (Section 5.2, Figure 3) confirm this empirically: when a standard MoE is restricted to using only experts selected as relevant for a given MMLU domain, performance degrades sharply—a 10% absolute drop when retaining 25% of experts, falling well below the performance of a memory-matched dense model trained from scratch. This demonstrates that the expert groupings in standard MoEs do not correspond to coherent, domain-level capabilities that can be extracted and used independently.

Prior Approaches and Where They Fall Short

The paper situates its contribution against three distinct lines of prior work, each of which addresses aspects of the modularity problem but falls short in critical ways:

1. Expert Pruning and Post-Hoc Selection

A significant body of work attempts to extract task-specific expert subsets from already-trained MoEs through various selection and pruning strategies (Section 2). Methods like Easy-EP [21] use small validation sets to identify which experts are most relevant for a given downstream task and then discard the rest. The paper acknowledges this line of work but identifies a fundamental limitation:

"This work introduces an MoE that enables selective use of expert subsets for a given downstream task."

The implied critique is that post-hoc expert selection is attempting to recover modularity that was never deliberately built in during training. The paper tests this directly by applying Easy-EP—a state-of-the-art expert pruning method—to both standard MoEs and EMO (Section 5.2, Figure 4). For standard MoEs, Easy-EP outperforms naive router-based selection when larger subsets are retained, but "performance still degrades sharply as the subset size decreases." The paper's conclusion is explicit:

"This suggests that even state-of-the-art selection methods cannot overcome the lack of localized domain-specific capabilities."

In other words: you cannot prune your way to modularity if the model's experts never learned domain-coherent representations in the first place. The routing patterns of standard MoEs distribute domain-relevant computation diffusely across many experts, meaning that any subset small enough to provide meaningful memory savings will inevitably exclude experts that are genuinely necessary.

2. Domain-Labeled Expert Training

A more direct approach is to enforce expert specialization by partitioning training data into predefined domains and training separate experts on each domain. Methods like FlexOlmo [7] and Branch-Train-MiX (BTX) [6] instantiate this idea: train individual expert models on math data, biomedical data, code data, etc., then merge them into a single MoE. While this enables standalone use of expert subsets—you can simply activate the math-trained experts for math tasks—the paper identifies several critical limitations:

Reliance on human-defined priors: The domain taxonomy must be specified in advance. This injects human biases about what constitutes a meaningful domain and where boundaries should be drawn. Real-world data does not always fit neatly into pre-specified categories—a document might blend mathematical notation with code, or scientific prose with historical context.
Rigidity for new domains: "Having fixed domains also restricts flexibility, making it difficult for the model to be applied to new domains during inference" (Section 3). If a downstream application requires capabilities that don't align with the training-time domain partitions, the model has no mechanism to compose relevant experts.
Performance ceiling: Constraining experts to fixed domains limits the model's ability to learn cross-domain patterns that emerge from the data. The paper explicitly states this approach "limits the model's overall performance" (Section 1), suggesting that domain-partitioned training trades away general-purpose quality in exchange for modularity.

3. Training Objectives for Expert Diversity

A third line of work promotes interpretability or diversity across experts through training objectives, but without the explicit goal of making expert subsets independently usable. The paper cites several such approaches [24, 25, 26, 27] that aim to reduce expert redundancy or promote specialization. The limitation is clear:

"such approaches do not ensure that expert subsets are usable in isolation"

Making experts diverse is not the same as making them modular. Experts could be highly specialized—each handling a narrow slice of the input distribution—but if that specialization is along non-domain axes (e.g., one expert handles long-range dependencies, another handles local context), then no subset can independently handle a complete domain-specific task.

The Closest Prior Work: ModuleFormer

The paper identifies ModuleFormer [29] as the closest prior work sharing the goal of training a modular MoE that supports standalone use of expert subsets. ModuleFormer introduces an objective that maximizes mutual information between tokens and experts. However, the paper reports a critical negative finding (Section 2):

"We attempted to reproduce ModuleFormer and found that they do not perform better than standard MoEs, and degrades significantly when less than 40% of experts are retained."

This places a concrete performance floor that EMO must exceed: previous attempts at emergent modularity either underperform standard MoEs at full scale or fail to maintain performance when subsets are small. EMO aims to achieve both—matching standard MoE performance as a full model while maintaining robustness down to very small expert subsets (12.5% retention).

How This Paper Positions Itself

The paper's core conceptual contribution is a shift in framing: rather than treating modularity as something to be recovered post-hoc or imposed through human labels, EMO treats modularity as a first-class training objective that emerges from a simple, self-supervised constraint. The key insight, articulated in Section 3, is that document boundaries provide a weak but sufficient supervisory signal for inducing domain-level expert groupings:

"Our key observation is that tokens within the same document usually come from the same domain. We therefore treat document boundaries as a weak supervisory signal: for each document, the router selects a shared expert pool, and all tokens in that document choose their active experts only from this pool. Different documents can use different pools, allowing modular expert subsets to emerge directly from the training data."

This approach is positioned as a synthesis that avoids the pitfalls of prior work:

No human-defined domains needed: Unlike FlexOlmo and BTX, EMO does not require domain labels on pretraining data. The document pool constraint is entirely self-supervised—it only requires knowing where documents begin and end, which is trivially available in any pretraining corpus.
Modularity is built in, not recovered: Unlike post-hoc expert pruning methods, EMO's modularity emerges during pretraining as a consequence of the training objective. Expert subsets are inherently coherent because the training process forces them to be.
Full model performance is preserved: Unlike ModuleFormer and fixed-domain approaches, EMO matches standard MoE performance as a full model (Table 1), demonstrating that modularity need not come at the cost of general capability.

The paper also positions EMO within a broader vision of composable model architectures. The abstract frames this explicitly:

"Altogether, our results demonstrate a path toward modular, memory-efficient deployment of large, sparse models and open new opportunities for composable architectures."

This suggests that the goal is not merely efficient deployment (though that is a primary motivation), but a more fundamental shift in how language models are built, maintained, and deployed—moving from monolithic artifacts toward systems with identifiable, isolatable, and recombinable components.

The Practical Stakes

To understand why this problem matters, consider the trajectory of MoE scaling: models like DeepSeek-V3 employ hundreds of experts with extreme sparsity (activating only a handful per token). As this ratio grows—more total experts, same or fewer active experts—the memory overhead of inactive experts becomes the dominant deployment cost. The paper's approach offers an orthogonal efficiency axis: rather than reducing the number of active parameters per token (the traditional MoE efficiency story), EMO reduces the number of parameters that must be loaded at all for a given task. For a domain-specific deployment, one could serve a 128-expert EMO model using only 16 experts (12.5%) with minimal performance degradation, representing roughly an 8× reduction in memory requirements.

This is not merely incremental. It opens the possibility of deploying models that are too large to fit in any single GPU's memory by distributing only the necessary expert subsets, enabling genuinely large-scale models to be served in resource-constrained environments—on-device, at the edge, or in settings where GPU memory is the binding constraint rather than FLOPs.

3. Technical Approach

3.1 Reader Orientation

EMO is a Mixture-of-Experts language model trained with a modified routing objective that makes expert subsets independently usable for domain-specific tasks. It solves the problem of monolithic deployment—where the full model must be loaded even for narrow-domain applications—by constraining all tokens within a document to select their active experts from a shared pool, causing expert groupings aligned with semantic domains to emerge automatically during pretraining without any human-provided domain labels.

3.2 Big-Picture Architecture (Diagram in Words)

EMO has four major components that interact during training and inference:

Base Transformer Backbone — a decoder-only Transformer with each feedforward sublayer replaced by a sparse mixture of n = 128 experts (127 routed + 1 shared). At each token position, exactly k = 8 routed experts are activated from the pool of 127.
Router Network — a learned linear layer at each MoE layer that produces logits over the nr = 127 routed experts given a hidden state. These logits are softmaxed to produce a probability distribution, and the top-k experts are normally selected. In EMO, the router operates in two stages: first selecting a document-level expert pool, then constraining per-token selection to within that pool.
Document-Level Expert Pool Constraint — the core EMO mechanism. For each document during training, the router computes an average routing distribution across all tokens, selects the top-d experts to form a shared pool, and forces every token in that document to route only to experts within this constrained set. The pool size d is sampled randomly per document from a uniform distribution between k and nr.
Global Load Balancer — an auxiliary loss computed over aggregated routing statistics across all data-parallel groups, encouraging uniform expert utilization across documents (rather than within micro-batches) to prevent the document-pool constraint from causing expert collapse.

Information flows as follows during training: a document enters → the router computes per-token routing probabilities → these are averaged across tokens → the top-d experts form the document pool → each token's routing probabilities are masked to only include pool experts and renormalized → the top-k experts within the pool are activated → the token is processed by those experts → the language modeling loss and load balancing loss are computed → gradients flow back through the entire pipeline, including through the pool selection mechanism.

3.3 Roadmap for the Deep Dive

First, the standard MoE formulation (Section 3.1), which establishes the baseline architecture, routing mechanics, and training objectives that EMO modifies.
Second, EMO's document-level pool constraint (Section 3.2), walking through how the pool is selected, how per-token routing is constrained, and why document boundaries serve as a weak supervisory signal.
Third, the load balancing challenge and its resolution (Section 3.3, Consideration 1), explaining why standard micro-batch load balancing conflicts with document-pool routing and how global load balancing resolves this tension.
Fourth, the dynamic pool size mechanism (Section 3.3, Consideration 2), detailing why training with a fixed pool size limits inference-time flexibility and how random sampling of d enables robust performance across all subset sizes.
Fifth, the key training and architectural choices validated through ablation (Appendix A), including the shared expert, pre-norm architecture, hyperparameter selection, and annealing experiments.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a training methodology paper whose core idea is that modularity in Mixture-of-Experts models can emerge from a simple, self-supervised constraint—restricting all tokens in a document to route through a shared expert pool—applied during standard autoregressive pretraining, without requiring domain labels, task-specific data partitioning, or post-hoc expert selection.

Standard Mixture-of-Experts Architecture and Training

Before describing EMO's modifications, the paper establishes the standard MoE formulation that serves as both the baseline architecture and the foundation upon which EMO builds.

Architecture. EMO uses a decoder-only Transformer language model where each feedforward sublayer is replaced by a sparse mixture of expert networks. The total number of experts n consists of nr routed experts (selected dynamically per token) and ns shared experts (always active for every token): n = nr + ns. In the main experiments, nr = 127, ns = 1, and k = 8 experts are activated per token.

Per-Token Routing. Given the hidden state xt at token position t, the router produces logits over the routed experts via a learned linear transformation:

$r(x_t) \in \mathbb{R}^{nr}$

where r(xt) is a vector of unnormalized scores (logits) for each of the nr routed experts.

The routing probabilities are obtained by applying softmax to these logits:

$p_t = \text{softmax}(r(x_t))$

where pt is a probability vector of length nr that sums to 1, representing the router's learned affinity of token t for each expert.

What it computes: a learned mapping from the token's hidden representation to a probability distribution over experts. For each expert i, (pt)i represents the unnormalized routing weight—how strongly the router believes expert i should process token t, before any selection or renormalization.

Why this form: the softmax ensures a proper probability distribution (non-negative, summing to 1), which enables gradient-based training of the router. Alternatives like hard assignment (e.g., routing to exactly one expert deterministically) would block gradient flow and prevent the router from learning which experts are appropriate for which tokens.

Expert Selection. From the routing distribution, the top-k experts are selected:

$K_t = \text{Top-}K(p_t, k) \subseteq \{1, \dots, nr\}$

where Kt is the set of indices of the k experts with the highest routing probabilities for token t. Only these k experts are activated; all others are bypassed. This is the sparsity mechanism that makes MoEs computationally efficient despite having many total parameters.

Feedforward Output. The MoE feedforward output for token t is the sum of the outputs of the activated routed experts (weighted by their routing probabilities) plus the outputs of all shared experts:

$\text{FFN}_{\text{out}}(x_t) = \sum_{i \in K_t} (p_t)_i \, E_i(x_t) + \sum_{j=1}^{n_s} E_j^{(s)}(x_t)$

where Ei(xt) is the output of the i-th routed expert applied to hidden state xt, (pt)i is the routing weight for that expert, and E_j^{(s)}(xt) is the output of the j-th shared expert.

What it computes: the final feedforward output for token t as a weighted combination of the selected routed experts plus all shared experts. The routing weights (pt)i serve as gating coefficients—experts with higher routing probabilities contribute more to the output, while unselected experts contribute nothing.

Why this form: the weighted sum preserves gradient flow through the router (the weights (pt)i are differentiable) while maintaining sparsity (only k routed experts are actually computed). Shared experts are always active to provide a stable computational base and capture general-purpose transformations that should not be sparsified.

Language Modeling Objective. The model is trained using the standard autoregressive language modeling loss:

$L_{CE} = -\sum_{t=1}^{T} \log P(x_t \mid x_{<t})$

where T is the number of tokens in the sequence and P(xt | x<t) is the model's predicted probability of token xt given all previous tokens.

What it computes: the negative log-likelihood of the training data under the model—the standard cross-entropy objective that encourages the model to assign high probability to the correct next token at each position.

Why this form: this is the fundamental objective for autoregressive language modeling. It provides a principled probabilistic training signal and is standard across virtually all LLM pretraining, ensuring fair comparison with baselines.

Load Balancing Loss. To prevent the router from collapsing to always selecting a small subset of experts (which would waste the remaining experts and degrade training efficiency), MoE training includes a load balancing auxiliary loss:

$L_{LB} = n_r \sum_{i=1}^{n_r} \bar{f}_i \cdot \bar{P}_i$

where nr is the number of routed experts, \bar{f}_i is the fraction of tokens routed to expert i, and \bar{P}_i is the average routing probability assigned to expert i across all tokens.

What it computes: a scalar penalty that is minimized when every expert receives an equal fraction of tokens (\bar{f}_i = 1/nr) and an equal average routing probability (\bar{P}_i = 1/nr). The product \bar{f}_i \cdot \bar{P}_i is large when either the token fraction or routing probability for expert i deviates from uniformity, and the sum across experts penalizes overall imbalance.

Why this form: the product formulation jointly penalizes under-utilization (low \bar{f}_i) and over-utilization (high \bar{f}_i). Alternative formulations that penalize only the fraction or only the probability would allow the router to find degenerate solutions—e.g., assigning high probability to all experts but only selecting a few, or selecting all experts equally but with very low confidence on the non-selected ones.

Router Z-Loss. An additional regularization term penalizes large router logits:

$L_{RZ} = \text{(logit regularization term)}$

where the exact formulation is not detailed in the main text but serves to prevent the router logits from growing unbounded, which would cause the softmax to saturate (producing near-one-hot distributions) and reduce gradient signal.

Full Training Objective. The complete objective combines these components:

$L = L_{CE} + \alpha L_{LB} + \beta L_{RZ}$

where α and β are hyperparameters controlling the strength of the auxiliary losses. Through ablations (Appendix A.4), the paper settles on α = 1e-1 for the load balancing coefficient and a learning rate of 4e-3.

EMO's Document-Level Expert Pool Constraint

The core innovation of EMO is a modification to the routing mechanism that enforces within-document expert consistency. Rather than allowing each token to independently select its top-k experts from the full set of nr routed experts, EMO first selects a document-level expert pool and constrains all tokens in the document to route within that restricted set.

Document Expert Pool Selection. For a document with T tokens, the router first computes the average routing distribution across all tokens:

$\bar{p} = \frac{1}{T} \sum_{t=1}^{T} p_t \in \mathbb{R}^{nr}$

where pt is the routing probability vector for token t (computed identically to the standard MoE formulation) and \bar{p} is the document-level average routing distribution—a vector of length nr where each entry i represents the average routing probability assigned to expert i across all tokens in the document.

What it computes: an aggregate measure of which experts the document "wants" to route to, by averaging the per-token routing preferences. If tokens in a code document consistently assign high probability to experts 3, 17, and 42, then \bar{p} will have elevated values at those indices, providing a document-level signal of expert relevance.

Why this form: averaging is the simplest aggregation that preserves the router's learned preferences while providing a single document-level signal. Alternatives like max-pooling would capture only the strongest preferences per token but lose information about moderate but consistent preferences across many tokens. The average naturally upweights experts that are consistently preferred across the document.

The document expert pool D is then selected as the top-d experts according to this average distribution:

$D = \text{Top-}K(\bar{p}, d) \subseteq \{1, \dots, n_r\}$

where d is the document pool size hyperparameter and D is the set of expert indices that form the constrained pool.

Constrained Routing. Once the document pool is established, each token's routing distribution is masked to only include experts in D and renormalized:

$\hat{p}_t(i) = \begin{cases} \frac{p_t(i)}{\sum_{j \in D} p_t(j)} & \text{if } i \in D \\ 0 & \text{otherwise} \end{cases}$

where \hat{p}_t(i) is the renormalized routing probability for expert i at token t.

What it computes: a cleaned probability distribution where experts not in the document pool receive zero probability, and the remaining experts' probabilities are scaled up proportionally so that \hat{p}_t sums to 1. This ensures that the routing mechanism remains well-defined (a proper probability distribution) while enforcing the constraint of using only pool experts.

Why this form: renormalization preserves the relative preferences among pool experts—if the original router preferred expert A over expert B by a 2:1 ratio within the pool, that ratio is maintained after renormalization. Simply zeroing out non-pool experts without renormalization would produce an improper distribution (summing to less than 1) and break the probabilistic interpretation.

The routed experts for token t are then selected from within the pool:

$R_t = \text{Top-}K(\hat{p}_t, k)$

The feedforward output is computed identically to the standard MoE, but using the constrained routing distribution:

$\text{FFN}_{\text{out}}(x_t) = \sum_{i \in R_t} (\hat{p}_t)_i \, E_i(x_t) + \sum_{j=1}^{n_s} E_j^{(s)}(x_t)$

The Intuition Behind Why This Works. The paper's key insight is that document boundaries provide a weak but sufficient supervisory signal for domain-level expert grouping. The reasoning proceeds as follows:

Tokens within a document usually share a domain. A document from a code repository contains code-related tokens throughout; a biomedical abstract contains biomedical terminology throughout; a news article contains journalistic prose throughout. This is a statistical regularity of pretraining corpora—not a guarantee, but reliable enough to serve as a training signal.
The router learns to map tokens to experts based on the tokens' content. During training, the language modeling objective pushes the router to select experts that minimize the next-token prediction loss. If certain experts are genuinely better at processing code tokens, the router will learn to assign higher probabilities to those experts for code tokens.
The document-pool constraint forces expert preferences to be consistent. By requiring all tokens in a document to use the same pool, the constraint acts as a regularizer that pushes the router to assign related tokens to related experts. If a document contains both code syntax tokens and code comment tokens, the pool constraint encourages the router to find a set of experts that handles both well, rather than routing them to entirely disjoint expert sets.
Different documents can use different pools. Critically, the constraint applies per-document, not globally. A math document can select a math-relevant pool while a legal document selects a legal-relevant pool. This preserves the model's ability to specialize across the full diversity of the training corpus, while enforcing that within any single document, expert usage is coherent.
The constraint is applied during training only (not during inference), so it shapes the learning dynamics without permanently restricting the model's routing behavior. At inference time as a full model, the standard per-token top-k routing is used, allowing experts from any pool to be activated. However, because the training constraint caused experts to organize into domain-coherent groups, the full model's routing naturally tends to co-activate related experts for domain-consistent inputs.

Why Document Boundaries Rather Than Explicit Domain Labels. The paper explicitly argues against using human-defined domain labels, which is the approach taken by methods like FlexOlmo and BTX. The rationale includes:

Ambiguity and bias in domain labels: "this formulation requires domain labels across pretraining data, which can be ambiguous, difficult to obtain, and injects human biases" (Section 3). A document containing both mathematical derivations and code examples—or one discussing the history of science alongside scientific concepts—would be forced into a single pre-specified category, potentially distorting the model's learning.
Rigidity for deployment: "Having fixed domains also restricts flexibility, making it difficult for the model to be applied to new domains during inference" (Section 3). If the training-time domains are Math, Code, and Biomedical, the model has no mechanism to form a pool relevant to a new domain like Game Development that blends code, creative writing, and visual design concepts.
Document boundaries are universally available: Every pretraining corpus already segments text into documents. No additional annotation, labeling pipeline, or human effort is needed. The signal is coarse (documents may contain topic shifts, mixed content, or fragmented text) but abundant—trillions of tokens across billions of documents provide enough statistical power for the constraint to be effective.

The Hyperparameter d Controls Granularity. The pool size d is the primary knob controlling the trade-off between specialization and flexibility:

At one extreme, d = k (pool size equals active expert count) forces all tokens in a document to use exactly the same k experts. This produces maximally specialized pools—a code document can only use the 8 experts its pool selected—but severely limits expressivity since every token in the document must share the exact same expert set, regardless of the token's specific role.
At the other extreme, d = nr (pool size equals total routed experts) recovers the standard MoE formulation—the "pool" is all experts, so the constraint has no effect. This provides maximum flexibility but no modularity pressure.
Intermediate values (k < d < nr) balance these forces: the document has a larger pool than the number of active experts per token, so different tokens can activate different subsets within the pool, but the pool is small enough that experts within it develop coherent, domain-aligned specializations.

In the main experiments, d is not fixed to a single value. Instead, it is sampled randomly for each document during training (discussed in detail below).

Load Balancing: Resolving the Conflict Between Pool Constraints and Expert Utilization

A central technical challenge arises from the interaction between the document-level pool constraint and the standard load balancing mechanism. Understanding this conflict—and its resolution—is essential to why EMO trains successfully.

The Conflict. In standard MoE implementations, the load balancing loss is typically computed over micro-batches (the subset of documents processed on a single GPU before gradient synchronization). The auxiliary loss LLB encourages uniform expert utilization within each micro-batch. This creates a direct tension with EMO's document-pool constraint:

The micro-batch load balancer wants tokens across documents to spread across many experts uniformly. If a micro-batch contains 4 documents, the balancer pushes each document's tokens to activate a diverse set of experts across the entire micro-batch.
The document-pool constraint wants tokens within each document to concentrate on a small subset of experts. If a document selects a pool of size d, its tokens should predominantly route to those d experts.

These two objectives are fundamentally opposed at the micro-batch level: the load balancer is trying to maximize per-micro-batch entropy over expert usage, while the pool constraint is trying to minimize per-document entropy (concentrating usage within the pool). The paper describes this as "the load-balancing loss is computed over only a few documents. While this local implementation reduces cross-device communication and simplifies distributed training, it also encourages tokens from the same document to spread across many experts, directly opposing the shared-pool constraint and causing unstable training" (Section 3.3, Consideration 1).

The Resolution: Global Load Balancing. The paper adopts global load balancing, a technique introduced by Qwen [3, 31], which computes load balancing statistics over aggregated data across all data-parallel groups rather than within individual micro-batches.

The standard (micro-batch) load balancing loss is defined as:

$L_{LB} = \frac{1}{n_p} \sum_{j=1}^{n_p} \left[ n_r \sum_{i=1}^{n_r} f_i^j \cdot P_i^j \right]$

where np is the number of data-parallel groups, f_i^j is the fraction of tokens in micro-batch j routed to expert i, and P_i^j is the average routing probability for expert i in micro-batch j.

What it computes: the loss sums over micro-batches independently, computing per-micro-batch load balance and then averaging. Expert i receives a penalty in micro-batch j if it is under- or over-utilized relative to other experts within that same micro-batch.

Why this fails for EMO: because each micro-batch contains only a few documents (potentially all from similar domains), the per-micro-batch statistics f_i^j can legitimately be skewed—a micro-batch of all code documents should route more to code-specialized experts, which is correct behavior that the load balancer would incorrectly penalize.

The global load balancing variant modifies the frequency computation:

$\bar{f}_i = \sum_{j=1}^{n_p} f_i^j$

$L_{LB} = \frac{1}{n_p} \sum_{j=1}^{n_p} \left[ n_r \sum_{i=1}^{n_r} \bar{f}_i \cdot P_i^j \right]$

where \bar{f}_i is the fraction of tokens routed to expert i aggregated across all data-parallel groups (global batch), rather than within a single micro-batch.

What it computes: the loss now uses the global token fraction \bar{f}_i (same for all micro-batches) while keeping the per-micro-batch routing probabilities P_i^j. This means expert i is penalized for under- or over-utilization relative to the global average, not relative to its micro-batch peers.

Why this resolves the conflict: the global load balancer allows individual micro-batches (and thus individual documents) to have skewed expert usage—a code document can route 90% of its tokens to code experts—as long as, across many diverse micro-batches spanning many domains, each expert receives roughly uniform utilization overall. The document-pool constraint enforces within-document consistency, while the global load balancer enforces across-document diversity. These objectives are now complementary rather than conflicting: the pool constraint makes documents use coherent expert subsets; the global balancer ensures different documents use different subsets.

Empirical Validation. The paper shows in Figure 7 (Appendix A.1) that training EMO with micro-batch-level load balancing leads to unstable training with large gradient norm spikes, while global load balancing produces "consistent and reliable training dynamics." The gradient norm curves visually demonstrate the instability: frequent spikes reaching values of 6-8 in the local variant versus stable values around 2-3 in the global variant.

The specific implementation requires an all-reduce operation across data-parallel groups to aggregate f_i^j into \bar{f}_i. Since the paper uses data parallelism only (no tensor or pipeline parallelism), this corresponds to computing load balancing over the global batch up to gradient accumulation steps.

Dynamic Pool Size: Why Random Sampling of d is Essential

The second critical technical consideration is the choice of pool size d during training. The paper discovers that using a fixed pool size during training severely limits inference-time flexibility.

The Problem with Fixed d. When EMO is trained with a constant pool size d = 32, the model "overfits" to expert subsets of size 32 (Table 2, Appendix A.2). At inference time, when experts are selected for a downstream domain, performance is good when exactly 32 experts are retained but degrades significantly when other subset sizes are used—either smaller (8 or 16 experts) or larger (64 experts). The paper states:

"Fixing a single expert pool size d works well during training but limits inference-time flexibility. The model 'overfits' only to expert sets of size d and performs poorly when deployed as expert subsets that isn't of size d." (Section 3.3, Consideration 2)

This happens because the training distribution of pool sizes shapes the model's learned routing patterns. If every document during training uses a pool of exactly 32 experts, the router learns to organize expertise assuming 32-expert clusters. At inference, if you try to use only 8 experts, the router's learned groupings don't decompose cleanly into 8-expert subsets—capabilities that were distributed across 32 experts now get truncated.

The Solution: Random d During Training. To enable robust performance across all subset sizes, the paper treats d as a random variable and samples it independently for each document during pretraining:

$d \sim U\{k, \dots, n_r\}$

where U{k, ..., nr} is the discrete uniform distribution over integers from k (the number of active experts per token) to nr (the total number of routed experts).

What it computes: for each document in the training batch, a random integer d is drawn uniformly from the range [k, nr]. This pool size determines how many experts the document can route through. Some documents get small pools (forcing concentrated, highly specialized expert usage), others get large pools (allowing more diffuse, flexible routing), and everything in between.

Why this form: the uniform distribution ensures the model sees the full spectrum of pool sizes with equal frequency during training. This prevents the router from developing routing patterns that only work for a specific pool size. The lower bound k is the minimum meaningful pool size (a pool smaller than k would make it impossible to select the required k active experts per token). The upper bound nr recovers the standard MoE routing (no effective constraint), ensuring the model retains the ability to use all experts when beneficial.

Why not a different distribution? The paper doesn't ablate this choice, but the uniform distribution has a straightforward justification: it makes no assumption about which pool sizes will be important at deployment time. A distribution skewed toward small pools would bias the model toward extreme modularity at the cost of full-model performance; one skewed toward large pools would produce weak modularity. The uniform distribution is the neutral choice that maximally exposes the model to the full range.

Empirical Validation. Table 2 (Appendix A.2) demonstrates the practical importance of this design choice. With a fixed d = 32 and nr = 128, MMLU performance at inference with 8 experts retained is 29.6 (without fine-tuning). With dynamic d ~ U(8, 128), the same 8-expert setting achieves 33.7—an improvement of over 4 percentage points. Similarly, with 16 experts: 34.4 (fixed) vs. 36.4 (dynamic). The dynamic training produces consistently better or equal performance at every retained expert count from 8 to 128.

Shared Experts and Architecture Improvements

The paper introduces several architectural modifications relative to the OLMoE baseline that improve both standard MoE and EMO performance.

Shared Experts. Following DeepSeek-MoE [5], EMO includes ns = 1 shared expert that is always active for every token, regardless of routing decisions. The shared expert's output is added to the routed experts' outputs in the feedforward computation.

The ablation in Table 2 shows that including the shared expert improves MMLU performance from 31.9 to 33.6 at full model scale (no fine-tuning). The shared expert likely serves as a repository for general-purpose transformations—layer normalization, positional information, basic syntactic processing—that are needed by all tokens regardless of domain. By offloading these universally-needed computations to a dedicated expert, the routed experts can specialize more sharply to domain-specific patterns.

Pre-Norm with Removed QK-Norm. The paper replaces OLMoE's ReorderedNorm (a variant of post-normalization with reordered layer operations) with standard pre-normalization and removes the query-key normalization (QK-norm). Figure 10 (Appendix A.5) shows that on both standard MoEs and EMO, pre-norm with no QK-norm achieves lower training loss than ReorderedNorm. The paper does not provide extensive analysis of why, but pre-norm is known to enable more stable training in Transformers by normalizing inputs before each sublayer rather than after, preventing activation magnitudes from growing across layers.

Total Expert Count. The paper uses n = 128 total experts (127 routed + 1 shared) rather than OLMoE's 64. This increased expert count provides more capacity for specialization while keeping the active parameter count (1B) the same.

Hyperparameter Selection and Ablation Process

Due to limited compute budget, the paper conducts a sequential ablation process, first tuning the learning rate, then the load balancing coefficient, for both the standard MoE baseline and EMO. The ablations are performed on models trained for 3000 steps (130B tokens), with the best configuration carried forward to the full 1T-token training runs.

Standard MoE Hyperparameters. Starting from OLMoE's defaults (lr = 4e-4, lb = 1e-2):

Learning rate is swept across {4e-4, 4e-3, 4e-2} with lb fixed at 1e-2. As shown in Figure 8, lr = 4e-3 achieves the lowest training loss. lr = 4e-2 causes training instability visible as loss spikes, while lr = 4e-4 converges too slowly.
With lr = 4e-3 fixed, the load balancing coefficient is ablated between 1e-2 and 1e-1. The paper reports "we do not observe significant differences between 1e-1 and 1e-2, and choose the former because it had slightly higher training stability."

The final configuration for the standard MoE baseline is lr = 4e-3, lb = 1e-1.

EMO Hyperparameters. Using the same methodology:

With lb = 1e-1 (carried forward from the standard MoE ablations), the learning rate is swept across {4e-4, 4e-3, 4e-2}. Figure 9 shows lr = 4e-3 offers the best training loss by 3000 steps.
Minor loss spikes observed with lr = 4e-3 are resolved by implementing global load balancing (Section 3.3, Consideration 1—discussed above).

The final configuration for EMO is also lr = 4e-3, lb = 1e-1, matching the standard MoE baseline and ensuring that any performance differences are attributable to the routing objective rather than hyperparameter choice.

Training Duration and Annealing. Both the standard MoE and EMO are trained from scratch on 1 trillion tokens from the OLMoE pretraining corpus, followed by an additional 50B-token linear annealing phase. The annealing phase ramps the learning rate down to zero, which is standard practice for final model quality improvements. For ablations, smaller models are trained on 130B tokens.

Expert Selection at Inference Time

For selective expert use during evaluation, the paper deploys only a task-specific subset of experts. The process operates as follows:

Router-Based Selection. For a given downstream domain (e.g., MMLU math), the system:

Processes a small validation set through the full model, collecting router probabilities at each layer for each token.
Aggregates routing probabilities across all tokens in the validation set, ranking experts by their average routing probability.
Retains the top-d experts in each layer and discards the rest, producing a domain-specific subset of size d per layer.

What it computes: a simple frequency-based ranking of expert relevance—experts that the router consistently assigns high probability to for domain-relevant tokens are retained, others are discarded.

Why this works for EMO but not standard MoEs: In EMO, experts within a domain-relevant pool have genuinely specialized to that domain's patterns, so retaining the top-d by average routing probability captures the coherent expert group. In standard MoEs, domain-relevant computation is distributed diffusely across many experts (because specialization is at the lexical rather than semantic level), so discarding any non-trivial fraction of experts inevitably discards necessary capabilities.

Easy-EP Selection. As an alternative, the paper applies Easy-EP [21], a state-of-the-art expert pruning method that uses a more computationally expensive selection procedure (details of which are in the cited paper, not reproduced here). The key finding (Figure 4) is that EMO's performance is largely insensitive to the choice of selection method—router-based and Easy-EP produce similar results—while standard MoEs benefit from Easy-EP at larger subset sizes but still degrade sharply as subsets shrink. This demonstrates that modularity is a property of the trained model, not an artifact of clever selection.

Sample Efficiency. Section B.2 demonstrates that expert selection for EMO is remarkably sample-efficient. With as few as a single few-shot example, expert subsets maintain strong performance on MMLU and MMLU-Pro. The paper hypothesizes this is because "the presence of few-shot demonstrations in each validation datapoint... may provide sufficient token-level signals" to identify relevant experts. Even with zero-shot prompts and only 5 validation examples, performance degradation is "modest."

Annealing a Standard MoE into Modularity

An interesting additional experiment (Section B.4, Table 3) asks: can modularity be induced after standard MoE pretraining, or does it require training from scratch?

The paper takes a standard MoE pretrained on 1T tokens and anneals it on 50B tokens using EMO's document-level expert pool objective (denoted "EMO-anneal"). The results show that EMO-anneal "trains successfully and exhibits signs of modularity" but underperforms EMO trained from scratch across most benchmarks and expert subset sizes. For example, on MMLU with 8 retained experts, EMO-anneal achieves 32.1 versus EMO's 36.1 (no fine-tuning).

This suggests that while post-hoc modularity induction is partially possible—the annealing phase can reorganize existing expert specializations to be somewhat more domain-coherent—the full benefits require the document-pool constraint to shape routing patterns throughout the entire pretraining process, from the initial random initialization through the full 1T-token curriculum.

Selective Expert Use Workflow

Combining these components, the end-to-end workflow for deploying EMO on a domain-specific task:

Pretraining: Train EMO from scratch on 1T tokens with d ~ U{8, ..., 128}, global load balancing, shared experts, and pre-norm architecture.
Expert Selection (at deployment time): For the target domain, run a small validation set (as few as 1-5 examples) through the full model, collect routing probabilities, rank experts by average routing probability, and retain the top-d experts per layer.
Subset Extraction: Physically extract the retained expert parameters and their corresponding router weights, discarding all other parameters. This produces a much smaller model checkpoint.
Inference (optional fine-tuning): Run the selected expert subset on the target task, optionally fine-tuning it on domain data. The paper shows results both with and without fine-tuning (Figure 3).
Composition: To serve multiple domains, maintain separate expert subsets and load the appropriate one per query, or compose subsets by including experts relevant to multiple domains.

4. Key Insights and Innovations

Innovation 1: Modularity as a First-Class Training Objective, Not a Post-Hoc Extraction

The dominant assumption in MoE research—both in architecture design and deployment—has been that expert specialization is an emergent property of standard training that can be exploited after the fact. The field's approach to selective expert use has been almost entirely extractive: train a standard MoE with conventional objectives, then apply pruning algorithms (Easy-EP [21], expert dropout, importance scoring [15, 22, 23]) to identify and retain task-relevant experts. The implicit model is that modularity is latent in the trained model and just needs to be uncovered.

EMO's central conceptual move is to invert this assumption: modularity is not something you recover from a trained model, but something you must build into the training process itself. The paper makes this explicit when it shows that even state-of-the-art post-hoc selection (Easy-EP) cannot salvage standard MoE performance under small expert subsets (Figure 4): "This suggests that even state-of-the-art selection methods cannot overcome the lack of localized domain-specific capabilities." The failure is not in the selection algorithm—it's in the model's routing structure, which never organized expertise along domain-coherent lines in the first place.

This is a fundamental reframing rather than an incremental improvement. It changes the question from "how do we identify which experts matter for a task?" to "how do we train so that coherent expert groups form in the first place?" The distinction matters because it redirects research effort: improving pruning algorithms for standard MoEs is, under this view, optimizing a fundamentally limited approach. The paper's evidence that EMO is robust to selection method (router-based and Easy-EP produce similar results, Figure 4) reinforces this—when modularity is trained in, selection becomes trivial; when it isn't, no amount of selection sophistication can compensate.

This framing also challenges the implicit assumption in MoE scaling that expert specialization is an unqualified good that naturally improves with scale. Prior work celebrated that experts in large MoEs specialize—to punctuation, prepositions, proper names [16, 17]—as evidence that the architecture "works." EMO recasts this same behavior as a failure mode for deployability: lexical-level specialization is precisely what prevents domain-level expert isolation, because every document contains punctuation, prepositions, and proper names, causing every domain to activate a diffuse set of experts.

Innovation 2: Document Boundaries as Self-Supervised Domain Signals

The paper's second distinctive contribution is the recognition that document boundaries—a structural artifact present in every pretraining corpus but previously ignored as a training signal—can substitute for human-provided domain labels in inducing modularity.

Prior work on training domain-specialized MoEs took one of two paths. Methods like FlexOlmo [7] and BTX [6] required explicit domain labels on pretraining data, forcing researchers to pre-commit to a domain taxonomy and label billions of tokens. This is expensive, introduces human bias about where domain boundaries should be drawn, and creates rigidity—the model cannot form expert groups for domains not anticipated at training time. At the other extreme, methods like ModuleFormer [29] attempted to induce modularity through purely internal objectives (mutual information maximization) with no structural signal at all, but the paper's reproduction effort found this approach underperformed standard MoEs and degraded severely at small subset sizes.

EMO occupies a novel intermediate position: it provides structural guidance (tokens from the same document should route similarly) without specifying what that structure should correspond to (the model discovers which documents are similar and which experts form coherent groups). Document boundaries serve as a weak but abundant supervisory signal—weak because a document may contain topic shifts or mixed content, but abundant because trillions of tokens across billions of documents provide enough statistical power for domain-level patterns to dominate over noise.

This is significant beyond the specific method because it identifies a general principle: structural artifacts of data organization (document boundaries, section breaks, source metadata) can substitute for explicit semantic labels in inducing emergent structure during pretraining. The paper doesn't explore this generalization, but the implication is clear—other structural signals (URL domains for web data, repository structure for code, citation context for academic text) could serve similar roles for different kinds of modularity.

The evidence for this insight is both quantitative and qualitative. Quantitatively, EMO expert subsets maintain near-full-model performance even at 12.5% retention (Figure 3), demonstrating that document-boundary-constrained training produces genuinely isolatable capability groups. Qualitatively, the clustering analysis (Figure 5) shows that EMO experts organize along semantic domains (health, politics, code) rather than lexical features (prepositions, proper names), confirming that the document-pool constraint shifted specialization from syntactic to semantic granularity. The contrast with standard MoE clustering—where the same analysis produces clusters like "prepositions," "copula verbs," and "definite articles"—makes the shift starkly visible.

Innovation 3: The Load Balancing–Modularity Tension as a Diagnostic Insight

Beyond the specific solution (global load balancing), the paper identifies a previously unarticulated tension in MoE training: the standard load balancing objective and any form of structured routing are fundamentally in conflict at the micro-batch level.

This is a diagnostic contribution rather than a methodological one—it explains why prior attempts at structured or specialized MoE training may have been unstable or underperforming, even when the underlying idea was sound. The paper articulates the mechanism clearly: micro-batch load balancing penalizes the exact behavior that modularity constraints try to enforce (concentrated expert usage within documents), creating opposing gradient signals that manifest as training instability (the gradient norm spikes in Figure 7).

What makes this insight significant is that it identifies a design principle that extends beyond EMO: any training objective that encourages structured expert usage (whether document-level, task-level, or modality-level) must be paired with a load balancing mechanism that operates at the same or larger granularity as the structure being enforced. If you want documents to use coherent expert subsets, load balance across documents, not within them. If future work wanted experts to specialize by language, the same principle would apply—load balance across languages, not within language-specific micro-batches.

The empirical validation (Figure 7) is not just a hyperparameter ablation but a demonstration of the principle: EMO with micro-batch load balancing is unstable, while EMO with global load balancing trains smoothly and produces modular expert groups. The specific implementation (all-reduce over data-parallel groups to compute global token fractions \bar{f}_i) is a practical instantiation of the principle, but the conceptual move—recognizing why the conflict exists and at what granularity it must be resolved—is the durable contribution.

This insight also connects to a broader theme in distributed training: as models and training objectives become more structured (mixture-of-experts, mixture-of-depths, domain-adaptive training), the granularity at which auxiliary losses are computed becomes a critical design dimension. The paper provides a clean case study of why and how this matters.

Innovation 4: The Proof That Semantic Modularity Requires Training-Time Induction

The paper provides what amounts to a controlled experiment demonstrating that semantic-level modularity does not emerge from standard MoE training, and that inducing it requires explicit constraints during pretraining.

This finding has the character of a negative result with positive implications: standard MoEs, despite exhibiting expert specialization that looks impressive under analysis (clusters of experts handling specific token types), do not develop the kind of domain-level capability grouping that would enable selective deployment. The paper shows this through multiple converging lines of evidence:

Quantitative: standard MoE performance collapses under expert subset restriction, falling below a memory-matched dense model trained from scratch (Figure 11).
Qualitative: clustering analysis reveals lexical rather than semantic specialization (Figure 5).
Selection-method invariance: even sophisticated post-hoc selection (Easy-EP) cannot recover usable subsets (Figure 4).
Activation similarity: domain-level expert activation patterns in standard MoEs show diffuse similarity across all domains (cosine similarity > 0.6 for most pairs), while EMO produces clearly differentiated patterns (cosine similarity < 0.4) that align with semantic relationships (Figure 6).

The implications extend beyond EMO. This finding suggests that the field's enthusiasm about expert specialization in large MoEs [16, 17, 18] may have been focused on the wrong kind of specialization. Experts specializing in prepositions or punctuation is real specialization—the routing patterns are genuine and reproducible—but it's specialization at a granularity that is useless for the modular deployment goal. The paper essentially argues that the kind of specialization that matters for practical modularity (domain-level, semantically coherent) requires different training incentives than the kind that emerges naturally (surface-level, lexically driven).

This insight also reframes the ModuleFormer result as more than a reproduction failure. The paper tried to reproduce ModuleFormer's mutual-information-based approach and found it "degrades significantly when less than 40% of experts are retained." This places a concrete boundary condition on the claim that modularity can emerge from purely internal objectives: without some structural signal that groups related tokens together (document boundaries, in EMO's case), the model doesn't learn the right kind of groupings, no matter how clever the internal objective.

Innovation 5: Memory-Matched Pareto Improvement Over Training from Scratch

The paper demonstrates that expert subsets extracted from a single EMO model can outperform models trained from scratch under the same memory budget, forming a new Pareto frontier in memory-accuracy trade-off (Figure 1, right; Figure 11).

This is significant because it challenges a reasonable default assumption: that if you know your deployment memory budget (say, 32 experts' worth of parameters), you should simply train a model of that size from scratch. The paper shows instead that training a much larger model (128 experts) with EMO's modularity objective, then extracting only the subset you need, yields better performance than training the smaller model directly. On GSM8K, for instance, the 8-expert EMO subset nearly doubles the performance of a memory-matched dense model trained from scratch (Figure 11, bottom row).

What makes this a fundamental finding rather than an incremental win is that it establishes modularity not just as a deployment convenience but as a training efficiency strategy. The larger EMO model provides a richer representational space during training—all experts can co-specialize and learn from the full diversity of the pretraining corpus—while the document-pool constraint ensures that this richness organizes into coherent, extractable groups. When you extract a subset, you get experts that were trained as part of a larger, more capable system, benefiting from cross-domain learning during pretraining while remaining independently functional.

This finding connects to broader themes in transfer learning and multi-task training: training on more data and more tasks often produces representations that are better even for individual tasks than training on those tasks alone. EMO provides a mechanism for this principle to operate in the MoE architecture specifically, with the document-pool constraint serving as the organizational principle that prevents the larger model's knowledge from becoming too entangled to decompose.

The practical upshot is a shift in how practitioners should think about model training under memory constraints: rather than matching the training budget to the deployment budget, train a larger modular model and extract. This is a different kind of scaling story—not "scale the model to improve full-model performance" (the standard narrative) but "scale the model to improve the quality of the subsets you can extract."

5. Experimental Analysis

5.1 Evaluation Methodology

Dataset. The primary pretraining corpus is the OLMoE pretraining corpus [17], consisting of 1 trillion tokens, with an additional 50B-token linear annealing phase. Evaluation uses five benchmark suites: MC9 (an average over nine multiple-choice benchmarks: ARC-Easy, ARC-Challenge, BoolQ, CSQA, HellaSwag, OpenBookQA, PIQA, SocialIQa, WinoGrande), Gen5 (five generative tasks: CoQA, SQuAD, Natural Questions, TriviaQA, DROP), MMLU [45] (aggregated across 16 domain categories after excluding the "other" catch-all category), MMLU-Pro [46] (aggregated across 13 domain categories, also excluding "other"), and GSM8K [47]. For domain-specific evaluation, MMLU's original 57 subjects are grouped into 17 broader categories, and MMLU-Pro subjects into 14 categories.
Base model(s). The primary architecture is a 1B-active, 14B-total parameter MoE with n = 128 experts (127 routed + 1 shared), activating k = 8 experts per token. The baseline standard MoE and EMO share this identical architecture, differing only in training objective. The architecture improves on OLMoE [17] by adding a shared expert, using pre-norm instead of ReorderedNorm, and removing QK-norm. For memory-matched comparisons, a dense model with 8-expert-equivalent parameters and a standard MoE with 32 experts are trained from scratch. Both the standard MoE baseline and EMO are trained from scratch on 1T tokens from the OLMoE pretraining corpus, with ablations run on 130B-token checkpoints.
Metrics. For multiple-choice benchmarks (all MC9 tasks, MMLU, MMLU-Pro), the primary metric is raw accuracy (acc-raw) — the fraction of examples where the highest-scoring answer choice matches the ground truth, scored by selecting the answer choice with the highest log-likelihood. For generative tasks in Gen5, the metric is F1 score (for SQuAD, CoQA, Natural Questions, TriviaQA, DROP). For GSM8K, the metric is exact match. Performance under selective expert use is reported both with and without fine-tuning. Aggregate MMLU and MMLU-Pro results are macro-averaged across domains. For MMLU and MMLU-Pro, 40% of examples per domain are randomly sampled for validation/expert selection, with the remaining 60% used for evaluation.
Baselines. The paper compares against multiple baselines at different scales and settings:
- OLMoE [17]: a standard MoE with 1B active parameters trained on 5T tokens on the same data mixture, using an outdated architecture (ReorderedNorm, QK-norm, no shared expert, 64 total experts, micro-batch load balancing).
- Standard MoE (Reg. MoE): the paper's own improved MoE architecture (128 experts, shared expert, pre-norm, no QK-norm) trained from scratch on 1T tokens with standard per-token routing — serving as the primary head-to-head comparison against EMO.
- Dense @8: a dense model trained from scratch with parameter count matched to an 8-expert subset of the 128-expert MoE (130B-token setting only).
- Reg. MoE @32: a standard MoE with 32 experts trained from scratch, memory-matched to a 32-expert subset (130B-token setting only).
- Random selection: for selective expert use, a sanity-check baseline where experts are selected uniformly at random rather than by routing probability or Easy-EP.
- Easy-EP [21]: a state-of-the-art post-hoc expert pruning method applied to standard MoEs and EMO for expert subset selection (Section 5.2, Figure 4).
- ModuleFormer [29]: the paper reports a reproduction attempt, finding it "does not perform better than standard MoEs, and degrades significantly when less than 40% of experts are retained" (Section 2), though detailed ModuleFormer results are not included in the main experimental tables.
Generation budget / compute accounting. All models are compared at matched active parameters (1B). For selective expert use, the budget is measured by the number of retained experts per layer d, with the subset sizes swept across {128 (full), 64, 32, 16, 8}. Memory-matched comparisons control for total parameter count loaded at inference. The full-model evaluations in Table 1 compare models at equivalent active parameters but different total parameters (EMO and Reg. MoE at 14B total vs. OLMoE at ~7B total for the 64-expert variant) and different training token budgets (1T vs. 5T). Training compute is measured in tokens processed.
Cross-validation / statistical protocol. For MMLU and MMLU-Pro domain-specific evaluation, 40% of examples are randomly sampled for validation (used for expert selection and optional fine-tuning) and the remaining 60% for testing. For tasks other than MMLU and MMLU-Pro, the original train and validation splits are merged into a combined set for both expert selection and finetuning. When fine-tuning is applied, input tokens are masked and optimization occurs only over output tokens, with one epoch, batch size 32, and learning rate 5 × 10⁻⁵ (standard Hugging Face settings). For GSM8K with small validation sets (n=1, 5, 10), three random seeds are used.

5.2 Main Quantitative Results

Full-Model Evaluation: EMO Matches Standard MoE Performance

Table 1 reports full-model performance across all evaluation suites for models trained on 1T tokens. The headline result is that EMO achieves comparable overall performance to a standard MoE trained with the same architecture and data budget, demonstrating that the document-level pool constraint does not degrade the model's general capabilities.

At the 1T-token scale, comparing EMO to the standard MoE baseline:

MC9: EMO achieves 63.1 vs. Reg. MoE's 63.9 — a difference of 0.8 percentage points.
Gen5: EMO achieves 57.9 vs. Reg. MoE's 59.7 — a difference of 1.8 points.
MMLU: EMO achieves 42.8 vs. Reg. MoE's 42.4 — EMO slightly outperforms by 0.4 points.
MMLU-Pro: EMO achieves 18.5 vs. Reg. MoE's 19.3 — a difference of 0.8 points.
GSM8K: EMO achieves 12.0 vs. Reg. MoE's 13.9 — the largest gap at 1.9 points.

The pattern is consistent: EMO is within 1–2 percentage points of the standard MoE across all benchmarks. Critically, both models substantially outperform the OLMoE baseline trained on 5T tokens (5× more data) in several categories — for instance, EMO's 42.8 on MMLU matches OLMoE's 42.8, and EMO's 57.9 on Gen5 exceeds OLMoE's 57.6. This validates the paper's architectural improvements (shared expert, pre-norm, increased expert count) independently of the modularity objective.

At the 130B-token scale (ablations), the pattern holds similarly: EMO and standard MoE are comparable (e.g., MMLU: 38.1 vs. 37.5; MMLU-Pro: 15.5 vs. 15.8), and both significantly outperform a dense model with matched active parameters (MMLU: 33.0 for Dense, vs. 37.5–38.1 for MoEs), confirming the benefits of sparsity even at moderate training budgets.

The key takeaway from Table 1 is that modularity does not impose a performance tax: the document-pool constraint reshapes how expertise is organized without reducing how much the model learns.

Selective Expert Use: EMO Maintains Performance Under Extreme Subset Restriction

Figure 3 presents the paper's central empirical contribution: performance of expert subsets across MMLU, MMLU-Pro, and GSM8K for models trained on 1T tokens, evaluated both without fine-tuning (top row) and with fine-tuning (bottom row). The findings are stark.

Standard MoEs degrade catastrophically. The green bars tell a clear story: as retained experts shrink from 128 (full model) to 8, standard MoE performance collapses:

MMLU (no fine-tune): 42.4 → 39.4 (64 experts, −3.0) → 30.8 (32 experts, −11.6) → 24.6 (16 experts, −17.8) → 22.7 (8 experts, −19.7). The drop from 128 to 32 experts already exceeds 10 absolute points.
MMLU-Pro (no fine-tune): 19.3 → 16.8 (64) → 13.2 (32) → 9.5 (16) → 9.9 (8). Performance nearly halves at 16 experts.
GSM8K (no fine-tune): 13.9 → 4.9 (64) → 2.4 (32) → 2.3 (16) → 2.9 (8). The model essentially breaks — random guessing performance on a math reasoning task.

With fine-tuning (bottom row), the standard MoE benefits modestly but the same pattern persists. MMLU at 32 experts: 34.2 (fine-tuned) vs. 30.8 (no fine-tune), but still far below the 42.4 full-model performance. GSM8K remains essentially broken at small subsets even with fine-tuning: 4.5 at 8 experts.

EMO remains robust. The purple bars show a qualitatively different pattern:

MMLU (no fine-tune): 42.8 (full) → 42.5 (64 experts, −0.3) → 41.4 (32 experts, −1.4) → 39.9 (16 experts, −2.9) → 36.1 (8 experts, −6.7). At 25% expert retention (32 out of 128), the drop is approximately 1 absolute point.
MMLU-Pro (no fine-tune): 18.5 → 18.2 (64) → 17.6 (32) → 16.6 (16) → 14.7 (8). At 12.5% retention (16 experts), the drop is roughly 2 points.
GSM8K (no fine-tune): 12.0 → 11.0 (64) → 11.7 (32) → 12.2 (16) → 6.9 (8). Strikingly, at 16 experts (12.5% retention), performance actually improves slightly over the full model (12.2 vs. 12.0), and even at 8 experts, the model retains more than half its full performance.

With fine-tuning, EMO subsets recover near-full-model performance at surprisingly small sizes:

MMLU (fine-tuned): 43.6 (full) → 43.3 (64) → 41.7 (32) → 40.1 (16) → 37.3 (8). At 32 experts, performance is 41.7 vs. 43.6 — a drop of less than 2 points.
GSM8K (fine-tuned): 27.8 (full) → 27.1 (64) → 27.5 (32) → 28.3 (16) → 23.3 (8). At 32 and 16 experts, fine-tuned EMO subsets match or exceed full-model performance (27.5 and 28.3 vs. 27.8).

The paper describes these results as "≈1% drop at 25% parameters and ≈3% drop at 12.5%" for the without-fine-tuning setting, and similar patterns with fine-tuning. These numbers are approximately consistent with Figure 3, with the 12.5% figure corresponding to the 16-expert column.

Expert Subsets Outperform Memory-Matched Models Trained from Scratch

Figure 1 (right) and the more detailed Figure 11 (Appendix B.1) compare EMO expert subsets against models trained from scratch with equivalent memory budgets. This comparison addresses the question: given a fixed memory budget, is it better to train a smaller model directly or to extract a subset from a larger modular model?

The 130B-token results (Figure 11, bottom row) show:

MMLU (fine-tuned): EMO @32 experts achieves 38.5 vs. Reg. MoE @32 (trained) at 37.1 — a 1.4-point advantage. EMO @8 experts achieves 34.5 vs. Dense @8 (trained) at 33.2 — a 1.3-point advantage.
MMLU-Pro (fine-tuned): EMO @32 achieves 16.3 vs. Reg. MoE @32 (trained) at 14.7. EMO @8 achieves 14.1 vs. Dense @8 at 12.7.
GSM8K (fine-tuned): EMO @8 achieves 13.3 vs. Dense @8 at 7.6 — nearly double the performance. EMO @16 achieves 15.2 vs. 14.3 for Dense @8.

The paper's claim that "EMO expert subsets push the Pareto frontier in memory-accuracy trade-off" (Figure 1 caption) is supported: at equivalent memory budgets, EMO subsets consistently match or outperform purpose-trained smaller models, with the advantage being most dramatic on reasoning-heavy tasks like GSM8K.

EMO Is Robust to Expert Selection Method

Figure 4 compares three expert selection strategies — random selection, router-based selection (the primary method used throughout the paper), and Easy-EP [21], a state-of-the-art expert pruning method — applied to both standard MoEs and EMO.

For standard MoEs (green lines), the story is nuanced:

Easy-EP (green, dashed) outperforms router-based selection (green, solid) when moderately large subsets are retained — e.g., at 64 experts on MMLU, Easy-EP achieves higher accuracy than router-based. This is consistent with Easy-EP's design as a more sophisticated selection algorithm.
However, as the subset size shrinks, Easy-EP performance still "degrades sharply" (Section 5.2), falling well below useful levels. On GSM8K at 16 experts, even Easy-EP-selected standard MoE subsets achieve near-zero performance.
Random selection (dotted lines) converges quickly to random performance, confirming that expert selection is non-trivial.

For EMO (purple lines), the findings are qualitatively different:

Router-based and Easy-EP selection produce nearly identical results across all subset sizes and all benchmarks. On MMLU, the solid and dashed purple lines are essentially superimposed. This is a key finding: when modularity is trained in, the selection method barely matters because the experts are already organized into coherent groups — the router's natural preferences are sufficient to identify relevant subsets.
Random selection underperforms (as expected), but EMO with random selection degrades more gracefully than standard MoEs — further evidence that EMO's expert groupings are more coherent, so even a random subset is more likely to contain a meaningful fraction of relevant experts.

The paper draws the explicit conclusion: "This highlights that modularity must be learned during training, rather than recovered through post hoc expert selection" (Section 5.2).

Sample Efficiency of Expert Selection

Figure 12 (Appendix B.2) ablates the validation data requirements for expert selection in EMO. The key results:

With few-shot prompts (the default setting, where both validation examples for expert selection and test examples include few-shot demonstrations), EMO shows "little degradation as validation set size decreases — even down to a single example" (Appendix B.2). On MMLU at 16 retained experts, performance is approximately constant whether the full validation set, 100 examples, 10 examples, 5 examples, or a single example is used for expert selection. The authors hypothesize that few-shot demonstrations provide sufficient token-level signals for identifying relevant experts.
With zero-shot prompts, performance degrades more noticeably as validation set size decreases, but "the drop remains modest even with only 5 validation examples." The paper does not provide exact numbers for this degradation curve beyond the qualitative description.
An unexpected finding on GSM8K: "performance improves as the validation set size decreases." The paper speculates that "smaller validation sets produce more focused estimates of expert relevance, whereas aggregating across multiple examples can smooth these signals and yield less specialized expert subsets." This suggests a non-trivial relationship between selection data quantity and subset quality that may be task-dependent.

Modularity Requires Training from Scratch (but Annealing Partially Works)

Table 3 (Appendix B.4) compares EMO (trained from scratch with the document-pool constraint for the full 1T tokens) against "EMO-anneal" (a standard MoE trained on 1T tokens, then annealed on 50B tokens using the document-pool constraint).

Across all benchmarks and subset sizes, EMO outperforms EMO-anneal:

MMLU, 8 experts (no fine-tune): EMO 36.1 vs. EMO-anneal 32.1 (4-point gap).
MMLU, 32 experts (fine-tuned): EMO 41.7 vs. EMO-anneal 39.9.

However, EMO-anneal does show signs of modularity — it significantly outperforms the un-annealed standard MoE at small subset sizes (compare EMO-anneal at 8 experts: 32.1 vs. standard MoE at 8 experts: 22.7 from Figure 3). This demonstrates that the document-pool constraint can partially reorganize expert specializations even after standard pretraining, but the full benefits require the constraint throughout the entire training process.

Domain-Level Activation Similarity Analysis

Figure 6 provides a complementary analysis to the clustering results, measuring cosine similarity between domain-level expert activation vectors across 24 human-labeled domains from WebOrganizer [48]. The activation vectors are computed by first averaging router probabilities across tokens within each document, then averaging these document-level vectors across all documents in a given domain.

The key finding: EMO produces distinctly differentiated activation patterns across domains, while standard MoEs show diffuse similarity. Specifically:

In EMO, cosine similarity between domain pairs is predominantly below 0.4, especially in deeper layers (Layers 10 and 15), indicating that different domains activate markedly different expert patterns. Related domains (e.g., software_development and software) show higher similarity, while unrelated domains are clearly separated.
In standard MoEs, cosine similarity between most domain pairs exceeds 0.6, indicating that all domains activate experts in roughly similar proportions. There is little domain-level differentiation.
Across both models, early layers (Layer 0) show limited domain structure, with differentiation emerging progressively in deeper layers — "domain-level expert specialization emerges progressively in deeper layers" (Section 5.3).

5.3 Ablation Studies and Robustness Checks

Shared experts: Including a shared expert (ns = 1) improves EMO performance. Table 2 (Appendix A.2) compares: without shared expert, nr = 128, d = 32 → MMLU 31.9 (full model, no fine-tune); with shared expert, nr = 127, ns = 1, d = 32 → MMLU 33.6. The improvement is ~1.7 points on MMLU. This is consistent with prior work (DeepSeek-MoE [5]).

Dynamic vs. fixed pool size d: Training with a single fixed pool size (d = 32) severely limits flexibility at inference. Table 2 shows: fixed d = 32 achieves MMLU 33.6 at full model but only 29.6 at 8 experts (no fine-tune) — a drop of 4 points. Dynamic d ~ U(8, 128) achieves 38.1 at full model and 33.7 at 8 experts — a more uniform performance profile across all subset sizes. With fine-tuning, the gap is similar: fixed d = 32 achieves 31.7 at 8 experts vs. dynamic d achieving 34.5. The paper's conclusion that "models trained with a fixed d perform well at that specific expert subset size but degrade when evaluated at other subset sizes" is well-supported.

Pre-norm vs. ReorderedNorm: Figure 10 (Appendix A.5) shows that on both standard MoEs and EMO, using pre-norm with removed QK-norm achieves lower training loss than OLMoE's ReorderedNorm. The curves are separated by a visible margin by 3000 training steps. Ablations were conducted without global load balancing, shared experts, and dynamic d, isolating the norm architecture effect.

Global vs. local load balancing: Figure 7 (Appendix A.1) demonstrates that EMO trained with standard micro-batch load balancing produces unstable training with frequent gradient norm spikes reaching values of 6–8, while global load balancing produces stable training with gradient norms consistently around 2–3. This ablation confirms the theoretical tension between document-pool routing and micro-batch load balancing described in Section 3.3.

Learning rate and load balancing coefficient: Figures 8 and 9 (Appendix A.4) show training loss curves for both standard MoE and EMO across hyperparameter sweeps. For the standard MoE, lr = 4e−3 significantly outperforms lr = 4e−4 (lower loss by 3000 steps), while lr = 4e−2 causes instability. For EMO, the same lr = 4e−3 is best. The load balancing coefficient sweep (1e−2 vs. 1e−1) shows minimal difference in loss, with 1e−1 chosen for "slightly higher training stability." The paper acknowledges these ablations were "in increments of 10x" due to limited compute budget, so finer-grained tuning might yield further improvements.

EMO-anneal (post-hoc modularity induction): Table 3 shows that annealing a standard MoE on the document-pool objective partially induces modularity but underperforms training from scratch. This is a notable finding because it suggests that expert routing patterns, once established during standard training, are not easily reorganized — the initial training creates path dependencies that constrain later reorganization.

5.4 Critical Assessment

The experiments provide strong evidence for the paper's central claims, but several limitations warrant attention.

Claim: EMO matches standard MoE performance while enabling modularity. The evidence from Table 1 supports this at the 1B-active scale and 1T-token budget. EMO is within ~1–2 points of the standard MoE across all benchmarks, with the largest gap on GSM8K (12.0 vs. 13.9). On MMLU, EMO slightly outperforms (42.8 vs. 42.4). However, two caveats apply. First, this is a single model scale — whether EMO continues to match standard MoEs at larger scales (e.g., 10B+ active parameters) is untested. The document-pool constraint could impose a regularization effect that becomes more or less pronounced at different scales. Second, the baseline comparison is to the paper's own improved standard MoE architecture, which already significantly outperforms OLMoE — this is a strong baseline, but the architectural improvements (pre-norm, shared expert) benefit both models equally, so the comparison isolates the routing objective effect. A missing ablation would be EMO vs. standard MoE at equivalent total parameters rather than equivalent active parameters — EMO uses 128 experts while OLMoE uses 64, making the total parameter comparison slightly asymmetric.

Claim: EMO enables selective expert use with minimal degradation. The evidence from Figure 3 is the paper's strongest result. At 25% expert retention (32/128), the drop is approximately 1% absolute on MMLU and MMLU-Pro; at 12.5% (16/128), the drop is approximately 2–3%. These numbers are genuinely impressive. However, several observations:

The "1% drop at 25% retention" figure is most accurate for MMLU and MMLU-Pro without fine-tuning. On GSM8K without fine-tuning, EMO at 32 experts actually improves slightly (11.7 vs. 12.0 full model), which is interesting but also means the claim is domain-specific. With fine-tuning, the pattern varies: MMLU at 32 experts drops ~2 points (43.6 → 41.7), which is still small but exceeds 1%.
The comparison against "random" as a sanity check (gray bars in Figure 3) is useful but the paper could strengthen the argument by comparing against additional degenerate baselines, such as selecting experts by parameter magnitude or by activation frequency on a generic corpus (rather than domain-specific validation data).
The most dramatic failure of standard MoEs is on GSM8K. The paper provides qualitative generation examples in Appendix B.5 showing that 8-expert standard MoE subsets produce repetitive or nonsensical outputs (e.g., "Olivia $2005 2005 2005 2005..."), while EMO 8-expert subsets produce coherent reasoning chains, sometimes with correct answers. However, these are selected examples, not a systematic analysis of failure modes. It would be informative to know what fraction of standard MoE subset failures are due to degenerate generation (like the repetitive example) vs. plausible-but-wrong reasoning.

Claim: Expert clusters in EMO align with semantic domains. The clustering analysis (Figure 5) and domain similarity analysis (Figure 6) provide converging evidence, but both have interpretive caveats. The cluster labels in Figure 5 are assigned by Claude Code — a post-hoc interpretation that could introduce anthropomorphic bias. The paper acknowledges this by publishing the interactive visualization (emovisualization.netlify.app), which lets readers inspect the raw clusters, but the main-text presentation relies on the assigned labels. The domain similarity analysis (Figure 6) uses WebOrganizer's 24 human-labeled domains, which is more objective but limited to the domain taxonomy that WebOrganizer provides. An analysis using unsupervised topic modeling (e.g., LDA) would complement these supervised analyses and avoid potential label bias.

Claim: EMO expert subsets outperform memory-matched models trained from scratch. The evidence for this (Figure 1, right; Figure 11) is solid but limited to the 130B-token training budget. The memory-matched baselines (Dense @8, Reg. MoE @32) are trained on the same 130B tokens, making the comparison fair at that budget. However, the paper does not train memory-matched baselines for the full 1T-token models, which would be the most compelling demonstration. If EMO @32 experts (extracted from a 1T-trained, 128-expert model) outperforms a standard 32-expert MoE trained on 1T tokens, that would definitively establish the "train large, deploy small" advantage. The current evidence is from 130B tokens only, and the paper does not state whether the training budget was held constant in FLOPs or in tokens — the 128-expert EMO model processes 130B tokens with 128 experts total, while the 32-expert baseline processes 130B tokens with 32 experts total, creating an asymmetry in total FLOPs.

Missing experiments that would strengthen the paper:

Scale analysis: Results at only one architecture size (1B active, 128 experts). Does modularity improve, degrade, or remain constant as the number of experts increases? The paper positions EMO as especially valuable for "large, sparse models" (Section 6), but all results are at a moderate scale. A scaling curve across expert counts (e.g., 64, 128, 256, 512) at fixed active parameters would test whether the document-pool constraint remains effective as sparsity increases.
Full 1T-token memory-matched baselines: As noted above, comparing 1T-trained EMO subsets against 1T-trained memory-matched models would be the definitive demonstration of the "train large, deploy small" advantage.
Ablation on the uniform distribution for d: The paper convincingly shows that dynamic d is better than fixed d, but does not ablate the distribution shape. Would a distribution skewed toward smaller d (e.g., Beta or geometric) produce better modularity? A uniform distribution treats all pool sizes equally, but it's possible that smaller pools (which enforce stronger constraints) contribute more to modularity than larger pools.
Comparison against domain-label-supervised methods: The paper argues that document boundaries are superior to explicit domain labels, but does not compare EMO against a FlexOlmo/BTX-style approach at matched scale. This would directly test the claim that self-supervised document-boundary training matches or exceeds human-supervised domain partitioning.
Cross-domain composition experiments: The paper claims EMO supports "composable architectures" (abstract, Section 6) but provides no experiments where expert subsets from multiple domains are combined. This is a stated future opportunity, not an evaluated capability. The paper should be more cautious about claiming compositionality without experimental support.
Layer-wise analysis of modularity: Figure 6 shows that domain structure emerges in deeper layers. Does modularity (as measured by subset performance) vary by layer? Are some layers more amenable to expert pruning than others? A layer-wise version of the selective expert use experiment would reveal whether all layers benefit equally from the document-pool constraint.

Potential weaknesses:

The "other" category exclusion is justified but underreported. Figure 13 shows that EMO expert subsets underperform memory-matched baselines on MMLU's "other" category. The paper argues this is a "property of modular models" and excludes it from aggregate metrics. This is a reasonable choice — a catch-all category is not a coherent domain, so domain-specific expert subset selection is ill-defined. However, the paper's abstract and introduction emphasize modularity without mentioning this limitation, which could mislead readers to expect that EMO supports subset deployment for any task rather than only well-defined domain tasks.
Expert selection requires labeled validation data. Although the paper shows sample efficiency (as few as 1 example, Figure 12), the selection process still requires domain-labeled examples. This is a weaker requirement than domain-labeling the entire pretraining corpus (which FlexOlmo requires), but it means EMO's modularity is not fully unsupervised — deployment to a new domain requires at least a handful of labeled examples to identify relevant experts.
The 130B-token baselines may not be fully optimized. The Dense @8 and Reg. MoE @32 baselines were trained on 130B tokens, but the paper does not describe hyperparameter tuning for these models. If the baseline training recipes are suboptimal, the comparison overstates EMO's advantage. The paper's careful hyperparameter tuning for the main 128-expert models suggests attention to fair comparison, but the smaller baselines' tuning is not detailed.
Single model family, single corpus. All experiments use the OLMoE pretraining corpus and an architecture derived from OLMoE. Whether EMO's modularity transfers to other corpora (e.g., more heavily curated datasets, multilingual data) or other model architectures (e.g., different expert counts, different sparsity levels, different base model families) is unknown. The paper's claim that EMO "demonstrate[s] a path toward modular... deployment" is appropriately hedged, but the single-setting evaluation limits generality.
Evaluation skew toward multiple-choice and knowledge tasks. The evaluation suite (MC9, MMLU, MMLU-Pro, Gen5, GSM8K) emphasizes factual knowledge and reasoning, with limited coverage of open-ended generation, dialogue, or creative tasks. The modularity benefits might not extend to tasks where domain boundaries are less clear or where capability overlap is essential.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Not Accounted for in the Headline Efficiency Gains

The assumption or constraint. The entire compute-optimal framework — and therefore EMO's selective expert use — depends on identifying which experts are relevant for a given downstream domain before deploying the subset. The paper's method for doing so requires running a validation set through the full 128-expert model to collect routing probabilities, then ranking experts by their average activation. The paper acknowledges this cost implicitly but does not include it in any performance or efficiency calculation:

"For each domain, we assume access to a small validation set to identify relevant experts" (Section 4.2).

While the paper demonstrates that this validation set can be remarkably small — "even a single few-shot example is sufficient to select an effective expert subset" (Appendix B.2) — a single example still requires a forward pass through the full 14B-parameter model. For a deployment scenario where the goal is to avoid loading the full model at all, this creates an inherent circularity: you must first load and execute the complete model to determine which subset you can safely load.

The consequence. In a realistic deployment, the total cost is expert selection (full-model forward pass on validation data) plus subset inference (forward pass on test data with reduced parameters). For high-throughput settings where many queries are processed after a one-time expert selection step, the amortized cost may be negligible — selecting experts once, then serving thousands of queries with the subset. But for low-volume or ad-hoc use cases (a single user wanting to answer one math question), the expert selection cost could exceed the subset inference cost, eliminating the practical memory advantage. The paper's headline claim that "retaining only 25% of experts incurs just a 1% absolute drop" (Section 1) measures accuracy degradation but not total computational cost including selection overhead.

This limitation is analogous to the difficulty estimation cost problem in the test-time compute scaling paper — in both cases, the reported efficiency gains assume a pre-computed oracle (which experts are relevant, which questions are hard) without accounting for the cost of obtaining that oracle.

What evidence exists in the paper. The paper does not measure or report the computational cost of expert selection relative to subset inference. Figure 12 (Appendix B.2) studies the data requirements for expert selection (how many validation examples are needed), but not the compute requirements. The paper also does not compare end-to-end latency or memory usage between (a) loading the full model once, selecting experts, then serving with a subset, versus (b) simply serving with the full model for all queries.

Mitigation status. The paper does not address this limitation directly. The sample efficiency result (1–5 examples suffice) mitigates the concern partially — the expert selection cost is at worst a small constant overhead per domain — but does not eliminate it. The paper does not propose methods for zero-shot expert selection (identifying relevant experts from the task description alone, without any forward passes) or for amortizing selection cost across queries in a principled way. This is a significant practical gap for deployment scenarios where the full model is genuinely too large to load at all (e.g., on-device settings), since even a single full-model forward pass may be infeasible.

Results Are Demonstrated at a Single Model Scale and Sparsity Level

The assumption or constraint. All experiments in the paper use a single architecture: 1B active parameters, 14B total parameters, n = 128 total experts (127 routed + 1 shared), and k = 8 experts activated per token, trained on 1 trillion tokens. The paper positions EMO as especially valuable for "large, highly sparse models" (Section 6):

"As MoE models scale to trillions of parameters, deploying or adapting them becomes increasingly resource-intensive... Modularity offers an orthogonal path: selectively using small subsets of experts for a given domain, enabling more accessible deployment and adaptation, particularly well-suited for large, highly sparse models."

However, the paper provides no empirical evidence at larger scales, higher sparsity ratios, or different expert counts.

The consequence. Several aspects of EMO's behavior could change with scale, and the direction of change is not obvious a priori:

Does the document-pool constraint become more or less effective as sparsity increases? If a model has 512 experts with only 4 active per token (sparsity ratio 128:1), the pool constraint forces all tokens in a document to select from a shared subset — but the pool size d must still accommodate k = 4 active experts per token while being small enough to enforce domain coherence. The tension between specialization and flexibility may shift at extreme sparsity ratios. The paper's d ~ U{k, ..., nr} sampling produces pool sizes ranging from 8 to 128; at 512 experts with k = 4, the range would be 4 to 512, and the uniform distribution would sample pool sizes near 512 much more frequently (since the range is larger), potentially weakening the modularity constraint.
Does modularity remain compatible with full-model performance at larger scales? Table 1 shows EMO matches standard MoE performance at 1B active / 14B total, with the largest gap being 1.9 points on GSM8K (12.0 vs. 13.9). This gap could widen, narrow, or remain constant at larger scales. If the gap widens, the modularity-performance tradeoff becomes less favorable; if it narrows or reverses, EMO becomes strictly preferable.
Do expert subsets continue to outperform memory-matched baselines at larger scales? The paper's "train large, deploy small" advantage (Figure 1, right) is shown only at 130B tokens with small models. At larger scales, the memory-matched baselines (e.g., a 64-expert standard MoE vs. a 64-expert subset from a 512-expert EMO) would themselves be substantial models trained on substantial data, and the relative advantage might shrink.

What evidence exists in the paper. No scaling experiments are provided. The paper does not train models at multiple scales (e.g., 64, 128, 256, 512 experts at fixed active parameters), nor does it vary the sparsity ratio (k/n). All results — both full-model evaluation (Table 1) and selective expert use (Figure 3) — are from the single 128-expert architecture. The paper's discussion of large-scale deployment (Section 6) is entirely forward-looking and cites DeepSeek-V3 [2] and Kimi K2 [50] as examples of models that could benefit, but provides no EMO results at those scales.

Mitigation status. The paper does not attempt to address this limitation. The scaling behavior of the document-pool constraint is flagged implicitly as future work (the "Future Directions" section discusses "Modular Development and Maintenance" but not scaling studies). This is a significant gap for a paper whose primary motivation is improving the deployability of "large, sparse models." The 1B-active scale is representative of research-scale models but is an order of magnitude smaller than production MoEs (DeepSeek-V3 has ~37B active, ~671B total; Mixtral 8×22B has ~39B active, ~141B total). Until EMO is validated at larger scales, the claim that it is "particularly well-suited for large, highly sparse models" remains speculative.

The "Other" Category Limitation Reveals a Fundamental Constraint on Modularity

The assumption or constraint. EMO's modularity relies on the assumption that downstream tasks correspond to coherent domains that align with the document-level groupings that emerge during pretraining. When tasks are general or mixed-domain — specifically, the "other" catch-all categories in MMLU and MMLU-Pro — this assumption breaks down. The paper acknowledges this explicitly in Appendix B.3:

"When the deployment task is general (e.g MMLU other and MMLU Pro categories, which serve as a 'catch-all' for MMLU subjects), EMO expert subsets of size 32 and 8 experts struggle to match the Reg MoE @ 32 and Dense @8 baseline models trained from scratch."

The paper further states: "We view this phenomenon as a property of modular models, and believe it provides concrete evidence that EMO works in selective expert use because it has groups of experts that have localize capabilities."

The consequence. This is not a minor edge case — it reveals a fundamental tradeoff inherent to modular architectures. EMO works by concentrating domain-relevant capabilities into specific expert groups. This concentration is what enables selective deployment (extract the math group for math tasks) but it also means that tasks requiring broad, cross-domain capabilities suffer because no single small subset contains the full diversity of knowledge. The paper's phrasing — "a property of modular models" — is accurate but understates the practical implication: deploying EMO selectively requires knowing in advance that your task is domain-specific enough to benefit from subset extraction.

For many real-world applications, this distinction is blurry. A customer support chatbot may need to answer questions spanning product knowledge (domain-specific), general conversation (not domain-specific), and troubleshooting (mixing technical and commonsense reasoning). A coding assistant may need to generate code (domain-specific) while also explaining concepts in natural language (general). In these settings, the decision of when to use a domain-specific subset versus the full model becomes a deployment-time engineering challenge that the paper does not address.

The paper excludes the "other" category from aggregate MMLU and MMLU-Pro results (Section 4.2, footnote 1: "Aggregated results exclude the 'other' category; see §C and B.3 for details"). While methodologically justified (a catch-all is not a coherent domain for expert selection), this means the headline numbers overstate EMO's robustness on the full MMLU benchmark. The abstract and introduction do not mention this exclusion, which could mislead readers to expect that EMO's modularity benefits apply uniformly across all task types.

What evidence exists in the paper. Figure 13 (Appendix B.3) provides the quantitative evidence on the MMLU and MMLU-Pro "other" categories for models trained on 130B tokens. On MMLU other, EMO @32 experts achieves 28.9 accuracy (no fine-tune) vs. Reg. MoE @32 (trained) at 45.4 — a gap of 16.5 points. On MMLU-Pro other, EMO @32 experts achieves 10.5 vs. 18.4 for the memory-matched baseline. These gaps are substantially larger than the corresponding gaps on domain-specific categories (where EMO subsets outperform the baselines). The paper does not report "other" category results for the full 1T-token models, making this limitation less visible in the main results.

Mitigation status. The paper does not address this limitation beyond acknowledging it. There is no proposed method for detecting whether a given task is "domain-specific enough" for selective deployment, no mechanism for dynamically falling back to the full model when a subset is inadequate, and no exploration of composable subsets (combining multiple domain-specific expert groups for mixed-domain tasks). The "Future Directions" section mentions "Fine-grained Control" — selectively enabling or disabling expert clusters based on the application — but frames this as a benefit (excluding harmful clusters in child-facing applications) rather than addressing the fundamental limitation that general-purpose tasks may require the full model.

Training Data Requirements for Modularity Are Incompletely Characterized

The assumption or constraint. EMO's document-pool constraint relies on the assumption that "tokens within the same document usually come from the same domain" (Section 3). The paper treats this as a "weak supervisory signal" and uses document boundaries from the OLMoE pretraining corpus as the only grouping mechanism. This assumption's validity depends on properties of the pretraining data — specifically, the average semantic coherence of documents and the distribution of document lengths — that are not characterized or ablated.

The consequence. If the pretraining corpus contains many short documents (where the pool constraint has little effect because there are few tokens to average over), documents with mixed-domain content (where the "shared domain" assumption is violated), or documents where domain boundaries do not align with document boundaries (e.g., concatenated web pages, multi-topic threads), the signal provided by document boundaries may be too noisy to induce coherent expert groupings. The paper shows that EMO works well on the OLMoE corpus, but does not establish why or characterize the minimum data quality requirements for the approach to succeed.

This matters for practitioners who might want to apply EMO to different corpora — e.g., code repositories (where a single file may mix code, comments, and documentation in multiple languages), multilingual data (where code-switching within documents is common), or curated datasets with different document length distributions. Without understanding which corpus properties are necessary or sufficient for modularity to emerge, transferring EMO to new settings becomes a trial-and-error process.

Furthermore, the paper does not ablate the effect of document length or document coherence on modularity quality. Would EMO trained on a corpus of very short documents (e.g., tweets, search queries) still develop coherent expert groupings? Would a corpus of artificially concatenated documents (violating the "shared domain" assumption) break modularity? These questions are not addressed.

What evidence exists in the paper. The paper provides qualitative evidence that the OLMoE corpus has sufficient document-level coherence: the clustering analysis (Figure 5) shows that EMO clusters align with semantic domains and that "tokens within the same document are largely grouped together" in EMO clusters. The domain similarity analysis (Figure 6) uses WebOrganizer's 24 human-labeled domains and shows EMO produces differentiated activation patterns. However, this only establishes that the approach works for this specific corpus — it does not characterize the corpus properties that enable this success.

The paper includes no ablation where document boundaries are artificially degraded (e.g., by shuffling sentence order within documents, merging unrelated documents, or splitting documents at sentence boundaries) to test sensitivity to document coherence. The annealing experiment (Appendix B.4, Table 3) shows that applying the document-pool constraint only during annealing partially induces modularity, but this is a training regime ablation, not a data quality ablation.

Mitigation status. The paper does not address this limitation. The choice of the OLMoE pretraining corpus is stated without justification for why it is appropriate beyond being a standard open-source MoE training corpus. The paper does not propose methods for assessing whether a given corpus is suitable for EMO training, nor does it explore data preprocessing strategies (e.g., filtering for document length, splitting concatenated documents) that might improve modularity. The "Future Directions" section does not mention data quality or corpus properties as a research direction.

The Full-Model Performance Gap on Generative Reasoning Tasks Warrants Investigation

The assumption or constraint. While EMO matches standard MoE performance on most benchmarks (Table 1), there is a consistent performance gap on generative reasoning tasks, particularly GSM8K: EMO achieves 12.0 vs. Reg. MoE's 13.9 on GSM8K — a gap of 1.9 absolute points or approximately 14% relative. This gap is larger than the gaps on knowledge-focused benchmarks (MMLU: +0.4, MMLU-Pro: −0.8, MC9: −0.8). The paper discusses this in the full-model evaluation (Section 5.1) but does not analyze whether the gap represents a systematic weakness of the document-pool constraint for reasoning tasks or a statistical fluctuation.

The consequence. If the GSM8K gap is systematic, it suggests that the document-pool constraint may impose a tradeoff between modularity and certain types of reasoning capability. Math reasoning (as in GSM8K) requires multi-step logical inference where tokens from different "domains" (mathematical notation, natural language, numerical computation) must interact flexibly. The document-pool constraint forces tokens within the same problem to route through a shared expert pool, which might limit the model's ability to dynamically reallocate computation across qualitatively different reasoning steps — even though the full document is "math," a single GSM8K problem may require the model to parse a word problem (language understanding), extract numerical relationships (symbolic reasoning), perform arithmetic (computation), and format an answer (language generation). These sub-tasks might benefit from different expert specializations, but the pool constraint restricts all tokens to the same subset.

This would be a concerning finding if it scales: larger EMO models might show widening gaps on reasoning-heavy benchmarks even as they match performance on knowledge-heavy ones, making EMO less suitable for applications where reasoning quality is paramount (code generation, mathematical problem-solving, logical inference).

However, the gap could also be a statistical artifact. At 130B tokens (Table 1, bottom), the GSM8K gap between EMO (4.2) and Reg. MoE (5.2) is 1.0 point — proportionally similar but smaller in absolute terms. The paper does not report confidence intervals, so it is unclear whether the 1.9-point gap at 1T tokens is significant or within normal training variance.

What evidence exists in the paper. Table 1 provides the primary evidence. Additional suggestive evidence: in the selective expert use experiments, EMO shows anomalous behavior on GSM8K — at 16 experts (12.5% retention) without fine-tuning, EMO actually outperforms the full model (12.2 vs. 12.0, Figure 3, top row), and the performance curve across subset sizes is non-monotonic (11.0 at 64 experts, 11.7 at 32, 12.2 at 16, 6.9 at 8). This non-monotonicity is not observed on MMLU or MMLU-Pro and suggests that GSM8K performance in EMO may be more sensitive to expert selection noise or that the relationship between subset size and reasoning capability is more complex.

The qualitative generation examples in Appendix B.5 show that even at 128 experts (full model), both EMO and the standard MoE sometimes fail on GSM8K problems. The examples demonstrate that EMO subsets produce coherent reasoning (albeit sometimes incorrect) while standard MoE subsets degenerate, but they do not shed light on why full-model EMO underperforms full-model standard MoE.

Mitigation status. The paper acknowledges the gap implicitly (it is visible in Table 1) but does not investigate its cause or propose mitigations. The "Future Directions" section does not mention reasoning-specific limitations. This is a missed opportunity: an analysis of whether the GSM8K gap is driven by specific types of errors (e.g., arithmetic mistakes vs. misunderstanding the problem statement) or specific routing patterns (e.g., whether EMO's router makes systematically different expert assignments on math tokens vs. the standard MoE) would clarify whether this is a fundamental tradeoff or a correctable weakness.

Expert Subset Selection Requires Domain-Labeled Validation Data, Limiting Zero-Shot Deployment

The assumption or constraint. EMO's expert selection assumes access to a validation set with example inputs for the target domain, from which routing probabilities are aggregated to rank experts. The paper demonstrates sample efficiency — "even a single few-shot example is sufficient" (Appendix B.2) — but this still requires (a) knowing the domain of the task in advance, (b) having at least one labeled example from that domain, and (c) running the full model to collect routing statistics. This is a weaker requirement than FlexOlmo [7] and BTX [6], which require domain labels on the entire pretraining corpus, but it is not zero-shot.

The consequence. For truly novel domains or tasks where no labeled examples exist, EMO provides no mechanism for expert selection. A user cannot simply describe their task in natural language ("I need to answer questions about 19th-century Russian literature") and have EMO identify the relevant experts — examples must be provided. This is a practical limitation for ad-hoc use, exploratory analysis, or rapidly changing task distributions.

Furthermore, the expert selection process assumes that the validation examples are representative of the test distribution. If the validation examples are drawn from a slightly different distribution than the test queries (e.g., the validation set contains easy math problems and the test set contains hard ones; the validation set contains factual questions and the test set contains reasoning questions), the selected expert subset may not be optimal. The paper does not test sensitivity to distribution shift between validation and test data.

This limitation is related to — but distinct from — the more general concern about modularity on mixed-domain tasks (Limitation 3 above). Even for a well-defined domain, the expert selection process requires labeled data that may not be available. In contrast, deploying a standard MoE or dense model requires no per-domain setup at all — you simply run the full model.

What evidence exists in the paper. Figure 12 (Appendix B.2) ablates the number of validation examples but not the quality or representativeness of those examples. The paper tests expert selection with 1, 5, 10, 100, and all available validation examples, showing that performance degrades only modestly as examples decrease. However, all validation examples in this ablation are drawn from the same distribution as the test set (the standard MMLU/MMLU-Pro splits), so representativeness is perfect by construction. The paper does not test scenarios where validation and test distributions differ — for example, selecting experts using MMLU math questions and evaluating on a harder math benchmark, or selecting experts using English-language examples and evaluating on translated equivalents.

The paper also does not compare against zero-shot expert selection baselines: for instance, using the task description alone (without examples) to predict which experts should be relevant, or using generic activation statistics from the pretraining corpus (rather than task-specific validation data) to rank experts.

Mitigation status. The paper does not address this limitation directly. The sample efficiency result (1 example suffices) mitigates the concern partially — the labeling burden is minimal — but does not eliminate the need for any labeled data or the sensitivity to distribution shift. The "Future Directions" section mentions "Fine-grained Control" and "Modular Development and Maintenance" but not zero-shot expert selection or robustness to distribution shift. An important practical improvement would be a method for selecting experts based solely on a textual task description (e.g., embedding the description and finding experts whose pretraining activation patterns are similar), which would enable truly zero-shot modular deployment.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around Mixture-of-Experts models from one where modularity is something to be recovered from already-trained models to one where it must be built in during pretraining as a first-class objective. Before EMO, the field's approach to selective expert deployment was almost entirely extractive: train a standard MoE with conventional per-token routing, then apply increasingly sophisticated pruning algorithms (Easy-EP, expert dropout, importance scoring) to identify task-relevant subsets. The implicit assumption was that expert specialization—which undeniably emerges in standard MoEs—would naturally support modular deployment if only we could find the right selection method.

EMO demonstrates that this assumption is fundamentally wrong for domain-level modularity. The paper's central empirical finding is not just that EMO works better than standard MoEs under subset restriction (which would be an incremental improvement), but that no post-hoc selection method, no matter how sophisticated, can extract usable domain-specific subsets from a standard MoE (Figure 4). The failure is architectural, not algorithmic: standard MoE training produces experts specialized along lexical axes (prepositions, punctuation, proper names) rather than semantic axes (math, code, biomedical), and no amount of clever selection can reassemble lexical experts into a domain-coherent group because the underlying capability was never localized in the first place. When Easy-EP—a state-of-the-art pruning method—fails to salvage standard MoE performance at small subset sizes, the implication is clear: the research program of post-hoc expert pruning for domain-specific deployment is optimizing a fundamentally limited approach.

This reframing has several downstream effects on what research directions become more or less attractive:

Less attractive: developing ever-more-sophisticated expert pruning algorithms for standard MoEs. If the bottleneck is not selection quality but the underlying routing structure, improving selection algorithms yields diminishing returns. The paper's evidence that EMO is largely insensitive to selection method (router-based and Easy-EP produce nearly identical results) reinforces this—when modularity is trained in, selection becomes trivially easy; when it isn't, selection is fundamentally limited regardless of sophistication.
More attractive: training objectives that induce structured expert usage. EMO's document-pool constraint is one instantiation of a broader principle: structural signals available in pretraining data (document boundaries, section breaks, source metadata) can substitute for explicit domain labels in guiding expert specialization. This opens a design space of training constraints that operate at different granularities—language-level, modality-level, task-level—each paired with an appropriate load balancing mechanism.
More attractive: investigating load balancing granularity as a first-class design dimension. The paper's diagnostic insight that micro-batch load balancing conflicts with structured routing (Section 3.3, Consideration 1; Figure 7) identifies a general principle: any training objective that encourages structured expert usage must be paired with a load balancing mechanism that operates at the same or larger granularity as the structure being enforced. This principle extends beyond EMO—future work on domain-adaptive training, multi-lingual MoEs, or modality-specific expert routing will need to match load balancing granularity to the intended structure.
Less attractive: treating expert specialization in MoEs as an unqualified good. Prior work celebrated that experts in large MoEs specialize to surface-level features as evidence that the architecture "works." EMO recasts this same behavior as a failure mode for deployability—lexical-level specialization is precisely what prevents domain-level expert isolation because every document in every domain contains the same surface-level features (punctuation, prepositions, common verbs). The paper's clustering analysis (Figure 5) makes this visually stark: standard MoE clusters are named "prepositions," "proper names," "copula verbs"; EMO clusters are named "health, medical & wellness," "U.S. politics & elections," "source code." These are qualitatively different kinds of specialization, and only the latter supports modular deployment.

The paper also provides a unified explanation for conflicting prior findings about expert specialization. Some work found specialization at surface-level patterns [16, 17], while other work found apparent specialization may reflect geometric properties of the representation space that are difficult to interpret [19]. EMO's analysis suggests both findings are correct under standard training—experts do specialize, but along axes that are not aligned with human-interpretable domains, making them appear simultaneously specialized (under statistical analysis) and non-specialized (under semantic analysis). The paper resolves this apparent contradiction by showing that different training objectives produce different kinds of specialization, and the kind that matters for practical modularity requires explicit incentives.

Finally, the paper introduces modularity as a training efficiency strategy, not just a deployment convenience. The finding that EMO expert subsets can outperform memory-matched models trained from scratch (Figure 1, right; Figure 11) suggests a counterintuitive principle: training a larger modular model and extracting subsets can be more compute-efficient than training smaller purpose-built models directly. This inverts the standard mental model where training budget should match deployment budget, and instead suggests that over-training with modularity constraints produces better small models than directly training small models. If this finding holds at larger scales (an open question—see Limitations), it would change how organizations allocate pretraining compute: rather than training a separate model for each deployment tier, train one large EMO model and extract appropriately sized subsets.

Follow-Up Research This Work Enables

Scaling laws for modularity: how does subset quality vary with total expert count, sparsity ratio, and training tokens? The paper demonstrates EMO at a single scale (128 experts, 8 active per token, 1T tokens), but the approach is motivated by making "large, highly sparse models" (Section 6) more deployable. A natural scaling study would train EMO at multiple expert counts (64, 128, 256, 512) while holding active parameters constant (~1B), measuring both full-model performance and subset performance at fixed retention percentages (e.g., 25%, 12.5%, 6.25%). The key question: does the document-pool constraint remain effective as sparsity increases, or does the uniform sampling of d ~ U{k, ..., nr} cause the constraint to weaken (since the range of d grows, and large d values dilute the modularity pressure)? A negative result—modularity degrading at high expert counts—would identify a fundamental limit to EMO-style training at the scales where it is most needed (e.g., DeepSeek-V3 with hundreds of experts). A variant experiment would fix the total expert count at 128 but vary k (the number of active experts per token), testing whether lower k (sparser routing) produces sharper expert specialization under the document-pool constraint.

Zero-shot expert selection: can relevant experts be identified from task descriptions without any forward passes through the full model? The current expert selection process requires running a validation set through the full 128-expert model, which creates an inherent circularity for memory-constrained deployment (you must load the full model to determine which subset to load). A concrete follow-up would train a lightweight "expert recommender" model that takes a textual task description (e.g., "multiple-choice questions about high school mathematics") and predicts which expert indices are relevant, trained on (task description, expert activation vector) pairs from EMO's own routing behavior across many MMLU/MMLU-Pro domains. The evaluation would measure: (a) whether zero-shot-selected subsets match the quality of validation-set-selected subsets, (b) the computational cost of the recommender relative to a full-model forward pass, and (c) generalization to held-out task descriptions not seen during recommender training. If successful, this would remove the primary practical obstacle to deploying EMO in truly memory-constrained settings (on-device, edge) where even a single full-model forward pass is infeasible.

Cross-domain expert composition: when a task requires capabilities spanning multiple domains, can expert subsets be combined? The paper demonstrates that domain-specific subsets work well in isolation, but acknowledges (Appendix B.3) that the "other" catch-all category in MMLU sees degraded subset performance because no single small subset contains the necessary breadth. A concrete follow-up would test compositional deployment: for a task blending math and code (e.g., generating code to solve a math problem), select the math-relevant experts and the code-relevant experts separately (using domain-specific validation data) and deploy their union as the active subset. The key measurements: (a) does the union's performance match full-model performance better than either subset alone? (b) How does performance scale with the size of the union—does a 16-expert union (8 math + 8 code) match or exceed a 16-expert single-domain subset? (c) Can the model handle negative composition—excluding specific expert clusters (e.g., those associated with "spam, adult, gambling & low-quality content" from Figure 5) to create safer deployments without degrading task performance? The paper's clustering analysis (Figure 5) provides a concrete list of semantically interpretable clusters that could be selectively enabled or disabled, making this experiment immediately actionable.

Data quality requirements for emergent modularity: what corpus properties are necessary for document-pool training to work? The paper assumes that "tokens within the same document usually come from the same domain" (Section 3) but does not characterize this assumption's validity or sensitivity. A controlled experiment would take the OLMoE corpus and systematically degrade document coherence at different levels: (a) shuffle sentences within documents (preserving per-sentence coherence but breaking document-level topic consistency), (b) concatenate unrelated documents (violating the shared-domain assumption while preserving within-document structure), (c) split documents at paragraph boundaries (reducing the number of tokens per document-pool constraint). Training EMO on each degraded corpus and measuring resulting modularity (via subset performance curves) would establish the minimum data quality requirements. A negative result—modularity persisting even under substantial degradation—would suggest the constraint is more robust than the paper's motivation implies; a positive result—modularity breaking at specific degradation thresholds—would provide practical guidance for corpus preparation when applying EMO to new data sources.

Layer-wise modularity: do all layers benefit equally from the document-pool constraint, or can the constraint be applied selectively? Figure 6 shows that domain-level expert activation similarity emerges progressively in deeper layers, with early layers showing limited domain structure. This suggests that the document-pool constraint may be more important in some layers than others. A concrete experiment would apply the document-pool constraint to only a subset of layers—e.g., only the first half, only the second half, only every other layer—and measure both full-model performance and subset quality. The hypothesis: applying the constraint only in deeper layers (where domain specialization naturally emerges) might preserve modularity while reducing the training-time constraint burden (since early layers, which handle more generic syntactic processing, could route freely). A negative result—modularity requiring the constraint at all layers—would suggest that domain-level specialization in deeper layers depends on constrained routing in earlier layers to establish consistent token representations before they reach domain-specific processing. A positive result—modularity preserved with sparse constraint application—would enable more flexible training recipes and potentially better full-model performance (since early layers could route more freely).

Annealing standard MoEs into modularity at larger scales and longer horizons. Table 3 shows that annealing a standard MoE on the document-pool objective for 50B tokens (5% of total training) partially induces modularity but underperforms training from scratch. The experiment raises the question: is the gap because the annealing duration was too short, or because routing patterns established during standard pretraining create path dependencies that cannot be fully reorganized? A concrete follow-up would take a standard MoE trained on 1T tokens and anneal it with the document-pool constraint for varying durations (50B, 100B, 200B, 500B tokens), measuring whether extended annealing closes the gap with from-scratch EMO training. If the gap closes with sufficient annealing, this would be a highly practical finding—standard MoEs could be made modular post-hoc without retraining from scratch, dramatically reducing the barrier to adoption. If the gap persists, it would provide fundamental insight into the path dependence of routing pattern formation: once experts have specialized along lexical axes, do they resist reorganization to semantic axes regardless of training duration? This question connects to broader issues in continual learning and representation plasticity.

Practical Applications and Downstream Use Cases

Domain-specific API endpoints with memory-proportional serving costs. A cloud LLM provider could train a single large EMO model (e.g., 128 experts, 14B total parameters) and serve it as multiple lightweight API endpoints, each loading only the domain-relevant expert subset. For a coding endpoint, load only the ~25–30 experts most relevant to code (roughly 25% retention), achieving near-full coding performance (based on MMLU Computer Science and GSM8K results) while using ~75% less GPU memory per endpoint. For a biomedical endpoint, load a different 25-expert subset targeting health, biology, and chemistry domains. The total memory across all endpoints could still be lower than serving the full model multiple times, since expert subsets for related domains overlap (Figure 6 shows high activation similarity for related domains). The concrete benefit: a cloud provider running the full 14B model on an 80GB GPU could instead serve 3–4 domain-specific endpoints on the same GPU, each using only ~3.5B parameters of loaded experts, enabling multi-tenant deployment without proportional memory scaling.

On-device deployment of capability-specific models extracted from a large cloud-trained parent. A mobile device manufacturer could train a large EMO model in the cloud (e.g., 512 experts, ~50B total parameters) and then extract lightweight subsets for on-device deployment: a 32-expert subset for code completion (~2B parameters), a 16-expert subset for text summarization (~1B parameters), and an 8-expert subset for basic conversational ability (~500M parameters). Each subset is extracted from the same parent model using a small number of domain-specific examples (Figure 12 shows as few as 1–5 examples suffice for expert selection), then quantized and shipped as part of an OS update. When the parent model is improved through continued pretraining or fine-tuning, only the relevant expert parameters need updating on-device, rather than replacing the entire model. This is a different model update paradigm—modular updates rather than monolithic replacements—that the paper explicitly envisions in Section 6 ("Modular Development and Maintenance").

Targeted capability removal for safer deployment. EMO's expert clusters align with interpretable semantic domains (Figure 5), including clusters the paper identifies as "spam, adult, gambling & low-quality content." For a child-facing application, these clusters could be permanently excluded at deployment time—simply don't load those experts—producing a model that retains general language capabilities while being structurally incapable of generating content from the excluded domains. This is qualitatively different from prompt-based safety filtering or output classifiers because the capability is removed at the parameter level: the model literally cannot generate text from an excluded domain because the necessary parameters are not loaded. The paper's evidence that domain-specific expert subsets retain near-full performance on their target domains (Figure 3) suggests that excluding specific clusters should not degrade performance on unrelated domains—the math experts don't need the adult-content experts to function. A concrete deployment would involve auditing expert clusters, determining which correspond to undesirable capabilities, and shipping a model checkpoint that excludes those expert indices. The paper does not evaluate this use case directly, but the clustering analysis (Figure 5) and the "Fine-grained Control" discussion (Section 6) suggest it is viable with existing EMO models.

Cost-efficient fine-tuning for domain adaptation. When adapting a large language model to a specific domain (e.g., legal document analysis, medical literature review), standard practice requires fine-tuning the full model, which is computationally expensive and risks catastrophic forgetting on general capabilities. With EMO, a practitioner could (a) select the domain-relevant expert subset using a small labeled validation set (5 examples suffice, per Figure 12), (b) fine-tune only those experts (freezing the rest), and (c) optionally reintegrate the fine-tuned experts into the full model. The paper's preliminary experiment on expert subset fine-tuning and reintegration (Section 6, "Modular Development and Maintenance") reports that "the resulting model improves over the original full model, though it does not yet match the performance of the standalone subset." This suggests the approach is viable but needs refinement. A concrete workflow: a legal tech company maintains one EMO model as their base, and each client (law firm) gets a customized subset fine-tuned on their specific document corpus. Updates to individual clients' models require fine-tuning only their subset, not the full model, reducing adaptation costs proportionally to the subset size (e.g., 4–8× cheaper for 12.5–25% expert retention).

When to Prefer This Method

The paper does not position EMO against named alternative deployment strategies (e.g., distillation, quantization, or standard MoE with expert offloading) with explicit tradeoff analyses. It demonstrates superiority over post-hoc expert pruning (Easy-EP, Figure 4) and memory-matched models trained from scratch (Figure 1, right), but does not provide decision rules for choosing between EMO and other deployment efficiency techniques. The primary comparison is against standard MoE training—EMO should be preferred when you need the ability to deploy domain-specific expert subsets, and the document-pool constraint's minimal impact on full-model performance (Table 1) means there is little downside to using it even when modularity is not the primary goal.