UniPool: A Globally Shared Expert Pool for Mixture-of-Experts

ArXiv: 2605.06665

🎯 Pitch

Even high-performing MoE models waste expert capacity: randomizing routers in deep layers drops accuracy by only ~1 point—hinting at cross-layer redundancy. UniPool treats expert capacity as a single global pool shared across all layers, matching or outperforming standard MoE with up to 58% fewer expert parameters, proving expert budgets can scale sublinearly with depth.

---

1. Executive Summary

This paper proposes UNIPOOL, a Mixture-of-Experts architecture that replaces per-layer private expert ownership with a single globally shared expert pool accessed by independent per-layer routers, combined with a pool-level auxiliary loss and NormRouter for stable training under sharing. Across five LLaMA-architecture model scales (182M–978M parameters) trained on 30B tokens from the Pile dataset, UNIPOOL consistently improves validation loss over matched vanilla MoE baselines by up to 0.0386, while reduced-pool variants achieve this with only 41.6%–66.7% of the vanilla expert-parameter budget—establishing that expert parameters need not grow linearly with depth and can instead scale sublinearly through cross-layer reuse. Routing-randomization probes further reveal that UNIPOOL's shared experts carry substantially more task-level information than layer-private deep experts, with single-layer randomization dropping average accuracy by 4.1 points on UNIPOOL versus only 1.3–1.5 points on vanilla MoE, confirming that global sharing converts depth-induced redundancy into reusable specialization rather than merely compressing already-similar parameters.

2. Context and Motivation

The Core Problem: Expert Parameters Are Tied Linearly to Depth, and That Is Wasteful

The fundamental architectural question this paper tackles is: why should every transformer layer own a private set of expert FFNs, and what happens if we break that ownership? In the standard MoE design—deployed across virtually all production MoE systems from Mixtral to DeepSeek-V3—each transformer layer maintains its own fixed set of $E$ expert feed-forward networks, and a layer-specific router selects a sparse subset of those private experts for each token. This design couples the number of layers $L$ to the total number of stored expert FFNs as $E \times L$, making expert parameters grow linearly with model depth.

This coupling is not a theoretical requirement of MoE routing; it is a convention that has been inherited from dense architectures and propagated through successive generations of MoE scaling without formal justification. The paper frames this as the expert-budget allocation rule—a rigid assignment that treats expert capacity as a per-layer resource rather than a global architectural budget. The consequence is architectural: adding layers necessarily allocates new private expert capacity, regardless of whether those new experts learn distinct transformations or merely replicate functions already present in other layers.

Why This Matters: The Waste Is Not Theoretical

The paper identifies three converging lines of evidence suggesting that per-layer expert ownership induces substantial redundancy, providing concrete numbers from recent analyses:

Within-layer parameter similarity. Huang et al. (2025) showed that same-layer expert weight matrices in Qwen and DeepSeek MoE models share a dominant subspace with pairwise cosine similarity above 0.9—meaning the expert weight matrices within a single layer are pointing in nearly the same direction. This is not a minor overlap; it suggests the experts are learning highly correlated transformations despite being architecturally separate.

Expert pruning as a diagnostic. Bai et al. (2025) demonstrated that pruning roughly half the experts in Mixtral 8×7B costs only ~8% relative quality loss, with the strongest intra-layer similarity concentrated in deep layers. If half the experts can be removed with minimal quality degradation, they cannot all be encoding non-redundant specialization—a substantial fraction of the expert parameter budget is functionally duplicative.

Re-routing to similar experts preserves accuracy. Wu et al. (2025) showed that tokens re-routed from their originally selected expert to the most-similar same-layer expert preserve accuracy with up to 2× decoding speedup on Qwen1.5-MoE, DeepSeek-V2-Lite, Qwen3-30B-A3B, and OLMoE. This is the strongest behavioral evidence: if selecting a different expert (that happens to be similar in weight space) produces functionally equivalent outputs, then the per-layer router is not making a sharp functional partition over its private expert set—it is selecting from a set of near-duplicates.

These prior findings document redundancy in expert parameters and outputs, but they treat it as a post-hoc compression target: the redundancy exists, and the goal is to prune or route around it after training. This paper asks a more fundamental question: why does the redundancy arise in the first place, and can it be prevented architecturally?

The Paper's Own Diagnostic: Routing Randomization as a Redundancy Probe

To directly test whether the per-layer router carries meaningful signal at depth, the authors introduce a simple but revealing experiment: replace the learned top-k router in a single deep-half MoE layer with uniform random assignment, and measure the downstream accuracy drop. The logic is straightforward: if deep-layer experts encode truly distinct specializations, then randomly choosing among them should hurt performance noticeably—the router's selection decision is load-bearing. If the experts are largely interchangeable, the drop should be small.

The results in Table 1 are striking. On three production MoE models (Qwen1.5-MoE, DeepSeek-V2-Lite, Qwen3-30B-A3B), randomizing routing in one deep-half layer drops average accuracy by only 1.0–1.6 points across five standard benchmarks (ARC-Easy, ARC-Challenge, PIQA, HellaSwag, WinoGrande). For context, this means that on a 30B-parameter class model (Qwen3-30B-A3B), the choice among same-layer private experts at a deep layer carries only about 1 point of task-level information on average. The router is not committing to a sharp functional partition; the experts within a layer are substitutes for each other in ways that matter very little for downstream behavior.

This routing observation aligns with and extends the prior parameter- and output-level redundancy evidence. The three signals—weight-space similarity above 0.9, pruning tolerance to ~8% quality loss, and routing randomization costs of 1.0–1.6 points—converge on the same conclusion: per-layer private expert ownership encourages every layer to independently rediscover similar transformations from a thin gradient signal, producing deep-layer redundancy that is then removed after training via pruning or similar-expert re-routing. The current approach addresses the symptom rather than the cause.

Why Prior Approaches Fall Short

The paper identifies several existing lines of work that touch on expert sharing or parameter efficiency, but argues that none addresses the structural problem:

Layer-private MoE scaling work (GShard, Switch Transformer, ST-MoE, Mixtral, DeepSeek series) retains per-layer expert ownership and focuses on improving routing stability, load balancing, or expert granularity within that paradigm. These works make the per-layer MoE work better, but they do not question whether the per-layer allocation itself is optimal. The paper acknowledges that these systems have pushed sparse MoE to impressive scales—"expert capacity is an effective way to scale language models"—but argues they share a blind spot: expert capacity is treated as a per-layer resource rather than a reusable global budget.

Parameter sharing across transformer layers has been explored in Universal Transformers (Dehghani et al., 2019) and ALBERT (Lan et al., 2020), but these models share all parameters across depth—attention weights, FFN weights, everything. UNIPOOL applies sharing selectively to only the MoE expert FFNs while retaining layer-specific attention blocks and routers. This targeted sharing respects an important structural fact about sparse MoE models: expert FFNs constitute a large fraction of stored parameters (often the majority), but routers at different depths can still benefit from learning distinct token-to-expert policies because the hidden-state context changes across the residual stream. Sharing everything destroys this layer-specific routing flexibility; sharing only the experts preserves it.

MoEUT (Csordás et al., 2024), the closest prior work, cyclically repeats a small group of shared transformer blocks across depth with per-layer entropy balancing. MoEUT shares entire transformer blocks (including attention and FFN experts together), while UNIPOOL shares only the FFN experts as a single global pool and leaves routers and attention per-layer. The paper's claim is that this more targeted sharing better matches the structure of sparse MoE models: routers should remain layer-specific because they need to route based on different hidden-state contexts, while the expert computations they route to can be shared.

Post-hoc compression methods (pruning, similar-expert re-routing) treat redundancy as something to be removed after training. The paper's position is that this is addressing the symptom: the redundancy arises because per-layer ownership forces each layer to independently discover similar functions from limited gradient signal. By making experts globally shared, the architectural constraint that causes the redundancy is removed—each expert now accumulates gradient signal from $L$ layers rather than one, and depth-induced duplication is converted into architectural reuse during training, not removed afterward.

How This Paper Positions Itself

The paper frames its contribution as an architectural intervention rather than a new training technique or a post-hoc compression method. The core claim is that per-layer expert ownership is a convention, not a requirement, and that replacing it with a global shared pool—combined with the right stabilizing mechanisms (pool-level balancing, NormRouter)—consistently improves both efficiency and quality across scales.

The paper also positions itself within the broader MoE scaling literature as contributing a new scaling hyperparameter: pool size. Under the shared-pool design, expert parameters need not grow as $E \times L$; they can grow sublinearly with depth. The reduced-pool experiments in Figure 2(a) directly test this: UNIPOOL variants using only 41.6%–66.7% of the vanilla expert-parameter budget match or outperform the layer-wise baselines. This reframes expert capacity allocation as a budget optimization problem: given a fixed total parameter or compute budget, should it be spent on more layers (depth), more experts per shared pool (breadth), or some combination? The paper's empirical answer—backed by the 830M scale showing both the largest loss reduction and the smallest winning expert fraction (41.6% of baseline)—is that allocating capacity toward depth and reusable expert pools can be more effective than allocating it primarily to width or to private per-layer capacity.

A key part of this positioning is the paper's explicit acknowledgment that this is not a simple drop-in replacement. Two non-trivial challenges must be solved for shared-pool MoE to work at all:

What is the right load-balancing objective when expert ownership becomes global? In layer-private MoE, the Switch auxiliary loss is applied independently at each layer to prevent dead experts within that layer. Under sharing, an expert unused by layer $l$ may be heavily used by other layers; treating it as dead within layer $l$ is misaligned with where parameters are actually allocated. The paper's pool-level auxiliary loss addresses this by aggregating token-to-expert assignments across all sharing layers and applying a single balancing objective over the global pool, preventing globally dead experts without forcing every layer to use every shared expert.

How to maintain stable routing into a much larger expert candidate set? Softmax-based routers are designed for layer-specific experts where the candidate set is small (typically 8 experts). In UNIPOOL, routers at different depths all select from the same pool of $M = 8L$ experts—a much larger candidate set—and layer-dependent hidden-state scale variations can translate into inconsistent routing sharpness across layers. The paper adopts NormRouter (Zheng et al., 2025), which replaces softmax gating with L2-normalize-then-ReLU scoring combined with a learnable scaling factor, explicitly because this formulation is well-matched to shared-pool routing: normalization makes scores less sensitive to layer-specific hidden-state scale, ReLU induces sparse competition over the large pool, and the learnable scale lets each router adjust routing strength adaptively.

The paper's ablation results (Table 5) demonstrate that these are not independent improvements: using a shared pool with the standard per-layer auxiliary loss actually worsens validation loss compared to vanilla MoE (1.9480 vs. 1.9317 at 182M) because the balancing objective is misaligned, and using NormRouter alone in a layer-private setting slightly degrades performance (1.9375 vs. 1.9317) because its benefits are tied to routing over a larger and sparser candidate set. The full UNIPOOL design—shared pool + pool-level auxiliary loss + NormRouter—is a co-designed system where each component addresses a specific failure mode that sharing introduces.

The Deeper Implication: A Structural Reframing of Expert Capacity

Beyond the immediate architectural contribution, the paper implies a more fundamental reframing: expert capacity should be treated as a reusable global budget, not a per-layer entitlement. This connects to a view of MoE routing where the set of expert transformations invoked across depth—the path a token takes through the shared pool—is the relevant object of study, rather than the per-layer expert index. Under this view, the vanilla MoE constraint that the $l$-th choice must come from layer $l$'s private expert set is an artificial restriction; relaxing it allows the same expert functions to be reused across depths where appropriate, and the total number of distinct expert tensors touched by a token becomes a descriptive statistic of the architecture's effective computation rather than being hard-coded to equal $L$.

This reframing is what makes the reduced-pool results in Figure 2(a) interpretable. They are not merely a parameter-efficiency finding (though they are that too); they are a direct test of the hypothesis that layer-private expert allocation over-provisions capacity. If the vanilla MoE organization contains functionally redundant experts, then a smaller globally shared pool should be able to match or surpass the larger per-layer allocation—which is exactly what the experiments show across four scales and with increasingly aggressive pool reduction at deeper architectures (66.7% at 182M down to 41.6% at 830M). The monotonic decrease in the smallest winning pool fraction with depth suggests that deeper backbones create more opportunities for cross-layer reuse, making the private allocation proportionally more wasteful.

3. Technical Approach

3.1 Reader Orientation

UNIPOOL is an architectural modification to Mixture-of-Experts transformers that replaces the conventional per-layer private expert sets with a single globally shared pool of expert feed-forward networks, while keeping layer-specific routers, and introduces two stabilization mechanisms—a pool-level auxiliary loss and NormRouter-based routing—to make training under sharing both stable and beneficial. The system solves the problem that per-layer expert ownership ties total expert parameters linearly to model depth and encourages duplicated function learning across layers, and it does so by converting depth-induced redundancy into architectural reuse: every layer routes into the same expert pool, so each expert receives gradient signal from all layers and can specialize on transformations that are genuinely reusable across depth, while the total expert-parameter count decouples from the layer count and can be tuned as an independent scaling hyperparameter.

3.2 Big-Picture Architecture

UNIPOOL's design replaces $L$ independent expert sets (one per transformer layer) with one shared expert pool of size $M$, while preserving per-layer routers and per-layer attention blocks. The architecture has five major components:

• Shared Expert Pool — a single collection of $M$ expert feed-forward networks, stored once and referenced by every MoE layer, replacing the $L \times E$ private experts in vanilla MoE. This is the central architectural change: expert parameters are global, not per-layer.
• Per-Layer Routers — $L$ independent router modules (one per transformer layer) that each select top-$k$ experts from the shared pool for the tokens at that layer. Routers remain layer-specific because different depths in the residual stream require different routing patterns, even though the underlying expert computations are shared.
• NormRouter Scoring — a routing mechanism that replaces standard softmax gating with L2-normalize-then-ReLU scoring plus a learnable scalar, providing scale-stable, naturally sparse routing into the large shared candidate pool.
• Pool-Level Auxiliary Loss — a load-balancing objective that aggregates token-to-expert utilization statistics across all layers and applies a single balancing pressure over the global pool, preventing globally dead experts without forcing every layer to use every shared expert.
• Per-Layer Attention Blocks — the self-attention components of each transformer layer, which remain per-layer and are not shared, preserving depth-specific context aggregation.

Information flows as follows: a token's hidden state enters layer $l$ → the per-layer attention block produces a context-aware representation → the per-layer NormRouter computes sparse routing scores over all $M$ shared experts → top-$k$ experts are selected → the token is dispatched to those experts in the shared pool → the expert outputs are combined with the router's gating weights → the result is added to the residual stream and passed to layer $l+1$. The pool-level auxiliary loss is computed after the full forward pass by aggregating token-dispatch statistics from all layers and applied as a single global regularization term.

3.3 Roadmap for the Deep Dive

• First, the shared expert pool architecture—the core architectural modification that replaces per-layer private expert sets with a single global pool accessible by all layers through independent routers, and how this changes the FFN computation at each layer.
• Second, the pool-level auxiliary loss—why the standard per-layer Switch auxiliary loss becomes misaligned under sharing, the mathematical derivation of the pool-level objective, and the one-step-behind implementation that avoids cross-layer tensor dependencies.
• Third, NormRouter—the routing mechanism adopted for shared-pool routing, including the mathematical formulation (L2-normalize-then-ReLU scoring with a learnable scale), the Monte Carlo initialization procedure for the calibration constant, and why these properties are specifically matched to the challenges of routing into a large shared candidate pool.
• Fourth, the co-design logic—why the shared pool, pool-level auxiliary loss, and NormRouter must be combined as a system, supported by ablation evidence showing that partial combinations (shared pool with per-layer aux loss, NormRouter alone in layer-private MoE) degrade performance.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an architectural design paper whose core idea is that MoE expert capacity should be treated as a global reusable budget rather than as per-layer private resources, and that this reframing—when accompanied by the right stabilization mechanisms—consistently improves both quality and parameter efficiency across model scales. The paper contributes a concrete architecture (UNIPOOL) that implements this reframing and provides empirical evidence across five scales that the shared-pool design works and that expert parameters can grow sublinearly with depth.

---

The Shared Expert Pool Architecture

In a standard MoE transformer with $L$ layers and $E$ experts per layer, each layer $l$ maintains its own disjoint set of expert feed-forward networks $\{e_{l,1}, e_{l,2}, \ldots, e_{l,E}\}$ and a layer-specific router $r_l$. The feed-forward output at layer $l$ for a token with hidden state $x$ is computed as:

$\text{FFN}_l(x) = \sum_{i \in \text{Top-k}(r_l(x))} g_{l,i}(x) \cdot e_{l,i}(x)$

where $\text{Top-k}(r_l(x))$ returns the indices of the $k$ experts with the highest router scores for token $x$ at layer $l$, $g_{l,i}(x)$ is the gating weight assigned by router $r_l$ to expert $i$ for token $x$ (typically the softmax probability or NormRouter score normalized over the selected set), and $e_{l,i}(x)$ is the output of layer $l$'s $i$-th private expert FFN when applied to $x$.

What it computes: for each token at each layer, select a sparse subset of that layer's private experts based on the router's scoring, compute each selected expert's FFN output on the token, weight those outputs by the router's gating weights, and sum them. The output is a single vector that replaces the dense FFN output and is added to the residual stream.

Why this form: the per-layer private ownership means expert $e_{l,i}$ is only ever accessed by layer $l$. Each expert receives gradient signal from only the tokens that pass through its owning layer, which constitutes a fraction $1/L$ of the total gradient budget that would be available if the expert were shared across all layers. This is the structural feature that the paper argues causes deep-layer experts to converge to similar transformations—each expert sees only a thin slice of the total gradient signal, so multiple experts independently learn to represent the same useful functions rather than specializing.

In UNIPOOL, this per-layer ownership is eliminated. A single global shared pool $\mathcal{E} = \{e_1, e_2, \ldots, e_M\}$ of $M$ expert FFNs is instantiated once. Each layer $l$ retains its own router $r_l$, but the router now selects from the shared pool rather than from a layer-private set:

$\text{FFN}_l(x) = \sum_{i \in \text{Top-k}(r_l(x))} g_{l,i}(x) \cdot e_i(x)$

where the critical difference from the vanilla formulation is that $e_i$ in this equation is the same parameter tensor regardless of which layer $l$ invokes it. If layer 3 and layer 7 both select expert $e_5$, they both compute $e_5(x)$ using the exact same weights; the expert's parameters are shared by reference, not copied.

What it computes: the same operation as vanilla MoE—sparse expert selection, weighted combination—but the expert implementations invoked are drawn from a single global collection rather than from layer-specific collections. The per-token active computation (number of expert FFN evaluations) remains $k \times L$, matched to vanilla MoE when $M = E \times L$ and $k = 1$, because each layer still selects and evaluates $k$ experts; the change is in which expert parameters those evaluations use.

Why this form: the architectural motivation is to convert depth-induced redundancy into architectural reuse. In vanilla MoE, each expert receives gradient signal from one layer only. In UNIPOOL, each expert receives gradient signal from every layer that routes tokens to it, which can be up to $L$ layers. This increases the effective gradient budget per expert by up to a factor of $L$, while simultaneously reducing the total number of distinct expert parameter tensors from $L \times E$ to $M$. The hypothesis—tested empirically in Section 6.2 and the reduced-pool experiments—is that this concentrated gradient signal produces more specialized, more reusable expert functions, and that the total number of experts needed can therefore be smaller than $L \times E$.

The pool size $M$ as a configuration choice. The pool size $M$ is not forced to equal $L \times E$ (the matched vanilla budget). In the main experiments (Table 2), $M$ is set to $8L$—matching the total expert FFN count of vanilla MoE with 8 experts per layer—so that the comparison isolates the effect of sharing rather than changing total parameters. In the reduced-pool experiments (Figure 2a), $M$ is deliberately set to a fraction of $8L$ (e.g., $M = 64$ or $48$ at $L = 12$ where the matched pool size is 96), keeping top-1 routing so active parameters and per-token FLOPs remain matched to the vanilla baseline while stored expert parameters decrease. The architectural parameters are: number of layers $L$, hidden size $H$, pool size $M$, active experts per token per layer $k$. The total number of stored expert FFN parameters is proportional to $M \times H^2$ (ignoring constant factors from the SwiGLU structure), compared to $L \times E \times H^2$ for vanilla MoE.

Routers remain per-layer. The paper is explicit that routers $r_l$ are not shared across layers. The justification is that different depths in the residual stream carry different representational content—early layers process local syntactic patterns, deeper layers process more abstract semantic features—so the decision of which expert to invoke should depend on depth-specific context. Sharing only the expert implementations (which are stateless function approximators) while keeping routers per-layer (which are context-sensitive selectors) respects this structural asymmetry. This is a key difference from MoEUT, which shares entire transformer blocks including routers, and from Universal Transformers, which share all parameters across depth.

Expert parallelism and implementation. The paper implements UNIPOOL in Megatron-LM by instantiating the expert pool once and reusing the same experts module across all MoE layers via reference sharing. All models use expert parallelism of 1 (no expert-parallel distribution) and grouped GEMM for efficient expert computation. This means the shared pool is replicated across devices in data-parallel training but is architecturally one logical pool; distributing experts across devices with expert parallelism is noted as future work.

---

The Pool-Level Auxiliary Loss

The standard Switch Transformer auxiliary loss (Fedus et al., 2022) for a single layer $l$ with $E$ private experts is defined as:

$\mathcal{L}^{(l)}_{\text{aux}} = \alpha \cdot E \cdot \sum_{i=1}^{E} f^{(l)}_i \cdot P^{(l)}_i$

where $\alpha$ is a hyperparameter controlling the loss weight (set to $1 \times 10^{-2}$ in the vanilla MoE baselines), $f^{(l)}_i$ is the fraction of tokens in the micro-batch at layer $l$ that are dispatched to expert $i$ (an empirical count divided by total tokens), and $P^{(l)}_i$ is the mean routing probability assigned to expert $i$ by router $r_l$ at layer $l$ (the average of the softmax or scoring function output for expert $i$ over all tokens in the micro-batch at that layer).

What it computes: for a single layer, multiply the observed token fraction per expert by the mean routing probability per expert, sum over all $E$ experts, and scale by $\alpha \cdot E$. The product $f^{(l)}_i \cdot P^{(l)}_i$ is large when an expert receives many tokens and has high routing probability—indicating concentration of traffic—or when an expert receives few tokens but not because of low routing probability, which would indicate the router is ignoring it despite high scores.

Why this form: the product structure penalizes both over-utilization (high $f$, high $P$) and under-utilization with non-trivial routing probability (low $f$, non-trivial $P$). The scaling by $E$ normalizes so that uniform routing ($f_i = 1/E, P_i = 1/E$ for all $i$) yields $\mathcal{L}_{\text{aux}} = \alpha$. This loss is applied independently at each MoE layer, meaning the notion of a "dead expert" is defined per layer: expert $i$ at layer $l$ is dead if $f^{(l)}_i \approx 0$ for that specific layer, regardless of what happens at other layers.

Under UNIPOOL's shared expert pool, this per-layer definition of deadness becomes misaligned with where parameters are actually allocated. An expert $e_j$ in the shared pool may receive zero tokens from layer $l$ but receive many tokens from layers $l+1, l+2, \ldots$; treating it as dead within layer $l$ and penalizing the router for not using it at that layer conflicts with the architectural goal of allowing different layers to specialize on different subsets of the shared pool. The per-layer auxiliary loss would force every layer to spread traffic over the entire pool, preventing the cross-layer specialization that sharing is designed to enable.

The pool-level loss derivation. The paper defines the global average token fraction $\bar{f}_i$ across all $L$ sharing layers as:

$\bar{f}_i = \frac{1}{L} \sum_{l=1}^{L} f^{(l)}_i$

where $f^{(l)}_i$ is the same per-layer token fraction as before (fraction of tokens at layer $l$ dispatched to shared expert $i$). The key property: $\bar{f}_i$ does not depend on $l$—it is a single scalar per expert computed by averaging over all layers. The global average routing probability $\bar{P}_i$ is defined analogously:

$\bar{P}_i = \frac{1}{L} \sum_{l=1}^{L} P^{(l)}_i$

The pool-level auxiliary loss is then:

$\mathcal{L}_{\text{pool}} = \alpha_{\text{pool}} \cdot M \cdot \sum_{i=1}^{M} \bar{f}_i \cdot \bar{P}_i$

where $\alpha_{\text{pool}}$ is the pool-level loss coefficient (set to $1 \times 10^{-2}$ for the 182M scale and $2 \times 10^{-2}$ for the 469M, 650M, 830M, and 978M scales), and $M$ is the number of experts in the shared pool.

What it computes: for the entire shared pool, multiply the global average token fraction per expert (across all layers) by the global average routing probability per expert, sum over all $M$ experts, and scale by $\alpha_{\text{pool}} \cdot M$. Unlike the per-layer loss, a shared expert is only considered under-utilized if it receives few tokens globally—across all layers combined. An expert that is heavily used by layers 1–4 but ignored by layers 5–8 will have a healthy $\bar{f}_i$ and will not be penalized, even though individual layers may route zero tokens to it.

Why this form: this precisely aligns the load-balancing objective with the architectural ownership: expert parameters are owned globally, so the dead-expert criterion should be global. The scaling by $M$ (instead of by the per-layer expert count $E$) normalizes against the pool size. The coefficient $\alpha_{\text{pool}}$ is tuned to be larger than the per-layer $\alpha$ at higher scales ($2 \times 10^{-2}$ vs. $1 \times 10^{-2}$ for vanilla MoE's per-layer aux loss at 469M+), reflecting that maintaining balanced utilization over a larger shared pool requires stronger regularization.

Decomposition into per-layer computation. The paper provides a critical algebraic manipulation showing that the pool loss decomposes into a sum of per-layer terms that can be computed independently during the forward pass, avoiding cross-layer tensor dependencies that would break activation checkpointing or require full-graph materialization:

$\mathcal{L}_{\text{pool}} = \alpha_{\text{pool}} \cdot M \cdot \sum_{i=1}^{M} \bar{f}_i \cdot \bar{P}_i$

$= \alpha_{\text{pool}} \cdot M \cdot \sum_{i=1}^{M} \bar{f}_i \cdot \left( \frac{1}{L} \sum_{l=1}^{L} P^{(l)}_i \right)$

$= \frac{1}{L} \sum_{l=1}^{L} \left( \alpha_{\text{pool}} \cdot M \cdot \sum_{i=1}^{M} \bar{f}_i \cdot P^{(l)}_i \right)$

The critical step: $\bar{f}_i$ can be pulled out of the inner sum over $l$ because it does not depend on the layer index—it is a global average computed from the token fractions of all layers. Each summand is now a per-layer computation: it depends on that layer's routing probabilities $P^{(l)}_i$ and the globally averaged token fractions $\bar{f}_i$, but does not require routing probabilities from other layers.

What this decomposition means operationally: each layer can compute its own contribution to the pool loss using only its local routing probabilities $P^{(l)}_i$ and the global token-fraction statistics $\bar{f}_i$, which are computed once per micro-batch from the previous micro-batch's dispatch data (the "one-step-behind" scheme). The total pool loss is the mean of these per-layer contributions.

One-step-behind implementation. Computing $\bar{f}_i$ requires knowing the token dispatch statistics from all $L$ layers, which are only available after the full forward pass completes. To avoid cross-layer tensor dependencies that would require waiting for all layers to finish before computing any loss (breaking the layer-by-layer forward computation graph needed for activation checkpointing), the paper uses a one-step-behind scheme:

1. During micro-batch $t$, each layer dispatches tokens to experts as usual and records its per-layer token fractions $f^{(l)}_i$.
2. After all layers complete their forward pass, the global token fractions $\bar{f}_i$ are updated by averaging the $f^{(l)}_i$ from micro-batch $t$ (without gradient tracking).
3. During micro-batch $t+1$, each layer computes its pool loss contribution using the $\bar{f}_i$ from micro-batch $t$ and its current routing probabilities $P^{(l)}_i$ (which do carry gradients).
4. The total pool loss is the mean of per-layer contributions, and gradients flow only through the $P^{(l)}_i$ terms back to the router parameters.

Why one-step-behind works: the token-dispatch distribution $\bar{f}_i$ changes slowly during training (it is a moving average of batch-level dispatch patterns), so using the previous micro-batch's statistics introduces negligible bias while completely avoiding cross-layer synchronization in the computation graph. The router parameters are the only parameters updated through this loss path (the $\bar{f}_i$ term carries no gradient), which is appropriate because load balancing is a routing problem rather than an expert-weight problem.

Why this design over alternatives: the paper's ablation in Table 5 demonstrates that a shared pool with the original per-layer auxiliary loss (computed independently at each layer using only that layer's $f^{(l)}_i$ and $P^{(l)}_i$) degrades validation loss compared to vanilla MoE (1.9480 vs. 1.9317 at 182M, a +0.0163 increase). The heatmap in Figure 3a shows why: with per-layer balancing, aggregate traffic collapses onto a small subset of shared experts (the "global expert collapse" phenomenon), because each layer independently spreads its traffic over the full pool, but since the pool is now $M = 8L$ experts rather than $E = 8$, the balancing pressure per expert becomes diluted by a factor of $L$, and the router converges to using only a fraction of the pool. The pool-level loss (Figure 3b) restores balanced global usage by operating at the granularity where parameters are actually allocated—the pool—while allowing layer-specific routing patterns (visible in the per-layer heatmap as different layers using different subsets of the pool).

---

NormRouter for Shared-Pool Routing

Standard MoE routers compute gating weights via softmax over logits $z = W h$, where $W \in \mathbb{R}^{E \times d}$ is the router's weight matrix (one row per expert, each row a $d$-dimensional embedding), $h \in \mathbb{R}^d$ is the token's hidden state at that layer, and $E$ is the number of candidates (8 per layer in vanilla MoE). The gating weight for expert $i$ is:

$g_i = \frac{\exp(z_i)}{\sum_{j=1}^{E} \exp(z_j)}$

and the top-$k$ experts are selected by taking the $k$ largest $g_i$ values.

In UNIPOOL, routers at different depths all select from the same pool of $M = 8L$ experts (e.g., $M = 96$ at $L = 12$), which is a much larger candidate set than the 8 experts per layer in vanilla MoE. The paper identifies two problems with softmax routing in this setting:

1. Layer-dependent scale sensitivity. The magnitude of the hidden state $||h||_2$ varies across layers in the residual stream (it tends to grow with depth as information accumulates). Since softmax logits $z = Wh$ scale linearly with $||h||_2$, deeper layers produce logits with larger magnitudes, which translates into sharper (lower-entropy) softmax distributions. This means routing sharpness is coupled to layer depth in a way that is not controllable—early layers route diffusely, deep layers route sharply—which is undesirable when all layers compete for the same expert pool.

2. No natural sparsity over large candidate sets. Softmax produces a dense probability distribution over all $M$ experts for every token, even though only $k$ experts will be selected ($k = 1$ in top-1 routing). This dense computation of logits and probabilities over 96–384 experts per token per layer adds non-trivial overhead and provides no inductive bias toward sparse expert usage.

NormRouter formulation. The paper adopts NormRouter (KERN) from Zheng et al. (2025), which replaces softmax gating with an L2-normalize-then-ReLU scoring function. The score for expert $i$ given token hidden state $h$ is computed as:

$s_i = \sigma \cdot c \cdot \max\left(0, \frac{z_i}{||z||_2 + \epsilon}\right)$

where $\sigma$ is a learnable scalar parameter (initialized to 1.0), $c$ is a fixed constant determined by Monte Carlo estimation at initialization time, $z = W h$ is the vector of raw logits (one per expert in the shared pool, so $z \in \mathbb{R}^M$), $||z||_2$ is the L2 norm of the logit vector, $\epsilon$ is a small constant for numerical stability (preventing division by zero), and $\max(0, \cdot)$ is the ReLU activation that zeros out negative values.

What it computes, step by step:

1. Compute raw logits $z = Wh$ as in standard routing, projecting the $d$-dimensional hidden state to an $M$-dimensional score vector via the router weight matrix.
2. L2-normalize the logit vector: divide each $z_i$ by $||z||_2 + \epsilon$. This produces a unit-norm direction vector where each component is in $[-1, 1]$, and the relative magnitudes of components reflect directional alignment rather than absolute scale.
3. Apply ReLU: set all negative components to zero. Since the normalized vector has both positive and negative components (roughly half, for random initialization), this naturally produces sparse scores—approximately $M/2$ experts receive zero score for any given token.
4. Multiply by the fixed calibration constant $c$ and the learnable scale $\sigma$. The constant $c$ ensures that the selected top-$k$ scores have approximately unit magnitude at initialization (so that gradient scales are well-behaved); $\sigma$ lets each router learn the appropriate routing sharpness during training.
5. After computing scores $s_i$ for all $M$ experts, select the top-$k$ experts by taking the $k$ largest $s_i$ values. The gating weights $g_i$ used in the FFN combination (Equation 2) are the selected scores (or a normalized version over the selected set).

Why L2 normalization: it makes the routing scores depend only on the direction of the logit vector $z$, not its magnitude. Since $||h||_2$ varies across layers but the router weight matrix $W$ can learn to compensate, L2 normalization decouples routing sharpness from hidden-state scale. Two layers with different $||h||_2$ values but similar directional preferences will produce similar normalized score distributions, making routing behavior consistent across depth. This is important because in a shared pool, inconsistent routing sharpness across layers would cause some layers to dominate expert utilization while others are crowded out—a form of cross-layer interference that L2 normalization prevents.

Why ReLU: it induces natural sparsity without requiring an explicit sparsification mechanism (like top-$k$ thresholding or entropy regularization). For an $M$-dimensional normalized vector with roughly symmetric positive and negative components, ReLU zeros out approximately half the experts, reducing the effective candidate set and making the top-$k$ selection more decisive. This sparsity is well-suited to the shared-pool setting where $M$ is large (96–384) and the router should focus on a small subset of relevant experts rather than computing dense scores over the entire pool.

Why a learnable scale $\sigma$: even though L2 normalization fixes the score magnitudes to be roughly unit-scale (after the $c$ calibration), different layers may need different routing sharpness. Early layers might benefit from more diffuse routing (lower effective temperature, achieved by smaller $\sigma$) to explore different expert combinations, while deeper layers might benefit from sharper routing (larger $\sigma$) as the representation becomes more specialized. The learnable $\sigma$ adapts per layer during training, controlled by the gradient signal from the language modeling objective and the auxiliary loss.

Monte Carlo initialization of $c$. The constant $c$ is designed to ensure that at initialization (before any training), the NormRouter scores for the top-$k$ selected experts have approximately unit magnitude. This calibration stabilizes early training by preventing the router scores from being either vanishingly small (which would produce tiny gradient signals) or explodingly large (which would saturate the ReLU and produce degenerate sharp routing).

The constant is computed via Algorithm 1 in Appendix H. The procedure:

1. Sample $N = 10^5$ random vectors $x \sim \mathcal{N}(0, I_M)$ from an $M$-dimensional standard normal distribution. These simulate random logit vectors before the router weights have learned any structure.
2. For each sampled $x$, compute $y = \text{ReLU}(x / ||x||_2)$—the L2-normalized vector with negative components zeroed out.
3. Sort the components of $y$ in descending order, take the top-$k$ components $y_{(1)}, y_{(2)}, \ldots, y_{(k)}$, and compute their L2 norm $||y_{:k}||_2$.
4. Record the reciprocal $1 / ||y_{:k}||_2$ in a set $S$.
5. After all $N$ samples, set $c$ to the mean of $S$:

$c = \text{mean}\left(\left\{ \frac{1}{||\text{top-k}(\text{ReLU}(x/||x||_2))||_2} : x \sim \mathcal{N}(0, I_M) \right\}\right)$

What this computes empirically: the expected value of $1 / ||\text{top-k}(\text{ReLU}(x/||x||_2))||_2$ under standard normal logits. Multiplying by this $c$ after L2-normalization and ReLU ensures that $\mathbb{E}[c \cdot ||\text{top-k}(\text{ReLU}(z/||z||_2))||_2] \approx 1$ at initialization. This is a one-time computation performed before training begins, and $c$ is then fixed for the entire training run.

Why this initialization matters: without $c$, the random logits $z = Wh$ at initialization (with $W$ sampled from $\mathcal{N}(0, 0.01^2)$ and $h$ from the residual stream distribution) would produce ReLU-normalized scores whose top-$k$ magnitudes are arbitrary—potentially very small, leading to flat routing distributions and weak gradient signal to the router, or very large, leading to effectively one-hot routing from the start. The calibration constant anchors the initial routing scores to a controlled scale, giving the router a well-conditioned starting point from which the learnable $\sigma$ can then adjust sharpness.

Auxiliary loss compatibility. The NormRouter is fully compatible with both the standard per-layer auxiliary loss and UNIPOOL's pool-level auxiliary loss. The routing scores $s_i$ replace the softmax probabilities $P^{(l)}_i$ in the auxiliary loss computation. Specifically, in the pool loss decomposition:

$\mathcal{L}_{\text{pool}} = \frac{1}{L} \sum_{l=1}^{L} \left( \alpha_{\text{pool}} \cdot M \cdot \sum_{i=1}^{M} \bar{f}_i \cdot P^{(l)}_i \right)$

the $P^{(l)}_i$ term is now computed from NormRouter scores rather than softmax probabilities. For NormRouter, $P^{(l)}_i$ is defined as the mean of the raw scores $s_i$ (after $\sigma$ and $c$ scaling, before top-$k$ selection) over all tokens in the micro-batch at layer $l$, providing a continuous signal about how strongly the router favors each expert that can be balanced by the auxiliary loss.

Router weight initialization. The paper uses an initialization standard deviation of 0.01 for all parameters including router weights (Table 8), following standard LLaMA-style initialization. The NormRouter's calibration constant $c$ is computed once per configuration (depending on $M$ and $k$) at initialization time using Algorithm 1.

Why NormRouter over alternatives: the paper's ablation in Table 5 shows that replacing vanilla softmax routing with NormRouter alone in a layer-private MoE slightly degrades validation loss (1.9375 vs. 1.9317 at 182M), indicating that NormRouter is not a universally stronger router—its benefits are specific to routing into a large, shared candidate pool. The paper hypothesizes that NormRouter's scale-stable, sparse scoring is particularly well-matched to the shared-pool setting because: (1) L2 normalization prevents depth-dependent routing sharpness from interfering with cross-layer expert competition; (2) ReLU-induced sparsity naturally prunes the large candidate set, reducing interference from irrelevant experts; and (3) the learnable $\sigma$ lets each router adapt its effective temperature to the shared-pool dynamics during training.

---

Co-Design: Why All Three Components Must Be Combined

The paper's ablation study in Table 5 provides direct evidence that UNIPOOL's three components—shared pool, pool-level auxiliary loss, and NormRouter—must be combined as a system, and that partial combinations either degrade performance or fail to capture the full benefit. The ablation is conducted at the 182M scale ($L = 12$, $H = 768$, $M = 96$ for shared-pool configurations, $E = 8$ per layer for vanilla MoE) with all models trained on 30B Pile tokens under matched optimization settings.

Shared pool with per-layer auxiliary loss and softmax routing (1.9480 vs. vanilla MoE's 1.9317, +0.0163). This configuration takes the shared pool architecture but retains the standard per-layer Switch auxiliary loss (one independent balancing objective per layer) and standard softmax routing. The result is a degradation in validation loss compared to vanilla MoE. The mechanism behind this failure is visible in Figure 3a: the per-layer auxiliary loss forces each layer to spread its traffic over all $M = 96$ experts, but since the loss is applied independently at each layer with the same coefficient $\alpha$, the effective balancing pressure per expert is $1/M = 1/96$ compared to $1/E = 1/8$ in vanilla MoE—a factor of 12× weaker. The router responds by collapsing aggregate traffic onto a small subset of experts (the dark vertical bands in the heatmap and the highly non-uniform aggregate usage bar), because the auxiliary loss is too weak to prevent concentration. This "global expert collapse" means most of the shared pool goes unused, defeating the purpose of sharing.

Shared pool with pool-level auxiliary loss and softmax routing (1.9180 vs. vanilla MoE's 1.9317, −0.0137). Replacing the per-layer auxiliary loss with the pool-level loss while keeping softmax routing partially recovers the benefit of sharing. The pool-level loss correctly identifies globally dead experts and applies balancing pressure at the right granularity. However, softmax routing still suffers from layer-dependent scale sensitivity and dense score computation over the large pool, preventing the full efficiency gain that NormRouter enables.

Vanilla MoE with NormRouter (1.9375 vs. 1.9317, +0.0058). Replacing softmax with NormRouter in a layer-private MoE (no sharing, $E = 8$ per layer) slightly degrades validation loss. The paper interprets this as evidence that NormRouter's benefits are specific to routing over a larger and sparser candidate set. In the layer-private setting with only 8 candidates, softmax's dense probability computation is not a bottleneck, and the layer-dependent scale sensitivity is less problematic because each layer owns its private experts and there is no cross-layer competition for expert capacity.

Vanilla MoE with sigmoid gating and no auxiliary loss (1.9239 vs. 1.9317, −0.0078). This ablation tests whether simply removing load balancing (using sigmoid gating, which does not require auxiliary losses) could match UNIPOOL's performance. The improvement over standard vanilla MoE is modest (−0.0078), substantially smaller than UNIPOOL's full improvement (−0.0288), showing that loosening load balancing alone is not sufficient to capture the benefits of shared-pool design.

Full UNIPOOL (shared pool + pool-level auxiliary loss + NormRouter, 1.9029 vs. 1.9317, −0.0288). The combination of all three components yields the largest validation loss reduction, larger than the sum of the individual improvements from pool-level aux loss (−0.0137) and NormRouter (+0.0058, applied to vanilla MoE). This super-additive improvement is evidence of co-design: the pool-level loss provides the right utilization objective for the shared pool, and NormRouter provides the routing mechanism that can effectively distribute traffic across the large candidate set while respecting layer-specific preferences.

Intermediate sharing scope (G = 6, 4, 2; with pool-level aux loss and NormRouter per group). The sharing-scope ablation varies $G$, the number of expert-pool groups across depth. $G = 12$ recovers vanilla MoE (each layer is its own group of 8 private experts). $G = 1$ is full UNIPOOL (one global pool of 96 experts). $G = 6$ means the 12 layers are partitioned into 6 groups of 2 layers each, with each group sharing its own pool of $8 \times 2 = 16$ experts. $G = 4$ is 4 groups of 3 layers, each sharing 24 experts. $G = 2$ is 2 groups of 6 layers, each sharing 48 experts. Each group uses its own pool-level auxiliary loss and NormRouter, so sharing is confined within groups.

The results in Table 5 and the training trajectory in Appendix Figure 4d show a monotonic improvement as sharing scope broadens: $G = 6$ (1.9121, −0.0196), $G = 4$ (1.9099, −0.0218), $G = 2$ (1.9213, −0.0104), and $G = 1$ (1.9029, −0.0288). The $G = 2$ result is slightly worse than $G = 4$, which the paper does not explicitly explain but may reflect an unfavorable grouping boundary where layers with very different routing preferences are forced to share. The key finding is that broader sharing (smaller $G$) generally improves validation loss, supporting the hypothesis that cross-layer expert reuse is beneficial and that the optimal sharing scope is global ($G = 1$).

The aux-free vanilla baseline (1.9239). This configuration uses sigmoid gating without any auxiliary loss in a layer-private MoE, testing whether the entire auxiliary loss mechanism (per-layer or pool-level) is necessary or whether the router can learn balanced routing on its own. The result (−0.0078 vs. standard vanilla MoE) is a modest improvement, far from UNIPOOL's −0.0288, confirming that UNIPOOL's gains come from architectural sharing and the matched stabilization mechanisms rather than from simply removing or weakening load-balancing constraints.

What the co-design principle means architecturally: UNIPOOL is not a modular improvement that can be adopted piecemeal. The shared pool is the core architectural change that enables expert reuse and gradient concentration; the pool-level auxiliary loss is the necessary load-balancing objective that aligns with global parameter ownership; and NormRouter is the necessary routing mechanism that provides scale-stable, sparse access to the large shared candidate pool. Removing any one component breaks the system, either through global expert collapse (shared pool without pool-level aux loss), through depth-dependent routing instability (shared pool without NormRouter), or through misaligned balancing pressure (any component mismatch). The ablation results support this co-design claim empirically, and the paper's theoretical motivation—that sharing changes both the load-balancing domain and the routing scale—provides the mechanistic explanation.

4. Key Insights and Innovations

Innovation 1: Expert Capacity as a Global, Reusable Budget Rather Than a Per-Layer Entitlement

The most fundamental conceptual move in this paper is reframing what expert parameters are in the MoE architecture. The entire production MoE ecosystem—from GShard through Switch Transformer, Mixtral, and the DeepSeek series—treats expert FFNs as a per-layer resource: each layer gets its own private set, and adding depth means allocating new private expert capacity. This is not a theorem; it is an inherited convention from dense architectures where every layer's FFN is architecturally distinct. The paper argues that this convention is not merely suboptimal but structurally wasteful: it couples total expert parameters linearly to depth ($E \times L$), forces each expert to learn from only $1/L$ of the total gradient budget, and creates conditions where deep-layer experts converge to near-duplicate transformations that post-hoc pruning methods then remove.

What distinguishes this from prior parameter-sharing work (ALBERT, Universal Transformers, MoEUT) is targeted sharing guided by a diagnosis of where the redundancy lives. ALBERT and Universal Transformers share everything across depth—attention, FFN, the works—which destroys layer-specific routing flexibility and depth-dependent representational specialization. MoEUT shares entire transformer blocks cyclically, which is closer but still bundles routing with expert computation. UNIPOOL shares only the expert FFNs, keeping routers and attention per-layer. This targeted design emerges from the paper's diagnostic work (Section 3, Table 1): the redundancy is in the expert implementations (their weight similarities, their near-substitutability under random routing), not in the routing decisions (which experts to invoke at which depth). Sharing only the implementations preserves depth-specific routing while concentrating gradient signal on a smaller set of genuinely non-redundant expert functions.

This is a fundamental reframing, not an incremental parameter-efficiency trick. The paper is not saying "you can use fewer parameters and get similar quality"—though that is empirically true (Figure 2a: 41.6–66.7% of expert parameters suffice). It is saying that the unit of expert capacity allocation is wrong in the standard architecture. When expert capacity is global, the total number of experts becomes a configuration parameter decoupled from depth, and the question shifts from "how many experts per layer?" to "how large a pool does the model need, given $L$ layers that will reuse it?" The reduced-pool experiments in Figure 2(a) are not just an efficiency result; they are a direct test of the hypothesis that per-layer ownership over-provisions. The monotonic decrease in the smallest winning pool fraction with depth (66.7% at 182M/12 layers down to 41.6% at 830M/48 layers) provides quantitative support: deeper models create more reuse opportunities, making private allocation proportionally more wasteful.

Innovation 2: Routing Randomization as a Redundancy Diagnostic, Not a Robustness Test

Table 1 introduces a simple but conceptually sharp diagnostic: if you randomize the router in one deep-half layer and the accuracy barely moves, the experts in that layer are functionally interchangeable. This is a genuinely novel way to probe expert specialization, and it is distinct from prior redundancy analyses in important ways.

Prior work on MoE redundancy focused on parameters and outputs. Huang et al. (2025) showed expert weight matrices share a dominant subspace with cosine similarity above 0.9. Bai et al. (2025) showed pruning half of Mixtral's experts costs only ~8% relative quality. Wu et al. (2025) showed re-routing to similar experts preserves accuracy with up to 2× decoding speedup. These are all post-hoc measurements of similarity after training, answering "how similar are the experts we ended up with?"

The routing-randomization probe answers a different question: how much task-level information does the per-layer router actually carry at depth? It is a behavioral test, not a structural similarity measurement. If the router is selecting among meaningfully distinct expert specializations, randomizing that selection should impose a noticeable cost. The observed cost—only 1.0–1.6 accuracy points across three production models (Qwen1.5-MoE, DeepSeek-V2-Lite, Qwen3-30B-A3B) on five benchmarks—is strikingly small. For context, a 1-point accuracy drop on a model in the 30B parameter class means the choice among same-layer private experts at deep layers carries almost no discriminative signal about the downstream task.

This diagnostic is significant beyond the evidence it provides for UNIPOOL's motivation. It is a general-purpose probe that any MoE practitioner can apply to their trained model to assess whether their per-layer expert budget is carrying its weight. A model where deep-layer randomization costs 5+ points has genuinely specialized deep-layer experts; a model where it costs 1 point has a redundancy problem. The paper demonstrates this by applying the same probe to its own UNIPOOL models (Table 4), where the cardinality-matched randomization cost rises to 4.1 points—the shared experts have become load-bearing because they are no longer interchangeable. The probe thus serves double duty: it motivated the architecture, and it validates that the architecture solved the problem it set out to address.

Innovation 3: Load Balancing Must Match the Parameter Ownership Domain, Not the Routing Domain

The standard Switch auxiliary loss (Fedus et al., 2022) applies load balancing independently at each MoE layer because, in layer-private MoE, that is where parameters are owned. The paper's insight—which is simple in retrospect but non-obvious ex ante—is that when expert parameters become globally shared, the domain of the load-balancing objective must shift to match. Dead-expert prevention should be defined over the global pool, not per layer, because a globally dead expert wastes parameters (it is stored but never used anywhere) while a locally dead expert in a shared pool may be heavily used by other layers and is therefore not wasteful at all.

This insight is fundamental rather than incremental because it identifies a structural coupling between parameter ownership and regularization that the field had not articulated. Prior work on MoE load balancing—Switch's auxiliary loss, BASE layers' linear assignment, expert-choice routing, hash-based routing, auxiliary-loss-free methods—all operate within the per-layer paradigm and take the ownership structure as given. The paper shows that when you change the ownership structure (per-layer → global), the old balancing objective becomes actively harmful: a shared pool with per-layer auxiliary loss suffers global expert collapse (Figure 3a) and degrades validation loss below vanilla MoE (Table 5, +0.0163). This is a concrete failure mode that the field had no reason to anticipate because no one had tried decoupling expert ownership from layer structure.

The pool-level auxiliary loss (Equation 5) is the natural correction—aggregate token fractions across layers, apply a single balancing objective over the pool—but the conceptual contribution is not the loss formula itself (it is a straightforward generalization). The contribution is identifying that the auxiliary loss must be co-designed with the ownership structure, and that mismatching them creates qualitatively new failure modes (global collapse under per-layer balancing) that do not arise in layer-private MoE. This principle—that load balancing should operate at the granularity where parameters are allocated—is architecture-agnostic and would apply to any hierarchical or grouped expert sharing scheme, not just the global pool.

Innovation 4: The Shared Pool Converts Redundancy Into Specialization Rather Than Merely Removing It

The paper draws a sharp distinction between two ways of dealing with expert redundancy: remove it after training (pruning, similar-expert re-routing) or prevent it from arising during training (architectural sharing). This is more than a "prevention is better than cure" platitude; it is a claim about the mechanism by which sharing produces gains.

The evidence comes from the routing-randomization comparison in Table 4. In vanilla MoE (both production models in Table 1 and the paper's own trained models in Table 4), deep-layer expert choice carries little task-level information (1.0–1.6 point randomization cost). This means the per-layer routers are not learning to select among functionally distinct experts at depth—they are selecting from a set of near-substitutes. Post-hoc pruning can identify and remove the least-used or most-similar of these substitutes, recovering parameter efficiency, but it does not change the fact that the model never learned to use those experts for genuinely distinct computations.

UNIPOOL changes the training dynamics structurally. Every shared expert receives gradient signal from up to $L$ layers rather than one, which is up to a 12–48× increase in gradient budget per expert at the tested scales. This concentrated signal, combined with the competition among layers for limited pool capacity (the pool size $M$ is finite and layers must share), forces experts to specialize on transformations that are genuinely reusable across depth. The evidence that this specialization actually happens is the jump in randomization cost: 4.1 points on UNIPOOL vs. 1.3–1.5 points on vanilla MoE at matched scales (Table 4). The shared-pool router is making load-bearing decisions because the experts it selects among are no longer interchangeable—they have differentiated under the pressure of cross-layer competition.

This insight reframes the efficiency gains in Figure 2(a) as a training-dynamics advantage, not just a parameter-count advantage. The reduced-pool UNIPOOL variants using 41.6% of vanilla expert parameters do not just match the baseline by being parameter-efficient; they match or surpass it because the experts they do have are better-trained—each has seen gradient signal from more layers and has been forced to specialize to survive in the shared pool. This is a fundamentally different mechanism from pruning, which removes experts but leaves the surviving ones trained under the same thin-gradient conditions as before.

Innovation 5: Depth as a Reuse Multiplier, Not Just a Compute Multiplier

The paper's most counterintuitive empirical finding is that deeper models benefit more from sharing, and can tolerate more aggressive pool reduction. At 182M (12 layers), UNIPOOL needs 66.7% of vanilla expert parameters to match the baseline. At 469M (24 layers), it needs 50%. At 830M (48 layers), it needs only 41.6%. This monotonic trend—the smallest winning pool fraction shrinks with depth—is not a secondary result; it is the paper's strongest evidence that per-layer expert ownership is a structural inefficiency whose cost compounds with depth.

The standard MoE scaling intuition is that depth adds layers, and each layer needs experts, so expert parameters scale linearly with depth. The paper's finding violates this: the more layers you have, the smaller the fraction of per-layer expert budget you need, because each additional layer creates more opportunities to reuse the shared pool. This is the architectural analog of a scaling law: in vanilla MoE, expert parameters $\propto L$; in UNIPOOL, expert parameters can grow as $o(L)$ (sublinearly) while maintaining or improving quality.

The 830M vs. 978M comparison sharpens this point. The 830M model is deeper (48 layers, hidden 1024) with fewer active parameters than the 978M model (24 layers, hidden 1536), and it has fewer stored UNIPOOL parameters (5.081B vs. 5.742B). Yet UNIPOOL achieves its largest loss reduction over vanilla MoE in the deeper 830M model (−0.0386), and the 830M UNIPOOL model achieves a lower absolute validation loss than the wider 978M UNIPOOL model (1.6923 vs. 1.6999). The interpretation is that depth creates reuse sites for the shared pool, and this reuse benefit can outweigh the raw parameter-count advantage of a wider model. This is a budget-allocation insight with practical implications: given a fixed parameter budget for an MoE model, allocating it toward depth and a shared expert pool may be more effective than allocating it toward width and private per-layer experts.

This innovation is fundamental because it changes how one thinks about the depth-width tradeoff in MoE architectures. Depth is no longer just a way to add sequential computation; it is a multiplier on expert reuse—each additional layer is another opportunity to invoke the shared pool, increasing the gradient budget per expert and improving specialization. Under the shared-pool design, the optimal architecture for a given parameter budget may be substantially deeper than conventional MoE scaling would suggest, because the per-layer cost of adding depth (in terms of new expert parameters) drops dramatically when the pool is shared.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All models are trained on the Pile dataset (Gao et al., 2020), an 800GB corpus of diverse text for language modeling. Training runs for 60,000 iterations with batch size 512 and sequence length 1,024, totaling approximately 30B tokens. The paper does not specify a separate held-out validation set beyond what is used for perplexity reporting; validation loss is presumably computed on a standard Pile validation split.

• Base model(s). All experiments use LLaMA-style transformer backbones (Touvron et al., 2023) with RMSNorm (Zhang and Sennrich, 2019), SwiGLU activation (Shazeer, 2020), rotary positional embeddings (RoPE; Su et al., 2024), grouped query attention with 4 KV heads, and untied input/output embeddings. Five active-parameter scales are evaluated: 182M (12 layers, hidden 768), 469M (24 layers, hidden 1024), 650M (36 layers, hidden 1024), 830M (48 layers, hidden 1024), and 978M (24 layers, hidden 1536). Full architectural details including layer counts, hidden sizes, attention heads, and FFN dimensions are provided in Appendix Table 6. The paper states it chose these scales to cover a range from small to moderate-sized models where multiple runs and ablations are feasible.

• Metrics. The primary metric is validation loss (cross-entropy) computed on a held-out portion of the Pile dataset, measured after 30B training tokens. Validation perplexity is also reported as $\exp(\text{loss})$. For downstream evaluation, accuracy (%) is reported on seven zero-shot benchmarks: ARC-Easy and ARC-Challenge (Clark et al., 2018), PIQA (Bisk et al., 2020), HellaSwag (Zellers et al., 2019), WinoGrande (Sakaguchi et al., 2021), LAMBADA (Paperno et al., 2016), and RACE (Lai et al., 2017). The paper reports the unweighted mean across tasks as "Avg." All downstream evaluations use the LM Evaluation Harness framework (the paper does not explicitly name the framework, but the task selection is standard for this tool).

• Baselines. Three baselines are compared:

  • Dense: the same LLaMA backbone with a standard dense FFN (no expert routing, no MoE), serving as a lower bound to confirm that sparse expert routing itself is beneficial.
  • Vanilla MoE: 8 private expert FFNs per layer with top-1 softmax routing and the standard per-layer Switch auxiliary loss (Fedus et al., 2022) with coefficient $\alpha = 1 \times 10^{-2}$. This is the primary comparison point against UNIPOOL under matched expert-parameter budgets.
  • Vanilla MoE with sigmoid gating and no auxiliary loss (appearing only in the ablation study, Table 5): tests whether simply loosening load-balancing constraints in a layer-private setting can match UNIPOOL's gains.

• Generation budget / compute accounting. The paper measures compute implicitly through matched active parameters and per-token FLOPs, not generation count. In the main experiments (Table 2), vanilla MoE uses $E = 8$ private experts per layer with top-1 routing, so active expert evaluations per token per layer is 1. UNIPOOL uses $M = 8L$ shared experts (matching the total expert FFN count of vanilla MoE) with top-1 routing, so active expert evaluations per token per layer is also 1. Both architectures perform $L$ expert FFN evaluations per token; the comparison isolates the effect of sharing versus private ownership at matched compute. In the reduced-pool experiments (Figure 2a), $M$ is reduced below $8L$ while keeping top-1 routing, so active parameters and per-token FLOPs remain matched to vanilla MoE while stored expert parameters decrease. The paper does not report wall-clock time or throughput, acknowledging this as a limitation (Appendix A).

• Cross-validation / statistical protocol. For the 182M scale main results, validation loss is averaged over three random seeds to check variance. For larger scales (469M, 650M, 830M, 978M), one run per configuration is used due to training cost. The paper does not use cross-validation for hyperparameter selection; architecture hyperparameters (pool size, auxiliary loss coefficients, NormRouter settings) are fixed per scale before training. The only selection based on validation data is implicit in the reduced-pool sweep (Figure 2a), where different pool sizes are trained and compared post hoc to find the smallest winning pool size.

Main Quantitative Results

Validation Loss and Perplexity Across Scales

The headline result from Table 2 is that UNIPOOL consistently outperforms vanilla MoE across all five model scales on validation loss and perplexity after 30B Pile tokens. The absolute numbers are:

Perplexity values (Table 2) follow the same pattern: UNIPOOL achieves lower perplexity at every scale, with improvements ranging from 0.17 (978M) to 0.22 (830M) absolute perplexity points.

Several patterns in these numbers are noteworthy:

Both MoE methods substantially outperform dense baselines. At 182M, vanilla MoE reduces validation loss by 0.1103 over dense (2.042 → 1.9317), and UNIPOOL further reduces it by 0.0288. This confirms that sparse expert routing is effective at these scales and that the shared-pool architecture improves upon it.

The improvement is not monotonic in model scale. The largest improvement (−0.0386) occurs at 830M (48 layers, hidden 1024), not at the largest model (978M, 24 layers, hidden 1536, where the improvement is −0.0172). The paper interprets this as evidence that UNIPOOL's benefit is tied to depth (number of layers creating reuse opportunities) rather than raw parameter count. The 830M model has twice the depth of the 978M model (48 vs. 24 layers), and the deeper model with a shared pool achieves a larger gain over its matched vanilla baseline and a lower absolute validation loss (1.6923 vs. 1.6999 for the 978M UNIPOOL model). The paper explicitly frames this as a budget-allocation finding: "for this architecture family, allocating capacity toward depth and reusable expert pools can be more effective than allocating it primarily to width."

The 978M gap is the smallest. The paper addresses this directly, arguing it is expected rather than contradictory: the 978M model allocates capacity primarily to width (hidden 1536 with only 24 layers), creating fewer cross-layer reuse opportunities for the shared pool. UNIPOOL's marginal gain is strongest when the architecture exposes more sites (layers) that can reuse the global pool. This is a conditional finding: the benefit of sharing depends on architectural depth, not just total parameters.

Reduced-Pool Results: Sublinear Expert Scaling

Figure 2(a) reports validation-loss change relative to vanilla MoE as a function of the fraction of vanilla expert parameters retained in the shared pool. The key quantitative finding is that UNIPOOL can match or outperform the layer-private baseline with a substantially smaller expert pool:

• At 182M ($L = 12$, matched pool size $M = 96$): the smallest winning pool uses 66.7% of vanilla expert parameters ($M = 64$, achieving validation loss 1.9215 vs. vanilla MoE's 1.9317).
• At 469M ($L = 24$, matched $M = 192$): the smallest winning pool uses 50% ($M = 96$, achieving −0.007 relative to vanilla MoE).
• At 650M ($L = 36$, matched $M = 288$): the smallest winning pool uses 50% ($M = 144$, achieving −0.011 relative to vanilla MoE).
• At 830M ($L = 48$, matched $M = 384$): the smallest winning pool uses 41.6% ($M = 160$, achieving −0.013 relative to vanilla MoE).

The paper explicitly notes the monotonic trend: "the smallest winning fraction shrinks monotonically with depth, so deeper backbones tolerate progressively smaller shared pools." At 830M with 48 layers, UNIPOOL beats the layer-private baseline while using fewer than half the expert parameters.

The reduced-pool variants tested at each scale (Figure 2a markers):

• 182M: $M = 64$ (66.7%), $M = 48$ (50%)
• 469M: $M = 128$ (66.7%), $M = 96$ (50%), $M = 64$ (33.3%)
• 650M: $M = 144$ (50%), $M = 128$ (44.4%), $M = 96$ (33.3%)
• 830M: $M = 160$ (41.6%), $M = 128$ (33.3%)

At every scale, the $\Delta$ validation loss curve relative to vanilla MoE trends downward (improving) as the pool gets smaller, up to a point. For example, at 469M, the losses relative to the vanilla baseline are approximately −0.002 ($M = 128$, 66.7%), −0.007 ($M = 96$, 50%), and then the curve crosses zero around $M = 64$ (33.3%). The paper does not report exact values for all intermediate points, but the pattern is clear from the plot: there is an optimal pool size below the matched budget, and reducing too far eventually degrades performance as the pool becomes capacity-limited.

The paper interprets these results as a test of the over-provisioning hypothesis from Section 3: "if vanilla MoE's layer-private expert sets duplicate useful functions, then a smaller globally shared pool should be able to match or surpass the larger per-layer allocation. The reduced-pool results support this prediction."

Granularity Scaling

Figure 2(b) shows that UNIPOOL's benefit composes with finer-grained MoE configurations on the 182M backbone. The specific configurations tested:

• 8E/top-1: vanilla MoE validation loss 1.9317, UNIPOOL 1.9029
• 16E/top-2: vanilla MoE approximately 1.910 (from the plot), UNIPOOL approximately 1.885 (estimated from plot; precise values not tabulated)
• 32E/top-4: vanilla MoE approximately 1.890 (from the plot), UNIPOOL approximately 1.860 (estimated from plot)

Both methods improve with larger expert counts, consistent with the fine-grained MoE scaling findings of Krajewski et al. (2024), and UNIPOOL maintains its advantage over vanilla MoE at each granularity level. The paper does not provide exact validation loss values for the granularity sweep in Table 2, reporting them only in the downstream evaluation table (Table 3 bottom block) as accuracy numbers.

Downstream Evaluation

Table 3 reports zero-shot accuracy on seven benchmarks. The overarching finding is that the validation loss improvements translate to downstream task gains, though with more variability than the clean validation loss trend.

Main scales (8E/top-1):

• At 182M, UNIPOOL improves the average accuracy from 38.74 (vanilla MoE) to 39.61 (+0.87 points), with gains on ARC-E (+1.01), ARC-C (+0.51), PIQA (+1.25), HellaSwag (+0.68), WinoGrande (flat), LAMBADA (+1.78), and RACE (+0.86). The task-level pattern is uneven: LAMBADA shows the largest relative gain, while WinoGrande shows no improvement.
• At 469M, UNIPOOL improves average accuracy from 41.62 to 43.11 (+1.49 points), with consistent gains across all seven tasks. The largest absolute improvements are on LAMBADA (+1.98) and ARC-E (+2.65).
• At 650M, UNIPOOL improves from 43.04 to 43.79 (+0.75 points), a smaller margin than at 469M. Notably, WinoGrande slightly degrades (53.04 → 52.49, −0.55).
• At 830M, UNIPOOL achieves its largest average improvement: 43.82 → 45.67 (+1.85 points), with gains on six of seven tasks (WinoGrande is essentially flat at +0.16). This aligns with the validation loss pattern where 830M showed the largest UNIPOOL gain (−0.0386).
• At 978M, the average improvement is modest: 43.91 → 44.07 (+0.16 points). Three tasks show small gains (ARC-E +1.10, PIQA +1.20, LAMBADA +0.31), while three show small degradations (ARC-C −0.94, WinoGrande +0.16 effectively flat, RACE −1.05). This aligns with the smaller validation loss improvement at this scale (−0.0172).

Granularity sweep (182M backbone):

• 16E/top-2: vanilla MoE 40.33, UNIPOOL 41.22 (+0.89 points)
• 32E/top-4: vanilla MoE 41.49, UNIPOOL 42.62 (+1.13 points)

Both methods improve with more experts and higher top-k, and UNIPOOL's advantage widens slightly at 32E/top-4 (+1.13) compared to 8E/top-1 (+0.87). This supports the claim that UNIPOOL composes with finer-grained expert decomposition.

The paper does not provide statistical significance tests or confidence intervals for the downstream numbers. Given that the test sets for some of these benchmarks are relatively small (e.g., ARC-Challenge has 1,172 questions, WinoGrande has 1,267), task-level fluctuations of ±0.5–1.0 points may not be statistically reliable, and the paper's emphasis on the average across seven tasks is a reasonable mitigation strategy.

Training Dynamics

Appendix Figure 4 provides the validation loss curves over the 60,000 training iterations. The key observations:

Panels (a)–(c) show UNIPOOL vs. vanilla MoE at 182M, 469M, and 650M. In all three cases, UNIPOOL stays below the vanilla MoE curve after the initial warmup phase (first ~600 iterations at 1% warmup fraction). The gap is not just a final-checkpoint artifact; it emerges early and is sustained throughout training. At 182M, the gap widens steadily after about 10K steps. At 469M and 650M, the two curves track closer during mid-training but diverge in the final 10K–20K steps.

Panel (d) shows the sharing-scope ablation at 182M over the full training trajectory. The ordering of the curves matches the endpoint ablation results (Table 5): global sharing (G=1, full UNIPOOL) is the lowest-loss configuration throughout most of training, vanilla MoE (G=12) is the highest-loss endpoint, and intermediate grouping configurations (G=2, 4, 6) generally lie between them. The paper notes that "broader expert sharing improves the optimization trajectory itself, rather than merely selecting a better final checkpoint." This is an important claim because it distinguishes UNIPOOL's benefit from, say, a better initialization or a regularization effect that only matters at convergence—the sharing architecture provides a sustained optimization advantage.

Ablation Studies and Robustness Checks

All ablations are conducted at the 182M scale ($L = 12$, $H = 768$) with results reported in Table 5 and discussed in Sections 3.4 and 5.4.

• Shared pool with per-layer auxiliary loss and softmax routing: degrades validation loss to 1.9480 compared to vanilla MoE's 1.9317 (+0.0163, worse than vanilla MoE). This configuration takes the architectural change (shared pool) but retains the standard per-layer Switch auxiliary loss and softmax routing. The failure mechanism is visualized in Figure 3a: the per-layer auxiliary loss with a much larger per-layer candidate set ($M = 96$ vs. $E = 8$) provides insufficient balancing pressure, causing aggregate traffic to collapse onto a small subset of shared experts. This is the "global expert collapse" phenomenon—the visual evidence in Figure 3a shows dark vertical bands in the heatmap (a few heavily used experts) and a highly non-uniform aggregate usage bar. The paper identifies this as the key reason that simply replacing private experts with a shared pool fails: the balancing objective must be co-designed with the architectural change.

• Shared pool with pool-level auxiliary loss and softmax routing: improves validation loss to 1.9180 (−0.0137 vs. vanilla MoE). Replacing the per-layer auxiliary loss with the pool-level loss partially recovers the sharing benefit, because the balancing objective is now aligned with global parameter ownership. However, this configuration still uses softmax routing rather than NormRouter, and the improvement (−0.0137) is less than half of the full UNIPOOL improvement (−0.0288). The paper attributes the remaining gap to softmax's unsuitability for routing into a large shared candidate pool.

• Vanilla MoE with NormRouter (no sharing): achieves 1.9375, slightly worse than standard vanilla MoE with softmax (1.9317, +0.0058). This is a critical negative result: NormRouter is not a universally stronger router. Applied in the standard layer-private setting with only $E = 8$ candidates per layer, it marginally degrades performance. The paper interprets this as evidence that NormRouter's advantages (scale stability, natural sparsity) are specifically matched to the shared-pool setting where routers select from $M = 96$ candidates and layer-dependent hidden-state scales can cause cross-layer routing interference. In the small-candidate layer-private setting, softmax's dense probability distribution works fine and the scale-sensitivity issue is less pronounced because there is no cross-layer competition for expert capacity.

• Vanilla MoE with sigmoid gating and no auxiliary loss: achieves 1.9239 (−0.0078 vs. standard vanilla MoE). This tests whether simply removing load-balancing constraints (using sigmoid gating, which does not require an auxiliary loss) can match UNIPOOL's gains. The modest improvement (−0.0078) is far smaller than UNIPOOL's full improvement (−0.0288), confirming that UNIPOOL's gains are not explained by loosening or removing auxiliary loss pressure. Load balancing is not the problem; the architectural sharing plus matched stabilization is.

• Full UNIPOOL (shared pool + pool-level aux loss + NormRouter): achieves 1.9029 (−0.0288 vs. vanilla MoE), the best result in the ablation table. The improvement is super-additive: the sum of the individual improvements from pool-level aux loss (−0.0137 applied to shared pool with softmax) and NormRouter (+0.0058 applied to vanilla MoE, noting this was a degradation rather than improvement) does not account for the full UNIPOOL gain. This super-additivity is the paper's evidence for co-design: the components are not independent improvements that sum linearly; they interact, with NormRouter being specifically beneficial in the shared-pool context where it prevents the routing instability that would otherwise limit the benefit of the pool-level loss.

• Intermediate sharing scope (G = 6, 4, 2): tests varying the number of expert-pool groups across depth. Each group shares its own pool of experts with its own pool-level auxiliary loss and NormRouter. Results: G = 6 (1.9121, −0.0196), G = 4 (1.9099, −0.0218), G = 2 (1.9213, −0.0104). The G = 2 result is anomalous—it is worse than G = 4 despite having broader sharing within each group (6 layers per group vs. 3 layers). The paper does not explain this inversion, but it may reflect an unfavorable grouping boundary where layers with very different routing preferences are forced into the same pool. The key finding is that global sharing (G = 1, 1.9029) outperforms all grouped configurations, supporting the claim that cross-layer reuse benefits from the widest possible sharing scope. The training trajectories in Appendix Figure 4d show this ordering is maintained throughout training, not just at the final checkpoint.

• Expert utilization visualization (Figure 3): while not a standard ablation, the heatmap comparison between Figure 3a (shared pool + per-layer aux loss + softmax) and Figure 3b (UNIPOOL with pool-level aux loss + NormRouter) provides visual evidence for the stabilization mechanisms. In Figure 3a, the aggregate usage bar (bottom) shows extreme non-uniformity—a few experts dominate—while the per-layer heatmap (top) shows limited layer-specific routing diversity. In Figure 3b, the aggregate usage bar is approximately uniform (matching the dashed "Uniform (0.125)" reference line), while the per-layer heatmap shows clear layer-specific patterns (different layers favor different expert subsets). This demonstrates that the pool-level loss achieves balanced global usage without forcing every layer to use every expert—exactly the design intent.

Critical Assessment

Claim 1: "UNIPOOL consistently improves validation loss and perplexity over matched vanilla MoE baselines." This claim is directly supported by Table 2. Across all five scales, UNIPOOL achieves lower validation loss and perplexity. The improvements range from −0.0172 (978M) to −0.0386 (830M), and both MoE methods outperform dense baselines by substantial margins. The paper provides training curves (Appendix Figure 4) showing the gap is sustained throughout training, not a final-checkpoint artifact. However, the claim is supported only on a single dataset (30B Pile tokens) with a single model family (LLaMA-style). The generalization to other data distributions, longer training horizons (30B tokens is modest for these model scales), or other model families is untested. The consistent trend across five scales is encouraging but does not constitute a scaling law.

Claim 2: "Reduced-pool variants using only 41.6%–66.7% of the vanilla expert-parameter budget match or outperform layer-wise MoE." This claim is supported by Figure 2(a) and the text in Section 5.2 describing the smallest winning pool fractions. The evidence is specific: at each of four scales (182M, 469M, 650M, 830M), a pool smaller than the matched vanilla budget achieves validation loss at or below the vanilla MoE baseline. The monotonic trend—deeper models tolerate more aggressive pool reduction—is a strong and non-obvious result. However, there are important caveats: (1) the winning fractions are determined by post-hoc comparison of trained models, not by a principled selection procedure; (2) the reduced-pool runs at each scale are sparse (two or three pool sizes tested), so the true optimal pool size for each scale may lie between the tested points; (3) storage savings from reduced pools are real but the paper does not report throughput or memory measurements (acknowledged in Appendix A); (4) the comparison is against a vanilla MoE baseline that may itself be over-parameterized at these scales—the claim is that UNIPOOL can match with fewer parameters, not that the vanilla baseline was optimal to begin with.

Claim 3: "UNIPOOL's shared experts carry substantially more task-level information than layer-private deep experts." This claim is supported by the routing-randomization comparison in Table 4. On the paper's own trained models at 469M and 978M, single-layer deep-half randomization drops average accuracy by 1.3/1.5 points on vanilla MoE (replicating the production-model pattern) versus 4.1 points on UNIPOOL under the cardinality-matched protocol. This is a clean, controlled comparison: the models are trained under matched data and optimizer settings, differing only in the architecture. The large jump in randomization cost provides direct behavioral evidence that UNIPOOL experts are less substitutable. Weaknesses: (1) the cardinality-matched protocol for UNIPOOL samples from each layer's top-8 most-used shared experts rather than uniformly from the full pool—this is well-motivated (keeping choice set size matched) but introduces a layer-specific dependency that complicates interpretation; (2) the evaluation is on only five benchmarks (Table 11), and task-level variation is substantial (e.g., on UNIPOOL at 469M, PIQA drops from 69.10 to 60.62 under randomization while WinoGrande drops from 52.72 to 50.61); (3) the paper reports only average accuracy in the main text (Table 4), burying the per-task breakdown in Appendix Table 11.

Claim 4: "Expert parameters need not grow linearly with depth; they can grow sublinearly while remaining more efficient and effective than vanilla MoE." This claim, stated in the abstract and Section 7, is the paper's headline scaling insight. The evidence comes from the reduced-pool results in Figure 2(a): at every tested scale, a sub-vanilla pool matches or outperforms the baseline. The monotonic decrease in the winning fraction with depth (66.7% → 50% → 41.6%) is the key pattern. However, this claim should be qualified: the paper demonstrates sublinear scaling at the tested scales (12–48 layers, 182M–830M active parameters), not at arbitrary scales. Extrapolating to hundreds of layers or billion-parameter models is speculative. The paper acknowledges this limitation explicitly (Appendix A: "validating UNIPOOL at billion-parameter scales with longer training horizons is an important direction"). Additionally, the claim that expert parameters "need not" grow linearly is supported, but the claim that they "can grow sublinearly while remaining more efficient" requires a stronger efficiency metric than validation loss alone—the paper does not report throughput, memory, or FLOPs comparisons in the reduced-pool regime (Appendix A acknowledges this gap).

Claim 5: "Pool size is an explicit depth-scaling hyperparameter." This framing is conceptually valuable but is not experimentally validated as a scaling law. The paper shows that pool size affects validation loss at each scale and that the optimal pool size (for matching the vanilla baseline) varies with depth, but it does not fit a functional relationship between depth and optimal pool size or provide guidance for choosing $M$ at new scales. The experiments test only a handful of pool sizes per scale (2–3), making the "optimal" pool size a coarse estimate. A true scaling analysis would sweep pool size more densely across a wider range of depths and model sizes, potentially revealing a power-law relationship analogous to the Chinchilla scaling laws for pretraining.

Genuine weaknesses in the experimental design:

Single dataset, single model family. All training is on 30B Pile tokens with LLaMA-style backbones. The Pile is a diverse corpus, so overfitting to dataset idiosyncrasies is less of a concern than it would be for a narrow-domain dataset, but the architectural findings are unvalidated on other data distributions (code, multilingual text, domain-specific corpora) and other model architectures (non-LLaMA transformer variants, encoder-decoder models). The paper's production-model routing probes (Table 1) use Qwen and DeepSeek models, which are different families, but these probes only motivate the problem—they do not validate that UNIPOOL works on those families.

No throughput or memory benchmarks. The paper measures success entirely in terms of validation loss and downstream accuracy. The shared-pool architecture has the same total expert FFN count as vanilla MoE in the matched setting, so storage is identical; in the reduced-pool regime, storage decreases. However, routing into a single global pool of $M = 8L$ experts (where $L$ can be 48) introduces all-to-all communication patterns in distributed training that may differ from per-layer expert parallelism. The paper uses expert parallelism of 1 in all experiments, so these distributed-systems concerns are untested. For a method whose stated advantage includes "efficiency," the absence of throughput or memory measurements is a significant gap.

30B tokens is modest for these model scales. The largest model tested (978M active parameters) is trained on only ~30B tokens, for a token-to-parameter ratio of roughly 30.7. The Chinchilla-optimal ratio for dense models is approximately 20 tokens per parameter, so 30B tokens is in the right ballpark for the 978M scale, but the paper uses a cosine schedule that completes at 60K steps—a fixed training horizon rather than a compute-optimal horizon. It is unknown whether UNIPOOL's advantage persists, shrinks, or grows with longer training. The training curves in Appendix Figure 4 show no sign of convergence at 60K steps (loss is still declining), so the reported numbers are checkpoint-at-budget, not converged values.

No combination of UNIPOOL with other MoE improvements. The paper compares UNIPOOL against a basic vanilla MoE with top-1 softmax routing and per-layer Switch auxiliary loss. It does not test whether UNIPOOL's benefits are additive with other known MoE improvements: z-loss router regularization (Zoph et al., 2022), expert choice routing (Zhou et al., 2022), shared experts alongside routed experts (DeepSeekMoE-style; Dai et al., 2024), or auxiliary-loss-free balancing (Wang et al., 2024). The granularity sweep (Figure 2b) shows UNIPOOL composes with fine-grained experts, which is a start, but more comprehensive composition tests would strengthen the claim that UNIPOOL is a general architectural improvement rather than a technique that works best on simple baselines.

The G=2 anomalous result is unexplained. In the sharing-scope ablation, G=2 (two groups of 6 layers sharing 48 experts each) performs worse than G=4 (four groups of 3 layers sharing 24 experts each): 1.9213 vs. 1.9099. The paper notes this without explanation. This could be a noise artifact (only one run per ablation), or it could indicate that grouping non-adjacent or semantically dissimilar layers into the same pool forces unproductive competition. Understanding when and why intermediate sharing scopes can underperform is important for practitioners who might consider grouped sharing as a middle ground, but the paper does not investigate this anomaly.

No confidence intervals or statistical testing. The 182M results are averaged over three seeds, but the paper does not report standard deviations or confidence intervals. For larger scales, single runs are used. The downstream evaluation (Table 3) reports accuracy to two decimal places without any indication of variance. For tasks like WinoGrande (1,267 test examples), a single percentage-point difference corresponds to roughly 13 examples, which could easily arise from sampling noise in model training. The paper's conclusions about which specific tasks benefit most from UNIPOOL are therefore less reliable than the aggregate validation loss trend.

The paper's strongest evidence is in the routing-randomization comparison (Table 4) and the reduced-pool monotonic trend (Figure 2a). These are clean, controlled experiments with clear quantitative results that directly support the paper's central narrative: that layer-private ownership causes redundancy, and that sharing prevents it. The weakest evidence is in the downstream evaluation, where task-level improvements are small and variable, and the absence of throughput benchmarks limits the practical case for adoption. A deployment-focused reader would want to know: "if I switch to UNIPOOL, what happens to my training throughput, inference latency, and memory usage?" The paper does not answer this.

Experiments that would have strengthened the paper:

1. Training to convergence or a much longer horizon (100B+ tokens) at one or two scales to test whether the UNIPOOL advantage is sustained or narrows. The current training curves show loss still declining at 60K steps, so the 30B-token comparison is a snapshot, not a converged measurement.

2. Throughput and memory benchmarking at least at one scale in the matched and reduced-pool configurations, using standard expert-parallel distributed training. This is the most obvious missing experiment for a methods paper proposing an architectural change to MoE training.

3. Scaling sweeps over both depth and pool size simultaneously to estimate a functional relationship between $L$, $M$, and validation loss. The current experiments vary $L$ and $M$ independently but never jointly sweep both at a fixed total parameter budget to find the optimal allocation.

4. A stronger vanilla MoE baseline incorporating modern stabilization techniques (router z-loss, auxiliary-loss-free balancing) to test whether UNIPOOL's gains persist against a more competitive baseline rather than the basic Switch-style vanilla MoE.

5. Application of the routing-randomization probe to different depth regions (early vs. middle vs. deep layers) in both vanilla MoE and UNIPOOL, to localize where redundancy is worst and where sharing helps most. Table 4 averages over deep-half layers only; a full depth-profile would be more informative.

6. Tests on a non-Pile dataset (e.g., Dolma; Soldaini et al., 2024) or a code-heavy corpus to assess domain generalization.

6. Limitations and Trade-offs

The Cost of Difficulty Estimation (for the Routing-Randomization Probe) Is Not Accounted For, and No Online Estimator Exists

The assumption or constraint. The routing-randomization probe in Section 3 (Table 1) and Section 6.1 (Table 4) is the paper's central diagnostic for identifying expert redundancy in vanilla MoE and validating that UNIPOOL eliminates it. However, this probe requires per-layer identification of the top-8 most-used experts from a held-out validation split to construct the cardinality-matched randomization protocol for UNIPOOL. The paper states this explicitly in Appendix F: "We therefore first identify each layer's top-8 most-used shared experts on a held-out Pile validation split, then sample uniformly from that per-layer top-8 set during the intervention." This is an offline analysis performed after training; it is not a technique that can be applied during training to dynamically assess whether experts are specializing, nor is its cost accounted for in any training or evaluation budget.

The consequence. The routing-randomization probe is the paper's strongest evidence that UNIPOOL experts are genuinely more specialized than vanilla MoE experts (4.1 vs. 1.3–1.5 point accuracy drops under randomization). But this evidence is post-hoc and dependent on a specific experimental protocol (pre-computing per-layer top-8 usage statistics on a validation split). A practitioner training a UNIPOOL model from scratch has no way to know during training whether the shared experts are specializing or collapsing into the redundancy pattern that the probe is designed to detect—they can only check after training is complete. If the shared pool does not induce specialization (e.g., due to a mis-tuned pool auxiliary loss coefficient or an insufficiently large pool), this failure would only be detectable after the full training budget is spent. Additionally, the probe's cost itself—multiple evaluation passes with randomized routing, one per deep-half layer—makes it impractical as a monitoring tool during large-scale training.

What evidence exists in the paper. The paper provides the probe results only for fully trained models (Tables 1, 4, 10, 11) and does not report any intermediate-checkpoint randomization results or attempt to correlate the probe's findings with training dynamics. The sharing-scope ablation in Appendix Figure 4d shows validation loss trajectories that diverge early (full UNIPOOL stays below vanilla MoE after warmup), which suggests that specialization benefits emerge during training, but there is no direct behavioral evidence (e.g., randomization cost at intermediate checkpoints) to confirm this. The paper's claim that "the shared pool reduces expert substitutability" (Section 6.1) is supported only at the final checkpoint.

Mitigation status. Not addressed. The paper treats the routing-randomization probe purely as an analysis tool, not as a method that needs to be made practical or online. No lightweight proxy for expert specialization (e.g., tracking the entropy of per-layer routing distributions, measuring expert usage overlap across layers, or computing pairwise expert weight similarities during training) is proposed or evaluated. A practitioner who wants to monitor whether UNIPOOL is working as intended during training has no guidance from the paper.

---

UNIPOOL Is Evaluated on a Single Model Family, a Single Dataset, and a Fixed Training Horizon of Only 30B Tokens

The assumption or constraint. All UNIPOOL experiments use LLaMA-style transformer backbones (RMSNorm, SwiGLU, RoPE, grouped query attention) trained on exactly 30B tokens from the Pile dataset with a cosine learning rate schedule completing at 60,000 iterations. The paper acknowledges this explicitly in Appendix A: "Our experiments are conducted at 182M–978M parameter scales with 30B training tokens. While the consistent improvement across five scales is encouraging, validating UNIPOOL at billion-parameter scales with longer training horizons is an important direction." The production MoE models used for the motivating redundancy probes (Qwen1.5-MoE, DeepSeek-V2-Lite, Qwen3-30B-A3B) are from entirely different model families trained on different data distributions with different architectures, but these are only used to motivate the problem—they are never used to validate UNIPOOL.

The consequence. Three distinct generalization risks are conflated but unresolved:

Model family dependence. The LLaMA architecture has specific properties—SwiGLU activations, RoPE positional embeddings, grouped query attention—that may interact with shared-pool routing in ways that do not generalize to other architectures. For example, the hidden-state scale variation across depth that the paper cites as motivation for NormRouter (Section 4.3) may be more or less severe under different normalization schemes or activation functions. The production-model routing probes show that the redundancy problem exists across families (Qwen, DeepSeek), but they do not show that UNIPOOL is the right solution for those families.

Dataset dependence. The Pile is a diverse English-language text corpus, but it has specific properties (domain mixture, token distribution, sequence length characteristics) that may influence how cross-layer expert reuse emerges. A dataset with different properties—more code, multilingual text, highly structured data—might produce different routing patterns and different optimal pool sizes. The paper does not test on any dataset other than the Pile, not even on another standard pretraining corpus like Dolma or C4.

Training horizon dependence. All models are trained for exactly 60,000 iterations (~30B tokens). The training curves in Appendix Figure 4 show that validation loss is still declining at the final checkpoint at all scales. This means the reported comparisons are snapshot comparisons at an arbitrary training budget, not converged comparisons. It is possible—and the paper provides no evidence either way—that the UNIPOOL vs. vanilla MoE gap narrows, widens, or even reverses with longer training. For example, if vanilla MoE's redundant experts eventually specialize under sufficient training (the redundancy is a transient phenomenon due to limited gradient signal per expert at early stages), then UNIPOOL's advantage might be most pronounced at the modest token budgets tested and diminish at larger budgets. Conversely, if the concentrated gradient signal in UNIPOOL genuinely produces better-trained experts, the gap might widen further. The paper's findings cannot distinguish between these scenarios.

What evidence exists in the paper. The paper provides training curves only up to 60K steps (Appendix Figure 4). The production-model probes (Table 1) are on fully trained models (Qwen1.5-MoE, DeepSeek-V2-Lite, Qwen3-30B-A3B were trained on trillions of tokens), confirming that redundancy persists at scale in vanilla MoE, which supports the hypothesis that the problem is structural rather than transient. However, this does not tell us whether UNIPOOL's advantage over vanilla MoE is structural or transient—it only tells us that the problem UNIPOOL addresses is structural.

Mitigation status. The paper acknowledges the scale and training horizon limitations in Appendix A and suggests future work. It partially addresses the model family concern by showing that redundancy exists across three production model families (Table 1), but this only validates the problem, not UNIPOOL as the solution for those families. No experiments on additional datasets or longer training horizons are reported.

---

The Routing-Randomization Probe's Cardinality-Matched Protocol for UNIPOOL Introduces a Layer-Specific Dependency That Complicates Interpretation of the Core Specialization Claim

The assumption or constraint. The paper's central claim about specialization—that "the shared pool reduces expert substitutability" and "UNIPOOL routers select reusable computations that are less interchangeable than layer-private deep experts" (Section 6.1)—is supported by comparing randomization costs: vanilla MoE drops 1.3–1.5 points under uniform random routing within a layer's 8 private experts, while UNIPOOL drops 4.1 points under the "cardinality-matched" protocol. However, the two randomization protocols are not symmetric in what they test. In vanilla MoE, randomizing a deep-half layer means uniformly sampling from the 8 private experts owned by that specific layer—a true test of within-layer expert substitutability, because the candidate set is the same for every deep-half layer. In UNIPOOL, the cardinality-matched protocol samples from each layer's per-layer top-8 most-used shared experts, which are identified from a held-out validation split (Appendix F). This means the candidate set for randomization is layer-specific: layer 12's top-8 experts are potentially completely different from layer 24's top-8 experts. The randomization is therefore sampling from a set that already reflects the learned routing preferences of that specific layer.

The consequence. The 4.1-point drop on UNIPOOL conflates two distinct sources of information loss: (1) the loss from replacing a specific expert choice with a random choice among candidates that the layer actually uses, which would be the cardinality-matched analog of the vanilla MoE test, and (2) the loss from potentially selecting an expert that the layer uses frequently but in a different context, which reflects the fact that UNIPOOL layers have genuinely different routing preferences (a desirable property that the per-layer heatmap in Figure 3b visually demonstrates). The protocol is measuring something about specialization, but it is not a clean comparison to the vanilla MoE probe, because the vanilla MoE probe does not pre-filter candidates by usage frequency—it samples uniformly from all 8 experts regardless of how often each is used. A truly symmetric protocol would sample uniformly from all 8 experts available to that layer in both architectures, but in UNIPOOL all $M = 8L$ experts are available to every layer, making uniform sampling over the full pool a different test (which the paper does report in Appendix Table 11, where the full-pool random cost is 3.8 points at 469M).

What evidence exists in the paper. Appendix Table 11 reports both the cardinality-matched protocol (marked with †) and the standard full-pool random protocol (unmarked) for UNIPOOL. At 469M, full-pool random drops accuracy by 3.8 points vs. 4.1 for the matched protocol; at 978M, full-pool random drops by 4.0 points vs. 4.1 for the matched protocol. The two protocols produce similar numbers, which the paper could use to argue that the protocol choice does not drive the conclusion. However, the paper does not discuss this comparison or acknowledge the asymmetry. The main text (Table 4 and Section 6.1) reports only the cardinality-matched protocol with the interpretation that it demonstrates reduced expert substitutability, without addressing how the layer-specific candidate set affects that interpretation.

Mitigation status. Partially addressed by the inclusion of both protocols in Appendix Table 11, but the interpretation in the main text does not engage with the asymmetry. A cleaner comparison would be to report the randomization cost for vanilla MoE also using a cardinality-matched protocol (sampling from the top-8 most-used experts per layer, which for vanilla MoE with 8 experts per layer is essentially the same as uniform sampling since all 8 are candidates), or to explicitly discuss why the two architectures require different protocols and what that means for the comparison. The paper's framing of the 4.1-point drop as evidence that "UNIPOOL routers select reusable computations that are less interchangeable than layer-private deep experts" is valid given the similar full-pool and matched-protocol numbers, but the reasoning is not fully articulated.

---

No Throughput, Memory, or Communication Benchmarks Are Reported, Leaving the Practical Deployment Case Unresolved

The assumption or constraint. The paper evaluates UNIPOOL solely through validation loss, perplexity, and downstream accuracy. It does not report wall-clock training throughput, inference latency, GPU memory usage, or expert-parallel communication patterns for any configuration. The paper states this explicitly in Appendix A: "We do not report wall-clock throughput comparisons in this work. At the matched setting (M = 8L), UNIPOOL has the same total expert FFN count as vanilla MoE, so the architectural change is that all layers share a single pool by reference rather than that the parameter count itself decreases. Storage and memory savings emerge only in the reduced-pool regime (Section 5.2), where smaller pools achieve matched or better quality with strictly fewer expert parameters. The pool auxiliary loss also introduces a small overhead from cross-layer statistic accumulation, and routing into a larger candidate pool may affect token-dispatch efficiency under expert parallelism; a detailed throughput and expert-parallel scaling study is left for future work."

The consequence. A practitioner evaluating whether to adopt UNIPOOL faces several unresolved practical questions:

In the matched setting ($M = 8L$), does the shared pool introduce overhead that negates the validation loss gains? The shared pool architecture means that every MoE layer's router must compute scores over $M = 8L$ experts (e.g., 384 experts at $L = 48$) rather than 8 experts per layer. At $L = 48$ with hidden size 1024, the router weight matrix per layer is $384 \times 1024$ in UNIPOOL vs. $8 \times 1024$ in vanilla MoE—a 48× increase in router parameters per layer and a corresponding increase in the cost of computing routing logits. NormRouter's L2 normalization and ReLU add additional operations per token per layer. The paper argues that ReLU-induced sparsity means only a fraction of experts receive non-zero scores (roughly half after L2 normalization and ReLU), but the full $M$-dimensional logit vector must still be computed before normalization. This router computation overhead may be small relative to the expert FFN computation (which dominates MoE FLOPs), but the paper provides no measurements to confirm this.

In the reduced-pool setting, do the storage savings translate proportionally to memory and throughput improvements? The paper's headline efficiency result—matching vanilla MoE with only 41.6–66.7% of the expert parameters—implies substantial storage savings. But in distributed training with expert parallelism, the communication patterns for a single global pool of $M$ experts are different from $L$ per-layer pools of $E$ experts each. With per-layer expert parallelism, each layer's all-to-all dispatch is independent and can be overlapped with computation from other layers. With a global shared pool, all layers dispatch to the same set of experts, which might create contention or require different parallelization strategies. The paper uses expert parallelism of 1 in all experiments (Table 7), so these distributed-systems concerns are entirely untested—the models are trained with all experts on a single device, which is feasible at the tested scales (up to 978M parameters) but would not scale to the billion-parameter regimes where MoE is most commonly deployed.

What is the one-step-behind pool auxiliary loss overhead? The pool-level auxiliary loss requires accumulating token-dispatch statistics across all layers and using the previous micro-batch's statistics in the current micro-batch's loss computation (Appendix G). This introduces a cross-micro-batch dependency that could complicate gradient accumulation, activation checkpointing, or pipeline parallelism. The paper states the overhead is "small" but provides no measurements.

What evidence exists in the paper. None. The paper provides zero throughput, memory, or communication measurements. All efficiency claims are based on parameter counts and validation loss, not on wall-clock time or hardware utilization. The paper acknowledges this limitation in Appendix A but treats it as scope for future work rather than as a gap that limits the actionable conclusions of the current paper.

Mitigation status. Not addressed. The paper explicitly defers throughput and expert-parallel scaling studies to future work. For a methods paper proposing an architectural change to MoE training—where practical throughput is often the binding constraint on adoption—this is a significant gap. The paper's claim that "storage and memory savings emerge only in the reduced-pool regime" is correct as stated (fewer parameters means less memory), but it does not account for the router overhead, the pool auxiliary loss overhead, or the distributed communication patterns that might erode or even reverse these savings in practice.

---

The Vanilla MoE Baseline Is Basic and Does Not Incorporate Modern Stabilization Techniques That Could Narrow or Eliminate UNIPOOL's Advantage

The assumption or constraint. The vanilla MoE baseline throughout the paper uses top-1 softmax routing with the standard per-layer Switch auxiliary loss (coefficient $\alpha = 1 \times 10^{-2}$). This is the canonical MoE setup from Fedus et al. (2022), but it is not representative of the best practices in modern MoE training circa 2024–2025. The paper does not compare against vanilla MoE variants that incorporate:

• Router z-loss (Zoph et al., 2022), which penalizes large logit magnitudes to prevent routing collapse and is standard in ST-MoE and many production systems.
• Auxiliary-loss-free balancing (Wang et al., 2024), which uses expert-level biases updated via gradient descent on the load-imbalance signal rather than an explicit auxiliary loss, and has been shown to achieve better load balancing with less interference with the primary objective.
• Shared experts alongside routed experts (DeepSeekMoE-style; Dai et al., 2024), where a small number of "always-on" shared experts capture common transformations and the routed experts specialize on the residual—this is a different form of sharing that operates within a layer rather than across depth.
• Expert choice routing (Zhou et al., 2022), where experts select the top tokens rather than tokens selecting the top experts, which enforces perfect load balancing by construction.

The consequence. UNIPOOL's gains—up to 0.0386 validation loss reduction—are measured against a baseline that may be substantially suboptimal relative to what a practitioner would actually deploy. If a better vanilla MoE baseline (e.g., with auxiliary-loss-free balancing and z-loss) closes a significant fraction of the gap, then UNIPOOL's practical value proposition weakens. The paper's claim that UNIPOOL improves over "vanilla MoE" is true as stated, but "vanilla MoE" in this context is a 2022-vintage baseline, not the 2024 state of the art.

The ablation study in Table 5 provides partial evidence that the gap is not trivially closed: the sigmoid-gated aux-free baseline (1.9239 at 182M) improves over standard vanilla MoE (1.9317) by only −0.0078, far less than UNIPOOL's −0.0288. This suggests that simply improving load balancing in the per-layer setting does not capture UNIPOOL's benefits. However, this tests only one specific alternative (sigmoid gating without auxiliary loss), not the full suite of modern MoE stabilization techniques. Auxiliary-loss-free balancing with expert-level biases (Wang et al., 2024) is a different mechanism from sigmoid gating and might interact differently with per-layer expert ownership.

A more subtle concern: UNIPOOL's NormRouter and pool-level auxiliary loss are themselves modern stabilization techniques applied to the shared-pool architecture. The comparison of UNIPOOL (shared pool + NormRouter + pool aux loss) against vanilla MoE (private experts + softmax + per-layer aux loss) conflates two differences: the architectural change (shared vs. private) and the stabilization mechanisms (NormRouter vs. softmax, pool-level vs. per-layer aux loss). The paper's NormRouter-only ablation on vanilla MoE (Table 5: 1.9375, +0.0058 vs. softmax vanilla MoE) shows that NormRouter alone does not improve the layer-private baseline—it slightly degrades it—which supports the interpretation that the architectural change is the primary driver of gains. But this does not answer the question: would a vanilla MoE with better stabilization (auxiliary-loss-free balancing, z-loss) close the gap against UNIPOOL, even if NormRouter alone does not?

What evidence exists in the paper. Table 5 includes three ablations that partially address this: (1) vanilla MoE with NormRouter (1.9375, slightly worse than softmax vanilla MoE), (2) vanilla MoE with sigmoid gating and no auxiliary loss (1.9239, modest improvement), and (3) vanilla MoE with softmax (1.9317, the standard baseline). None of these incorporate the specific modern techniques listed above. The granularity sweep (Figure 2b) shows UNIPOOL maintains its advantage under 16E/top-2 and 32E/top-4 configurations, but these change the expert granularity, not the routing or balancing mechanism. The paper does not cite or compare against auxiliary-loss-free balancing, DeepSeekMoE-style shared experts, or z-loss regularization.

Mitigation status. Partially addressed by the NormRouter and sigmoid aux-free ablations, which show that better routing or balancing within the per-layer paradigm does not match UNIPOOL's gains. However, these are weak baselines compared to the known state of the art. The paper's strongest defense is the routing-randomization result (Table 4): even if a better per-layer MoE baseline achieves lower absolute loss, the redundancy probe shows that per-layer expert ownership structurally produces interchangeable experts at depth, which UNIPOOL eliminates. A properly tuned vanilla MoE might close the validation loss gap but would still exhibit the 1.3–1.5 point randomization cost, indicating that the redundancy problem is architectural rather than optimization-dependent. This argument is theoretically compelling but empirically untested against a strong baseline.

---

Hard Problems in Expert Specialization (Capacity-Limited Shared Pools, Layer-Routing Interference, and the Unexplained G=2 Anomaly) Remain Unresolved, and the Paper Provides No Diagnostic Tools for Detecting Them During Training

The assumption or constraint. The paper focuses on demonstrating that UNIPOOL works across five scales, and it reports aggregate metrics (validation loss, downstream accuracy) that summarize average-case behavior. It does not deeply investigate failure modes of the shared-pool architecture—conditions under which sharing is counterproductive or unstable—beyond the broad observation that "an overly small pool can introduce interference among depth-specific demands" (Section 6.2). Three specific unresolved issues emerge from the paper's own results:

The optimal pool size tradeoff is empirically observed but not mechanistically characterized. Figure 2(a) shows that reducing the pool too far eventually degrades performance (the $M = 48$ point at 182M is above the vanilla baseline; the $M = 64$ point at 469M is roughly at parity). The paper attributes this to "interference among depth-specific demands" when the pool is too small, but this is a post-hoc interpretation, not a demonstrated mechanism. There is no measurement of what kind of interference occurs—do different layers route the same expert to process tokens that require conflicting transformations? Do the expert weights oscillate between satisfying deep-layer and shallow-layer demands? The paper provides no gradient analysis, expert weight trajectory analysis, or per-layer loss decomposition that would illuminate the failure mode.

The G=2 anomaly in the sharing-scope ablation is unexplained. Table 5 shows that grouping the 12 layers into 2 pools of 6 layers each (G=2, validation loss 1.9213) performs worse than grouping into 4 pools of 3 layers each (G=4, 1.9099), despite having broader sharing within each group. The monotonic improvement from G=12 (vanilla) through G=6 through G=4 breaks at G=2, before recovering at G=1 (full UNIPOOL, 1.9029). The paper mentions this result without explanation, and Appendix Figure 4d shows the G=2 curve tracking near vanilla MoE for much of training. Is this a noise artifact (single run)? Does it indicate that forcing certain layers to share (e.g., layers 1–6 in one pool, 7–12 in another) creates a boundary where routing preferences change sharply, causing interference? Without investigation, a practitioner trying to deploy grouped sharing as a middle ground between vanilla MoE and full UNIPOOL has no guidance on how to choose group boundaries.

There is no diagnostic for detecting when the shared pool is capacity-limited. The paper's main results use $M = 8L$, matching the total expert count of vanilla MoE. The reduced-pool experiments (Figure 2a) find that smaller pools can match or outperform at the tested scales, but the optimal pool size varies with depth in ways that are empirically observed but not predicted. A practitioner scaling UNIPOOL to a new architecture ($L$ layers, hidden size $H$) has no formula or diagnostic for choosing $M$. Should $M$ scale with $\sqrt{L}$? With $\log L$? Is there a measurable signal during training—routing entropy collapse, expert gradient norm divergence, cross-layer expert usage overlap—that indicates the pool is too small?

The consequence. These unresolved issues mean that UNIPOOL is demonstrated to work under specific, well-tuned conditions but is not a robust, self-diagnosing architecture. A practitioner who deploys UNIPOOL with an inappropriately small pool, or groups layers suboptimally, or encounters a depth-dependent interference pattern, has no tools from this paper to detect or correct the problem during training—they would only discover it at evaluation time, after spending the training budget. The paper's contribution is an existence proof that shared-pool MoE can work and can be more efficient; it does not provide the engineering knowledge needed to make it work reliably at new scales or under new conditions.

What evidence exists in the paper. The reduced-pool sweep (Figure 2a) shows the existence of an optimal pool size at each scale and that performance degrades on both sides (too large and too small). The G=2 anomaly (Table 5) shows that intermediate sharing scopes can underperform. The expert utilization heatmap (Figure 3) shows that pool-level balancing achieves uniform aggregate usage, but this is a necessary condition for the pool to work, not a sufficient diagnostic for whether the pool size is appropriate. There is no analysis of expert weight trajectories, gradient conflicts, or per-layer loss decomposition that would illuminate the failure modes.

Mitigation status. Not addressed. The paper treats these as observations within the experimental results rather than as open problems requiring investigation. The sharing-scope ablation is presented as evidence that broader sharing is better (G=1 > G=4 > G=6 > G=12), with the G=2 result noted but not interpreted. The pool-size sweep is presented as evidence for sublinear scaling, with the interference regime acknowledged in passing. For a paper that proposes a new architectural paradigm for MoE, the absence of even a preliminary diagnostic framework for when the paradigm is working versus failing is a gap that limits the method's deployability.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a structural reframing of expert capacity allocation in Mixture-of-Experts architectures—from a per-layer private resource to a global, reusable budget—and demonstrates that this reframing consistently improves both quality and parameter efficiency across scales. The magnitude of change is architectural rather than algorithmic: UNIPOOL does not propose a new training objective, a new routing formula, or a new expert design; it proposes changing the fundamental ownership structure of expert parameters in a way that has been assumed fixed since the inception of sparse MoE.

What makes this a meaningful shift rather than an incremental parameter-sharing trick is the co-design logic. The paper demonstrates that changing the ownership structure forces changes to the load-balancing objective (pool-level auxiliary loss) and the routing mechanism (NormRouter), and that mismatching any component actively degrades performance (the shared pool with per-layer aux loss worsens validation loss by 0.0163 at 182M). This is not "you can share parameters to save memory"—which ALBERT and Universal Transformers established years ago—it is "sharing changes the training dynamics of expert specialization in ways that require coordinated architectural redesign, and doing that redesign correctly converts depth-induced redundancy into genuine functional specialization." The routing-randomization evidence (4.1-point accuracy drop on UNIPOOL vs. 1.3–1.5 on vanilla MoE under matched randomization protocols) is critical here: it demonstrates that the mechanism of improvement is not just parameter efficiency but a qualitative change in what the experts are—less interchangeable, more load-bearing, more specialized.

This work reconciles a tension in the MoE literature between two lines of evidence. On one hand, scaling work (Switch Transformer, GShard, Mixtral, DeepSeek) shows that adding more experts per layer improves performance, suggesting that expert diversity is valuable. On the other hand, redundancy analyses show that within-layer experts are highly similar (cosine similarity >0.9, pruning tolerance to ~8% quality loss, routing randomization costs of 1.0–1.6 points), suggesting that much of that diversity is illusory. UNIPOOL resolves this tension by showing that the redundancy is architecturally induced by per-layer private ownership: experts are diverse in principle but cannot achieve that diversity under thin per-layer gradient signals. When the same total number of experts is shared across depth—so each expert receives gradient signal from up to $L$ layers rather than one—the diversity becomes real (routing randomization cost jumps to 4.1 points), and the total number of experts needed to achieve a given quality level decreases (41.6–66.7% of the vanilla budget suffices at the tested scales). The apparent contradiction between "experts are useful" and "experts are redundant" is resolved by recognizing that the redundancy is a training artifact of per-layer ownership, not an inherent property of sparse expert routing.

This work also redirects the conversation around MoE scaling. The dominant scaling paradigm—train larger models with more layers and more experts per layer, both growing linearly—is implicitly questioned by the reduced-pool results. If expert parameters can grow sublinearly with depth while maintaining or improving quality, then the conventional MoE scaling recipe (proportional scaling of layers and expert count) leaves efficiency on the table. The paper establishes pool size as a new scaling hyperparameter decoupled from depth, analogous to how Chinchilla scaling laws established that data and model size should be scaled jointly rather than model size alone. The implication is that future MoE scaling studies should not assume $E \times L$ expert parameters as a given; they should treat the depth-pool allocation as an optimization problem in its own right. The 830M vs. 978M comparison—where a deeper model with fewer total parameters achieves lower validation loss—is a concrete data point supporting this reframing.

The work makes certain research directions more attractive. Post-hoc expert pruning and similar-expert re-routing—which treat redundancy as something to remove after training—now appear as symptomatic treatments of a problem that can be architecturally prevented. Research effort might shift from "how do we compress a trained MoE?" to "how do we train an MoE that doesn't develop the redundancy in the first place?" Similarly, work on improved auxiliary losses for MoE (auxiliary-loss-free balancing, expert-level bias adjustment) now has a new design dimension to consider: the domain over which load balancing is defined (per-layer vs. pool-level vs. hierarchical). UNIPOOL's pool-level loss is a specific instance of a more general principle—load balancing should match the parameter ownership domain—that could inform future balancing mechanisms even for non-global sharing schemes.

Conversely, this work makes certain directions less attractive in isolation. NormRouter, as applied in layer-private MoE, slightly degrades performance (Table 5: 1.9375 vs. 1.9317). This suggests that developing better routers for the standard per-layer setting—without changing the ownership structure—may have limited upside, because the bottleneck is not the routing formula but the expert set that the router selects from. If the experts are near-substitutes (as the randomization probe shows), then no router can extract meaningful specialization from them; the architectural structure must change first. This does not mean router research is pointless, but it suggests that architectural co-design (router + ownership structure + load balancing) is more promising than treating the router as an independent component to optimize.

Follow-Up Research This Work Enables

Training UNIPOOL to compute-optimal convergence (100B+ tokens) at one representative scale, measuring whether the validation loss gap widens, narrows, or reverses relative to vanilla MoE. The paper's training curves (Appendix Figure 4) show loss still declining at 60K steps (~30B tokens) for all models, making the reported comparisons snapshots at an arbitrary training budget. The production-model routing probes (Table 1) suggest that redundancy persists in vanilla MoE even after trillions of training tokens, so the problem UNIPOOL addresses is structural, but we do not know whether the advantage of addressing it structurally is sustained, amplified, or diminished with longer training. A simple experiment: take the 469M scale (where the gap is −0.0346 at 30B tokens) and train both vanilla MoE and UNIPOOL to 100B or 300B tokens on the same data, evaluating validation loss and routing-randomization cost at regular intervals. If the gap widens, the concentrated gradient signal in UNIPOOL genuinely produces better-trained experts over time; if the gap narrows, vanilla MoE's experts eventually specialize under sufficient gradient budget, and UNIPOOL's advantage is primarily an acceleration phenomenon rather than a capacity phenomenon. This distinction has direct implications for whether UNIPOOL should be adopted in large-scale training runs where budget is measured in trillions of tokens.

A joint sweep over depth $L$ and pool size $M$ at fixed total parameter budgets, fitting a functional relationship between depth, pool size, and validation loss analogous to Chinchilla-style scaling laws for shared-pool MoE. The paper demonstrates that deeper models tolerate smaller pool fractions (66.7% at $L=12$, 50% at $L=24$ and $L=36$, 41.6% at $L=48$) but tests only 2–3 pool sizes per scale. A systematic sweep—e.g., train models at $L \in \{12, 24, 36, 48, 72\}$ with pool sizes $M$ ranging from $0.25 \times 8L$ to $2 \times 8L$, all at matched active FLOPs, measuring validation loss at a fixed token budget—could reveal whether the optimal pool size follows a power-law relationship with depth ($M \propto L^\alpha$ for some $\alpha < 1$), or whether there is a sharp transition at some depth beyond which sublinear scaling breaks down. The 830M vs. 978M comparison hints that depth matters more than raw parameter count, but a controlled sweep would isolate the effect. Such scaling laws would be directly useful to practitioners choosing architectures for large-scale MoE training: given a parameter budget and target depth, what pool size should be allocated?

Measuring expert weight specialization trajectories under shared-pool vs. layer-private training to identify when and why experts diverge or collapse into redundancy. The paper's routing-randomization probe (Section 6.1) provides a behavioral measure of expert specialization at the final checkpoint, but there is no analysis of the training dynamics that produce this outcome. A follow-up could track per-expert weight cosine similarity to other experts in the same pool (or same layer, for vanilla MoE) across training steps, along with per-expert gradient norm and gradient cosine similarity (measuring whether different layers' gradient signals to the same shared expert conflict or align). The hypothesis: in vanilla MoE, deep-layer expert weights converge to high pairwise similarity because each expert sees only $1/L$ of the total gradient signal, and the per-layer loss landscape has many near-equivalent local minima for expert weights; in UNIPOOL, the concentrated gradient signal from $L$ layers creates a sharper loss landscape where redundant experts are quickly driven apart. Visualization of expert weight trajectories (e.g., PCA projections over training steps) would make this mechanism interpretable and could inform better pool initialization or expert re-initialization strategies for UNIPOOL.

Stress-testing UNIPOOL with aggressive pool reduction, combined with a diagnostic for detecting capacity-limited shared pools during training. The reduced-pool experiments in Figure 2(a) identify an optimal pool size post-hoc, but provide no signal that a practitioner could monitor during training to determine whether their pool is too small. A targeted stress-test: take the 182M model and train UNIPOOL with $M \in \{96, 64, 48, 32, 24, 16\}$ (the paper already has $M=96$, $64$, and $48$; extend to smaller pools), measuring not just final validation loss but also per-expert gradient norm variance, cross-layer expert-usage overlap (Jaccard similarity between layer routing distributions), and the entropy of the per-layer routing distribution. The goal is to identify a measurable correlate of the "interference among depth-specific demands" that the paper hypothesizes occurs when pools are too small. For instance, if cross-layer expert-usage overlap increases sharply below a certain pool size threshold, that threshold might be detectable early in training (well before convergence) from routing statistics, providing an online diagnostic for pool sizing. This is especially important for the unexamined G=2 anomaly (where two groups of 6 layers sharing 48 experts each underperforms four groups of 3 layers sharing 24 experts each)—understanding which layers interfere when forced to share would inform grouping strategies for intermediate sharing scopes.

Combining UNIPOOL's global shared pool with DeepSeekMoE-style shared experts, testing whether per-layer always-on experts and a cross-layer routed pool are complementary or redundant. DeepSeek-V2 and V3 (Dai et al., 2024; DeepSeek-AI, 2024) use a hybrid architecture: a small number of "shared experts" that process every token at every layer (always active, no routing), plus a larger number of "routed experts" selected by the per-layer router. The shared experts capture common transformations needed across all tokens; the routed experts specialize on the residual. This is conceptually orthogonal to UNIPOOL's cross-layer sharing: DeepSeek shares experts within a layer (all tokens see the shared experts) while UNIPOOL shares experts across layers (all layers access the same pool). A natural combination: each layer has a small set of private always-on shared experts (as in DeepSeek) plus access to a global routed pool shared across all layers (as in UNIPOOL). The hypothesis is that the private shared experts handle depth-invariant common transformations (reducing pressure on the global pool to represent them), while the global routed pool handles depth-specific specializations that benefit from cross-layer gradient signal. Testing this at the 469M or 650M scale—comparing validation loss against pure UNIPOOL, pure DeepSeek-style, and the standard vanilla MoE baseline—would clarify whether these two forms of sharing are additive or whether the global pool already subsumes what per-layer shared experts provide. This is a natural integration experiment given that DeepSeek-style shared experts are a proven production technique and UNIPOOL targets a different dimension of sharing.

Applying UNIPOOL to a code generation model trained on a code-heavy corpus, measuring whether cross-layer expert reuse patterns differ between natural language and structured code, and whether the optimal pool size shifts. All UNIPOOL experiments use the Pile dataset, which is predominantly natural language. Code has different statistical properties—more repetitive syntactic structures, longer-range dependencies, token distributions dominated by keywords and identifiers—that could change how cross-layer expert reuse emerges. A simple replication: take the LLaMA architecture at 182M or 469M scale, train on The Stack or a similar code corpus for an equivalent token budget, and compare UNIPOOL vs. vanilla MoE on code generation benchmarks (HumanEval, MBPP) in addition to validation loss. The specific question: does the optimal pool fraction (smallest winning $M$ as a fraction of $8L$) differ between natural language and code? If code's repetitive structure means fewer distinct expert transformations are needed, the optimal pool might be even smaller than the 41.6–66.7% range observed on the Pile. Conversely, if code's syntactic variety demands more distinct transformations, the optimal pool might be larger. This is a domain-generalization test that would determine whether UNIPOOL's efficiency gains are universal or distribution-dependent—a critical question for practitioners training MoE models on code, math, or multilingual corpora.

Practical Applications and Downstream Use Cases

Reducing stored parameters for on-device or memory-constrained MoE deployment. The paper's reduced-pool results show that UNIPOOL matches vanilla MoE quality using only 41.6–66.7% of the expert parameters at the tested scales. At the 830M scale, this means 5.081B stored parameters for UNIPOOL vs. the matched vanilla MoE baseline with proportionally more expert parameters (the paper reports 5.081B for UNIPOOL but does not report the vanilla MoE stored parameter count separately; based on the architecture, vanilla MoE at 48 layers with 8 private experts per layer stores $48 \times 8 = 384$ expert FFNs vs. UNIPOOL's $M = 384$ in the matched setting, so the matched comparison has identical stored parameters). The savings emerge when using a reduced pool: at 830M with $M = 160$ (the smallest winning pool, 41.6% of matched), the stored expert parameters are reduced by 58.4% relative to the vanilla allocation, while validation loss improves by −0.013. For a deployment scenario where model weights must fit in limited device memory (edge devices, consumer GPUs, mobile inference), this is a direct storage-to-quality tradeoff: UNIPOOL with a 41.6% pool provides better quality than vanilla MoE at full expert capacity, using substantially less memory. The paper does not provide memory measurements, but the parameter count reduction is architecturally guaranteed: smaller $M$ means proportionally fewer expert weight matrices stored. The caveat is that the router weight matrices per layer are larger in UNIPOOL ($M \times d$ instead of $E \times d$), but since $d$ (hidden size) is typically 1024–1536, the router parameters are a tiny fraction of the expert FFN parameters (which scale as $M \times d \times d_{\text{ffn}}$, where $d_{\text{ffn}} = 4d$ for SwiGLU, so expert parameters dominate by a factor of ~$4d$). This use case is most directly supported by the paper's data, though the paper acknowledges (Appendix A) that throughput measurements are missing.

More efficient expert-parameter allocation in large-scale MoE training, where depth and expert count are architectural decisions made before training begins. The paper's central reframing—pool size as a depth-scaling hyperparameter decoupled from layer count—is directly actionable for teams designing MoE architectures. The standard design process for a new MoE model involves choosing the number of layers $L$, the number of experts per layer $E$, and the active experts per token $k$, with the implicit constraint that total expert parameters scale as $E \times L$. UNIPOOL removes that constraint: the architect can choose $L$ and $M$ independently, and the paper's data suggests that $M$ can grow sublinearly with $L$ ($M = 8L^{0.5}$ to $8L^{0.7}$ would be consistent with the observed winning fractions of 66.7% at $L=12$, 50% at $L=24$, 41.6% at $L=48$). For a hypothetical large-scale training run targeting a trillion-parameter MoE model, this could mean allocating fewer total parameters to experts and more to depth (more layers, each reusing the shared pool), or keeping total parameters fixed and increasing depth with a smaller per-layer-equivalent expert budget. The 830M vs. 978M comparison—where the deeper 830M model achieves lower validation loss than the wider 978M model despite fewer total parameters—provides a concrete data point for this allocation strategy. The practical decision this enables: "we have a compute budget for training; should we spend it on a model with 96 layers and a shared pool of 384 experts, or 48 layers with 8 private experts per layer (also 384 total experts)?" The paper's evidence favors the deeper shared-pool configuration, though this remains to be validated at larger scales.

Diagnostic tool for evaluating expert utilization in existing MoE models, independent of whether UNIPOOL is adopted. The routing-randomization probe introduced in Section 3 (replacing the learned router in a single deep-half layer with uniform random assignment and measuring the downstream accuracy drop) is a lightweight, architecture-agnostic method for assessing expert specialization in any trained MoE model. It requires no retraining, no weight analysis, and no modification to the model architecture—only the ability to intervene on routing decisions during inference, which is straightforward in most MoE implementations. The paper demonstrates the probe on three production models (Qwen1.5-MoE, DeepSeek-V2-Lite, Qwen3-30B-A3B) and on its own trained models, providing a reference range: a drop of 1.0–1.6 points indicates high expert substitutability (redundancy), while a drop of 4.1 points indicates specialization. A team deploying or fine-tuning an MoE model could run this probe as part of their model evaluation pipeline to determine whether the model's expert budget is being used efficiently. If deep-layer randomization costs less than 2 points on their target benchmarks, the model has substantial expert redundancy and would benefit from either post-hoc pruning, architectural sharing (UNIPOOL), or further training with stronger specialization incentives. If the cost is 4+ points, the experts are load-bearing and pruning or sharing would be riskier. The probe is specifically practical because it requires only inference (no gradient computation) and can be averaged over multiple layers to produce a single summary statistic. The paper provides per-task breakdowns in Appendix Tables 10 and 11, which practitioners can compare against their own task distributions.

When to Prefer This Method

The paper explicitly positions UNIPOOL against vanilla per-layer MoE with top-1 softmax routing and per-layer Switch auxiliary loss. The comparison is architectural rather than algorithmic, and the decision rule is not presented as a universal binary but as a depth-dependent tradeoff informed by the paper's scaling results. Based on the evidence:

Prefer UNIPOOL over vanilla per-layer MoE when:

• The architecture is deep (24+ layers) and expert parameters constitute a large fraction of total parameters. The paper's data shows that the winning pool fraction decreases monotonically with depth (66.7% at 12 layers → 41.6% at 48 layers) and that the largest validation loss improvement (−0.0386) occurs at the deepest tested architecture (830M, 48 layers). Deeper models create more reuse opportunities for the shared pool, and the per-layer gradient dilution problem that causes redundancy is more severe with more layers. The 978M model (24 layers but wider) showed a smaller gain (−0.0172), consistent with the interpretation that depth drives the benefit. Concrete threshold: at $L \geq 24$, the paper's data supports UNIPOOL providing a meaningful improvement; at $L=12$, the improvement is smaller (−0.0288 at 182M) but still positive.
• The training compute budget allows for the minor overhead of pool-level auxiliary loss computation and NormRouter routing, and no expert-parallel distribution is required. The paper's experiments use expert parallelism of 1, meaning all experts reside on a single device. For models that require expert parallelism to fit in memory (which is common at larger scales), the distributed communication patterns for a global shared pool are untested and may introduce overhead that the paper does not measure. If the model fits on-device without expert parallelism (which is true up to the paper's tested scales of ~5B stored parameters), the architectural change is a pure parameter-reference modification with the same total expert FFN count in the matched setting—no throughput penalty beyond the router overhead, which the paper asserts is small but does not measure. If expert parallelism is required, a practitioner should benchmark throughput before adopting.
• The training horizon and data distribution are similar to the paper's setup (30B+ tokens of diverse English text, LLaMA-style backbone). The paper does not provide evidence for UNIPOOL on other data distributions (code, multilingual, domain-specific corpora) or on non-LLaMA architectures. The production-model routing probes suggest the redundancy problem is cross-family, but whether UNIPOOL is the right solution for Qwen or DeepSeek architectures is untested. A practitioner using a non-LLaMA backbone should first validate that their architecture exhibits the deep-layer redundancy pattern (via the routing-randomization probe) before committing to UNIPOOL.
• The deployment scenario benefits from reduced stored parameters (edge deployment, memory-constrained inference, multi-tenant serving where model copies must fit in VRAM). The reduced-pool results provide a concrete path to smaller models with matched or better quality: at 48 layers, a pool of 160 experts (41.6% of the vanilla 384-expert budget) outperforms the layer-private baseline while using substantially less storage. The paper does not provide inference latency measurements, so this recommendation applies to storage-constrained rather than latency-constrained deployments until throughput benchmarks are available.

Prefer vanilla per-layer MoE over UNIPOOL when:

• The architecture is shallow (few layers, or layers with very different functional roles where cross-layer sharing is unlikely to help). The paper does not test models with $L < 12$, but the trend in winning pool fraction (66.7% at $L=12$, 50% at $L=24$, 41.6% at $L=48$) suggests that the benefit of sharing decreases as depth decreases. Below some threshold (possibly $L=6$ or $L=8$), the overhead of the pool-level auxiliary loss and NormRouter routing may outweigh the benefit of sharing, and the standard per-layer MoE setup is simpler and well-understood.
• The model uses expert parallelism across many devices, and the communication patterns for a global shared pool have not been benchmarked. The paper explicitly defers this to future work (Appendix A). A practitioner training at scales where expert parallelism is necessary (e.g., hundreds of experts distributed across dozens of GPUs) should not adopt UNIPOOL without first profiling the all-to-all communication cost of routing all layers into a single shared pool versus the independent per-layer all-to-all in vanilla MoE. The risk is that global pool routing creates cross-layer synchronization dependencies that prevent overlapping communication with computation, eroding throughput.
• Existing infrastructure is tightly coupled to per-layer expert ownership (custom kernels for per-layer expert dispatch, load-balancing monitors that assume per-layer utilization statistics, checkpoint formats that assume separate expert weights per layer). The paper's implementation in Megatron-LM uses reference sharing (the same expert module object is reused across layers), which is architecturally clean but may conflict with systems that expect layer-specific expert parameters. Adopting UNIPOOL in a production training pipeline may require non-trivial engineering work beyond the research implementation.
• The training budget is very small (fewer than 10B tokens for models in the tested size range). The paper's training curves (Appendix Figure 4) show that the UNIPOOL vs. vanilla MoE gap emerges after the initial warmup phase and widens through training. At very small token budgets, the gap may not have fully materialized, and the added complexity of the pool-level auxiliary loss and NormRouter may not be justified by the endpoint gain. The paper provides no data below 30B tokens.