Coupling Experts and Routers in Mixture-of-Experts via an Auxiliary Loss

ArXiv: 2512.23447

🎯 Pitch

Standard MoE routers learn expert capabilities only through indirect gradients, leading to misrouted tokens that sabotage specialization—yet simply forcing each expert’s own router embedding to trigger the strongest activation from that expert dramatically improves performance across model scales, all while adding less than 1% training overhead.

1. Executive Summary

This paper proposes expert-router coupling (ERC) loss, a lightweight auxiliary loss that tightly aligns router decisions with expert capabilities in Mixture-of-Experts models. Training MoE language models from 3B to 15B parameters on trillions of tokens from the dolma-v1.5-sample dataset, the authors introduce a clustering-based approach where each expert's router embedding is perturbed with bounded noise to serve as a proxy for its assigned tokens, then fed through all experts to enforce that each expert responds most strongly to its own proxy and that each proxy activates its corresponding expert most strongly (a form of contrastive coupling without denser activation). The ERC loss improves downstream task accuracy across multiple benchmarks while incurring only 0.2–0.8% training overhead and no inference overhead, establishing that weak expert-router coupling — not imperfect router orthogonality — is the critical bottleneck limiting MoE performance, and that this coupling can be strengthened efficiently only when the auxiliary loss's complexity is independent of token count rather than scaling with it, as in prior dense-activation methods like Autonomy-of-Experts.

2. Context and Motivation

The Core Problem: Routers Don't Know Their Experts

In a standard Mixture-of-Experts model, the router is a linear classifier that takes a token $x \in \mathbb{R}^d$ and outputs a weight vector $w = \text{softmax}(xR^\top)$ indicating which experts should process that token. The top-$K$ experts are selected, and their outputs are combined. This architecture implicitly assumes that the router's parameter matrix $R \in \mathbb{R}^{n \times d}$ contains an accurate representation of what each expert is good at — row $R[i]$ should encode the kind of tokens that expert $i$ handles well, so that when a token's inner product with $R[i]$ is high, expert $i$ is the right choice.

The problem is that nothing enforces this assumption. The router is trained only through the final task loss — the cross-entropy of the language modeling objective. The gradient signal flows back through the selected experts to the router, but this is indirect feedback. The router never directly observes what each expert's parameters actually look like or what the expert would do with tokens it didn't receive. As the paper puts it:

"Without direct access to expert parameters (and therefore their true capabilities), routers resort to trial-and-error learning of routing strategies, often resulting in misrouted tokens whose gradients interfere with expert specialization."

This is a representation alignment problem: the router's internal model of expert capabilities (embedded in $R$) can drift away from the experts' actual capabilities (embedded in $W_g$, $W_p$, $W_o$). When this decoupling occurs, two things go wrong:

1. Expert specialization suffers. Tokens that should go to expert $i$ may get routed elsewhere, and expert $i$ receives tokens it isn't best suited for. The expert never gets a clean, consistent signal about what it should specialize in.
2. Router decisions become suboptimal. Even if an expert has learned to handle a certain type of token well, the router might not know this, so those tokens never reach that expert.

This problem is structural, not incidental. It follows directly from the MoE design decision to keep the router as a separate linear layer that never inspects expert parameters. The paper's post-hoc analysis (Table 2 in Appendix A) makes the severity concrete: evaluating a trained vanilla MoE model, achieving zero ERC loss across all layers requires setting $\alpha = 5$. This means the model's natural state has off-diagonal activation norms $M[i, j]$ that are up to 5 times larger than allowed under even the loosest coupling constraint ($\alpha = 1$). The router and experts are significantly decoupled.

Why This Matters: The Scaling Implications

This problem grows in importance as models scale. Modern MoE architectures push toward hundreds or thousands of experts with very low activation rates ($K \ll n$, often $K = 8$ out of $n = 256$). In such high-sparsity regimes, the cost of a routing mistake is severe: when only 3% of experts are activated per token, misrouting a token to the wrong expert means the token gets processed by a specialist that may not have the right capabilities, while the correct expert sits idle. There is no redundancy to absorb routing errors.

Moreover, the router receives gradients only for the experts it selects. If expert $i$ is never selected for tokens of type $\mathcal{T}$ (because the router never routes them there), expert $i$ never gets training signal on $\mathcal{T}$, so it never develops capabilities for $\mathcal{T}$, so the router continues to have no reason to route $\mathcal{T}$ to expert $i$. This is a self-reinforcing cycle of decoupling that the standard training objective cannot break.

The paper frames this as a fundamental limitation:

"Traditional MoEs offer no explicit constraints to guarantee [that the router possesses accurate representations of expert capabilities]."

This isn't just a theoretical concern — it directly limits model quality. If routers don't know what their experts can do, the entire "mixture" concept degenerates: experts become less specialized, the model effectively wastes its increased parameter count, and performance converges toward that of a much smaller dense model.

Prior Approaches and Their Shortcomings

The paper identifies three categories of prior work that attempt to address aspects of this problem, each with significant limitations.

Dense Activation: Autonomy-of-Experts (AoE)

AoE [25] takes the most direct approach: eliminate the separate router entirely. Instead, every token is fed through a low-rank projection $W^i_{\text{down}} \in \mathbb{R}^{d \times r}$ for all $n$ experts, producing $n$ intermediate activation norms. These norms become the routing signal: the top-$K$ experts by activation norm are selected to continue processing, and the others are terminated early. The router is now embodied in the experts themselves — there is no possible decoupling because the routing decision is computed from the same parameters that process the token.

This approach works. The paper acknowledges AoE as "a competitive yet more computationally expensive MoE variant" and shows in Figure 3(a) that AoE achieves the best downstream accuracy among all compared methods at 3B scale. However, AoE has a fatal practical limitation:

"The computational overhead of AoE scales with the number of tokens during both training and inference."

Specifically, AoE introduces $2T(n - K)dr$ additional FLOPs per layer, where $T$ is the number of tokens (often millions per batch), $n$ is the number of experts, $K$ is the number selected, $d$ is the hidden dimension, and $r$ is the factorization rank. This scales linearly with $T$, making it prohibitive for large-scale training. The paper reports that AoE requires 1.6× more training hours and 1.3× higher memory than a vanilla MoE at 3B scale. When attempting to scale to 15B parameters with $n = 256$ experts, the AoE method "failed to train due to being overly costly."

A follow-up work, Union-of-Experts [46], reduced this overhead by using only a small subset of neurons per expert for routing, but this only mitigates the token-dependent cost — it doesn't eliminate it. As long as every token touches every expert (even partially), the computational principle contradicts the core motivation of MoE: sparse activation.

Competition-Based Routing (CompeteSMoE)

Pham et al. [30] proposed using experts' final output norms to supervise the router logits, essentially training the router to predict which experts would produce the strongest responses. This achieves coupling without the inference overhead of AoE. However, during training, the model is fully dense-activated — every token goes through every expert to collect the supervision signal. This contradicts the sparsity principle that makes MoE tractable for large-scale training. The paper includes it "only for background discussion, not as a baseline" for this reason.

Contrastive Learning on Experts or Routers Alone

Several works have applied contrastive losses to expert outputs to encourage specialization. Guo et al. [11] minimizes pairwise similarity between the outputs of selected experts, encouraging them to produce different features. Other works [13, 21] explicitly orthogonalize expert parameters. These approaches improve specialization but suffer from two problems:

1. Computational cost: computing pairwise similarities between $K$ selected experts per token scales as $\mathcal{O}(K^2 T)$, which becomes expensive at scale.
2. The decoupling problem remains: orthogonalizing experts or making their outputs different doesn't tell the router which expert specializes in what. The router is still trained through the indirect task loss, so it may not learn to route tokens to the right orthogonal expert.

Similarly, router orthogonalization [1] — enforcing $\hat{R}\hat{R}^\top = I$ to make router embeddings spread out and distinct — helps with load balancing (by encouraging experts to cover different parts of the input space) but doesn't couple routers to experts. The paper shows that router orthogonalization yields "only limited gains" (Figure 6(c)), because the baseline MoE already has nearly orthogonal routers (average absolute cosine similarity of 0.15, corresponding to angles of 81°–99°). The problem isn't that routers are too similar — it's that they don't encode expert capabilities accurately.

The critical insight from this result is that router orthogonality and expert-router coupling are distinct problems:

"Even if both routers and experts are orthogonalized, there is no guarantee that each $R[i]$ will be aligned with $W^i_g$. Therefore, the ERC loss cannot be reduced to contrastive techniques applied individually to routers or experts."

Load Balancing and Stability Losses

Existing auxiliary losses for MoE — load balancing loss [8, 31, 43] and z-loss [50] — address training stability and expert utilization, not expert-router coupling. Load balancing ensures tokens are distributed evenly across experts so all experts get used and receive gradients. This is necessary but orthogonal: an MoE can have perfectly balanced load and still have misrouted tokens because the router doesn't know which expert is best for which token. The paper confirms this by applying a consistent load balancing loss (weight 0.01) in all experiments and showing that the ERC loss provides additional gains beyond what balanced routing alone achieves.

How This Paper Positions Itself

The paper frames the ERC loss as occupying a previously empty point in the design space of MoE auxiliary losses. The key requirement is the following set of three design principles (Section 3):

"(1) Routers must be retained in MoE architectures to preserve routing efficiency. (2) An auxiliary loss that enables interaction between experts and routers can strengthen their coupling. (3) The loss must have complexity independent of the number of input tokens and must not introduce activation density beyond that of a vanilla MoE."

These principles directly address the failures of prior work:

• Principle 1 rules out AoE-like approaches that eliminate the separate router. The router provides the $O(TdK)$ sparse computation that makes MoE efficient; removing it and computing routing from expert activations necessarily costs $O(Tdn)$ at minimum.
• Principle 2 rules out purely router-side or purely expert-side regularization (orthogonalization, output contrast). Interaction is needed to create alignment between the two components.
• Principle 3 rules out CompeteSMoE-style dense activation during training and any loss whose computation scales with $T$.

The ERC loss satisfies all three by operating entirely on the router embedding matrix $R \in \mathbb{R}^{n \times d}$, not on input tokens. It creates $n^2$ "interactions" by perturbing each $R[i]$ with bounded noise and feeding all $n$ perturbed embeddings through all $n$ experts' $W_g$ matrices. This produces a fixed-size matrix $M \in \mathbb{R}^{n \times n}$ that captures expert-router coupling, and the loss is computed on this matrix alone. The computational cost is $2n^2 Dd$ FLOPs — independent of batch size $T$, independent of the number of tokens being processed. As the paper notes:

"a cost that is negligible in practical pre-training setups where $K$ is often in the millions"

The contrast with AoE's $2T(n - K)dr$ overhead is stark. With $T = 3 \times 10^6$, $n = 256$, $K = 8$, $d = 1536$, $D = 768$, and $r = 512$, the AoE overhead is roughly $2 \times 3 \times 10^6 \times 248 \times 1536 \times 512 \approx 1.2 \times 10^{15}$ FLOPs per layer versus the ERC loss's $2 \times 256^2 \times 768 \times 1536 \approx 1.5 \times 10^{11}$ — a difference of roughly four orders of magnitude. The ERC loss does in fixed cost what AoE does in cost proportional to batch size.

The Clustering Interpretation as Unifying Framework

The paper introduces a novel conceptual framing that motivates the entire approach: MoE routing as clustering. The router parameters $R$ are interpreted as $n$ cluster centers in $\mathbb{R}^d$. For an input token $x$, the inner product $xR[i]^\top$ measures similarity to cluster center $i$, and the top-$K$ similarity scores determine routing. Under this view:

• The set of tokens routed to expert $i$, denoted $X_i$, forms a cluster around $R[i]$.
• Expert $i$ should specialize in processing tokens from its cluster $X_i$.
• $R[i]$ serves as a compact proxy for the entire cluster $X_i$.

This interpretation is what makes the ERC loss possible without dense activation. Instead of feeding all $T$ tokens through all experts (to see which expert responds to which token), the paper feeds just the $n$ cluster centers through all $n$ experts. If $R[i]$ is a good proxy for $X_i$, then how expert $j$ responds to $R[i]$ approximates how expert $j$ would respond to tokens in $X_i$. The bounded random noise $\delta_i$ ensures $R[i]$ generalizes from a single point to a neighborhood, capturing variation within the cluster without crossing cluster boundaries.

This reframing from "tokens and experts" to "clusters and centers" is what enables the efficiency breakthrough. It reduces an $\mathcal{O}(Tn)$ interaction problem to an $\mathcal{O}(n^2)$ one.

Why Weak Coupling Is the Overlooked Bottleneck

The paper's ablation comparing ERC loss with router orthogonalization (Figure 6(c)) makes a compelling case that weak coupling, not poor router geometry, is the primary bottleneck. The baseline MoE already has nearly orthogonal router embeddings — they're well-separated in space and cover different directions. Yet performance improves substantially with the ERC loss. This implies that even with well-structured router embeddings, the model suffers from the decoupling problem: $R[i]$ might point in a perfectly distinctive direction, but that direction doesn't correspond to what expert $i$ actually does well.

This insight explains why prior work on router regularization and load balancing achieved limited gains — they optimized the geometry of routing without addressing whether the routing function actually matches expert capabilities. The ERC loss is the first auxiliary loss specifically designed to create this match, making it a novel contribution rather than an incremental improvement over existing regularization techniques.

The Unexplored Dimension: Controllable Specialization

Beyond the immediate practical benefit of improved performance, the paper positions the ERC loss as enabling a new kind of scientific investigation into MoE behavior. Prior work on expert specialization — whether through output orthogonalization [11, 13, 21], shared experts [6], or massive numbers of tiny experts [12, 29] — treated specialization as a binary property: more is better. The ERC loss, through its $\alpha$ parameter, provides continuous control over the degree of specialization, and through the noise bound $\epsilon$, provides quantitative tracking of specialization during training.

This is significant because the paper's own experiments reveal that the relationship between specialization and performance is non-monotonic. At $n = 64$, $\alpha = 1$ (the weakest coupling constraint that still exceeds vanilla MoE) achieves the best performance — stronger constraints with $\alpha < 1$ actually hurt. At $n = 256$, $\alpha = 0.5$ is optimal. This suggests a specialization-collaboration trade-off that depends on model architecture and has been unexplored in prior work, largely because there was no tool to systematically vary and measure specialization during pre-training. The ERC loss provides that tool, and the paper explicitly calls for future work to develop automated methods for finding the optimal $\alpha$ for a given model configuration.

3. Technical Approach

3.1 Reader Orientation

The paper proposes a lightweight auxiliary loss function, called the Expert-Router Coupling (ERC) loss, that is added to the standard language modeling objective during MoE pre-training. It solves the problem of routers and experts drifting apart — where the router's internal representation of what each expert can do becomes misaligned with the experts' actual learned capabilities — by creating a direct, computationally cheap interaction between router parameters and expert parameters at every training step, forcing them to stay mutually consistent.

3.2 Big-Picture Architecture (Diagram in Words)

The ERC loss operates on a standard MoE layer with the following components and information flow:

1. Router parameter matrix $R \in \mathbb{R}^{n \times d}$: Each row $R[i]$ is interpreted as a cluster center representing the set of tokens routed to expert $i$.

2. Noise injection module: Before computing the ERC loss, each $R[i]$ is multiplied element-wise by bounded random noise $\delta_i$ drawn from a uniform distribution, producing a perturbed proxy token $\tilde{R}[i]$. This perturbation is bounded so that $\tilde{R}[i]$ stays closer to $R[i]$ than to any other $R[j]$, ensuring it remains a valid representative of expert $i$'s token cluster.

3. Expert input projections $W_g^j \in \mathbb{R}^{d \times D}$ for all $j = 1, \ldots, n$: Each perturbed proxy $\tilde{R}[i]$ is matrix-multiplied by every expert's $W_g$ matrix, producing $n^2$ intermediate activation vectors of dimension $D$.

4. Activation norm matrix $M \in \mathbb{R}^{n \times n}$: The L2 norm of each activation vector is computed, where $M[i, j]$ is the activation norm of expert $j$ in response to proxy token $\tilde{R}[i]$. The diagonal $M[i, i]$ represents how strongly expert $i$ responds to its own proxy; off-diagonals $M[i, j]$ represent cross-expert responses.

5. ERC loss computation: Two constraints are enforced on $M$ for all $i \neq j$: (a) $M[i, j] < \alpha M[i, i]$ — each proxy activates its own expert more than any other expert; (b) $M[j, i] < \alpha M[i, i]$ — each expert is activated more by its own proxy than by any other proxy. Violations of these constraints incur a penalty, summed over all pairs and averaged.

6. Integration with standard MoE forward pass: The ERC loss is computed as a side computation during training only. The actual token routing still uses the clean (unperturbed) $R$ to compute router logits $xR^\top$, select top-$K$ experts, and produce the layer output. The ERC loss scalar is added to the total training loss. At inference time, the ERC loss is not computed — there is zero overhead.

3.3 Roadmap for the Deep Dive

• First, the clustering interpretation of MoE routing, which is the conceptual foundation justifying why perturbing router embeddings and measuring expert responses provides a valid signal for coupling.
• Second, the three-step procedure for computing the ERC loss — noise injection, activation norm computation, and constraint enforcement — because these are the core mechanism.
• Third, the mathematical formulation of the bounded noise, including the derivation of the maximum noise level $\epsilon_i$, since this ensures proxy tokens remain valid cluster representatives.
• Fourth, the ERC loss function itself, unpacking the two inequality constraints and how they translate to the final $\mathcal{L}_{\text{ERC}}$ expression, including the role of $\alpha$.
• Fifth, the efficiency analysis, comparing the ERC loss's $\mathcal{O}(n^2)$ fixed cost against AoE's $\mathcal{O}(Tn)$ token-dependent cost, to show why the clustering interpretation enables a fundamentally more efficient coupling mechanism.
• Sixth, the design choices and hyperparameter defaults ($\alpha = 1$, loss weight of 1, $\tilde{R}W_g$ as the activation source) and the reasoning behind them, since these are the practical knobs a practitioner needs to apply the method.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a method paper whose core idea is that expert-router coupling can be achieved efficiently by operating on router embeddings as cluster centers rather than on input tokens, reducing the coupling computation from a $T$-dependent cost to a fixed $n$-dependent cost.

The Clustering Interpretation of MoE Routing

The paper reframes MoE routing as an implicit clustering process. In a standard MoE layer, the router computes a weight for each expert as $xR[i]^\top$ — the inner product between the input token $x \in \mathbb{R}^d$ and the $i$-th row of the router parameter matrix $R \in \mathbb{R}^{n \times d}$. The authors observe that this is mathematically equivalent to measuring the similarity between $x$ and a set of $n$ cluster centers, where $R[i]$ is the center for cluster $i$.

Under this interpretation:

• The set of tokens $X_i = \{x \mid \text{expert } i \text{ is selected for } x\}$ forms a cluster around $R[i]$.
• The router's job is to assign each token to the nearest cluster centers (by inner product, which equals cosine similarity scaled by norms when all $R[i]$ have comparable norms — an assumption the paper verifies holds in practice).
• Expert $i$ should specialize in processing tokens from its cluster $X_i$.

This reframing is not just conceptual — it enables a computational strategy. If $R[i]$ is a good representative of the cluster $X_i$, then how expert $j$ responds to the single vector $R[i]$ provides an approximation of how expert $j$ would respond to a typical token in $X_i$. Instead of feeding all $T$ tokens through all $n$ experts (costing $\mathcal{O}(TnDd)$), the method feeds only $n$ cluster centers through all $n$ experts (costing $\mathcal{O}(n^2 Dd)$). The $T$-dependence is eliminated entirely.

The key insight is that this approximation is not heuristic — if the router is functioning correctly as a cluster-based router, then $R[i]$ genuinely represents the tokens in $X_i$, and the approximation is accurate. The ERC loss's job is precisely to make the router function as a correct cluster-based router by enforcing that expert responses to $R[i]$ are consistent with what the router's assignments imply.

To account for the fact that $R[i]$ is a single point while $X_i$ is a distribution of tokens, the method perturbs $R[i]$ with bounded random noise to create $\tilde{R}[i]$, simulating variation within the cluster without crossing into neighboring clusters. The noise bound is derived formally and computed dynamically per layer per training step.

Step 1: Generating Perturbed Proxy Tokens

For each expert $i$, the method creates a perturbed proxy token:

$\tilde{R}[i] = R[i] \odot \delta_i$

where $\odot$ denotes element-wise (Hadamard) multiplication, and $\delta_i \in \mathbb{R}^d$ is a random vector with each component drawn independently from a uniform distribution $\mathcal{U}(1 - \epsilon_i, 1 + \epsilon_i)$.

What this computes: element-wise multiplication of the router embedding $R[i]$ by a random scaling factor per dimension. Each dimension $k$ of $R[i]$ is multiplied by a factor between $1 - \epsilon_i$ and $1 + \epsilon_i$, producing a perturbed vector $\tilde{R}[i]$ that is a randomly scaled version of the original cluster center. This produces a point that lies near $R[i]$ but deviates from it by a controlled amount per dimension.

Why multiplicative noise rather than additive: additive noise would shift $R[i]$ by a fixed amount regardless of the magnitude of $R[i]$ in each dimension. Multiplicative noise scales the perturbation proportionally to the original value, which means dimensions where $R[i]$ has large magnitude (and thus has strong influence on routing) get larger perturbations, while near-zero dimensions (which do not affect routing) stay near zero. This preserves the directional structure of $R[i]$ while introducing variation in its magnitude along each axis.

Why bounded noise: the critical requirement is that $\tilde{R}[i]$ must remain in the cluster of expert $i$, meaning it must be closer to $R[i]$ than to any other cluster center $R[j]$ for $j \neq i$. The bound $\epsilon_i$ is derived to guarantee exactly this property. If unbounded noise were used, $\tilde{R}[i]$ could cross into another expert's cluster, violating the assumption that it represents tokens assigned to expert $i$ and making the coupling signal misleading.

Determining the Noise Bound $\epsilon_i$

The paper derives the maximum allowable $\epsilon_i$ from the geometry of the cluster centers. Let $j = \arg\min_{j^* \neq i} \|R[i] - R[j^*]\|$ be the nearest other cluster center. The requirement is that the perturbed point $\tilde{R}[i]$ must satisfy:

$\|\tilde{R}[i] - R[i]\| < \|\tilde{R}[i] - R[j]\|$

What this inequality means: the Euclidean distance from the perturbed point to its original center $R[i]$ must be strictly less than its distance to the nearest other center $R[j]$. This guarantees that if we were to assign $\tilde{R}[i]$ to a cluster by nearest-neighbor, it would still be assigned to cluster $i$.

Through algebraic manipulation (expanding squared norms, applying the Cauchy-Schwarz inequality, and solving for the worst-case noise configuration), the paper derives the sufficient condition:

$\epsilon_i \leq \frac{\|R[i] - R[j]\|}{2\|R[i]\|}$

Why this specific bound: the derivation (provided in Appendix B) starts by expanding the squared distance inequality, collecting terms involving the random noise components $\delta_{i,k}$, and bounding the worst-case sum of these terms using the component-wise constraint $\delta_{i,k} \leq 1 + \epsilon_i$ for terms with positive coefficients and $\delta_{i,k} \geq 1 - \epsilon_i$ for terms with negative coefficients. The resulting expression involves the sum of absolute values $\sum_k |R_{i,k}(R_{j,k} - R_{i,k})|$, which is upper-bounded via Cauchy-Schwarz by $\|R[i]\| \cdot \|R[i] - R[j]\|$. This substitution yields a cleaner but slightly tighter bound, guaranteeing the original inequality holds while being computationally simpler.

Why use the maximum value: the paper sets $\epsilon_i$ to the right-hand side of the inequality — the largest noise level that still guarantees the perturbed point stays in its cluster. Using a smaller $\epsilon_i$ would mean less variation in the proxy token, making it a less good representative of the diversity within $X_i$. Using a larger $\epsilon_i$ risks crossing cluster boundaries. The maximum safe value maximizes the proxy's representativeness while maintaining correctness.

Dynamic recomputation: the bound $\epsilon_i$ is computed at every training step for every MoE layer, because both $\|R[i]\|$ and the inter-center distances $\|R[i] - R[j]\|$ change as the model trains. As experts become more specialized (their $W_g$ parameters diverge), the coupled router centers $R[i]$ move apart, increasing $\|R[i] - R[j]\|$ and allowing larger $\epsilon_i$. The noise level thus tracks specialization dynamically — a property the paper exploits in Section 4.4 for analyzing expert specialization.

Step 2: Computing the Activation Norm Matrix $M$

For each pair $(i, j)$, the perturbed proxy $\tilde{R}[i]$ is passed through expert $j$'s input projection:

$\text{activation}_{i,j} = \tilde{R}[i] \cdot W_g^j \in \mathbb{R}^D$

where $W_g^j \in \mathbb{R}^{d \times D}$ is the first of the three parameter matrices in expert $j$'s SwiGLU feed-forward network. The L2 norm of this activation vector is then computed:

$M[i, j] = \left\|\tilde{R}[i] \cdot W_g^j\right\|$

What this computes: a scalar representing how strongly expert $j$'s input projection responds to the proxy token from cluster $i$. If expert $j$'s parameters are well-suited to process tokens from cluster $i$, the norm of the intermediate activation will be large. The resulting matrix $M \in \mathbb{R}^{n \times n}$ captures all pairwise proxy-expert responses: row $i$ shows how all experts respond to proxy $i$; column $j$ shows how expert $j$ responds to all proxies. The diagonal $M[i, i]$ is the self-response: how strongly expert $i$ responds to its own cluster's proxy.

Why $W_g$ specifically rather than other activations: the paper ablated five candidates: $\tilde{R}W_g$, $\tilde{R}W_p$, $\text{SiLU}(\tilde{R}W_g)$, the post-SwiGLU activation $\text{SiLU}(\tilde{R}W_g) \odot \tilde{R}W_p$, and the final expert output $(\text{SiLU}(\tilde{R}W_g) \odot \tilde{R}W_p)W_o$. As shown in Figure 6(a), $\tilde{R}W_g$ achieves the best downstream performance among all choices. Using the final output achieves comparable performance but incurs higher computational cost (since it requires the full expert forward pass including $W_p$, gating, element-wise multiplication, and $W_o$ projection). Using only $W_g$ is both the most effective and the cheapest — it requires just one matrix multiplication per expert-proxy pair. The authors hypothesize that $W_g$ captures the core feature extraction capability of the expert before any nonlinearity, which is sufficient for assessing alignment.

Why the L2 norm as the coupling signal: prior work [9, 22, 25] has established that the activation norm of MLPs represents how well their capabilities match their inputs. Higher norm indicates stronger feature matching — the expert's learned weight vectors align well with the token's features, producing large inner products and thus large activation magnitudes. The norm-based selection in AoE [25] directly builds on this principle, using intermediate activation norms as routing scores. The ERC loss inherits this justification: if expert $i$ is genuinely specialized for tokens in cluster $X_i$, then its activation norm for $\tilde{R}[i]$ (a representative of $X_i$) should exceed its activation norm for other clusters' proxies.

Computational pattern: the paper implements this step using a single Einstein summation (einsum) operation, as shown in the pseudocode (Figure 8): torch.einsum('jDd,id->ijD', self.experts.Wg, R). Here, j indexes experts, D indexes the intermediate dimension, d indexes the input dimension, and i indexes the proxy tokens. This batches all $n^2$ matrix multiplications into one efficient tensor operation, avoiding a Python loop over $i$ and $j$. The L2 norm is then taken along the last dimension (dim=-1) to produce $M \in \mathbb{R}^{n \times n}$.

Why $n^2$ operations and not $n \times T$: the breakthrough is that the computation operates on $n$ proxy tokens (one per expert), not on $T$ actual input tokens. The number of expert-proxy interactions is thus $n \times n = n^2$, independent of batch size. In a typical pre-training run where $T$ is in the millions while $n$ is in the hundreds, $n^2$ is negligible compared to $Tn$. This is the computational insight that makes the ERC loss viable where AoE is not.

Step 3: Enforcing the Two Coupling Constraints

The ERC loss imposes two constraints on the matrix $M$ for all pairs $(i, j)$ where $i \neq j$:

Constraint 1 (row constraint): $M[i, j] < \alpha \cdot M[i, i]$

What this constrains: for a fixed proxy token $\tilde{R}[i]$ (row $i$), the activation it elicits from any other expert $j$ must be strictly less than $\alpha$ times the activation it elicits from its own corresponding expert $i$. In operational terms: each cluster's proxy token must activate its designated expert more than it activates any other expert, by at least a factor of $1/\alpha$.

Why this matters for expert specialization: if $M[i, i]$ is large and $M[i, j]$ is small for all $j \neq i$, then expert $i$ has developed parameters that respond specifically and strongly to tokens from cluster $i$ while remaining relatively unresponsive to tokens from other clusters. This is the definition of specialization: expert $i$ is optimized to best match the features of its assigned token cluster $X_i$. Without this constraint, expert $i$ might respond equally strongly to tokens from cluster $j$, meaning it has not developed cluster-specific capabilities — it remains a generalist.

Constraint 2 (column constraint): $M[j, i] < \alpha \cdot M[i, i]$

What this constrains: for a fixed expert $i$ (column $i$), the activation it produces for any other cluster's proxy $\tilde{R}[j]$ must be strictly less than $\alpha$ times the activation of expert $i$ for its own proxy $\tilde{R}[i]$. In operational terms: each expert must respond more strongly to its own cluster's proxy than to any other cluster's proxy, again by at least a factor of $1/\alpha$.

Why this matters for accurate routing: if expert $i$ is most activated by $\tilde{R}[i]$ (and, by extension, by tokens in $X_i$), then the router's representation $R[i]$ accurately encodes what expert $i$ is good at. When the router computes $xR[i]^\top$ for a real token $x$, if that token is similar to $R[i]$ (high inner product), it should genuinely benefit from expert $i$'s processing because expert $i$ has demonstrated strong responsiveness to such tokens. Without this constraint, $R[i]$ might point in a direction that expert $i$ does not actually specialize in, causing the router to send tokens to expert $i$ that expert $i$ processes poorly.

The two constraints together: these constraints are symmetric in the roles of $i$ and $j$ but enforce different aspects of coupling. Constraint 1 is about expert behavior — does each expert actually specialize? Constraint 2 is about router accuracy — does each router embedding faithfully represent its expert's specialty? The paper summarizes:

"Constraint 1 ensures the proxy token $\tilde{R}[i]$ activates its corresponding expert $i$ more than any other expert $j$... Constraint 2 requires that expert $i$ responds more strongly to its own proxy token $\tilde{R}[i]$ than by any other $\tilde{R}[j]$."

The ERC Loss Function

The two constraints are converted into a differentiable loss by summing hinge-style penalties over all off-diagonal pairs:

$\mathcal{L}_{\text{ERC}} = \frac{1}{n^2} \sum_{i=1}^{n} \sum_{j \neq i} \bigl(\max(M[i, j] - \alpha M[i, i], 0) + \max(M[j, i] - \alpha M[i, i], 0)\bigr)$

where $n$ is the number of experts, $M \in \mathbb{R}^{n \times n}$ is the activation norm matrix defined above, and $\alpha \in [0, 1]$ is a scalar hyperparameter controlling coupling strength.

What it computes: for each ordered pair $(i, j)$ with $i \neq j$, two penalty terms are evaluated. The first term $\max(M[i, j] - \alpha M[i, i], 0)$ penalizes violations of Constraint 1: if the off-diagonal element $M[i, j]$ exceeds the threshold $\alpha M[i, i]$, the excess is added as a penalty. If $M[i, j]$ is already below the threshold, the $\max(\cdot, 0)$ function returns zero — no penalty for satisfied constraints. The second term $\max(M[j, i] - \alpha M[i, i], 0)$ analogously penalizes violations of Constraint 2. The double sum aggregates penalties over all $n(n-1)$ ordered off-diagonal pairs, and the $1/n^2$ factor normalizes by the total number of elements in the matrix, making the loss magnitude independent of $n$.

How the penalty mechanism works: the $\max(\cdot, 0)$ function is a hinge loss — it is zero when the constraint is satisfied and positive linear when violated. The gradient is zero for satisfied constraints, so the optimization focuses only on pairs where coupling is inadequate. For a violated pair, the gradient pushes $M[i, j]$ down (by updating expert $j$'s parameters to respond less to proxy $i$) or pushes $M[i, i]$ up (by updating expert $i$'s parameters to respond more to its own proxy), or both. This creates a competitive pressure: expert $i$ is encouraged to become better at processing its own cluster's tokens, while simultaneously discouraging other experts from competing for those same tokens.

Why the hinge form rather than a ratio loss: a natural alternative would be to penalize the ratio $M[i, j] / M[i, i]$ directly when it exceeds $\alpha$. However, ratio-based losses are numerically unstable when $M[i, i]$ is small (early in training or for under-specialized experts) — small absolute changes produce large ratio changes, leading to gradient spikes. The hinge form $\max(M[i, j] - \alpha M[i, i], 0)$ is additive rather than multiplicative, avoiding division and providing stable gradients throughout training.

Why normalize by $n^2$: without normalization, the loss magnitude would grow quadratically with the number of experts $n$. Since the paper experiments with both $n = 64$ and $n = 256$ configurations, the $1/n^2$ normalization ensures the ERC loss weight (fixed at 1 in experiments) has consistent effect across model scales without requiring manual tuning.

Why sum both $M[i, j] - \alpha M[i, i]$ and $M[j, i] - \alpha M[i, i]$: these two terms are not redundant because $M$ is not symmetric in general. $M[i, j]$ is how expert $j$ responds to proxy $i$, while $M[j, i]$ is how expert $i$ responds to proxy $j$. Both must be constrained: each proxy must distinguish its own expert from all others (Constraint 1, across rows), and each expert must prefer its own proxy over all others (Constraint 2, across columns). Constraining only rows or only columns would leave the other dimension unregulated, allowing, for example, an expert to be equally responsive to all proxies (column constraint violated) even if each proxy preferentially activates its own expert (row constraint satisfied).

The role of $\alpha$: $\alpha$ controls the strictness of the coupling constraints. When $\alpha = 0$, the constraints require $M[i, j] < 0$ and $M[j, i] < 0$ (impossible since norms are non-negative), so the loss forces $M[i, j] \to 0$ for all $j \neq i$ — maximum specialization, where each expert responds only to its own cluster's proxy and not at all to others. When $\alpha = 1$, the constraints require only that the self-response $M[i, i]$ exceeds the cross-responses — a weaker condition that still enforces coupling but allows experts to partially respond to other clusters' tokens. Values of $\alpha$ between 0 and 1 interpolate between these extremes. At $\alpha \geq 5$ (as shown in Appendix A, Table 2), the constraints are so weak that even a vanilla MoE (without ERC loss) achieves zero penalty, meaning the loss provides no training signal.

Why $\alpha = 1$ is the default: the paper states "we use $\alpha = 1$ by default if not specified." This choice represents the weakest coupling constraint that still exceeds what a vanilla MoE achieves spontaneously (which, per Table 2, requires $\alpha = 5$ to reach zero loss). It provides meaningful coupling pressure without forcing extreme specialization that might hurt the model's ability to route flexibly. The paper's exploration of $\alpha \in \{0.4, 0.6, 0.8, 1.0\}$ in Section 4.4 shows that $\alpha = 1$ is optimal for the $n = 64$ configuration but not for $n = 256$ (where $\alpha = 0.5$ is better), demonstrating that the optimal strictness depends on model architecture.

How the Loss is Integrated into Training

The ERC loss is computed as a side computation during the forward pass of each MoE layer when the model is in training mode (self.training == True). The pseudocode in Figure 8 shows the integration:

def forward(self, x):
    erc_loss = 0.0
    if self.training:
        R = self.get_noisy_router(self.R)
        M = torch.norm(torch.einsum('jDd,id->ijD', self.experts.Wg, R), dim=-1)
        erc_loss = self.erc_loss(M)

    logits = x.view(-1, x.shape[-1]) @ self.R.T
    scores = logits.softmax(dim=-1)
    expert_weights, expert_indices = torch.topk(scores, dim=-1)
    return self.experts(x, expert_weights, expert_indices), erc_loss

What happens step by step:

1. Training mode check: the ERC loss is computed only during training. At inference, erc_loss = 0.0 is returned, and the rest of the forward pass proceeds identically to a vanilla MoE. There is zero inference overhead.

2. Noisy router generation: the get_noisy_router function computes the per-expert noise bound $\epsilon_i$ dynamically (as described above), draws random noise, and returns the perturbed $\tilde{R}$. This step is wrapped in torch.no_grad() because the noise computation involves operations (norm, cdist, argmin) that should not propagate gradients — the noise is treated as an input to the ERC loss, not something to be optimized.

3. Activation norm matrix $M$: the einsum efficiently computes all $n^2$ matrix-vector products, and the L2 norm along the last dimension produces $M \in \mathbb{R}^{n \times n}$.

4. ERC loss computation: the erc_loss function (detailed below) applies the hinge penalties and returns a scalar.

5. Standard MoE forward pass: the router logits are computed using the clean (unperturbed) self.R, not $\tilde{R}$. The top-$K$ selection, expert computation, and output aggregation proceed exactly as in a vanilla MoE. The perturbed router is used only for computing the ERC loss.

6. Return: the layer output and the ERC loss scalar are returned. The training loop sums the ERC losses from all MoE layers with weight 1.0 and adds them to the language modeling cross-entropy loss and the load balancing loss.

Why the router logits use clean $R$: the noise perturbation is only for probing expert responses — it simulates token-level variation within each cluster. For actual routing decisions, the clean $R$ is the correct representation because it is the cluster center that the router is learning as the optimal routing direction. Perturbing $R$ for routing would introduce unnecessary stochasticity into the token-to-expert assignment, potentially degrading training stability. The paper explicitly notes: "the perturbed $\tilde{R}$ is used only for loss computation; routing still uses the clean $R$ to compute router logits, as in standard MoEs."

The erc_loss function implementation:

def erc_loss(self, M):
    row_diff = (M - self.alpha * torch.diag(M).unsqueeze(1))
    row_diff_clamped = torch.clamp(row_diff, min=0.0)

    col_diff = (M - self.alpha * torch.diag(M).unsqueeze(0))
    col_diff_clamped = torch.clamp(col_diff, min=0.0)

    mask = torch.ones_like(M) - torch.eye(M.size(0), device=M.device)
    total_diff = (row_diff_clamped + col_diff_clamped) * mask

    return total_diff.mean()

What this does operationally:

• torch.diag(M).unsqueeze(1) extracts the diagonal $M[i, i]$ as a column vector of shape $(n, 1)$. Subtracting $\alpha$ times this from $M$ computes $M[i, j] - \alpha M[i, i]$ for all $(i, j)$ pairs simultaneously via broadcasting — row $i$ of the difference matrix uses $M[i, i]$ as the threshold.

• torch.clamp(diff, min=0.0) implements the $\max(\cdot, 0)$ hinge operation, zeroing out negative values (satisfied constraints).

• The column difference col_diff uses unsqueeze(0) instead of unsqueeze(1), creating a row vector of shape $(1, n)$. This computes $M[j, i] - \alpha M[i, i]$ — a different pattern where the threshold varies by column rather than by row.

• The mask removes the diagonal (where $i = j$) from the penalty sum, since the constraints apply only to off-diagonal pairs.

• total_diff.mean() averages over all $n^2$ elements (including the zeroed diagonal), implementing the $1/n^2$ normalization.

Why two separate difference computations: the row constraint $M[i, j] < \alpha M[i, i]$ uses $M[i, i]$ as the reference for all $j \neq i$ — one threshold per row. The column constraint $M[j, i] < \alpha M[i, i]$ uses $M[i, i]$ as the reference for all $j \neq i$ — one threshold per column entry. These are structurally different (row-major vs. column-major application of the diagonal threshold), requiring separate broadcasting operations. A naive implementation might loop over $i$ and $j$, but the vectorized form with broadcasting is dramatically more efficient on GPU.

Efficiency Analysis: Why This Approach Scales

The paper provides both theoretical FLOP analysis and empirical throughput measurements to demonstrate the ERC loss's efficiency advantage over prior coupling methods.

Theoretical FLOP cost per MoE layer:

• Vanilla MoE forward: $6TKDd$ FLOPs, where $T$ is the number of tokens, $K$ is the number of activated experts, $d$ is the hidden dimension, and $D$ is the intermediate FFN dimension. This comes from two $T \times d \times D$ matrix multiplications ($W_g$ and $W_p$, totaling $4TdD$ FLOPs each for the two experts' worth) plus one $T \times D \times d$ multiplication ($W_o$, costing $2TDd$ FLOPs), multiplied by $K$ experts.

• ERC loss overhead: $2n^2 Dd$ FLOPs, where $n$ is the number of experts. This comes from $n^2$ matrix-vector products, each multiplying a $1 \times d$ vector by a $d \times D$ matrix, costing $2dD$ FLOPs.

• AoE overhead: $2T(n - K)dr$ FLOPs, where $r$ is the low-rank factorization dimension. This comes from all $n$ experts processing each token through their $W_{\text{down}}$ projections (costing $T \times n \times 2dr$ FLOPs), minus the $K$ experts that would have processed the token anyway with the original $W_g$ (avoiding double-counting their cost as overhead).

The crucial scaling difference: in the ERC loss, the overhead is multiplied by $n^2$ — the number of experts squared. In AoE, the overhead is multiplied by $T$ — the number of tokens, which is typically orders of magnitude larger. For the 15B-parameter configuration used in the paper ($T = 3 \times 10^6$, $n = 256$, $K = 8$, $d = 1536$, $D = 768$, $r = 512$):

• Base MoE cost: $6 \times 3 \times 10^6 \times 8 \times 1536 \times 768 \approx 1.7 \times 10^{14}$ FLOPs per layer
• ERC overhead: $2 \times 256^2 \times 768 \times 1536 \approx 1.5 \times 10^{11}$ FLOPs per layer (approximately 0.09% of base cost)
• AoE overhead: $2 \times 3 \times 10^6 \times (256 - 8) \times 1536 \times 512 \approx 1.2 \times 10^{15}$ FLOPs per layer (approximately 7× the base cost)

The ERC overhead is roughly four orders of magnitude smaller than the AoE overhead and is a negligible fraction ($< 0.1\%$) of the base forward pass cost. This analysis ignores the cost of computing the noise and constructing $M$ itself (the cdist and norm operations), but these are also $\mathcal{O}(n^2 d)$ and $\mathcal{O}(n^2 D)$ respectively, not scaling with $T$.

Empirical throughput: the paper reports actual training throughputs for the 15B configuration under realistic parallelism (data parallelism dp_size = 64, expert parallelism ep_size = 8):

• Baseline (vanilla MoE): 62.03B tokens/day
• ERC-loss-augmented MoE: 61.52B tokens/day
• Overhead: 0.82%

For the 3B configuration with $n = 64$ and no expert parallelism (dp_size = 32, ep_size = 1), the overhead is even smaller at approximately 0.18%. These measurements include the distributed communication overhead and noise computation costs, confirming that the theoretical FLOP advantage translates to real-world training speed.

Why AoE cannot practically scale to high sparsity: as $n$ increases and $K$ stays fixed, the term $(n - K)$ in AoE's overhead grows. At the extreme of $n = 256$ and $K = 8$, AoE processes 248 experts per token that are then discarded — a 97% waste rate. As the paper reports: "the AoE method failed to train due to being overly costly" at 15B scale with $n = 256$. In contrast, the ERC loss's overhead grows only with $n^2$ and is independent of $T$, making it the only coupling method tested that scales to high-sparsity regimes.

Why the ERC loss has zero inference overhead: at inference, the self.training flag is False, so the entire ERC loss computation is skipped. The model performs exactly the same operations as a vanilla MoE: compute $xR^\top$, softmax, top-$K$ selection, and $K$ expert forward passes. The ERC loss influences the model only through the parameter values it shapes during training — the $R$ and $W_g$ matrices have been optimized to satisfy the coupling constraints, but no extra computation is needed at inference to maintain that coupling.

Design Choices and Hyperparameter Defaults

The paper makes several concrete design decisions, each with explicit justification:

Loss weight fixed at 1: the ERC loss is added to the total training loss without a tunable coefficient. The paper states: "the loss weight of the ERC loss is fixed at 1." This simplifies hyperparameter tuning and works because the loss is self-scaling: the $1/n^2$ normalization ensures the loss magnitude is consistent across model sizes, and the hinge mechanism means the loss decreases toward zero as coupling improves, eventually providing a small but persistent gradient rather than dominating the language modeling objective.

$\alpha = 1$ by default: this is the weakest coupling that still exceeds the vanilla MoE's natural decoupling (which, per Table 2, would need $\alpha \geq 5$ to achieve zero ERC loss). The paper recommends starting with $\alpha = 1$ for practitioners, then experimenting with lower values: "For practitioners implementing the ERC loss, we recommend starting with $\alpha = 1$, which eliminates expert decoupling and should provide some gains. Further improvement may be achieved by experimenting with lower $\alpha$ values, depending on the specific configuration of your model."

Using $\tilde{R}W_g$ as the activation source: the paper ablates five candidates (Figure 6(a)) and selects $\tilde{R}W_g$ as the default because it achieves the best performance while being the cheapest to compute. Using the full expert output (involving $W_p$, SiLU activation, element-wise multiplication, and $W_o$ projection) achieves comparable accuracy but costs more FLOPs. Simpler alternatives ($\tilde{R}W_p$, $\text{SiLU}(\tilde{R}W_g)$) underperform, suggesting that the raw linear projection captures alignment information that the subsequent nonlinearities and gating may partially obscure.

Multiplicative rather than additive noise: multiplicative noise ($\tilde{R}[i] = R[i] \odot \delta_i$) scales perturbations proportionally to the magnitude of each dimension, preserving the directional structure of $R[i]$. Additive noise ($\tilde{R}[i] = R[i] + \delta_i$) would shift the vector by an absolute amount, potentially dominating small-magnitude dimensions and changing the effective direction of the proxy. The multiplicative form ensures that the noise simulates token-level variation in activation strength along the dimensions that matter for routing.

Bounded noise via dynamic $\epsilon_i$: rather than using a fixed noise level (which would be too large for some experts and too small for others as cluster centers move during training), the method computes $\epsilon_i$ per expert per layer per training step. This adapts to the current geometry of the cluster centers and ensures that no $\tilde{R}[i]$ crosses into another expert's territory, regardless of how the router parameters evolve.

Compatibility with load balancing: the ERC loss is applied alongside a standard load balancing loss with weight 0.01. Figure 3(b) shows that the ERC loss does not interfere with load balancing — the load balance loss difference between MoE with and without ERC loss is on the order of $10^{-5}$, negligible relative to the load balance loss magnitude of approximately $10^{-2}$. This is because the ERC loss operates on the relationship between $R$ and $W_g$, not on the distribution of routing probabilities across tokens, which is what load balancing regulates.

No gradient flow through noise sampling: the get_noisy_router function is wrapped in torch.no_grad(). The noise $\delta_i$ is treated as a fixed input to the ERC loss computation, not as a parameter to be optimized. This prevents the model from learning to manipulate the noise distribution (e.g., by making all $R[i]$ identical so $\epsilon_i = 0$ and no perturbation occurs) rather than learning genuine expert specialization.

Why the Architecture Avoids Common Failure Modes

Avoiding norm manipulation as a trivial solution: A natural concern is whether the model could minimize the ERC loss by simply manipulating parameter norms rather than learning meaningful coupling. For example, increasing $\|R[i]\|$ would increase all $M[i, j]$ (since $M[i, j] = \|\tilde{R}[i] W_g^j\|$), but the ERC loss only penalizes relative magnitudes, so this wouldn't help. More subtly, the model might try to make $M[i, i]$ large by increasing $\|W_g^i\|$ while decreasing $\|W_g^j\|$ for $j \neq i$. The paper addresses this in Appendix A.2:

"any attempt to reduce one term of the ERC loss by manipulating norms will simultaneously increase other terms"

Specifically, increasing $\|R[i]\|$ helps satisfy the row constraint for row $i$ by making $M[i, i]$ larger and thus raising the threshold $\alpha M[i, i]$. But it simultaneously hurts the column constraint for column $i$ by making $M[i, j]$ larger (for other rows $j$), potentially violating $M[i, j] < \alpha M[j, j]$. The dual structure of the loss — penalizing both row-wise and column-wise violations — creates a competitive balance that makes norm manipulation ineffective as a minimization strategy. The only stable solution is to align each $R[i]$ with its corresponding $W_g^i$ so that $M[i, i]$ is genuinely large through feature alignment rather than norm inflation.

The empirical evidence supports this: Table 3 shows that the average norms of $R[i]$ and $W_g^i$ are comparable between the baseline and ERC-loss-trained models, while the standard deviations are smaller for the ERC-loss model (indicating more consistent norms). Meanwhile, the ERC loss values drop from significant positive numbers in the baseline to zero in the ERC-loss model. The coupling is achieved through alignment, not norm inflation.

Avoiding the $T$-dependent cost trap: The fundamental design constraint is that the coupling mechanism must not scale with the number of tokens. All prior approaches that achieved effective coupling — AoE [25], CompeteSMoE [30] — violated this constraint by requiring every token to interact with every expert (or every expert's routing-relevant parameters) during training. The ERC loss achieves the same goal by operating on the router embeddings as cluster centers, which are already summaries of the token distribution. The $n$ cluster centers encode the same routing-relevant information that $T$ tokens would provide, but at a fixed cost independent of $T$. This is the core architectural innovation.

4. Key Insights and Innovations

Innovation 1: Expert-Router Coupling as a Distinct, Previously Unidentified Bottleneck in MoE Training

The paper's most fundamental contribution is not the ERC loss itself but the identification and isolation of weak expert-router coupling as a critical, independent failure mode in Mixture-of-Experts models. Before this work, the MoE literature recognized several problems — load imbalance (tokens concentrate on a few experts, leaving others idle), training instability (large router logits causing gradient spikes), and insufficient expert specialization (experts learn redundant, overlapping functions). The field developed auxiliary losses for each: load balancing loss [8, 31, 43] for the first, z-loss [50] for the second, and output orthogonalization or contrastive losses [11, 13, 21] for the third.

What the field had NOT recognized is that even with balanced routing, stable training, and orthogonalized expert outputs, the router can still be wrong about which expert is best for which token. The router's internal representation of expert capabilities — embedded in its parameter matrix R — can drift away from the experts' actual learned capabilities — embedded in W_g, W_p, W_o. This decoupling means tokens get sent to experts that aren't the best match, and experts receive gradients from tokens they can't effectively learn from.

The paper provides two pieces of diagnostic evidence that elevate this from speculation to demonstrated fact:

First, the post-hoc ERC loss analysis (Table 2). Evaluating a trained vanilla MoE model, achieving zero ERC loss across all layers requires setting α = 5. This means that in the trained model's natural state, off-diagonal activation norms M[i, j] frequently exceed the diagonal M[i, i] by up to a factor of 5. The router embedding for expert i — which should be the direction that most strongly activates expert i — is not the direction that most strongly activates expert i. The coupling is quantitatively, severely broken.

Second, the router orthogonalization ablation (Figure 6c). The baseline MoE already has nearly orthogonal router embeddings (average absolute cosine similarity of 0.15, corresponding to angles between 81° and 99°). Enforcing perfect orthogonality provides only marginal gains. This demonstrates that the problem is not router geometry — the embeddings are already well-separated in space. The problem is that each embedding's direction doesn't correspond to what its expert actually does. This is a clean dissociation: router orthogonality (a geometric property) and expert-router coupling (a semantic alignment property) are different quantities, and the latter is what limits performance.

This insight is significant because it redirects the research agenda. Prior work on improving MoE models focused on making experts more different from each other (output orthogonalization, contrastive losses) or making routing more balanced (load balancing losses). The ERC paper shows that neither approach addresses the core problem: the router needs to know which specialist handles what. This is a fundamentally different kind of requirement — it's about alignment between two model components, not about the internal properties of either component alone. Future MoE research can now target coupling directly rather than treating it as an emergent property that will appear if experts are sufficiently specialized.

The paper frames this as analogous to the proposer-scorer mismatch problem in reinforcement learning and inference-time compute, where a proposal distribution and a scoring function drift apart when trained independently. The ERC loss is to MoE what a verifier is to a generator: the component that ensures the selection mechanism (router) remains faithful to the capabilities of the selected components (experts).

Innovation 2: Enabling Fixed-Cost Coupling Through the Clustering Interpretation of Routing

The paper's second major contribution is the conceptual reframing of MoE routing as implicit clustering and the computational strategy this reframing enables. This is not a method that incrementally improves on AoE or CompeteSMoE — it's a fundamentally different approach to the coupling problem that sidesteps the T-scaling bottleneck that made prior dense-activation methods impractical.

The conceptual move: prior work conceptualized the coupling problem as "for each token, determine which expert responds best." This naturally leads to O(Tn) computation — every token must interact with every expert (or its routing-relevant parameters) to collect the coupling signal. AoE does exactly this: all n experts process every token up to the W_down projection, and the resulting activation norms determine routing. CompeteSMoE does this during training: all experts process every token to produce output norms that supervise the router.

The ERC paper reframes the problem as "each router embedding R[i] is a cluster center representing the set of tokens X_i routed to expert i." Under this view, the coupling question becomes: does expert i respond most strongly to its own cluster center, and does cluster center i most strongly activate expert i? Answering this question requires only n proxy tokens (one per cluster center) interacting with n experts — an O(n^2) problem, not an O(Tn) one.

This reframing is not obvious because it relies on an assumption that R[i] genuinely represents X_i. In a decoupled model, this assumption fails — R[i] may point in a direction unrelated to the tokens actually routed to expert i. The ERC loss resolves this circularity: it simultaneously enforces that R[i] represents X_i (by making expert i respond most strongly to R[i]) AND that X_i is routed to expert i (by making R[i] point toward expert i's specialty). The loss bootstraps the clustering interpretation into being correct.

Why this is fundamental rather than incremental: prior coupling methods (AoE, CompeteSMoE) were conceptually straightforward — "let every expert see every token and use the result." They differed in implementation (which part of the expert to use, when to compute the signal) but shared the same O(Tn) scaling and the same limitation: they don't scale to high-sparsity regimes with large n and small K. The ERC loss operates on a different order of the problem (expert count rather than token count) by leveraging a different abstraction (cluster centers rather than individual tokens). This is a qualitative shift in approach, not an optimization of the existing paradigm.

The practical consequence is that the ERC loss is the first coupling method demonstrated to scale to n = 256 experts with K = 8 active — a sparsity level of 97%. AoE "failed to train due to being overly costly" at this scale. The paper's throughput measurements confirm this: 0.82% overhead for the ERC loss versus prohibitive cost for AoE. This isn't a 2× or 3× improvement — it's the difference between feasible and infeasible.

The clustering interpretation also provides a unifying language for thinking about MoE behavior. Router embeddings become "centers," expert W_g matrices become "responsive directions," the activation norm matrix M becomes a "coupling matrix" that can be inspected to diagnose which experts are failing to specialize or which router embeddings are misaligned. This conceptual vocabulary is valuable beyond the loss itself — it gives researchers and practitioners a way to think about and debug MoE models that the standard "router logits and expert outputs" framing doesn't provide.

Innovation 3: Continuous, Quantitative Control Over Expert Specialization

The paper's third contribution is establishing the ERC loss as a scientific instrument for studying expert specialization, not just an engineering tool for improving performance. This is a meta-contribution: the method enables new kinds of empirical investigation that were previously infeasible.

What was missing before: prior work on expert specialization [6, 11, 12, 13, 21, 29, 47] treated specialization as a binary property or an emergent consequence of architectural choices. You could add a shared expert to encourage specialization [6], or orthogonalize expert outputs [11], or increase the number of experts to make each more atomic [12], but you couldn't dial in a specific degree of specialization and observe the consequences. More critically, you couldn't quantitatively track specialization during training to understand how it evolves. This meant that claims about specialization — for instance, that "more specialization is always better" — were based on coarse comparisons between models with different architectures, confounded by other factors.

The ERC loss provides two features that change this:

Feature 1: The α parameter provides a continuous knob for specialization. When α = 0, the loss maximally penalizes cross-expert activation, forcing each expert to respond only to its own cluster's tokens and not at all to others — maximum specialization. When α = 1, the loss only requires that self-response exceeds cross-response — a weaker but still meaningful coupling that allows partial overlap in expert capabilities. Between these extremes, α interpolates the degree of specialization. This enables systematic sweeps: train the same model architecture with different α values, and observe how specialization affects downstream performance.

The results of such a sweep (Figure 5b) are revealing: at n = 64, α = 1 is optimal, and α = 0.4 actually underperforms α = 1. This means that maximum specialization is not optimal — there is a trade-off between specialization and collaborative flexibility. The optimal point depends on model architecture: at n = 256, the optimal α shifts to 0.5, suggesting that larger expert counts support (and benefit from) more specialization. This finding undercuts the implicit assumption in much prior work that "more specialization → better performance."

Feature 2: The noise bound ε_i provides a real-time, quantitative metric of specialization. Because ε_i is dynamically computed from the distances between cluster centers (ε_i = \|R[i] - R[j]\| / 2\|R[i]\|), and because the ERC loss couples router embeddings tightly to expert capabilities, ε_i tracks how differentiated the experts have become. When experts specialize — their W_g matrices diverge to handle different types of tokens — the coupled router embeddings move apart, increasing ε_i. When experts remain generalist, router embeddings stay close together, and ε_i is small.

Figure 5a demonstrates this: ε_i traces a clear trajectory during training that varies systematically with α. Smaller α (stronger coupling, more specialization) produces larger ε_i throughout training. The metric is available at every training step for every layer, providing a granular view of how specialization develops that was previously inaccessible without post-hoc analyses like t-SNE visualizations (which the paper also provides in Figure 4 as qualitative confirmation).

This capability is significant because it transforms expert specialization from a qualitative narrative ("experts become specialized during training") into a quantitative measurement that can be plotted, compared across configurations, and optimized. The paper explicitly calls for future work to develop "reliable quantitative metrics for specialization," and the ERC loss provides the first such metric that is tightly coupled to the training objective itself. This opens the door to automated specialization tuning — for instance, adjusting α online during training based on ε_i trajectories — that would be impossible without a quantitative signal.

Innovation 4: The Specialization-Collaboration Trade-Off as a First-Class Design Dimension

Building on the ability to control specialization, the paper surfaces a finding with important implications: the optimal degree of expert specialization is non-monotonic with respect to both the strictness of the coupling loss and the number of experts. This is neither obvious from prior work nor something the field has systematically studied.

The finding in detail: Figure 5b shows that at n = 64 experts, the relationship between α and downstream accuracy is non-monotonic. α = 1 achieves the best performance (~56.5% average accuracy). α = 0.8 underperforms (~56.3%). α = 0.4 underperforms further (~55.8%), falling close to the vanilla MoE baseline (~55.5%). Stronger coupling — more specialization — actually hurts. Yet when the model scales to n = 256 experts (15B parameters), the optimal α shifts to 0.5, and this value is explicitly searched for and reported.

Why this matters conceptually: prior work largely operated under the implicit assumption that expert specialization is uniformly desirable. Methods were designed to maximize specialization — orthogonalizing expert outputs [11, 13], using shared experts to force the remaining experts to specialize more [6], increasing expert count to extreme values [12]. The ERC paper's results show that this assumption is context-dependent and can be actively harmful. In a model with 64 experts, pushing experts to be maximally different from each other reduces performance because the model loses the ability to assemble effective K-expert combinations for each input. If every expert is hyper-specialized to a narrow cluster, and a token's processing requires capabilities spanning multiple clusters, no K = 8 subset of experts may contain all the needed skills.

The paper frames this as a collaboration requirement: "The core consideration is whether, among all \binom{n}{K} possible expert combinations, an effective K-expert set can be assembled for any given input." When n is small relative to the diversity of the input distribution, experts need to be more generalist so that any K-subset covers the needed capabilities. When n is large, individual experts can afford to be more specialized because the model can select from a larger pool to assemble a covering set.

This insight provides a principled explanation for why architectural choices like shared experts [6] work: the shared expert handles common capabilities, allowing the remaining n-1 experts to specialize more aggressively without breaking the covering property. It also predicts that the optimal α should scale with n/K — more experts per active slot means more room for specialization — which the paper partially validates with the n = 64 vs. n = 256 comparison, though a systematic study across n and K values is left to future work.

Why this is not incremental: this finding changes the optimization problem for MoE architecture design. Rather than searching only over architectural hyperparameters (number of experts, expert size, number of layers), designers must now also search over a specialization parameter that controls how sharply experts divide the input space. The ERC loss provides both the mechanism for controlling this parameter (via α) and the diagnostic signal for evaluating it (via ε_i). This adds a new dimension to the MoE design space that was previously implicit, unmeasured, and uncontrolled.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All pre-training experiments use the open-source dataset dolma-v1.5-sample [37]. The 3B-parameter models are trained on 500B tokens with a batch size of 3 million tokens. The 15B-parameter models are also trained on several trillion tokens (the exact token count for 15B is not specified beyond "a total of several trillion tokens" in the abstract, and the training configuration "largely follows" the 3B setup with increased model depth and expert count). For downstream evaluation, the 3B models are assessed on ARC-Challenge [4], CommonsenseQA [39], COPA [33], BoolQ [3], HellaSwag [48], OpenbookQA [26], SciQ [45], Social IQa [35], WinoGrande [34], and MMLU [14]. The 15B models are evaluated on a more challenging suite: MMLU [14], C-Eval [16], MMLU-Pro [44], AGI-Eval [49], BBH [38], MATH [15], GSM8K [5], and TriviaQA [17].

• Base models. All experiments use MoE language models trained from scratch. The primary configuration is a 3B-parameter model with 12 Transformer layers, hidden dimension d = 1536, intermediate FFN dimension D = 768, 16 attention heads per layer, n = 64 experts, and K = 8 experts selected per token. The implementation is based on OLMoE [27]. The number of activated parameters is approximately 500M. For scaling experiments, the model is expanded to 15B parameters by increasing n to 256 experts (keeping K = 8) and doubling the model depth, resulting in approximately 700M activated parameters. The 3B scale is chosen because it "represents the largest scale at which we could successfully train the AoE model under our available resources" — a practical constraint that makes the 3B comparison the only scale at which all three methods (vanilla MoE, ERC-loss MoE, AoE) can be compared head-to-head.

• Metrics. Pre-training quality is assessed via downstream task accuracy on the benchmark suites listed above. For the 3B models, both average accuracy across all 10 tasks and per-task accuracy are reported. For the 15B models, individual benchmark scores are provided. The paper does not report perplexity on a held-out validation set, which would be the standard pre-training metric — this is a notable omission, as downstream accuracy alone does not fully characterize language modeling quality and can be sensitive to evaluation protocol. Load balancing quality is monitored via the load balancing loss value (weight 0.01 in all experiments). Expert specialization is tracked visually via t-SNE projections of W_g parameters (Figure 4) and quantitatively via the noise bound ε (Figure 5a).

• Baselines. Three baselines are compared at 3B scale: (1) Vanilla MoE — the standard MoE architecture with top-K routing, trained with only the language modeling loss and load balancing loss. (2) AoE (Autonomy-of-Experts) [25] — an MoE variant where routing is performed by computing intermediate activation norms from all experts for every token, eliminating the separate router. AoE uses a low-rank factorization rank r = 512 to match the total parameter count of the vanilla MoE. This is described as "a competitive yet more computationally expensive MoE variant." (3) MoE + L_ERC — the proposed method, a vanilla MoE augmented with the ERC loss (loss weight 1, α = 1 by default). At 15B scale, only vanilla MoE and MoE + L_ERC are compared, because AoE "failed to train due to being overly costly." The paper also ablates against router orthogonalization loss [1] in Figure 6c.

• Generation budget / compute accounting. The paper does not perform generation-time comparisons (there is no inference-time search or sampling budget to account for). Instead, efficiency is measured in terms of training throughput (tokens per day) and memory usage. For the 3B models, the reported metrics are training hours and memory cost relative to the vanilla MoE baseline. For the 15B models, absolute throughput numbers are provided: 62.03B tokens/day for vanilla MoE vs. 61.52B tokens/day for ERC-loss MoE, representing a 0.82% overhead. The theoretical FLOP analysis in Appendix C.1 breaks down the per-layer computational cost in terms of matrix multiplication FLOPs, comparing vanilla MoE (6TKDd), ERC loss overhead (2n²Dd), and AoE overhead (2T(n-K)dr). This analysis establishes that the ERC loss overhead is independent of batch size T while AoE's scales linearly with T.

• Cross-validation / statistical protocol. The paper does not apply cross-validation for the pre-training experiments. Each model configuration is trained once from scratch (a single training run per setting) and evaluated on the downstream benchmarks. This is standard practice for large-scale pre-training experiments where multiple training runs are prohibitively expensive, but it means the reported differences between methods are point estimates without confidence intervals or error bars. For the α sweep in Section 4.4 (Figure 5), four separate training runs are performed (α ∈ {0.4, 0.6, 0.8, 1.0}), and for the ablation studies (Figure 6), additional individual training runs are performed for each variant. The per-task results in Figure 9 provide some indication of variance across tasks, but statistical significance of the aggregate improvements is not formally assessed. For the 15B experiments, only a single α value is reported (the optimal α = 0.5 is mentioned as a result of hyperparameter search but the search process is not detailed).

Main Quantitative Results

3B-Parameter Models: ERC Loss vs. Vanilla MoE vs. AoE

Aggregate downstream accuracy. Figure 3a reports the average accuracy across all 10 downstream tasks as a function of training tokens (from 100B to 500B). The MoE + L_ERC model (blue line) consistently outperforms the vanilla MoE (orange line) throughout training. At 500B tokens, the ERC-augmented model achieves approximately 56.5% average accuracy versus approximately 55.5% for the vanilla MoE — a gap of roughly 1 percentage point that emerges early (by ~200B tokens) and remains stable through the end of training. The AoE model (green line) achieves the highest accuracy overall, reaching approximately 56.8% at 500B tokens, but with the acknowledged 1.6× training time and 1.3× memory overhead.

The gap between ERC-loss MoE and vanilla MoE is modest in absolute terms (~1 percentage point) but should be interpreted in context: this is a pure architectural improvement with negligible computational cost. A 1% accuracy gain on a 10-task aggregate at 3B scale, achieved with <1% training overhead, represents a favorable cost-benefit ratio. Moreover, the gap is stable rather than shrinking — the curves in Figure 3a remain roughly parallel from 200B to 500B tokens, suggesting the benefit is not diminishing with continued training.

Task-specific results (Figure 9). The per-task breakdown reveals that the ERC loss improvement is not uniform across benchmarks. On some tasks (e.g., ARC-Challenge, CommonsenseQA, BoolQ, HellaSwag), the ERC-loss model shows clear separation from the vanilla MoE. On others (e.g., WinoGrande, MMLU), the gap is smaller or the curves overlap. AoE generally tracks above both MoE variants across most tasks but the margin varies. This heterogeneity is expected — different tasks draw on different capabilities, and expert specialization benefits some types of reasoning more than others — but the paper does not analyze which task characteristics correlate with larger ERC loss gains.

Load balancing compatibility. Figure 3b shows the load balancing loss over training for all three methods. The ERC-loss MoE and vanilla MoE have nearly identical load balancing loss curves (both converging to approximately 1.04 × 10⁻² at 500B tokens, with a difference on the order of 10⁻⁵). AoE shows a slightly higher load balancing loss (approximately 1.08 × 10⁻²), with a difference from vanilla MoE of approximately 4 × 10⁻⁴. The paper characterizes this as "still small" but "notably larger than the difference exhibited by ours." This result is important because it demonstrates that the ERC loss strengthens expert-router coupling without interfering with the load balancing objective — the two auxiliary losses are addressing orthogonal problems.

Efficiency comparison. At 3B scale, the paper reports that "MoE models with and without ERC loss have nearly identical throughput and memory costs." In contrast, AoE requires "1.6× more training hours and 1.3× higher memory usage, limiting further scaling due to impractical training times and out-of-memory issues." The exact throughput numbers (tokens/second) are not provided for the 3B configuration, only the relative multipliers. The theoretical analysis in Appendix C.1 supports these measurements: for the 3B configuration with n = 64, d = 1536, D = 768, the ERC loss adds 2 × 64² × 768 × 1536 ≈ 9.7 × 10⁹ FLOPs per layer, compared to AoE's 2T(64-8) × 1536 × 512 ≈ 8.8 × 10¹⁰ × T per-layer FLOPs, where T is in the millions.

15B-Parameter Models: Scaling Validation

Performance on challenging benchmarks. Table 1 reports results for the 15B-parameter configuration (n = 256, K = 8). The ERC-loss model outperforms the vanilla MoE on all eight benchmarks:

The improvements range from +0.4 (MATH) to +2.2 (AGI-Eval), with an average gain of approximately +1.3 percentage points across the eight benchmarks. The paper notes that "throughout this large-scale training, we observed no loss spikes or abnormal gradients," confirming that the ERC loss remains stable at scale.

The gains at 15B scale are broadly consistent with the 3B results in magnitude (roughly 1–2 percentage points), suggesting the benefit scales with model size. However, the 15B comparison is necessarily limited: AoE cannot be run at this scale, so we cannot assess whether the ERC loss narrows or widens the gap with dense-activation coupling methods as model size increases.

Optimal α shifts with expert count. The paper reports that for the 15B model with n = 256, the optimal α is 0.5, compared to α = 1 for the 3B model with n = 64. This result is mentioned in Section 4.4 but the supporting data (training curves at different α values for the 15B model) are not shown. The finding is consistent with the paper's hypothesis that larger n supports more specialization: with 256 experts and only 8 active per token, the model can afford more specialized experts because the probability of assembling an effective 8-expert combination from a larger, more specialized pool is higher.

Expert Specialization Analysis (Section 4.4)

Qualitative visualization. Figure 4 shows t-SNE projections of the W_g parameter rows for experts from layer 6 (the middle depth). For the vanilla MoE (Figure 4a), the points are scattered without clear cluster structure — different experts' parameters intermingle, indicating they have not developed distinct specializations. For the ERC-loss MoE (Figure 4b), the points form visible clusters corresponding to expert IDs. Experts 8, 16, 24, 32, 40, 48, and 56 are projected into roughly separable regions of the 2D space, indicating that their W_g parameters have diverged to encode different feature extraction patterns. This is consistent with the ERC loss's mechanism: by enforcing that each expert responds most strongly to its own proxy token, the loss drives W_g^i to align with R[i] and diverge from W_g^j for j ≠ i.

Quantitative tracking via ε. Figure 5a plots the noise bound ε (averaged or aggregated across layers — the paper doesn't specify the aggregation method) as a function of training tokens for four values of α. Three findings emerge:

1. ε decreases over training. As training progresses from 100B to 500B tokens, ε declines for all α values. This means cluster centers are becoming closer together relative to their norms — experts are becoming less sharply separated. This is somewhat counterintuitive given that Figure 4 shows more clustered expert parameters with the ERC loss. The resolution is that ε measures the distance between router embeddings normalized by their magnitude (ε_i = \|R[i] - R[j]\| / 2\|R[i]\|), not the absolute separation of W_g parameters. As expert parameters diverge (Figure 4b), the router embeddings that are coupled to them may maintain or even reduce their relative separation while still encoding distinct specializations. The paper does not fully explain this dynamic.

2. ε varies systematically with α. Smaller α (stricter coupling) produces larger ε values throughout training. At 500B tokens: α = 0.4 → ε ≈ 0.625; α = 0.6 → ε ≈ 0.575; α = 0.8 → ε ≈ 0.525; α = 1.0 → ε ≈ 0.475. This monotonic relationship confirms that ε tracks the coupling strength: stricter constraints force router embeddings further apart (relative to their norms) to satisfy the condition that each expert responds most strongly to its own proxy.

3. ε provides a real-time signal. The curves are smooth and well-separated, with clear trajectories that could be monitored during training to detect convergence, diagnose training instabilities, or trigger α adjustments. This is the first quantitative metric for MoE specialization that is available online during pre-training without requiring post-hoc visualizations.

Optimal specialization is non-monotonic. Figure 5b shows downstream accuracy (presumably the average across all 10 tasks at the final checkpoint, though the paper doesn't specify the exact evaluation point) as a function of α. The vanilla MoE baseline (horizontal dashed line) achieves approximately 55.5%. All ERC-loss models outperform the baseline: α = 0.4 → ~55.8%, α = 0.6 → ~56.3%, α = 0.8 → ~56.3%, α = 1.0 → ~56.5%. The curve is non-monotonic — performance first increases as α decreases from 1.0 to 0.6, then decreases from 0.6 to 0.4. The optimal α = 1.0 (weakest coupling that still exceeds vanilla MoE) contradicts the intuition that stronger specialization is better.

The paper interprets this as a specialization-collaboration trade-off: at α = 0.4, experts are too specialized, making it difficult to assemble effective K-expert combinations for diverse inputs. At α = 1.0, coupling is sufficient to align routers with experts without forcing hyperspecialization that hurts collaborative flexibility. The fact that α = 0.8 and α = 0.6 both underperform α = 1.0 suggests there is a "valley" in the performance landscape — moderate specialization is worse than either weak specialization (α = 1.0) or strong specialization at larger n (α = 0.5 for n = 256).

Ablation Studies and Robustness Checks

Choice of activation source for computing M (Figure 6a): Five candidates were compared: (a) \tilde{R}W_g (the default), (b) \tilde{R}W_p, (c) \text{SiLU}(\tilde{R}W_g), (d) post-SwiGLU activations (\text{SiLU}(\tilde{R}W_g) \odot \tilde{R}W_p), and (e) final expert output ((\text{SiLU}(\tilde{R}W_g) \odot \tilde{R}W_p)W_o). \tilde{R}W_g achieves the highest downstream accuracy (~56.5% at 500B tokens). The final output variant achieves comparable performance (~56.4%) but at higher computational cost. The other three variants underperform, with accuracy curves that fall between the ERC-loss model using \tilde{R}W_g and the vanilla MoE baseline. This result is non-obvious: one might expect that later-stage activations (closer to the expert output) would be more informative about expert capabilities. The paper attributes W_g's effectiveness to the fact that "the raw linear projection captures alignment information that the subsequent nonlinearities and gating may partially obscure" — essentially, the SiLU and element-wise multiplication introduce nonlinear distortions that reduce the signal about feature alignment between the proxy token and the expert's learned weight directions.

Importance of random noise δ (Figure 6b): Training an MoE with the ERC loss but without noise perturbation — computing M directly from the clean R instead of the perturbed \tilde{R} — substantially degrades performance. The noise-free variant (labeled "MoE + L_ERC - δ") achieves approximately 55.5–55.8% accuracy, barely above the vanilla MoE baseline (~55.5%). This demonstrates that coupling to the exact cluster center R[i] does not generalize to the tokens in X_i that R[i] represents. Without noise, the ERC loss optimizes for R[i] specifically, but the real tokens routed to expert i are not exactly R[i] — they're a distribution around it. The bounded noise simulates this distribution, ensuring the coupling constraint applies to a neighborhood of R[i] rather than the single point. The performance collapse when δ is removed confirms that the noise is not merely a regularization detail but a critical component of the method.

Router orthogonalization loss (Figure 6c): Replacing the ERC loss with a loss that enforces \hat{R}\hat{R}^\top = I (where \hat{R} is the row-wise normalized router matrix) yields only marginal gains over the vanilla MoE. The orthogonalization baseline (labeled "MoE + L_orthogonalization") achieves approximately 55.7% at 500B tokens versus ~55.5% for vanilla MoE — a gap of ~0.2 percentage points, compared to the ERC loss's ~1.0 percentage point gain. The paper explains this by noting that the baseline MoE already has nearly orthogonal router embeddings (average absolute cosine similarity 0.15, corresponding to angles of 81°–99°), so further orthogonalization provides little benefit. More importantly, this result demonstrates that router geometry and expert-router coupling are distinct properties: the router embeddings can be perfectly orthogonal (well-separated in space) while still being misaligned with expert capabilities. The ERC loss addresses the alignment problem directly, which is why it provides substantially larger gains.

Effect of α > 1 (Figure 7, Appendix A.1): Training with α = 2 yields limited improvement over the vanilla MoE (~56.0% vs. ~55.5% at 500B tokens), and α = 3 yields almost no improvement (curves nearly overlapping with the vanilla MoE). This is consistent with the post-hoc ERC loss analysis (Table 2): the vanilla MoE achieves zero ERC loss only at α ≥ 5, so α = 2 or 3 provides much weaker coupling pressure than α = 1, allowing the model to remain in a largely decoupled state. The paper frames this as "the router and experts will shift from a state of no mismatch toward looser coupling constraints, ultimately causing the model to degenerate into a vanilla MoE."

Verification that models do not minimize ERC loss through norm manipulation (Table 3, Appendix A.2): The average parameter norms of R[i] and W_g^i across layers are comparable between the baseline MoE and the ERC-loss MoE (e.g., for layer 0: \|R[i]\| = 1.85 ± 0.39 for baseline vs. 1.67 ± 0.31 for ERC-loss; \|W_g^i\| = 25.46 ± 3.93 vs. 24.14 ± 3.02). However, the ERC-loss model shows lower standard deviations for both quantities (more consistent norms across experts), and the ERC loss values drop from positive values (0.15–0.87) in the baseline to zero in the ERC-loss model. The paper argues that "any attempt to reduce one term of the ERC loss by manipulating norms will simultaneously increase other terms" due to the dual row-and-column constraint structure, making norm manipulation an ineffective strategy. The comparable average norms with reduced variance and zero ERC loss is evidence that the coupling is achieved through directional alignment (making R[i] point in directions to which W_g^i responds strongly) rather than through norm inflation.

MoE vs. ERC-MoE vs. AoE across individual tasks (Figure 9): The per-task breakdown across the ablation studies and the main comparison provides several additional observations:

• On ARC-Challenge, the ERC-loss MoE consistently outperforms vanilla MoE across all ablation variants, suggesting this benchmark (which requires reasoning and knowledge application) particularly benefits from improved routing.
• On HellaSwag, the gains are also consistent and substantial.
• On WinoGrande, the differences between methods are minimal across all configurations, suggesting this task's difficulty may be dominated by factors other than routing quality.
• On MMLU, the ERC-loss model shows modest but consistent gains.
• The final-output-based ERC loss variant (Figure 6a) tracks closely with the \tilde{R}W_g variant on most tasks, confirming that the choice of activation source matters primarily for efficiency rather than capability.

Critical Assessment

Does the ERC Loss Genuinely Improve Model Performance?

The central claim — that the ERC loss "significantly enhances model performance" — is supported by the reported results but with important caveats about magnitude and generality.

What the experiments demonstrate: At 3B scale on 500B training tokens, the ERC loss provides approximately +1.0 percentage point improvement in average downstream accuracy across 10 benchmarks (Figure 3a), from ~55.5% to ~56.5%. At 15B scale on several trillion tokens, the improvement is approximately +1.3 percentage points across 8 more challenging benchmarks (Table 1). These gains are consistent (all benchmarks improve, though by varying amounts) and stable (the gap doesn't shrink with continued training).

What the experiments do NOT demonstrate: Several factors limit the strength of the evidence:

1. Single training run per configuration. Each model is trained once. With pre-training variance known to be non-trivial (different random seeds can produce models with meaningfully different downstream performance), the reported differences of 1–2 percentage points could partially reflect seed variance rather than method effect. Confidence intervals or multi-seed results would substantially strengthen the claims.

2. No perplexity evaluation. The paper evaluates only downstream task accuracy, not language modeling perplexity on a held-out validation set. Perplexity is the standard pre-training metric and is less sensitive to evaluation protocol details than downstream accuracy. It's unclear whether the ERC loss improves the model's fundamental language modeling capability or primarily improves the specific skills tested by the downstream benchmarks (which may involve reasoning, knowledge retrieval, or pattern matching that benefits disproportionately from better routing).

3. The gains are modest in absolute terms. A 1–2 percentage point improvement on downstream benchmarks from a method with <1% training overhead is a favorable trade-off, but it represents a relatively small absolute improvement in model quality. This is not a weakness per se — auxiliary losses typically provide incremental gains — but it means the practical impact may be limited for applications where other factors (data quality, model scale, training duration) dominate performance.

4. The 15B results are sparse. Only eight benchmark scores are reported for the 15B models, with no training curves, no per-layer analysis, no ε trajectories, and no α sweep results shown. The claim that the ERC loss "effectively addresses the expert-router decoupling problem even at scale" is based on a single comparison at a single α value (0.5). The evidence at scale is thinner than at 3B.

Does the ERC Loss Narrow the Gap with Dense-Activation Methods?

The paper claims that the ERC loss "narrows the performance gap with a competitive yet more computationally expensive MoE variant [AoE]." At 3B scale (the only scale where AoE can be compared), this is quantitatively true: AoE achieves ~56.8% average accuracy, ERC-MoE achieves ~56.5%, and vanilla MoE achieves ~55.5%. The ERC loss closes roughly two-thirds of the gap between vanilla MoE and AoE (from a 1.3 percentage point gap to a 0.3 percentage point gap).

However, the practical significance of this finding is limited by two factors:

1. AoE is not a pure performance upper bound. AoE achieves the best accuracy in this comparison, but this is a single architecture at a single scale. It's possible that other coupling methods (not compared) or simply training longer would achieve the same or better accuracy than AoE without the overhead. The comparison establishes that ERC loss is efficient relative to one expensive baseline, not that it approaches the optimal coupling strategy.

2. The gap narrowing cannot be verified at scale. At 15B with n = 256, AoE cannot be trained at all, so we don't know whether the ERC loss would continue to narrow the gap with dense-activation methods at larger scales. The inability to run AoE at scale is itself a point in favor of ERC loss (it's the only viable coupling method at high sparsity), but it means the claim about "narrowing the gap" is constrained to the 3B regime.

Does the ERC Loss Enable Controllable Specialization?

The claim that the ERC loss provides "flexible control and quantitative tracking of expert specialization levels during training" is the most strongly supported and most novel contribution of the paper.

What the experiments demonstrate:

• Controllable specialization: Figure 5b shows that varying α produces systematically different levels of downstream performance, confirming that α controls a meaningful property of the model. The non-monotonic relationship (α = 1.0 is optimal, α = 0.4 underperforms) demonstrates that this control is not trivial — the optimal setting is not at either extreme.

• Quantitative tracking: Figure 5a shows that ε varies monotonically with α and traces a smooth trajectory during training. The relationship is consistent across α values and training steps, establishing ε as a reliable metric.

• Qualitative validation: Figure 4's t-SNE visualizations confirm that the quantitative ε measurements correspond to visible differences in expert parameter clustering — the ERC-loss model's experts form distinct clusters while the vanilla MoE's experts do not.

Limitations:

• Only relative comparison is provided. ε is shown to vary with α, but there is no absolute calibration: what value of ε corresponds to "sufficient" specialization? Is there a threshold below which performance degrades? The metric tracks specialization but doesn't tell the practitioner what target value to aim for.

• The ε-performance relationship is not directly established. Figure 5a shows ε values, and Figure 5b shows performance values, but the relationship between ε and performance for a given model configuration is not analyzed. Does higher ε always correspond to higher performance? For α = 0.4, ε is highest but performance is lowest, suggesting the relationship is non-monotonic.

• No dynamic α adjustment demonstrated. The paper suggests that ε could be used for online control of specialization, but this capability is not demonstrated. All experiments use fixed α throughout training.

Missing Experiments and Analyses

Several experiments would substantially strengthen the paper's claims:

1. Multi-seed training runs. Training 2–3 seeds for the 3B configuration (vanilla MoE and ERC-MoE) would provide variance estimates and allow statistical comparison. The current single-run results could reflect seed noise.

2. Perplexity evaluation. Reporting validation perplexity alongside downstream accuracy would clarify whether the ERC loss improves general language modeling or specifically helps with the skills tested by the benchmarks.

3. ERC loss trajectory during training. The ERC loss value itself (not just ε) over training would show whether the coupling constraints are satisfied early or continue to improve, and whether the loss converges to zero or plateaus at a positive value.

4. Per-layer analysis of coupling. Table 3 provides per-layer ERC loss values for the final checkpoint, but the dynamics of how coupling develops across layers during training are unexplored. Do early layers couple faster than later layers? Does coupling propagate from input to output?

5. Combining ERC loss with other specialization methods. The paper argues that ERC loss addresses a different problem than output orthogonalization or shared experts. Demonstrating that ERC loss provides additional gains on top of these methods (or vice versa) would strengthen the claim that coupling is an independent bottleneck.

6. Sensitivity to loss weight. The ERC loss weight is fixed at 1. The paper does not explore whether larger weights (which would more aggressively enforce coupling) improve or degrade performance, or whether the optimal weight varies with model scale or α.

7. Comparison with non-AoE dense-activation methods. While CompeteSMoE is discussed as related work, it is not implemented as a baseline. A comparison at small scale would contextualize the ERC loss's efficiency-performance trade-off against another coupling approach.

8. Inference-time analysis. The paper claims zero inference overhead, which is true by construction (the loss is not computed at inference), but there is no analysis of whether the ERC-trained model exhibits different routing patterns at inference — for instance, whether tokens are routed more consistently (lower entropy in the routing distribution) or whether expert utilization patterns differ from the vanilla MoE.

Where the Claims Hold Conditionally

• The performance improvement claim holds for the specific model configurations tested (3B with n = 64, K = 8; 15B with n = 256, K = 8) on the specific benchmarks reported. Generalization to other model scales, expert counts, activation rates, or task domains is plausible but unverified.

• The efficiency claim (0.2–0.8% overhead) holds under the specific parallelism configurations tested (data parallelism 32–64, expert parallelism 1–8) and assumes expert parallelism can distribute the n² computation. For configurations with different parallelism strategies (e.g., tensor parallelism, pipeline parallelism), the overhead characteristics may differ.

• The specialization control claim holds qualitatively (ε varies with α, α affects performance) but lacks the quantitative calibration that would make it actionable (what ε target to aim for, whether α should be adjusted during training).

• The claim that the ERC loss "cannot be reduced to contrastive techniques applied individually to routers or experts" is demonstrated only by comparison with router orthogonalization. A comparison with expert-output contrastive learning [11] would provide stronger evidence for this claim.

Overall, the experimental evidence establishes that the ERC loss is an effective and efficient method for improving MoE model performance by strengthening expert-router coupling, and that it enables quantitative investigation of expert specialization. The evidence is strongest at 3B scale (where detailed training curves, ablations, and comparisons with AoE are provided) and thinner but consistent at 15B scale. The primary limitations are the single-run nature of the pre-training experiments, the absence of perplexity evaluation, and the lack of demonstrated integration with other specialization methods or dynamic α control.

6. Limitations and Trade-offs

6.1 The Coupling Signal Requires Batch-Independent Overhead That Becomes Non-Negligible at Very High Expert Counts

The assumption or constraint. The ERC loss achieves its efficiency breakthrough by being independent of token count $T$: it computes $n^2$ expert-proxy interactions regardless of batch size, which the paper correctly identifies as negligible when "$K$ is often in the millions" (Section 3.3). However, this fixed cost grows quadratically with the number of experts $n$. At $n = 64$, the overhead is approximately $2 \times 64^2 \times 768 \times 1536 \approx 9.7 \times 10^9$ FLOPs per layer — trivial. At $n = 256$, it grows to approximately $1.5 \times 10^{11}$ FLOPs per layer. At $n = 1024$ (a scale already explored in recent MoE architectures), the overhead would be $2 \times 1024^2 \times Dd \approx 2.4 \times 10^{12}$ FLOPs per layer for the same hidden dimensions — roughly 25× larger than at $n = 256$.

The consequence. For architectures with very large expert counts (hundreds to thousands), the $n^2$ FLOP cost may become non-trivial relative to the base forward pass, particularly for models with small per-token computation (small $K$, small $d$, small $D$). The method's efficiency advantage over dense-activation approaches (which scale with $T$) erodes as $n$ grows, because $n^2$ eventually competes with $T(n - K)$ when $n$ is large enough relative to $T$. For a configuration with $n = 1024$, $K = 8$, and $T = 3 \times 10^6$, the ERC overhead would still be smaller than AoE's ($2.4 \times 10^{12}$ vs. $T(n-K)dr$), but the gap narrows. In extreme configurations (very large $n$, very small $T$ per device due to model parallelism), the ERC loss could become the dominant auxiliary cost.

What evidence exists in the paper. The paper provides detailed FLOP analysis for $n = 64$ and $n = 256$ (Appendix C.1, C.2) and confirms 0.18% and 0.82% overhead respectively. However, it does not analyze or discuss the $n^2$ scaling behavior beyond $n = 256$. The throughput measurements (Section 4.2, Appendix C.2) are limited to the two configurations tested. The paper does not provide a formula or rule of thumb for when the ERC loss overhead exceeds, say, 5% of total training cost, which would be the point at which practitioners might need to consider alternatives or optimizations (such as computing the loss on only a subset of experts per step).

Mitigation status. The paper does not address the quadratic scaling with $n$ as a limitation or discuss strategies for mitigating it. A natural mitigation — randomly sampling a subset of $m < n$ experts to compute the ERC loss on each step — is not proposed or evaluated. The clustering interpretation suggests this could work: coupling a random subset of experts per step, amortized over many steps, might achieve similar alignment at reduced cost. This is left entirely to future work. For practitioners, the practical guidance (Section 4.4) only addresses α selection, not when the ERC loss itself might become too expensive.

6.2 Performance Gains Are Modest in Absolute Magnitude and Unmeasured Against Statistical Variance

The assumption or constraint. The paper's central performance claim is that the ERC loss "significantly enhances model performance" (Section 6). At 3B scale, the gain is approximately +1.0 percentage point in average downstream accuracy over 10 benchmarks (Figure 3a). At 15B scale, the gain averages +1.3 percentage points across 8 benchmarks (Table 1). These improvements are reported from single training runs per configuration — each model is trained once from scratch without replication.

The consequence. Pre-training is known to exhibit non-trivial variance across random seeds. Different initializations, data orderings, and stochastic gradient noise can produce models whose downstream performance differs by amounts comparable to the reported ERC loss gains. Without confidence intervals or multi-seed results, a practitioner cannot determine whether the +1.0–1.3 percentage point improvement is reliably attributable to the ERC loss or could arise from seed variance alone. This is especially concerning for the 3B results where the absolute gain (~1.0 pp) is modest, and for the α sweep in Section 4.4, where the performance differences between α values are themselves small (~0.2–0.3 pp between adjacent α settings) and could easily fall within seed noise. The claim that the ERC loss "narrows the performance gap with [AoE]" — from a 1.3 pp gap to a 0.3 pp gap — would be substantially weaker if AoE or vanilla MoE variances overlapped with the ERC-MoE confidence interval.

Additional measurement uncertainty comes from the absence of perplexity evaluation on a held-out validation set. The paper evaluates only downstream task accuracy, which is sensitive to prompting format, few-shot example selection, and evaluation protocol. It is possible that the ERC loss improves the specific skills tested by the chosen benchmarks without improving the model's fundamental language modeling capability (as would be measured by perplexity). The paper provides no evidence to distinguish these possibilities.

What evidence exists in the paper. The paper presents no confidence intervals, no error bars, and no multi-seed results for any experiment. The 3B configuration is trained once per setting (vanilla MoE, ERC-MoE at α = 1, AoE, plus one run each for the ablation variants and α sweep). The 15B configuration is trained twice (vanilla MoE and ERC-MoE at α = 0.5). This is standard practice for large-scale pre-training research where multiple training runs are prohibitively expensive, but it means the reported differences are point estimates without statistical characterization. The per-task breakdown in Figure 9 provides some indication of variance across tasks — the ERC loss shows consistent gains on some tasks (ARC-Challenge, HellaSwag) and minimal gains on others (WinoGrande) — but this is cross-task variance, not within-task variance across training runs, and does not address the seed noise question.

Mitigation status. The paper does not address the statistical reliability of its results. This is a common limitation in large-scale training papers and is not unique to this work, but it directly affects the strength of the central performance claim. A practitioner deciding whether to adopt the ERC loss cannot assess whether the expected benefit (1.0–1.3 pp improvement) exceeds the expected variance of their own training setup. Future work with multi-seed runs at smaller scale (where replication is feasible) could establish whether the ERC loss gains are statistically reliable.

6.3 The Method Is Validated on a Single Model Architecture and Training Recipe

The assumption or constraint. All experiments use OLMoE-based [27] MoE language models with SwiGLU expert FFNs, trained on the dolma-v1.5-sample dataset [37] with the AdamW optimizer and a cosine learning rate schedule. The architectures tested are 3B parameters (12 layers, $d = 1536$, $D = 768$, $n = 64$, $K = 8$) and 15B parameters (doubled depth, $n = 256$, $K = 8$). Both use the same basic architectural template: standard Transformer layers with MoE FFNs, top-$K$ routing, SwiGLU activation, and no shared experts.

The consequence. The ERC loss is designed around the structure of SwiGLU MoE layers — it uses $W_g$ specifically as the coupling target, relies on the router computing $xR^\top$ as a similarity score, and assumes the clustering interpretation of routing (which depends on inner-product-based routing with roughly normalized embeddings). MoE architectures that deviate from these assumptions may not benefit from the ERC loss, or may require modifications:

• MoE variants without explicit routers (e.g., hash-based routing, learned index-based routing, or architectures where expert selection is hard-coded by position) cannot use the ERC loss because there is no $R$ matrix to perturb and align with $W_g$.
• MoE with non-SwiGLU experts (e.g., GELU-activated FFNs, or architectures where the first projection is not factorizable into $W_g$ and $W_p$) would need a different choice of activation source for computing $M$. The paper's ablation (Figure 6a) shows that $\tilde{R}W_g$ is the best activation source for SwiGLU, but this finding may not transfer to other expert structures.
• MoE with shared experts [6] or other architectural modifications that affect the routing-expert relationship may interact with the ERC loss in unknown ways. The paper notes that shared experts allow remaining experts to be "more specialized even with the same $n$" (Section 4.4 footnote), suggesting the optimal α might shift, but this interaction is not studied.
• Different training recipes (different optimizers, learning rate schedules, batch sizes, data mixtures) might affect how the ERC loss interacts with the language modeling objective. The paper uses a single training configuration (AdamW with $\beta = (0.9, 0.95)$, weight decay 0.1, learning rate $4 \times 10^{-4} \to 4 \times 10^{-5}$ cosine, load balancing weight 0.01) for all experiments.

More fundamentally, the ERC loss's mechanism depends on the clustering interpretation being a reasonable model of how routing works in practice. If a particular MoE architecture learns to route tokens in a way that doesn't correspond to cluster-based similarity (e.g., the router learns to encode task-specific or position-specific information rather than token-content similarity), the proxy tokens $\tilde{R}[i]$ may not be valid representatives of $X_i$, and the coupling signal may be misleading.

What evidence exists in the paper. The paper demonstrates effectiveness on two model sizes (3B and 15B) within a single architectural family. The appendix (Section C.2) discusses parallelism strategies (data parallelism, expert parallelism) that are common in practice, confirming the ERC loss works under realistic distributed training. However, there is no ablation across different expert architectures (GELU vs. SwiGLU, different $K$ values, shared vs. non-shared experts), different training datasets, or different optimizer configurations. The paper acknowledges that the optimal α depends on $n$ and $K$ (Section 4.4) but does not explore other architectural factors. The router orthogonality observation — that the baseline MoE already has nearly orthogonal routers (average cosine similarity 0.15) — is explicitly caveated: "we do not imply that all MoEs always have nearly orthogonal router embeddings, as this may depend on the data or specific architecture" (Section 4.5). If a model architecture produced less orthogonal routers, the ERC loss might interact differently with the routing geometry.

Mitigation status. The paper does not claim universality and acknowledges that the optimal α is architecture-dependent. The guidance to practitioners (Section 4.4) — "start with $\alpha = 1$" and experiment — implicitly acknowledges that results may vary. However, the paper does not provide diagnostic tools for practitioners to determine whether the ERC loss is working as intended in their specific setup (e.g., whether the ERC loss value decreases during training, whether ε shows meaningful trajectories, what values of ε are typical for well-coupled models). Without such diagnostics, a practitioner applying the ERC loss to a novel architecture cannot easily determine if poor results are due to suboptimal α, an architectural incompatibility, or a training issue.

6.4 Difficulty Estimation for Optimal α Requires Expensive Hyperparameter Search

The assumption or constraint. The ERC loss introduces a new hyperparameter $\alpha$ that controls the strictness of expert-router coupling. The paper's own results demonstrate that the optimal $\alpha$ is neither at an extreme ($\alpha = 1$ is optimal at $n = 64$, but $\alpha = 0.5$ is optimal at $n = 256$) nor trivially predictable from model configuration alone. The relationship between $\alpha$ and performance is non-monotonic (Figure 5b), and the paper provides only qualitative guidance: "smaller values of $n$ favor more generalist experts, while larger $n$ can support a higher degree of specialization" (Section 4.4). There is no quantitative model that predicts the optimal α from $n$, $K$, model depth, or other architectural parameters.

The consequence. A practitioner adopting the ERC loss for a new model configuration must perform a hyperparameter sweep over α to find the optimal value. At the 3B scale, this requires training multiple models from scratch — the paper's sweep covered 4 α values ($\{0.4, 0.6, 0.8, 1.0\}$) on 500B tokens each, totaling 2 trillion tokens of pre-training just for hyperparameter selection. At the 15B scale, the cost is proportionally higher. This undermines the "lightweight" characterization of the method: while the per-step overhead is negligible (0.2–0.8%), the total cost of adopting the method — including the α search — may be substantial for large-scale training where each training run represents a significant compute investment.

The problem is exacerbated by the fact that α is not the only hyperparameter. The choice of activation source for computing $M$ (Figure 6a), the loss weight, and potentially the noise distribution could also require tuning. The paper fixes all of these (loss weight = 1, activation source = $\tilde{R}W_g$, multiplicative uniform noise) based on ablations at 3B scale, but the optimal choices may themselves depend on model architecture and scale.

What evidence exists in the paper. The paper explicitly acknowledges this limitation in Section 4.4:

"we currently lack quantitative metrics to characterize 'large' or 'small' $n$ and $K$ across different models; as a result, determining the optimal trade-off remains largely empirical."

The paper's suggested mitigation — "start with $\alpha = 1$" and experiment — is practical advice that reduces the problem (α = 1 provides meaningful gains at both tested scales, even though it's suboptimal at $n = 256$) but does not eliminate the need for expensive search to achieve optimal results.

The $\epsilon$ metric (Figure 5a) provides a potential signal that could guide α selection without full training runs, but this capability is not developed. The paper does not establish a relationship between $\epsilon$ trajectories at early training stages and final model performance, nor does it propose using $\epsilon$ to dynamically adjust α during training (which would amortize the search cost into a single training run). The paper's ε analysis is purely descriptive (showing that ε varies with α) rather than prescriptive (showing how ε can guide α selection).

Mitigation status. The paper leaves the development of automated α selection methods to future work, explicitly calling for "reliable quantitative metrics for specialization, and an automated evaluation of the optimal specialization degree for a given model" (Section 4.4). The $\epsilon$ metric and its correlation with α is a promising direction but is not yet actionable for hyperparameter optimization. The "start with $\alpha = 1$" recommendation is a partial mitigation — it ensures some gains without search — but leaves performance on the table for configurations where $\alpha = 1$ is suboptimal.

6.5 The Loss Provides No Guarantees About Inference-Time Routing Behavior

The assumption or constraint. The ERC loss is computed during training only and imposes coupling constraints on router embeddings $R[i]$ and expert $W_g$ parameters through proxy tokens $\tilde{R}[i]$. At inference, no coupling computation occurs. The implicit assumption is that training-time coupling between $R$ and $W_g$ translates to improved routing decisions on real input tokens at inference — that if $R[i]$ is well-aligned with $W_g^i$ (as the ERC loss enforces), then tokens similar to $R[i]$ (which the router assigns to expert $i$ due to high inner product) will genuinely benefit from expert $i$'s processing.

The consequence. The ERC loss optimizes a proxy metric (activation norms of expert responses to perturbed router embeddings) and assumes this proxy correlates with actual routing quality on real tokens. This assumption could fail in several ways:

• Distribution mismatch between proxy tokens and real tokens. The bounded multiplicative noise $\delta_i$ ensures $\tilde{R}[i]$ is within the same cluster as $R[i]$, but real tokens in $X_i$ may follow a different distribution than uniform multiplicative perturbations around $R[i]$. If the actual token distribution in cluster $i$ is concentrated in a particular subregion, or has correlations across dimensions that the independent uniform noise doesn't capture, the coupling learned from $\tilde{R}[i]$ may not transfer perfectly to real tokens.

• The router does not guarantee that $xR[i]^\top$ is high only for tokens in $X_i$. The ERC loss constrains $R[i]$ to align with $W_g^i$, but it does not constrain the router's behavior at the decision boundary between clusters. Two tokens that are similar to each other might get routed to different experts based on small differences in their inner products with $R[i]$ versus $R[j]$. The ERC loss does not explicitly optimize for smooth or sensible routing boundaries.

• Token representations evolve during training. The ERC loss operates on router embeddings $R$, which are free parameters, not on the token representations $x$ that the router receives. The token representations are produced by the preceding attention and FFN layers, which also change during training. The ERC loss couples $R[i]$ to $W_g^i$, but if the token representations drift, the mapping from token $x$ to cluster center $R[i]$ (via inner product) may no longer correspond to the mapping from $R[i]$ to expert $i$ (which is what the ERC loss enforces). The paper provides no analysis of whether the router embeddings and token representations remain in a consistent space throughout training.

• No direct measurement of routing accuracy. The paper evaluates downstream task performance and load balancing, but does not directly measure whether the ERC loss actually improves routing decisions. For instance: do tokens routed to expert $i$ receive higher activation norms from expert $i$ than from other experts? Does the entropy of the routing distribution decrease (indicating more confident/consistent routing)? Are there fewer cases where the top-1 expert is clearly the wrong choice? Without such measurements, the mechanism by which the ERC loss improves performance remains somewhat opaque — it could be through better routing, better expert specialization, or some combination.

What evidence exists in the paper. The paper provides indirect evidence that coupling improves through several channels: the ERC loss value drops to zero (Table 3), ε varies systematically with α (Figure 5a), and expert parameters form more distinct clusters in t-SNE projections (Figure 4). However, none of these directly demonstrate that inference-time routing decisions are more accurate. The performance improvements on downstream benchmarks (Figure 3a, Table 1) are consistent with better routing but do not isolate the mechanism. The paper's ablation showing that router orthogonalization provides minimal gains (Figure 6c) provides some evidence that the ERC loss is doing something beyond improving router geometry — something related to expert alignment — but the chain from "$R[i]$ aligns with $W_g^i$" to "tokens are correctly routed at inference" has an unverified link.

Mitigation status. The paper does not address this limitation explicitly. It does not propose or conduct experiments that directly measure routing accuracy (e.g., by comparing expert activation norms for routed vs. non-routed tokens, or by analyzing per-token routing entropy). The clustering interpretation provides a conceptual justification for why proxy-token coupling should translate to real-token routing quality, but this justification is not empirically validated. This is a significant gap because it leaves open the possibility that the ERC loss improves performance through a mechanism other than better routing — for instance, by regularizing expert parameters to be more structured in general, independent of the coupling to the router.

6.6 The Method Targets Only One Dimension of Expert Specialization and Does Not Address Load Concentration or Expert Collapse

The assumption or constraint. The ERC loss strengthens alignment between router embeddings and expert capabilities — it ensures that $R[i]$ points in a direction to which $W_g^i$ responds, and that $W_g^i$ responds most strongly to $R[i]$. However, it does not directly constrain which tokens get routed where in terms of their content or the diversity of tokens within each expert's cluster. The ERC loss is compatible with load balancing (as demonstrated in Figure 3b) but does not itself contribute to or guarantee balanced routing.

The consequence. Several MoE failure modes that are distinct from weak coupling can persist or even be exacerbated under the ERC loss:

• Expert collapse / representation collapse. Despite coupling, an expert could learn to respond strongly to its own $R[i]$ while $R[i]$ itself drifts to represent a degenerate set of tokens (e.g., all punctuation, or all tokens at a specific position). The ERC loss would be satisfied — $M[i, i]$ is large, $M[i, j]$ is small — but the expert's specialization is trivial or unhelpful. The load balancing loss partially mitigates this by ensuring tokens are distributed across experts, but load balancing only constrains the quantity of tokens per expert, not their quality or diversity.

• Token concentration within experts. Even with balanced total load, an expert might receive tokens from a very narrow subset of the input distribution (e.g., only tokens from a specific domain, position, or syntactic role). The ERC loss does not penalize this — as long as the tokens routed to expert $i$ are similar to $R[i]$ and activate expert $i$ strongly, the coupling constraints are satisfied. Whether this narrow concentration is desirable (true specialization) or harmful (the expert overfits to a spurious feature of the training data) depends on the data distribution and is not controlled by the ERC loss.

• Interaction with load balancing. The paper demonstrates that the ERC loss is compatible with load balancing loss (Figure 3b), but this compatibility is shown only at one load balancing weight (0.01) and one α setting (1.0). At stricter α values, the ERC loss might indirectly affect load distribution — if experts become more specialized, certain token types might concentrate on fewer experts, potentially conflicting with load balancing objectives. The paper does not explore the interaction between α and load balancing weight.

• No constraint on what experts specialize in. The ERC loss ensures each expert specializes in something (the tokens in its cluster), but does not ensure that the specializations are diverse, complementary, or covering the full input distribution. It is possible for experts to develop overlapping or redundant specializations while still satisfying the coupling constraints, as long as each cluster center is distinct enough to satisfy $M[i, j] < \alpha M[i, i]$. The paper's t-SNE visualizations (Figure 4) show that experts trained with ERC loss form more distinct clusters, but this does not guarantee that the clusters cover meaningfully different aspects of the input.

What evidence exists in the paper. The paper does not analyze expert utilization patterns beyond load balancing loss (Figure 3b). There is no analysis of token diversity within experts (e.g., entropy of token types per expert, domain distribution per expert, or position distribution per expert). The t-SNE projections (Figure 4) show expert parameter clustering but do not show which tokens are assigned to each expert. The claim that the ERC loss enables "specialization" is based on the coupling mechanism (each expert responds most strongly to its own proxy) rather than on direct evidence that experts handle different types of tokens or develop complementary capabilities.

Mitigation status. The paper does not claim that the ERC loss solves load balancing, diversity, or expert collapse — it is explicitly described as targeting the "weak coupling between router decisions and expert capabilities" (Section 6). However, the paper's broader framing of specialization (Section 4.4) could be read as implying that stronger coupling leads to better specialization across all relevant dimensions. The limitation is that coupling is necessary but not sufficient for good specialization — an expert can be well-coupled to its router embedding while still learning unhelpful or redundant capabilities. The paper does not provide diagnostic tools for distinguishing "good" specialization from "bad" specialization, and does not discuss how the ERC loss should be combined with other techniques (output diversification, shared experts, domain-aware routing) to achieve truly complementary expert capabilities.

The $\epsilon$ metric (Figure 5a) tracks cluster separation but does not indicate whether the separated clusters correspond to useful divisions of the input space. A practitioner using the ERC loss cannot determine from $\epsilon$ alone whether their experts are developing meaningful or trivial specializations.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a diagnostic reframing rather than a paradigm shift. The ERC loss does not fundamentally alter the MoE architecture — experts still process tokens via top-K selection, routers remain linear classifiers, and the forward pass is unchanged at inference. What changes is the conceptual vocabulary the field can use to think about and debug MoE models: weak expert-router coupling is now isolated as a distinct failure mode, separable from load imbalance, training instability, and expert redundancy.

This matters because it redirects attention from expert-side interventions to alignment problems between model components. Prior work focused on making experts more different from each other (output orthogonalization, contrastive losses) or making routing more balanced. The ERC paper demonstrates that an MoE can have perfectly orthogonal routers, balanced load, and stable training — yet still underperform because the router doesn't know which expert handles what. This is a subtle failure mode that was invisible without a coupling-specific diagnostic. The post-hoc ERC loss evaluation (Table 2), where a trained vanilla MoE requires α = 5 to achieve zero coupling loss, provides the first quantitative evidence that decoupling is real and severe in standard training.

The clustering interpretation of routing is the paper's most generative conceptual contribution. By treating router embeddings as cluster centers rather than opaque parameter vectors, the paper makes visible what was previously implicit: the relationship between an expert's specialty and the tokens routed to it can be probed, measured, and optimized. This reframing opens the door to a family of cluster-based analyses — visualizing token-to-center distances, tracking cluster purity over training, measuring inter-center separation as a specialization metric — that were not previously part of the MoE toolkit. The ε metric (Figure 5a) is the first quantitative specialization tracker available online during pre-training, and its existence alone changes what empirical questions can be asked about MoE training dynamics.

The paper also resolves a latent tension in prior work between two competing narratives: (1) that expert specialization is universally desirable (motivating orthogonalization, shared experts, and massive expert counts), and (2) that MoEs with many experts sometimes underperform dense models of similar compute (motivating skepticism about routing quality). The ERC paper's finding that specialization and performance have a non-monotonic relationship (α = 1 outperforms α = 0.4 at n = 64; α = 0.5 is optimal at n = 256) provides a resolution: specialization helps, but only up to a point that depends on the expert-to-active-slot ratio. Pushing specialization too far breaks the model's ability to assemble effective K-expert combinations. This explains why prior work that maximized specialization through output orthogonalization [11, 13, 21] sometimes saw limited or negative gains — they were operating on the wrong side of the specialization-collaboration trade-off curve, without a knob to dial in the optimal point.

The practical efficiency breakthrough — achieving coupling with O(n²) rather than O(Tn) cost — shifts what is feasible for large-scale MoE training. AoE and CompeteSMoE demonstrated that dense interaction between tokens and experts improves performance, but their computational cost made them impractical at the high sparsity levels (n ≫ K) that modern MoE architectures target. The ERC loss is the first coupling method demonstrated to work at n = 256 with K = 8 while adding <1% training overhead. This doesn't make AoE obsolete — AoE still achieves the best accuracy at 3B scale (Figure 3a) — but it establishes that the coupling benefit can be largely recovered without the prohibitive cost, making coupling a viable consideration for architectures where it was previously dismissed as impractical.

However, this is not a paradigm shift. The ERC loss is an auxiliary loss — a training-time regularizer — not a new architecture or training algorithm. Its gains are meaningful but modest (+1.0–1.3 pp downstream accuracy). The fundamental MoE architecture and training procedure remain unchanged. The paper's primary legacy will likely be conceptual and diagnostic: establishing coupling as a first-class concern in MoE design, providing tools to measure it, and demonstrating that it can be controlled with a lightweight mechanism.

Follow-Up Research This Work Enables

Characterizing the coupling dynamics across model depth and training stages. The paper provides only layer-wise ERC loss values at the final checkpoint (Table 3). A natural extension is to track how coupling develops layer-by-layer during training: do early layers couple faster than late layers? Does coupling propagate from input to output, or does it emerge simultaneously? Table 3 hints at layer heterogeneity — layers 0 and 11 have higher baseline ERC loss values (0.87 and 0.50) than intermediate layers (0.15–0.28) — but the dynamics are unexplored. A strong follow-up would train a 3B model with the ERC loss, log M matrices and ε values per layer at intermediate checkpoints (e.g., every 10B tokens), and analyze whether coupling depth correlates with downstream task improvements. If certain layers resist coupling, layer-specific α values or loss weights could recover performance that uniform α leaves on the table.

Measuring whether the ERC loss actually improves routing accuracy on real tokens. The paper demonstrates that the ERC loss drops to zero (Table 3) and ε varies with α (Figure 5a), but never directly measures whether inference-time routing decisions improve. A crucial follow-up would instrument a trained ERC-MoE to compute, for a sample of real input tokens, the activation norms of the K selected experts versus the (n - K) non-selected experts. If coupling translates to better routing, the ratio of selected-expert activation norms to non-selected-expert activation norms should be higher for the ERC-trained model than for a vanilla MoE. A more ambitious version would track whether tokens that the ERC-MoE routes differently from the vanilla MoE (divergent routing decisions) show systematically higher activation norms under the ERC model's choices versus the vanilla model's choices. This experiment would close the chain from "R[i] aligns with W_g^i" to "tokens are actually routed to experts that process them well."

Automated α selection via early-training ε trajectories. The paper identifies the need for expensive α sweeps as a practical limitation of the method (Section 4.4). A direct follow-up would train a family of 3B models at α ∈ {0.2, 0.4, 0.6, 0.8, 1.0} but monitor ε at very early training stages (e.g., first 10B–50B tokens). The hypothesis is that ε trajectories diverge quickly and predict final performance, allowing α selection from a short preliminary training run rather than a full 500B-token training. If ε at 50B tokens predicts final accuracy with reasonable correlation, a practitioner could run a cheap sweep (5 configurations × 50B tokens = 250B total, comparable to one full training run) to select α, then commit the full budget to the chosen value. A negative result — where ε trajectories don't stabilize or don't predict final performance — would also be valuable, as it would establish that cheap α selection is not possible and that alternative approaches (dynamic α scheduling, learned α) are needed.

Combining ERC loss with output-level specialization methods. The paper argues that coupling and output diversification address different bottlenecks, but never tests whether combining ERC loss with expert output orthogonalization [11] or shared experts [6] provides additive gains. A direct experiment at 3B scale would train four configurations: (a) vanilla MoE, (b) MoE + output orthogonalization loss, (c) MoE + ERC loss, (d) MoE + both losses. If the combination outperforms either alone, it would confirm that coupling and diversification are complementary. If gains are sub-additive or the combination underperforms, it would suggest the methods partially address the same underlying problem, refining our understanding of what "specialization" means mechanistically. A version of this experiment using a shared expert architecture [6] is particularly interesting: the shared expert handles common capabilities, potentially allowing the remaining experts to tolerate stricter coupling (lower α) without breaking the K-expert covering property. The paper's footnote in Section 4.4 explicitly flags this interaction as unexplored.

Stress-testing the clustering assumption under distribution shift. The ERC loss assumes that R[i] is a valid proxy for the tokens routed to expert i. This assumption could break if the model is fine-tuned on a new domain or evaluated on out-of-distribution data. A stress-test experiment would pre-train an ERC-MoE on dolma-v1.5-sample (the paper's setup), then fine-tune it on a substantially different domain (e.g., code, scientific text, or a non-English language) and measure whether the ERC loss increases after fine-tuning (indicating recoupling is needed) and whether continuing to apply the ERC loss during fine-tuning improves adaptation. If the clustered token distributions shift significantly, the original R[i] embeddings may no longer represent the new token assignments, and re-applying the ERC loss during fine-tuning could be necessary to re-establish coupling. A negative result — where coupling survives distribution shift or where re-applying the ERC loss doesn't help fine-tuning performance — would establish the robustness limits of the clustering assumption and suggest that the ERC loss is primarily a pre-training technique rather than a general MoE regularizer.

Investigating whether the ERC loss serves as implicit verifier training for routing. The ERC loss's structure — proxy tokens, expert activation norms, a contrastive objective — bears structural similarity to how verifiers are trained for reasoning models (generating candidate solutions and scoring them). A speculative but intriguing direction is whether the ERC loss implicitly trains the router to act as a "capability verifier" for experts: by learning to align R[i] with W_g^i, the router may develop the ability to predict not just which experts will be selected but which experts would most benefit a given token. This could be tested by measuring whether router logits correlate with expert output quality (e.g., the reduction in next-token prediction loss when using expert i versus a random expert) in ERC-trained models versus vanilla MoEs. If the correlation is higher, it suggests the router has learned a more faithful internal model of expert capabilities — a form of emergent metacognition about the model's own components.

Practical Applications and Downstream Use Cases

Cost-efficient pre-training of high-sparsity MoE models. The most immediate practical application is for teams training MoE language models at scale with high sparsity (n ≫ K). For a model with n = 256 experts and K = 8 active (97% sparsity), the ERC loss adds 0.82% training overhead while improving downstream accuracy by approximately +1.3 percentage points across benchmarks (Table 1). For a training run costing millions of GPU-hours, a <1% increase in compute cost yielding a >1 percentage point accuracy gain represents a highly favorable return on investment. The method is a drop-in addition — fixed α = 1 provides gains without tuning, load balancing is unaffected (Figure 3b), and no inference overhead is incurred. For teams already using auxiliary losses (load balancing, z-loss), adding the ERC loss requires minimal engineering effort: approximately 10 lines of PyTorch (Figure 8) added to each MoE layer's forward pass.

Deployment of MoE models with dynamic specialization requirements. The α parameter provides a lever that could be exploited in deployment scenarios where the optimal specialization level varies by task or user. A service provider fine-tuning a single base ERC-MoE for multiple downstream applications could use different α values (or whether to apply the ERC loss at all during fine-tuning) to control how aggressively experts specialize to each domain. For a broad-coverage assistant (where collaborative flexibility matters), a higher α (weaker coupling, more generalist experts) might be preferred. For a specialized code-generation model (where expert specialization to syntax patterns could help), a lower α might be beneficial. The paper's finding that α = 1.0 is optimal at n = 64 but α = 0.5 is optimal at n = 256 provides initial calibration for this approach: larger expert counts can tolerate and benefit from more aggressive specialization.

Diagnostic monitoring during MoE pre-training. The ε metric (Figure 5a) provides a training-time signal that was previously unavailable: a quantitative, per-layer measure of expert specialization that can be logged and monitored without post-hoc analysis. ML teams training large MoE models could track ε trajectories to detect training anomalies (e.g., ε suddenly collapsing in certain layers, indicating loss of specialization), compare specialization patterns across different architectural choices (e.g., different K values, different expert sizes), or trigger automated interventions (e.g., temporarily increasing the ERC loss weight if ε drops below a threshold). The paper's demonstration that ε varies smoothly and systematically with α (Figure 5a) establishes that the metric is well-behaved and interpretable, making it suitable for integration into existing training monitoring dashboards.

Informing architectural decisions about expert count and routing configuration. The paper's finding that the optimal α shifts from 1.0 at n = 64 to 0.5 at n = 256 provides concrete guidance for teams deciding on MoE architecture: larger expert counts not only increase total capacity but also support a higher degree of productive specialization. This means the performance benefit of increasing n may be partly mediated by improved specialization (enabled by stronger coupling at lower α), which is an effect not captured by standard parameter-count or FLOP-based scaling analyses. A team choosing between (n = 64, K = 8) and (n = 128, K = 8) can now anticipate that the larger configuration may benefit from a lower α, and can budget a small-scale α sweep into their architecture selection process. The paper's framework — that the optimal specialization depends on whether an effective K-expert combination can be assembled from n specialized experts — provides a conceptual model for extrapolating these choices to even larger configurations.