ArXiv: 2605.10933

🎯 Pitch

A sparse MoE model can match a dense Transformer's performance using exactly the same total parameter budgetβ€”not just the same computeβ€”if you fix the router and activation functions. DECO achieves this by stabilizing expert activation ratios with a simple input-normalized SiLU and auto-tuning sparsity, then extracts a 2.93Γ— speedup on a Jetson edge device.

1. Executive Summary

This paper proposes DECO, a sparse Mixture-of-Experts architecture designed to match the performance of dense Transformers under identical total parameter budgets and training tokens β€” a stricter constraint than prior MoE-dense comparisons that allow MoE models to train on more data under the same computation budget. The architecture combines three named mechanisms: ReLU-based routing enhanced by learnable expert-wise router scaling (which adaptively calibrates the contribution of individual routed experts via learned per-expert coefficients), NormSiLU (a dual-stage activation function that prepends inter-expert mean normalization and intra-expert RMS normalization before the SiLU non-linearity to stabilize activation ratios and prevent vanishing activations), and non-gated MLP experts with adaptive sparsity regularization (which auto-scales the regularization coefficient to precisely control the activation ratio). Evaluated on language modeling perplexity and commonsense reasoning benchmarks across four parameter scales (0.11B–1.18B) using a diverse training data mixture, DECO activates only 20% of routed experts while matching dense model performance and outperforming established MoE baselines like DeepSeek-V3 and ReMoE, with a specialized acceleration kernel delivering a 2.93Γ— speedup over dense inference on Jetson AGX Orin β€” establishing that dense-comparable performance under matched total parameters and training tokens is achievable when the router and expert design are jointly optimized for activation stability and sparsity control.

2. Context and Motivation

The Core Problem: MoE's "Dense Incomparability" Within a Fixed Budget

Mixture-of-Experts architectures offer a compelling tradeoff: scale model capacity (total parameters) without proportionally scaling computation (the number of parameters actually used per token). This makes them attractive for large language models, where the cost of training and inference grows with model size. However, the paper identifies a fundamental gap in the existing MoE literature that becomes critical for a specific deployment scenario: end-side devices, such as edge GPUs, mobile processors, or embedded systems like the NVIDIA Jetson AGX Orin used in the paper's experiments.

The gap is best understood through three competing objectives that end-side deployment simultaneously demands (Section 1, Figure 1):

  1. High performance: The model must achieve accuracy comparable to a dense baseline of equivalent representational capacity.
  2. Low computation cost: The per-token computational budget (active parameters) must be substantially smaller than the dense model's, which is the standard MoE promise.
  3. Small storage overhead: The total parameter footprint β€” the sum of all expert weights, shared layers, attention, and embeddings β€” must fit within the constrained memory of edge hardware and must not trigger excessive memory-access latency from paging experts between storage tiers (e.g., from flash to GPU memory, or from high-bandwidth memory to shared memory on a Jetson device).

The key tension is between objectives (2) and (3): a standard MoE achieves low computation by having many experts (large total parameters) while activating few per token. But this massive total parameter count creates a storage problem. On a server-side GPU with abundant high-bandwidth memory, the memory-access cost of loading expert weights may still be manageable. On an edge device with limited memory and slower interconnects, the latency of repeatedly fetching expert parameters can erode or nullify the computational speedup from sparse activation. As the paper notes (Section 1):

"MoE with a huge number of total parameters demands substantial storage space. More critically, large MoE models may incur high memory-access costs when transferring experts between GPU high-bandwidth memory and shared memory, or when moving offloaded parameters from disk/flash storage to GPU/NPU memory of end-side devices."

This is not a hypothetical concern. Li et al. (2025), cited in the paper, also identifies that memory-access bottlenecks limit the practical efficiency of MoE inference, even on server hardware. For edge deployment, the problem is more acute.

The central research question the paper poses is therefore:

"Can a sparse MoE model achieve performance comparable to a dense model, given the same total parameter budget and the same number of training tokens?"

This is a stricter formulation than prior work. Most MoE research asks: can an MoE with more total parameters but fewer active parameters match or exceed a dense model when the MoE is allowed to train on more tokens (since MoE's lower per-token computation means more tokens can be processed within a fixed FLOPs budget)? The paper explicitly contrasts with this approach, citing Li et al. (2025):

"A closely related study by Li et al. (2025) identifies the optimal settings of DeepSeek-V3-style MoE architectures that enable them to surpass dense models under matched total parameters and computation budget. However, due to the low per-token computation of sparse MoE, in that work, MoE settings are trained on substantially more tokens under the same computation budget. We adopt a stricter setting that requires exactly the same number of training tokens."

The distinction is crucial for end-side deployment: if an MoE achieves dense parity only by training on 2–3Γ— more data, the total storage and training cost advantage may disappear. The paper's question is: can an MoE match a dense model pound for pound β€” same total parameters, same training data β€” while still achieving the sparse activation advantage?

Why This Matters: The "Ideal Triangle" and Edge Deployment

The practical significance extends beyond academic curiosity. As LLMs become ubiquitous, the deployment landscape is fragmenting: large models serve cloud APIs, but there is growing demand for on-device inference in privacy-sensitive applications, low-latency scenarios (autonomous systems, real-time translation), and bandwidth-constrained environments. The prevailing approaches for shrinking models for edge deployment include:

  • Pruning: Removing weights from a trained dense model, which can reduce storage but often degrades performance and does not fundamentally change the compute-per-token.
  • Quantization: Reducing numerical precision (e.g., INT4, INT8), which reduces both storage and compute but with precision-accuracy tradeoffs.
  • Knowledge distillation: Training a smaller student model on a larger teacher's outputs, which reduces both storage and computation but typically at a significant performance cost relative to the teacher.

MoE offers a different point in the design space: keep total capacity (total parameters) moderately sized for edge storage, use sparse activation to reduce computation, and achieve performance closer to a dense model of the same total size rather than a compressed or distilled approximation. If this works, it provides a qualitatively different deployment option β€” one where the model thinks smarter (routes each token to specialized sub-networks) rather than thinking cheaper (fewer parameters, lower precision).

However, existing MoE designs fail to achieve this vision because they are optimized for server-side deployment where total parameter count is a minor concern compared to FLOPs. The paper's "ideal triangle" (Figure 1) frames this as a simultaneous three-objective optimization that prior work has not attempted.

Where Prior Approaches Fall Short

The paper identifies four categories of prior work and diagnoses their limitations relative to the dense-comparability-under-matched-budget goal.

1. Conventional TopK routing with large expert pools. Architectures like Mixtral (Jiang et al., 2024), DeepSeek-V3 (Liu et al., 2024), and DeepSeek-MoE (Dai et al., 2024) use TopK routing: each token is dispatched to exactly KK experts, determined by the highest KK router logits after Softmax or Sigmoid normalization. These models achieve strong performance but have two shortcomings for the paper's setting:

  • Non-differentiability: The TopK operation is a hard selection β€” it discards gradient flow to non-selected experts, which can create training inefficiencies and makes sparsity control a discrete optimization problem rather than a continuous one.
  • Uniform activation ratio: Every token activates exactly KK experts, regardless of its difficulty or the content. This is inefficient β€” some tokens may genuinely need more expert computation, others less. But TopK cannot adapt.
  • Massive total parameters: To achieve low active computation with TopK, the total number of experts must be large (since active-per-token is fixed at KK). For example, a TopK MoE with K=2K=2 and 20% activation ratio implies 10 total experts per layer, which may be manageable, but as KK grows or the target activation ratio drops, the total parameter count balloons. This contradicts the storage objective.

2. Load balancing via auxiliary losses. The standard approach for ensuring uniform expert utilization in TopK MoE is an auxiliary loss (Fedus et al., 2022) that penalizes imbalanced routing. The paper acknowledges that auxiliary losses can interfere with the primary language modeling objective β€” a well-known tension in MoE training. DeepSeek-V3 partially addressed this with a loss-free load-balancing policy (Wang et al., 2024a), but the paper suggests that ReLU-based routing coupled with sparsity regularization provides a cleaner solution by naturally inducing sparsity through a differentiable mechanism rather than relying on hard constraints with auxiliary balancing.

3. ReLU-based routing in prior work. Two prior systems use ReLU for routing: ReMoE (Wang et al., 2024b) and BlockFFN (Song et al., 2025b). ReLU routing has inherent advantages: it is fully differentiable (gradients flow through all router logits, zero or not), it naturally induces sparsity (the ReLU zeroes out negative logits), and it enables token-dependent activation ratios (some tokens activate more experts, others fewer, depending on how many logits are positive). The paper builds on this foundation but identifies specific inadequacies:

  • ReMoE: Uses ReLU routing with adaptive L1-norm sparsity regularization. However, the paper finds that coupling ReLU routing with standard SiLU-activated experts leads to unstable training dynamics β€” specifically, a surging routed-expert activation ratio (Figure 4) and vanishing SiLU output magnitudes (Figure 6). These issues force the sparsity regularization to work harder (higher coefficient, as shown in Figure 5), which degrades performance.
  • BlockFFN: Designed for chunk-level speculative decoding acceleration with additional regularization terms (chunk sparsification loss and activation locality loss) that target union sparsity across consecutive tokens. The paper strips these objectives away (Appendix B, Table 7) for fair comparison and finds that its base architecture, like ReMoE, lacks the expert-side stabilization that DECO introduces.

4. Expert design: gated MLP as the default. Since Shazeer (2020), the SwiGLU architecture (SiLU-gated MLP) has been the de facto standard for Transformer FFN layers, and MoE models have largely inherited this design. However, the paper presents a novel empirical finding: gated MLP experts are fundamentally incompatible with ReLU-based routing. As shown in Table 3 and Figure 7, when DECO uses gated experts, the routed-expert activation ratio destabilizes dramatically during training. The paper hypothesizes that the multiplicative interactions in gated MLPs (the element-wise product of the gate and the up-projection outputs) produce highly dynamic output scales that send massive gradient signals back to the router. Because ReLU-based routing couples activation directly to the logit threshold (unlike TopK's hard selection, which masks logit instability), this gradient surge directly destabilizes the activation ratio.

Critically, this interaction is specific to ReLU-based routing. The paper shows that for TopK routing (DeepSeek-V3), gated vs. non-gated experts make little difference β€” the hard TopK selection insulates the routing decision from logit fluctuations. This explains why prior MoE research, which predominantly used TopK, never identified gating as a problem.

How This Paper Positions Itself

DECO positions itself at the intersection of three research threads, integrating and refining insights from each:

  • From the ReLU-routing literature (ReMoE, BlockFFN): It adopts the core insight that ReLU provides differentiable, token-adaptive sparsity, but addresses the training instability that prior work left unresolved.
  • From the expert-design literature (DeepSeek-V3, DeepSeek-MoE): It incorporates shared experts and learnable router scaling (extending DeepSeek-V3's scalar factor to per-expert vectors) but reconceives expert design from the ground up β€” replacing gated MLPs with non-gated ones and replacing vanilla SiLU with NormSiLU.
  • From the sparsity-control literature: It builds on ReMoE's adaptive coefficient mechanism but replaces L1-norm sparsification with router entropy, which the paper finds more numerically stable.

The paper's theoretical contribution (Section 3.2, Appendix E) derives a gradient-bounding argument for why NormSiLU stabilizes training: by normalizing the pre-activation distribution, NormSiLU constrains the Frobenius norm of the gradient with respect to the up-projection weights to be bounded by O(∣∣Wup∣∣Fβˆ’1)O(||W_{up}||_F^{-1}), preventing gradient explosion while maintaining differentiability. This is not just an empirical trick β€” it has a principled justification rooted in the Jacobian structure of layer normalization and SiLU.

Perhaps most importantly, the paper positions itself against the grain of MoE scaling research that treats total parameters as "free" (to be scaled up aggressively since only a fraction are active). By imposing the hard constraint of matched total parameters and training tokens, it forces a re-examination of architectural choices that are typically optimized for the unlimited-parameter regime. The paper's findings β€” that non-gated experts outperform gated ones with ReLU routing, that NormSiLU is necessary for stable training, that expert-wise scaling matters β€” only surface under this constraint. In a less constrained setting, these effects might be masked by simply adding more experts or training on more data.

This positioning is reinforced by the paper's final discussion (Section 6), which connects DECO to the literature on intrinsic activation sparsity in dense models (Song et al., 2025a; Luo et al., 2024; Zhang et al., 2024). These studies show that even in standard SwiGLU FFNs, only ~30–40% of neurons contribute meaningfully to any given token's output. The paper frames this as evidence that "dense models can be regarded as a special form of sparse MoE, where the SiLU-activated gating projection functions as a 'router', and neurons within up-down projections serve as 'experts'." Under this interpretation, DECO is not adding sparsity to a dense model β€” it is making the sparsity that already exists architecturally explicit and optimizable, which is why dense-comparable performance under matched budgets is achievable in principle.

3. Technical Approach

3.1 Reader orientation

DECO is a specific recipe for building a Mixture-of-Experts Transformer where the router and the experts are jointly redesigned so that training is stable and the model learns to activate only a small fraction of its experts per token without degrading performance. The paper solves the problem of how to make a sparse MoE match a dense model when both have exactly the same total number of parameters and exactly the same number of training tokens β€” a stricter goal than prior MoE work, which typically allowed MoE models to train on more data because their per-token computation was lower.

3.2 Big-picture architecture

A DECO model is a standard decoder-only Transformer with one modification: in place of a dense feed-forward network, most layers contain an MoE block with the following components. First, a ReLU-based router takes the token's hidden representation and produces a non-negative score for each expert; only experts receiving a positive score are activated. Second, learnable expert-wise scaling factors multiply these router scores element-wise to calibrate each expert's contribution, accommodating the fact that different experts may have systematically different output magnitudes. Third, non-gated MLP experts process the token (one up-projection, a NormSiLU activation, one down-projection), avoiding the multiplicative gating interactions that destabilize ReLU-based routing. Fourth, NormSiLU β€” a dual-stage normalization prepended to the SiLU non-linearity β€” stabilizes the internal activation statistics of experts, preventing both the routed-expert activation ratio from surging and the SiLU outputs from vanishing. Fifth, an adaptive sparsity regularization term, computed from the entropy of the normalized router scores, is added to the training objective with a coefficient that automatically increases or decreases to keep the average activation ratio at a pre-specified target. All components are trained end-to-end with the language modeling loss plus the regularization term.

3.3 Roadmap for the deep dive

  • First, the ReLU-based router and the learnable expert-wise scaling factors, because they determine which experts process each token and with what weight, forming the core routing mechanism that the rest of the design depends on.
  • Second, the NormSiLU activation function and its dual-stage normalization, because the paper's central empirical finding is that ReLU-based routing fails when paired with standard SiLU experts, and NormSiLU is the proposed fix β€” understanding why requires knowing what NormSiLU does mechanistically.
  • Third, the choice of non-gated MLP experts over gated ones, because this interacts with both the router (gated experts destabilize ReLU-based routing) and the activation function (NormSiLU operates on the intermediate state of a non-gated expert).
  • Fourth, the adaptive sparsity regularization, because it is the control mechanism that keeps the activation ratio at the target level, and its design (entropy-based, auto-scaling coefficient) is informed by the stability properties that NormSiLU and non-gated experts provide.
  • Fifth, how the pieces compose into a full DECO expert block, with the formal equations and the SparseLinear operators that make inference efficient.

3.4 Detailed, sentence-based technical breakdown

This is primarily an architectural design paper whose core idea is that a sparse MoE can match a dense Transformer at matched total parameters and training tokens, but only if the router and expert designs are jointly optimized for activation stability β€” namely, by replacing TopK with ReLU-based routing, replacing gated MLP experts with non-gated ones using NormSiLU, and using adaptive entropy-based sparsity regularization to precisely control the activation ratio.

Data flow through a DECO MoE layer (pipeline in words)

Before diving into individual mechanisms, it is useful to trace what happens to a single token's hidden state as it passes through one MoE layer in DECO. Understanding the full pipeline makes each component's role clear.

Step 1: Router computation. The token's hidden state $x \in \mathbb{R}^{d_h}$ (where $d_h$ is the model's hidden dimension) is multiplied by a learned weight matrix $W_{\text{router}} \in \mathbb{R}^{d_h \times N_e}$, where $N_e$ is the number of routed experts. This produces $N_e$ raw logits, one per expert. A ReLU function is applied element-wise: any negative logit becomes zero, any positive logit is passed through unchanged. The result is a sparse vector of non-negative scores $\text{ReLU}(W_{\text{router}}^T x)$. This sparsification is what makes the routing "sparse" β€” experts with zero scores are not activated. Importantly, because ReLU is differentiable everywhere (unlike TopK's hard selection, which has zero gradient for non-selected experts), gradients flow through the router to all experts during backpropagation, even those that received zero activation on a given forward pass (the gradient is zero when the input is negative, but the path exists and is non-zero when the input becomes positive in other contexts).

Step 2: Expert-wise scaling. The sparse router scores are multiplied element-wise by a learned vector $\alpha \in \mathbb{R}^{N_e}$, producing scaled scores $p = \alpha \odot \text{ReLU}(W_{\text{router}}^T x)$. Each expert gets its own scalar coefficient that can grow or shrink during training to match the expert's typical output magnitude. This is the "learnable expert-wise router scaling" mechanism.

Step 3: Shared expert computation (always-on). In parallel with the routed path, the token's hidden state is also processed by a shared expert β€” a standard dense FFN (non-gated MLP with an activation function) that is always active for every token. The shared expert's output is added to the weighted sum of the routed experts' outputs. The shared expert is not part of the routing mechanism; it provides a "baseline" computation that ensures every token receives at least some processing even if few or no routed experts are activated. The paper follows DeepSeek-V3's shared expert design, with the shared expert's intermediate dimension $d_s$ being a configurable hyperparameter (Table 5 shows $d_s = 128$ for all DECO configurations, which is twice the routed expert dimension $d_e = 64$).

Step 4: Routed expert computation (sparse, only for activated experts). For each routed expert $e$ that received a positive scaled score $p_e > 0$, the token's hidden state $x$ undergoes a sparse linear operation with two learned weight matrices: an up-projection $W_{\text{up}}^{(e)} \in \mathbb{R}^{d_e \times d_h}$ that expands the hidden state to the expert's intermediate dimension $d_e$, and a down-projection $W_{\text{down}}^{(e)} \in \mathbb{R}^{d_h \times d_e}$ that projects back to the model's hidden dimension. Between the two projections, the NormSiLU activation is applied: the intermediate state is first centered by subtracting the cross-expert mean of the up-projection outputs (inter-expert mean normalization), then scaled to unit RMS norm along the expert dimension (intra-expert RMS normalization), and finally passed through the SiLU non-linearity $x \cdot \sigma(x)$. The output is a vector in $\mathbb{R}^{d_h}$.

Step 5: Weighted aggregation. The outputs of all activated routed experts are summed, each weighted by its scaled router score $p_e$. The shared expert's output is added to this sum. The result is the MoE layer's output, which is then passed through the standard residual connection and layer normalization before proceeding to the next Transformer layer (attention or the next MoE block).

Step 6: Sparsity regularization (training only). During training, the router entropy loss $\mathcal{L}_{\text{ent}}$ is computed from the normalized router scores (across all tokens in the batch) and added to the language modeling loss. The coefficient of this regularization term is adaptively adjusted each iteration to keep the average activation ratio β€” the fraction of routed experts with $p_e > 0$ across all tokens β€” at the user-specified target (e.g., 20%). At inference time, the regularization is not applied; the ReLU router naturally produces sparse activations based on the learned weights.

ReLU-based routing and learnable expert-wise scaling

What the router computes. The core routing equation in DECO is:

p=Ξ±βŠ™ReLU(WrouterTx)p = \alpha \odot \text{ReLU}(W_{\text{router}}^T x)

where $x \in \mathbb{R}^{d_h}$ is the input hidden state for a single token, $W_{\text{router}} \in \mathbb{R}^{d_h \times N_e}$ is a learned weight matrix mapping from the hidden dimension to one logit per routed expert, $\alpha \in \mathbb{R}^{N_e}$ is a learned vector of per-expert scaling factors, $\odot$ denotes element-wise multiplication, and $p \in \mathbb{R}^{N_e}$ is the final vector of routing weights, one per expert.

What it computes operationally: The hidden state $x$ is linearly projected to $N_e$ raw logits. The ReLU function zeroes out any expert whose logit is negative, retaining only positive logits unchanged. The result is a sparse vector where only experts with positive logits receive non-zero weights. The per-expert learnable coefficients $\alpha$ then scale each retained weight, producing the final routing scores $p$. An expert $e$ is "activated" for this token if and only if $p_e > 0$. The number of active experts varies per token β€” some tokens may activate many experts (if many logits are positive), others few (if most logits are negative).

Why ReLU instead of TopK: TopK routing selects exactly $K$ experts per token by taking the $K$ largest logits, setting all others to zero. This has three disadvantages that ReLU addresses. First, TopK is non-differentiable with respect to the selection decision β€” the gradient is zero for non-selected experts, so the router receives no signal about whether non-selected experts might have been better choices. ReLU is differentiable (the gradient is 0 when the input is negative and 1 when positive), so during backpropagation, the router weights that produced negative logits receive gradient pressure to increase if those experts would have been useful. Second, TopK enforces a uniform activation ratio across all tokens β€” every token activates exactly $K$ experts, regardless of whether the token is "easy" (requiring little computation) or "hard" (requiring more). ReLU allows token-dependent activation: the number of active experts is the number of positive logits, which can vary. Third, ReLU naturally induces sparsity β€” for any reasonable initialization and training trajectory, a substantial fraction of logits will be negative, so only a fraction of experts are activated on average β€” without requiring a hard selection operator.

Why learnable expert-wise scaling (and why vectorized, not scalar): The router scores $p_e$ serve two purposes: they determine which experts are activated (via the ReLU threshold), and they serve as weights when aggregating expert outputs. The second role creates a problem: different experts may produce outputs with systematically different magnitudes. If one expert's outputs are consistently larger than another's, and both receive similar router scores, the larger-magnitude expert will dominate the aggregated output regardless of whether it is actually more relevant to the current token.

DeepSeek-V3 addressed a related issue by introducing a single scalar scaling factor applied uniformly to all router scores, which balanced the overall contribution between the shared expert and the set of routed experts. DECO extends this from a single shared scalar to a per-expert learned vector $\alpha \in \mathbb{R}^{N_e}$. This allows each expert to have its own coefficient, which can grow or shrink during training to compensate for that expert's characteristic output scale.

The paper provides empirical justification for this design choice by measuring expert output norms on the trained DECO (Medium) model. In the first MoE layer, the average output norm across all experts is 0.23, but the maximum exceeds 1.14 β€” a nearly 5Γ— range. A single scalar could not compensate for this heterogeneity; only per-expert coefficients can.

Ablation evidence (Table 1): The paper compares three settings: "Fixed" (a constant scaling factor of 1.0, effectively no scaling), "Scalar" (a single learnable scalar shared by all experts, initialized to 0.1), and "DECO" (the full per-expert learnable vector, initialized to 0.1 for all entries). For the Small model, DECO achieves PPL 34.36 vs. Fixed at 34.43 and Scalar at 35.16. For Medium, DECO achieves PPL 27.74 vs. Fixed at 27.94 and Scalar at 27.92. The "Fixed" setting performs reasonably because the initial value of 0.1 was chosen well (Table 8 shows that initialization values around 0.1–0.25 work best). However, the "Scalar" setting underperforms, confirming that a single shared coefficient is insufficient β€” heterogeneity across experts requires per-expert adaptation.

Initialization (Table 8): The paper sweeps initialization values for $\alpha$ from 0.01 to 1.0 on the Small model. Performance is non-monotonic: PPL ranges from 34.26 (at 0.25) to 35.66 (at 0.025). The best results cluster around 0.1–0.25. All DECO main experiments use an initialization of 0.1.

How routing connects to the forward pass: The scaled router scores $p_e$ are used linearly: the routed MoE output for a token is $\sum_{e: p_e > 0} p_e \cdot \text{Expert}_e(x)$. The shared expert output is added separately without router weighting. The router scores themselves are not normalized (no Softmax, no Sigmoid) β€” the ReLU and scaling provide the only transformation. This is a deliberate departure from standard MoE designs that use Softmax to produce a probability distribution over experts. In DECO, the scores are unbounded non-negative weights, and the model must learn to produce appropriate magnitudes through the combination of the router weights $W_{\text{router}}$ and the scaling factors $\alpha$.

NormSiLU: dual-stage normalization before SiLU

The problem NormSiLU solves. The paper identifies two coupled pathologies that emerge when ReLU-based routing is paired with standard SiLU-activated experts (the "SiLU" ablation in Figures 4–6):

  1. Surging activation ratio: During early training, the fraction of routed experts that are active (have positive router logits) increases rapidly from the initialization level toward 40–50%, far exceeding the target of 20%. The adaptive sparsity regularization eventually pulls this back (Figure 4), but doing so requires a much higher regularization coefficient (Figure 5), which competes with the language modeling objective and degrades final performance (Table 2, "SiLU" row: Small PPL 35.69 vs. 34.36 for NormSiLU).

  2. Vanishing SiLU output magnitudes: The average absolute output of the SiLU activation inside experts drops to very small values (Figure 6, "SiLU" trace stabilizes near 0.2–0.3, compared to ~0.6–0.8 for NormSiLU). This means expert neurons are under-utilized β€” their outputs are so small that they contribute negligibly to the final token representation, wasting the parameters allocated to those experts.

NormSiLU addresses both issues simultaneously through a dual-stage normalization applied to the pre-activation intermediate state inside each expert.

NormSiLU algorithm (Algorithm 1, step by step). Given a token's hidden state $x \in \mathbb{R}^{d_h}$ and the up-projection weights $W_{\text{up}} \in \mathbb{R}^{N_e \times d_e \times d_h}$ for all $N_e$ experts, the computation for the activated experts proceeds as follows:

Step 1: Sparse up-projection. For each active expert, a sparse linear operation computes the intermediate state:

xup(e)=Wup(e)β‹…xx_{\text{up}}^{(e)} = W_{\text{up}}^{(e)} \cdot x

where $W_{\text{up}}^{(e)} \in \mathbb{R}^{d_e \times d_h}$ and $x_{\text{up}}^{(e)} \in \mathbb{R}^{d_e}$. For inactive experts, this computation is skipped entirely (this is what "sparse linear" means β€” only the rows of the weight tensor corresponding to active experts are loaded and multiplied).

Step 2: Inter-expert mean normalization. The average up-projection weight across all experts is pre-computed (and fixed during inference; it can be updated periodically during training):

WΛ‰up=1Neβˆ‘e=1NeWup(e)∈RdeΓ—dh\bar{W}_{\text{up}} = \frac{1}{N_e} \sum_{e=1}^{N_e} W_{\text{up}}^{(e)} \in \mathbb{R}^{d_e \times d_h}

The average intermediate state that would be produced if all experts were active is:

xavg=WΛ‰upβ‹…x∈Rdex_{\text{avg}} = \bar{W}_{\text{up}} \cdot x \in \mathbb{R}^{d_e}

This average is subtracted from the actual intermediate state of each active expert:

xupβ€²(e)=xup(e)βˆ’xavgx'^{(e)}_{\text{up}} = x_{\text{up}}^{(e)} - x_{\text{avg}}

What this achieves: The intermediate state for each expert is centered around the cross-expert mean. Without this step, the up-projection outputs could drift to systematically positive or negative values depending on the expert's learned weight pattern. The SiLU function $\text{SiLU}(z) = z \cdot \sigma(z)$ is roughly linear for large positive $z$ (since $\sigma(z) \approx 1$, so $\text{SiLU}(z) \approx z$) and zero for large negative $z$ (since $\sigma(z) \approx 0$, so $\text{SiLU}(z) \approx 0$). If the pre-activation values drift far from zero, SiLU either passes them through linearly (no non-linearity benefit) or squashes them to zero (vanishing activations). Centering around zero ensures that SiLU operates in its most non-linear, information-rich regime for a larger fraction of neurons.

Step 3: Intra-expert RMS normalization. The centered intermediate state is then normalized to unit RMS within each expert:

xupβ€²β€²(e)=RMSNorm(xupβ€²(e))x''^{(e)}_{\text{up}} = \text{RMSNorm}(x'^{(e)}_{\text{up}})

where RMSNorm is the standard root-mean-square normalization (dividing by the RMS of the vector, with a learnable scale parameter, but without subtracting the mean since the mean has already been handled by the inter-expert step):

RMSNorm(v)=v1deβˆ‘i=1devi2+Ο΅βŠ™Ξ³\text{RMSNorm}(v) = \frac{v}{\sqrt{\frac{1}{d_e} \sum_{i=1}^{d_e} v_i^2 + \epsilon}} \odot \gamma

with a learnable vector $\gamma \in \mathbb{R}^{d_e}$ and a small $\epsilon$ for numerical stability.

What this achieves: The activation magnitudes are normalized to a consistent scale across tokens and across training steps. This prevents the SiLU outputs from either exploding (causing training instability) or vanishing (causing under-utilization). The learnable scale parameter $\gamma$ allows the model to adjust the overall magnitude if needed, but the normalization provides a stable baseline.

Step 4: SiLU activation. The normalized, centered state is passed through the SiLU non-linearity:

yup(e)=xupβ€²β€²(e)β‹…Οƒ(xupβ€²β€²(e))y^{(e)}_{\text{up}} = x''^{(e)}_{\text{up}} \cdot \sigma(x''^{(e)}_{\text{up}})

Step 5: Down-projection. The activated state is projected back to the model's hidden dimension via a second sparse linear operation:

y(e)=Wdown(e)β‹…yup(e)y^{(e)} = W_{\text{down}}^{(e)} \cdot y^{(e)}_{\text{up}}

Why dual-stage rather than a single LayerNorm or RMSNorm: A single normalization across the entire concatenated expert dimension (all $N_e \cdot d_e$ values for all experts) would, in theory, provide similar statistical stabilization. The paper provides a theoretical justification for this in Appendix E, showing that a global LayerNorm bounds the gradient norm by $O(||W_{\text{up}}||_F^{-1})$, preventing gradient explosion. However, a global normalization is fundamentally incompatible with sparse inference: it requires materializing the intermediate states for all experts (to compute the global mean and variance), which defeats the computational advantage of MoE β€” the whole point is to avoid computing inactive experts.

NormSiLU's dual-stage design decomposes the full normalization into two parts that are compatible with sparsity. The inter-expert mean normalization uses only the fixed average weight $\bar{W}_{\text{up}}$, which can be pre-computed once and reused β€” it does not require computing the actual intermediate states of inactive experts. The intra-expert RMS normalization is applied independently within each active expert, using only that expert's own intermediate state. Together, they approximate the effect of a global normalization while preserving the $O(\text{active parameters})$ computational complexity of MoE inference.

Ablation evidence (Table 2, Figures 4–6). The paper tests three ablations of NormSiLU: "w/o Mean" (removes inter-expert mean normalization, keeps intra-expert RMSNorm), "w/o RMS" (removes intra-expert RMSNorm, keeps inter-expert mean normalization), and "SiLU" (vanilla SiLU, no normalization). Key findings:

  • Figure 4 (activation ratio trend): "SiLU" and "w/o RMS" show surging activation ratios that peak around 45–50% before regularization pulls them back. "w/o Mean" and NormSiLU show stable, nearly flat trajectories. The intra-expert RMS normalization is primarily responsible for preventing the activation ratio surge.
  • Figure 5 (regularization coefficient): "SiLU" and "w/o RMS" require coefficients that are 2–5Γ— larger than "w/o Mean" and NormSiLU to achieve the same target sparsity. Since higher regularization competes with the language modeling loss, this directly harms performance.
  • Figure 6 (SiLU output magnitudes): "SiLU" and "w/o Mean" show vanishing output magnitudes (stabilizing at ~0.2–0.3). "w/o RMS" and NormSiLU show healthy, larger magnitudes (~0.6–0.8). The inter-expert mean normalization is primarily responsible for preventing vanishing activations.
  • Table 2 (final performance): On Small, NormSiLU achieves PPL 34.36; "w/o Mean" achieves 34.57; "w/o RMS" achieves 35.77; "SiLU" achieves 35.69. Both normalization steps contribute, with RMS normalization providing a larger absolute gain (1.41 PPL improvement over "w/o RMS") but mean normalization also being important (0.21 PPL improvement over "w/o Mean"). The full NormSiLU is the best configuration.

The causal mechanism linking NormSiLU to activation ratio stability: The paper does not spell this out explicitly, but the mechanism can be inferred from the data. When expert activations vanish (low SiLU output magnitudes), the expert outputs become very small. The router, which receives gradient signals through the expert outputs, gets weak or noisy gradients about which experts are useful. In response, the router may increase logits across the board (trying to amplify the weak signals), which drives more experts above the ReLU threshold β€” hence the surging activation ratio. NormSiLU prevents the vanishing in the first place, so the router receives clean, well-scaled gradient signals, and the activation ratio remains stable. This interpretation is consistent with the observation that "w/o Mean" (which still has vanishing activations, per Figure 6) shows a stable activation ratio in Figure 4 β€” but this is because the RMSNorm in "w/o Mean" normalizes the already-vanished activations to unit RMS, which actually amplifies noise rather than recovering useful signal, leading to the poor final PPL (34.57 vs. 34.36).

Non-gated MLP experts vs. gated experts

What a gated MLP expert is. In standard SwiGLU (Shazeer, 2020), a feed-forward layer computes:

SwiGLU(x)=(Wgateβ‹…xβ‹…SiLU(Wupβ‹…x))β‹…WdownT\text{SwiGLU}(x) = (W_{\text{gate}} \cdot x \cdot \text{SiLU}(W_{\text{up}} \cdot x)) \cdot W_{\text{down}}^T

where $W_{\text{gate}}$ and $W_{\text{up}}$ are two separate up-projection matrices, their outputs are multiplied element-wise (the "gate" controls how much of the up-projected signal passes through), and the result is projected back down. This introduces a multiplicative interaction between two linear projections of the input.

What a non-gated MLP expert is. In a non-gated MLP (which DECO uses):

MLP(x)=NormSiLU(Wupβ‹…x)β‹…WdownT\text{MLP}(x) = \text{NormSiLU}(W_{\text{up}} \cdot x) \cdot W_{\text{down}}^T

There is only one up-projection, no gating multiplication, and the activation function is applied directly to the intermediate state. The computation is simpler, with fewer parameters for the same intermediate dimension, and β€” critically β€” has no multiplicative interactions between parallel projections.

The empirical finding (Table 3): gated experts destabilize ReLU-based routing. The paper tests three architectures β€” DeepSeek-V3 (TopK routing), ReMoE (ReLU routing), and DECO (ReLU routing + NormSiLU + expert scaling) β€” each with both gated and non-gated experts, matched for total parameters. The results reveal a stark interaction:

  • For TopK routing (DeepSeek-V3): Gated vs. non-gated is nearly neutral. At Small, PPL is 35.09 (gated) vs. 35.16 (non-gated); at Medium, 28.19 vs. 28.28. The difference is small and slightly favors gated, consistent with the established SwiGLU advantage in dense models. TopK's hard selection of exactly $K$ experts masks any router instability caused by gated expert dynamics.

  • For ReLU routing (ReMoE and DECO): Gated experts are dramatically worse. At Small, DECO (gated) achieves PPL 39.77 vs. DECO (non-gated) at 34.36 β€” a 5.41 PPL gap. ReMoE shows the same pattern: PPL 36.02 (gated) vs. 35.13 (non-gated) at Small. The gap persists across scales (Medium: 32.46 vs. 27.74 for DECO; Large: 22.13 vs. 21.87 for DECO). Non-gated MLP experts are strongly preferred for ReLU-based routing.

The hypothesized mechanism (Section 4.4, Figure 7). The paper attributes this to gradient dynamics. In a gated MLP, the output depends on the product of the gate projection and the up-projection. Small changes in the input can produce large changes in the output because the two projections can interact multiplicitively β€” the output scale is not a simple function of the input norm. These dynamic output scales send high-variance gradient signals back through the router (since the router's gradient depends on the expert outputs). In a TopK MoE, the TopK selection insulates the router: only the top $K$ experts are selected regardless of their logit magnitudes, so logit fluctuations do not change which experts are active. In a ReLU-based MoE, the activation decision is directly coupled to the logit value (positive = active, negative = inactive), so any logit fluctuation directly changes the activation pattern.

Figure 7 visualizes this: DECO (GA β€” gated) shows a wildly unstable activation ratio that surges from ~20% to ~45% before being aggressively pulled back by regularization. DECO (NG β€” non-gated) shows a flat, stable trajectory at the target ratio. The gated variant requires much more regularization to control the instability, and that regularization directly hurts the language modeling objective.

Additional nuance: why TopK MoE can use gated experts. The paper's finding does not contradict the established superiority of SwiGLU in dense models or TopK MoE models. The instability is specific to the combination of ReLU-based routing (where activation depends on logit thresholds) and gated MLPs (which produce volatile output scales). DeepSeek-V3, which uses TopK, can safely use gated experts. The paper's contribution is identifying this interaction as a design constraint for ReLU-based MoE architectures, which prior work (ReMoE, BlockFFN) did not recognize or address.

Parameter efficiency of non-gated vs. gated. A gated MLP with intermediate dimension $d_e$ has $2 \cdot d_h \cdot d_e$ up-projection parameters (gate + up) and $d_h \cdot d_e$ down-projection parameters, totaling $3 \cdot d_h \cdot d_e$. A non-gated MLP has $d_h \cdot d_e$ up-projection parameters and $d_h \cdot d_e$ down-projection parameters, totaling $2 \cdot d_h \cdot d_e$. For the same total parameter budget, non-gated experts can be either wider (larger $d_e$) or more numerous (more experts $N_e$). The paper matches total parameters by adjusting other dimensions β€” Table 5 shows all DECO configurations use $d_e = 64$ for routed experts, and the gated vs. non-gated comparisons in Table 3 control for total parameter count.

Adaptive sparsity regularization

Why regularization is needed. ReLU-based routing naturally induces sparsity (negative logits become zero), but the level of sparsity β€” what fraction of experts are active on average β€” is not guaranteed to hit any particular target. Without explicit control, the activation ratio could drift to undesirable values: too high (wasting the computational savings of MoE) or too low (losing representation power). The regularization term provides a differentiable signal that encourages the router to produce the desired sparsity level.

What the router entropy loss computes. The router entropy is defined as:

p1=∣p∣Sum(∣p∣)p_1 = \frac{|p|}{\text{Sum}(|p|)}

Lent=βˆ’p1Tln⁑(p1+Ο΅)\mathcal{L}_{\text{ent}} = -p_1^T \ln(p_1 + \epsilon)

where $p \in \mathbb{R}^{N_e}$ is the vector of scaled router scores from Equation 1, $|p|$ denotes element-wise absolute value, $\text{Sum}(|p|)$ is the sum of the absolute values of all router scores, $p_1 \in \mathbb{R}^{N_e}$ is the normalized probability distribution over experts (non-negative, summing to 1), $\epsilon$ is a small constant for numerical stability, and $\mathcal{L}_{\text{ent}}$ is a scalar entropy value.

What it computes operationally: First, the absolute values of the router scores are normalized to form a probability distribution $p_1$ β€” this ensures that experts with higher router scores (whether positive or negative, hence the absolute value) contribute more to the entropy. Second, the entropy of this distribution is computed as the negative dot product of $p_1$ with its element-wise natural logarithm. Entropy is maximized when all $N_e$ experts have equal probability ($p_1^{(e)} = 1/N_e$ for all $e$), and minimized (approaches 0) when one expert has probability 1 and all others have probability 0.

Why entropy rather than L1 norm (as used in ReMoE): ReMoE uses an L1-norm sparsification loss on the router scores: $\mathcal{L}_{\text{sparse}} = ||p||_1$. Minimizing the L1 norm encourages small router scores, which drives more logits negative (since ReLU zeroes out negatives, a smaller L1 norm implies fewer positive logits, hence higher sparsity). However, L1 regularization can interact poorly with the adaptive coefficient mechanism β€” as the coefficient increases, the L1 loss can drive all scores toward zero, making the activation ratio collapse. Entropy provides a smoother, more numerically stable signal: high entropy means the router is distributing weight across many experts (low sparsity), and the regularization term penalizes this by adding $\lambda \mathcal{L}_{\text{ent}}$ to the training objective, where $\lambda$ is the adaptive coefficient. The paper states that entropy is chosen "to improve numerical stability" compared to L1.

Adaptive coefficient mechanism. Rather than using a fixed coefficient $\lambda$, which would require careful tuning and might not maintain a stable sparsity level as training dynamics evolve, DECO adaptively adjusts $\lambda$ each iteration:

Ξ»t+1={Ξ»tβ‹…Ξ·ifΒ currentΒ sparsity<targetΒ sparsityΞ»t/Ξ·ifΒ currentΒ sparsityβ‰₯targetΒ sparsity\lambda_{t+1} = \begin{cases} \lambda_t \cdot \eta & \text{if current sparsity} < \text{target sparsity} \\ \lambda_t / \eta & \text{if current sparsity} \geq \text{target sparsity} \end{cases}

where "current sparsity" is the fraction of routed experts with zero activation ($p_e = 0$), averaged over the training batch, "target sparsity" is the user-specified level (e.g., 0.80, corresponding to a 20% activation ratio), and $\eta > 1$ is a multiplier (set to 1.002 for Small/Medium and 1.001 for Large/XLarge; see Table 5). The initial coefficient $\lambda_{\text{init}}$ is $1 \times 10^{-8}$ for all models.

What this does: If the model is activating too many experts (sparsity below target), $\lambda$ is multiplied by $\eta$, increasing the penalty on entropy and pushing the router toward more concentrated, sparser distributions. If sparsity exceeds the target (too few experts active), $\lambda$ is divided by $\eta$, relaxing the penalty and allowing the router to activate more experts. Over training, $\lambda$ oscillates around the value needed to maintain the target sparsity, with the oscillation amplitude determined by $\eta$ (smaller $\eta$ = tighter control but slower adaptation).

How the regularization connects to the total loss. The full training objective is:

Ltotal=LLM+Ξ»β‹…Lent\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{LM}} + \lambda \cdot \mathcal{L}_{\text{ent}}

where $\mathcal{L}_{\text{LM}}$ is the standard cross-entropy language modeling loss, $\mathcal{L}_{\text{ent}}$ is the router entropy defined above, and $\lambda$ is the adaptively updated coefficient. Only $\lambda$ is updated by the adaptive rule; the router weights $W_{\text{router}}$ and scaling factors $\alpha$ are updated by the optimizer (AdamW) based on the gradient of $\mathcal{L}_{\text{total}}$.

The interaction with NormSiLU and non-gated experts. The adaptive regularization works because NormSiLU and non-gated experts provide a stable baseline activation trend. If the activation ratio were wildly unstable (as in Figure 7, "GA" trace), the adaptive coefficient would need to swing dramatically to compensate, potentially overshooting and causing training instability. The stability provided by NormSiLU (Figure 4) and non-gated experts (Figure 7) means the regularization only needs to make small corrections, keeping $\lambda$ small (Figure 5 shows NormSiLU's coefficient stabilizes at ~0.01–0.02) and minimizing interference with the language modeling objective.

The regularization at inference time. The router entropy loss is applied only during training. At inference, the router produces sparse activations based on the learned weights, and no regularization gradient is computed. The activation ratio at inference may differ slightly from the training value due to distribution shift between training and inference data, but the paper's results (Figure 3) show that performance is robust.

Full DECO expert block equations

Putting all components together, the MoE layer output for a token with hidden state $x \in \mathbb{R}^{d_h}$ is computed as follows.

Router:

p=Ξ±βŠ™ReLU(WrouterTx)p = \alpha \odot \text{ReLU}(W_{\text{router}}^T x)

where $W_{\text{router}} \in \mathbb{R}^{d_h \times N_e}$ and $\alpha \in \mathbb{R}^{N_e}$.

Routed expert computation (for each active expert $e$ with $p_e > 0$):

xup(e)=SparseLinear(x,Wup(e))x_{\text{up}}^{(e)} = \text{SparseLinear}(x, W_{\text{up}}^{(e)})

xupβ€²(e)=NormSiLU(x,Wup,xup(e))x'^{(e)}_{\text{up}} = \text{NormSiLU}(x, W_{\text{up}}, x_{\text{up}}^{(e)})

y(e)=SparseLinear(xupβ€²(e),Wdown(e))y^{(e)} = \text{SparseLinear}(x'^{(e)}_{\text{up}}, W_{\text{down}}^{(e)})

where $W_{\text{up}} \in \mathbb{R}^{N_e \times d_e \times d_h}$ is the tensor of all experts' up-projection weights, $W_{\text{up}}^{(e)} \in \mathbb{R}^{d_e \times d_h}$ is the slice for expert $e$, $W_{\text{down}} \in \mathbb{R}^{N_e \times d_h \times d_e}$ is the down-projection tensor, and SparseLinear denotes a matrix multiplication that only loads and computes with the weight rows/columns for active experts.

Note on NormSiLU's interface: The function NormSiLU(x, W_up, x_up) takes three arguments: the original input $x$ (needed to compute the inter-expert mean normalization via the average weight $\bar{W}_{\text{up}}$), the full up-projection weight tensor $W_{\text{up}}$ (to compute the mean), and the specific expert's intermediate state $x_{\text{up}}^{(e)}$. During inference, $\bar{W}_{\text{up}}$ is pre-computed, so the normalization only requires multiplying $x$ by this fixed average matrix (an $O(d_h \cdot d_e)$ operation shared across all active experts for this token) and then applying intra-expert RMSNorm per active expert.

Shared expert computation (always active):

yshared=FFNshared(x)y_{\text{shared}} = \text{FFN}_{\text{shared}}(x)

where $\text{FFN}_{\text{shared}}$ is a standard dense feed-forward network, presumably with a non-gated MLP structure and an activation function. The paper states (Section 3.2, Equation 2 description) that shared experts are used but does not specify their exact internal architecture beyond indicating they are a standard FFN.

Aggregation:

yMoE=yshared+βˆ‘e:pe>0peβ‹…y(e)y_{\text{MoE}} = y_{\text{shared}} + \sum_{e: p_e > 0} p_e \cdot y^{(e)}

Residual connection (standard Transformer):

xout=x+yMoEx_{\text{out}} = x + y_{\text{MoE}}

followed by layer normalization before the next sub-layer (attention or the next MoE block).

Training-only: sparsity regularization. During training, the router entropy loss $\mathcal{L}_{\text{ent}}$ is computed as described above and added to the total loss, with the coefficient $\lambda$ adaptively updated per iteration.

Summary of design choices and their justifications

  • ReLU-based routing over TopK: Differentiable (gradients flow to all experts), token-adaptive activation ratios, naturally induces sparsity without hard selection. The cost is that activation ratios must be controlled via regularization rather than being fixed by construction, which motivates the adaptive sparsity regularization design.
  • Learned expert-wise vector scaling over fixed scalar: Accommodates systematic heterogeneity in expert output magnitudes (empirically, a ~5Γ— range in output norms), preventing some experts from dominating regardless of relevance.
  • Non-gated MLP experts over gated (SwiGLU): Essential for stable training with ReLU-based routing β€” the multiplicative interactions in gated MLPs produce volatile output scales that destabilize router gradients and cause activation ratio surges. For TopK routing, this is not an issue because the hard selection masks logit instability.
  • NormSiLU over vanilla SiLU: The inter-expert mean normalization centers pre-activations around zero, preventing SiLU output vanishing and keeping expert neurons well-utilized. The intra-expert RMS normalization stabilizes the activation ratio trend, reducing the required regularization strength. The dual-stage decomposition is necessary because a single global normalization across all experts would violate the sparse computation constraint of MoE inference.
  • Router entropy over L1-norm regularization: More numerically stable β€” entropy provides a smoother optimization landscape that interacts better with the adaptive coefficient mechanism, avoiding the collapse-to-zero problem that L1 regularization can cause.
  • Adaptive coefficient over fixed coefficient: Maintains precise sparsity control across training without manual tuning. The multiplier $\eta$ is kept close to 1 (1.001–1.002) to ensure slow, stable adaptation. The small initial coefficient ($1 \times 10^{-8}$) means the regularization starts negligible and only grows if sparsity deviates from the target, preventing early-training interference with representation learning.
  • Shared expert in every MoE layer: Ensures every token receives a baseline computation regardless of routing decisions, preventing degenerate cases where a token activates zero routed experts. Following DeepSeek-V3, the shared expert is separate from the routed experts and its output is added unweighted.
  • Dense FFN in the first layer: Table 5 notes that "All MoE baseline settings and DECO use a dense FFN at the first layer, and adopt sparse MoE for the remaining $N_{\text{layer}} - 1$ layers." This is a common practice in MoE Transformers: the first layer's representations are not yet rich enough for meaningful expert specialization, so a dense FFN is used to bootstrap the representations before the first MoE layer.

4. Key Insights and Innovations

Innovation 1: Diagnosing a specific architectural incompatibility β€” gated MLP experts and ReLU-based routing β€” that prior work overlooked entirely

The paper's most conceptually distinctive move is not proposing a new component, but rather identifying a hidden interaction between two design choices that the field treated as independent. The standard MoE architecture stacks two well-established building blocks: TopK routing (from the original MoE literature) and SwiGLU gated MLP experts (from Shazeer, 2020). When researchers began exploring ReLU-based routing as a differentiable alternative to TopK (ReMoE, BlockFFN), they carried over the gated MLP expert design without questioning the compatibility β€” because, after all, SwiGLU had been established as strictly superior to non-gated MLPs in dense Transformers, and TopK MoE models like DeepSeek-V3 had used gated experts successfully.

DECO's fundamental insight is that this compatibility assumption is false. Section 4.4 (Table 3, Figure 7) demonstrates that gated MLP experts, when paired with ReLU-based routing, produce a catastrophic destabilization of the activation ratio that is entirely absent with TopK routing. The mechanism β€” multiplicative interactions in the gated MLP producing volatile output scales that send high-variance gradient signals back to the router, which then destabilizes the logit threshold that controls ReLU activation β€” is specific to the ReLU + gated combination. TopK masks this instability because its hard selection of exactly K experts is invariant to logit fluctuations, making the choice of gating largely neutral (Table 3, DeepSeek-V3 rows: gated PPL 35.09 vs. non-gated 35.16 at Small).

This is a diagnostic insight, not an incremental refinement. The paper is not saying "non-gated experts are generally better" β€” they are worse for TopK routing, consistent with the SwiGLU literature. Rather, it is saying: the interaction between router type and expert type matters enormously, and it has been invisible to the field because prior ReLU-MoE work (ReMoE with PPL 36.02 gated vs. 35.13 non-gated; BlockFFN, which used non-gated experts but for chunk-sparsity acceleration reasons, not stability reasons) did not systematically ablate this interaction. The 5.41 PPL gap between DECO (gated) and DECO (non-gated) at the Small scale (Table 3) is not a subtle effect β€” it is an existential one, making the difference between an MoE that works and one that fails catastrophically.

Significance beyond performance: This finding redefines the design space for MoE architectures. It means that the choice of router and the choice of expert are not independent optimization axes β€” they are coupled through gradient dynamics that depend on the router's activation mechanism. Future ReLU-based (or threshold-based) routing designs cannot simply inherit the expert architecture from TopK MoE literature; they must either use non-gated experts or solve the gradient instability problem through other means. The paper does not claim to have fully characterized why gated experts produce this instability (the gradient explanation in Section 4.4 is described as a hypothesis, not a proof), but the empirical demonstration that the interaction exists and is of first-order importance is a genuine contribution to MoE understanding.

Evidence anchors: Table 3 (the 5.41 PPL gap for DECO gated vs. non-gated at Small), Figure 7 (the wildly unstable activation ratio for DECO with gated experts), and the contrast with DeepSeek-V3 (gated vs. non-gated nearly identical under TopK).

Innovation 2: Demonstrating that normalization inside experts controls router-level dynamics, revealing a previously invisible coupling between expert-internal statistics and routing behavior

Prior work on MoE training stability focused primarily on the router: load-balancing auxiliary losses (Fedus et al., 2022), loss-free balancing (Wang et al., 2024a), and sparsity regularization (Wang et al., 2024b; Song et al., 2025b). The implicit assumption was that expert-internal dynamics β€” the activation functions, normalization schemes, and output magnitudes inside each expert β€” were an independent concern, affecting representation quality but not routing stability.

DECO overturns this assumption through a clean ablation of NormSiLU (Section 4.3, Figures 4–6). The paper shows that replacing vanilla SiLU with NormSiLU inside experts directly controls two router-level phenomena: the surging activation ratio (Figure 4) and the required regularization strength (Figure 5). Specifically, removing the intra-expert RMS normalization from NormSiLU causes the activation ratio to surge from ~20% to ~45% during early training, forcing the adaptive regularization coefficient to increase 2–5Γ— to compensate β€” and this increased regularization directly degrades final performance (PPL 35.77 for "w/o RMS" vs. 34.36 for full NormSiLU at Small). Removing the inter-expert mean normalization causes vanishing SiLU output magnitudes (Figure 6), leading to under-utilized expert parameters and degraded performance (PPL 34.57 for "w/o Mean").

The conceptual leap is recognizing that these are not two separate problems (router instability and expert under-utilization) but two symptoms of the same underlying mechanism: when expert activations vanish or drift, the gradient signal back to the router becomes weak or noisy, destabilizing the logit landscape that the ReLU threshold operates on. By normalizing the expert-internal statistics, NormSiLU stabilizes the gradients flowing back to the router, which in turn stabilizes the activation ratio, which in turn reduces the needed regularization strength, which in turn preserves performance. The causal chain is: expert-internal statistics β†’ router gradients β†’ activation ratio β†’ regularization burden β†’ performance.

This is not merely an engineering trick. The paper provides a theoretical justification (Appendix E) showing that a global normalization would bound the up-projection gradient norm by O(||W_up||_F^{-1}), preventing gradient explosion. The key design move β€” decomposing the global normalization into inter-expert mean and intra-expert RMS stages β€” is what makes this compatible with sparse inference (a global normalization would require materializing all experts' intermediate states, defeating the purpose of MoE sparsity). So NormSiLU is not just "normalization helps training" β€” it is a specific, principled decomposition that achieves what a global normalization would achieve while preserving the O(active parameters) inference cost of MoE.

Significance beyond performance: This finding establishes that expert-internal design is a first-class lever for controlling MoE training dynamics, not a secondary concern. It implies that future MoE architectures should co-design the router's activation mechanism and the experts' internal normalization, rather than treating them as independent modules. It also provides a diagnostic framework: if an MoE exhibits unstable activation ratios during training, the root cause may not be in the router or the regularization β€” it may be in the expert activation function's statistics. This is a new diagnostic concept for MoE debugging that the field did not have before.

Evidence anchors: Figure 4 (activation ratio trends for NormSiLU vs. ablations), Figure 5 (regularization coefficient trends), Figure 6 (SiLU output magnitudes), Table 2 (final performance for each ablation), and Appendix E (theoretical gradient bound).

Innovation 3: Reframing dense-comparability as an architectural optimization problem under a strict matched-budget constraint, revealing that existing MoE architectures are over-designed for the unlimited-parameter regime

Most MoE research evaluates architectures under a paradigm where total parameters are treated as "cheap" β€” the goal is to maximize performance for a given active parameter budget or FLOPs budget, with total parameter count being a secondary concern (scaling laws work by Krajewski et al., 2024; Tian et al., 2025 follows this pattern). Under this paradigm, the optimal design tends toward large numbers of small experts, aggressive sparsity (e.g., Top-2 out of 64 experts), and substantial total parameter counts that dwarf the active count. This makes sense for server-side deployment where memory is abundant and the primary cost is FLOPs.

DECO's reframing β€” asking whether an MoE can match a dense model under identical total parameters and identical training tokens β€” inverts the optimization criteria. Under this constraint, the design space is radically different. You cannot compensate for architectural inefficiencies by adding more experts (total parameter budget is fixed). You cannot compensate for slower learning dynamics by training on more data (token budget is fixed). Every design choice must earn its parameter and data cost against a dense baseline that uses the same budget in a uniform, non-sparse way.

This reframing is what surfaces the architectural insights that the paper's other innovations depend on. In the unlimited-parameter regime, the instability of gated experts with ReLU routing might be masked by simply adding more experts, or by training longer, or by accepting a higher regularization coefficient because the extra parameters provide headroom. Under the matched-budget constraint, these pathologies become existential β€” they directly prevent the model from reaching dense parity. Similarly, the need for NormSiLU and expert-wise scaling would be less apparent if the model could simply throw more experts at the problem.

This is a framing innovation rather than a technical one β€” but it is an important one because it opens a new research question that the MoE field had not systematically asked. The paper's discussion (Section 6) connects this to the literature on intrinsic activation sparsity in dense models, suggesting that dense-comparability is achievable in principle because dense models already exhibit sparse activation patterns (~30–40% of neurons active per token). DECO is, in this view, not adding sparsity to a dense model but making the existing sparsity architecturally explicit. This is a conceptual bridge between the MoE and activation sparsity literatures that prior work had not drawn.

What this is not: This is not a theoretical scaling law or a formal proof of dense-comparability. It is an empirical demonstration, at modest scales (0.11B–1.18B), that dense-comparability is achievable under the matched-budget constraint with the right architectural choices. The paper does not claim that DECO will match dense models at all scales β€” indeed, Figure 8 raises the possibility that the required activation ratio for parity decreases with scale, which, if it holds, would mean DECO becomes more efficient as models grow, but this is stated as a hypothesis requiring further investigation, not a claim.

Significance beyond performance: The matched-budget framing provides a new benchmark for MoE architecture evaluation that is directly relevant to edge deployment. It also suggests that the MoE literature's focus on massive expert pools and server-side scaling may have led to architectural choices (like gated experts and complex load-balancing schemes) that are suboptimal for the constrained regime. The paper is, in effect, arguing that MoE architectures should be evaluated at matched total parameters, not just matched active parameters β€” a methodological contribution to how the field should compare architectures.

Evidence anchors: Figure 3 (DECO matches or exceeds Dense across four scales at 20% activation), Figure 8 (the activation ratio required for dense parity decreases from 15% at Small to 10% at Medium), and the explicit contrast with Li et al. (2025) in Section 1 (which allowed MoE to train on more tokens under the same computation budget).

Innovation 4: A practical demonstration that architectural sparsity translates to real-device speedup on edge hardware, closing the loop from theoretical efficiency to measured inference throughput

Many MoE papers report theoretical FLOPs reductions (fewer active parameters per token) but do not demonstrate that these reductions translate to wall-clock speedups on actual hardware β€” particularly edge hardware, where memory bandwidth, cache hierarchy, and interconnection constraints can make sparse computation less efficient in practice than on paper. The gap between theoretical FLOPs and measured throughput is a well-known challenge in sparse model deployment: loading expert parameters from memory, managing irregular memory access patterns, and kernel launch overheads can erode the computational savings.

DECO's acceleration kernel (Section 5, Table 4) demonstrates that the sparsity-induced savings are real and substantial on two devices: a server-class NVIDIA RTX 4090 (2.56Γ— speedup over dense inference) and an edge device, NVIDIA Jetson AGX Orin (2.93Γ— speedup). The higher speedup on the Jetson is notable β€” it suggests that the memory-access savings from sparse activation are relatively more beneficial on bandwidth-constrained edge hardware, which aligns with the paper's motivation.

The kernel's design choices matter: it uses CUTLASS (Thakkar et al., 2023) to leverage tensor cores, and it replaces TopK's vector-wise sorting with DECO's element-wise ReLU activation, which has lower computational complexity. The paper reports that the ReLU-based kernel outperforms a TopK baseline at the same 20% activation ratio β€” a finding that directly validates the architectural choice of ReLU over TopK for inference efficiency, not just for training differentiability.

This is an engineering contribution rather than a conceptual one, but it is significant because it closes the argument: the paper's central claim is that DECO enables dense-comparable performance with reduced computation and storage for end-side deployment. The benchmark results on real hardware confirm that the "reduced computation" part is not merely theoretical β€” it translates to the Jetson AGX Orin, an actual edge device of the type the paper targets.

Evidence anchors: Table 4 (speedup numbers across six Spec-Bench tasks on two devices), and the kernel implementation details in Section 5 (CUTLASS-based, leveraging tensor cores, ReLU vs. TopK complexity advantage).

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The paper uses two evaluation datasets. For language modeling, it evaluates perplexity (PPL) on the C4 English validation set (Raffel et al., 2020). For downstream task performance, it evaluates average accuracy across a suite of commonsense reasoning benchmarks: PIQA (Bisk et al., 2020), SIQA (Sap et al., 2019), HellaSwag (Zellers et al., 2019), ARC-Easy, ARC-Challenge (Clark et al., 2018), WinoGrande (Sakaguchi et al., 2020), and LAMBADA (Paperno et al., 2016). All downstream evaluations are conducted using the LMEval harness (Gao et al., 2024). The training data itself is a diverse mixture including FineWeb (Penedo et al., 2024), Nemotron-CC (Su et al., 2025), the Pile (Gao et al., 2020), Wikipedia, and other collected raw corpora β€” covering web text, math, and code β€” with the mixing ratio "carefully tuned through experiments on small-scale models" (Appendix B).

  • Base model(s). All experiments use a custom LLaMA-style Transformer (Touvron et al., 2023) as the dense backbone. Four parameter scales are tested: Small (0.11B), Medium (0.24B), Large (0.53B), and XLarge (1.18B). These scales are deliberately modest β€” the paper targets end-side deployment where models must fit within constrained device memory. Across all scales, the dense baseline uses SwiGLU FFNs, attention, and embedding layers that are held identical within each parameter-matched group. The key hyperparameters per scale are detailed in Table 5: hidden dimensions (dh) range from 768 (Small) to 1792 (XLarge), layer counts (Nlayer) from 16 to 32, and training tokens (ntokens) from ~3.93B to ~48.3B β€” consistently approximately 40Γ— the parameter count. The learning rate follows a WSD schedule (Hu et al., 2024; Dubey et al., 2024) with 100 warmup steps and 1000 decay steps. All MoE variants, including DECO, use a dense FFN at the first layer and sparse MoE for all remaining layers (Appendix B, Table 5).

  • Metrics. Two primary metrics are reported: C4 validation perplexity (PPL, lower is better), measuring the model's language modeling quality on held-out web text, and average downstream task accuracy (Task, higher is better), computed as the macro-average of accuracy scores across the seven commonsense reasoning benchmarks listed above. Individual benchmark scores are provided in Appendix F (Tables 9–12) for full transparency. The paper treats both metrics as equally important β€” a model must match or exceed the dense baseline on both to be considered "dense-comparable."

  • Baselines. Five baselines are compared against DECO at every scale (Figure 3):

    • Dense: A standard LLaMA-style Transformer with SwiGLU FFNs, no sparsity. This is the primary target for "dense-comparable" performance.
    • TopP (Huang et al., 2024): An MoE architecture that selects experts by a cumulative probability threshold p, activating experts until their cumulative router score reaches at least p. This enables token-dependent activation ratios, like ReLU routing, but uses a different selection mechanism (Softmax-based thresholding rather than ReLU).
    • DeepSeek-V3 (Liu et al., 2024): A state-of-the-art TopK-routed MoE with gated MLP experts. Notably, the paper tests DeepSeek-V3 with both gated experts (the original design) and non-gated experts, and reports the best-performing variant β€” gated for DeepSeek-V3 β€” in the main Figure 3 comparison (Section 4.4, Table 3 confirms this choice is justified).
    • ReMoE (Wang et al., 2024b): A ReLU-based routing MoE with adaptive L1-norm sparsity regularization. The paper tests both gated and non-gated expert variants, and reports the best-performing variant β€” non-gated β€” in the main comparison.
    • BlockFFN (Song et al., 2025b): A ReLU-based routing MoE originally designed for chunk-level speculative decoding acceleration. For fair comparison, the paper replaces BlockFFN's original two-loss regularization (chunk sparsification + activation locality) with DECO's single-loss adaptive sparsity regularization, having verified that this "one-loss" variant outperforms the original "two-loss" design (Appendix B, Table 7: Small PPL 35.27 vs. 35.62; Medium PPL 28.16 vs. 28.65).

    All MoE baselines are configured with the same routed-expert activation ratio (~20% on training data), the same shared expert intermediate dimension, and closely matched expert parameter counts to ensure a controlled comparison.

  • Generation budget / compute accounting. The paper does not report inference-time generation metrics (tokens/sec, latency) in the main evaluation section β€” those appear only in Section 5 (Practical Inference Acceleration), which measures decoding speed. Instead, the primary comparison is architectural: all models within a scale group are trained on the same number of total parameters and the same number of training tokens. This is the paper's central methodological constraint: compute is measured by training tokens Γ— total parameters, and this product is held constant across all models at a given scale. This is stricter than prior MoE work (e.g., Li et al., 2025) which allowed MoE models to process more tokens under the same FLOPs budget because their per-token computation is lower.

  • Cross-validation / statistical protocol. The paper conducts repeated experiments to assess statistical reliability. For the Small and Medium scales, five independent runs with different random seeds are performed for both DECO and the Dense baseline (Appendix D, Table 6). Results are reported as mean Β± standard deviation. DECO (Small) achieves PPL 34.69Β±0.218 and Task 36.84Β±0.174, compared to Dense (Small) at 34.43Β±0.105 and 36.78Β±0.090. DECO (Medium) achieves PPL 27.53Β±0.276 and Task 39.13Β±0.217, compared to Dense (Medium) at 27.73Β±0.307 and 38.98Β±0.096. The paper notes that the narrow standard deviations confirm DECO does not suffer from high run-to-run variance, and that DECO "actually slightly outperforms [Dense] on the 'Medium' scale" β€” a finding that would not be evident from single-run results. No cross-validation or held-out strategy selection is described for hyperparameter tuning, which appears to be done via grid search on small-scale models followed by power-law extrapolation to larger scales (Appendix B).

Main Quantitative Results

Overall performance: DECO matches dense models at 20% activation across four scales

The headline result appears in Figure 3, which shows C4 validation perplexity and average downstream task accuracy for DECO and all five baselines at each of the four scales. At the target activation ratio of 20%, DECO achieves:

  • Small (0.11B): PPL 34.36 vs. Dense 34.40; Task 36.90 vs. Dense 36.80. DECO is statistically indistinguishable from Dense and slightly edges ahead on task accuracy (+0.10 percentage points, though the repeated runs in Table 6 show overlapping confidence intervals).
  • Medium (0.24B): PPL 27.74 vs. Dense 27.85; Task 39.18 vs. Dense 39.01. DECO outperforms Dense on both metrics, with the gap widening to +0.11 PPL and +0.17 points task accuracy. The repeated runs (Table 6) confirm this is reliable: DECO's mean PPL across five runs is 27.53Β±0.276 vs. Dense 27.73Β±0.307.
  • Large (0.53B): PPL 21.87 vs. Dense 21.93; Task 42.81 vs. Dense 42.80. DECO achieves virtual parity.
  • XLarge (1.18B): PPL 18.38 vs. Dense 18.41; Task 47.38 vs. Dense 46.98. DECO slightly outperforms Dense on PPL (+0.03) and task accuracy (+0.40 points) β€” the latter gap is the largest absolute advantage at any scale.

The pattern is striking: DECO does not merely approximate dense performance β€” it matches or slightly exceeds it at all four scales, while activating only 20% of its routed experts per token. This is the paper's central empirical claim.

DECO outperforms all existing MoE baselines under matched conditions

Figure 3 also reveals a clear performance hierarchy among MoE architectures at matched total parameters, training tokens, and activation ratio:

  • TopP is the weakest, with PPL 40.98 (Small), 33.55 (Medium), 26.60 (Large), and 22.22 (XLarge) β€” consistently 3–7 PPL points worse than DECO. Task accuracy follows the same pattern (34.71, 36.34, 38.67, 41.10).
  • BlockFFN is the next best, achieving PPL 35.27 (Small), 28.16 (Medium), 22.94 (Large), 19.28 (XLarge). It trails DECO by 0.9–1.0 PPL at Small/Medium and by 0.5–1.1 PPL at Large/XLarge.
  • ReMoE and DeepSeek-V3 are comparable to each other but both trail DECO. At Small: ReMoE PPL 35.13, DeepSeek-V3 PPL 35.09 vs. DECO 34.36. At Medium: ReMoE 28.99, DeepSeek-V3 28.19 vs. DECO 27.74. The gap narrows at larger scales but DECO retains a clear advantage (Large: 21.87 vs. 22.19/22.49; XLarge: 18.38 vs. 18.69/18.99).

The task accuracy rankings mirror the PPL rankings, with DECO achieving the highest score at every scale (Small: 36.90; Medium: 39.18; Large: 42.81; XLarge: 47.38).

This is significant because DeepSeek-V3 represents the state-of-the-art in TopK-routed MoE, and ReMoE represents the prior state-of-the-art in ReLU-routed MoE. DECO's consistent advantage β€” achieved with non-gated experts and NormSiLU, the two key architectural divergences from these baselines β€” validates that the paper's design choices provide gains beyond simply adopting ReLU routing.

Activation ratio scaling: DECO achieves dense parity at lower ratios as scale increases

Figure 8 evaluates DECO across a wide range of activation ratios (5% to 25%) at the Small and Medium scales:

  • Positive correlation with performance: Both scales show monotonic improvement in PPL as the activation ratio increases from 5% to 25%. This is expected β€” higher activation ratios mean more expert computation per token, providing greater representational capacity.
  • Scale-dependent parity thresholds: The Small setting reaches Dense-level PPL (~34.40) at approximately a 15% activation ratio, while the Medium setting achieves parity at only about 10%. This suggests that as the total model scale increases, DECO can match dense models with proportionally fewer active experts β€” the required activation ratio for dense-comparability decreases with model size. The paper explicitly flags this as a hypothesis requiring larger-scale verification: "further investigation on larger-scale models is required to validate this scaling trend" (Section 4.5).
  • Superiority at matched activation: At the standard 20% ratio, Medium DECO achieves PPL 27.74, which is below the Dense Medium PPL of 27.85 β€” DECO is not just matching but exceeding dense performance at this scale and ratio.

The practical implication: if this trend holds at larger scales, DECO could achieve dense-comparable performance at very low activation ratios (perhaps 5% or less) for multi-billion-parameter models, making the computational savings even more dramatic than the 5Γ— reduction implied by 20% activation.

Effect of MoE hyperparameters: shared expert size and expert granularity

Appendix A explores two additional MoE-specific hyperparameters that affect performance:

Shared expert size (Appendix A.1, Figure 9): Holding total parameters approximately constant, the paper varies the shared expert's intermediate dimension ds relative to the fixed routed expert dimension (de = 64). When ds is around 1–2Γ— de (64–128 for Small, 64–128 for Medium), DECO achieves dense-comparable PPL and is relatively insensitive to the exact shared expert size. However, when ds grows to 3–4Γ— de (192–256), PPL degrades significantly β€” for Small, from ~34.4 to ~34.9; for Medium, from ~27.7 to ~28.2. This degradation occurs because the fixed total parameter budget forces a tradeoff: a larger shared expert means fewer routed experts, and the loss of routed expert diversity (reduced Ne) harms performance more than the increased shared capacity helps. The paper concludes that "a single shared expert with a scale comparable to that of a routed expert appears to be optimal," corroborating findings by Tian et al. (2025).

Expert granularity (Appendix A.2, Figure 10): Granularity g is defined as g = 4 Β· d_h / d_e, which measures how many times the total expert parameters could fit into a single dense FFN of the same hidden dimension (a higher g means more, smaller experts). The paper evaluates g from approximately 40 to 250 for both Small and Medium scales:

  • For Medium, PPL improves monotonically as g increases above ~120. Finer-grained experts (more experts, each smaller) perform better, consistent with the "fine-grained MoE" findings of DeepSeek-MoE (Dai et al., 2024).
  • For Small, the trend is less pronounced β€” performance fluctuates near the dense baseline and shows reduced sensitivity to granularity. The paper hypothesizes that lower-capacity experts at this scale struggle to capture the "long-tail" data distribution or develop meaningful specialization.
  • At very low granularity (g < 60), both scales show performance near the dense baseline. For g > 60, DECO consistently achieves dense parity at both scales.

All main experiments use de = 64 for routed experts across all scales (Table 5), corresponding to granularities that increase with dh: 48 (Small), 64 (Medium), 80 (Large), 112 (XLarge) β€” all in the regime where dense parity is achievable per Figure 10.

Practical inference acceleration: 2.93Γ— speedup on edge hardware

Section 5 reports wall-clock decoding speeds on Spec-Bench (Xia et al., 2024), a benchmark covering six task categories: machine translation (MT), translation (Trans.), summarization (Summ.), question answering (QA), math, and retrieval-augmented generation (RAG). The evaluation uses two devices: an NVIDIA RTX 4090 (24GB, server-class) and a Jetson AGX Orin (64GB, edge device). Three configurations are tested:

  • Dense: Standard autoregressive decoding with no sparsity optimizations.
  • TopK: A TopK-routed MoE at 20% activation ratio, using the same acceleration framework (FR-Spec; Zhao et al., 2025b).
  • DECO: The proposed architecture with the custom CUTLASS-based acceleration kernel, also at 20% activation.

Results (Table 4):

  • RTX 4090: DECO achieves an average of 223.75 tokens/sec across tasks, compared to Dense at 87.28 tokens/sec β€” a 2.56Γ— speedup. TopK achieves 196.99 tokens/sec (2.26Γ— speedup over Dense). DECO outperforms TopK by 13.6% at the same activation ratio.
  • Jetson AGX Orin: DECO achieves 44.24 tokens/sec vs. Dense at 15.09 tokens/sec β€” a 2.93Γ— speedup. TopK achieves 40.04 tokens/sec (2.65Γ— speedup). DECO outperforms TopK by 10.5%.

The speedup is larger on the Jetson (2.93Γ— vs. 2.56Γ—), consistent with the paper's motivation: memory-access savings from sparse activation are relatively more impactful on bandwidth-constrained edge hardware. The DECO kernel's advantage over TopK at identical sparsity levels is attributed to the element-wise ReLU activation having lower computational complexity than the vector-wise sorting operations required by TopK.

The speedup is not uniform across tasks β€” it ranges from 2.86Γ— (Trans.) to 3.02Γ— (Math) on the Jetson, suggesting that the sparsity pattern varies with input characteristics, though the paper does not analyze this variation further.

Ablation Studies and Robustness Checks

Learnable expert-wise router scaling (Table 1): The paper compares three scaling designs within DECO: "Fixed" (constant scaling factor of 1.0, equivalent to no scaling), "Scalar" (a single learnable scalar coefficient shared by all experts, initialized to 0.1), and "DECO" (the full per-expert learnable vector Ξ±, initialized to 0.1 for all entries). On Small, Fixed achieves PPL 34.43 and Scalar 35.16 vs. DECO 34.36 β€” the scalar variant actually performs worse than no scaling, while the vectorized variant provides a modest improvement. On Medium, Fixed achieves 27.94, Scalar 27.92, and DECO 27.74 β€” the vectorized advantage (0.20 PPL over Fixed, 0.18 over Scalar) is consistent but not dramatic. The key finding is not that per-expert scaling provides enormous gains, but that a single shared scalar is insufficient β€” it underperforms even the no-scaling baseline at Small β€” because it cannot accommodate the heterogeneity in expert output scales (the paper reports a ~5Γ— range in output norms across experts in the first MoE layer). The vectorized scaling is a necessary design refinement, not a major performance driver.

Scaling factor initialization (Table 8): Sweeping the initialization value of Ξ± from 0.01 to 1.0 on DECO (Small) reveals non-monotonic behavior: PPL ranges from 34.26 (at 0.25) to 35.66 (at 0.025), with the best values clustered around 0.1–0.25. The paper selects 0.1 as the initialization for all main experiments. The performance variation (1.40 PPL range) is substantial, indicating sensitivity to this hyperparameter, though the paper does not discuss whether this sensitivity diminishes at larger scales.

NormSiLU component ablation (Table 2, Figures 4–6): The most comprehensive ablation evaluates four activation variants inside experts: full NormSiLU, "w/o Mean" (removes inter-expert mean normalization, keeps RMSNorm), "w/o RMS" (removes intra-expert RMSNorm, keeps mean normalization), and "SiLU" (vanilla SiLU, no normalization). Results are reported across three diagnostic dimensions tracked throughout training:

  • Final performance (Table 2): NormSiLU achieves Small PPL 34.36 and Task 36.90. Removing either normalization step degrades performance: "w/o Mean" yields PPL 34.57 (Task 36.85); "w/o RMS" yields PPL 35.77 (Task 36.78); "SiLU" yields PPL 35.69 (Task 36.46). The largest degradation comes from removing RMS normalization (1.41 PPL gap vs. NormSiLU), while removing mean normalization causes a smaller but meaningful 0.21 PPL gap. At Medium, the pattern holds: NormSiLU (27.74/39.18) > "w/o Mean" (27.77/38.88) > "w/o RMS" (27.97/39.04) > "SiLU" (28.05/38.85).

  • Activation ratio trajectory (Figure 4): "SiLU" and "w/o RMS" show a surging activation ratio that peaks at ~45–50% around training step 2000–4000, before the adaptive regularization pulls it back to ~20%. "w/o Mean" and NormSiLU show flat, stable trajectories near 20%. Conclusion: Intra-expert RMS normalization is the component that prevents the activation ratio surge.

  • Regularization coefficient trajectory (Figure 5): "SiLU" and "w/o RMS" require coefficients that rise to ~0.05–0.06 during training, compared to ~0.01–0.02 for "w/o Mean" and NormSiLU β€” a 3–5Γ— difference. Higher regularization directly competes with the language modeling loss, explaining the performance gap.

  • SiLU output magnitudes (Figure 6): "SiLU" and "w/o Mean" show vanishing output magnitudes that stabilize at ~0.2–0.4, while "w/o RMS" and NormSiLU maintain healthy magnitudes of ~0.6–0.8. Conclusion: Inter-expert mean normalization is the component that prevents vanishing SiLU activations, ensuring expert neurons remain well-utilized.

The key insight is that both normalization steps are necessary but address different problems: RMS normalization prevents router-level instability (surging activation ratio β†’ high regularization β†’ performance loss), while mean normalization prevents expert-internal under-utilization (vanishing activations β†’ wasted parameters β†’ performance loss). The two problems are coupled β€” vanishing activations in experts feed back to destabilize the router β€” and NormSiLU solves both simultaneously.

Expert gating compatibility (Table 3, Figure 7): The paper tests gated (GA) vs. non-gated (NG) MLP experts across three architectures: DeepSeek-V3 (TopK routing), ReMoE (ReLU routing), and DECO (ReLU routing + NormSiLU + expert scaling). The results are stark:

  • DeepSeek-V3: Gated vs. non-gated is nearly neutral. At Small, gated achieves PPL 35.09 vs. non-gated 35.16 (Task 36.31 vs. 36.73 β€” non-gated slightly better on task). At Medium, 28.19 vs. 28.28 (Task 38.82 vs. 38.47). At Large, 22.19 vs. 22.33. The small differences slightly favor gated for PPL and non-gated for task accuracy, but the overall picture is that TopK routing is insensitive to expert gating.

  • ReMoE: Non-gated is clearly superior. At Small, gated PPL 36.02 vs. non-gated 35.13 (Task 36.61 vs. 36.60). At Medium, 29.55 vs. 28.99 (Task 38.15 vs. 38.57). At Large, 22.78 vs. 22.49 (Task 42.27 vs. 42.48).

  • DECO: Non-gated is dramatically superior. At Small, gated PPL 39.77 vs. non-gated 34.36 β€” a 5.41 PPL gap. The gap persists at Medium (32.46 vs. 27.74) but narrows substantially at Large (22.13 vs. 21.87, only a 0.26 PPL gap). The Large-scale result suggests that the gating instability may partially resolve at larger scales, though non-gated remains preferable.

Figure 7 visualizes the mechanism: DECO (GA) shows a wildly unstable activation ratio that surges from ~20% to ~45% around step 5000 and requires aggressive regularization to control. DECO (NG) maintains a flat, stable trajectory at the target ratio. The paper's gradient-based explanation (multiplicative interactions in gated MLPs β†’ volatile output scales β†’ high-variance router gradients β†’ logit instability β†’ activation ratio surge) is consistent with the evidence, though not formally proven.

BlockFFN regularization comparison (Table 7): To ensure fair comparison, the paper replaces BlockFFN's original two-loss regularization (chunk sparsification + activation locality) with DECO's single-loss adaptive sparsity regularization. The justification: BlockFFN's original objectives target speculative decoding efficiency, not the pure performance-sparsity tradeoff that DECO addresses. Table 7 confirms that the "one-loss" variant outperforms the original "two-loss" at both Small (PPL 35.27 vs. 35.62, Task 36.53 vs. 36.23) and Medium (PPL 28.16 vs. 28.65, Task 38.44 vs. 38.18) scales at the same 20% target activation ratio. This ensures that BlockFFN's underperformance relative to DECO (Figure 3) is not due to suboptimal regularization but to architectural differences.

Repeated runs (Table 6): Five independent training runs with different random seeds for both DECO and Dense at Small and Medium scales. The narrow standard deviations (DECO Small PPL: Β±0.218; Dense Small PPL: Β±0.105; DECO Medium PPL: Β±0.276; Dense Medium PPL: Β±0.307) confirm that the results are not sensitive to random initialization or data ordering. The paper notes that DECO (Medium) achieves a lower average PPL than Dense (Medium) across the five runs (27.53 vs. 27.73), though the standard deviations overlap β€” so the claim of "slightly outperforming" Dense should be interpreted as a consistent trend rather than a statistically significant gap.

Training data heterogeneity (Section 6, reported in text only): The paper reports a subsidiary experiment on FineWeb (Penedo et al., 2024), a less heterogeneous dataset than the default mixture. Under this condition, DECO (Small) achieves PPL 30.14 vs. Dense (Small) 29.76 β€” a 0.38 PPL gap in Dense's favor, reversing the parity seen on the diverse mixture. However, DECO still exceeds Dense on task accuracy (38.65 vs. 38.03). The paper hypothesizes that "high-entropy, multi-domain datasets are inherently better suited for sparse MoE" because the model can activate specialized parameter subsets for different domains. This is presented as a discussion point rather than a full experiment, and no larger-scale or multi-dataset ablation is provided.

Critical Assessment

Claim 1: DECO matches dense model performance under identical total parameter budgets and training tokens. This claim β€” the paper's central "dense-comparability" thesis β€” is supported by Figure 3 at four scales, but the evidence is qualified in several important ways.

What the experiments actually demonstrate: At all four scales (0.11B–1.18B), the specific DECO configuration tested β€” 20% activation ratio, shared expert dimension 128, routed expert dimension 64, non-gated experts with NormSiLU, adaptive entropy regularization β€” achieves PPL and task accuracy that are statistically indistinguishable from or slightly better than the dense SwiGLU baseline. The repeated runs (Table 6) confirm this is not a single-seed artifact.

What the experiments do not demonstrate: (1) Scale extrapolation is unverified. The paper explicitly flags that the apparent trend toward requiring lower activation ratios at larger scales "requires further investigation" (Section 4.5). The largest model tested is 1.18B parameters β€” modest by contemporary standards. Whether DECO maintains dense parity at 7B, 13B, or 70B parameters is unknown. The architectural innovations might interact with scale in unanticipated ways (e.g., the gated-vs-non-gated gap partly closes from Small to Large in Table 3, suggesting some training instabilities may resolve at scale).

(2) The activation ratio matters substantially. Figure 8 shows that DECO at 20% achieves parity, but at 10% (Small) or 5% (Small), DECO underperforms Dense. The claim is therefore specific to the 20% activation ratio at the scales tested. The paper's suggestion that the required ratio decreases with scale is a hypothesis, not a demonstrated fact. If the required ratio stays at 20% for larger models, the computational advantage (5Γ— fewer active parameters) is fixed but the absolute active parameter count grows, which may erode the deployment advantage for large models.

(3) Dense-comparability is sensitive to data heterogeneity. The FineWeb experiment (Section 6) shows that on less diverse data, DECO (Small) trails Dense (Small) by 0.38 PPL, though it still wins on task accuracy. This is a single data point at one scale, but it raises the possibility that DECO's advantage is partly a function of the diverse training mixture, and that on homogeneous corpora (e.g., domain-specific text, code-only), the gap to dense might widen. The paper does not explore this dimension systematically.

(4) The dense baseline uses SwiGLU, but DECO uses NormSiLU with non-gated MLPs. Some of DECO's performance may come from the activation function difference rather than the MoE structure per se. A dense model using NormSiLU (instead of SwiGLU) is not evaluated, so we cannot separate how much of the parity is due to DECO's MoE architecture vs. its expert-internal design. If a dense NormSiLU model outperforms dense SwiGLU, then DECO's parity with SwiGLU-dense may understate the gap.

Claim 2: DECO outperforms established MoE baselines (DeepSeek-V3, ReMoE, BlockFFN, TopP). This claim is well-supported by Figure 3, with DECO achieving lower PPL and higher task accuracy than all baselines at all four scales. The margins are substantial at Small/Medium scales (0.73–0.93 PPL better than the next-best baseline) and narrower but consistent at Large/XLarge (0.32–0.61 PPL). The baselines are configured fairly: matched total parameters, matched training tokens, matched activation ratio, matched shared expert dimension, matched expert parameter counts.

Caveats: The BlockFFN baseline uses DECO's sparsity regularization rather than its original two-loss design β€” the paper justifies this with Table 7, showing the one-loss variant outperforms the original, but this means the comparison is not against the published BlockFFN architecture. The DeepSeek-V3 baseline does not include all DeepSeek-V3 innovations (e.g., auxiliary-loss-free load balancing, multi-token prediction) β€” it is DeepSeek-V3-style routing and expert design, not the full system. This is reasonable for an architectural comparison but means the claim is "DECO routing + expert design > DeepSeek-V3 routing + expert design" rather than "DECO > DeepSeek-V3."

Claim 3: DECO's sparsity translates to tangible inference speedups on edge hardware (2.93Γ— on Jetson AGX Orin). This claim is supported by Table 4, which shows measured decoding speeds on real hardware. The speedup is consistent across task types and is larger on the edge device (2.93Γ—) than on the server GPU (2.56Γ—), matching the paper's motivation.

Caveats and missing comparisons: (1) The speedup is measured relative to a dense model with no sparsity optimizations. A dense model could potentially benefit from similar kernel-level optimizations (though it has no sparsity to exploit). The paper compares DECO against "Dense" with "standard autoregressive decoding without sparsity optimizations" β€” this is a weaker dense baseline than a fully optimized dense inference pipeline would provide. (2) The comparison kernel (TopK at 20% activation) uses the same FR-Spec framework, but the dense baseline does not β€” it is unclear whether FR-Spec's optimizations for dense inference would close part of the gap. (3) The paper does not report memory usage or storage footprint on the Jetson β€” the speedup is computational only, and the total parameter count (which determines storage) is matched by construction, so the storage benefit is the 0Γ— vs. larger-MoE comparison rather than an absolute reduction vs. the dense model. (4) The kernel is evaluated on Spec-Bench, which tests decoding speed under specific prompt distributions β€” performance may vary on other workloads.

Missing experiments that would strengthen the evaluation:

  • Ablation at larger scales (3B–7B parameters). The convergence of the gated-vs-non-gated gap at Large and the hypothesis of decreasing required activation ratio both demand verification at scales where they would have practical impact.
  • Dense NormSiLU baseline. Disentangling the activation function effect from the MoE architecture effect requires a dense model with NormSiLU-activated FFNs (or at minimum, gated NormSiLU).
  • Ablation of the shared expert. How much does DECO's performance drop if the shared expert is removed? The shared expert is always active, contributing to all tokens β€” its presence means that even at 20% routed expert activation, the total active parameter count is somewhat higher than 20% of the FFN parameters. The paper does not quantify this.
  • Memory bandwidth and storage benchmarks on the Jetson. The paper motivates DECO with storage overhead and memory-access costs (Section 1) but only reports inference throughput β€” not model loading time, memory footprint during inference, or I/O latency when swapping experts.
  • Comparison with a compute-matched dense model. The paper matches total parameters and training tokens, but an MoE at 20% activation uses fewer FLOPs per token during both training and inference. A compute-matched comparison β€” where the dense model is trained on proportionally more tokens or uses wider layers to match the total FLOPs β€” would test whether DECO's efficiency is due to the MoE structure or simply to using fewer FLOPs per parameter. The paper explicitly rejects this comparison (Section 1: "We adopt a stricter setting that requires exactly the same number of training tokens"), which is a valid methodological choice, but it means the paper does not answer the question of whether DECO is more efficient than an iso-FLOP dense model.

Experimental strengths: The paper's experimental design has several genuine strengths. The repeated runs (Table 6) are rare in pretraining-scale MoE papers and add credibility. The consistent scaling across four model sizes enables trend analysis (even if extrapolation is uncertain). The side-by-side gated vs. non-gated comparison across three architectures is a clean, controlled experiment that isolates a specific interaction (router type Γ— expert gating). The real-hardware benchmarks on two devices address the deployment motivation directly. The dual-metric evaluation (PPL + downstream task accuracy) guards against the risk that language modeling improvements do not transfer to useful capabilities β€” though the paper does not evaluate on more challenging benchmarks (MMLU, GSM8K, HumanEval) that would test whether DECO's parity holds for reasoning and knowledge-intensive tasks beyond commonsense QA.

6. Limitations and Trade-offs

Limitation 1: Dense-comparability has been demonstrated only at modest scales (≀1.18B parameters), and scaling behavior β€” particularly the required activation ratio β€” is an extrapolation, not a verified fact

The assumption or constraint. The paper's central claim is that DECO matches dense model performance under matched total parameters and training tokens. This claim is demonstrated at four scales from 0.11B to 1.18B. However, Figure 8 reveals that the activation ratio required for dense parity decreases with scale β€” 15% for Small (0.11B) but only 10% for Medium (0.24B) β€” and the paper explicitly states that "further investigation on larger-scale models is required to validate this scaling trend" (Section 4.5). No experiments are conducted beyond 1.18B.

The consequence. A practitioner cannot assume that DECO will maintain dense-comparable performance at 7B, 13B, or 70B parameters β€” the scales where MoE architectures are most commonly deployed. The trend toward lower required activation ratios is encouraging, but it could plateau, reverse, or interact with other architectural choices (e.g., the gated-vs-non-gated gap partly closes from Small to Large in Table 3, suggesting some instabilities resolve with scale, but whether they fully resolve is unknown). If the required activation ratio does not continue to decrease, the computational advantage (5Γ— fewer active FFN parameters at 20% ratio) may not compensate for the architectural complexity and training instability risk at larger scales. Conversely, if the required ratio drops to 5% or lower, DECO would become dramatically more efficient β€” but the paper provides no evidence for or against this possibility.

What evidence exists in the paper. Figure 8 (the two-scale comparison of PPL vs. activation ratio) and the explicit caveat in Section 4.5. Table 6 provides repeated-run variance at Small and Medium, confirming that the single-scale results are reliable, but provides no information about larger scales. The convergence of the gated-vs-non-gated gap in Table 3 (5.41 PPL at Small, 0.26 at Large) is indirect evidence that training dynamics change with scale, but the direction and magnitude of further changes are unknown.

Mitigation status. The paper does not mitigate this limitation. It acknowledges it transparently (Section 4.5, Section 7 Limitations) and states that a larger product-level DECO model is under development (Limitations section), but no larger-scale results are presented. A practitioner considering DECO for a production-scale model should treat the scaling extrapolation as an unverified hypothesis requiring internal validation.

Limitation 2: The difficulty of achieving dense-comparability is sensitive to training data heterogeneity, and the mechanism is not characterized

The assumption or constraint. The paper's main experiments use a carefully tuned diverse data mixture including web text, code, math, and multiple curated corpora (Appendix B). Section 6 reports a subsidiary experiment on FineWeb, a less heterogeneous dataset, where DECO (Small) trails Dense (Small) in perplexity (30.14 vs. 29.76) β€” a 0.38 PPL gap that reverses the parity observed on the diverse mixture. The paper hypothesizes that "high-entropy, multi-domain datasets are inherently better suited for sparse MoE" because the model can route domain-specific tokens to specialized expert subsets, but provides no systematic analysis of which aspects of data diversity matter, how much diversity is needed, or whether the gap widens with scale.

The consequence. A practitioner deploying DECO on a domain-specific corpus (e.g., medical text, legal documents, a single programming language) cannot assume dense-comparable performance based on the paper's results. The FineWeb result suggests that on less diverse data, DECO may underperform an equivalently sized dense model β€” and the paper does not test whether increasing the activation ratio, adjusting the shared expert size, or other hyperparameter changes can recover parity. The hypothesis that diversity helps because it enables expert specialization is intuitively plausible, but it also implies that DECO's advantage over dense models may be fundamentally tied to data heterogeneity. If the deployment data is homogeneous, DECO may waste parameters on experts that never develop distinct specializations, reducing effective capacity relative to a dense model that allocates all parameters uniformly.

What evidence exists in the paper. Only a single data point: the FineWeb PPL comparison reported in Section 6 text (no figure, no table, no downstream task breakdown beyond the average, no Medium/Large-scale replication). Task accuracy on FineWeb-trained DECO is 38.65 vs. Dense 38.03 β€” DECO still wins on task despite trailing on PPL, which complicates the interpretation. The paper does not analyze why task accuracy and PPL diverge on FineWeb, nor does it test whether the diversity-dependence interacts with model scale.

Mitigation status. The paper acknowledges this as an open question requiring "more studies to verify this explanation" (Section 6) but does not conduct controlled experiments varying data diversity, expert count, or activation ratio to characterize the relationship. This is flagged as future work, not addressed in the current paper.

Limitation 3: DECO has only been evaluated on pretraining perplexity and commonsense reasoning benchmarks β€” supervised fine-tuning, reinforcement learning, and challenging reasoning tasks are entirely untested

The assumption or constraint. All experiments in the paper evaluate base pretrained models on language modeling perplexity (C4) and commonsense reasoning accuracy (PIQA, SIQA, HellaSwag, ARC, WinoGrande, LAMBADA). The paper explicitly acknowledges in the Limitations section that "we do not conduct experiments involving the supervised fine-tuning (SFT) or reinforcement learning (RL) stages" and that "MoE architectures may encounter unique challenges during post-training, such as RL instability resulting from fluctuating router activations (Zheng et al., 2025)." No experiments on more challenging benchmarks β€” MMLU, GSM8K, HumanEval, BBH β€” are reported.

The consequence. This is a significant gap for two reasons. First, pretraining metrics do not guarantee downstream capability transfer. A model that matches dense perplexity may still underperform on reasoning tasks if the MoE routing fails to compose knowledge across experts effectively, or if sparse activation introduces subtle representational pathologies that perplexity does not capture. The paper evaluates only on commonsense QA, which primarily tests shallow factual association rather than multi-step reasoning or knowledge integration. Second, post-training stability is a known MoE failure mode. The paper's own citation (Zheng et al., 2025) indicates that RL training can cause router activation patterns to fluctuate, potentially destabilizing the sparsity control that DECO's adaptive regularization maintains during pretraining. If DECO's architecture is sensitive to distribution shift between pretraining and SFT/RL data β€” analogous to how it is sensitive to data heterogeneity (Limitation 2) β€” the dense-comparability demonstrated in pretraining may not survive post-training. A practitioner intending to instruction-tune or RLHF-tune a DECO model has no evidence from this paper about whether the architecture will remain stable or maintain its performance advantage.

What evidence exists in the paper. None. The Limitations section flags this as a concern and states that the authors are "currently training a larger, product-level DECO model optimized for edge-device deployment" that will involve identifying and mitigating post-training issues, but no results are presented or promised for the current paper.

Mitigation status. No mitigation in the current paper. This is deferred entirely to future work, specifically a product-level model that is not described or evaluated here. A practitioner should treat DECO as a pretraining architecture whose post-training behavior is an open research question.

Limitation 4: The inference acceleration benchmarks omit critical deployment metrics β€” memory footprint, model loading time, and expert-swapping latency β€” that are central to the paper's end-side motivation

The assumption or constraint. The paper motivates DECO through the "ideal triangle" of end-side deployment: high performance, low computation cost, and small storage overhead (Section 1, Figure 1). The storage concern is specifically about the memory-access cost of loading MoE expert parameters: "large MoE models may incur high memory-access costs when transferring experts between GPU high-bandwidth memory and shared memory, or when moving offloaded parameters from disk/flash storage to GPU/NPU memory of end-side devices." However, the acceleration benchmarks (Section 5, Table 4) report only decoding throughput (tokens/sec) on two devices β€” an RTX 4090 and a Jetson AGX Orin. No measurements are reported for: total GPU memory consumption during inference, the time required to load the model from storage to device memory, the overhead of expert parameter swapping if the model does not fit entirely in device memory, or the latency of individual forward passes (as opposed to throughput over many tokens).

The consequence. The throughput numbers (2.93Γ— speedup on Jetson) demonstrate that DECO's sparse computation translates to faster token generation, but they do not validate the storage-overhead claim that distinguishes DECO from other MoE architectures. Since DECO is matched for total parameters with the dense baseline, the on-disk model size is identical β€” the storage advantage over dense models is zero by construction. The paper's storage argument is implicitly a comparison against larger MoE models (with more total parameters) that would not fit on the Jetson, but this comparison is never made explicit or measured. A practitioner evaluating DECO for edge deployment needs to know: does the model load fast enough for the use case? Does it fit in available memory alongside other applications? Does the sparse activation pattern cause irregular memory access that increases per-token latency variance? The paper provides none of this information.

Furthermore, the throughput comparison uses an optimized dense baseline that may not be fully representative. The "Dense" baseline in Table 4 uses "standard autoregressive decoding without sparsity optimizations," while the DECO kernel uses FR-Spec with CUTLASS tensor-core optimizations. The paper does not report what a fully optimized dense inference pipeline (using the same FR-Spec framework) would achieve, so the speedup numbers may partially reflect kernel engineering quality rather than architectural sparsity per se.

What evidence exists in the paper. Table 4 (decoding speeds) and Section 5 (kernel description). Memory usage, loading time, and latency are not measured. The paper does not claim to have measured these β€” it simply does not address them, despite making storage a central motivation.

Mitigation status. None. The paper treats throughput measurements as sufficient evidence for the end-side deployment value proposition, but they only address the "low computation cost" vertex of the ideal triangle. The "small storage overhead" vertex β€” the one that distinguishes DECO's matched-parameter approach from traditional MoE β€” is asserted rather than empirically validated. A practitioner should not assume that DECO's end-side deployment advantages over a dense model extend beyond throughput without conducting their own memory profiling.

Limitation 5: The dense-comparability claim is specific to SwiGLU-activated dense models β€” DECO uses NormSiLU with non-gated MLPs, and the paper does not disentangle the activation function improvement from the MoE architectural benefit

The assumption or constraint. The dense baseline in all experiments is a standard LLaMA-style Transformer using SwiGLU (SiLU-gated MLP) feed-forward layers. DECO's experts use NormSiLU-activated non-gated MLPs. These differ in two respects: the activation function (NormSiLU vs. vanilla SiLU in the gating path of SwiGLU) and the gating mechanism (no gate multiplication vs. element-wise gating). The paper demonstrates that NormSiLU provides substantial benefits over vanilla SiLU within DECO (Table 2: 1.33 PPL improvement from "SiLU" to "NormSiLU" at Small), but it never evaluates a dense model using NormSiLU or non-gated MLP layers. This means the paper cannot answer the question: how much of DECO's ability to match the dense baseline comes from the MoE architecture, and how much comes from simply having a better activation function?

The consequence. It is possible that a dense model with NormSiLU-activated FFNs would outperform the SwiGLU dense baseline, in which case DECO's "dense-comparability" is partly β€” or largely β€” an activation function improvement rather than an MoE achievement. If a dense NormSiLU model achieves, say, PPL 34.0 at Small (vs. 34.40 for SwiGLU-dense), then DECO's 34.36 would represent a 0.36 PPL penalty for the MoE architecture relative to the best possible dense model, rather than parity. The paper cannot rule this out. Conversely, if NormSiLU does not help dense models (perhaps because the gating in SwiGLU already provides some normalization effect), then DECO's parity is genuinely attributable to the MoE design. The missing ablation makes it impossible to attribute the performance correctly.

This matters for a practitioner choosing between architectures: if the goal is the best possible performance at a given parameter budget, and NormSiLU provides independent benefits, the optimal choice might be a dense NormSiLU model rather than a sparse DECO model β€” or a DECO model with an even lower activation ratio, since part of the parity comes from the activation function. Without the dense NormSiLU baseline, these tradeoffs cannot be evaluated.

What evidence exists in the paper. Table 2 shows that NormSiLU substantially outperforms vanilla SiLU within DECO. Table 3 shows that non-gated experts outperform gated experts for ReLU-based routing. But no experiment removes the MoE routing entirely while keeping NormSiLU non-gated MLPs, and no experiment puts NormSiLU into a dense SwiGLU-style architecture. The paper does not acknowledge this missing ablation.

Mitigation status. Not addressed. The paper implicitly treats the choice of activation function as part of the DECO architectural package rather than an independently variable component. This is a reasonable design choice β€” DECO is proposed as an integrated architecture, not a modular toolkit β€” but it means the paper's central claim of "MoE matches dense" conflates the routing structure with the activation function. Future work could resolve this by evaluating a dense NormSiLU baseline, but the current paper does not.

7. Implications and Future Directions

How This Work Changes the Landscape

DECO makes three contributions that shift how the MoE field should think about architecture design, though the shift is a reframing and new diagnostic rather than a paradigm overthrow.

The most consequential shift is methodological: DECO demonstrates that MoE-dense comparisons must control for total parameters and training tokens simultaneously, not just active parameters or FLOPs. Prior MoE research β€” including the closely related work by Li et al. (2025) that the paper explicitly contrasts against β€” operated under a looser equivalence: an MoE with fewer active parameters can train on more tokens under the same FLOPs budget, and the comparison against dense is made at matched FLOPs, not matched tokens. This is a reasonable metric for server-side deployment where total parameters are less constrained, but it obscures whether the MoE architecture itself is parameter-efficient β€” whether it extracts more performance per parameter than a dense model, or simply compensates for architectural inefficiency by training on more data. DECO's matched-parameter, matched-token constraint forces the architecture to earn its keep: every expert, every routing parameter, every normalization layer must contribute at least as much per parameter as the dense SwiGLU baseline they replace. Under this stricter test, most existing MoE designs fail β€” Figure 3 shows TopP, BlockFFN, ReMoE, and even DeepSeek-V3 all trailing the dense baseline by meaningful margins, while only DECO matches or exceeds it.

This reframing matters because it changes what "better than dense" means. The field has largely accepted that MoE models win on the FLOPs-matched axis β€” they have fewer active parameters per token, so they can be larger in total while costing less to run. This is true but limited: it tells you that MoE is good for scaling up total capacity within a compute budget. It does not tell you whether MoE is good for fitting a model into a fixed storage budget while maintaining performance β€” the exact question that edge deployment and memory-constrained inference raise. By showing that the answer is "yes, but only if you redesign the router and expert together," DECO opens a new axis for MoE evaluation: parameter efficiency under storage constraints. This is not a replacement for the FLOPs-matched axis but a complement, and future MoE papers will likely need to report both (matched-parameter and matched-FLOP comparisons against dense) to fully characterize their architectures.

The second shift is diagnostic: DECO identifies a specific, previously invisible coupling between router and expert design β€” the incompatibility of gated MLP experts with ReLU-based routing. This is not a general finding that gated experts are bad or that ReLU routing is fragile. It is a specific interaction: the multiplicative gating in SwiGLU produces volatile output scales that send destabilizing gradient signals to the router, and ReLU routing, unlike TopK, couples activation directly to logit thresholds, so logit instability directly destabilizes the activation ratio. Table 3 makes this interaction stark: for TopK routing (DeepSeek-V3), gated vs. non-gated is a 0.07 PPL difference at Small; for ReLU routing (DECO), it is a 5.41 PPL gulf. This is not a subtle second-order effect β€” it is a first-order failure mode that makes the architecture non-functional.

The diagnostic value extends beyond DECO. Any future routing mechanism that uses a threshold-based activation (not just ReLU, but any continuous function where the activation decision depends on crossing a learned threshold) will need to account for this interaction. The paper's hypothesized mechanism β€” multiplicative interactions in the expert producing high-variance gradients that destabilize threshold-based routing β€” provides a conceptual framework for diagnosing similar failures in other threshold-based or reinforcement-learning routing schemes. Conversely, the finding that TopK's hard selection masks this instability explains why the MoE field never identified it before: the dominant routing paradigm (TopK) was immunized against the problem by construction. This is a classic case of a failure mode that is invisible under the standard experimental conditions and only surfaces under a specific architectural combination β€” and once surfaced, it explains why some prior ReLU-based MoE attempts (the paper notes that ReMoE and BlockFFN did not systematically ablate gating, and ReMoE's default configuration likely suffered from this issue) showed unstable training dynamics that were attributed to other causes.

The third shift is conceptual: DECO connects MoE sparsity to the intrinsic activation sparsity of dense models, reframing MoE not as "adding experts" but as "making existing sparsity architecturally explicit." Section 6 cites studies showing that standard SwiGLU FFNs activate only ~30–40% of their neurons meaningfully per token. Under this lens, a dense model is already a sparse MoE β€” the SiLU gating acts as a soft router, and the parameter columns of the up/down projections act as single-neuron "experts." DECO's contribution is to replace this unstructured, fine-grained sparsity with structured, coarse-grained sparsity (entire expert sub-networks rather than individual neurons) that is compatible with hardware-efficient sparse computation (sectioned weight matrices, predictable memory access patterns). This reframing explains why dense-comparable performance is achievable in principle: DECO is not forcing a dense model to become sparse; it is reorganizing sparsity that already exists into a form that hardware can exploit. The activation ratio of 20% for DECO should be compared not to 100% for dense, but to the ~30–40% natural activation sparsity of the dense SwiGLU β€” from this perspective, DECO is achieving a 2Γ— improvement in structured sparsity (20% vs. ~35% effective utilization) while maintaining performance, which is a more modest but more mechanistically grounded claim than "5Γ— fewer active parameters."

This reframing also suggests that future improvements in MoE efficiency may come not from more sophisticated routing but from better alignment between the MoE's expert boundaries and the natural specialization patterns that emerge in dense models. If dense models already learn specialized parameter subsets for different input patterns, then an MoE router that learns to replicate those boundaries with minimal overhead could approach dense performance at very low activation ratios. This is a different research direction than the dominant MoE paradigm of "more experts, more sparsity, more capacity" β€” it suggests that the goal should be to match the specialization already present in dense models, not to create new specialization that dense models lack.

The work also partially reconciles a tension in the MoE literature about expert design. Since Shazeer (2020), SwiGLU has been treated as universally superior to non-gated MLPs β€” and DECO's own results confirm this for TopK routing (Table 3, DeepSeek-V3 rows) and for dense models more broadly. But DECO shows that this superiority is conditional on the routing mechanism: when routing is threshold-based, non-gated MLPs become superior because they avoid the gradient instability. This resolves the apparent contradiction between "SwiGLU is always better" and the empirical success of non-gated experts in some ReLU-MoE implementations (BlockFFN used non-gated experts, but for chunk-sparsity reasons, not stability reasons β€” DECO provides the mechanistic explanation for why non-gated experts were necessary even without the chunk-sparsity motivation). The field now has a clear conditional rule: gated experts for TopK and dense models; non-gated experts for threshold-based routing. This replaces a universal claim with a context-dependent one, which is a more accurate and more useful engineering principle.

Follow-Up Research This Work Enables

Scaling the dense-comparability claim to 7B+ parameters to determine whether the required activation ratio continues to decrease with scale. The most urgent open question from DECO is whether the trend in Figure 8 β€” where Medium reaches dense parity at 10% activation while Small requires 15% β€” continues to larger models. If a 7B DECO model matches a 7B dense model at, say, 5% activation, the practical implications are transformative: a 7B-parameter model with the inference cost of a 350M-parameter model. If the trend plateaus or reverses, DECO's advantage is bounded at the 5Γ— reduction already demonstrated. A strong follow-up would train DECO and dense baselines at 3B, 7B, and 13B parameters (matching total parameters and training tokens at each scale), sweep activation ratios from 5% to 25%, and fit a scaling law relating required activation ratio to total parameter count. The critical measurement is not just whether parity is achieved, but at what activation ratio β€” and whether that ratio follows a power-law decrease, a logarithmic plateau, or some other functional form. The paper's own caveat that "further investigation on larger-scale models is required" (Section 4.5) makes this the most direct and important extension.

Testing DECO through supervised fine-tuning and RLHF to determine whether the sparse routing remains stable under distribution shift. The paper explicitly flags post-training as an untested risk (Limitations), citing Zheng et al. (2025)'s finding that MoE architectures can experience router activation fluctuations during RL. A focused follow-up would take a pretrained DECO model (e.g., the Medium or Large checkpoints), perform standard instruction tuning on a dataset like Alpaca or OpenHermes, and then conduct RLHF or DPO alignment. The key measurements are: (1) Does the average activation ratio drift from the pretraining target during SFT? (2) Does the per-expert utilization balance degrade (some experts becoming "dead" while others are overloaded)? (3) Does the RL reward signal cause the adaptive sparsity regularization β€” which is not active during post-training β€” to be missed, leading to activation ratio collapse or surge? (4) Does the final chat model maintain dense-comparable performance on benchmarks like MT-Bench or AlpacaEval? A negative result β€” e.g., finding that the activation ratio surges to 40% during RL, destroying the inference efficiency advantage β€” would not invalidate DECO as a pretraining architecture but would mean that post-training stabilization techniques (perhaps a frozen router, or continued sparsity regularization during RL) are essential for practical deployment.

Replacing the shared expert with a learned gating mechanism to test whether DECO's always-on computation is necessary or can be made sparse. DECO's shared expert β€” a dense FFN that processes every token β€” is a design choice inherited from DeepSeek-V3. It ensures that even if a token activates zero routed experts (all router logits negative), the token still receives some computation. But the shared expert is always active, contributing to the per-token FLOPs regardless of the token's difficulty. A natural extension would replace the shared expert with a small pool of "shared experts" that are also sparsely routed (perhaps with a separate, independently regularized router), or with a learned gating mechanism that decides whether the shared expert is needed at all for a given token. The experiment would measure: (1) What fraction of tokens actually require the shared expert (i.e., would produce degenerate outputs without it)? (2) Can a sparse shared-expert pool maintain performance while reducing the always-on computation? (3) How does the optimal shared-expert size scale with total model size and activation ratio? The paper's findings on shared expert size (Appendix A.1, Figure 9) suggest that an oversized shared expert is harmful because it crowds out routed experts, but the lower bound β€” what is the minimum shared capacity needed β€” is unexplored.

Training a dense NormSiLU baseline to disentangle the activation function improvement from the MoE architectural benefit. This is the most important missing ablation from the current paper, as discussed in Limitation 5. A clean experiment would train three models at matched total parameters and tokens: (a) standard dense SwiGLU (the current Dense baseline), (b) dense NormSiLU with non-gated MLPs (identical FFN structure to DECO's experts but no routing β€” every token passes through all FFN parameters), and (c) DECO at 20% activation. If (b) significantly outperforms (a), then part of DECO's parity is an activation function gain that could be achieved without MoE β€” the optimal edge deployment might be a dense NormSiLU model, which avoids routing overhead entirely. If (b) and (a) are comparable, then DECO's parity is genuinely attributable to the MoE routing (the model is matching dense performance despite using fewer parameters per token, not just benefiting from a better activation). If (b) underperforms (a) β€” which is possible since SwiGLU's gating provides a form of input-dependent conditioning that non-gated MLPs lack β€” then DECO's achievement is even more impressive, since it overcomes an inherent disadvantage of non-gated experts through routing. The experiment would also clarify whether NormSiLU should be adopted broadly in dense architectures or is specifically beneficial in the MoE context.

Evaluating DECO on knowledge-intensive and reasoning benchmarks to determine whether dense-comparable performance extends beyond commonsense QA. The paper's downstream evaluation is limited to commonsense reasoning tasks (PIQA, SIQA, HellaSwag, ARC, WinoGrande, LAMBADA). These primarily test shallow factual association and simple inference β€” they do not stress the model's ability to compose knowledge across multiple facts, perform multi-step mathematical reasoning, or generate coherent long-form text. A critical stress test would evaluate DECO against the dense baseline on: (a) MMLU (57 subject areas testing factual knowledge), (b) GSM8K and MATH (multi-step mathematical reasoning), (c) HumanEval and MBPP (code generation), and (d) a long-context retrieval task like Needle-in-a-Haystack. The concern is that MoE routing might fragment knowledge β€” if facts about a topic are distributed across multiple experts, and the router only activates a subset, the model might fail to integrate information that a dense model would naturally combine. This is a known concern in MoE models (the "knowledge fragmentation" hypothesis) and DECO's design β€” with its shared expert providing a base of always-available knowledge and its ReLU routing allowing token-dependent expert combinations β€” might either mitigate or exacerbate it. A negative result on MMLU or GSM8K would substantially narrow DECO's applicability claims, even if commonsense QA parity is maintained.

Characterizing the diversity-dependence of DECO's performance through controlled data ablation. The FineWeb experiment in Section 6 is a single data point that raises more questions than it answers. A systematic follow-up would construct a continuum of training data diversity: from a single domain (e.g., Wikipedia only, or code only), through a two-domain mixture (Wikipedia + code), up to the full diverse mixture used in the paper. At each diversity level, train DECO and Dense at matched scale (Small or Medium being most practical) and measure the PPL gap between them. The hypothesis is that the gap narrows monotonically with diversity β€” but the shape of the curve matters. If the gap closes sharply after 2–3 domains, then DECO's advantage emerges with even modest diversity and is robust for most practical deployments. If the gap requires the full 5+ domain mixture to close, then DECO is only suitable for highly heterogeneous training distributions. Additionally, test whether increasing the number of experts (Ne) at fixed parameter count (higher granularity) improves performance on homogeneous data β€” the intuition being that more experts allow finer specialization even when the data diversity is low, since different patterns within a single domain might still benefit from routing. This experiment would directly address the paper's hypothesis that "high-entropy, multi-domain datasets are inherently better suited for sparse MoE" (Section 6) and provide practical guidance for when DECO can be expected to work.

Practical Applications and Downstream Use Cases

On-device language models for privacy-sensitive applications (healthcare, legal, personal assistants). The most direct application of DECO is deploying capable language models entirely on edge devices, avoiding cloud round-trips that expose sensitive data. The paper's headline numbers support this: on a Jetson AGX Orin (a representative edge device), DECO achieves 44.24 tokens/sec vs. 15.09 tokens/sec for an equivalent dense model β€” a 2.93Γ— speedup that brings the decoding rate from "usable but sluggish" to "comfortably interactive" for real-time applications. Since DECO matches dense performance under the same total parameter budget, the model that fits in the Jetson's 64GB memory delivers the same accuracy as a dense model that would run 3Γ— slower. For a healthcare application requiring HIPAA-compliant on-device processing of patient notes, or a legal application processing confidential contracts, this speedup is the difference between a viable product and an unacceptably slow one. The 20% activation ratio means the device's memory bandwidth β€” typically the bottleneck on edge hardware β€” is far less taxed, reducing both latency and energy consumption per query.

Cost-efficient batch inference for data processing pipelines (document classification, entity extraction, synthetic data generation). Organizations that run large-scale batch inference β€” processing millions of documents through an LLM for labeling, extraction, or data synthesis β€” care primarily about throughput per dollar. DECO's 2.56Γ— speedup on an RTX 4090 (223.75 vs. 87.28 tokens/sec) translates directly to processing 2.56Γ— more documents per GPU-hour at the same hardware cost, while matching the accuracy of a dense model with the same storage footprint. This is distinct from typical MoE cost savings, which come from using fewer FLOPs per token but often require more total GPU memory (to hold the larger expert pool). DECO's matched-parameter design means the GPU memory requirement is identical to the dense baseline β€” no need for higher-memory instances or expert offloading. For a company processing 100 million documents per month, a 2.56Γ— throughput improvement with no accuracy loss and no memory increase is a direct 60% reduction in GPU rental costs. The ReLU-based kernel's advantage over TopK (223.75 vs. 196.99 tokens/sec on RTX 4090, a 13.6% further gain) means DECO is not just matching dense efficiency but exceeding the efficiency of standard MoE approaches at the same sparsity level.

Edge-deployed coding assistants with low latency requirements. Coding assistants (like GitHub Copilot or local code completion tools) require low latency β€” the model must produce suggestions within a few hundred milliseconds to feel responsive. Running a 1B–3B parameter model on a developer's laptop GPU (or even CPU) with acceptable latency is challenging for dense models. DECO at a 1.18B total parameter scale (the paper's XLarge configuration) achieves dense-comparable performance at 20% activation, meaning the per-token computation is equivalent to a ~236M-parameter dense model. On a laptop with an integrated GPU or Apple Silicon Neural Engine, this could mean the difference between real-time code completion and noticeable lag. The storage footprint is the same as a 1.18B dense model (~2.4GB in FP16), which fits comfortably in most development machines' memory. While the paper doesn't benchmark on code-specific hardware or measure Time-To-First-Token (which matters more for code completion than average throughput), the 2.93Γ— Jetson speedup suggests that the latency reduction on similar-class integrated GPUs would be substantial. A production deployment would need to validate code generation quality specifically (HumanEval, MBPP scores), which the paper does not measure, but the architectural efficiency claim is directly transferable.

Federated learning or distributed fine-tuning on heterogeneous edge devices. In federated learning scenarios, many edge devices collaboratively train or fine-tune a shared model without centralizing data. The primary constraints are per-device computation, memory, and communication bandwidth. DECO's architecture offers an interesting property for this setting: different devices might naturally develop different expert specializations based on their local data distributions (e.g., a device used primarily for medical queries would route more tokens to medical-specialized experts). The ReLU-based routing, being differentiable and token-adaptive, might allow the router to learn device-specific activation patterns during local training, while the shared expert ensures a baseline of general capability. During federated aggregation, the router weights and expert weights could be averaged across devices, with the expert-wise scaling factors Ξ± naturally accommodating the heterogeneity of expert utilization across the fleet. This is speculative β€” the paper provides no federated learning experiments β€” but the architectural properties (sparse activation reducing per-device FLOPs, matched total parameters keeping storage constant, learned per-expert scaling handling output magnitude heterogeneity) align well with federated constraints. A practical deployment would need to test whether the adaptive sparsity regularization remains stable when applied independently on each device with different data distributions, and whether expert specialization persists or washes out during aggregation.

When to Prefer This Method

The paper positions DECO explicitly against two alternatives: (1) dense Transformers, and (2) conventional TopK-routed MoE architectures. The conditions for preferring DECO follow directly from the constraints the paper imposes and the results it demonstrates.

Prefer DECO over a dense Transformer when:

  • The deployment target is an edge device or memory-constrained environment where total parameter count is limited by storage, but per-token inference speed matters β€” DECO delivers 2.56–2.93Γ— speedup at matched total parameters and matched performance (Table 4).
  • The training data is heterogeneous (multiple domains: web text, code, math, etc.) β€” the FineWeb result (Section 6) suggests DECO's advantage over dense narrows or reverses on homogeneous data.
  • The model scale is modest (below ~2B parameters, within the range tested) β€” the scaling trend toward lower required activation ratios is a hypothesis, not a demonstrated fact, and at larger scales the architectural risk is uncharacterized.
  • You are willing to accept that post-training stability (SFT, RLHF) is unproven for DECO and may require additional stabilization techniques.

Prefer DECO over a TopK-routed MoE when:

  • You need token-adaptive computation (variable number of active experts per token) rather than fixed-K activation β€” ReLU routing provides this by construction, while TopK enforces uniform activation.
  • You can train with a differentiable sparsity objective rather than discrete load-balancing losses β€” DECO's adaptive entropy regularization may interfere less with the language modeling objective than auxiliary load-balancing losses, though the paper does not directly compare against DeepSeek-V3's full loss-free balancing system.
  • You are using ReLU-based routing (or any threshold-based routing) β€” if so, DECO's non-gated experts and NormSiLU are demonstrated necessities for stable training (Table 3, Figures 4–6), and using gated experts with ReLU routing will produce catastrophic activation instability.

Prefer a dense Transformer or TopK MoE over DECO when:

  • Post-training (SFT, RLHF) is a critical part of your pipeline, and the stability of DECO's routing under distribution shift has not been validated for your use case.
  • Your training data is homogeneous (single domain), where the paper's limited evidence suggests DECO may underperform dense models.
  • You need the absolute best possible performance at any cost and can scale total parameters aggressively β€” DECO is optimized for the constrained regime, and conventional MoE with massive expert pools may outperform it when total parameters are not limited.
  • You are deploying at very large scale (10B+ parameters) where the paper provides no evidence, and the risk of unanticipated scaling behavior is unacceptable without internal validation.